Design a semantic search engine

30-second answer

• A central architectural choice is hybrid retrieval: merge BM25 and dense-vector results with Reciprocal Rank Fusion (RRF). Keyword search catches exact matches that embeddings miss, while embeddings catch synonyms that keywords miss.
• The query pipeline runs in under 200ms P95: embed the query (5ms), fan out to BM25 and vector search in parallel (~30ms each), fuse ranked lists via RRF (1ms), rerank top-50 to top-10 with a cross-encoder (20-30ms), return results.
• Fine-tuning a small embedding model (e5-small or bge-small) on domain-specific query-document pairs can improve recall over a generic large model at lower inference cost. Measure the gain and cost on the target query distribution before adopting it.
• Scaling to 100M documents requires IVF-PQ quantization, which compresses 600GB of raw vectors down to ~150GB with less than 5% recall loss, and horizontal sharding across a Qdrant or Weaviate cluster.
• Embedding model upgrades require re-indexing every document. Plan for this from day one with a blue-green index strategy so model improvements do not require an outage.

Requirements and assumptions

Functional requirements

• Users can submit natural language queries and receive the top-K most semantically relevant documents, ranked by relevance score.
• The system indexes documents from multiple sources (web pages, PDFs, internal knowledge bases) through an ingestion pipeline that chunks, embeds, and stores content.
• The system supports both semantic similarity search and exact keyword matching (hybrid search) so queries like "error code E-4012" return exact matches while "how to fix a leaky faucet" matches "plumbing repair tutorial for dripping tap."
• Users can apply metadata filters (date range, source, category, language) alongside semantic search to narrow results without sacrificing relevance.
• The system provides a relevance feedback mechanism where users can upvote or downvote results, feeding signal back into the ranking model.
• The system detects and handles multi-lingual queries, returning results in the same language or cross-lingually when requested.

Non-functional requirements

• P95 query latency under 200ms end-to-end (from query received to results returned).
• Support for 100M indexed documents with up to 10M new documents ingested per day.
• Query throughput: 1,000 queries per second sustained.
• Search relevance: NDCG@10 above 0.65 on the domain evaluation set, measured weekly.
• Cost: under $2,000/month for embedding inference and vector storage at 100M document scale.
• Availability: 99.9% uptime with zero-downtime index updates and model upgrades.

The hardest engineering problem here: embedding model upgrades. When you switch from text-embedding-3-small to a fine-tuned model, every single document must be re-embedded and re-indexed. At 100M documents, that is 3-5 days of continuous GPU processing. Without a blue-green index strategy, you either serve stale embeddings or suffer downtime.

Assumptions

• The document store is authoritative; the text index and vector index are derived artifacts with document and embedding-model versions.
• Search results can be evaluated offline with labeled query-document pairs and online with clicks or other feedback, but neither signal is treated as perfect ground truth.
• Freshness and relevance are separate goals. A small hot index may serve new documents while the larger index is optimized for stable query latency and recall.

5-minute approach

Start with the two workloads: asynchronous ingestion and low-latency query serving. Ingest versioned chunks into both a sparse index and a dense vector index, then fan out queries to both, fuse the rankings, rerank a small candidate set, and return filtered results.

• Choose chunk boundaries and metadata before choosing an embedding model; bad chunks cannot be repaired by a larger vector.
• Use BM25 for exact terms and dense retrieval for meaning, with metadata filters applied in both paths.
• Version embeddings and build a replacement index beside the active one so upgrades are an atomic serving decision.
• Measure relevance, freshness, latency, and index health separately; a fast irrelevant result is still a search failure.

The entities and endpoint below define the document, chunk, index, query, and feedback boundaries.

Core entities

Document

• doc_id, source (web/pdf/knowledge_base), raw_text, url, title, language, created_at, updated_at, metadata (category, author, tags)

Chunk

• chunk_id, doc_id, chunk_index, text, token_count, embedding_vector (float32[]), embedding_model_version, created_at

SearchQuery

• query_id, raw_query, query_embedding, filters (date_range, source, language), top_k, user_id, timestamp

SearchResult

• result_id, query_id, chunk_id, rank, rrf_score, dense_score, sparse_score, rerank_score, clicked (boolean), feedback (upvote/downvote/null)

EmbeddingModel

• model_id, model_name, dimensions, index_name, is_active, deployed_at, ndcg_score, total_docs_embedded

API design

POST /api/search - execute a hybrid search query

Request: {
  "query": "how to fix a leaky kitchen faucet",
  "top_k": 10,
  "filters": {
    "source": "knowledge_base",
    "language": "en",
    "date_after": "2025-01-01"
  },
  "include_scores": true
}
Response: {
  "query_id": "qry_sem_7x2k",
  "results": [
    {
      "chunk_id": "chk_48291",
      "doc_id": "doc_plumbing_guide",
      "title": "Plumbing Repair Tutorial: Dripping Taps",
      "snippet": "To repair a dripping tap, first turn off the water supply valve under the sink...",
      "scores": { "rrf": 0.89, "dense": 0.84, "sparse": 0.72, "rerank": 0.93 },
      "rank": 1
    },
    {
      "chunk_id": "chk_51002",
      "doc_id": "doc_home_repair",
      "title": "Home Repair: Kitchen Sink Issues",
      "snippet": "A leaking faucet wastes up to 3,000 gallons per year. Replace the O-ring...",
      "scores": { "rrf": 0.81, "dense": 0.79, "sparse": 0.68, "rerank": 0.87 },
      "rank": 2
    }
  ],
  "latency_ms": 142,
  "model_used": "bge-small-finetuned-v2"
}

The primary search endpoint. Embeds the query, runs parallel dense + sparse retrieval, fuses with RRF, reranks, and returns the final ranked list. Metadata filters are applied as pre-filters on the vector DB side.

POST /api/documents/ingest - ingest a document into the search index

Request: {
  "url": "https://example.com/plumbing-guide",
  "source": "web",
  "metadata": { "category": "home_repair", "language": "en" }
}
Response: {
  "doc_id": "doc_plumbing_guide",
  "chunks_created": 12,
  "total_tokens": 3840,
  "embedding_model": "bge-small-finetuned-v2",
  "status": "indexed",
  "latency_ms": 2400
}

Triggers the ingestion pipeline: fetch content, chunk, embed each chunk, and index in both the vector DB and BM25 index. Returns once indexing is complete.

POST /api/feedback - submit relevance feedback on a search result

Request: {
  "query_id": "qry_sem_7x2k",
  "chunk_id": "chk_48291",
  "feedback": "upvote"
}
Response: {
  "status": "recorded",
  "feedback_id": "fb_9x2m"
}

Captures user relevance signals for offline analysis. Aggregated feedback data feeds into periodic fine-tuning of the embedding model and reranker.

GET /api/documents/{doc_id} - retrieve a document and its chunks

Response: {
  "doc_id": "doc_plumbing_guide",
  "title": "Plumbing Repair Tutorial: Dripping Taps",
  "url": "https://example.com/plumbing-guide",
  "chunks": 12,
  "indexed_at": "2026-04-10T14:30:00Z",
  "embedding_model": "bge-small-finetuned-v2"
}

Administrative endpoint for inspecting indexed documents and verifying that ingestion completed correctly.

45-minute interview approach

Treat this as a search-system design question with an AI retrieval component. Spend the time on ranking quality, freshness, and index operations rather than only vector storage.

• 0–5 min — Clarify scope: ask corpus and document types, filters, languages, freshness SLA, query shape, feedback signals, and whether results are snippets or full documents.
• 5–10 min — Requirements and estimates: state document/chunk counts, ingest rate, QPS, P95 latency, NDCG target, storage budget, availability, and update cadence.
• 10–18 min — Interfaces and data model: define ingest, search, feedback, document, chunk, embedding-version, index, and ranking-feature records.
• 18–28 min — High-level design: draw ingestion queue, parser/chunker, embedding service, BM25 index, vector index, query fan-out, RRF fusion, reranker, and result API.
• 28–38 min — Deep dive: explain chunking, hybrid retrieval, model selection, blue-green reindexing, sharding/quantization, and relevance feedback.
• 38–42 min — Scale and operations: cover hot/cold indexes, backfill throughput, shard balance, cache behavior, model rollout, freshness lag, and cost.
• 42–45 min — Trade-offs and close: compare dense-only with hybrid, reranking with tail latency, and smaller vectors with recall; state how a stale or partial index behaves.

High-level design and data flow

A semantic search engine has two distinct pipelines that share an embedding model but serve different workloads. The ingestion pipeline (offline) processes documents in bulk: it fetches raw content, splits it into overlapping chunks of 256-512 tokens, embeds each chunk with the active model, and writes the resulting vectors to a vector database while simultaneously indexing the raw text in Elasticsearch for BM25 search. The query pipeline (online) handles user searches in real time: it embeds the query, fans out to both the vector DB and Elasticsearch in parallel, merges the ranked lists with RRF, and optionally reranks the top candidates with a cross-encoder.

The embedding model sits behind a dedicated inference service that serves both pipelines. During ingestion, it processes batches of chunks (throughput-optimized, 64-128 chunks per batch). During queries, it processes single queries with minimal latency (latency-optimized, 5ms per embed). Using one serving configuration for both workloads can make ingestion slow or query latency high, so separate the serving pools.

A metadata store (PostgreSQL) tracks documents, chunks, model versions, and feedback signals. This is the source of truth for what is indexed, which model produced each embedding, and which documents need re-embedding after a model upgrade.

Draw both pipelines from the start. Offline ingestion and online serving are different workloads with different optimization strategies.

Here is the query pipeline animated step by step. The key insight is the parallel fan-out: BM25 and vector search run simultaneously, and the total latency is the max of the two (not the sum).