Design a RAG chatbot

30-second answer

• Two pipelines drive everything: an async ingestion pipeline (chunk, embed, store) and a real-time query pipeline (retrieve, rerank, assemble, generate).
• Hybrid retrieval (BM25 + semantic, fused with Reciprocal Rank Fusion) outperforms either alone, especially for exact product names and error codes.
• Add a reranker after retrieval to cut from top-50 to top-5 chunks before the LLM call. The 15-30ms extra latency buys a 20-30% accuracy gain.
• Guardrails are not optional: a citation check verifies structure, and a RAGAS faithfulness score gates hallucinations before the response ships.
• A semantic cache for common questions hits 60-70% on FAQ traffic, collapsing both latency and LLM spend.

Requirements and assumptions

Functional requirements

• Users ask natural-language questions about the company's product documentation (50K documents).
• The system retrieves relevant content and returns a grounded answer with source citations.
• Responses arrive in under 3 seconds end-to-end, including LLM generation.
• New or updated documents are available for retrieval within 10 minutes of ingestion.
• The system refuses out-of-scope questions rather than hallucinating an answer.

Non-functional requirements

• 100K daily active users with peak concurrency around 10K simultaneous requests.
• P95 latency under 3 seconds; P99 under 5 seconds.
• Zero-hallucination SLA: no response claims a feature exists unless that claim is supported by a cited source.
• Embedding model upgrades require full re-embedding of the corpus without user-visible downtime.
• Cost target: under $0.02 per query at scale.

Assumptions

• Product documentation is the source of truth and has stable document IDs, versions, ownership, and update timestamps.
• Retrieval may return no trustworthy evidence. The generator must refuse or ask a clarifying question when the evidence does not support an answer.
• Ingestion and query traffic are independent workloads. An embedding-model migration may run beside the serving index without changing the active index until validation is complete.

5-minute approach

Draw two pipelines. The ingestion path parses, chunks, embeds, and indexes versioned documents; the query path performs hybrid retrieval, reranks the best candidates, and generates a cited answer only from the retrieved evidence.

• Combine keyword search for exact identifiers with dense retrieval for paraphrases, then fuse and rerank results.
• Put document version, tenant, language, and access-control filters into retrieval rather than relying on the model to enforce them.
• Make freshness, embedding versions, and deletion propagation observable; use a blue-green index for upgrades.
• Gate the response on evidence coverage and return a useful “not found” path when retrieval is weak.

The entities and endpoint below define the document, chunk, index, and answer contract.

Core entities

Document (source of truth, ingestion time)

• id, source_url, title, raw_text, metadata (product area, version), updated_at, content_hash

Chunk (derived at ingestion)

• chunk_id, document_id, text (200 tokens), embedding (1536-dim float array), parent_chunk_id

Query

• query_id, user_id, question_text, session_id, created_at

Response

• response_id, query_id, answer_text, source_chunk_ids[], faithfulness_score, latency_ms

EvalResult (offline quality tracking)

• eval_id, response_id, faithfulness, context_relevance, answer_relevancy, human_label

API design

POST /api/chat (main query endpoint)

Request:  { "question": "Does the product support SSO?", "session_id": "abc123" }
Response: { "answer": "Yes, SSO via SAML 2.0 ...", "sources": [{"url": "...", "title": "..."}] }

POST /api/ingest (trigger document ingestion)

Request:  { "document_url": "https://docs.example.com/sso", "priority": "normal" }
Response: { "job_id": "job_456", "status": "queued" }

GET /api/ingest/status/{job_id\d}

Response: { "job_id": "job_456", "status": "complete", "chunks_created": 24, "duration_ms": 3200 }

GET /api/health

Response: { "status": "ok", "vector_db": "healthy", "llm_provider": "healthy", "p95_latency_ms": 2100 }

45-minute interview approach

Keep the discussion on designing a grounded retrieval-and-generation system, with search quality and freshness treated as first-class requirements.

• 0–5 min — Clarify scope: ask corpus size, document types, access-control rules, update frequency, citation expectations, supported languages, and out-of-scope behavior.
• 5–10 min — Requirements and estimates: state users and QPS, corpus/chunk counts, freshness window, P95 latency, cost per query, availability, and hallucination tolerance.
• 10–18 min — Interfaces and data model: define ingestion, search, answer, document-version, chunk, embedding, citation, and feedback records.
• 18–28 min — High-level design: draw ingestion queue and index, query embedding, BM25/vector fan-out, fusion, reranking, prompt assembly, LLM, and faithfulness gate.
• 28–38 min — Deep dive: explain chunking, hybrid retrieval, embedding upgrades, cache safety, access filters, and how unsupported claims are refused.
• 38–42 min — Scale and operations: cover hot/cold indexes, sharding, backfills, queue lag, model rollout, relevance monitoring, and cost controls.
• 42–45 min — Trade-offs and close: compare larger chunks with context, reranking with latency, and freshness with indexing cost; state the outage and no-evidence behavior.

High-level design and data flow

Two completely separate pipelines share a single vector database. The ingestion pipeline runs asynchronously and can process thousands of documents per hour without touching the query path. The query pipeline is the hot path where every user request flows in real time.

The ingestion pipeline starts with a document loader that fetches raw HTML/Markdown from the docs system, strips navigation chrome, and passes clean text to the chunker. Chunking uses a sentence-aware splitter at 200 tokens with 20-token overlap to avoid cutting ideas mid-sentence. A parent-child strategy stores both a 200-token chunk (retrieved) and its 1,000-token parent document (sent to the LLM), giving precise retrieval without losing context.

The query pipeline is where the latency budget matters most. Embedding the user's question (15ms), hybrid retrieval (25ms), reranking top-50 to top-5 (25ms), and the LLM call (600-1,500ms) are the four dominant costs. You hit under 3 seconds by using a fast LLM (GPT-4o-mini or Claude Haiku) and parallelising steps where the dependency graph allows it.