Retrieval augmented generation

TL;DR

RAG retrieves relevant document chunks at query time and injects them into the context window so the model answers from your data, not from training memory.
Two pipelines: ingestion (chunk, embed, store) and query (embed, retrieve, rerank, assemble, generate).
Hybrid retrieval (BM25 + semantic via Reciprocal Rank Fusion) is a strong baseline in many workloads, but its lift should be verified against a dense-only or sparse-only baseline.
Reranking with a cross-encoder can be high-leverage and adds latency and cost. Measure the lift on your corpus before making it a required stage.
The RAGAS framework measures faithfulness, answer relevancy, context precision, and context recall, giving you real metrics instead of vibes.
The "lost in the middle" problem means chunk ordering matters as much as chunk selection.

RAG retrieves evidence at request time and gives that evidence to the model before generation. The system therefore has two jobs: keep an accurate, authorized index, and retrieve a small set of useful context for each query. Better generation cannot compensate for missing documents or an incorrect access filter.

5-minute explanation

Ingestion normalizes documents, preserves metadata and permissions, chunks content, and indexes it. Query time embeds or tokenizes the question, retrieves candidates, optionally reranks them, assembles a bounded context, and asks the model to answer from that context. The answer path needs an explicit no-answer behavior and citations or evidence links when users need to verify claims.

Measure the stages separately: ingestion completeness and freshness, retrieval recall and precision, reranker lift, context use/faithfulness, answer quality, latency, and cost. Treat retrieved text as untrusted input: enforce authorization in the retrieval layer and defend against prompt injection in documents.

The problem it solves

Your company has 10,000 internal documents. An LLM trained through 2024 knows nothing about them. You could fine-tune the model on those documents, but fine-tuning is expensive, your documents update weekly, and fine-tuned models hallucinate about their training data almost as often as base models do.

You need the model to answer questions grounded in specific, current, private documents. Fine-tuning bakes knowledge into weights. RAG gives knowledge at read time.

The model doesn't memorize your data. It reads the relevant piece of it right before answering, every time.

What is it?

Retrieval Augmented Generation (RAG) is a pattern where, before calling the LLM, you retrieve the top-K most relevant documents (or document chunks) from a corpus and include them in the context window. The model then answers using those retrieved chunks as its primary source.

Think of it like a librarian. You don't memorize every book in the library. When someone asks a question, you walk to the right shelf, pull the relevant pages, read them, and then answer. RAG makes the LLM work the same way: retrieve first, then generate.

RAG was formalized in a 2020 paper from Meta AI (Lewis et al.) and is now a common pattern for grounding LLMs in private or frequently updated data. It can address training-cutoff problems and support citations, but it does not guarantee factuality when retrieval or generation fails.

How it works

There are two distinct pipelines. Ingestion runs offline (or on a schedule). Query runs at request time. Understanding which pipeline to optimize for which problem is the most important RAG debugging skill.

Ingestion pipeline

The ingestion pipeline runs whenever your data changes. For most teams, that means a nightly batch job plus an event-driven trigger for high-priority document updates. The pipeline's job: take raw documents, split them into retrieval-friendly chunks, compute embeddings, and write them to your vector store with metadata for filtering.

Teams can spend weeks optimizing retrieval when the actual problem is an ingestion pipeline that silently drops or mis-parses documents. Verify ingestion completeness and freshness before debugging ranking.

Query pipeline

Total latency budget for a production RAG call: 800ms to 1.5s. Query rewriting adds 100-200ms (optional but valuable). Embedding is fast (10-30ms). Parallel retrieval takes 30-80ms. Reranking is the hidden cost at 50-200ms. LLM generation dominates at 300-800ms.

Engineering implication: use this latency breakdown to set a budget and identify the stage to optimize; the values are starting assumptions, not a promise about every deployment.

Chunking strategies

How you split documents is the first place most RAG systems fail. Chunks that are too large cause the "lost in the middle" problem. Chunks that are too small lose context that spans sentence boundaries.

Fixed-size chunking (e.g., 512 tokens with 50-token overlap) is simple and works for uniform documents like API docs or FAQ pages. The overlap prevents hard splits at sentence boundaries. It's the default most teams start with, and that's fine.

Semantic chunking splits when the topic changes, detected by embedding similarity between consecutive sentences. It can improve retrieval precision on heterogeneous collections, but the lift depends on the corpus and it requires an embedding call per sentence during ingestion.

Parent-child chunking stores small chunks (256 tokens) for retrieval and large chunks (1,024 tokens, the parent) for context injection. You retrieve the small chunk for precision, then inject the larger parent for context. This is often a strong option for mixed document types, but the storage, token, and retrieval tradeoffs should be benchmarked.

Strategy	Chunk Size	Best For	Key Tradeoff
Fixed-size	256-512 tokens	Uniform docs, quick start	Splits mid-topic, loses context
Semantic	Variable (100-800 tokens)	Mixed document types	Slower ingestion, needs embedding per sentence
Parent-child	256 retrieval / 1,024 context	Production systems	More storage, index complexity
Sentence window	Single sentence + window	Precise Q&A	Very granular, high chunk count

Retrieval: sparse, dense, and hybrid

Sparse retrieval (BM25): keyword-based, TF-IDF variant. Fast, no embedding needed. Excellent at exact matches: product codes, error messages, specific terminology. Falls apart on paraphrases. "How do I log in?" won't match a document about "authentication procedures."

Dense retrieval: embed both the query and documents, find nearest neighbors in vector space. Catches semantic similarity across paraphrases. Misses exact terminology matches when vocabulary differs from training data. A query for "ERR_CONNECTION_RESET" might not retrieve the doc titled with that exact error code.

Hybrid retrieval (BM25 + dense): retrieve candidates from both, merge with Reciprocal Rank Fusion (RRF). BM25 catches exact terms that dense search can miss, while dense search catches paraphrases that BM25 can miss. This is a strong baseline in many systems, but measure the lift on the target corpus and query mix.

RRF is elegant: for each document, compute 1 / (k + rank_sparse) + 1 / (k + rank_dense) where k = 60 (standard constant). No tuning needed. Documents that rank well in both systems bubble to the top.

Engineering implication: explain the BM25-versus-semantic tradeoff and choose hybrid search when the evaluation shows enough lift to justify its cost.

Reranking

Most teams retrieve too many candidates (top-50) and then need to select the best 5-10 for the context window. Reranking is how you do that selection.

A cross-encoder model sees the query and a candidate chunk side-by-side and scores their relevance together. This is fundamentally different from bi-encoder retrieval, which encodes the query and chunk separately. Cross-encoders are 10-50x slower but significantly more precise because they see both texts simultaneously.

Run the cheaper bi-encoder retrieval first to get a bounded candidate set, then run the cross-encoder only on those candidates. This two-stage architecture limits reranking cost while improving precision when the reranker is well matched to the corpus.

Cohere Rerank, ColBERT v2, and BGE-Reranker are example options. Teams sometimes skip reranking to protect latency; whether that is a mistake depends on the measured retrieval quality and the task's latency budget.

Engineering tip: compare hybrid + rerank with a baseline

"I'd compare hybrid retrieval (BM25 + semantic via RRF) and a cross-encoder reranker on the top-50 candidates with a simpler baseline." The useful design signal is the measured quality, latency, and cost tradeoff.

Context assembly

Retrieval gives you chunks. Assembly determines what actually goes in the context window and in what order. This step is underrated and often ignored entirely.

The "lost in the middle" finding (Liu et al., 2023) showed that LLMs recall information at the start and end of long contexts better than in the middle. If you have 5 retrieved chunks, the most critical one should be first or last, not buried in position 3. Order your chunks by relevance: highest relevance at the edges, lowest in the middle.

Deduplicate aggressively. Parent-child chunking can return parent and child chunks that overlap significantly. Embedding similarity-based deduplication before context assembly prevents wasting tokens on near-duplicate content.

Token budgeting matters too. If your model has a 128K context window, don't fill it all with retrieved chunks. Leave room for the system prompt, grounding instructions, conversation history, and the model's own generation. Reserve a measured portion for retrieved context and tune it against context use and answer quality rather than treating 60-70% as universal.

Gotcha: retrieval success does not equal generation success

A chunk being retrieved doesn't mean the model will use it. If your system prompt has vague grounding instructions, the model will still hallucinate even with the right chunk present. Always include explicit instruction: "Answer only using the provided context. If the context doesn't contain the answer, say so."

RAG query pipeline (animated)

User Query

>Waiting for input...

Query Rewrite

>Ready...

Hybrid Retrieve

>BM25 + dense ready...

Rerank (Top-5)

>Cross-encoder loaded...

Assemble Context

>Token budget: 8K...

LLM Generate

>GPT-4o ready...

0/7

Press play to run the simulation, or drag the timeline to a step.

RAG query pipeline: from user question to grounded answer

Key variants / types

RAG has evolved through distinct generations. Understanding where your system sits in this progression helps you identify what to improve next.

Variant	Architecture	Key Feature	Best For	Limitation
Naive RAG	Retrieve top-K, concatenate, generate	Simple, fast to build	Prototypes, uniform docs	No reranking, poor on ambiguous queries
Advanced RAG	+ query rewriting, reranking, HyDE	Higher retrieval precision	Production Q&A systems	Added latency (100-300ms)
Modular RAG	Pluggable components per stage	Swap retrievers, rankers, generators	Teams with diverse doc types	Architecture complexity
Agentic RAG	Agent decides when and whether to retrieve	Conditional retrieval, multi-hop reasoning	Complex research queries	Unpredictable latency, harder to debug

Naive RAG is what most tutorials teach. Embed the query, retrieve top-5 chunks, stuff them into the prompt, call the LLM. It works for demos but breaks in production because it doesn't handle ambiguous queries, vocabulary mismatches, or noise in retrieved chunks.

Advanced RAG adds the components that make production systems work: query rewriting (expanding the user's query for better recall), HyDE (Hypothetical Document Embeddings, where you generate a hypothetical answer first and use that as the retrieval query), cross-encoder reranking, and structured context assembly. This is where most production systems should be.

Modular RAG treats each pipeline stage as a pluggable component. You might use BM25 for one document type and dense retrieval for another, or swap rerankers based on query type. It's the right architecture when you have heterogeneous data sources with different retrieval characteristics.

Agentic RAG gives an AI agent control over the retrieval process. The agent decides whether to retrieve at all, can issue multiple retrieval queries, synthesize results across queries, and decide when it has enough information to answer. This is the frontier, used in systems like Perplexity and multi-hop research assistants. The tradeoff: latency is unpredictable (the agent might make 1 or 5 retrieval calls), and debugging is significantly harder.

The honest recommendation: start with Advanced RAG. Ship it. Only move to Modular or Agentic when you have evidence that Advanced RAG's limitations are hitting you.

When to use / when to avoid

When to use RAG

You need to answer questions from private, proprietary, or frequently-updated documents
Your documents are too large to fit in a single context window
You need to cite sources for answers (compliance, legal, customer trust)
You're grounding a customer-facing assistant in a product knowledge base
Your data changes faster than you can retrain (weekly or more frequent updates)

When to avoid RAG

The task is a capability problem, not a knowledge problem (the model can't write code, not that it lacks data). Use fine-tuning instead.
Your data source is a live database or API with real-time freshness requirements. Use function calling to query it directly.
Your entire corpus fits in the context window (under 100K tokens). Just stuff it in. No retrieval needed.
The expected question types are narrow and predictable. A simpler lookup table or search might outperform the complexity of RAG.

RAG vs fine-tuning vs function calling

RAG adds knowledge at query time. Fine-tuning changes model behavior permanently. Function calling connects the model to live data sources. These are complements, not competitors. Many production systems use all three.

Real-world examples

Notion AI (2023): Notion's Q&A feature is an example of RAG over a user's workspace. Documents can be chunked and embedded at write time, then filtered and retrieved at query time. The example illustrates that document permissions and workspace metadata are part of the retrieval design.

GitHub Copilot Chat (2024): Copilot's chat mode is an example of retrieval over a codebase. Files can be chunked by function and class boundary, embedded, and stored per repository. Syntactic boundaries are a useful starting point for code search, but recall should be measured against fixed-size and other structure-aware strategies.

Elastic + OpenAI production stack: Elasticsearch can provide the BM25 layer in a hybrid RAG pipeline. Teams can combine sparse retrieval with a vector store or kNN search and merge the candidates with RRF. The example shows how a search platform can supply both lexical and dense retrieval; throughput and architecture depend on the deployment.

Perplexity (2024): An example of agentic RAG in which a system chooses search queries, retrieves from web and indexed sources, synthesizes across multiple retrievals, and generates answers with inline citations. Multi-hop retrieval can improve coverage while adding latency and cost; the acceptable tradeoff is product- and workload-specific.

Limitations and tradeoffs

Retrieval quality is a ceiling. If the right evidence is not retrieved, the model cannot ground its answer in it. RAG quality is bounded by retrieval and access filtering.
Latency chain. Query embedding + search + reranking + LLM call creates multiple hops. Reranking can add material latency; measure it in the target deployment.
Index maintenance. A stale index can be worse than no index. Documents that change need an ingestion and deletion strategy; incremental indexing is useful when full re-indexing is too expensive.
Long-tail queries. Highly specific queries about niche topics may not retrieve good chunks even with hybrid search. Fallback strategies (query rewriting, HyDE) help but add complexity and latency.
Evaluation is hard. Unlike classification tasks, there's no single accuracy number. You need to measure retrieval quality and generation quality separately, hence frameworks like RAGAS.

The fundamental tension: more retrieval stages can improve quality but add latency and operational cost. Every RAG system is navigating a precision-versus-speed tradeoff.

RAG evaluation with RAGAS

RAGAS (Retrieval Augmented Generation Assessment) is one framework for evaluating RAG pipelines. Its metrics help separate retrieval quality from generation quality:

Metric	What It Measures	Diagnoses
Faithfulness	Does the answer stick to the retrieved context?	Hallucination despite correct retrieval
Answer Relevancy	Does the answer address the question asked?	Off-topic generation
Context Precision	Are the retrieved chunks actually relevant?	Retrieval returning noise
Context Recall	Did retrieval find all the needed information?	Missing relevant chunks

RAGAS is useful for diagnostic separation. If faithfulness is low but context precision is high, investigate generation (grounding instructions, context dilution, or model behavior). If context recall is low, investigate retrieval (embeddings, missing chunks, or vocabulary mismatch). Confirm metric behavior with human review because automated metrics are proxies.

RAG failure diagnosis

When your RAG system produces bad answers, use this decision tree to pinpoint the root cause:

This flowchart is a useful debugging aid for RAG systems. For example, context precision of 0.85 with faithfulness of 0.6 suggests investigating grounding before changing retrieval, while still checking the metric definitions and a human-reviewed sample.

Practical checklist

Verify ingestion completeness, document versions, deletes/tombstones, freshness, provenance, and access metadata.
Enforce tenant and document permissions during retrieval; do not rely on the generation prompt to filter data.
Build a representative evaluation set with retrieval recall/precision, reranker lift, context use, faithfulness, answer quality, latency, and cost.
Choose chunk boundaries and overlap from document structure and measured retrieval behavior, not a universal token size.
Add hybrid search or reranking when the evaluation shows lift that justifies their latency and operating cost.
Define no-answer, conflicting-source, stale-source, and citation behavior explicitly.
Treat retrieved documents as untrusted prompt input; test prompt injection, data exfiltration, and citation mismatch.
Trace query rewrites, candidate IDs/scores, filters, final context, model/configuration version, and user-visible evidence.

Failure modes to watch

Missing, stale, duplicated, or incorrectly permissioned chunks produce a confident wrong answer.
Chunking splits the evidence or query rewriting drifts away from the user's intent.
Reranking adds latency without improving the answer, or context dilution causes the model to ignore the best chunk.
The model contradicts or cites context it did not use.
Retrieved text contains instructions that manipulate the generation step.

Optional: explaining this in a design review or interview

When to bring it up

RAG comes up in almost every AI system design interview. Any question involving "how would you build a Q&A system," "how do you ground the model in company data," or "how do you reduce hallucination" is a RAG question. Bring it up immediately when the problem involves private data, frequently-updated data, or source citation requirements.

Depth calibration

Junior/mid-level: Know the two pipelines (ingestion + query). Be able to draw the basic architecture. Know that RAG reduces hallucination by grounding in retrieved context.
Senior: Explain hybrid retrieval (BM25 + dense + RRF), cross-encoder reranking, and chunking strategies. Know the latency budget. Mention RAGAS for evaluation.
Staff/principal: Discuss Agentic RAG, multi-hop retrieval, incremental indexing at scale, evaluation pipelines, the precision-latency tradeoff, and when RAG is the wrong pattern entirely.

Common questions and strong answers

Interviewer asks	Strong answer
"How would you ground the LLM in our company docs?"	"RAG: chunk the docs, embed them, store in a vector DB. At query time, retrieve top-K relevant chunks via hybrid search and inject them into the context with a grounding prompt."
"How do you handle hallucination?"	"Explicit grounding instructions in the system prompt, cross-encoder reranking to surface the right chunks, and edge-first ordering to avoid the lost-in-the-middle problem."
"BM25 or vector search?"	"Both. Hybrid retrieval with RRF. BM25 catches exact matches that dense misses. Dense catches paraphrases that BM25 misses. The combination wins on every benchmark."
"How do you evaluate a RAG system?"	"RAGAS framework: faithfulness, answer relevancy, context precision, context recall. The four metrics separate retrieval problems from generation problems."
"What's the latency budget?"	"Query rewrite 100ms, embed 20ms, hybrid retrieval 50ms, reranking 100ms, LLM generation 500ms. Total p50 around 800ms. Reranking is the hidden cost most teams don't budget for."
"When would you NOT use RAG?"	"When the problem is capability, not knowledge (use fine-tuning). When data is live and real-time (use function calling). When the corpus fits in the context window (just stuff it in)."

Common interview mistakes

Mistake	Why it's wrong	Say this instead
"Just embed everything and do cosine similarity"	Ignores BM25, reranking, chunking strategy, and context assembly. This is naive RAG that fails in production.	"I'd use hybrid retrieval with BM25 and dense search, then rerank with a cross-encoder before context assembly."
"RAG eliminates hallucination"	RAG reduces but does not eliminate hallucination. The model can still ignore retrieved context or confabulate from training data.	"RAG reduces hallucination on factual questions by grounding in context, but you still need strong grounding instructions and evaluation."
Skipping the ingestion pipeline	Candidates jump straight to query-time architecture without discussing how documents get into the system.	"There are two pipelines. Ingestion handles chunking, embedding, and storage. Query handles retrieval, reranking, and generation."
"I'd use the biggest context window possible"	More context often hurts. The lost-in-the-middle problem means the model ignores middle chunks. A 128K window full of noise is worse than 5 precise chunks.	"I'd retrieve fewer, higher-quality chunks and order them edges-first. Precision beats volume."
Not mentioning evaluation	Candidates build the system but have no plan to measure whether it works.	"I'd set up RAGAS to measure faithfulness and context precision separately, so I can diagnose whether problems are retrieval or generation."

Test your understanding

Quick recap

RAG retrieves relevant document chunks at query time and injects them into the context window so the LLM answers from your data.
Two pipelines: ingestion (chunk, embed, store with metadata) and query (rewrite, embed, retrieve, rerank, assemble, generate).
Hybrid retrieval (BM25 + dense + RRF) with cross-encoder reranking is the production default. Don't skip either.
Parent-child chunking gives you retrieval precision (small chunks) and context richness (large parents). Start here for mixed document types.
The "lost in the middle" effect means chunk ordering matters. Put highest-relevance chunks at the start and end of the context block.
RAGAS separates retrieval problems (context precision/recall) from generation problems (faithfulness/answer relevancy). Use it to diagnose, not guess.
The fundamental tradeoff is precision vs latency. Every additional pipeline stage (rewriting, reranking, multi-hop) improves quality and adds milliseconds.

Vector databases for AI - The storage layer that makes RAG retrieval fast. Understand ANN search, HNSW, and metadata filtering.
Embeddings - RAG depends on embedding quality. Garbage embeddings mean garbage retrieval, regardless of everything else.
Context engineering - Context assembly in RAG is a subset of context engineering. The grounding prompt design directly determines faithfulness.
LLM evaluations - RAGAS is the RAG-specific evaluation framework, but it sits within the broader LLM evaluation ecosystem.