RAG architectures

TL;DR

• Naive RAG (chunk, embed, retrieve top-K, stuff into prompt) breaks on ambiguous queries and poor chunk boundaries. It's a starting point, not a destination.
• Advanced RAG fixes specific failure modes by adding query rewriting before retrieval and reranking after it.
• Modular RAG treats each stage as a swappable component. Any stage can be independently replaced, AB-tested, or skipped.
• HyDE (Hypothetical Document Embedding) solves vocabulary mismatch: generate a fake answer, embed it, retrieve against that embedding instead of the raw query.
• Reranking (retrieve a wider candidate set, then keep a smaller set) can improve precision for complex queries. The added latency and cost are worth it only when evaluation shows a meaningful benefit.

The Problem It Solves

Your knowledge base has the right information. The LLM still gives wrong answers. This is the core RAG frustration, and it usually means retrieval is failing, not generation.

Raw vector similarity search retrieves chunks that are semantically close to the query, but "close" is not the same as "relevant." A user asking "what's our refund policy for SaaS subscriptions?" might get chunks about refund accounting processes instead of customer-facing policy text. The embedding model doesn't understand the query's intent, just its surface structure.

RAG architectures exist to close this gap between syntactic similarity and actual relevance. Different architectures address different failure modes, so knowing which architecture to reach for depends on which failure mode you're seeing.

What Is It?

RAG (Retrieval-Augmented Generation) grounds an LLM's responses in external documents by retrieving relevant chunks at query time and including them in the prompt. The LLM generates an answer conditioned on those retrieved chunks rather than relying solely on its training data.

There are three useful levels of RAG architecture: naive RAG (the baseline pipeline), advanced RAG (targeted improvements at each stage), and modular RAG (a compositional framework where stages are replaceable). Each level addresses a different set of retrieval and operations concerns.

How It Works

Naive RAG

The baseline pipeline: split documents into chunks, embed each chunk, store in a vector database, embed the user query at query time, retrieve the top-K most similar chunks, stuff them into the prompt, generate.

This works well for simple factual retrieval over clean, well-structured documents. It breaks when queries are ambiguous, when the relevant information spans multiple chunks, or when chunk boundaries cut through conceptual units.

Advanced RAG

Advanced RAG adds three stages to the naive pipeline.

Pre-retrieval: Query rewriting transforms the user's query before embedding it. HyDE (Hypothetical Document Embedding) asks the LLM to generate a hypothetical answer, embeds that answer, and retrieves against it. This can help when the hypothetical answer shares vocabulary with the documents, but it adds a model call and should be evaluated against simpler rewrites.

Retrieval: Hybrid search combines dense vector search with sparse BM25 keyword matching. Dense retrieval handles semantic similarity; BM25 handles exact keyword matches (product names, IDs, technical terms). MMR (Maximal Marginal Relevance) diversifies the result set so you don't retrieve five nearly identical chunks.

Post-retrieval: Reranking uses a cross-encoder model that sees both the query and each candidate chunk together. A bi-encoder (used for initial retrieval) scores query and document independently; a cross-encoder scores them jointly, which is slower but can be more precise. A common pattern is to retrieve a wider candidate set with a fast bi-encoder and rerank a smaller set with a cross-encoder. Tune the candidate and final counts on representative queries.

Modular RAG

Modular RAG treats the pipeline as a graph of swappable components. Each stage (query transformation, retrieval, reranking, context assembly, generation) is an independent module with defined inputs and outputs. You can swap any module, AB-test two implementations of the same stage, or add new stages without touching the rest of the pipeline.

Frameworks such as LlamaIndex and LangChain support modular RAG. The main operational benefit is that clear interfaces between stages make each failure easier to isolate and test.

Context Assembly: The Underrated Stage

How you assemble retrieved chunks matters. LLMs pay more attention to content at the beginning and end of the context window than in the middle (the "lost in the middle" effect). Put the most relevant chunk first.

Remove duplicate chunks before assembly. Downstream from retrieval, you'll often get near-identical chunks from different document sections. Deduplicate by cosine similarity above 0.97 before building the prompt.

Include metadata (source URL, document date, section title) alongside each chunk. This gives the LLM enough information to assess recency and source authority, which reduces hallucination and allows the model to cite sources accurately.

Implementation Sketch

This is a simplified implementation showing the core advanced RAG pipeline. Production systems add error handling, caching, and observability around each stage.

async def advanced_rag_query(user_query: str) -> str:
    # Stage 1: Query rewriting with HyDE
    hypothetical_answer = await llm.generate(
        f"Write a detailed answer to: {user_query}",
        model="gpt-4o-mini"  # Fast model for HyDE
    )
    hyde_embedding = embed(hypothetical_answer)

    # Stage 2: Hybrid retrieval (dense + sparse)
    dense_results = vector_db.search(hyde_embedding, top_k=50)
    bm25_results = keyword_index.search(user_query, top_k=20)
    candidates = merge_and_deduplicate(dense_results, bm25_results)

    # Stage 3: Cross-encoder reranking
    scored = reranker.score(query=user_query, documents=candidates)
    top_chunks = sorted(scored, key=lambda x: x.score, reverse=True)[:5]

    # Stage 4: Context assembly (relevance-ordered, with metadata)
    context = "\n\n".join(
        f"[Source: {c.metadata['url']} | {c.metadata['date']}]\n{c.text}"
        for c in top_chunks
    )

    # Stage 5: Grounded generation
    return await llm.generate(
        f"Answer based ONLY on the provided context.\n\n"
        f"Context:\n{context}\n\nQuestion: {user_query}",
        model="gpt-4o"
    )

RAG Failure Diagnosis

When your RAG system gives bad answers, the failure is in exactly one of three places. This is the diagnostic framework I use.

• Empty or irrelevant retrieval: The retrieved chunks don't contain the answer. Fix: improve chunking strategy, add query rewriting or HyDE, or check that your embedding model handles the domain vocabulary.
• Right chunks, wrong answer: Retrieval is correct but the LLM still hallucinates or misinterprets. Fix: improve the generation prompt, add explicit instructions to stay grounded in the provided context, or add reranking to surface the most relevant chunk first.
• Inconsistent quality: Sometimes correct, sometimes wrong on the same question. Fix: add MMR for diversity in retrieval, increase top-K and add reranking, or check chunk boundary quality.

RAG vs Fine-Tuning

Use RAG when your knowledge changes frequently (product docs, pricing, policies). Use fine-tuning when you need the model to have a specific communication style, domain vocabulary, or format that doesn't change often. Most production systems need both: fine-tuned base model for style and tone, RAG for current factual knowledge.

When It Shines

• You need LLM answers grounded in proprietary or frequently updated documents
• You want to cite sources and reduce hallucination in factual domains
• You need to add knowledge without the cost and time of fine-tuning
• Your queries vary in complexity and a naive pipeline gives inconsistent quality

Real-World Usage

Perplexity AI runs a full advanced RAG pipeline: query rewriting, hybrid search across live web content and indexed documents, reranking, and context assembly with source citations. The reranking stage is what allows Perplexity to show high-precision answers for technical queries, not just semantically similar paragraphs.

GitHub Copilot's workspace feature uses modular RAG to retrieve relevant code context (open files, imported modules, recent edits) before generating suggestions. The context assembly stage trims to fit the context budget while prioritizing the most recently edited files.

Customer service chatbots at scale (Intercom, Zendesk AI) use naive RAG for straightforward FAQ retrieval and advanced RAG with reranking for complex multi-step support queries. The architecture is tiered by query complexity.

Failure Modes and Pitfalls

• Latency stack: Each advanced RAG stage adds latency. HyDE adds one LLM call, reranking adds 100-500ms. Profile your pipeline; the gains may not justify the cost for simple use cases.
• Chunk quality determines ceiling: No retrieval improvement compensates for bad chunking. Semantic chunking (split on meaning, not fixed character count) is worth the implementation cost.
• Threshold calibration is ongoing: The similarity threshold for retrieval and the reranking cutoff both require calibration against real query distributions. They drift as your content and user base evolve.
• Agentic RAG is powerful but unpredictable: Letting an agent decide when and what to retrieve is the most flexible pattern but hardest to debug. Reserve it for use cases where the query distribution is highly variable.

Trade-offs

Add stages one at a time and retain a baseline so each change has a measurable quality and latency impact.

Practical Checklist

Before shipping this pattern:

• Define the contract and scope. State the inputs, outputs, invariants, and use cases the pattern covers.
• Pin what you evaluate. Record prompt, model, retrieval, policy, and dependency versions so changes are attributable.
• Set explicit budgets. Bound latency, tokens, storage, retries, and any added model calls.
• Measure the trade-off. Compare quality, safety, cost, and latency with representative inputs, including adversarial and edge cases.
• Plan invalidation and rollback. Decide how stale or unsafe results are expired and how to return to a known-good version.
• Instrument and review. Log decisions and failure reasons with secrets and personal data redacted.

Test Your Understanding

Quick Recap

1. Naive RAG (chunk, embed, retrieve, generate) is a starting point. Most production use cases need at least one advanced RAG improvement to reach acceptable quality.
2. HyDE solves vocabulary mismatch by embedding a hypothetical answer instead of the raw query. It's the single highest-leverage improvement for queries that use different terminology than your documents.
3. Reranking (retrieve 50, rerank to 5 with a cross-encoder) delivers 15-30% accuracy improvement for complex queries at the cost of 100-500ms added latency.
4. Diagnose before improving: check whether the failure is in retrieval, generation, or context assembly. Each has a different fix.
5. Modular RAG treats each pipeline stage as a swappable component. Define clear interfaces between stages and each becomes independently testable and AB-testable.
6. Agentic search (tool-based iterative retrieval) is a viable alternative for smaller corpora with frequent changes, but vector RAG still wins for large-scale semantic retrieval.
7. The "lost in the middle" effect means chunk ordering matters. Put the most relevant chunk first in your assembled context.

Related Patterns

• Retrieval-augmented generation: The foundational concept that RAG architectures implement. Start here if you're new to grounding LLMs in external data.
• Vector databases for AI: ANN indexes, embedding storage, and similarity search internals are essential for debugging retrieval quality issues.
• LLM evals: Your RAG pipeline needs eval infrastructure to measure quality across stages. Evals tell you which stage is failing.
• Prompt management: The generation prompt determines how well the LLM uses retrieved context. Prompt versioning directly impacts RAG output quality.
• User feedback flywheel: Production RAG quality depends on continuous improvement driven by real user feedback signals.