Retrieval in RAG

Why retrieval is the hard part

In Retrieval-Augmented Generation, the LLM is rarely the bottleneck — retrieval is. If the right passage never enters the context window, no amount of prompt engineering or model upgrade will recover it. "Garbage in, garbage out" applies to context just as much as to training data.

• Generation quality is upper-bounded by retrieval quality.
• Most production RAG failures are recall failures, not reasoning failures.
• Most cost wins come from retrieving less, more precisely — not from cheaper models.

Anatomy of a retrieval pipeline

A modern retrieval stack has five stages. Each one is independently tunable, and weakness in any single stage caps end-to-end quality.

  ┌──────────┐   ┌──────────┐   ┌──────────┐   ┌──────────┐   ┌──────────┐
  │  Ingest  │──▶│  Chunk   │──▶│  Embed   │──▶│  Search  │──▶│  Rerank  │──▶ context
  └──────────┘   └──────────┘   └──────────┘   └──────────┘   └──────────┘
   parse/clean    split + meta   dense/sparse   ANN + filter   cross-encoder

Ingestion & parsing

Source documents come in messy: PDFs with multi-column layouts, HTML with nav chrome, Confluence exports with macros. Use layout-aware parsers (Unstructured, LlamaParse, Docling) to recover real structure — headings, tables, code blocks — before chunking. Strip boilerplate, deduplicate, and preserve canonical URLs as metadata.

Chunking strategy

Chunks are the unit of retrieval. Too small and you lose context; too large and embeddings become diluted and recall drops. There is no universal best size — measure it on your data.

• Recursive character splitting (~512–1024 tokens, ~10–15% overlap) is a strong default.
• Semantic / structural chunking respects headings, sentences, or code blocks — better for technical docs.
• Parent-document retrieval: embed small chunks, return the larger parent passage for generation.
• Always attach metadata to each chunk: source URL, title, section, timestamp, permissions.

Embeddings

Embedding models map text into a vector space where semantic similarity ≈ geometric proximity (usually cosine). The choice of model determines the ceiling of dense retrieval quality.

• Pick a model trained for retrieval (asymmetric query/document objectives), not for classification.
• Dimensionality (768 / 1024 / 1536 / 3072) trades quality for storage and latency.
• Matryoshka / MRL embeddings let you truncate vectors at query time for cheap coarse search.
• Re-embed when you change models — embeddings are NOT portable across model families.

from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import PGVector

embedder = OpenAIEmbeddings(model="text-embedding-3-small")
store = PGVector.from_documents(
    documents=chunks,
    embedding=embedder,
    collection_name="docs",
    connection_string=PG_URL,
)

Vector (dense) search

Approximate Nearest Neighbour indexes — HNSW, IVF, ScaNN, DiskANN — make similarity search sub-linear. Tune index parameters (M, efConstruction, efSearch for HNSW) to trade recall for latency. Always benchmark recall@k against an exact brute-force baseline on a held-out query set.

Keyword & hybrid search

Dense embeddings are weak on rare tokens, product SKUs, error codes, and exact phrases. BM25 (or a sparse model like SPLADE) handles those cases. Hybrid search runs both and fuses results — typically with Reciprocal Rank Fusion — to combine semantic recall with lexical precision.

# Reciprocal Rank Fusion across two retrievers
def rrf(rankings: list[list[str]], k: int = 60) -> list[str]:
    scores: dict[str, float] = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)

fused = rrf([dense.search(q, 20), bm25.search(q, 20)])

Metadata filtering & access control

Pre-filter by structured metadata before similarity search: tenant_id, language, document type, recency, ACL. Filtering at the index level is faster and safer than post-filtering — it also prevents leaking documents the user is not allowed to see, which is the most common RAG security bug.

Reranking

Retrievers optimise for recall — they cast a wide net. A cross-encoder reranker (Cohere Rerank, BGE-reranker, Voyage rerank) then re-scores the top-N candidates by jointly encoding (query, passage), promoting precision. The pattern: retrieve 50–100, rerank to top 5–10, pass those to the LLM.

Query transformations

User queries are noisy: short, ambiguous, conversational. Transforming the query before search routinely beats any embedding-model upgrade.

• Query rewriting — turn a follow-up like "and for Postgres?" into a standalone query.
• Multi-query — generate N paraphrases, retrieve for each, union the results.
• HyDE — have the LLM draft a hypothetical answer, then embed that to retrieve.
• Step-back prompting — retrieve for a broader, conceptual version of the question first.
• Decomposition — split a multi-hop question into sub-questions and retrieve per step.

Routing & multi-index

Real systems have multiple corpora (docs, tickets, code, FAQ). A router — either a small classifier or an LLM with structured output — picks the right index, or runs several in parallel and merges. Self-querying retrievers go further: the LLM emits both a semantic query and structured metadata filters from the natural-language question.

Context assembly

Once you have your top passages, assembly matters. Order by relevance or chronology, deduplicate near-identical chunks, include source attributions for citations, and respect the context budget — leave room for the system prompt, history, and the model's response.

• Beware of the "lost in the middle" effect: place the most important passages at the start or end.
• Contextual compression — summarise or extract only the relevant sentences from each passage.
• Always include a stable doc_id / URL per chunk so the model can cite sources.

Retrieval metrics

You cannot improve what you do not measure. Build a labelled eval set (50–500 query/expected-document pairs is enough to start) and track retrieval metrics independently from end-to-end answer quality.

• Recall@k — does the right document appear in the top k? (the most important retrieval metric)
• MRR / nDCG — rank-aware metrics that reward putting the right doc higher.
• Context precision — fraction of retrieved chunks actually used in the answer.
• Faithfulness — does the generated answer stay grounded in the retrieved context?

Common retrieval failure modes

Most RAG bugs reduce to one of these. Diagnose by inspecting traces in LangSmith / Phoenix / RAGAS before changing the model.

• Chunk-boundary loss — the answer spans two chunks and neither alone is retrieved.
• Vocabulary mismatch — query uses different terms than the source; fix with hybrid or query rewriting.
• Stale data — index not refreshed; add incremental sync and a freshness filter.
• Embedding/model drift — quality degrades after a model swap without re-embedding.
• Tenant leakage — missing ACL filter returns documents from another customer.