AI Stack Docs

Retrieval

Vectorless RAG

Retrieval-augmented generation without embeddings, vector indexes, or ANN search. The LLM (and classical IR) does the heavy lifting of finding what matters in your corpus.

Why go vectorless?

  • No embedding model, no vector DB, no re-indexing on every model swap.
  • Citations point to real document locations (page, section, row), not opaque chunks.
  • Works great over small-to-mid corpora (≲ a few thousand docs) and structured data.
  • Avoids semantic drift, chunk-boundary loss, and "vector says similar but it isn't" failures.
  • Easier to debug — every retrieval step is a query you can read.

Core retrieval strategies

1. Lexical search (BM25 / full-text)

The workhorse of vectorless RAG. BM25 ranks documents by term frequency and inverse document frequency. Postgrestsvector, SQLite FTS5, Elasticsearch, OpenSearch, Tantivy, and Meilisearch all do this well.

sql
-- Postgres full-text retrieval
SELECT id, title, ts_rank_cd(search_tsv, query) AS score
FROM documents, plainto_tsquery('english', $1) AS query
WHERE search_tsv @@ query
ORDER BY score DESC
LIMIT 20;

2. LLM-as-router (agentic navigation)

Give the model a table of contents, file tree, or document index and let it pick which sections to open. Repeat until it has enough context to answer. This is how tools like PageIndex and file-search agents work.

ts
// Loop: model picks the next section to read
while (!model.hasEnoughContext()) {
  const choice = await llm.chooseSection({
    question,
    toc: docToc,
    alreadyRead,
  });
  alreadyRead.push(await loadSection(choice.id));
}
const answer = await llm.answer({ question, context: alreadyRead });

3. Structured filters & SQL

For anything tabular, generate SQL (or a typed query DSL) instead of embedding rows. Pair with column descriptions and few-shot examples. Text-to-SQL beats vector search whenever the answer depends on exact values, joins, or aggregates.

4. Metadata & faceted lookup

Filter by author, date, tag, product, jurisdiction, etc. before any ranking happens. A small, well-tagged corpus + filters often outperforms a large vector index.

5. Knowledge graph traversal

Entities and relations indexed in Neo4j, Memgraph, or even Postgres. The model issues Cypher / SQL hops to follow the graph instead of nearest-neighbor lookups.

A typical vectorless pipeline

question
   │
   ▼
┌─────────────────────────┐
│ Query rewrite (LLM)     │  expand acronyms, generate keywords
└─────────────────────────┘
   │
   ▼
┌─────────────────────────┐
│ Router (LLM)            │  pick: BM25 | SQL | TOC-walk | graph
└─────────────────────────┘
   │
   ▼
┌─────────────────────────┐
│ Execute query           │  Postgres FTS / SQL / file read
└─────────────────────────┘
   │
   ▼
┌─────────────────────────┐
│ Verify & refine (LLM)   │  enough context? if not, loop
└─────────────────────────┘
   │
   ▼
answer + citations (doc, page, row)

Query rewriting

Lexical retrieval lives and dies by the query. Use the LLM to:

  • Extract keywords and required entities from the question.
  • Generate synonyms and alternate phrasings (multi-query).
  • Decompose multi-hop questions into independent sub-queries.
  • Pick the right index/table to hit (routing).

Ranking & reranking

Pull a wide candidate set with BM25 (top 50–100), then have the LLM (or a cross-encoder used purely as a scorer) rerank the top results for relevance before they enter the answer prompt. No embeddings required — the reranker reads the text directly.

ts
const candidates = await bm25.search(query, { k: 50 });
const ranked = await llm.rerank({
  query,
  passages: candidates.map(c => c.text),
  topK: 8,
});

When vectorless is the right call

Pick vectorless when…

  • Corpus is small or highly structured.
  • Exact terms / IDs / numbers matter.
  • Auditability and citations are non-negotiable.
  • You want a single SQL/Postgres stack.
  • Documents change frequently.

Reach for vectors when…

  • Corpus is large (10k+ unstructured docs).
  • Users phrase questions very differently from source text.
  • Cross-lingual or multimodal retrieval is required.
  • Latency budget can't afford an LLM-in-the-loop router.

Failure modes to watch

  • Keyword mismatch: user says "revenue", docs say "turnover". Mitigate with query expansion.
  • Router loops: the LLM keeps requesting more sections. Cap iterations and token budget.
  • Stale TOCs: regenerate document indexes when sources change.
  • Long-tail recall: BM25 misses paraphrased answers — add a reranker or hybridize with embeddings later.

Tooling

  • Indexes: Postgres FTS, SQLite FTS5, Elasticsearch / OpenSearch, Meilisearch, Typesense, Tantivy.
  • Frameworks: LangChain BM25Retriever, LlamaIndex KeywordTableIndex & SummaryIndex, Haystack BM25Retriever, PageIndex.
  • Rerankers: Cohere Rerank, Voyage Rerank, or any chat model with a scoring prompt.