RAG: the levers, in the order to pull them

There is an order to improving retrieval, and getting the order wrong is the most
common way teams spend a month on RAG and end up roughly where they started.

The four levers, in order

Filter first. Metadata filters — current version, permitted for this user, right
product line — remove wrong answers before anything is scored. This is the
cheapest and most reliable lever and it is routinely skipped, because filtering
feels like plumbing and ranking feels like intelligence.

Then hybrid retrieval. Combine keyword and vector search. Vector search handles
paraphrase, which is what people usually want it for. Keyword search handles exact
terms — product codes, error numbers, proper nouns — which is precisely where pure
vector search is weakest, because "ERR-4402" and "ERR-4403" are extremely similar
in embedding space and completely different in meaning.

Then rerank. A second-stage model scores the shortlist properly rather than by
distance. It costs latency and money, and it is usually the single largest quality
gain available.

Then tune k. How many chunks to return.

Tuning k first is the classic mistake. Raise it and you add noise; lower it and
you lose recall. You are trading two problems against each other without fixing
either, and it feels like progress because the numbers move.

The symptom names the lever

This is the part worth memorising, because scenario questions give you a symptom.

The agent answers plausibly but from the wrong document. A filtering problem.
The right document may well have been retrieved — it just arrived alongside a
wrong one that was equally similar.

It misses material that definitely exists. A recall problem. Either the
chunking split the answer across a boundary, or the query needed keyword matching
that pure vector search did not provide.

The right chunk is retrieved but ranked fifth. A reranking problem. Retrieval
worked; ordering did not.

It answers from the old version of the policy. A metadata problem. Nothing on
the chunk said which version it was, so nothing could filter on it.

Chunking, briefly

A chunk is the unit that gets embedded, retrieved and injected. Too large and you
inject noise and pay for it; too small and you sever the context that made the
passage meaningful.

Split on structure, not character count — headings, sections, clauses.
Documents already carry the author's idea of where one thought ends, and a
character-count split ignores it. Add a little overlap so a sentence at a boundary
does not lose its antecedent.

And the one that fixes more problems than any tuning parameter: every chunk should
carry the document it came from, its version, its effective date and its
permissions. Then retrieval can filter rather than hope.

Most "our RAG is bad" problems are metadata problems, not embedding problems.

Embedding quality

Three things degrade it, and all three fail quietly.

Domain mismatch — a general model may not distinguish two terms your industry
treats as completely different.

Mixed content — embedding a chunk that is half prose and half table produces a
vector representing neither well.

Inconsistent preprocessing — if documents are cleaned one way at indexing and
queries another, you have built a systematic mismatch into the system.

And the self-inflicted one worth stating plainly: if you change embedding model,
re-embed everything. A mixed index where some vectors came from one model and
some from another produces distances that are meaningless across the boundary. It
does not throw an error. It just returns slightly worse results forever.

The trap

Groundedness does not check whether the right thing was retrieved.

An answer can score 0.91 on groundedness and be perfectly, faithfully supported by
the 2023 version of the policy instead of the 2026 one. The metric is doing exactly
what it claims — measuring whether the answer follows from the retrieved material —
and it has no opinion at all about whether that material should have been retrieved.

Which means retrieval has to be evaluated separately: for a set of known
questions, is the right document in the top k, and is it the current version?