Tuning — the four causes, and the order to work in

When an agent is not performing, there are only four reasons, and they have
different fixes. Diagnosing before touching anything is the whole skill here.

The four causes

Content gap. The answer genuinely does not exist anywhere. The fix is to
create or connect the content.

Retrieval failure. The content exists but is never found. The fix is
structural: restructure it, add real vocabulary, remove duplicates, improve the
metadata.

Instruction failure. It finds the right thing and does the wrong thing with
it. The fix lives in agent instructions, topics and prompt actions.

Scope mismatch. People are asking for something it was never meant to do. The
fix is to extend the scope deliberately, or reset expectations — and that second
option is a legitimate answer, not a failure.

Tune in this order

Cheapest and most reversible first.

Instructions — free, instant, reversible.
Grounding — add or correct the content.
Retrieval configuration — chunking, metadata, source selection.
Topic design — make the critical paths explicit.
Model — try a different one before you tune one.
Fine-tuning — last, and it teaches form rather than facts.

The sequence matters because each step costs more and is harder to undo than the
one before it. Teams that start at the bottom of this list spend months proving
something they could have tested in an afternoon.

The most expensive mistake

A retrieval failure misdiagnosed as a content gap.

You commission new content that already exists. The duplicate then makes retrieval
worse, because there are now two similar documents for the retriever to choose
between. So the symptom intensifies, and the natural response is to commission
more content.

Always check whether the answer is already there and simply not being found.

The feedback loop

Fallback log → backlog → change → re-evaluate.

Two qualifications worth keeping. User feedback is a signal, not an instruction —
it tells you where to look, not what to do. And track whether a fix actually moved
the metric, because a change that felt right and changed nothing is common.

Using AI to tune AI

Cluster failed interactions to find themes. Draft candidate content from real
questions. And a human still approves what ships.

Exam trap. Adding content to fix a retrieval problem makes it worse — you
have added another near-duplicate for the retriever to choose between. The
diagnostic question is always: does this answer already exist somewhere the
agent can reach?

The one line to carry: find out whether the answer is missing or merely
unfindable, because the two look identical from the outside and have opposite
fixes.