Drift and quality regression

Two things get run together in conversation and the exam separates them, so it is
worth being precise.

Drift is behaviour changing when you changed nothing. The model was updated
underneath you, the data moved, the users started asking different things.

Quality regression is a measurable drop against a baseline, usually traceable
to something you did — a new prompt, a retrieval change, a different model.

The second is the ordinary software problem. The first is the one that is specific
to this work, and it is the one people have no instrument for.

Why live traffic cannot see drift

Here is the thing that makes drift genuinely hard, and it is not obvious.

You cannot detect drift by watching production traffic, because the input moves
at the same time as the behaviour.

Yesterday's questions are not today's questions. Users change what they ask,
seasonally and in response to events and because your product changed. So when your
output distribution shifts, you have no way to tell whether the system got worse or
the questions got harder. Both look identical in aggregate.

Live monitoring can tell you that something changed. It structurally cannot tell
you what.

The only instrument that works

A fixed evaluation set, run on a schedule, unchanged.

Hold the input still and any movement in the output is real. That is the entire
logic, and it is why the design constraint is fixed — the moment you improve the
set, you have broken the comparison you were relying on.

Which has an uncomfortable consequence: it has to have been running before the
problem started. A baseline established after someone noticed the quality drop
tells you what the degraded system scores. It cannot tell you what you lost.

If you take one operational habit from this whole guide, it is worth making it this
one — get a modest fixed set running on a weekly schedule early, even a poor one.
A mediocre set with six months of history is worth more than an excellent set
created this morning.

What to watch alongside it

The evaluation set is the instrument. These are the signals that tell you where to
look when it moves:

Guardrail trigger rate. A sudden rise is either an attack or a change that
broke something legitimate — and both are things you want to know about within the
hour.

Refusal rate. Usually moves after a prompt or guardrail change, and usually in
the direction of over-caution.

Escalation rate. The agent finishing on its own less often.

Tokens per successful outcome. The general-purpose one. Almost every kind of
degradation shows up as more tokens for the same result.

The trap

"Quality has fallen but we haven't deployed anything" is the drift signature,
and it appears in scenarios precisely because the instinct is to reach for more
monitoring.

More monitoring of live traffic will not answer it. The answer is a fixed set on a
schedule — and if the scenario is testing whether you know that, it is also quietly
testing whether you know it needed to exist beforehand.