The metrics — and the dangerous one

A card about metrics, and about one metric in particular that will mislead you if
you read it alone.

What to measure, and what it actually tells you

Usage and sessions tell you about adoption. Watch out: high usage with low
resolution is churn, not success. People coming back repeatedly can mean the
thing is useful or that it never worked the first time.

Resolution rate tells you whether the user got what they came for. It needs a
definition everyone agrees on, or the number means nothing.

Escalation rate tells you how often a human is needed. And falling escalation
is not automatically good — it can equally mean people stopped bothering to ask.

Containment or deflection tells you how often it handled things alone. This is
the dangerous one, and it gets its own section below.

Latency tells you whether people will wait. Reasoning and extra hops both add
to it.

Token cost per interaction tells you what it costs at real volume. It is the
lever behind router and context discipline.

The dangerous one

Containment rises when the agent is good — and also when users give up.

Those two produce the same number. A team optimising containment without watching
anything else can drive it upward while the experience quietly gets worse, and
the dashboard will congratulate them the whole way.

Read metrics in pairs

That is the defence, and it generalises well beyond this exam.

Containment with satisfaction. Escalation with resolution. Usage with repeat
questions.

A single number can always be gamed by a failure. Pair it with something that
moves in the opposite direction when things go wrong.

Signals that point at content, versus retrieval

Content problems show up as the same question appearing in the fallback log
repeatedly, users rephrasing two or three times, or high abandonment on one topic.
The material is missing or wrong.

Retrieval problems look different: the content exists but is never cited, the
wrong document comes back confidently, or answers are correct for a neighbouring
topic. The material is there and cannot be found properly.

Telling those apart saves a great deal of wasted effort, because they have
completely different fixes.

Exam trap. Containment rate on its own is the trap metric. It goes up when
the agent is good — and also when users give up and stop escalating. Any answer
that optimises containment without a satisfaction or resolution counterpart is
the wrong one.

The one line to carry: every metric can be improved by a failure, so read them
in pairs.