The evaluation set and human review

Everything on the evaluation cards depends on having a set of cases with known-good
outcomes. This card is about building one that is worth having.

Three properties that matter more than size

Drawn from reality. Real questions users actually asked — not invented ones.
Invented sets are systematically too clean: they ask one thing at a time, they use
the right terminology, and they assume the user knows what they want. Real questions
do none of that.

Covering the tail. The ambiguous, the out-of-scope, the multi-part, and the ones
where the right answer is a refusal. The typical case is not where systems fail,
so a set made of typical cases measures the thing you were never worried about.

That refusal category is worth calling out. An evaluation set with no cases where
the correct behaviour is "I cannot help with that" cannot distinguish a helpful
system from an over-eager one.

Versioned, and grown from failure. Every production incident becomes a case.
That is how the set stops being a snapshot of what somebody thought hard about once
and becomes a record of how the system actually fails.

Why it is the most valuable thing the team builds

The evaluation set outlives the model, the prompt, and usually the framework.

Models get replaced. Prompts get rewritten. Someone migrates from one framework to
another. Through all of it, the set is what tells you whether the new thing is
better than the old thing — and without it, every change is a matter of opinion and
whoever is most confident wins.

Treat it as a first-class artefact in the repository, not a notebook on someone's
machine.

Human review: sample deliberately

The syllabus asks for a human review process, and the default instinct — sample
randomly, review a fixed percentage — is a poor use of expensive attention.

The disagreement set is where the learning is. Cases where the automated judge
and a human reviewer differ tell you either that your judge is miscalibrated or that
your rubric is wrong. Both are worth knowing. Random sampling mostly re-confirms the
cases that were already easy.

Two other priorities worth building in: low-confidence runs, and high-impact
ones — the cases where being wrong costs the most, regardless of how confident the
system was.

Components and end-to-end

Both, and for different reasons.

Component evaluation tells you where it broke. End-to-end tells you whether it
worked.

Teams that only run end-to-end know something is wrong and cannot localise it. Teams
that only run components end up with four green dashboards and an unhappy user,
because every part worked and the whole did not.

The trap

A criterion is not a criterion without three parts: a metric, a threshold, and a
named consequence.

"We measure groundedness" is a dashboard. "Groundedness below 0.8 blocks the
release" is a gate.

And the threshold has to be agreed before you see the number — because a threshold
chosen after the fact is not a standard, it is a description of where you currently
are, dressed up as one.