The evaluator families, and what each measures

The dashboard said the agent was performing well. Groundedness 0.89, relevance
0.91, no safety flags. The support team said it was useless.

Both were right, and the gap between them is this card.

The evaluators were scoring answers. The agent's actual job was to resolve
cases — and it was producing beautifully grounded, impeccably relevant answers to
questions the customer had not asked, then failing to call the tool that would have
closed the ticket.

Nobody had evaluated the agent. They had evaluated its prose.

The four families

General-purpose quality — coherence, fluency. Measures the writing.

RAG-specific — groundedness, relevance. Measures the answer against what was
retrieved.

Safety and security — hate and unfairness, violence, protected material.

Agent-specific — eleven of them, covered on the next card. These are the ones
that measure whether it did the work.

Plus custom evaluators "tailored to your domain-specific requirements", for
the rule that only you have.

The agent-specific family is the one candidates forget exists, and it is the one
that answers the story above.

The four components the syllabus names

Here is a detail worth noticing. The syllabus asks you to "design and implement
evaluations for memory, knowledge, tools, and prompts".

Read that list again. Answer quality is not on it.

Memory — does it recall what it should, and not recall what it should not?
Two failure directions, and the second is a security test as much as a quality one.

Knowledge — is retrieval returning the right material? Evaluate this separately
from answer quality, because for a set of known questions the real questions are: is
the right document in the top k, and is it the current version?

Tools — did it call the right tool, with the right arguments, at the right time?
The failure modes are distinct: wrong tool, right tool with bad arguments, a call it
did not need, and a call it should have made and did not.

Prompts — does a change improve things overall? This is regression evaluation,
and it is what turns prompt engineering from superstition into evidence.

Evaluate components and end-to-end

Both, not either.

Component evaluation tells you where it broke. End-to-end tells you whether it
worked.

Teams that only do end-to-end know something is wrong and cannot localise it. Teams
that only do components end up with four green dashboards and an unhappy user —
which is exactly the situation this card opened with.

The tooling

Foundry benchmark compares models on public data or your own.

Azure AI Evaluation SDK tests specific model endpoints.

And evaluation is not a phase — Microsoft describes three stages across the
lifecycle, ending in "automated quality gates into CI/CD pipelines", which is the
release card.

The trap

Groundedness does not check whether the right thing was retrieved.

If a scenario says the answers are well-supported but cite the wrong version of a
document, groundedness is the distractor — it is measuring exactly what it claims
to measure and it has no opinion whatsoever about whether the retrieved material
should have been retrieved.

The answer is retrieval evaluation, run separately.