Card 11 of 38· Domain 2 · Generative AI and agents
Evaluating models and apps
The four evaluator families and what each catches, scoring and thresholds, and why groundedness — not relevance or fluency — is the metric that catches a confident lie.
Evaluation is how you find out whether the thing is any good, and the exam wants you to know which evaluator answers which question. Most of the marks come from one distinction, covered at the end.
The four families
| Family | The evaluators in it | What it catches |
|---|---|---|
| General quality | Coherence, fluency | Logical consistency and language quality |
| Retrieval | Groundedness, relevance | Fabrication — this is the hallucination detector |
| Safety | Hate and unfairness, violence, sexual, self-harm, indirect attack, protected material | Harmful or unsafe output |
| Agent-specific | Tool call accuracy, task completion, intent resolution | Whether the agent actually did the job |
The families map onto genuinely different failure modes. A system can be coherent and fluent while inventing facts, or perfectly grounded while failing to complete the task it was given. One family will not tell you about another.
Scoring
Retrieval evaluators score from 1 to 5, and the pass threshold is 3.
Groundedness Pro is different: it returns true or false, via Content Safety, rather than a graded score.
Automatic evaluation works by using a model as a judge over a dataset of prompts. Worth sitting with for a second — you are asking one language model to grade another's homework. It scales in a way human review does not, and it inherits the judge's blind spots.
Choosing the right evaluators
If you are testing a direct model, drop the intent-resolution and business evaluators. They are built for multi-turn agent evaluation, where there is a conversation and a goal to resolve. Pointed at a single model call they measure nothing meaningful.
If you have no evaluation dataset, the wizard will generate a synthetic one.
Before and after deployment
Pre-production: the AI red teaming agent, built on PyRIT, probes the system adversarially before real users arrive.
Post-production: continuous evaluation, scheduled evaluation and scheduled red teaming — the mechanisms from card 7.
One practical note with a cost attached: playground evaluations are on by default and billed by consumption. Easy to leave running.
Groundedness is the fabrication metric
This is the trap, and it is worth being precise about because the three metrics sound similar and measure different things.
- Groundedness — is the answer anchored in the source material?
- Relevance — is the answer on topic?
- Fluency — is it well written?
If a question asks how to detect made-up answers, the evaluator is groundedness. Not relevance. Not fluency.
The reason is worth holding onto beyond the exam: a confidently well-written lie scores highly on both relevance and fluency. It is on topic and it reads beautifully. Only groundedness asks the question that catches it — is any of this actually supported by the source?