The evaluation set — the durable asset

Card 22 said evaluation is the only thing that catches a confidently wrong
answer. This card is about the thing evaluation runs against, and why it outlasts
everything else you build.

What makes an evaluation set worth having

Real questions. Taken from actual users, not invented by the build team.
Including the badly-worded ones, because those are what people really type. The
fallback log is the best source you have.

Covers the tail. Not just the easy top ten intents. Edge cases, ambiguous
asks, out-of-scope questions — and the cases where refusing correctly is the right
answer, which people routinely forget to include.

Versioned and growing. Every production failure becomes a new test case, and
the set is versioned alongside the agent.

Validating custom AI models

Define the criteria before building. What does acceptable look like,
numerically? Which dimensions matter — accuracy, groundedness, tone, latency,
cost?

Deciding that afterwards means deciding it in the presence of the result, which
is not really deciding.

Compare against the incumbent. The bar is relative: no worse on any dimension,
better on at least one. Otherwise the change is not justified, however
interesting the new thing is.

Validate prompts too. Copilot prompt best practice is testable — structure,
examples, explicit output format, and behaviour when the answer is missing.

Using Copilot to build the test cases

What it is good at. Generating variations of a real question. Drafting edge
cases you would not have thought of. Clustering production failures into themes.

What still needs a human. Deciding the expected answer. Approving the rubric.
Judging anything consequential.

That division is worth carrying beyond this exam: the machine is good at
generating candidates and clustering, and unqualified to decide what correct means.

End-to-end scenarios

Design tests that cross several Dynamics 365 apps, and test the handoffs
specifically — that is where multi-app solutions break, and each app tested alone
will pass.

Exam trap. The evaluation set — not the agent — is the asset that survives.
Models change, platforms change, the agent gets rebuilt; the set of real
questions with agreed correct answers is what lets you prove the new thing is
not worse.

The one line to carry: build the evaluation set as though the agent is
temporary, because it is.

Goes with: card 41 for the metrics themselves: precision, recall and F1, mAP
for detection, and the Foundry evaluator families for anything generative.

---

Currency (19 Aug 2026). Evaluations can now be triggered from the Power Automate
connector, with a REST API in preview for CI/CD. That moves evaluation out of manual
testing and into the release pipeline — see cards 27 and 44.