The eleven agent evaluators

The previous card put the agent-specific evaluators in a box and moved on. This
card opens the box, because there are eleven of them and the way Microsoft groups
them tells you more than the list does.

System evaluation and process evaluation

Microsoft splits the agent evaluators by what they look at:

"System evaluation — to examine the end-to-end outcomes of the agentic
system. Process evaluation — to verify the step-by-step execution to achieve
the outcomes."

In plainer terms: system evaluation asks did it get there. Process evaluation
asks did it get there sensibly.

Both matter, and neither substitutes for the other. An agent can pass every
process check and still fail the task. It can also complete the task by a route
you would never have signed off — right answer, alarming method.

The system evaluators

Task Completion — did the agent fully complete the requested task, with a
usable deliverable that meets the requirements?

Task Adherence — did it follow the rules and constraints in its instructions?

Intent Resolution — did it correctly identify what the user actually wanted?

Task Navigation Efficiency — did it take the expected steps, or wander? This
one compares against an expected sequence, so it needs ground truth.

Customer Satisfaction — how satisfied would a user be across the whole
conversation, measured across helpfulness, completeness, clarity, tone, resolution
and adaptability.

The process evaluators

All five are about tool use, and the interesting thing is that they are not
redundant. Each one catches a different failure.

Tool Call Accuracy — the right tool, with the right parameters.

Tool Selection — the right tools chosen, and no unnecessary ones alongside.

Tool Input Accuracy — were all the parameters correct, judged strictly.

Tool Call Success — did the calls actually succeed, or did they error or time
out? This one is purely technical.

Tool Output Utilization — did the agent understand and use what the tool
returned?

That last one deserves a moment. The call succeeded. The arguments were right. The
tool returned the correct answer. And the agent then ignored it and answered from
somewhere else. Every other check on that list passes. No other evaluator sees
this failure, and in practice it is common.

There is also a Quality Grader, which rolls relevance, abstention and answer
completeness into a single evaluator — and adds groundedness and context coverage
when you supply context.

Two operational facts that decide whether you can use them at all

The first is the one that catches teams out. Microsoft says to avoid
tool_call_accuracy, tool_input_accuracy, tool_output_utilization,
tool_call_success and groundedness when the conversation includes calls to
Azure AI Search, Bing Grounding, Bing Custom Search, SharePoint Grounding, Code
Interpreter, Fabric Data Agent or Web Search. Support is limited for those.
Supported: File Search, Function Tool, MCP and knowledge-based MCP.

Read that list again and notice what it means. The evaluators are weakest exactly
where enterprise agents are most common — grounded on SharePoint, searching an
Azure AI Search index. Know that before you promise anyone a quality dashboard.

The second: in multi-agent systems, the system evaluators apply "to the main
orchestrator or the final agent responsible for task completion", not to every
agent individually. Process evaluators go per-agent.

What the outputs look like

Most return a binary Pass/Fail. Intent Resolution and Tool Call Accuracy score
on a 1–5 scale and then threshold to pass or fail. Customer Satisfaction returns
a 1–5 Likert scale. Several of the eleven are marked preview.

The trap

Task Completion and Task Adherence are the likeliest distractor pair on the
whole exam.

If a scenario says the agent did exactly what was asked but broke a policy stated
in its instructions, the answer is Task Adherence. The task completed
perfectly — that is precisely the point. Reaching for Task Completion because the
word "task" appears in the stem is the mistake the pair is designed to catch.