Release, rollback and testing

Most of this topic is ordinary software engineering. Three things are not, and
those three are where the exam lives.

The prompt is a deployable artefact. It changes behaviour more than most code
changes do, and it changes more often. If it lives in a portal rather than the
repository, you have an untracked production dependency.

The model is a dependency you do not control. It can be updated or deprecated
underneath you. Your release process has to handle a change you did not initiate,
which ordinary CI/CD simply assumes away.

"Passing" is a threshold, not a boolean. A unit test passes or fails. An
evaluation returns a score, and somebody has to decide in advance what score blocks
a release.

The three methodologies compose

They are usually presented as alternatives. They are not.

DTAP — Development, Test, Acceptance, Production — is sequential environments
with promotion gates. It suits regulated work where each gate has an owner and
evidence. It is slow by design and the slowness is the feature.

Blue/green runs two identical production environments and switches traffic
between them. Rollback is a traffic switch, which is the fastest recovery
available. It costs double the production footprint during a release.

Canary gives the new version a small share of live traffic first, and you watch.
It suits changes whose effects only appear at real volume with real users — which
describes most agent changes, because you cannot fully evaluate quality offline.

In practice: DTAP for the promotion path, canary for the production release,
blue/green underneath for instant rollback. A scenario naming a regulated
environment points at DTAP; one worried about quality at real volume points at
canary; one demanding fast recovery points at blue/green.

Rollback is harder than it looks

Rollback assumes going back to the previous state fixes things. For agents there
are four reasons it might not.

The prompt and the code must roll back together. A prompt written for v2 running
against v1 code is a state neither was tested in.

Memory persists across the rollback. Long-term memory written by the new version
is still there afterwards. If the new version stored something in a new shape, the
old version now reads data it does not understand.

In-flight runs resume. A long-running agent may be mid-workflow, and durable
execution means it picks up where it left off — possibly into code you just
reverted.

The model version may not roll back. If the change was a model update, reverting
your code does not revert the model.

The practical version of this: rehearse the rollback, including the memory
question. "We can roll back" is a claim that is true of the container and often
false of the state around it, and you find out which during an incident.

Testing, and where determinism belongs

Four kinds, and the fourth does not exist in ordinary software.

Unit — your functions, tools, parsers, payload construction. Deterministic.
Integration — tool calls, retrieval, MCP servers, downstream services.
Regression — known past failures never returning.
Automated evaluations — behaviour and quality against a scored set. Not
deterministic; threshold-based.

The organising principle is push determinism down. Everything you can test
without a model, test without a model: payload construction, filter logic, parsing,
guardrail rules, schema validation. All fast, all cheap, all reliable. Teams that
skip this end up with slow, expensive, flaky suites that mostly re-test the model.

Evaluations sit at the top — the slowest, most expensive, least deterministic layer,
and the only one that measures what the system is actually for.

A habit worth adopting: every production incident becomes two artefacts. A
deterministic test if the cause was deterministic, and an evaluation case
regardless. That is how the suites grow from real evidence rather than imagination.

Quality gates

Microsoft's own framing is that observability lets you "integrate automated quality
gates into CI/CD pipelines." Making that work needs four decisions.

Which metrics gate? Not all of them. Groundedness and task completion are
gate-worthy; fluency usually is not.

What threshold? Agreed in advance, from the current baseline.

What about run-to-run variation? Two runs of the same set on the same system
differ slightly. Your threshold has to sit outside that noise, or you will block
releases at random and the team will quickly learn to override the gate.

Who can override, and is it logged? There will be a legitimate emergency. An
override that is possible and recorded is healthy; one that is impossible gets
bypassed by other means.

Infrastructure as code

For agents the estate is larger than it first appears: the Foundry project, model
deployments and capacity, agent definitions, toolboxes and their versions,
connections, Key Vault and its access policies, private endpoints and networking,
monitoring and alert rules, and RBAC assignments.

Test the infrastructure too — that private endpoints really are private, that the
agent identity holds only the roles it should, that the vault is not publicly
reachable. Those are assertions you can run in a pipeline, and they are exactly the
controls a regulated review will ask you to evidence.

The trap

RBAC is the configuration that drifts fastest, because it gets changed by hand
during incidents and never changed back.

Expressing role assignments in code and asserting them in the pipeline is what turns
a security design into something that stays true six months later. Everything else
in your IaC describes what you built; this part describes what you promised.