CI/CD and production monitoring

This card covers one of the five topics the official video series never teaches. It is examinable anyway, which is exactly why it is worth your attention rather than your skimming.

Two separate things live here: getting an agent deployed automatically, and knowing whether it is still any good once it is out there.

The deployment pipeline, in four steps

Continuous integration and continuous deployment — CI/CD — means changes flow to production through an automated pipeline rather than by hand. The tooling here is azd, the Azure Developer CLI.

1. azd pipeline config — detects whether you are on GitHub or Azure DevOps, creates the service principal, writes the secrets and generates the workflow file.
2. azd provision — creates or updates the infrastructure from your Bicep templates.
3. azd deploy — builds the container, pushes it to the registry, and creates a new hosted-agent version.
4. azd ai agent eval run — the regression gate. It fails the build when scores drop below your threshold.

Step four is the interesting one. Without it a pipeline will happily ship a version that is worse than the one it replaced, because nothing in steps one to three has any opinion about quality.

Pipeline details that trip people up

• Authentication is OIDC federated, not a stored secret. That means the pipeline proves its identity to Azure rather than holding a key. It needs the id-token: write permission on the job.
• The runner has no extensions installed. Run azd ext install microsoft.foundry before anything else.
• Put --no-prompt on every azd ai command. Miss it, and a missing value causes the job to hang waiting for input that will never come — the build does not fail, it just sits there.
• Project context comes from the FOUNDRY_PROJECT_ENDPOINT variable, or from azd ai project set.
• Provisioning needs Contributor plus Foundry Owner. Contributor alone will not do it.

The three legs of observability

Knowing your system works in production rests on three different things, and they answer different questions.

• Tracing — OpenTelemetry data flowing into Application Insights, capturing model calls, tool calls and agent decisions. This tells you what happened in one request.
• Monitoring — operational metrics: token consumption, latency, error rates. This tells you how the system is behaving in aggregate.
• Evaluation — quality, safety and agent-specific evaluators. This tells you whether the answers are any good, which neither of the other two can.

A system with tracing and monitoring but no evaluation will report itself perfectly healthy while giving people worse and worse answers.

Four ways to watch it after launch

• Continuous evaluation — scores live production traffic at a sampled rate.
• Scheduled evaluation — runs a fixed test dataset on a schedule.
• Scheduled red teaming — recurring adversarial probing, on the assumption that new weaknesses appear over time.
• Azure Monitor alerts — fire when a quality threshold is breached or harmful output is detected.

Drift and live quality are not the same question

This is the distinction the exam tests, and it is genuinely useful to have straight.

Drift means the system's behaviour has changed over time. To detect it you need scheduled evaluation: the same dataset, run repeatedly. Because the inputs are held constant, a moving score can only mean the system moved.

Continuous evaluation samples real production traffic. It tells you how you are doing right now, and it tells you nothing reliable about drift — because the inputs are changing too. A falling score might mean the system degraded, or it might mean people started asking harder questions this month. You cannot separate the two.

If a question asks how to detect drift, the answer is scheduled evaluation against a fixed dataset. Every time.