The AI Red Teaming Agent

Foundry ships a tool for attacking your own system, and its purpose is stated
plainly: to "proactively find safety risks associated with generative AI systems
during design and development" by "simulating the behavior of an adversarial user
who is trying to cause your AI system to misbehave."

What it is built on

PyRIT — Microsoft's open-source Python Risk Identification Tool — combined with
Foundry's Risk and Safety Evaluations.

Worth knowing both names, because the question can come from either direction.

Three things it does

Automated scans for content risks. It probes model and application endpoints by
simulating adversarial behaviour, at a scale no human team would attempt.

Evaluation of probing success. Each attack-response pair is scored, producing
metrics including the Attack Success Rate (ASR).

Reporting and logging. A score card of attack techniques and risk categories to
"help you decide if the system is ready for deployment", with findings logged,
monitored and tracked over time in Foundry.

Why ASR is the number that matters

ASR is the number to put in front of a release board, and the reason is worth
stating.

It is a measured percentage rather than an assurance. It moves when you change
your defences, so it responds to work. And it can be tracked across releases,
which turns "is this safe enough" from an argument between people with different
risk appetites into a trend anyone can read.

That is a meaningful upgrade on the usual alternative, which is a security review
producing a document that says the team has considered the risks.

Shift-left, and the NIST frame

Microsoft situates this within NIST's Govern, Map, Measure, Manage, focusing on
the last three:

Map — identify the relevant risks and define the use case.
Measure — evaluate those risks at scale.
Manage — mitigate in production and monitor, with an incident-response plan.

And the argument for automating it is honest about why the manual version does not
scale:

"Manual AI red teaming process is time and resource intensive. It relies on the
creativity of safety and security expertise… This process can create a
bottleneck for many organizations."

Shift-left here means running these scans "throughout the design, development,
and pre-deployment stage" — moving "from costly reactive incidents to more
proactive testing frameworks that can catch issues before deployment."

What it does not replace

Automated red teaming does not replace human red teaming. It makes human time
affordable, by clearing the known attack classes automatically so your scarce
expertise is spent on the novel ones.

Treating the scan as the whole programme is the mistake — it tests the attacks
somebody has already thought of, which is exactly the category an inventive attacker
is not limited to.

The trap

Track ASR alongside false-positive rate.

An Attack Success Rate of zero is not automatically good news. A system that refuses
anything faintly unusual will score beautifully and be unusable, and the score card
will not tell you — because refusing a legitimate request is not an attack success.

The pair of numbers is the honest measurement. Either one alone can be gamed by
moving in one direction.