The agent that nearly paid the wrong supplier

This autumn, four personal AI agents launched in seven weeks: OpenAI's dots, Meta's Muse, xAI's Grok Bot and Instinct. Each gets its own computer in the cloud, signs into your apps and keeps working after you close the tab. Boards noticed. So did a lot of CIOs, who are now being asked a simple question: can we have one of those for the business?

The honest answer is "yes, but you won't be building one agent." You'll be building a small team of them, and every decision along the way carries a risk someone will later audit.

So this is a story. The company is fictional. The decisions are real, and so is the guidance behind them, which comes from Anthropic's and Microsoft's own published advice on building agents. I've kept the consumer products to the odd line, because they show the same patterns in the wild.

Meet Fernhill Supplies: a UK distributor of catering equipment, around 1,200 staff, a few thousand suppliers. Its accounts-payable team is drowning. Invoices arrive by email as PDFs. Each has to be matched to a purchase order and a delivery note. Mismatches mean emails back and forth with suppliers. Then someone approves the payment.

The finance director wants the backlog gone. The board wants "AI agents". The CIO has to make it work, and make it survive an audit.

Chapter one: the first answer is "not everything needs an agent"

The CIO starts with what the technology actually is.

A language model does one thing. Text goes in, text comes out. It can't open an inbox, read a PDF or pay anyone. When an AI product "checks your email", ordinary code is doing the checking. The model writes a request, something like fetch the latest invoices. The code runs it and pastes the result back in as text. The model reads that and decides what to ask for next.

That loop is what people mean by an agent: a model that asks for actions, with code that carries them out, round after round, until the job is done. The actions it may ask for are its tools. Every power the agent has is a tool someone chose to connect. So is every risk.

The first design meeting looks at the invoices themselves. Most of them are boring. The invoice matches the purchase order, the goods arrived, the amounts agree. Nothing about that needs judgement. A fixed sequence of steps, run the same way every time, does it faster and more predictably. That's a workflow, and it's been possible for years.

Both vendors say the same thing to anyone building agents. Anthropic:

we recommend finding the simplest solution possible, and only increasing complexity when needed. This might mean not building agentic systems at all.

Microsoft's agent framework is blunter: "If you can write a function to handle the task, do that instead of using an AI agent."

So Fernhill splits the problem. Clean matches go through a plain workflow. The agent gets the exceptions: the price that doesn't match, the delivery that was short, the supplier who invoiced twice. Those are the cases where the next step depends on what you just found out. That's exactly where an agent earns its place.

The cheapest agent is the one you didn't need to build.

Chapter two: one agent that knew too much

The first prototype is a single agent with every tool it might need. It can read the inbox, search the purchasing system, check the warehouse records, email suppliers and draft payments. It has one long set of instructions covering all of it.

It works in the demo. Then it meets real volume, and quality slides.

The cause is the context window: the amount of text a model can hold in mind at once. Every instruction, every tool description and every document it reads has to fit inside. A messy dispute with twenty emails and three PDFs fills it. Details fall out. Tools with similar names get confused.

The fix is to split the work. Fernhill ends up with four agents, each with a smaller job and its own context:

Intake — reads incoming email and pulls out the invoice details.
Matching — compares invoice, purchase order and delivery records, and explains any gap.
Supplier queries — writes to the supplier to resolve the gap.
Payment proposal — prepares the payment for a human to approve.

That's a multi-agent system. Anthropic built its own research product this way, with a lead agent handing pieces of work to helpers working in parallel. It measured the gain: the multi-agent version "outperformed single-agent Claude Opus 4 by 90.2% on our internal research eval."

It also measured the bill:

In our data, agents typically use about 4× more tokens than chat interactions, and multi-agent systems use about 15× more tokens than chats.

Tokens are the units models read, write and charge by, about three-quarters of a word each. More agents means more text passed around, which means more cost.

The CIO's most important split isn't about quality, though. It's about permissions. The payment agent is kept apart because it needs access no other agent should have. If the supplier-queries agent is ever tricked, it has no route to the money. That reason for splitting gets less attention than the others, and it's usually the easiest to defend.

Chapter three: deciding who goes first

Four agents need an arrangement. Who passes work to whom, and in what order? This is the orchestration pattern, and Microsoft's architecture guidance names five:

Sequential — each agent hands its output to the next, like a production line.
Concurrent — several agents work on the same thing at once, and their results are combined.
Handoff — one agent passes the whole task to a specialist.
Group chat — agents discuss the problem together, with a manager deciding who speaks.
Magentic — a manager agent plans, delegates, checks progress and re-plans when something fails.

Fernhill doesn't pick one on general merit. It lets each constraint in the process pick. The normal exception path is sequential, because order matters: you can't propose a payment before the match is resolved. When a supplier replies with a question about a different order, that's a handoff. A genuinely tangled dispute, where nobody knows the steps in advance, needs a manager that can re-plan.

Group chat is ruled out early. Supplier pricing is confidential, and in a group chat every agent sees everything.

Microsoft warns about the flexible pattern too:

Magentic orchestrations are the most variable because the manager agent continues to iterate until it builds a viable plan, which makes it hard to predict the total cost.

The consumer agents live almost entirely in that last pattern. Ask one to "sort out my car insurance renewal" and nobody knows the steps in advance. That's why they're impressive, and why they're hard to predict.

Chapter four: what the agents said to each other, and what they remembered

The agents now pass work between them. The lazy way is to forward the whole conversation each time. Fernhill doesn't, for three reasons: it's noisy, it's expensive, and it hands each agent information it shouldn't have. Instead each handoff is a deliberate payload, a short structured note with only what the next agent needs.

A process that runs for weeks also needs memory. The matching agent learns that one supplier always invoices before delivery. The supplier agent learns who to email at each firm. That's useful, and it's also where two new problems start.

The first is poisoning. If an agent can write to a memory that other agents trust, one bad write spreads to all of them as fact. Microsoft's red team tested this on an email assistant. A single planted email told it to remember to forward sensitive messages to an outside address. It worked on 4 runs in 10. Then the team made the assistant more diligent about checking its memory:

After modifying the agent's system prompt to encourage the agent to check its memory before responding, the attack success rose to over 80%.

The feature meant to make it more helpful made it easier to poison. Fernhill's fix is to stop the agents writing straight to shared memory. Updates go through a check first.

The second problem arrives as a letter. A sole-trader supplier asks Fernhill to erase the personal data it holds about them. Under UK data protection law, Fernhill has a month to deal with that request. Where is that data now? In the invoices, yes. But also in what the agents remembered.

The consumer products show how awkward this is. OpenAI's help page for dots says disconnecting an app "does not delete information your dot has already obtained from it." The only way to remove it is to delete the dot. Fernhill decides memory has to be something it can search, export and delete per person. Otherwise it fails the first request that arrives.

Chapter five: whose name is on the payment?

Until now, the agents mostly read things. The payment agent is different. It acts. So the question becomes: whose authority is it using?

There are three possible answers, and they behave very differently.

Its own identity — the agent has its own account with its own narrow permissions.
On behalf of a person — it carries a specific user's permissions for that one request, and no more.
A shared account or key — one powerful login that every request uses.

The prototype used the third. A developer had connected it to the finance system with a service account that could do almost anything. It worked perfectly in testing, because the testers had broad access anyway.

It also meant that when something went wrong, nobody could tell whether a person or an agent had done it. Every action looked the same in the logs.

Fernhill moves each agent to its own identity. In Microsoft environments that's what Entra Agent ID is for: each agent gets its own entry in the company directory, so it can be given narrow permissions, audited, and switched off in one place. Agent 365 adds a registry where admins "can now view all their agents in a single, centralized registry". Microsoft's own least-privilege guidance frames the shift:

the security question shifts from whether an agent can complete a task to whether it should be allowed to perform each action, against which resources, and under whose authority.

Identity also has a physical side: where do the passwords and API keys actually sit? Anthropic learned this on its own hosted agents. Code the model wrote ran in the same container as the credentials, "so a prompt injection only had to convince Claude to read its own environment." The fix:

The structural fix was to make sure the tokens are never reachable from the sandbox where Claude's generated code runs.

Meta's Muse follows the same rule for consumers: the agent only ever holds a stand-in token, swapped for the real one outside its environment after an action is approved.

If an agent logs in as a person, the logs will say the person did it.

Chapter six: the email that changed the bank details

Three weeks after go-live, the supplier agent reads an email that looks like it's from a regular supplier. It says the supplier has changed banks, gives new account details, and asks for the outstanding invoices to be paid there.

Every finance team knows this scam. What's new is how it reaches an agent. The email doesn't need to fool a person. It only needs to include text the agent will read as an instruction. That's prompt injection: instructions hidden inside content the agent was only meant to read.

It slips past the usual defences because of where it arrives. Fernhill had filters on what staff typed into the system. But this text came in through a tool, the inbox. It entered the agent's context after those filters had already run.

Nobody in the industry claims to have solved this. Meta says "Prompt injection remains an open problem in the industry". Anthropic says protection inside the model "will never be 100% effective". Microsoft's 2026 failure taxonomy records an advert that "visually mimicked a legitimate "Approve" button" to fool an agent working on screen.

In a multi-agent system the danger multiplies, because one fooled agent passes its mistake along. Anthropic's warning is short: "Agents are stateful and errors compound." Microsoft's advice is to "Validate agent output before you pass it to the next agent."

Fernhill puts checks at four points: what comes in, which tools an agent may call, what those tools return, and what goes out. Then it adds one rule no model can talk its way past. Bank details can't be changed by any agent. A change goes to a person, who confirms it with the supplier through a known phone number.

The fraudulent email gets as far as a flagged exception. The money doesn't move.

Chapter seven: the approval that stopped being read

Every payment needs a human approval. That was in the design from day one, and it's what every consumer agent offers too. Dots asks each time before "permanently deleting data, installing or running software from an unrecognized source, or granting new security-sensitive access."

Fernhill's rule for where to put an approval is simple: put it where the action can't be undone. A payment can't. A draft email to a supplier can be. So drafts flow freely, and payments stop for a person.

Microsoft's failure taxonomy adds a rule the CIO takes seriously:

Deterministic invocation: agents themselves should not determine if/when human in the loop controls are invoked.

In other words, the agent never decides when to ask. The rules do.

Then a quieter problem shows up. The approvers are approving everything. They see forty requests a day, nearly all correct, and they stop reading. Anthropic measured the same effect in its own products:

Our telemetry showed users approved roughly 93% of permission prompts. The more approvals a user sees, the less attention they pay to each, becoming over time much less diligent in their supervision.

The fix is another agent. A reviewer checks every proposed payment against the purchase order, the supplier's history and the approved bank details. Routine payments get a lighter touch. Anything unusual reaches a person with a clear reason attached. Fewer approvals, so the ones that remain get read.

That's also how the consumer products are built. Muse has a separate agent called Sentinel, "the sole permission authority for connector actions and network egress". Dots and Grok Bot each have an automatic review step. OpenAI keeps its review controls "outside the environments dots can change", so the working agent can't switch them off.

But the reviewer is a model too, and models miss things. Anthropic published the miss rate for the automated checker in its own coding product: "The 17% false-negative rate on real overeager actions is the honest number."

Chapter eight: the audit

Nine months in, the system has cleared the backlog. Then the questions arrive, from more directions than the CIO expected.

The external auditors test controls over payments every year. They don't care that an AI was involved. They want to see who approved each payment, that the person preparing it wasn't the person approving it, and evidence for both. "The agent proposed it and the reviewer agent approved it" is not segregation of duties.

The data protection lead wants to know what personal data the agents read, how long memory keeps it, and how an erasure request is honoured. New, higher-risk processing like this is likely to need a data protection impact assessment before it goes live, not after.

The cyber insurer wants to know what an agent can reach, and whether the bank-details scam is covered by a control or by luck.

Large customers send security questionnaires asking whether AI processes their data, where it runs, and who can see it. Where it runs matters: Grok Bot's documentation says its computers "run in the United States today". Muse launched in the US and Canada only. Dots isn't on the Pro plan in the UK.

The EU, if Fernhill sells into Europe: the EU AI Act already expects firms to support the AI literacy of staff who use these systems.

All of them ask, in different words, for the same three things.

Show me what happened. Not a summary: the run itself, every step and tool call, enough to replay it. Monitoring tells you whether the system is still healthy. This is observability, being able to see inside one specific run. They're different, and auditors want the second. Agents rarely fail loudly. They drift, and nothing crashes.

Show me who was allowed to do it. The agent identities, their permissions, and the registry of every agent in use.

Show me it can't run away. Every handoff re-sends text, so costs grow quietly. Fernhill had set a monthly cloud budget. It turned out a budget only sends an alert. It doesn't stop anything. The real brake is a hard limit inside the system itself: a cap on loops, depth and spend for each job.

Chapter nine: what the CIO wrote down

Afterwards, the CIO turns the nine months into a set of questions that any new agent project at Fernhill has to answer before it goes live. They work just as well for judging a product someone wants to buy.

Need — would a workflow or a plain function do this job?
Shape — why several agents, and which constraint chose the pattern?
Identity — does each agent act as itself, as a named user, or through a shared key?
Credentials — can an agent's own code ever reach a password or key?
Memory — what does it keep, who checks what it writes, and can one person's data be deleted?
Inputs — which outside content reaches it, and are tool results checked like user input?
Approvals — which actions always need a person, and who can turn that off?
Evidence — can you replay any run, and is there a hard ceiling on loops and spend?
Location — where does it run, and under whose law?

None of these is about which model to use. Every one of them is about what surrounds the model.

The trap

The most reassuring sentence in any agent pitch, consumer or enterprise, is that a second system checks the first. Fernhill has one. Every personal agent launched this autumn has one. It's real, and it helps.

But a reviewer that is itself a model is a judgement, not a wall. The same email that fooled the supplier agent could be written to fool the reviewer. It misses a measurable share of bad actions. xAI's documentation says plainly that its reviewer "does not review every side effect", memory writes among them.

What saved Fernhill wasn't the reviewer. It was the rule no model could argue with: an agent cannot change bank details. The controls that hold are all like that. A key the agent never sees. A network it can't reach. An approval it isn't allowed to skip. An identity that can be switched off in one place.

So when someone shows you a multi-agent system, theirs or a vendor's, don't ask how clever the checking agent is. Ask what the agents physically can't do, even on the day they're fooled.