Multi-agent risk is not single-agent risk at scale: the five failure classes that only exist between agents

> ../research/multi_agent_risk.md

§ 01 · A funding call that is really an admission

In June, Google DeepMind and four partners opened a funding call of up to ten million dollars for technical research into multi-agent safety. Grants run one to two years, in two tiers, with proposals due 8 August 2026 and awards announced in the autumn. Informational webinars ran on 30 June and 23 July.

The interesting part is not the money. It is the exclusion list. The call explicitly does not want proposals on single-agent alignment, interpretability, robustness, jailbreak defence, or prompt injection protection. Those are, roughly, the entire published corpus of applied agent safety work. What it does want is the science of agent networks, infrastructure for identity and reputation and commitment between agents, and methods for overseeing populations of deployed agents.

Read plainly, that is a well-funded group of people saying: the controls we have govern one agent, the systems we are shipping are not one agent, and there is not yet a field of research for the gap. The stated view from DeepMind's alignment side is that the field does not exist and they would like it to.

For anyone running a multi-agent system in production right now, and that is most teams who got past a single-loop prototype, this is worth more than the usual funding announcement. It is a public statement about the coverage of your control set.

> deepmind.google/blog/investing-in-multi-agent-ai-safety-research

── Where the gap is ──

§ 02 · Why per-agent controls do not reach it

Every control we have written about in this Lab operates on one agent. Permission boundaries scope what an agent may touch. The kill switch stops an agent. The approval queue puts a human in front of an agent's write. Instrumentation records what an agent decided. Trust boundaries define where an agent's inputs stop being trusted.

Each of those is necessary. None of them observes the system.

The reason is structural. A per-agent control evaluates a predicate over one agent's state and one agent's actions. A multi-agent failure is a predicate over the relationship between several agents' states and actions, and that predicate is not evaluable from inside any one of them. We traced the concrete version of this in the false consensus postmortem: four agents, every one behaving correctly against its own contract, and a confident false output at the end. There was no agent to blame, because there was no agent that was wrong. Auditing each component individually returns a clean result and tells you nothing.

This is the same reason a code review of four correct functions does not tell you the program is correct. Correctness of parts is not composed.

── The five classes ──

§ 03 · The five failure classes

The research literature converging on this problem, particularly the Cooperative AI Foundation's work on multi-agent risks, sorts the space into five classes. They are worth learning as a vocabulary, because a failure you cannot name is a failure you will misdiagnose as a bug in whichever agent happened to speak last.

Collusion. Agents converge on a joint behaviour that serves their individual objectives and defeats the constraint the operator intended. The textbook case is pricing agents arriving at a coordinated floor without any instruction to coordinate, but the production version is smaller and duller: two agents settling into a pattern of mutual approval because each treats the other's output as evidence, and the loop is cheaper than checking.

Conflict. The opposite dynamic. Agents with partially opposed objectives escalate, consume resources, and produce oscillation rather than convergence. A retry loop between two services is a trivial instance. A negotiation between a procurement agent and a supplier agent, both budget-optimising, is not trivial at all.

Destabilising dynamics. Feedback in the interaction graph amplifies rather than damps. Each agent responds rationally to what it observes, the responses are inputs to each other, and the system oscillates or runs away. This is a control theory failure, not a model failure, and no amount of improving any single agent fixes it.

Emergent agency. The population exhibits goal-directed behaviour that no individual agent was given. This is the class that sounds least serious and is hardest to reason about. The practical form is that a pipeline develops a stable strategy, in the game-theoretic sense, that nobody designed, and the strategy survives changes to individual agents because it is a property of the arrangement.

Novel security vulnerabilities. Attacks that exist only because agents talk to each other: instructions injected into one agent that propagate as trusted content into a second, authority laundering where a low-privilege agent induces a high-privilege one to act on its behalf, and identity confusion where the chain of who asked for what dissolves after two hops. We have written the first-hop version of this. The multi-hop version is worse, because the compromised content arrives at the second agent already carrying the first agent's credibility.

── What this means in practice ──

§ 04 · The three properties that decide your exposure

You do not need all five classes to be plausible in your system. Which ones apply is determined by three structural properties, and you can assess them this afternoon.

Is the interaction graph acyclic. A pipeline where output flows one direction and never returns is a fundamentally different risk object from a system where agents can respond to each other. Cycles are the precondition for destabilising dynamics, conflict and most collusion. Many production systems acquired a cycle by accident, usually because a retry, a review step, or a shared queue closed a loop nobody drew.

Does one agent's output become another agent's trusted input. If agent B treats agent A's output as evidence rather than as claims requiring independent grounding, then A's errors do not stay in A. This is the mechanism behind false consensus and behind multi-hop injection, and it is almost always introduced for a good reason: independent verification is expensive and the outputs looked fine.

Do the agents share an objective, a budget, or a resource. Shared scarce resources create the incentive structure that produces conflict and collusion. Two agents drawing on one rate limit, one budget, or one approval queue are in a game with each other whether or not anyone intended that.

Zero of three, and your multi-agent system is a pipeline with a naming convention, which is fine. Two or three of three, and you have a system whose behaviour is not determined by its components.

── What to instrument ──

§ 05 · What to instrument in a system you already run

The controls for this are immature, which is precisely the point of the funding call. But the observability is not, and observability is the prerequisite for everything that comes later.

Record the interaction graph, not just the traces. Most instrumentation captures what each agent did. Very little captures which agent's output was in which agent's context at the time it decided. Emit that edge explicitly: source agent, target agent, artifact identifier, and whether the target treated it as evidence or as a claim. Without this, no multi-agent failure is reconstructable after the fact.

Preserve provenance across hops. Every artifact carries the chain of who produced it and on what basis, and that chain survives being consumed by another agent. Two hops is enough to lose it entirely if nothing is enforcing it, and provenance is the only thing that lets you answer, after an incident, whether a conclusion was ever grounded in anything.

Measure agreement, and be suspicious of it. In any system where several agents contribute to one conclusion, unanimity is a metric worth alerting on, not a comfort. Track the rate at which downstream agents diverge from upstream ones. A rate that trends toward zero means the later agents have stopped evaluating and started deferring, which is when the system's error correction quietly stops existing.

Bound the system, not just each agent. Per-agent turn ceilings and budgets do not compose into a system ceiling. A total budget, a total turn count and a wall-clock deadline for the whole interaction, enforced outside every agent, is what stops conflict and runaway dynamics from being unbounded. The kill switch has to reach the system, not one process.

Give the population a shared identity chain. Who acted, on whose behalf, under what authority, maintained across hops rather than reset at each agent. This is the requirement the joint guidance from the national security agencies put on individual agents in the spring, extended to the case where the actor is a chain rather than a component.

── The honest position ──

§ 06 · The honest position

There is no mature control set for this yet. Anyone selling you one is describing per-agent controls with a multi-agent label.

What is available today is a diagnosis, a vocabulary, and a set of things to record so that when the controls arrive you have the data they need. Our own position, and it is a conservative one, is that multi-agent architectures should be chosen deliberately rather than arrived at. A single agent with several tools has a smaller failure surface than four agents with one tool each, and most systems that describe themselves as multi-agent are the second thing wearing the language of the first.

Where a genuine multi-agent architecture is warranted, the human gate belongs at the system boundary rather than inside any one agent, which is the argument the approval queue pattern makes about routing only the decisions that need a person. In a multi-agent system the decision that needs a person is usually the system's conclusion, not any individual step, and it is precisely the step that looks most confident.

The research call closes on 8 August. If anyone reading this operates a multi-agent system at scale and has traces of the interaction graph, that data is currently scarcer than the funding.

── End of note ──

ORBIRESEARCH