Multi-Agent Orchestration for Physical Security: The 2026 Agentic AI Briefing on the Coordination Layer, Agent Handoff, Shared World-State, and the Supervisor Pattern Between Detection and Response
Why the coordination layer, not the detection model, decides whether agentic security works: a research-grounded framework for agent handoff, shared world-state, and the supervisor pattern between detection and response.
The agentic security conversation has moved past the single autonomous agent. The 2026 architectures buyers are being shown stitch together several specialized agents: one that classifies detections, one that gathers corroborating context, one that decides what action to recommend, one that drives notifications and access control. That is a multi-agent system, and multi-agent systems introduce a failure surface that single-agent platforms do not have. The dominant failure mode is not bad detection. It is bad coordination: agents losing context on handoff, issuing conflicting actions, or waiting on each other while a threat is live. This report assembles the primary research on multi-agent failure and maps it onto physical security, where the cost of a coordination breakdown is measured in response seconds, not retried API calls.
Multi-agent orchestration is the layer of an agentic physical security platform where independent AI agents divide a workflow and hand work to one another. It is becoming the default architecture for the same reason it became the default in software agents generally: a single model asked to watch every camera, reason about every event, decide every response, and drive every integration is a single point of failure with no separation of concerns. Splitting the work into specialized agents is sound engineering. But the moment a workflow involves more than one autonomous agent, the system inherits a second category of risk that has nothing to do with how well any individual agent detects a weapon. It has to do with whether the agents stay coordinated under load, under ambiguity, and under failure.
This report does three things. First, it summarizes the first systematic primary research on why multi-agent LLM systems fail, and translates each failure category into its physical security equivalent. Second, it maps a coordination architecture onto the detection-to-response pipeline, borrowing the command-and-control doctrine that emergency management spent five decades refining. Third, it gives security leaders a structured way to interrogate the orchestration claims now appearing in agentic security RFPs, because the orchestration layer is where these platforms will quietly succeed or publicly fail.
What the First Multi-Agent Failure Taxonomy Actually Found
The most important empirical result in multi-agent systems was published not by a vendor but by an academic group. In "Why Do Multi-Agent LLM Systems Fail?" a team led by researchers at UC Berkeley, including Ion Stoica and Joseph Gonzalez, assembled the first large-scale dataset of multi-agent system failures: more than 1,600 annotated execution traces across seven popular multi-agent frameworks, validated by expert human annotators with high inter-annotator agreement. From that dataset they built the Multi-Agent System Failure Taxonomy, or MAST, the first data-backed map of how these systems break. The headline finding is sobering for anyone about to depend on a multi-agent pipeline: observed failure rates ran as high as 86.7 percent on some framework-and-task combinations, and the gains multi-agent designs promised over single agents were frequently minimal.
What matters more than the failure rate is its structure. MAST clusters fourteen distinct failure modes into three categories, and the distribution tells security architects exactly where to look. Roughly 41.8 percent of failures were specification and system-design issues: ambiguous role definitions, poor task decomposition, duplicate agent roles, missing termination conditions. About 36.9 percent were inter-agent misalignment: communication breakdowns, context lost during handoff between agents, conflicting outputs, and format mismatches. The remaining roughly 21.3 percent were verification failures: inadequate validation and error propagation. Read those three numbers again with a security workflow in mind. Nearly four in five multi-agent failures originate before any agent's reasoning quality even enters the picture, in how the agents were specified and how they coordinate.
"Despite enthusiasm for Multi-Agent LLM Systems, their performance gains on popular benchmarks are often minimal. This gap highlights a critical need for a principled understanding of why MAS fail."
Cemri, Pan, Yang, et al., "Why Do Multi-Agent LLM Systems Fail?", arXiv:2503.13657 (UC Berkeley, 2025)The physical security translation of the MAST categories is direct, and it is uncomfortable. Specification failures look like two agents both believing they own the dispatch decision, or a detection agent that never receives a clear termination signal and keeps re-escalating a resolved event. Inter-agent misalignment looks like a triage agent that drops the confidence score and camera context when it hands an event to the response agent, so the action agent acts on a verdict it cannot inspect. Verification failures look like no agent in the chain checking whether the recommended lockdown actually matches the detected threat before the notification fires. None of these are detection problems. Every one of them is a coordination problem, and coordination is the thing a multi-agent architecture is supposed to be good at.
The Coordination Layer: Where Detection Becomes Response
Orchestration is the connective tissue between an agent that knows something and an action that changes the physical world. In a physical security platform, the orchestration layer is responsible for four things that no individual agent owns: it decomposes an incoming event into the tasks each specialized agent will perform, it routes work to the right agent in the right order, it carries shared state so that context survives every handoff, and it arbitrates when two agents reach conflicting conclusions. When any of those four responsibilities is unowned or poorly engineered, the MAST failure modes appear.
The most studied orchestration pattern, and the one that maps cleanly onto security operations, is the supervisor pattern. A single supervisor or orchestrator agent classifies the incoming event, decomposes it into sub-tasks, dispatches each to a specialized worker agent, and integrates their outputs into a single recommendation. Industry analyses of production multi-agent deployments report that the supervisor pattern accounts for the large majority of systems that survive contact with production, precisely because it gives the coordination responsibilities a clear owner. It also has a documented cost: every layer of supervision adds latency, because each supervisory reasoning step takes time before any worker begins. In a domain where the entire value proposition is compressing the seconds between threat-visible and responder-notified, that latency is not an abstraction. It is the metric.
Classifies the incoming event, decomposes it into sub-tasks, sequences the worker agents, holds the shared world-state, and arbitrates conflicting outputs. Owns the coordination responsibilities no single worker owns. Every consequential recommendation passes through a human checkpoint before action.
Two design commitments in that diagram do the real work. The first is that the handoff between agents carries the full evidence packet, not a compressed verdict. MAST identifies context loss on handoff as a leading inter-agent failure, and the security version is acute: an action agent that receives only the string "weapon, camera 12" cannot do anything but obey, while one that receives the object class, confidence, camera, timestamp, and annotated frame can be checked by a human and audited after the fact. The second is that authority for consequential action never sits with the orchestrator. The supervisor coordinates; it does not get to order a lockdown. That boundary is the multi-agent expression of the human-in-the-loop principle developed in our Human-in-the-Loop AI Security framework, and it is what keeps an orchestration bug from becoming a physical action.
The Interoperability Question: MCP, A2A, and the New Coordination Standards
Multi-agent orchestration inside a single vendor's platform is one problem. Coordination between agents built by different vendors, on different frameworks, is a second and harder one, and 2025 was the year the industry started standardizing it. Two protocols now anchor the conversation. The Model Context Protocol, covered at the tool-integration level in our Agentic Action Layer technical reference, standardizes how a single agent connects to tools and data. The Agent2Agent protocol, or A2A, standardizes the layer above it: how autonomous agents discover, authenticate, and delegate work to one another across framework and organizational boundaries.
A2A's trajectory is worth a security buyer's attention because it signals where the architecture is heading. Google announced A2A in April 2025 and donated it to the Linux Foundation in June 2025, establishing it as a vendor-neutral standard with founding support from Amazon Web Services, Cisco, Microsoft, Salesforce, SAP, and ServiceNow. By its one-year anniversary in April 2026 the project reported support from over 100 technology companies and shipped A2A Protocol v1.0, its first stable, production-ready release, adding cryptographically signed agent identity for trust in multi-agent workflows. The framing the maintainers themselves use is the clarifying one: MCP manages an agent's internal tool integration, while A2A handles the external coordination between autonomous entities. Both layers have to be right for a multi-vendor security stack to coordinate safely.
A physical security buyer rarely buys one monolithic agent. The realistic 2026 stack has a detection platform, a video management system, an access-control system, and a mass-notification system, increasingly each with its own agentic layer. Whether those layers can coordinate without brittle, custom point-to-point integrations is now a protocol question. Asking a vendor how its agents coordinate, and whether they speak emerging standards like A2A and MCP rather than a proprietary handshake, is the orchestration-era version of asking whether a camera speaks ONVIF. It is a lock-in and a resilience question at once.
Borrowing Command Doctrine: What NIMS Already Solved
The coordination problems multi-agent AI is rediscovering were solved, in the human domain, decades ago. The discipline that solved them is incident command. After reviewing fragmented command-and-control failures across major incidents, FEMA formalized the National Incident Management System in 2004, building on the Incident Command System that emerged from 1970s Southern California wildfire response. NIMS exists for exactly the reason multi-agent orchestration is hard: when multiple autonomous responders converge on one event, the default outcome is conflicting directives, duplicated effort, and dropped handoffs unless a coordination structure prevents it. Two NIMS principles translate almost without modification into agent orchestration.
The first is unity of command. In NIMS, every responder reports to exactly one supervisor, which eliminates the confusion of conflicting orders from multiple bosses. The multi-agent analog is direct: each worker agent should answer to one orchestrator, and the system should never allow two agents to both believe they own a decision. MAST's specification-failure category, the largest single bucket of multi-agent failures, is in large part a unity-of-command failure: duplicate roles and ambiguous ownership. The second principle is span of control. Five decades of incident-command practice converged on a ratio of one supervisor to between three and seven subordinates, with five as the rule-of-thumb optimum, because a supervisor stretched past that number loses the ability to track what each subordinate is doing. An orchestrator agent coordinating too many workers hits the same wall, and the practical implication is that very wide multi-agent systems need hierarchy, supervisors of supervisors, rather than one orchestrator trying to track twenty workers at once.
"Unity of command means that each individual involved in incident operations reports to only one supervisor. This eliminates the confusion caused by multiple, conflicting directives."
FEMA, National Incident Management System, on the command principle that maps directly onto agent orchestrationThe lesson for security buyers is that the orchestration layer is not a novel engineering frontier with no precedent. It is a new implementation of a coordination problem with a mature, battle-tested doctrine attached. A vendor whose multi-agent architecture cannot articulate its unity-of-command boundaries, who owns each decision, who arbitrates conflicts, and how wide each supervisor's span runs, is reinventing a wheel that emergency management already machined to tolerance. The questions to ask are old questions in new clothing.
Three Coordination Models, Compared
The agentic security market is converging on three structural answers to how detection becomes response. Each is a defensible engineering position with distinct failure modes, and procurement teams should evaluate them as architectures, not as marketing claims about autonomy.
| Coordination Model | How Work Flows | Strengths | Failure Modes to Probe |
|---|---|---|---|
| Single monolithic agent | One model ingests detections, reasons about context, decides a response, and drives integrations end to end. | No inter-agent handoff surface, so no coordination failures by construction; simplest to audit as one reasoning trace. | Single point of failure; no separation of concerns; one prompt or model regression degrades the entire pipeline; hard to scale across detection modalities without the one agent becoming unwieldy. |
| Orchestrated multi-agent (supervisor pattern) | A supervisor decomposes the event, dispatches specialized worker agents, holds shared state, and integrates outputs into one recommendation for a human checkpoint. | Clean separation of concerns; specialized agents are individually testable; scales across modalities; coordination has a single owner; maps to mature command doctrine. | Supervisory latency at each layer; context loss on handoff if the evidence packet is compressed; the orchestrator itself becomes a critical component requiring its own monitoring. |
| Decentralized peer-to-peer agents | Agents coordinate directly with one another, negotiating handoffs without a central supervisor, increasingly over emerging standards such as A2A. | No single bottleneck; resilient to one agent failing; native fit for multi-vendor stacks where no party owns the whole pipeline. | Hardest to reason about and audit; conflict arbitration is emergent rather than owned; the unity-of-command and span-of-control failures NIMS warns about are most likely here without strong protocol guarantees. |
These models are not mutually exclusive, and the market is hybridizing. A platform can run a supervisor pattern internally while coordinating peer-to-peer with a separate access-control vendor's agent over A2A. The analytical error is treating "multi-agent" as a feature that is present or absent. The right questions are structural: who owns coordination, what survives each handoff, where authority for consequential action sits, and what the system does when one agent in the chain fails or stalls. For the integration-level mechanics beneath these models, the Agentic Action Layer reference covers the tool-calling surface, and the Agentic AI Safety Case covers how autonomy tiers are scoped per action.
Engineering the Coordination Layer Against Its Known Failures
Because the MAST research names the failure modes, a security program can specify controls against each of them. The practices below translate the multi-agent failure literature and incident-command doctrine into requirements a security director can write into an RFP and an architect can verify.
Preserve the full evidence packet through every handoff
Context loss on handoff is a leading inter-agent failure. The control is a non-negotiable rule that every agent-to-agent handoff carries the structured evidence packet, object class, confidence, camera, timestamp, and annotated frame, rather than a compressed verdict. A downstream agent should never act on a conclusion it cannot inspect, and a human should never be asked to confirm one. Evidence preservation is also what makes the whole pipeline auditable after the fact.
Enforce unity of command in the agent graph
Specification and design issues are the single largest MAST failure category, and duplicate or ambiguous ownership is at their core. The control is an explicit map of which agent owns which decision, with no decision owned by two agents. Borrowed straight from NIMS: every worker answers to one orchestrator, and the orchestrator, not a worker, arbitrates conflict.
Bound the orchestrator's span of control
An orchestrator tracking too many workers degrades the same way a human supervisor does past seven subordinates. The control is to cap the worker count under any one supervisor and introduce hierarchy when the pipeline grows wide, rather than letting one orchestrator track an unbounded fan-out. Wide systems need supervisors of supervisors, not a heroic single coordinator.
Make termination and conflict explicit
Missing termination conditions and unresolved conflicting outputs are named MAST modes. The control is an explicit termination signal for every workflow, so a resolved event stops re-escalating, and a defined arbitration rule for when two agents disagree, so the system has a deterministic answer rather than a race condition. In a security pipeline, an unresolved conflict between a "lockdown" and a "stand down" recommendation cannot be left to chance.
Instrument the loop, not just the agents
Single-agent accuracy metrics do not describe a multi-agent system's health. The control is loop-level instrumentation: end-to-end coordination latency, handoff success and context-preservation rates, conflict-arbitration frequency, and per-agent stall and timeout rates. These are exactly the reasoning-trace and logging artifacts the Audit and Governance framework for agentic physical security AI requires, and they double as the documentation regulators and insurers increasingly request.
The NIST AI Risk Management Framework's GOVERN function expects organizations to document the configuration and known failure modes of their AI systems. For a multi-agent deployment, that documentation now has to cover the coordination layer explicitly: the agent graph, decision ownership, handoff contracts, and conflict-arbitration rules. The full procurement context is analyzed in our NIST AI RMF standards-compliance briefing; the orchestration-specific takeaway is that "how the agents coordinate" is becoming a compliance artifact, not just an engineering detail.
What This Means for Your Next Procurement
Security leaders evaluating AI weapon detection or broader computer-vision risk platforms with agentic capabilities can convert this report into five orchestration-specific RFP questions:
- Is your platform a single agent or a multi-agent system, and if multi-agent, what coordination pattern, supervisor, peer-to-peer, or hybrid, governs how the agents work together?
- What exactly is carried in a handoff between agents, and can you show that the full detection evidence packet, not a compressed verdict, survives to the action layer?
- Which agent owns each consequential decision, where does the human checkpoint sit, and how does the system prevent two agents from both acting on the same event?
- What is your end-to-end coordination latency distribution, not the average, from threat-visible-in-frame to a recommendation reaching the human checkpoint?
- Do your agents coordinate over emerging open standards such as A2A and MCP, or a proprietary handshake, and what happens to the pipeline when one agent in the chain stalls or fails?
Vendors with a coherent orchestration architecture answer these with diagrams and latency distributions. Vendors without one answer with the word "autonomous." That difference is observable in a one-hour technical session, before any pilot begins, and it predicts deployment success better than headline detection accuracy. For the broader evaluation methodology that surrounds these questions, see our proof-of-concept methodology report, and for the autonomous-architecture foundation, the Agentic Security Operations Center architecture reference.
Frequently Asked Questions
What is multi-agent orchestration in physical security?
Multi-agent orchestration is the architecture layer where several specialized AI agents divide a security workflow and hand work to one another: a detection agent classifies an event, a context agent corroborates it, a decision agent recommends a response, and an action agent executes on human authorization. Orchestration is the connective tissue that decomposes the event, routes work to the right agent, carries shared state across every handoff, and arbitrates when agents disagree. It is distinct from how any single agent detects a threat; it governs how the agents stay coordinated.
Why do multi-agent AI systems fail more than single agents?
Because they add a failure surface single agents do not have. The first systematic study of multi-agent LLM failures, the MAST taxonomy from a UC Berkeley-led team, analyzed over 1,600 traces and found failure rates as high as 86.7 percent, with nearly 79 percent of failures coming from specification and design issues plus inter-agent misalignment, categories that exist only because there is more than one agent. The lesson is not that multi-agent systems are bad, but that their dominant risk is coordination, not detection quality.
What is the supervisor pattern in agentic security?
The supervisor pattern uses one orchestrator agent to classify an incoming event, decompose it into sub-tasks, dispatch each to a specialized worker agent, and integrate their outputs into a single recommendation. Industry analyses report it accounts for the majority of multi-agent systems that survive production, because it gives coordination a clear owner. Its documented cost is latency, since each supervisory step takes time before workers begin, which matters acutely in security where the whole value is compressing response seconds.
How does the NIMS incident command system relate to AI agent coordination?
The coordination problems multi-agent AI is rediscovering were solved in the human domain by incident command. FEMA's National Incident Management System enforces unity of command (each responder reports to exactly one supervisor) and span of control (one supervisor per three to seven units), the same principles that prevent agent orchestration from producing conflicting directives and overloaded coordinators. A multi-agent security architecture that cannot articulate its unity-of-command boundaries is reinventing doctrine emergency management already proved.
What are MCP and A2A, and why do they matter for security buyers?
The Model Context Protocol standardizes how a single agent connects to tools and data; the Agent2Agent protocol, or A2A, standardizes how autonomous agents discover, authenticate, and delegate work to one another across frameworks and organizations. A2A was donated to the Linux Foundation in June 2025 and reached a production-ready v1.0 by April 2026 with support from over 100 companies. For buyers, whether a vendor's agents coordinate over open standards rather than a proprietary handshake is both a lock-in question and a resilience question, the agentic-era equivalent of asking whether a camera speaks ONVIF.
Does multi-agent orchestration mean the AI acts without human approval?
No, and the two questions should be kept separate. Orchestration governs how agents coordinate; authority governs who approves consequential action. In a sound architecture the orchestrator coordinates the workflow but never holds authority for consequential actions such as a lockdown; those pass through a human checkpoint. An orchestration bug should be able to misroute or stall work, but it should never be able to trigger a physical action on its own. That separation is the multi-agent expression of the human-in-the-loop principle.
How does IntelliSee approach multi-agent coordination?
IntelliSee's design keeps detection, corroboration, recommendation, and execution as separable responsibilities, preserves the full annotated evidence packet (object class, confidence, camera, timestamp, frame) through every handoff so downstream steps and human reviewers can inspect what they act on, and keeps authority for consequential actions at a human checkpoint rather than with any coordinating layer. The platform performs no facial recognition, stores no video, and collects no personal information, and it holds DHS SAFETY Act Designation. Security teams can request a risk assessment to map their current architecture against the coordination framework in this report.
Conclusion: Coordination Is the New Detection
For a decade the AI security conversation was about detection accuracy, and the next platforms will keep competing on it. But the architectures buyers are now being shown are multi-agent, and the first rigorous evidence on multi-agent systems is unambiguous: between two platforms with comparable detection quality, deployment outcomes will be decided by the coordination layer. Whether context survives every handoff, whether decision ownership is unambiguous, whether the orchestrator's span stays bounded, and whether authority for consequential action stays with a human rather than a coordinator. These are not detection questions. They are command-and-control questions, and emergency management answered the human version of them five decades ago.
Security leaders do not need to read the multi-agent systems literature. They need to procure as if its central finding is true, because it is: in a multi-agent system, the coordination layer is where it succeeds or fails, and that layer should be specified, interrogated, and audited with the same rigor once reserved for the detection model.
IntelliSee provides no-cost risk assessments that map your existing detection, video management, access control, and notification infrastructure against the coordination framework in this report. Contact our team to schedule a structured conversation about how an agentic layer would coordinate across your stack.
Continue the research: The Agentic Action Layer: MCP, Tool Calling, and the Engineering Patterns That Connect AI Reasoning to Response | Human-in-the-Loop AI Security: Trust Calibration and the Verification Layer | Audit and Governance for Agentic Physical Security AI
Primary Sources and Citations
- Cemri, M., Pan, M.Z., Yang, S., Agrawal, L.A., Chopra, B., Tiwari, R., Keutzer, K., Parameswaran, A., Klein, D., Ramchandran, K., Zaharia, M., Gonzalez, J.E., & Stoica, I. (2025). Why Do Multi-Agent LLM Systems Fail? arXiv:2503.13657 (v3, October 2025). MAST taxonomy and MAST-Data, 1,600+ annotated traces across 7 frameworks.
- Linux Foundation / Google Open Source. (2025-2026). Agent2Agent (A2A) Protocol. Announced April 2025; donated to the Linux Foundation June 23, 2025; A2A Protocol v1.0 and one-year anniversary, April 2026.
- Federal Emergency Management Agency. National Incident Management System (NIMS). Unity of command and span-of-control (one supervisor per 3-7 units) management characteristics; Incident Command System foundations.
- National Institute of Standards and Technology. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1, GOVERN function on human-AI configuration and documented failure modes.
- Anthropic. (2024). Model Context Protocol. Open standard for connecting AI systems to tools and data sources.
More intelligence like this
New IntelliSee research drops monthly at most. Subscribe and get the next sector playbook, technology briefing, or threat intelligence report in your inbox the day it ships.
Request a Risk Assessment
Talk to an IntelliSee security specialist. No sales pitch — a structured conversation about your environment, your threat profile, and whether computer vision is the right fit.
Request a Risk Assessment