How to Evaluate Agentic Physical Security AI: The 2026 Briefing on Agent Hijacking, Red-Team Evals, and Continuous Evaluation

To evaluate agentic physical security AI, you have to start from what makes it different: an agentic system does not just detect, it decides, and then it acts. It reads a camera stream, reasons about what it is seeing, calls tools that lock a door, escalate to a monitoring center, or dispatch to a public-safety answering point, and it does this with a degree of autonomy that a passive alerting system never had. That autonomy is the entire value proposition, and it is also the entire risk. The question every security director, risk officer, and procurement team now has to answer is not whether the model is accurate. It is whether the agent behaves the way it is supposed to behave when an adversary, a malformed input, or an unfamiliar scene is trying to make it behave otherwise.

That question is an evaluation problem, and the evaluation science for agentic AI matured sharply between early 2025 and mid-2026. Federal standards bodies, open-source research groups, and the adversarial-security community converged on a shared vocabulary for how autonomous agents fail and a shared set of methods for measuring those failures before deployment and continuously after it. This briefing is a technical reference on that evaluation architecture: what agent hijacking is, how the benchmark harnesses actually work, why a one-shot accuracy test is the wrong instrument, and how a physical security buyer should structure a proof-of-concept and an ongoing evaluation program around an agent that is allowed to touch the physical environment.

The evaluation problem agentic security inherits from the rest of AI

An agent is an AI system that plans over multiple steps and calls tools to affect the world, rather than returning a single answer. The moment a model can call tools, the security surface changes fundamentally, and it changes in a way that traditional detection accuracy does not capture. A gun-detection model can be scored on precision and recall against a labeled test set. An agent that receives a gun detection and then decides whether to trigger a lockdown has to be scored on something harder: does it take the correct action, only the correct action, and nothing an attacker planted in its input asked it to do instead.

The core failure mode has a name. NIST's Center for AI Standards and Innovation defines agent hijacking as a form of indirect prompt injection, in which an attacker inserts malicious instructions into data that an agent ingests, causing it to take unintended, harmful actions. As NIST frames it, this is a modern incarnation of an old computer-security problem: a system that lacks a clean separation between trusted instructions and untrusted data. Current LLM-based agents combine developer instructions and task-relevant data into one input stream, and an attacker who can shape any part of that stream, an email, a file, a caption, a sign held up to a camera, a manipulated metadata field, can try to smuggle in a command the agent was never meant to follow. NIST published this analysis in its January 2025 technical blog on strengthening agent-hijacking evaluations, drawing on the adversarial-machine-learning taxonomy it maintains in NIST AI 100-2.

The reason this matters for physical security specifically is that the consequence of a hijacked action is not a leaked spreadsheet. It is a door that unlocks, an alarm that is suppressed, or a dispatch that is cancelled. The blast radius of an autonomous action in the physical world is measured in seconds and in people, which is why the evaluation bar for an agent that can act on a physical environment has to be higher than the bar for a chatbot. IntelliSee keeps a human in the verification loop for consequential actions for this reason, a design choice we examine in our briefing on human-in-the-loop AI security and trust calibration, and the evaluation methods in this report are how you would prove that the loop actually holds.

How an agent-hijacking benchmark actually works

The reference tool the field standardized on is AgentDojo, an open-source evaluation environment built by researchers at ETH Zurich and published as arXiv:2406.13352. AgentDojo is worth understanding in detail, because its structure is the template a physical security buyer should borrow when designing an agent proof-of-concept. It is not a static list of questions. It is a dynamic environment in which an agent is given a set of tools and asked to complete realistic user tasks while adversarial content is planted in the data it will encounter along the way.

The published benchmark contains 97 realistic user tasks and 629 security test cases across four simulated environments, workspace management, banking, travel booking, and an online store, with 70 tools and 27 injection targets. The fundamental unit is the hijacking scenario: the agent is asked to complete a legitimate task, but the environment contains an injection task, a malicious instruction the attacker is trying to get the agent to perform instead. Crucially, AgentDojo provides ground-truth evaluation functions that programmatically verify two separate things at once: did the agent complete the legitimate task (utility), and did it also complete the malicious injection task (security failure). That joint measurement of utility and security is the single most important idea for a physical security buyer to take away, because a defense that stops every attack by refusing to do anything useful is not a defense. It is a broken product.

The image below is the concrete anchor for what all of this is protecting. It is an actual IntelliSee detection output: a weapon flagged in a building entrance with a bounding box and a confidence score. In an agentic pipeline, this detection is the legitimate task that starts the chain. Everything downstream, the escalation, the notification, the lockdown decision, is what an evaluation harness stress-tests to confirm the agent acts on this signal and only this signal.

Live Detection CAM 01 · ENTRANCE

Why one test run is the wrong instrument

When NIST's technical staff ran AgentDojo against a leading commercial model, they surfaced four evaluation insights that overturn the intuition most buyers bring to a proof-of-concept. The first is that a model robust to yesterday's attacks is not robust, full stop. NIST found that a hardened version of a leading model resisted previously published hijacking attacks well, but when its red team, working with the UK AI Security Institute, developed attacks optimized for that specific model, the attack success rate rose from 11% for the strongest baseline attack to 81% for the strongest new attack. The lesson is that evaluation has to be adaptive and adversarial. A canned test suite that a vendor has already trained against tells you almost nothing.

The second insight is that aggregate scores hide the risk that matters. NIST showed that averaging attack success across a mix of injection tasks produced a single number, 57%, that concealed enormous variation underneath. Some injection tasks, sending a benign email, succeeded far more often than the average, while a high-consequence task like mass data exfiltration succeeded less often but was catastrophic when it did. For a physical security agent, this maps directly: an evaluation that reports one blended number is useless. You need the success rate broken out by consequence tier, so that a low-frequency but high-severity failure, an agent that can be induced to suppress an alarm, is never buried under a pile of harmless ones.

The third insight is the one that most directly changes how a proof-of-concept should be run. Because language models are probabilistic, the same attack tried repeatedly produces different outcomes, and an attacker who can retry cheaply will eventually get through. When NIST re-ran a set of injection tasks 25 times each instead of once, the average attack success rate climbed from 57% to 80%. A single clean test pass is not evidence of safety. It is a coin flip that happened to land heads. Any credible agent evaluation, in a lab or in a POC, has to measure attack success over many attempts, not one.

The fourth insight closes the loop: shared evaluation frameworks have to be continuously improved to keep pace with attackers. NIST extended AgentDojo with new high-consequence injection tasks, remote code execution, database exfiltration, and automated phishing, then open-sourced its improvements as AgentDojo-Inspect. The evaluation harness is a living system, not a certificate you earn once.

The shared threat vocabulary: OWASP and MITRE ATLAS

Evaluation needs a taxonomy of what to test for, and two authoritative catalogs now supply it. The OWASP Gen AI Security Project released the OWASP Top 10 for Agentic Applications 2026 in December 2025, developed with input from more than a hundred practitioners. It ranks the ten most critical risk classes for autonomous agents, and its ordering is a direct signal of where evaluation effort belongs. Agent Goal Hijack sits at the top as ASI01, the same failure mode NIST measures, followed by Tool Misuse and Exploitation (ASI02), Agent Identity and Privilege Abuse (ASI03), Agentic Supply Chain Compromise (ASI04), Unexpected Code Execution (ASI05), and Memory and Context Poisoning (ASI06), through to Rogue Agents at the bottom of the list. For a security agent that holds credentials to a video management system, an access-control platform, and a mass-notification tool, ASI02 and ASI03 are not abstract, compromising the agent is functionally equivalent to compromising every tool it can reach.

Where OWASP supplies the risk ranking, MITRE ATLAS supplies the adversary playbook. ATLAS, the Adversarial Threat Landscape for Artificial-Intelligence Systems, is a living knowledge base of real-world adversary tactics and techniques against AI systems, modeled on the ATT&CK framework the cybersecurity community already knows. As of 2026 it documents on the order of 16 tactics and roughly 170 techniques drawn from observed attacks and red-team demonstrations, and its late-2025 and early-2026 updates added a wave of techniques aimed squarely at autonomous agents: context and memory poisoning, agent configuration tampering, tool-credential harvesting, and exfiltration through tool invocation. A practical detail for security operations teams is that a large share of ATLAS mitigations map onto controls a SOC already runs, which makes the framework something you can operationalize rather than a purely academic reference. We map how these agentic components fit together in our reference on the agentic action layer and tool-calling architecture, and the coordination risks in multi-agent orchestration and the supervisor pattern.

The four-layer evaluation stack for a security agent

Pulling the standards together produces a concrete evaluation architecture. It is not one test. It is four layers, each measuring a different property, and a credible agent, whether built in-house or bought, should be assessed against all four before it is granted any authority to act, and continuously afterward. The infographic below lays out the stack.

The value of the four-layer view is that it exposes how much of the real risk lives below the accuracy line. Layer 1 is where most vendor marketing operates, and it is genuinely necessary, an agent that misses weapons is useless before any of the harder questions apply. But layers 2 through 4 are where an autonomous system either earns the right to act or does not, and they are precisely the layers a one-page accuracy claim skips. A buyer who evaluates only layer 1 has confirmed the agent can do its job and learned nothing about whether it can be turned against them.

Evaluating a security agent versus a general-purpose agent

The public benchmarks were built around office-agent environments, email, banking, travel, and a physical security buyer has to translate them. The structure carries over cleanly, but the tasks, the injection surfaces, and the consequence tiers are different. The table below maps the general-purpose evaluation concept to its physical security equivalent.

The highlighted row is the design principle that separates a defensible physical security agent from a reckless one. In an office setting, an agent that is unsure can usually pause and ask. In physical security, the safe default on a consequential action, an unlock, an alarm suppression, a dispatch cancellation, is to fail closed and route to a human who can verify, rather than to act autonomously on an uncertain or possibly manipulated signal. This is the same logic that governs autonomy tiers and operational guardrails in our agentic AI safety case framework, and it is what layer 3 of the evaluation stack is built to measure.

From pre-deployment test to continuous evaluation

The most important shift in the 2026 evaluation consensus is that evaluation is not a one-time gate before launch. It is a program that runs for the life of the deployment. The reason is drift: the model provider ships an update without a version bump, the camera fleet changes, new scene types appear as the deployment expands, and an agent that passed every pre-deployment test silently starts behaving differently. Infrastructure metrics stay green during a pure quality regression, which is exactly why they miss it. The only mechanism that surfaces this class of failure is a continuous-evaluation loop.

In practice that loop has three moving parts. First, an offline regression suite that reruns the full battery of capability, adversarial, and safe-degradation scenarios after every model, prompt, or tool change, blocking a rollout if scores fall below defined thresholds. Second, a canary set of held-out scenarios replayed on the live production model on a schedule, teams running daily canary replays report catching silent drift on their production models that no infrastructure metric would have flagged. Third, production scoring, sampling a slice of live events, scoring them with automated evaluators and human review, and, critically, promoting every real-world failure into a permanent test case so the same failure can never recur unnoticed. This last practice, failures become tests, is what turns an evaluation program into a ratchet that only tightens.

An agent that passed every test on the day it launched can quietly become a different system three model updates later. Continuous evaluation is not a nice-to-have bolted onto deployment. It is the only instrument that sees a quality regression before an incident does.

For a physical security buyer, the continuous-evaluation requirement translates into a small number of concrete contract questions. Does the vendor run an offline regression suite, and will they show you the thresholds. Do they replay canary scenarios against the live model, and how often. When a real-world miss or false action happens in your deployment, does it become a permanent test case, and can you see that it did. A vendor who cannot answer these is selling a system that was evaluated once and is now running on faith. The auditability and logging that make this possible are the same evidence base we describe in our framework on audit and governance for agentic physical security AI, and the accountability questions they raise are the subject of our briefing on agentic AI liability and risk allocation.

What this means for a proof-of-concept

The through-line for a buyer is that the proof-of-concept is where evaluation either happens or does not, and most POCs are structured to demonstrate capability, layer 1, and nothing else. A POC that reflects the 2026 evaluation consensus looks different. It runs adversarial injection scenarios, not just clean detections, and it runs each of them many times to expose the multi-attempt escalation NIST documented. It reports results by consequence tier, so a rare but catastrophic failure is visible rather than averaged away. It deliberately feeds the agent out-of-distribution and degraded scenes to confirm it fails closed rather than guessing. And it establishes, before signing, how the continuous-evaluation loop will operate once the system is live. This is a natural extension of the discipline in our gun-detection proof-of-concept methodology, raised to the standard an autonomous, tool-calling agent demands.

None of this is a reason to avoid agentic security. Autonomy is what compresses the interval between a detection like the one above and a protective response, and that compression is where the safety and loss-cost benefit lives. The point is narrower and more useful: the systems that deserve autonomy are the ones that can prove, under adversarial and repeated testing, that they act correctly, resist manipulation, degrade safely, and stay that way over time. Evaluation is how that proof is produced. A vendor who welcomes a layer-2-through-4 evaluation is telling you something a glossy accuracy number never can. To pressure-test an agentic deployment against this standard, request a risk assessment, or start with the foundations in our definitive 2026 guide to proactive computer vision and the architecture in how IntelliSee works.

Frequently asked questions

Continue the research

This briefing is part of IntelliSee Intelligence's Agentic AI stream. To go deeper into the architecture and accountability of autonomous physical security, continue with our Agentic AI Safety Case framework for autonomy tiers and operational guardrails, our reference on the agentic action layer and tool-calling patterns, and our analysis of human-in-the-loop verification and trust calibration. For the full library, visit the IntelliSee Intelligence Hub.