Edge vs. Cloud AI Inference in Physical Security: A Technology Briefing on Where the Model Runs, Latency Reality, Privacy Architecture, and Hybrid Patterns Reshaping Buyer Evaluation
A primary-source-grounded reference on where AI video inference runs, the latency arithmetic, regulatory implications, and the hybrid edge-cloud patterns now dominating real-world deployments.
Where the AI model runs has quietly become the most consequential procurement question in physical security.
Edge vs cloud AI inference in physical security used to be an architecture trivia question. It is now a procurement question with measurable consequences for latency, privacy posture, regulatory exposure, and operating cost. Buyers who ask vendors only what the model detects, without asking where the model runs, are missing the layer of the architecture that determines whether the system can actually meet its detection-time, data-residency, and resilience promises.
This briefing is a primary-source-grounded reference for security directors, IT architects, and procurement officers evaluating AI physical security platforms. It defines the four common (and often conflated) meanings of "edge AI," walks the millisecond-by-millisecond latency math, summarizes the privacy and regulatory implications, and lays out the four hybrid patterns that now dominate real-world deployments. It draws on published silicon specifications from NVIDIA, Hailo, and Axis Communications; the ONVIF Profile M interoperability standard; IEC 62676-4:2025; and the NIST AI Risk Management Framework.
Why edge vs cloud AI inference became a procurement question in 2026
Three forces converged between 2023 and 2026 to push the where-does-the-model-run question from infrastructure trivia to executive procurement scrutiny.
First, silicon caught up to the workload. A modern edge AI accelerator now delivers what required a server-class GPU a generation ago. NVIDIA's Jetson AGX Orin module provides up to 275 TOPS (trillion operations per second) of AI compute in a footprint suitable for an on-premises 1U appliance or even a camera-adjacent enclosure, drawing roughly 60 watts at full load. At the lower end of the spectrum, the Hailo-8 accelerator delivers 26 TOPS at a typical 2.5-watt draw, with 8.25 watts at maximum utilization. Axis Communications shipped its ninth-generation ARTPEC system-on-chip in early 2025, tripling on-camera analytics throughput compared to its predecessor and bringing meaningful deep-learning inference directly onto the camera itself. These are not theoretical specifications; they are commercially shipping silicon that resets the assumption that "real" AI requires a data-center destination.
Second, latency requirements tightened. The threat-detection promises that physical security vendors began making in 2022 and 2023 (sub-30-second detection-to-alert, sub-10-second weapon detection, "real-time" perimeter response) are achievable only when the latency budget is allocated correctly. End-to-end cloud-routed inference commonly consumes 100–300 milliseconds before any application logic runs, while edge inference on the same model can complete in 5–20 milliseconds. The difference is not academic when downstream response systems depend on it. A 250-millisecond detection delay multiplied across a 30-frames-per-second analysis pipeline is the difference between catching a behavior at frame one and reacting to it at frame eight or nine.
Third, the regulatory environment changed. The patchwork of U.S. state biometric privacy laws (BIPA, CUBI, MHMDA), the EU AI Act's risk categorization of physical security AI, federal supply-chain restrictions under NDAA Section 889, and emerging healthcare and education sector-specific data-residency rules all interact with the question of where AI processing occurs. A vendor that pipes raw video into a hyperscale cloud endpoint for inference creates a fundamentally different compliance profile than one that runs the same model on a dedicated appliance behind the customer's firewall. See the related Intelligence briefings on biometric privacy compliance and the EU AI Act for how this interacts with downstream legal exposure.
What "edge AI" actually means: four definitions vendors routinely conflate
The phrase "edge AI" carries at least four distinct technical meanings in physical security vendor marketing. Buyers who do not distinguish between them are not comparing the same architecture across vendors.
1. On-Camera Inference
The model runs on a deep-learning-capable system-on-chip inside the camera itself, such as Axis Communications' ARTPEC-9 or Hanwha's Wisenet-equivalent silicon. Detection happens before the video stream ever leaves the camera. Bandwidth and latency are minimized; the trade-off is that the model size, frame rate, and detection class diversity are constrained by what the embedded SoC can sustain.
2. On-Premises Appliance Inference
The model runs on a dedicated server or 1U rack-mounted appliance located inside the customer's network, processing feeds from multiple cameras. Compute is substantially higher than on-camera SoCs (a Jetson AGX Orin appliance handles dozens of feeds concurrently); the trade-off is a physical asset to install and maintain. This is the architecture IntelliSee uses for sustained multi-camera deployments.
3. Customer-Region Cloud Inference
The model runs in a cloud region near the customer's geography, with video streamed continuously to a hyperscale inference endpoint. Vendors frequently brand this as "edge cloud" or "regional edge." It reduces some of the latency penalty of distant data centers but does not eliminate network round-trip time, and it does not change the data-residency footprint: full video frames or compressed streams still leave customer premises.
4. Multi-Region Cloud Inference with Edge Caching
The detection model runs in the cloud, with metadata or model weights cached at distributed points of presence. This is the architecture pattern most distant from true edge inference but most commonly described as "edge AI" in vendor marketing. Latency, bandwidth, and privacy implications resemble traditional cloud architectures more than edge ones.
The distinction matters because vendor claims about latency, privacy, and resilience can be true at one level of "edge" and entirely untrue at another. A platform marketed as "edge AI" that streams full-resolution video to a cloud inference endpoint for model evaluation is not architecturally comparable to one that processes the same frames on an appliance inside the customer's server room. For a deeper reference on how computer vision architecture varies across deployment patterns, the technical reference on AI gun detection and the architectural shift from CCTV to real-time detection walk through the underlying model and runtime stack.
The latency arithmetic: where milliseconds actually accumulate
Latency in an AI video analytics pipeline is the sum of distinct stages, and the location of inference determines which stages dominate. Understanding this math is what separates buyers who can stress-test vendor claims from buyers who accept marketing latency numbers at face value.
A typical cloud-routed inference pipeline accumulates latency in this order: frame capture from the camera sensor (1–5 ms depending on shutter and pipeline), in-camera encoding (5–15 ms for H.264 or H.265, sometimes higher for AV1), network upload from camera to cloud ingress (50–150 ms typical, highly variable with network conditions), queue wait at the cloud inference service (10–50 ms depending on tenancy and concurrency), GPU inference itself (20–50 ms on a server-class accelerator for a typical CV model), result return network leg (50–150 ms), and finally local application response logic (5–20 ms). End-to-end totals routinely land between 150 and 400 milliseconds, with high-variance outliers when network conditions degrade.
The same model running on an on-premises edge appliance bypasses most of those stages. Frame capture, encoding, and a short LAN hop (under 1 ms) precede direct inference on a local accelerator (10–30 ms on a Jetson AGX Orin running a typical detection model), with results dispatched immediately to application logic over the same local network. End-to-end totals of 15–50 milliseconds are common, with low variance because the network leg is internal and predictable.
The 10-to-20-times latency advantage of edge over cloud inference is not a marketing claim; it is the structural consequence of where the milliseconds accumulate. The infographic below traces the latency arithmetic side-by-side and tags which stages are eliminated when inference moves on-premises.
Cloud vs. edge inference latency stages, the same model on the same camera frame
Both pipelines analyze the same 1080p frame with the same detection model. Only the inference location differs. The accumulated latency is structural, not theoretical.
The takeaway is not that cloud inference is unworkable for physical security. It is that the latency budget cloud inference consumes is real, network-variable, and accumulates whether the marketing literature mentions it or not. Some workloads tolerate it. Detection-time-critical workloads, particularly drawn-weapon detection and active-shooter alerting, do not.
Why edge inference is the BIPA-compatible default for physical security AI
State biometric privacy statutes like Illinois BIPA, Texas CUBI, and Washington's My Health My Data Act regulate the collection, transmission, and storage of biometric identifiers, not the act of looking at a video frame in real time. An architecture that processes a frame on a local appliance, computes a non-identifying detection signal (a drawn firearm, a person in a restricted zone), and discards the frame creates a fundamentally different data-protection profile than one that streams the same frame to a third-party cloud for inference. The model in both cases performs object detection, not facial recognition. But the data-residency footprint, the audit-trail complexity, and the breach-notification exposure are categorically different. Where the model runs is the architectural variable that determines which compliance regime applies.
Privacy, residency, and the regulatory case for edge processing
The NIST AI Risk Management Framework, published as NIST AI 100-1 in January 2023, identifies seven trustworthiness characteristics that AI systems should satisfy, two of which interact directly with the inference-location question: privacy enhancement and security/resilience. The framework does not prescribe edge or cloud; it requires that organizations be able to characterize, manage, and document where data flows and what privacy controls apply at each stage. The practical implication for physical security buyers is that the documentation burden is materially lower when video processing remains on-premises.
The same logic extends across the regulatory environment. In healthcare, HIPAA's Security Rule requires reasonable safeguards over the systems that might incidentally capture protected health information, and reduces administrative overhead substantially when video does not traverse third-party networks for inference. In education, the Family Educational Rights and Privacy Act (FERPA) and state student-privacy statutes create similar dynamics. In federal contracting, the supply-chain restrictions of NDAA Section 889 and FAR 52.204-25 (covered in the related supply-chain compliance briefing) interact with the inference-location question because cloud vendors with restricted-origin components in their stack create exposure that on-premises deployments simply avoid.
The IEC 62676-4:2025 standard, published October 9, 2025, replaces the 2014 edition of the video surveillance application guidelines and explicitly addresses bandwidth, latency, and lifecycle requirements for modern IP-based and AI-augmented video surveillance systems. The standard does not mandate edge inference, but it does require that operators specify and validate latency and image-handling requirements as part of the system design, which makes inference location an explicit specification line rather than an implicit vendor choice.
ONVIF Profile M, finalized June 30, 2021 and now ubiquitous across major analytics-capable cameras, is the interoperability standard that lets edge and cloud components from different vendors exchange analytics metadata without proprietary integration. Profile M is what makes hybrid edge-cloud architectures practically deployable: a camera with on-board detection can emit standardized metadata to a third-party VMS or cloud orchestrator, while inference continues to run where the operator chose to run it.
The four hybrid patterns reshaping real-world deployments
Most production physical security deployments in 2026 are neither pure-edge nor pure-cloud. They are hybrids that allocate specific functions to specific layers based on latency tolerance, regulatory posture, and operating cost. Four patterns dominate.
Pattern 1: Edge inference, cloud orchestration. Detection runs on-premises (on-camera, on-appliance, or both). Alerting, dashboarding, multi-site aggregation, and historical analytics happen in a cloud orchestration layer. Video itself does not leave the customer network; only the standardized metadata (detection events, confidence scores, location tags) traverses the boundary. This is the dominant pattern for healthcare, education, and federal-adjacent deployments where data-residency posture and detection-time requirements both matter. IntelliSee operates in this architecture: the model runs at the customer site, the orchestration layer aggregates events for cross-site analytics and reporting.
Pattern 2: Edge inference, on-premises orchestration, optional cloud reporting. Detection and orchestration both run inside the customer network, with optional one-way reporting to a cloud dashboard for executive visibility. This pattern is common in critical-infrastructure deployments and federal facilities where any outbound data flow is treated as a controlled boundary. It is the most defensible architecture against data-exfiltration and network-disruption threat models.
Pattern 3: Cloud inference with edge pre-filtering. A lightweight motion or change-detection model runs at the edge (or on-camera) and uploads only frames that show meaningful activity. Heavy inference happens in the cloud on the filtered subset. Bandwidth is reduced compared to streaming full video continuously, but the privacy and latency profile remains cloud-shaped because full frames of the events of interest are still transmitted off-premises.
Pattern 4: Cloud inference, cloud orchestration, edge alerting. Full or near-full video streams to the cloud for inference. Detection events return to local alerting endpoints. This is the most network-dependent of the four patterns and the most exposed to the latency arithmetic above. It is also the easiest to deploy from a hardware-procurement standpoint, which keeps it common in retail and certain commercial real estate deployments where the latency and privacy budgets are more forgiving.
Inference Location Trade-Offs Across Architecture Patterns
| Dimension | Pure Edge | Hybrid (Edge Inference, Cloud Orchestration) | Pure Cloud |
|---|---|---|---|
| End-to-end latency | 15–50 ms typical | Inference: 15–50 ms · Cross-site analytics: cloud-bound | 150–400 ms typical |
| Data leaving customer network | None for detection | Metadata only, no raw video | Full video frames |
| Network-outage resilience | Detection continues | Detection continues; cloud reporting pauses | Detection stops |
| Bandwidth profile | Internal LAN only | Low (metadata) | High (continuous video) |
| Operating cost driver | Appliance hardware lifecycle | Appliance + cloud orchestration tier | Continuous cloud inference + egress |
| Privacy posture | Strongest | Strong (no raw video off-premises) | Cloud-tenant model dependent |
| Suitable detection workloads | Latency-critical: weapons, falls, perimeter | Most production workloads | Forensic, lower-stakes alerting |
The reason hybrid Pattern 1 dominates in detection-time-critical deployments is that it preserves the latency, privacy, and resilience advantages of edge inference while still giving operators the cross-site visibility, fleet management, and executive reporting that a cloud orchestration layer provides. The cloud is doing reporting work, not detection work; the model is doing detection work, not reporting work; and the boundary between the two carries only the standardized metadata that ONVIF Profile M and the operator's own integration policies allow.
How security leaders should evaluate where inference runs in a vendor's stack
Security directors and procurement officers evaluating an AI physical security platform should treat the inference-location question with the same rigor as encryption, retention, and access-control questions. The following lines belong in any serious RFP or vendor questionnaire.
Specify the inference location for each detection category. Ask the vendor where, physically, the model evaluates a frame for each detection class. The answer should reference specific hardware, specific software runtime, and specific network topology. A vague answer ("our edge architecture runs at the edge") is a flag. A concrete answer ("the gun detection model runs on an on-premises 1U appliance in the customer server room, inference time 14–18 ms per frame on a Jetson AGX Orin module") is the bar.
Specify the data that leaves the customer network for each detection event. Ask whether any frames, video segments, or biometric features are transmitted off-premises during normal operation. Ask the same question for support, debug, and model-improvement scenarios. The bar is an explicit per-data-class disclosure, not a generic privacy statement.
Specify the resilience profile during network disruption. Ask what the system does when the customer's internet connection is unavailable. Detection should continue; reporting may pause. A platform whose detection layer stops working when the cloud is unreachable has an availability dependency that needs to be priced into the procurement decision.
Specify the standards conformance. ONVIF Profile M support for analytics metadata interoperability is now a baseline expectation. IEC 62676-4:2025 conformance for video surveillance system specification is an emerging baseline. NIST AI RMF alignment for risk documentation is a near-certain procurement requirement for federal and federal-adjacent buyers.
Specify the upgrade and lifecycle path for the inference hardware or service. Edge appliances have hardware refresh cycles; cloud services have versioning and deprecation policies. Both create future cost and risk. Ask for the vendor's published or contractual policy on each.
What "real-time" actually means once you trace the network path
Vendors describe their platforms as "real-time" with varying degrees of precision. Procurement teams can stress-test the claim with one question: trace the path of a single frame from the moment the sensor captures it to the moment a designated responder receives the alert, naming each hop and timing each transition. If the vendor cannot produce that trace, the "real-time" claim is not validated. If the trace shows full video frames crossing the customer-to-cloud boundary, the platform is real-time within cloud architectural constraints, which is not the same as real-time within the operator's latency budget. The trace is the documentation that turns marketing language into engineering specification.
Frequently asked questions about edge vs. cloud AI inference in physical security
Does edge inference mean the AI model runs inside the camera itself?
Not necessarily. "Edge" is a relative term, and in physical security it refers to any inference location that sits inside the customer's own network rather than in a third-party cloud. On-camera inference (the model runs on a deep-learning-capable SoC in the camera, such as Axis ARTPEC-9 or comparable silicon) is one form. On-premises appliance inference (a dedicated server in the customer's server room evaluates feeds from many cameras) is another. Both are edge architectures; both keep video off the public network. The architectural difference between them shows up in cost, model size, and concurrent camera capacity, but not in the regulatory or latency posture relative to a cloud-based alternative.
Why does edge inference have lower latency than cloud inference if cloud GPUs are faster?
Cloud GPUs are often faster at the inference step itself than edge accelerators (a server-class H100 outperforms a Jetson AGX Orin on raw throughput). But the inference step is only one stage of the end-to-end pipeline. Network upload to the cloud, queue wait at the inference service, and the return network leg add 150 to 300 milliseconds that an edge architecture eliminates entirely. The total advantage of edge is the sum of those eliminated stages, not the per-frame inference time itself. Latency is end-to-end, not just per-model.
Is cloud inference always wrong for physical security?
No. Cloud inference is appropriate for workloads where latency tolerance is high (forensic search, post-event analytics, license plate recognition for non-real-time use, business intelligence dashboards) and where the regulatory and data-residency posture is compatible with sending video off-premises. The architectural mistake is not using cloud inference; it is using cloud inference for workloads whose latency, privacy, or resilience requirements are incompatible with the cloud's structural properties. Weapon detection, active shooter alerting, fall detection, and behavioral health unauthorized-access detection sit in that incompatible category for most deployments.
How do hybrid architectures handle multi-site deployments?
The dominant hybrid pattern (edge inference, cloud orchestration) handles multi-site deployments by running detection independently at each site while aggregating standardized metadata to a central cloud orchestration layer. The orchestration layer provides cross-site visibility, fleet management, alert escalation, and executive reporting. Each site's detection remains operational even if the cloud orchestration layer is briefly unavailable, because the model itself runs locally. This is why the pattern dominates in regulated multi-site environments such as hospital systems, K-12 districts, and federal facility networks.
What happens to detection accuracy when models run on edge accelerators rather than server GPUs?
Modern edge accelerators (Jetson AGX Orin, Hailo-8 and -15, Axis ARTPEC-9, Ambarella CVflow SoCs) support the same model architectures used in data-center inference, typically with INT8 quantization that introduces marginal accuracy reduction (commonly 1–3 percentage points on standard benchmarks) in exchange for substantial throughput and power gains. For physical security detection tasks (weapons, falls, perimeter intrusion, loitering), the accuracy delta between an INT8-quantized model on an edge accelerator and an FP16 model on a server GPU is typically not the binding constraint; sensor quality, lighting conditions, and training-data coverage matter more. The briefing on computer vision under adversarial conditions covers these failure modes in depth.
How does ONVIF Profile M relate to the edge vs. cloud question?
ONVIF Profile M, released June 30, 2021, is the interoperability standard for analytics metadata and events between cameras, analytics services, video management software, and cloud services. It does not specify where inference must occur. It standardizes the format and protocol by which detection metadata is exchanged across architectural boundaries. Profile M is the reason hybrid edge-cloud deployments work practically: an on-camera or on-appliance detection layer can emit standardized metadata to a different vendor's VMS or to a cloud orchestrator without proprietary integration glue. For buyers, Profile M conformance on both the detection layer and the management layer is now a baseline expectation.
If we already have a cloud-based VMS, can we still add edge inference?
Yes, and this is one of the most common deployment patterns. Edge inference appliances integrate with cloud-based VMS systems through ONVIF Profile M, RTSP feeds, and standardized webhook or API event integration. The detection layer runs on-premises (preserving the latency and privacy advantages of edge), while the cloud VMS handles video management, retention, and operator workflows. The architectural decision is not all-or-nothing; it is a layered choice where each layer's location is selected on its own merits. The platform overview walks through the integration topology with major VMS vendors.
Continue the research
This briefing covers the architecture-level case for choosing where AI inference runs in a physical security deployment. For deeper reading on specific pieces of the technology stack:
- AI Video Analytics vs. Traditional CCTV — the foundational technology briefing on the shift from recording-first to real-time detection-first architectures.
- How AI Gun Detection Works — a technical reference on the model architecture, training pipeline, and inference runtime behind firearm detection.
- Computer Vision Under Occlusion, Low Light, and Adversarial Conditions — the failure-mode analysis that complements this briefing's architecture analysis.
- The Definitive 2026 Guide to Proactive Computer Vision — the cross-cutting reference that situates edge inference within the broader proactive-detection paradigm.
- How IntelliSee deploys — the operational view of the hybrid edge-inference, cloud-orchestration pattern in practice.
More intelligence like this
New IntelliSee research drops monthly at most. Subscribe and get the next sector playbook, technology briefing, or threat intelligence report in your inbox the day it ships.
Request a Risk Assessment
Talk to an IntelliSee security specialist. No sales pitch — a structured conversation about your environment, your threat profile, and whether computer vision is the right fit.
Request a Risk Assessment