Article 2

Digital Security Magazine 19th Edition

Agentic AI threat modeling:
When AI starts acting

Control, Trust and accountability at scale

AI is entering a new operational phase.

What began as conversational assistants and content-generation systems is rapidly evolving into autonomous, tool-enabled, goal-driven AI, capable of planning, orchestrating workflows, interacting with systems, and taking actions with limited human intervention.

This transition fundamentally changes the security equation.

Agentic AI introduces a different challenge: the system is no longer only generating outputs; it is increasingly able to act upon its environment.

The security question therefore shifts from, “Can the model produce unsafe content?” to “What is the system allowed to do, under which conditions, and with what operational impact?”

This evolution requires organizations to rethink threat modeling and move beyond model misuse toward the governance of autonomous operational behavior.

Why traditional threat modeling is no longer sufficient

Traditional application threat modeling frameworks assume relatively deterministic behavior, where applications execute predefined logic, workflows are statically designed, trust boundaries are known, and execution paths are predictable.

Agentic AI systems break many of these assumptions.

Execution paths may emerge dynamically at runtime based on prompts, retrieved content, memory state, external tool responses, delegated tasks, and interactions between agents. This introduces a new class of operational risk, where the challenge is not only protecting an application but governing autonomous behavior. In many cases, the challenge is no longer identifying a single vulnerability but understanding how autonomous decisions evolve and propagate across interconnected systems.

Threat modeling must therefore evolve from protecting software components alone to also governing autonomous operational behavior.

A layered threat modeling approach for Agentic AI

While layered threat modeling approaches already exist in traditional cybersecurity, agentic AI introduces new dimensions related to autonomy, orchestration, runtime adaptation, and delegated execution that require extending traditional models.

Threat modeling for agentic AI requires a layered approach that connects governance, operational behavior, technical attack paths, and runtime controls. It is therefore mandatory to adapt threat modeling to systems capable of autonomous operational behavior. Here’s how businesses can adopt a practical structure, organized into four layers. This layered structure helps organizations progressively connect governance assumptions, operational risks, threat scenarios, and concrete attack paths across increasingly autonomous AI ecosystems.

Governance and trust assumptions: The first layer focuses on operational governance questions. At this level, many failures are governance-related rather than purely technical. As autonomy increases, governance becomes a security control.

Agentic AI risk categories: The next layer defines persistent risk domains associated with autonomous AI systems, such as operational risks introduced by autonomous planning, tool invocation, memory persistence, and multi-agent interaction.

High-level threat scenarios: This layer translates abstract risks into operational compromised patterns such as insecure inter-agent communication, memory poisoning, and cross-system propagation. These scenarios capture how unsafe instructions spread between agents, workflows, and orchestration layers and can help organizations analyze operational behavior rather than isolated vulnerabilities.
For example, memory poisoning occurs when malicious, incorrect, or manipulated information becomes stored within an agent’s memory and subsequently influences future decisions or actions. Because memory can shape planning, reasoning, and tool usage over time, organizations should establish controls around memory governance, including validation of persistent information, memory lifecycle management, trust boundaries between sessions, and monitoring of memory updates. This helps reduce the risk of corrupted context propagating across autonomous workflows.

Operational attack paths and runtime manifestations:The final layer focuses on concrete implementation scenarios such as memory poisoning, covert agent messaging, API abuse, and other outputs. These scenarios are environment-specific and often become the basis for assessments, adversarial testing, red teaming, tabletop exercises, and control validation.

Cross-layer runtime governance

A major difference introduced by agentic AI is that risk cannot be evaluated solely at design time. Autonomous systems continuously adapt their execution paths based on context, memory, external interactions, and intermediate decisions. As a result, security increasingly becomes a runtime governance challenge rather than a static architecture concern.

The challenge shifts from protecting static applications to governing dynamic operational behavior.

Bracing for change

As AI systems begin to act rather than simply respond, security must evolve from protecting models to governing autonomous behavior.

The future of AI security will not be defined solely by model safety, but by the ability to govern operational autonomy at scale.

The question is no longer whether AI can generate unsafe outputs, but whether organizations are prepared to govern systems capable of autonomous action.

Connect with us and let’s make security a core part of your business’ AI governance.

Learn more about the Atos Group’s commitment to creating a sovereign, governed approach to Agentic AI and how it is accelerating intelligence for our clients: AI – Atos Sovereign Agentic Studios – Atos