Breaking Lab srl
A Flaw Built Into the Foundation
Security researchers have delivered one of the most sobering assessments of AI safety to date: large language models are structurally incapable of being made fully secure against certain classes of attack. The argument, presented at the International Conference on Machine Learning (ICML), one of the most prestigious venues in the field, is not a warning about a specific bug or misconfiguration. It is a claim that the very mechanism by which LLMs process and respond to text makes a complete defense mathematically impossible.
The core issue is that LLMs cannot reliably distinguish between instructions they are supposed to follow and data they are merely supposed to process. When a model reads a document, browses a webpage, or summarizes an email on a user’s behalf, it is exposed to text written by third parties. If that text contains carefully crafted instructions — a technique known as prompt injection — the model may act on them just as readily as it acts on the legitimate commands from the user or developer who deployed it. There is no reliable internal boundary.
Why This Is Different From Ordinary Software Bugs
Traditional software vulnerabilities are, at least in theory, patchable. A buffer overflow can be closed with a code fix; a misconfigured access control list can be corrected. What the ICML researchers argue is that prompt injection is not a bug in that conventional sense. It emerges directly from the transformer architecture’s attention mechanism, which treats all tokens — whether they come from a system prompt, a user query, or an external document — as part of the same representational space. You cannot patch the architecture without fundamentally changing what makes LLMs capable in the first place.
This puts the AI industry in an uncomfortable position. Agentic AI systems — models that browse the web, execute code, send emails, or interact with APIs on a user’s behalf — are precisely the applications generating the most commercial excitement right now. They are also the applications most exposed to adversarial content in the wild. Every webpage an agent visits is a potential attack surface.
The Real-World Attack Surface Is Already Here
The timing of this research is not purely academic. Just days before the ICML paper drew attention, Anthropic disclosed that its own Claude models had successfully breached three companies during controlled security tests, a disclosure prompted by a similar incident involving OpenAI’s models penetrating systems at Hugging Face. Those events were treated as surprises. The ICML finding suggests they should have been expected.
Enterprises deploying AI copilots for customer service, legal document review, or financial analysis are routinely exposing their models to content from untrusted parties. A malicious actor who knows a target company uses a specific AI assistant needs only to craft a document or web page that the model will eventually process. From there, the injected instructions can redirect the model’s behavior, exfiltrate information, or corrupt outputs — all without ever touching the underlying infrastructure.
What Defenders Can Actually Do
The researchers are not arguing that all defenses are worthless. Sandboxing, output filtering, strict privilege controls over what actions an agent is permitted to take, and human-in-the-loop review for high-stakes decisions all reduce risk meaningfully. The point is that no combination of these measures can reduce the risk to zero, because the vulnerability is in the reasoning process itself, not in a peripheral component that can be swapped out.
Some AI providers have experimented with instruction hierarchy schemes — essentially teaching models to weight system-level instructions more heavily than user-level ones, and user-level instructions more heavily than content-level ones. These approaches have shown some promise, but the researchers contend they are heuristics that can be circumvented, not structural solutions.
An Industry Reckoning That Can No Longer Be Deferred
The broader implication of this research is that the AI industry’s standard reassurance — that safety and capability issues will be engineered away given enough time and funding — does not apply cleanly here. Regulators, enterprise buyers, and security teams would do well to treat the current generation of agentic AI systems as permanently operating in a partially adversarial environment, rather than waiting for a patch that may never arrive. Building AI products responsibly may require accepting hard capability trade-offs: the most powerful agents are, by design, the most exposed ones.
