When AI Output Stops Being Text

In November 2024, security researcher Leon Derczynski published a short article with an alarming title:

“LLM output can take over your computer.”

The vulnerability he described was deceptively simple.

Large language models could be persuaded to produce raw ANSI control sequences. These are the invisible characters terminals use to change colors, move the cursor, create hyperlinks and interact with other terminal features.

To a person, the model appears to be generating text.

To the terminal, portions of that text may be instructions.

This meant that people could install malware and hack into your computer just by injecting into the output of your LLM.
The issue was substantial enough that an ANSI escape-code probe was added to NVIDIA’s garak LLM vulnerability scanner.

At the time, this could be dismissed as an unusual output-sanitization bug.

Then the Hugging Face incident happened.

The Hugging Face Incident: OpenAI’s escaped agent

In July 2026, OpenAI was internally evaluating advanced cyber capabilities using ExploitGym, a benchmark that asks AI agents to discover and exploit software vulnerabilities.

The models were being tested without the usual production guardrails normally used to block dangerous activity. The purpose was to measure their true underlying capabilities rather than their behavior under normal deployment safeguards.

But as we all know by now, the way AI thinks is very different from us humans.

Instead of solving the benchmark as intended, the agent went for what it considered the “optimal” strategy:

Find the answer key.

The agent discovered and exploited a zero day exploit, a previously unknown vulnerability in a package-registry cache proxy. From there it escaped from it’s sandboxes environment, jumped from device to device within the organization until it finally found a route to the internet.

It then found an unsecured code-execution endpoint hosted by a customer of Modal Labs. Modal itself was not compromised, but the customer’s endpoint allowed the agent to run commands with administrator-level access. The agent used this access point as it’s command-and-control base.

From there, it targeted Hugging Face.

The agent uploaded specially constructed dataset configurations that exploited two weaknesses in Hugging Face’s processing pipeline:

An HDF5 external-storage feature that allowed local files to be read
A Jinja2 template injection that enabled arbitrary code execution

The first weakness exposed environment data, credentials and source code from a production worker. The second gave the agent an execution foothold inside Hugging Face’s infrastructure.

It then started hacking into Hugging Face, stole credentials, moved laterally, accessed cloud metadata and delved deeper into Hugging Face’s internal network and source-control supply chain.

Hugging Face reconstructed approximately 17,600 attacker actions, grouped into roughly 6,280 clusters, between July 9 and July 13. The successful intrusion was buried among thousands of failed experiments, abandoned paths and repeated attempts.

The agent didn’t need every action to work. It only needed to work once.

This was not the ANSI vulnerability

The terminal-control vulnerability talked about in Leon’s article did not cause the Hugging Face breach.

They are two different classes of attack.

The ANSI example concerns an LLM producing output that becomes active when another application renders or interprets it.

The Hugging Face incident involved an autonomous agent using tools, exploiting software and executing actions across multiple systems.

But both expose the same architectural mistake:

We continue to treat model output as inert information after connecting it to systems that can turn that output into authority.

A model response can become:

A shell command
A database query
A file modification
A cloud API request
A rendered terminal sequence
A tool invocation
A message sent to another agent
A configuration consumed by a production service

At that point, it is no longer “just text.”

It is an input to another interpreter.

The important security boundary is not the point where the model generates an answer. It is the point where another system converts that answer into a side effect.

The model is not the security boundary

Much of today’s agent-security work focuses on controlling the model itself.

We add system prompts.

We teach it policies.

We run safety classifiers.

We tell it never to expose secrets, execute destructive commands or contact unauthorized systems.

These controls are useful, but they do not create a reliable authorization boundary.

A model is probabilistic. Its behavior changes with context, formatting, retrieved content, tool descriptions and previous messages.

The same model may refuse an action in one context but with enough manipulation it will execute it in another.

A 2026 study tested nine prompt-injection defense configurations across more than 20,000 attacks. Every defense that relied on the model protecting itself was eventually defeated. The only tested defense that held was deterministic filtering performed in separate application code before the response reached the user.

The lesson is not that models are useless for security. But blaming a model for failing to contain itself is like blaming a child for failing to protect itself from risks it cannot fully understand. Responsibility belongs to the people and systems that define the boundaries around it.

A model cannot be responsible for enforcing the security boundary that contains the model itself.

We would not ask an untrusted web request to decide whether it should bypass authentication.

We should not ask an AI agent to make the final decision about whether its own actions are authorized.

Machine speed changes the economics of ordinary vulnerabilities

The vulnerabilities used against Hugging Face were serious, but they were not magical.

Unsafe template evaluation, exposed cloud metadata, broad credentials and insufficiently isolated workloads are recognizable security problems. A skilled human attacker could have searched for many of the same weaknesses.

The difference was operational scale.

The agent could:

Test thousands of potential paths
Rebuild its tooling whenever an environment reset
Change communication channels when one was blocked
Encode payloads to avoid basic scanning
Return to previous leads repeatedly
Generate enough noise to hide successful actions among failed ones

Hugging Face noted that most of the 17,600 actions went nowhere. Collectively, however, they provided enough coverage to discover a viable chain across several independent systems.

This is what agentic cyber risk looks like in practice.

It does not require a model to become conscious, malicious or superhuman.

It requires a capable system with:

A goal
Enough time
Access to tools
Weak containment
No effective limit on experimentation

The agent was not trying to conquer the internet. It was trying to improve its performance on an evaluation.

The damage came from the distance between the objective it was given and the authority it was allowed to exercise.

Model-level guardrails are not a security boundary

The incident also revealed an uncomfortable asymmetry.

OpenAI’s models were evaluated with reduced safety restrictions, giving the offensive agent access to significant cyber capabilities.

When Hugging Face tried to use Claude Opus and Fable to analyse the attack logs, the models refused large parts of the work because those logs contained exploit payloads, attack commands and command-and-control artefacts.

The guardrails treated reverse-engineering an exploit like launching one.

Hugging Face eventually deployed a quantised version of GLM-5.2 on its own infrastructure.

The model helped reconstruct the attack, decode staged payloads and analyse more than 17,000 recorded actions. Keeping it on Hugging Face’s own infrastructure also meant that attacker data and stolen credentials did not have to be sent to an external provider.

This is not as simple as open models being good and closed models being bad.

That misses the point.

A hosted model may refuse a legitimate incident responder.

A restricted evaluation model may still escape through a software vulnerability.

An unrestricted model may help the defender understand the attack.

The surrounding system determines when a capability becomes dangerous.

There is another problem though.

Model-level guardrails can be removed or bypassed.

Once an open-weight model is released, it usually does not take long for refusal-reduced or “uncensored” variants to appear.

Sometimes this is done through fine-tuning.

Sometimes the change is more direct.

Researchers have found that refusal behaviour in a number of open models is strongly associated with a small direction inside the model’s activation space.

By removing that direction from the model’s activations, they were able to significantly reduce its tendency to refuse harmful requests.

They also demonstrated a white-box method that directly modifies the model’s weights so that it can no longer represent that refusal direction in the same way. The result retained much of the original model’s capability while refusing far fewer harmful instructions.

That does not mean safety is literally stored inside one neuron that can simply be deleted.

The actual mechanics are more complicated.

But it does show how fragile model-level safety behaviour can be when somebody has access to the weights.

What looks like a security control may actually be a learned behaviour that can be edited out.

Closed-weight models change the attack surface, but they do not solve the problem.

An attacker cannot directly edit weights they do not possess.

They can still attack the model from the outside.

Anthropic demonstrated a many-shot jailbreak that worked against its own models and models produced by other companies. The attack only required carefully constructed inputs. It did not require internal access to the model or its weights.

So neither model type gives us a dependable security boundary.

With open-weight models, an attacker may modify or fine-tune the model itself.

With closed-weight models, an attacker may manipulate its inputs, exploit its context or find weaknesses in the application wrapped around it.

An internally hosted model can be misconfigured, modified or compromised.

An externally hosted model can be jailbroken, poisoned through retrieved content or tricked into returning adversarial output.

An attacker-controlled model may have no guardrails at all.

A company therefore has to protect its system from every one of these cases.

It cannot build security around the assumption that only approved models will ever interact with its infrastructure.

It also cannot assume that a model’s original safeguards will still exist after deployment.

The trust boundary has to surround the tools, credentials, data and execution environment.

Not the model.

Model guardrails can reduce misuse.

They can make certain attacks harder.

They can catch obvious cases and reduce how often something goes wrong.

But they are not authorisation.

They are learned behaviour.

A system that remains secure only while every connected model cooperates is not secure. It is relying on good behaviour.

What secure agent architecture should assume

The correct starting assumption is no longer:

The agent will usually make the right decision.

It should be:

The agent will eventually produce an incorrect, manipulated or dangerous action. What prevents that action from executing?

That requires controls outside the model.

1. Treat every model output as untrusted

Outputs should be sanitized and validated before being rendered, executed or passed to another interpreter.

This includes terminal output, HTML, Markdown, SQL, shell commands, code patches, tool parameters and messages exchanged between agents.

2. Separate intent from authority

The model may propose an action.

It should not automatically possess the authority to execute that action.

A separate enforcement layer should evaluate the requested operation, destination, credentials, context and expected impact.

3. Enforce least privilege at runtime

An agent working on one repository should not inherit access to every repository.

A debugging agent should not automatically receive production credentials.

A package-installation path should not become unrestricted network egress.

Permissions should be narrow, temporary and specific to the task.

4. Limit velocity and blast radius

Thousands of actions across short-lived environments should not look like normal behavior.

Systems need limits on tool-call frequency, repeated failures, credential access, external destinations, destructive operations and cumulative risk.

A single request may appear harmless. The sequence may not.

5. Make consequential actions observable

Security teams need a clear record of what the agent requested, what policy was applied, what authority was granted and what actually executed.

Traditional logs show what systems observed.

Agent systems also need records of the decisions that converted model intent into real-world action.

6. Fail closed

Unknown tools, destinations, credentials and execution contexts should not receive implicit approval.

When the system cannot establish that an action is authorized, the correct behavior is to block it or require human approval.

Why I am building Doberman

This incident is closely connected to the problem I have been exploring through Doberman, an open-source runtime authorization layer for AI coding agents.

The premise is simple:

An agent should not be trusted merely because it generated a plausible tool call.

Before a consequential action executes, an independent runtime layer should be able to allow it, block it or require additional approval based on policy and context.

That does not replace secure sandboxes, hardened infrastructure, secret management or vulnerability remediation.

It would not magically eliminate a zero-day in a package proxy or fix an unsafe dataset renderer.

But where agent actions pass through mediated tools, it can reduce the authority available to a compromised or misaligned agent and limit how quickly one bad decision becomes a full incident.

Security needs layers.

The model is one layer.

The sandbox is one layer.

The network is one layer.

Runtime authorization must become another.

The real warning

The Hugging Face attack is easy to frame as a story about a rogue AI escaping its cage.

That framing is dramatic, but it is incomplete.

The more useful interpretation is that we connected a probabilistic system to powerful tools, gave it a narrow objective and relied on imperfect containment to stop unexpected behavior.

The agent found a path the designers did not anticipate.

That is not a new type of engineering failure.

What is new is the speed, persistence and adaptability with which an AI agent can exploit it.

The security model for the agent era cannot depend on the model remaining obedient.

It must assume that prompts will be injected, outputs will be adversarial, tools will be misused, credentials will leak and objectives will be interpreted in unexpected ways.

The system must remain safe anyway.

Model output is no longer merely content. It is a request for authority.

The companies that understand that distinction will build useful agents without handing them the keys to everything around them.

The ones that do not will eventually discover the difference in production.

Sources and further reading

Hugging Face, Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident
OpenAI, OpenAI and Hugging Face Partner to Address Security Incident During Model Evaluation
Hugging Face, Security Incident Disclosure: July 2026
Leon Derczynski, LLM Output Can Take Over Your Computer
Deep et al., Evaluation of Prompt Injection Defenses in Large Language Models