DobermanThe AI Security Playbook
My takeaways from RAND’s Practical Guide for Securing AI Models
TLDR: I think RAND’s guidebook is the best security standard for AI publicly available. This article is my summary of the guidebook so that you don’t have to read it with some of my own insights and opinons sprinkled in. I am hoping to incorporate all of these insights into my runtime security system for AI: Doberman.
DISCLOSURE: This article is my interpretation of the guidebook. It is not an official summary and it does not represent RAND’s views beyond the direct quotes. I am taking the ideas I found most important and expressing my own opinions
Abstract
The way people are thinking about AI security is still outdated and far too focused on the input stage with prompt injection and data poisioning. But for some reason after this first step miraculously people just assume everything afterwards can be trusted since it got passed this first stage. But this leaves a huge opening for attackers. Every action, tool call, external access and I/O should be gated. The model is only part of the surrounding system, and the supply chain is now much bigger than the model itself. I hate to say it, but agent autonomy needs to be limited, at least until we find a way to truly secure AI. What people think is the solution is sandboxing but this isn’t a Panacea as we saw with the hugging face incident. Memory is also a huge problem, though it is extremely helpful in improving the personal experience with AI, once it is corrupted it can lead to persiting damage, so there must be dedicated owner and defined scope. On the flipside we should be logging our Agents, just keeping track of token spend and other stats can be helpful for debugging but is practically useless information for security. We need to be keeping track of the entire journey of each agents workflow. This can be double edged sword; it makes incident response much easier but if it falls into the attackers hands it’s essentially a skeleton key.
Essentially, every security measure standalone is practically useless, so the best hope we have is to cobble everything together to create layers of redundancy to make up for each individual security layer’s blindspots.
Introduction
AI has revolutionized the way humans interact with computers, how computers interact with each other and the people in cybersecurity have been struggling to keep up with this rapid change.
For the past few years, the security discussion around AI has focused on model theft, data poisoning and harmful outputs. Those risks still exist but the systems being built today have much bigger threats to be worrying about.
Models can now browse the internet, retrieve internal documents, execute code, access credentials, call APIs and even directly control our computers. We aren’t working with chatbots that only produces text anymore. We are working with agents that act. So our defenses need to evolve alongside AI. For some people this is obvious but a large part of the industry is still stuck in the mindset of security for chatbots.
I recently went through RAND’s Practical Guide for Securing AI Models. It is a heavy read, but it contains some of the most useful thinking I have seen on how AI security needs to evolve.
Before I dive in, I want to make it clear that we should still be improving model safeguards, testing for jailbreaks, prompt injection and harmful outputs. But the security of an AI system cannot soley depend on whether the model chooses to behave correctly.
It needs a failsafe and that is the system surrounding the AI and what it allows the model to do when it misbehaves.
That means securing the full execution path:
Model → orchestrator → tools → infrastructure → data
We need to treat every input and output as untrusted, limiting tool permissions, tracking identity and provenance, restricting network access, sandboxing execution and authorizing consequential actions before they happen.
Where AI Security is currently at.
From what I am seeing in the AI security discussions on Linkedin/X/reddit is still focused on the prompting.
Can the model be jailbroken?
Can we detect prompt injection?
Can we train it to refuse harmful requests?
Can another model inspect the response before it reaches the user?
These are valid questions, but they are becoming less useful as the main security strategy.
I don’t think malicious prompting can ever be completely eliminated. Models are probabilistic, natural language is ambiguous and attackers will continue finding new ways to manipulate context. Fable 5, despite the meticulous efforts that Anthropic went into safeguarding the model, was still jailbroken in less than 48 hours.
We should definitely make prompt injection harder, but as per Murphy’s law we should accept that malicious instructions will eventually get through.
What’s critical is what happens after things go wrong.
If a model is only generating a restaurant recommendation, a bad response might end at a minor inconvenience, but if that same model is accessing a shell, customer database, GitHub organization or cloud account, the consequences can be dire.
Only focusing on “Can we make the model behave?” leaves a catastrophic hole when the safeguards fail and as we have seen repeatedly, these safeguards inevitably ALWAYS fail.
What would we should be focusing on now is:
“What is the model allowed to do when it does not behave?”
That distinction is where a lot of current AI security thinking falls apart.
The same model can be relatively harmless inside an isolated chatbot but even just by connecting it to a terminal suddenly just text output can become dangerous as I talked about in my previous post. When it’s connected to production systems these dangers could make or break entire companies (slight exaggeration but you get the idea). The capability of the model matters, but the authority we give it matters just as much.
Risk follows authority.
As Uncle Ben put it, “With great power comes great responsbility.” and the ower that AI has is beyond great.
The model is only one part of the system
One of RAND’s most useful observations is that many AI failures will not begin inside the model.
They will begin in the systems around it.
RAND identifies several major attack surfaces in high-risk AI systems:
Data inputs and retrieval
Tooling and code execution
Orchestration and application logic
Identity and model access
Deployment infrastructure
Output handling
This matches how modern AI applications are actually being built.
Nowadays, model rarely acts alone. There is usually an agent framework interpreting its output, a retrieval system pulling in external information, a memory layer storing context, a tool router selecting functions and a collection of internal services trusting the information being passed between them.
Each component can look secure in isolation but the connections between them and the system as a whole often remains vulnerable.
Imagine a coding agent reading a GitHub issue.
The issue contains a malicious instruction. The model treats that instruction as part of its task and generates a tool call. The orchestrator converts the output into a shell command. The shell inherits credentials from the host environment. The result is then stored in memory and affects the agent’s next action.
Which component failed?
The model followed the information it was given, the orchestrator executed a valid-looking tool call, the shell used credentials that were already available and the memory system stored what it was told to store.
Nothing necessarily broke at the individual component level. The vulnerability existed in the trust relationships between those components.
RAND describes this as implicit trust. Information moves from one part of the system to another without enough verification of its source, integrity or authority.
Once untrusted content manages to get in the workflow, it can quietly influence everything downstream.
Prompt filters cannot solve that problem by themselves.
Prompt injection becomes dangerous when it gains authority
Prompt injection is often treated like a content moderation problem.
Detect suspicious text. Remove it. Tell the model to ignore instructions inside retrieved documents. Add another classifier in front of the tool call.
That can help, but it misses the actual risk.
In an agentic system, prompt injection is an authorization problem.
The dangerous part is not that the model reads a malicious instruction. The dangerous part is that the instruction can influence an action the system has permission to perform.
A webpage telling an agent to reveal its system prompt is one thing, but one telling the agent to read environment variables and send them to an external server is another.
The difference is not the quality of the prompt injection. The difference is the authority available to the agent.
RAND highlights transitive prompt injection as a high-consequence threat. Untrusted content can pass through a model and influence generated code, tool calls or later decisions. The guide recommends as a solution to use controls such as sanitizing retrieved content, tracking provenance and blocking tool calls influenced by untrusted sources.
To me, the important part is the last one and exactly what I have been trying to implement in Doberman’s authorization layer.
We should assume that some injections will reach the model, that detection systems will fail eventually and that capable agents will occasionally find loopholes around the controls developers made.
The security boundary therefore cannot be the model’s willingness to ignore an instruction.
It has to sit at the point where generated intent becomes real execution.
Every consequential action needs authorization
As I mentioned at the start, RAND recommends deny-by-default authorization across the full execution path:
Model → orchestrator → tools → data stores
Identity and context should follow the request through every layer. Tools should operate with least-privilege permissions instead of inheriting broad credentials from the application or host machine. Connectors should fail closed when identity or policy information is missing.
This is basically zero-trust architecture applied to AI agents.
A tool call should not be trusted simply because it came from an approved model.
The action itself needs to be evaluated.
The system should know:
Who initiated the task
Which user or workload the agent represents
Which tool is being called
Which resource will be affected
Where the instruction originated
Which permissions are required
Whether the action can be reversed
Whether it matches the original task
What the agent has already done during the session
This means moving away from broad permissions like “This agent can access Supabase.”
( I still often do this as the lazy route, since configuring all those settings take too long for my attention span, but I’ve paid immensely when Claude decided the easiest way to fix my RLS was to delete my entire table from Supabase)
That permission is almost meaningless from a security perspective. Supabase access could mean reading a row in a table or deleting the ENTIRE database. (It was not fun to fix and it ruined my demo at the hackathon I was attending.)
A safer permission would look more like:
This agent may read edit table rules for the next 30 minutes. It may not access user information, change settings, delete rows or edit table rules without additional approval.
That difference determines the blast radius when the model is compromised, confused or simply wrong.
Authorization also needs to apply at the operation level. An agent that can read a database should not automatically be able to write to it. An agent that can create a pull request should not automatically be able to merge it. An agent that can inspect cloud infrastructure should not inherit permission to delete it.
The default should be no access, followed by narrow access for a specific task instead of the other way around.
Agent autonomy needs limits
A major problem with static permissions is that individually harmless actions can become dangerous when combined.
Reading one source file may be reasonable.
Reading an entire repository may still be reasonable in some situations.
Searching for credentials, creating an archive and opening an outbound connection is no longer reasonable.
The system needs to understand the sequence, not just the individual actions.
RAND recommends limiting agent autonomy through execution budgets, step limits, constrained loops and human approval for high-impact operations. It also points to the need to restrict dangerous tool combinations.
I think execution budgets will become an important part of agent security.
An agent should not only have a list of available tools. It should have a bounded operating envelope.
That envelope could limit:
Number of tool calls
Maximum loop depth
Total execution time
Files or records accessed
Amount of data transmitted
Network destinations contacted
Secrets requested
Infrastructure modified
Money spent
Privileged operations performed
This matters because agents are good at chaining capabilities together.
File readers, compression tools and HTTP clients are useful but when combined they can become an exfiltration pipeline.
A secure system needs to recognize when a sequence of acceptable actions is producing an unacceptable outcome.
I recognise that this sort of leads to a paradox, the point of Agent autonomy is to improve efficiency but limiting it and requiring human in the loop monitoring makes the autonomy obsolete. The best fix in my very biased opinion is a runtime authroization that creates dynamic permissions learning from user preferences which is the end goal of Doberman.
Sandboxing is super important but it’s also not the Panacea
No matter how many static guardrails and limits we set, to build a strong authorization, things will go wrong. (I feel like a broken record at this point, repeating this)
And there is a neverending list of where things could go wrong; A classifier could miss something. A rule forgets to cover the weird edge case. A developer adds a tool without understanding what it can really do. An agent combined three individually permitted actions and creates an outcome nobody predicted. Eventually, a blindspot will be found whether intentionally or on accident.
Sandboxing decides how bad things get when these action happens anyway.
Sandboxing is the exact recommendation that RAND makes. Having strong isolation around high-risk AI systems: containers, virtual machines, secure enclaves, default-deny network egress. For agentic systems, it also emphasizes constrained execution, action logging, and kill switches.
A coding agent usually doesn’t need unrestricted access to the host machine. It does not need every file in the user’s home directory. It definitely should not inherit personal SSH keys, cloud credentials, or production tokens because somebody launched it from the wrong shell.
And internet access should not be unlimited just because internet access makes the agent more useful.
Thebest security default is very straightforward:
Give the agent the smallest environment where it can still complete the task
That means read-only filesystems where possible, temporary workspaces, restricted network namespaces, short-lived credentials and resource limits.
It also means planning for the agent to fail in a real way, not just when it return a bad tasting lasagna recipe.
There should be a way to stop it, terminate the session, revoke whatever credentials it has, and roll back the changes it made. A kill switch shouldn’t be an enterprise add-on. Every system capable of taking consequential action needs one.
The uncomfortable part is that normal sandboxes may not be enough. AI systems are changing quickly, and incidents like the with Hugging Face show how fragile our assumptions about isolation can be.
Network access is part of the permission model
Egress is easy to underestimate because it rarely looks dramatic.
A team will block unknown websites and feel safe. Meanwhile, package managers, internal proxies, telemetry services, cloud metadata endpoints, and webhooks still remain available.
That is all part of network access.
An agent does not need a browser or a raw socket to move data out of an environment. Any system that accepts information and sends something elsewhere can become an egress channel.
A package installation request can carry information. So can a monitoring endpoint, webhook or even a DNS request. (Pretty obvious when you think about it, isn’t it
The guidebook suggests denying external network access by default and then enabling only what the system needs. That is the right starting point, but destination controls alone are too crude.
An agent installing an approved package from a trusted registry is not doing the same thing as an agent uploading a repository archive to an unknown server.
A weather API call is not the same as sending customer data to that API’s domain.
The destination could be identical but the action is not.
A domain allowlist cannot tell you why the request exists, what is inside the payload, or what sequence of model decisions led to it. It sees the address and thats about it.
Egress needs to be authorized like any other consequential action and these are the questions we should be asking to decide that authorization:
Who is sending the data?
What is being sent?
Why does the task require it?
Where is it going?
How much data is leaving?
Does the transfer match the task the user actually gave the agent?
Without that context, an allowlist can create the appearance of control while missing the parts that matter most.
Retrieved information should not automatically be trusted
A lot of systems scrutinize the initial user prompt, then become strangely relaxed about everything retrieved afterward.
(IMO that is the most backwards thinking imaginable.)
RAG pipelines, web pages, search results, emails, issue trackers, and internal documents all place unverified content inside the agent’s context.
RAND’s recommendation is validating retrieved documents, securing vector databases, recording document hashes, testing for indirect prompt injection, and preserving the origin of retrieved information.
The same rules should apply to tool output.
Search results can contain instructions. GitHub issues can contain instructions. Log Generated code can contain instructions aimed at the next model that reads it. Hidden instructions could literally be hidden anywhere, someone could hack into your friends twitter account and post something with malicious prompts.
One agent’s output becomes another agent’s input, and with how lazy people are with the internals of their agentic workflows, it’s essentially a black box and nobody knows where the instruction came from, but the system could and definitely should know.
It should preserve whether the source of where a piece of context came from, whether that’s a user, an internal database, a document that has never been verified etc.
That origin matters, trust should not increase merely because information passed through another component. A malicious instruction does not become safe because a model summarized it, a database stored it, or another agent repeated it in cleaner language.
An untrusted input should remain untrusted until verified
Memory is so useful but it can turn into our biggest kryptonite.
Memory is usually sold as a feature: Agents remembers preferences, picking up where it left off, keeping track of long-running work and the whole experience gets smoother.
But, it can also remember the wrong thing and in the wrong hands it could wreak havoc while going undetected. (I for one have not checked my context.md and memory files for at least a month.)
RAND warns about state leakage, cross-session contamination, and memory replay. It recommends limiting retention, restricting access, encrypting or redacting stored content, and auditing what the system keeps.
Agent memory needs a clear owner and a defined scope. It also needs provenance, expiration, tenant isolation, access controls, redaction, and reliable deletion.
Not optional deletion. Reliable deletion. (repeat for emphasis 😎)
A malicious instruction should not become trusted because it survived in memory for a week. A secret exposed during one task should not appear in another session. One user’s information should never drift into another user’s context.
Persistent memory creates persistent risk so the longer state survives, the more carefully somebody has to govern it.
The AI supply chain is bigger than the model (obviously)
Traditional supply-chain security still applies to AI systems it just that there are way more things in the chain now.
Teams have to think about model weights, training data, fine-tuning data, tokenizers, embeddings, vector indexes, prompts, plugins, tools, retrievers, evaluation environments, and model-generated code.
RAND recommends signed artifacts, checksums, software bills of materials, dependency scanning, provenance checks, and hardened CI/CD pipelines.
These matter because tools do more than give an agent new capabilities. They bring new code, dependencies, permissions, and trust relationships with them.
Compromised plugin bypass model safeguards, malicious package could execute during installation, poisoned retriever can decide what the model sees and corrupt evaluation environments can manipulate the results used to decide whether the system is safe enough to deploy.
Nobody has to attack the model if the surrounding system already trusts something malicious. It’s like letting the robber in the house and the next day you wake up like an idiot wondering where all your things have gone.
Package installation is a good example. It is often treated as routine developer convenience. For an agent, it can involve network access, arbitrary installation scripts, and executable third-party code.
We treat it as a harmless convenience but in reality it’s a security-sensitive action.
Logging should go beyond stats, it should explain why the action happened
Most AI observability products focus on prompts, responses, latency, and token usage.
This may be useful for debugging and optimizing your system but it’s essentially useless for security.
RAND recommends logging model access, prompts, outputs, tool calls, and system actions while preserving traceability across model versions, deployments, and configuration changes.
For an agent, the audit trail should show what it attempted, which identity it used, what information influenced the action, which policy allowed or blocked it, who approved it, and what changed.
It also needs to record which model and tool versions were running and what happened immediately before and after the action. (With how quickly iterations happen, a vulnerability could exist in one version and disappear quietly in the next)
The final output alone is not an incident record. To get even the faintest idea of what’s going on, security teams need to reconstruct the decision path, including the parts where the system nearly did something dangerous but stopped.
Logging creates a huge issue; Prompts and tool outputs may contain credentials, customer data, private documents, or source code. A company that logs everything without thinking about redaction can accidentally create a centralized collection of its most sensitive material.
This means that even though logging makes it much easier to respond to incidents, it also makes the attackers job much easier.
Logs need to be tamper-resistant, structured, and useful for investigation. They also need access controls, retention limits, and aggressive redaction of secrets and private data.
So logging can be a double-edged sword, more logging is not automatically more security.
Human approval only works when the approval means something
Human-in-the-loop is often presented as the final answer to agent risk.
Put a confirmation box in front of the action. Make the user click Allow. Problem solved.
Except usually the box says something vague like “Allow this action?” and gives the user no meaningful way to judge it.
What command will run?
Which resource will it affect?
What are the expected side effects?
Where did the request come from?
A user cannot authorize an action they cannot see.
Approval also gets weaker when it happens constantly. If somebody has to approve dozens of ordinary actions, they eventually stop reading. The confirmation becomes another button in the workflow. I for one have never turned off auto mode and for the glorious short period when -bypass-permissions existed, I used it religiously.
RAND recommends step-level approval for consequential actions and stronger authorization for highly sensitive operations.
A useful approval woudl be tied to the exact action, target resource, arguments, expected effect, requesting identity, alongside a unique request with a short expiration time.
Change the action and the approval should expire.
Replay the request later and it should fail.
Increase the risk after approval and the system should ask again.
The intended fuction of human approval is real authorization. It shouldn’t be a scapegoat method to last-minute transfer of responsibility from the system to the user.
Security should be in every step of the full lifecycle
RAND structures AI security across the entire lifecycle: design, development, deployment, operation, and retirement.
That matters because weaknesses introduced early do not magically disappear when the model goes to the next step:
Poor dataset controls affect behavior later.
Compromised development infrastructure affects model artifacts.
Weak deployment permissions expose production systems.
Missing monitoring lets abuse continue.
Bad retirement procedures leave weights, credentials, and sensitive datasets sitting around long after a system is supposedly gone.
Identity, access control, provenance, artifact integrity, supply-chain security, red-teaming, and incident response have to survive every stage.
The guide also makes clear that AI security is larger than prompt injection. It covers input manipulation, training-data manipulation, model manipulation, input extraction, training-data extraction, and model extraction.
Those threats affect confidentiality, integrity, and availability across language models, computer vision, audio, multimodal systems, reinforcement learning, and scientific AI.
AI security does not replace traditional cybersecurity, it should work in tandem, extending it into probabilistic systems, untrusted data pipelines, and software that can convert generated output into real actions.
The shift should be from model safety to execution security
For years, the industry has asked whether a model can generate unsafe content.
Agentic systems introduce a harder question:
“Can generated content cause unsafe execution?”
Model safeguards shape behavior while execution controls limit authority. Both matter. But when the model is manipulated, confused, or confidently pursuing the wrong objective, execution controls are what remain.
My reading of RAND’s guide is that secure agent systems need a small set of non-negotiable foundations:
Deny-by-default authorization
Least-privilege identities
Granular tool permissions
Provenance tracking
Sandboxed execution
Restricted egress
Execution budgets
Human approval for high-impact actions
Continuous monitoring
Kill switches and rollback
Controls that increase with system capability
None of this requires abandoning what security teams already know. It only requires applying those principles at a finer level and closer attention to detail because AI systems make decisions dynamically and move across tools, identities, and environments.
A chatbot answering a question and an agent modifying production infrastructure cannot share the same security model just because they use the same underlying LLM.
Their authority is different which means the consequences. are different too, so the controls need to reflect that.
What this means for agent builders
The model cannot be the security boundary and the prompt cannot be the policy engine.
The agent framework cannot be trusted merely because it is the component coordinating everything. And a tool call should not execute simply because the model asked for it.
There needs to be an independent enforcement layer between the agent’s intention and the environment’s execution. That layer should understand what the agent is allowed to do. It should evaluate consequential actions, preserve identity and provenance, track cumulative behavior, and block requests that exceed the agent’s authority.
This is the direction I am taking with Doberman.
I am not trying to build another prompt filter. I am not trying to build another classifier that guesses whether a tool call looks suspicious.
The goal is runtime authorization for AI agents.
As agents gain access to shells, credentials, production repositories, and customer data, we can stop pretending the main question is whether somebody can persuade the model to do something unsafe. Somebody definitely will. The real question is whether the surrounding system gives the model enough authority to make the unsafe action real.
Also published on Substack ↗