AI Security Playbook

1. Scope of this playbook

This playbook is for security and architecture teams building a threat model and defense program specifically for AI agents and LLM applications — prompt injection, tool-output poisoning, credential and access risks, and agentic-specific attack patterns. It assumes familiarity with traditional application security and focuses on what's genuinely different about securing AI systems. It complements, rather than replaces, the security sections already embedded in the Agent Platform, MCP, and Gateway playbooks — this is where those threads come together into a single threat model.

2. AI security maturity model

Self-assessment checklist

• Do you have a documented threat model for each production AI agent, covering prompt injection, tool-output poisoning, and credential misuse specifically?
• Is content an agent reads (web pages, documents, tool outputs) treated as untrusted input, with the same validation rigor as user-submitted form data?
• Have you run a red-team exercise against your agents specifically targeting prompt injection and tool misuse, or only against traditional application endpoints?
• Could you detect, today, if an agent's tool-calling pattern shifted anomalously — more calls, different targets, unusual sequences?
• If asked "what's our actual AI agent security incident rate," could you answer with data, or would the honest answer be "we're not sure we'd know"?

3. How much dedicated AI security investment is warranted?

Every organization running AI agents with any tool access or consequential action capability needs at minimum baseline controls — this isn't optional given how prevalent prompt injection already is. The case for a fuller threat-modeled program strengthens with the number of agents in production, the sensitivity of what they can access, and whether they take actions with real financial, operational, or reputational consequence. Given the gap between executive confidence and actual incident rates, treat any assumption that "our existing controls probably cover this" as unverified until tested.

4. Readiness checklist

• A documented threat model exists for each production agent, covering the specific tools and data it can access.
• Input validation treats all agent-consumed content — not just direct user prompts — as potentially adversarial, including retrieved documents, tool outputs, and web content.
• Credentials and access are scoped per agent following least privilege, matching (not exceeding) the access a human in an equivalent role would have.
• Monitoring can detect anomalous tool-calling patterns, not just traditional infrastructure anomalies.
• A red-teaming or adversarial testing process exists, even a lightweight one, run before major agent capability changes reach production.

5. Build vs. buy

6. Phased rollout playbook

Phase 0 — Threat modeling

Document a threat model for each production or near-production agent: what can it access, what could go wrong, what would an attacker try. This is foundational and often reveals gaps immediately, especially around what's treated as trusted versus untrusted input.

Phase 1 — Baseline controls

Apply input/output validation, scoped credentials, and logging consistently across agents. Treat this as a floor, not a ceiling — baseline controls alone won't catch sophisticated attacks.

Phase 2 — Adversarial testing

Run red-team exercises specifically targeting prompt injection, tool misuse, and credential escalation. Feed findings back into the threat model and controls, closing the loop rather than treating testing as a one-time exercise.

Phase 3 — Continuous defense

Move toward ongoing adversarial testing integrated into your deployment pipeline, so new agent capabilities are tested before they reach production rather than after an incident.

7. Governance & risk checklist

• Every production agent has a named security owner accountable for its threat model staying current.
• Security incidents involving AI agents are tracked and categorized distinctly from traditional application security incidents, so patterns specific to AI risk are visible.
• Threat models are reviewed whenever an agent's tools or access change, not just on a fixed calendar cadence.
• There's board or executive-level visibility into actual AI security incident data, closing the confidence gap between perceived and real risk.
• A tested incident response process exists specifically for AI agent incidents, since the investigation and containment steps differ from traditional application incidents.

8. Security checklist

• All content an agent reads — documents, web pages, tool outputs, retrieved context — is treated as untrusted input subject to validation, not implicitly trusted because it arrived through an internal system.
• Agent credentials are short-lived and scoped per tool and per task where possible, not long-lived broad service accounts.
• Tool outputs are validated before being acted on or fed back into further reasoning — a compromised or manipulated tool response is now part of your attack surface.
• Rate limits and cost ceilings exist per agent to contain the blast radius of a compromised or manipulated agent loop.
• Multi-agent and cross-boundary (A2A) interactions authenticate and validate incoming requests as untrusted input, not implicitly trusted internal traffic.

9. Cost model & ROI

The ROI case here is closer to insurance than to a typical feature investment: the cost of prevention is small and predictable, the cost of a serious incident is large and highly variable — and the current confidence gap suggests most organizations are underpricing that risk.

10. Organizational playbook

Security teams should own the threat-modeling framework, red-teaming program, and incident response process for AI-specific risks. Platform and agent-owning teams should implement the resulting controls and keep their agent's threat model current as capabilities change. This requires security teams to build genuine AI-specific expertise rather than applying traditional AppSec playbooks unchanged — the attack patterns (prompt injection, tool poisoning, agentic escalation) are different enough to need dedicated attention, not a checkbox added to an existing review.

11. Common failure patterns

• Trusted-by-default content: treating retrieved documents, web content, or tool outputs as safe because they arrived through an internal pipeline, when they're exactly the vector prompt injection exploits.
• Confidence without verification: assuming existing security policies cover AI agents without testing that assumption — the gap between executive confidence and actual incident rates suggests this is widespread.
• Static threat models: a threat model documented once at launch and never updated as an agent's tools and access expand.
• Detection-only defense: relying on injection detection tooling alone, when current tools catch only a minority of sophisticated attempts — defense needs to be layered, not single-point.
• Credential sprawl: agents accumulating broad, long-lived credentials because scoping them properly felt like it would slow down development.

12. AI security posture decision scorecard

An organization that scores well on baseline controls but poorly on adversarial testing is defended against known patterns only — given how fast agentic attack techniques are evolving, that's necessary but not sufficient. Weight accordingly.