#14.11 AI System Compromise

Playbook ID: PB-AISYS | Default severity: SEV-2 (escalate to SEV-1 if the AI system holds write or act authority into a production system of record, if a non-human identity it used has been replayed elsewhere, or if regulated personal data left through it; drop to SEV-3 only once you have proven the system was read-only over data the requester was already entitled to see) | Owner: Incident Commander

#When to run this

Open this playbook on any of:

• A model doing something it was never asked to do. An assistant returning content the requesting user has no rights to; an agent making a tool call it has never made before, or making a familiar call with parameters outside its normal range; model output containing a URL to a host nobody recognises.
• An egress alert on an AI runtime. A denied — or worse, permitted — outbound request from an agent, inference service or retrieval pipeline to a destination outside its allowlist.
• A tool-definition change. A pinned MCP server or tool definition whose digest no longer matches the approved one; a new server appearing in a developer or production agent configuration without a change record.
• Retrieval leakage reported by a human. A user says the assistant showed them another department's, another customer's or another tenant's content. This arrives as a support ticket, not as a security alert, roughly every time.
• Cloud detections on AI workloads. GuardDuty DefenseEvasion:IAMUser/BedrockLoggingDisabled, or UnauthorizedAccess:IAMUser/ResourceCredentialExfiltration.OutsideAWS on a Lambda or ECS task that runs an agent (GuardDuty IAM finding types).
• AI-tooling supply chain. A package in your AI stack pulled from a compromised release; a developer agent CLI invoked by a build step with permission-bypassing flags.
• Provider notification that your API keys or account were used for activity you did not authorize, or a sustained inference-volume anomaly consistent with systematic extraction.

Not for: deepfake or voice-clone social engineering of a person — that is 14.9, and no AI system of yours is compromised. An attacker who merely used AI to write their malware is an ordinary intrusion; run the playbook that matches what they touched. If the AI vendor themselves was breached, start with 14.5 and use this playbook to scope your own agent's blast radius. If the compromise landed in the cluster and the agent was the door, contain here and hand the cluster to 14.10. Regulatory notification detail lives in Chapter 15; the AI inventory, agent onboarding and governance controls this playbook assumes you already have are Chapter 7.

#What you are dealing with

A large language model reads instructions and data on the same channel. There is no reliable in-band way to tell the two apart, which means every document, email, ticket, wiki page, web fetch, PDF and tool description near your model is untrusted input to a privileged executor. That is not a bug someone is about to fix. It is the shape of the technology, and every incident in this playbook is a variation on it.

EchoLeak is the reference case and worth knowing by name. Disclosed in June 2025 by Aim Security, CVE-2025-32711 was a zero-click indirect prompt injection in Microsoft 365 Copilot, CVSS 9.3. A single crafted email carrying instructions hidden in HTML comments and white text was ingested into the RAG context. When the user later asked Copilot an unrelated question, the hidden instructions ran: retrieve sensitive tenant data, encode it into a URL, let the client auto-fetch it. The chain evaded Microsoft's cross-prompt-injection classifier, defeated link redaction using reference-style Markdown, and abused a Teams proxy. Microsoft patched server-side and reported no exploitation in the wild (arXiv, HackTheBox). The user did nothing. There was nothing for them to do.

The second pattern is the confused deputy, and it is the one that will actually hurt you, because the agent needs no exploit — only the access its runtime already carries. In May 2026 Sysdig observed an LLM-driven actor exploit CVE-2026-39987 in a marimo notebook, then autonomously enumerate escape primitives, mount the Docker socket, create a privileged container with a /:/host bind, read /etc/shadow and SSH keys, and replay a projected Kubernetes service-account token to dump the cluster Secret store — database credentials, AWS keys, OpenAI API keys. The agentic tells were unmistakable: it parsed and acted on a canary directive hidden in a JSON error response, and it unit-tested its own payload delivery with "hello" before running the escape scripts (Sysdig). The third pattern turns your own AI tooling into the attacker's hands: in the Nx s1ngularity compromise of August 2025, malicious package versions detected Claude Code CLI, Google Gemini CLI and Amazon Q CLI on developer machines and invoked them with permission-bypassing flags to sweep the filesystem for secrets — 2,349 credentials from 1,079 developer systems, then a second wave that used the stolen GitHub tokens to flip private repositories public (The Hacker News, GitGuardian).

The mistake teams make is treating this as a model problem. The first instinct is to open a ticket with the vendor and ask for a better prompt filter, which is roughly like responding to a burglary by asking the locksmith to make the front door more persuasive. Containment for an AI system is an identity and blast-radius operation, not a model operation: revoke the credential the agent runs as, disable the tool it abused, quarantine the corpus that carried the injection, and rotate everything the runtime could reach. The model can wait.

Be honest about the state of the discipline while you are in it. The attack classes are well documented — OWASP holds LLM01 Prompt Injection at number one for a second consecutive edition and now publishes a separate Top 10 for Agentic Applications with ASI01 Agent Goal Hijack, ASI02 Tool Misuse and ASI03 Identity & Privilege Abuse (OWASP GenAI, Agentic Top 10). The forensic corpus is thin. There is no credible confirmed reporting of a named real-world RAG-leakage or model-extraction breach, and every circulating statistic about RAG leak rates and costs traces to content farms rather than research. And Mandiant's conclusion from over 500,000 hours of 2025 incident response still stands: most intrusions come from human and systemic failures, not from AI (M-Trends 2026). Run this playbook when it applies. Do not let it displace the fundamentals. Actionable takeaway: the single question that drives every step below is what could this identity reach — not what did the model say.

#Roles for this incident

Table conventions. `TIP-OFF marks a step an adversary can observe. EVIDENCE` marks a step that destroys or degrades evidence if run out of order. Do not reorder around those markers without the IC.

#Phase 1 — Detection and Triage

-- CloudTrail Lake: every action taken by the agent's role across the window.
-- Trino dialect, SELECT-only, event data store ID as the FROM value.
-- Run with: aws cloudtrail start-query --query-statement "<this>"
SELECT eventTime, eventName, awsRegion, sourceIPAddress, readOnly,
       userIdentity.principalId, errorCode
FROM <event-data-store-id>
WHERE userIdentity.arn LIKE '%<agent-role-name>%'
  AND eventTime > '<window-start>'
ORDER BY eventTime

# Kubernetes: every pod running under the agent's service account, with its node.
# Verbatim from the EKS Best Practices Guide, Incident Response and Forensics.
kubectl get pods -o json --namespace <namespace> \
  | jq -r '.items[] | select(.spec.serviceAccount == "<service account name>") | "\(.metadata.name) \(.spec.nodeName)"'

#Phase 2 — Containment

Order matters here more than in almost any other playbook in this chapter, and the reason is specific: the agent's credential is the payload, the trace is the evidence, and the two are destroyed by opposite actions. Cut the channel before you touch the identity; capture the trace before you touch the pod.

# Entra ID — agent running as a user identity (a real account with a UPN). Users only:
# see the note below for the service principal case, which these cmdlets do not cover.
# Revoke-MgUserSignInSession invalidates refresh tokens and browser session cookies
# by resetting signInSessionsValidFromDateTime. It is a CAE critical event.
Connect-MgGraph -Scopes "User.ReadWrite.All","Directory.AccessAsUser.All"
$User = Get-MgUser -Search UserPrincipalName:'<agent-upn>' -ConsistencyLevel eventual
Update-MgUser -UserId $User.Id -AccountEnabled:$false
Revoke-MgUserSignInSession -UserId $User.Id

If the agent runs as a service principal, none of the block above applies to it. A service principal has no userPrincipalName, so Get-MgUser will not return it, and there is no session-revocation action for it — revokeSignInSessions is a user-only operation and Entra publishes no service-principal equivalent. Do not go looking for one mid-incident. The containment path is three separate actions: disable the service principal so it can no longer sign in; remove its client secrets and certificates so it cannot authenticate with credentials it already holds; and strip its authority — its app-role assignments and its delegated permission grants, using the commands below. Any access token it was already issued stays valid until it expires. That is the reason the egress cut at step 2.1 comes before the identity work, not after it.

# Where the AI tool reached your tenant through an OAuth grant, revoke the grant.
# Microsoft: "Normal remediation steps (for example, resetting passwords or requiring
# multifactor authentication) aren't effective against this type of attack."
Remove-MgOauth2PermissionGrant -OAuth2PermissionGrantId <id>
Remove-MgServicePrincipalAppRoleAssignment -ServicePrincipalId <sp-id> -AppRoleAssignmentId <id>

// AWS — the AWSRevokeOlderSessions inline policy the console attaches to a role.
// Denies sessions assumed before the timestamp, plus ~30 seconds of propagation slack.
// Requires PutRolePolicy on the role. Cannot be used on a service-linked role, and
// roles created from IAM Identity Center permission sets must be revoked in Identity Center.
{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Deny",
    "Action": "*",
    "Resource": "*",
    "Condition": { "DateLessThan": { "aws:TokenIssueTime": "<ISO-8601 timestamp>" } }
  }
}

# Kubernetes — what actually revokes a bound service-account token.
# Modern tokens are bound to an API object; if that object is gone or its uid
# does not match, authentication fails immediately (60s after deletionTimestamp
# where finalizers are pending).
kubectl delete pod <agent-pod> -n <ns>              # pod-bound token
kubectl delete serviceaccount <sa> -n <ns>          # every token for the SA
# Deleting the SA does NOT remove the grant. Strip RBAC too, or a recreated SA
# of the same name inherits it.
kubectl delete rolebinding <binding> -n <ns>

#Phase 3 — Eradication

#Phase 4 — Recovery

#Phase 5 — Post-Incident

#Decision points

#Communications and notification triggers

Chapter 15 carries the full matrix. Three things are specific to this scenario.

The clock most likely to bind you is not the AI one. If retrieval leakage or an injection-driven exfiltration moved personal data, GDPR Article 33's 72 hours runs from the controller becoming aware — and awareness usually lands at Phase 1 step 1.8, when the reachability list tells you what the identity could touch. Record that timestamp deliberately.

Two more triggers worth pre-deciding. Notify your model or tooling provider when the incident involves their platform — you are often a data point in a campaign they can see across customers, and their abuse and security channels are the fastest route to knowing whether you are alone. And tell your users something true and early where an assistant produced or exposed content it should not have. The people who report these incidents are almost always ordinary users who noticed something odd and bothered to say so, and how you answer them determines whether the next one bothers.

#Automation notes

Safe to automate ungated — everything that gathers and nothing that acts: exporting agent traces and hashing them the moment a case opens; snapshotting the corpus; diffing live tool and MCP server definitions against pinned digests and raising an alert on drift; enumerating what an agent's identity can reach, into a target list that is never executed automatically; assembling the normalized UTC timeline; firing the S3 Object Lock legal hold and short-retention log exports.

Reversible, scoped and rate-limited — automate with logging, no approval: applying deny-all egress to a single AI workload; disabling a single tool or MCP server; revoking one agent's session where that agent has a named owner and a tested restore path. This is the general gate rule from Chapter 17 applied here: automation may gather, enrich, correlate and recommend freely; it may act only where the action is reversible, scoped and rate-limited; irreversible or organization-wide actions require a named human approver.

Requires a human gate: deleting a service account other workloads share; taking a customer-facing corpus offline; rolling back a model or adapter; attaching a quarantine SCP; deleting an OIDC provider, which breaks every role that trusts it; purging agent memory before it has been preserved.

And one gate specific to this playbook. Do not run an AI triage agent over the evidence in an AI incident without a human between it and any action. The evidence in a prompt-injection case is attacker-authored text engineered to be read by a language model, which makes your analysis pipeline the next target in the chain. The documented failure modes of agentic triage — overconfident closure on weak proof, and hallucinated detail in investigation narratives — are survivable on a phishing alert. Here you would be handing the attacker's script directly to the responder. Summarize with a model if you like. Act on a human.

Actionable takeaway: automate the capture, gate the cut.

#Pitfalls