Inventory every AI system touching your data, govern it against a standard an auditor recognises, use it in the SOC where it is actually good, and build the human process checks that stop an AI-enabled attacker — because the technology ones do not.
Who needs this: CISO · Security Architect · SOC Lead · Detection Engineer · AI/ML Platform Owner · GRC Lead · Head of IT | Read time: 32 min | Maps to: CSF 2.0 GOVERN, IDENTIFY, PROTECT, DETECT · CIS Controls 1, 2, 3, 6, 8, 15, 16, 17 · ISO/IEC 27001 A.5.1, A.5.2, A.5.9–5.11, A.5.19–5.23, A.8.8 · ISO/IEC 42001 · NIST AI RMF (AI 100-1) + GenAI Profile (AI 600-1)
Cyber warriors, let's start with the two incidents that ended the debate about whether any of this is real.
In November 2025 Anthropic disclosed GTG-1002, which it assesses with high confidence to be a Chinese state-sponsored group that ran an agentic intrusion campaign against roughly thirty global targets — large technology companies, financial institutions, chemical manufacturers, government agencies. The AI performed 80–90% of the campaign, with humans stepping in only at decision gates. It did reconnaissance, identified databases, researched vulnerabilities, wrote exploit code, harvested credentials, triaged the stolen data by intelligence value, built backdoors, and then wrote up its own attack documentation. The safeguard bypass was not clever cryptography. The operators role-played as employees of a legitimate security firm doing authorized penetration testing, and they decomposed the work into small tasks that each looked innocuous on their own (Anthropic).
Six months later, Sysdig's threat research team watched the second one happen in a customer's cloud. An LLM-driven actor exploited a vulnerability in a marimo notebook, then autonomously enumerated container-escape primitives, mounted the Docker socket, created a privileged container with /:/host, read /etc/shadow and SSH keys, and replayed a projected Kubernetes service-account token against the API server to dump the entire cluster Secret store — database credentials, AWS keys, and, with a certain poetry, OpenAI API keys. The tell that it was an agent: it parsed and acted on a canary directive hidden inside a JSON error response, and it unit-tested its own payload delivery with "hello" before running the escape scripts (Sysdig). Note the thing that matters most in that chain: the agent never needed an exploit for the escalation. It only needed the access its own runtime already carried.
Now the counterweight, and please put this in your program's stated assumptions before you spend a dollar. Mandiant's conclusion from over 500,000 hours of 2025 incident response is that 2025 was not the year breaches directly resulted from AI; most intrusions still stem from human and systemic failures (M-Trends 2026). AI today is a force multiplier on TTPs you already know, not a new kill chain — with the two agentic exceptions above. Anyone selling you an "AI-native" replacement for identity hygiene, logging and patching is selling you a very expensive hat.
This chapter covers four distinct problems that the industry insists on blending into one slide. Keep them separate, because they have different owners, different budgets and different failure modes: securing the AI you build or buy, governing it, using it in defense, and defending against attackers who use it.
NIST gives you the spine. The preliminary draft of **NIST IR 8596, the Cybersecurity Framework Profile for Artificial Intelligence — the "Cyber AI Profile" — was released 16 December 2025 with a comment period that closed 30 January 2026. It aligns to CSF 2.0 and organises the whole domain around three focus areas: Securing AI System Components (Secure), Conducting AI-Enabled Cyber Defense (Defend), and Thwarting AI-Enabled Cyber Attacks (Thwart)** (NIST, NIST IR 8596 iprd). NIST is separately developing SP 800-53 Control Overlays for Securing AI Systems.
I use those three plus a fourth — Govern — because Secure/Defend/Thwart are engineering activities and none of them survive contact with a board, an auditor or a customer questionnaire without a management system behind them.
| Problem | Focus area | Who owns it | The failure that tells you it is broken |
|---|---|---|---|
| Securing AI you build or buy | Secure | Security Architect + AI Platform Owner | You cannot list the AI systems that touch regulated data |
| Governing AI | (Govern) | GRC Lead, accountable exec named | Your AI policy exists but no system has an impact assessment |
| Using AI in defense | Defend | SOC Lead | An agent closed an alert and left no evidence for why |
| Defending against AI-enabled attackers | Thwart | SOC Lead + Head of IT (service desk) | Your payment-change process trusts a voice |
Because IR 8596 is a preliminary draft, do not write "compliant with NIST IR 8596" on anything. Use it as the structure for your gap analysis and your target profile — that is what a CSF Profile is for.
Actionable takeaway: Split your AI work into Secure / Govern / Defend / Thwart on one page, name a single accountable owner per row, and refuse any AI initiative that cannot say which row it belongs in. Programs that treat "AI security" as one bucket end up funding the exciting quarter of it and none of the boring three.
Every AI governance framework lands on the same first requirement, and it is the one where most programs fail. The AI system inventory is the AI analogue of CIS Control 1 — asset inventory — and it fails for the identical reason asset inventory always fails: the organization acquires new assets faster than the process that records them.
Shadow AI is not an aberration to be stamped out; it is the default state. Someone in finance is pasting a reconciliation into a consumer chatbot right now, and they are doing it because it works and nobody gave them a sanctioned alternative. If your first move is a ban, your second move is losing visibility entirely, because the traffic moves to personal devices where you have no telemetry at all.
| Field | Why it is load-bearing |
|---|---|
| System name and business owner | An owner who is a person, not a department |
| Provider, model family, and whether it is hosted, API, or on-prem | Determines where data physically goes and which regulator cares |
| Data classes it can read | The scoping input for your impact assessment and your DPA |
| Data classes it can write or act on | Separates an assistant from an agent; changes the risk class entirely |
| Identity it runs as | Human-delegated, shared service account, or its own principal (see Chapter 4) |
| Tools/functions it may call, and their blast radius | The confused-deputy surface |
| Egress destinations it can reach | The exfiltration channel in every prompt-injection chain |
| Approval record: who approved it, when, against what assessment | The single field an auditor will ask for first |
| Retention and training-use terms | Whether your data becomes someone's training corpus |
| Sub-processors behind the provider | Where the fourth-party risk lives (Chapter 11) |
You do not need a shadow-AI discovery product to get to 80% coverage. Six passes, roughly a day of work each, all against data you are already paying to store:
# Pass 3: find AI provider SDKs and API endpoints across every repo you have cloned locally.
# Returns file:line hits. Adapt the pattern list to the providers your egress logs surfaced in pass 1.
grep -rInE 'anthropic|openai|@google/generative-ai|google\.generativeai|bedrock-runtime|azure\.ai\.(inference|openai)|litellm|langchain|llama_index' \
--include='*.py' --include='*.ts' --include='*.js' --include='*.go' --include='*.java' \
--include='requirements*.txt' --include='package.json' --include='go.mod' --include='pom.xml' \
./repos/ | sort -uThe cheap version: if you have no budget at all, do passes 1, 2 and 5 only, quarterly, into a spreadsheet with the ten fields above. Three queries and a card statement will find the overwhelming majority of your shadow AI, and an approval column with a name in it is worth more to your auditor than a discovery product with nobody reading its output.
Actionable takeaway: Run the six discovery passes this month and publish the inventory with a "last verified" date in the header. Then give the business a sanctioned tool with a real data agreement — because every hour a sanctioned option does not exist is an hour your data spends somewhere you cannot see.
Three catalogs, all current, all free. Use them; do not write your own taxonomy.
OWASP Top 10 for LLM Applications 2025 — LLM01 Prompt Injection · LLM02 Sensitive Information Disclosure · LLM03 Supply Chain · LLM04 Data and Model Poisoning · LLM05 Improper Output Handling · LLM06 Excessive Agency · LLM07 System Prompt Leakage · LLM08 Vector and Embedding Weaknesses · LLM09 Misinformation · LLM10 Unbounded Consumption. Prompt injection holds the top slot for the second consecutive edition; System Prompt Leakage, Vector and Embedding Weaknesses, and Unbounded Consumption are the new-for-2025 entries (OWASP GenAI, 2025 PDF). A 2026 LLM edition is published on the same project site (OWASP GenAI LLM Top 10 2026) — pin whichever edition you map against and re-baseline deliberately rather than tracking "latest", the same discipline Chapter 9 applies to ATT&CK versions.
OWASP Top 10 for Agentic Applications 2026, released 9 December 2025 and built by more than 100 contributors, uses ASI01–ASI10 identifiers: ASI01 Agent Goal Hijack, ASI02 Tool Misuse, ASI03 Identity and Privilege Abuse, ASI07 Insecure Inter-Agent Communication, plus Agentic Supply Chain Compromise, Unexpected Code Execution, Memory and Context Poisoning, Cascading Agent Failures, and Rogue Agents. It maps real incidents to each category and defines an Agentic Development Lifecycle (ADLC) (OWASP GenAI, Agentic Security Initiative).
MITRE ATLAS v5.6.0 (Adversarial Threat Landscape for AI Systems) mirrors ATT&CK's structure deliberately, with 16 tactics including two AI-specific ones — AI Model Access (AML.TA0000) and AI Attack Staging (AML.TA0001) (mitre-atlas/atlas-data). Older references say "ML Model Access" and "ML Attack Staging"; the terminology shifted. Because ATLAS mirrors ATT&CK, it drops straight into an existing threat-informed-defense practice with the tooling you already run.
The useful move is not reciting the lists. It is deciding, per AI system in your inventory, which categories actually apply — most systems face four or five — and writing detections and design constraints for those. A retrieval chatbot over internal documents has an LLM01/LLM02/LLM08 problem and essentially no LLM06 problem. An agent with write access to your ticketing system and your cloud account is the reverse: LLM06 Excessive Agency and ASI02 Tool Misuse are the whole game.
Actionable takeaway: For each system in your AI inventory, record which OWASP LLM and ASI categories are in scope and which are explicitly out of scope, with the reason. A threat model that claims all ten apply to everything is a threat model nobody will use twice.
Here is the mechanism, stated plainly, because most vendor material dances around it.
Large language models process instructions and data on the same channel. There is no in-band separator that reliably distinguishes "this is a command from my operator" from "this is content I was asked to read." A model that reads an email, a Jira ticket, a wiki page, a fetched web page, a PDF, or an MCP tool description is reading text that an attacker may have written, and that text is arriving on the same channel as your system prompt. Direct prompt injection is a user typing an attack into the box. Indirect prompt injection is an attacker planting the payload in content the model will later ingest on someone else's behalf — and that is the one that turns into a breach.
EchoLeak (CVE-2025-32711) is the reference case and belongs in your training deck by name. Disclosed in June 2025 by Aim Security, it was a zero-click indirect prompt injection in Microsoft 365 Copilot, rated CVSS 9.3. A single crafted email carried instructions hidden in HTML comments and white text. Copilot ingested it into RAG context. Later, when the user asked Copilot an ordinary question, the hidden instructions caused it to retrieve sensitive tenant data and encode that data into a URL that was then automatically fetched. The chain evaded Microsoft's cross-prompt injection classifier, defeated link redaction using reference-style Markdown, and abused a Teams proxy to complete the exfiltration — a full privilege escalation across LLM trust boundaries with no user interaction at all. Microsoft patched server-side and reported no in-the-wild exploitation (arXiv analysis, HackTheBox writeup).
Read that chain again and notice what it did to the defenses that were present. There was an injection classifier. It was evaded. There was link redaction. It was defeated with a Markdown syntax variant. This is why I will not tell you that any filter prevents prompt injection. Input and output classifiers are speed bumps: they raise the cost of the trivial attack and they will be bypassed by anyone who tries twice. Buy them if they are cheap and already in your stack. Do not architect on them.
What actually reduces risk is architectural, and it comes down to breaking the "lethal trifecta" — private data access, exposure to untrusted content, and the ability to communicate externally, all in one agent (Simon Willison). Remove any one leg and the chain does not complete.
| Design rule | What it stops | Cheap implementation |
|---|---|---|
| Egress allowlist for any model-driven fetch or render; block auto-fetched images and reference-style links to arbitrary hosts | The exfiltration leg of EchoLeak-class chains | Deny-by-default egress on the app's network path; allowlist your own domains |
| Separate the context that reads untrusted content from the context that holds secrets — do not let one session do both | The private-data leg | Two API calls and a validated hand-off, not one prompt |
Treat all model output as untrusted input to whatever consumes it: encode before rendering, parameterize before querying, never eval | LLM05 Improper Output Handling; XSS and injection downstream | Your existing output-encoding library |
| Human confirmation on every state-changing action, showing the resolved parameters, not the intent | LLM06 Excessive Agency; ASI02 Tool Misuse | A confirmation dialog and an audit line |
| Enforce the user's permissions at retrieval time, not the application's | Cross-user data disclosure via retrieval | Pass identity into the query filter; test with a low-privilege account |
| Log the full prompt, retrieved context, tool calls and outputs for privileged agents | You cannot investigate what you did not record | Structured logs to your existing SIEM (Chapter 9) |
Actionable takeaway: For every AI system that can reach private data, prove on paper that at least one leg of the lethal trifecta is severed, and make that proof a deployment gate. If you cannot sever a leg, the system requires a human confirmation on every action that leaves the boundary — no exceptions for "internal only" tools.
An agent is a program that holds credentials, reads attacker-influenced text, and takes actions. Put that way, it is the confused-deputy problem with a language model in the middle, and the confused deputy is one of the oldest failure modes we have.
The Model Context Protocol and its peers are where this gets concrete in 2026. The named attack classes in circulation are worth learning as a set:
(Simon Willison, Microsoft — The state of MCP security in 2026)
None of this is theoretical. The Sysdig case from this chapter's opening is the confused deputy in production: the agent inherited a projected Kubernetes service-account token from a mounted volume and replayed it against the API server to dump the cluster Secret store. No exploit was required for the escalation — only the standing access the runtime carried (Sysdig).
| # | Action | Who | Done when | Evidence to capture |
|---|---|---|---|---|
| 1 | Assign the agent its own identity — never a shared service account, never a human's delegated token as the standing credential. | Security Architect | Principal exists with a named owner and an expiry | Principal ID, owner, creation ticket |
| 2 | Scope that identity to the minimum permission set, then write down the revocation procedure and test it before the agent handles real data. | Ops Lead (Identity) | A dry-run revocation completed and timed | Revocation runbook ID, test timestamp and elapsed time |
| 3 | Define the egress allowlist and apply it at the network layer. | Security Architect | Deny-by-default confirmed by a blocked test request | Policy ID, denied-request log line |
| 4 | Pin every tool/server to a specific version and record a hash of its tool definitions. | AI Platform Owner | Pinned reference committed to the repo | Version, digest, commit SHA |
| 5 | Configure alerting on any change to a pinned tool definition, and require re-approval before the change takes effect. | Detection Engineer | A test modification raises an alert and blocks | Alert rule ID, test evidence |
| 6 | Classify each tool as reversible or irreversible; require a named human approver on every irreversible one. | Incident Commander (policy owner) | Classification recorded per tool | Tool register with the approval class |
| 7 | Remove ambient credentials from the runtime — no automatic service-account token mounts, no reachable instance metadata unless the agent genuinely needs it. | Ops Lead (Cloud) | Agent runs and the token/metadata path is absent | Manifest diff, negative test result |
| 8 | Turn on full agent logging — prompts, retrieved context, tool calls, parameters, outputs — into the SIEM before production traffic. | Detection Engineer | Events visible in the SIEM with the right retention | Sample event, index name, retention setting |
Steps 1 and 2 must precede everything else, and the reason is not tidiness. If the agent's identity is a shared service account, step 2 has no meaningful answer: you cannot revoke it during an incident without breaking every other consumer of that account, so under pressure you will not revoke it at all. That is how a contained incident becomes an uncontained one. Step 7 must precede step 8's production traffic for the reason the Sysdig case demonstrates — logging an escalation you could have made structurally impossible is a poor trade.
The concrete Kubernetes control in step 7 is automountServiceAccountToken: false on any pod that does not need to talk to the API server. Chapter 6 covers the cluster-side work and Chapter 4 covers agent identity, scoping and revocation as an identity discipline; do not build a parallel process here.
Actionable takeaway: Before an agent touches production, run the revocation test and record how long it took. An agent whose credentials you cannot kill in under five minutes is not a productivity tool, it is an unmanaged privileged account with a chat interface.
Retrieval leakage is a permissions problem wearing a machine-learning costume. The most common serious defect in enterprise RAG is that the index is built once, with the ingesting service's permissions, and then queried by everyone — so the retrieval layer silently returns the union of what the pipeline could read rather than what the asking user is allowed to read. The fix is not a model fix. Enforce the requesting user's authorization at query time, and test it with a deliberately low-privilege account before launch. OWASP catalogs the broader class as LLM08:2025 Vector and Embedding Weaknesses, which also covers embedding inversion, cross-tenant leakage in multi-tenant vector stores, poisoning at the embedding layer, and unvalidated user-supplied filters flowing into vector-database query strings (OWASP).
Poisoning got much worse than the industry assumed, and this one is verified. Anthropic, the UK AI Safety Institute and the Alan Turing Institute showed that roughly 250 malicious documents suffice to install a backdoor in models from 600M to 13B parameters — a near-constant absolute number, not a percentage of the training corpus. A 13B model trained on twenty times more data was backdoored by the same 250 documents (Anthropic, Alan Turing Institute). The comfortable assumption — that scale dilutes poison — is wrong. Poisoning does not get harder as models get bigger.
For most readers this is not a "we train foundation models" problem; it is a fine-tuning and data-provenance problem. If you fine-tune or continue-train on scraped, user-submitted or vendor-supplied corpora, you need provenance on that data and a review gate on new sources. Maps to LLM04:2025 Data and Model Poisoning and LLM03:2025 Supply Chain.
The AI supply chain is a live attack surface, not a hypothetical. In March 2026, the actor TeamPCP backdoored the aquasecurity/trivy-action GitHub Action; LiteLLM's CI auto-installed the poisoned scanner, which stole LiteLLM's PyPI publishing tokens; malicious litellm releases shipped days later with the payload injected directly into the distributed wheels, running credential harvesting, then lateral movement across Kubernetes clusters, then a persistent systemd backdoor. The campaign spanned GitHub Actions, Docker Hub, npm, PyPI and OpenVSX in five days (Resecurity, LiteLLM first-party update). Read that chain carefully: a security scanner was the delivery vehicle into an AI infrastructure package. The May 2026 "Mini Shai-Hulud" wave then targeted the AI developer supply chain specifically, across 170+ npm packages and 404 malicious versions (CSA Labs, Singapore CSA AD-2026-009).
Practical controls, all of which belong to Chapter 11's supply-chain discipline applied to your AI stack: pin GitHub Actions by commit SHA rather than tag, use short-lived OIDC credentials instead of long-lived publishing tokens, isolate publish jobs, and require an SBOM for AI components — the 2026 CISA minimum elements explicitly extend SBOM scope to AI systems and SaaS (CISA). Note also that SWID tags were removed as an accepted SBOM format in the 2026 revision; SPDX and CycloneDX are the two current formats. For your own model artefacts, apply the same provenance discipline: signed artefacts, a registry with access control, and a record of which training and fine-tuning data produced which version.
Model theft and extraction — high-volume query-based distillation, side-channel signals, and insider access to production artefacts — was LLM10:2023 Model Theft in the prior edition and remains a documented, practically demonstrated attack class (OWASP, Praetorian). If your model is a competitive asset, rate-limit per authenticated principal, monitor for the query patterns that characterize systematic distillation, and treat the production weights as crown-jewel data under Chapter 8's classification scheme.
Actionable takeaway: Before your next RAG system ships, run one test — query it as a user who should see nothing, and confirm they see nothing. If retrieval permissions are not enforced at query time, stop the launch. Everything else in this section is a slower burn; that one is a data breach on day one.
Two standards, and they are complements rather than alternatives. The common real-world pattern is ISO/IEC 42001 as the certifiable management system, with NIST AI RMF as the risk operating model running inside it.
ISO/IEC 42001:2023 is the first international AI management system (AIMS) standard, published December 2023. It uses ISO's Harmonized Structure (Clauses 4–10), so it slots alongside ISO 27001 and 9001 with shared context, leadership, planning, support, operation, evaluation and improvement machinery, and it requires a Statement of Applicability justifying inclusion or exclusion of Annex A controls (AWS, ISMS.online). If you already run a 27001 ISMS, the integration cost is far lower than the sales pitch suggests, because the clause structure is the same one your internal audit program already knows.
NIST AI RMF 1.0 (AI 100-1), released 26 January 2023, has four functions — GOVERN, MAP, MEASURE, MANAGE — with GOVERN at the centre, cross-cutting the other three (NIST). Its Generative AI Profile, NIST AI 600-1 (26 July 2024) enumerates twelve risk categories unique to or exacerbated by generative AI and maps suggested actions onto that core. The twelve: CBRN information or capabilities; confabulation; dangerous, violent or hateful content; data privacy; environmental impacts; harmful bias or homogenization; human-AI configuration; information integrity; information security; intellectual property; obscene, degrading and/or abusive content; and value chain and component integration (NIST AI 600-1). Use those twelve as your impact-assessment scoping checklist and you will not have to invent one.
Not what it must say. What it must produce, in artefacts an auditor can pick up:
The acceptable-use policy is the part employees will actually read, so keep it to one page and make it specific: which tools are sanctioned, which data classes may go into which tool, that customer and regulated data never enters an unsanctioned service, that AI output touching customers or code is reviewed by a named human, and that AI-assisted code is subject to the same review and provenance rules as any other. Add the two questions people genuinely need answered: where does my data physically go, and is it used for training. Answer both per sanctioned tool, in the policy, in plain language.
Most published guidance on this is now stale, so read this carefully. The Digital Omnibus on AI, adopted as Regulation (EU) 2026/1744, was published in the Official Journal on 24 July 2026 and entered into force 27 July 2026 — days before the original 2 August 2026 high-risk deadline.
Two practical consequences. If you ship a GPAI model above the systemic-risk threshold, you have a live serious-incident duty to the AI Office today. If you ship an Annex III high-risk system, the AI Act clock does not start until December 2027 — but GDPR, product liability and, if it is a product with digital elements, the Cyber Resilience Act still bite in the meantime. Chapter 15 owns the full notification matrix; do not build a separate AI notification process beside it.
Data sovereignty and sub-processors deserve their own line in your vendor process rather than a paragraph in a policy nobody opens. For each sanctioned AI tool, record the processing region, whether the provider commits to not training on your data, the sub-processor list, and the notice period for adding a sub-processor. Then re-read that list quarterly, because your SaaS vendors are adding AI sub-processors faster than they are sending you emails about it. Chapter 11 owns the contract clauses.
Actionable takeaway: Produce artefacts 1, 2, 3 and 7 from the list above in the next quarter — policy with an owner, inventory, impact assessments, and the wire from AI incidents into your existing IR process. Those four convert an AI governance slide into an AI governance program, and every remaining artefact is easier once they exist.
I use AI every day and it has made me measurably faster. It has also confidently told me things that were not true, in a tone of complete certainty, at exactly the moment I was tired enough to believe it. Both of those sentences have to be true at once for you to deploy this well.
High-volume, evidence-based tasks that are easy to verify after the fact: enrichment (reputation, geo and ASN, asset owner, user context, prior alert history), deduplication and correlation, ticket creation and routing, evidence collection, and closure of known-good alert classes with a documented rationale. Alert triage is the clearest production use case for AI agents today (Panther, Panther on triage agents).
Anything irreversible, anything whose blast radius scales with a false positive, and anything requiring organizational context the automation does not have. Concretely: auto-isolating a device, auto-containing at scale, auto-attaching a quarantine SCP with org-wide reach, auto-deleting an OIDC provider that every role trusts, auto-draining a node that is holding your evidence.
Two recur in production, and they are the two you must design against: overconfident closure backed by weak proof, and hallucinated detail in investigation narratives. Alongside them: hallucination on ambiguous alerts, blindness to novel attack patterns, and missing organizational context. The sharpest statement of the risk is worth memorizing — the agent acts on a confident hallucination before a human sees it (Panther, UnderDefense, Kaspersky).
The failure mode that scares me most is not the false negative. It is the beautifully written investigation narrative with three invented details that a tired analyst signs off at 04:00 because it reads like every good report they have seen. Fluency is not accuracy, and AI is extremely good at fluency.
Teams that succeed report the same phased pattern: enrichment first → summaries → autonomous closure of known-good alert classes, with each phase gated by measured analyst confidence in the previous one, not by a vendor's readiness assessment. Autonomy is then configured per action class — fully autonomous, human-on-the-loop, human-in-the-loop — with every agentic decision logged.
The order is not arbitrary. Enrichment is verifiable at a glance and fails safe. Summaries are where you learn your agent's specific hallucination signature on your data, and you need that knowledge before you grant it any authority. Skipping to autonomous closure means you find out about the hallucination signature from an incident review instead of a metric.
Measure two things and put them on the SOC dashboard: agent-closure rate, and spot-check accuracy from a random sample of agent-closed alerts re-reviewed by a human every week. If spot-check accuracy is not being measured, the closure rate is not a metric, it is a wish. Chapter 9 owns SOC metrics; Chapter 17 owns the automation architecture.
This is the highest-value defensive use of AI that nobody demos, because it is unglamorous. AI is genuinely good at drafting a Sigma rule from a threat report, at proposing field mappings across log schemas, and at generating the benign-sample edge cases a human would not think to test. It is bad at knowing whether the rule will drown your queue on Monday.
So keep the human structure and let AI fill it. Palantir's Alerting and Detection Strategy (ADS) framework requires nine documented sections per detection: Goal, Categorization (ATT&CK mapping), Strategy Abstract, Technical Context, Blind Spots and Assumptions, False Positives, Validation, Priority, Response (palantir/alerting-detection-strategy-framework). The two sections teams skip are the two in bold, and those are precisely the two an AI will confabulate most convincingly, because they require knowing your environment. Write those two yourself.
Then make the CI pipeline do the arguing. A minimum viable detection-as-code pipeline has four gates: schema and lint on every rule; conversion succeeds for every configured backend; the rule fires against a stored true-positive sample; and the rule does not fire against a stored benign sample. That fourth gate is what catches AI-generated detection slop before it reaches an analyst (SigmaHQ, Splunk on detection-as-code). Chapter 9 owns the pipeline in detail.
Automated timeline generation is the other quiet win — assembling a first-draft chronology from logs during an incident, for a human to correct. Chapter 17 covers it; the rule here is simply that the draft is labeled a draft and the Scribe owns the authoritative timeline.
Actionable takeaway: Deploy AI in the SOC in the order enrichment → summaries → closure, and do not advance a phase until you have four weeks of measured spot-check accuracy on the current one. And keep the analyst's judgment on escalations — augment your people, do not replace the judgment that decides when to wake the executive sponsor.
Now the other direction. Here is what is actually documented, with the vendor hype stripped out.
| Case | When | Outcome |
|---|---|---|
| Arup (Hong Kong office) | Jan–Feb 2024; victim named May 2024 | ~US$25.6M lost across 15 wire transfers in one day. Began with a phishing email impersonating the UK-based CFO; the employee's scepticism was overcome by a multi-person video conference in which every other participant was AI-generated (CNN) |
| WPP (CEO Mark Read) | May 2024 | Unsuccessful. WhatsApp account using a public photo → Teams meeting → voice clone plus YouTube footage of a senior exec, with the attacker impersonating Read in the meeting chat (OECD AI Incidents) |
| Ferrari | July 2024 | Blocked. An executive received a WhatsApp voice clone of the CEO authorizing a transfer, and challenged the caller with a shared-secret question — a recently recommended book — that the clone could not answer (AI Incident Database) |
| LastPass | April 2024 | Blocked. An employee received calls, texts and a WhatsApp voicemail with a voice clone of the CEO, and flagged the channel anomaly rather than detecting the fake (Adaptive Security) |
Read the last column of the three blocked cases and notice what stopped them. Not a detection product. A human process check — an out-of-band channel anomaly and a shared-secret challenge. That is the mitigation your playbook must encode, because it is the one with a documented record of working.
The structural backdrop: voice phishing was the #2 initial infection vector in 2025, at 11% of all Mandiant investigations (M-Trends 2026), and DBIR 2026 similarly finds voice and text phishing convert better than email (Help Net Security). The FBI's IC3 reported $20.877B in total 2025 losses across 1,008,597 complaints — the first year over one million — with BEC at $3.047B across 24,768 complaints, and introduced "AI-related" as a formal crime descriptor for the first time: 22,000+ complaints and roughly $900M in losses (FBI).
The service desk is the other front door. CISA's advisory on Scattered Spider / UNC3944 / Octo Tempest, last updated 29 July 2025, documents attackers researching employees on business platforms and social media, then calling the IT help desk posing as them to obtain password resets and MFA token transfers to attacker-controlled devices, often splitting the request across separate contacts to evade detection (CISA AA23-320A). Add a convincing voice clone to that call and the last remaining control — the agent's instinct that something sounded off — is gone.
| # | Control | Who owns it | Why it works |
|---|---|---|---|
| 1 | Payment and payee-change requests require callback on a number from the vendor master record, never one supplied in the request | Finance, with CFO sign-off | Removes the attacker's control of the channel; this is the Arup control |
| 2 | A shared challenge phrase for executive-authorized financial instructions, rotated quarterly, never sent over the channel it protects | Executive Sponsor | This is exactly what stopped the Ferrari attempt |
| 3 | Help-desk account-recovery and MFA re-enrolment require out-of-band verification against an authoritative source, with no exception for a caller in a hurry | Head of IT | Removes the urgency lever that CISA documents as the standard play |
| 4 | A tenant-wide freeze switch for help-desk-initiated MFA re-enrolment, pre-approved and tested | Incident Commander | Converts a slow policy fix into a containment action available in minutes |
| 5 | Train on the channel, not the artefact: a video call is not proof of identity, and neither is a familiar voice | Comms Lead + Head of IT | LastPass blocked its attack on a channel anomaly, not on fake detection |
| 6 | Two-person rule above a defined payment threshold, with the second person contacted independently | Finance | Requires the attacker to win twice through separate channels |
Note what is not in that table: deepfake detection software. If it is already bundled in your stack, fine, use it as a signal. Do not build the control on it, and do not let anyone tell the board it is the mitigation. The three cases that were stopped were stopped by process.
Detection-side, wire these to your SIEM: help-desk-initiated MFA method changes correlated with a sign-in from a new device within a short window; payee bank-detail changes in the ERP correlated with recent inbound contact to that employee; and executive-impersonation domain and display-name lookalikes in mail flow. Chapter 14.9 has the full deepfake and AI-enabled social-engineering playbook; Chapter 14.2 has BEC and payment fraud; Chapter 4 has help-desk verification as an identity control. This section exists to tell you which controls are load-bearing, not to duplicate the response steps.
Hoxhunt has run a longitudinal experiment since 2023 across more than 70,000 real-world simulations, pitting AI-generated phishing against elite human red-team spear phishing. AI went from **31% less effective in 2023, to 10% less in 2024, to 24% more effective by March 2025 — a 55-point swing in two years (Hoxhunt). Microsoft's MDDR 2025 states AI can make some phishing operations up to 50× more profitable by scaling targeting, and that Microsoft blocked $4B of fraud and scams between April 2024 and April 2025 (Microsoft MDDR 2025). ENISA's ETL 2025 found phishing and its variants — vishing, malspam, malvertising — accounted for roughly 60% of all initial infection vectors** across 4,875 EU incidents (ENISA ETL 2025).
The operational consequence is blunt: the spelling-and-grammar heuristic is dead, and any awareness training still teaching it is actively harmful because it gives people a confidence signal that no longer correlates with anything. Retrain on structure — unexpected urgency, a request to change a payment destination, a channel switch, an authentication prompt you did not initiate — and put the money into phishing-resistant MFA, which blocks over 99% of identity attacks even when the attacker already holds a valid username and password (Microsoft MDDR 2025). Chapter 4 owns that migration.
This is the fastest-moving item in the chapter and the one most distorted by marketing, so here is the honest picture — a discovery/exploitation gap.
The capability is real. Anthropic's Project Glasswing, announced April 2026 and gated to roughly fifty partner organizations, found more than 10,000 high- or critical-severity vulnerabilities in systemically important software, including a 27-year-old OpenBSD flaw and a 16-year-old FFmpeg bug that automated fuzzing had tested around five million times without finding (Anthropic). Anthropic's own red-team assessment reports that its prior model failed at autonomous exploit development almost entirely, while the newer one reached a 72.4% success rate in the Firefox JS shell (Anthropic red team).
And now the number that should govern your patch queue. VulnCheck found that of 1,061 vulnerabilities attributable to AI-assisted discovery, only 14 — 1.3% — have been confirmed exploited in the wild; for Glasswing specifically, 23,019 findings yielded 126 published CVEs, of which exactly one has been confirmed exploited (VulnCheck). One.
Translation for your vulnerability program: AI-discovered CVEs are inflating your patch queue far faster than they are inflating your actual exploitation risk. Expect vendor advisory volume to rise sharply without a proportional rise in incidents. Prioritize on exploitation evidence — KEV and EPSS — not on CVE count, and do not let a rising open-vulnerability number panic anyone into abandoning risk-based prioritization. Chapter 10 owns that model.
Actionable takeaway: Implement the callback-on-file-number rule and the executive challenge phrase this month; they cost nothing and they are the two controls with a documented record of stopping real deepfake fraud. Then tell your board plainly that AI-assisted vulnerability discovery is raising advisory volume, not exploitation rates, so that your KEV-driven prioritization survives the next scary headline.
If you read this chapter and felt the budget anxiety, here is the sequence I would run with nothing but staff time. The order is deliberate: each step makes the next one cheaper.
| Week | Do this | Why it comes here |
|---|---|---|
| 1–2 | Discovery passes 1, 2 and 5 — egress logs, OAuth grants, expense lines. Publish the inventory. | Everything downstream needs the list. Scoping without it is guesswork. |
| 3 | One-page acceptable-use policy with a named owner, and one sanctioned tool with a real data agreement. | A ban without an alternative moves the traffic somewhere you cannot see. |
| 4 | Payment callback rule and executive challenge phrase. Brief Finance and the service desk. | Zero cost, highest documented loss avoidance in this chapter. |
| 5–6 | Help-desk out-of-band verification runbook plus the tenant-wide MFA re-enrolment freeze switch, tested. | Turns the most-attacked human process into a controlled one. |
| 7–8 | For each AI system that reaches private data, prove one leg of the lethal trifecta is severed. Fix or gate the ones that fail. | Architectural, so it holds when the guardrail model does not. |
| 9–10 | RAG permission test with a low-privilege account. Own agent identity plus a timed revocation test for every agent. | The two tests that catch the day-one breaches. |
| 11–12 | Wire AI incidents into the existing IR process (RS.MA) and run one tabletop on the 14.11 scenario. | Governance you can evidence, using the process you already have. |
Chapter 21 sequences this against everything else competing for the same twelve weeks.
Actionable takeaway: Do weeks 1 through 4 even if you do nothing else. Inventory, policy, sanctioned tool, callback rule. That is four items, no procurement, and it moves you from "we have no idea" to "we know and we have a floor."
Securing AI is mostly asset management with better marketing; governing it is mostly writing down what you already decided; defending with it works right up to the moment you stop checking its homework; and defending against it comes down to whether a human being will pick up the phone and dial a number they already trusted. Inventory it, scope it, log it, and keep a person on the escalations — the machines are fast, but they have never once been accountable.
[IG1] [ID.AM] [CIS 1] [CIS 2] [A.5.9][IG1] [ID.AM] [DE.CM][IG1] [ID.AM] [CIS 3][IG1] [GV.PO] [A.5.1][IG1] [GV.RR] [A.5.2][IG1] [GV.SC][IG1] [RS.MA] [CIS 17] [A.5.24][IG1] [PR.AT] [CIS 14][IG1] [PR.AA] [CIS 6][IG1] [PR.AT] [CIS 14] [A.6.3][IG2] [PR.DS] [CIS 16][IG2] [PR.AA] [CIS 3][IG2] [PR.AA] [CIS 5] [CIS 6][IG2] [PR.PS] [CIS 4][IG2] [GV.SC] [DE.CM][IG2] [GV.RR] [RS.MI][IG2] [DE.CM] [CIS 8][IG2] [DE.AE] [RS.AN][IG2] [DE.CM] [CIS 8][IG2] [GV.SC] [CIS 16] [A.5.19][IG2] [ID.RA] [ISO 42001][IG2] [GV.SC] [CIS 15] [A.5.19][IG3] [GV.SC] [ID.RA][IG3] [GV.OC] [RS.CO][IG3] [PR.DS] [CIS 3]