How to write a playbook that a machine can execute and a human can take over mid-step, where to put the approval gates, and which automations will quietly hurt you.
Who needs this: SOC Manager, Detection Engineer, Automation Engineer, Incident Commander, CISO | Read time: 26 min | Maps to: DETECT (DE.AE, DE.CM), RESPOND (RS.MA, RS.AN, RS.MI, RS.CO), IDENTIFY (ID.IM), GOVERN (GV.RR) | CIS 8, 13, 17 | ISO A.5.24, A.5.26, A.5.28, A.8.15, A.8.16
Cyber warriors, here is the number that ended the debate about whether to automate: Mandiant's 2025 frontline data puts the median hand-off window between an initial-access broker and the group that buys the access at 22 seconds — down from more than eight hours in 2022 (M-Trends 2026). Twenty-two seconds. You cannot page a human, wait for them to find their laptop, and still be inside that window. Anything that has to happen in the first minute has to happen without a person in the path.
And here is the number that should stop you from automating everything: in the same dataset, global median dwell time was 14 days, and 52% of organizations found the intrusion themselves. The other 48% were told. A response program tuned entirely for the 22-second window and not at all for the 14-day investigation is a program that will contain the alert it saw and miss the intrusion it did not.
So this chapter is not "automate your SOC." It is a specific engineering claim: every step you write in a playbook should be convertible into a workflow step, and the playbook should run on two tracks at once — humans and machines working the same document, with explicit handover points where one hands control to the other. The machine takes the parts that are fast, repetitive, verifiable and reversible. The human keeps the parts that require organizational context, judgement about blast radius, and accountability. The interesting engineering is in the seam between them.
Chapter 2 covered how to write a playbook. Chapter 9 covered detection engineering. Chapter 13 covered incident command and severity. This chapter covers what happens when you point a robot at all three.
Most playbooks cannot be automated, and the reason is not the tooling. It is that the steps are written as sentences instead of as operations. "Investigate the affected host and determine scope" is a paragraph in a document, not a step. Nobody can tell you when it is finished, what it consumed, or what it produced.
A step is convertible when it has five properties. This is not a style preference — it is the minimum interface a workflow engine needs, and it is also, not coincidentally, exactly what a tired human needs at 03:00.
| Property | The test | What breaks without it |
|---|---|---|
| Atomic | The step does one operation against one system. If the verb list contains "and," split it. | Partial failure leaves the workflow in an undefined state; no responder knows which half completed. |
| Explicit precondition | Stated as a machine-checkable condition, not an assumption. "Diagnostic settings are exporting Entra sign-in logs to a retained store." | The automation runs against a system that cannot answer, and returns a confident empty result. |
| Machine-checkable done-when | An observable end state: an API returns a specific value, a record exists, a count is zero. Not "the host is contained" but "GET on the device returns isolationState: Isolated." | You cannot tell success from silent failure, so retries and rollback are impossible. |
| Idempotent | Running it twice produces the same end state as running it once. | Retry logic — the thing that makes automation reliable — becomes the thing that causes damage. |
| Reversible, or explicitly marked irreversible | Either the step names its own undo operation, or it is flagged as one-way and therefore gated. | Automation happily performs actions that no human would have signed off on. |
Two of these deserve a moment because they are where real playbooks fail.
Idempotency is a property of the API you call, not of your intention. AWS's session revocation is a good citizen: the console action attaches an inline policy named AWSRevokeOlderSessions to the role, denying sessions issued before a timestamp, and running it again simply refreshes that timestamp (AWS IAM). Deleting an OIDC identity provider is also idempotent — and also catastrophic, because "deleting an OIDC provider does not update roles that reference it. Any attempt to assume such roles will fail" (AWS CLI reference). Idempotent and safe are different words.
Preconditions are where automation lies to you most often. A workflow that queries Microsoft Defender XDR's CloudAppEvents table for OAuth activity returns nothing at all if Defender for Cloud Apps is not deployed with the Microsoft 365 activities connector enabled — the table is simply unpopulated, and the query succeeds (Microsoft Learn). A workflow that searches the Purview audit log for the Consent to application operation the moment an alert fires may find nothing, because "it can take from 30 minutes up to 24 hours for the corresponding audit log entry to be displayed in the search results after an event occurs" (Microsoft Learn). Google Workspace's OAuth Token log events carry a documented lag of "a couple of hours" (Google). An automated OAuth-abuse check that runs at T+2 minutes and reports "no malicious grants found" is not a control. It is a false negative with a timestamp on it, and a responder will read it as evidence.
Actionable takeaway: rewrite one existing playbook this week with the five-column discipline — Action / Who / Precondition / Done when / Evidence — and mark every step AUTO, AUTO+GATE, or HUMAN. You will find that a third of your steps cannot be converted because they were never really steps. Those are the ones failing at 03:00 too.
If you want a formal target to write toward, OASIS CACAO Security Playbooks v2.0 is the closest thing the field has to a normative machine-readable playbook schema. Its workflow step types — start, end, action, playbook-action, parallel, if-condition, while-condition, switch-condition — are precisely the control-flow primitives a human-readable playbook needs anyway, and its top-level properties include the ones home-grown playbooks always forget: valid_until and revoked (a playbook that expires), derived_from (provenance), signatures (integrity), and workflow_exception (what to do when the playbook itself fails) (OASIS CACAO v2.0). Open-source CACAO orchestrators exist (SOARCA) — which matters because it means your playbook logic can leave a vendor UI without being retyped.
I will be blunt about this, because vendor materials will not be.
Enrichment. Reputation lookups, geo/ASN resolution, asset owner and criticality, user context and manager, device posture, prior-alert history for the same entity. It is read-only, high-volume, and wrong answers are visible rather than destructive. This is the single highest-return automation in any SOC and the one that most reduces analyst time-to-first-judgement.
Deduplication and correlation. Collapsing forty alerts about one host into one case. Cheap, reversible, and it directly attacks the alert-fatigue problem — the peer-reviewed synthesis on alert fatigue in SOCs notes cited industry studies reporting false-positive rates as high as 99% (Tariq et al., ACM Computing Surveys 57(9), 2025).
Ticket creation, routing and case scaffolding. Open the case, attach the alert, populate the entity list, set the SLA clock, page the right rota. Every second of this a human spends is a second not spent thinking.
Evidence collection. Urgent rather than merely useful, because your evidence has a shorter life than your investigation. Microsoft Entra ID audit and sign-in logs retain 7 days on Free and 30 days on P1/P2, and "log retention changes aren't retroactive" (Microsoft Learn). CloudTrail console Event history is a hard 90 days of management events only (AWS). Automate the export, not the analysis.
Containment of well-scoped known-bad. Note all three qualifiers. Well-scoped: one host, one account, one key. Known: the detection has a validated true-positive history, not a hypothesis. Bad: a match on a KEV-listed exploit attempt or a confirmed-compromised credential, not an anomaly score.
Timeline generation and status updates. Its own section below — the most underrated automation in the building.
Scoping decisions. Deciding that an incident is bigger than the alert requires knowing what else is in the blast radius, and that knowledge lives in an asset inventory that is, in most organizations, aspirational.
Severity judgement. Severity keys off business impact, and NIST is explicit that prioritization depends on "asset criticality, functional impact of the incident, data impact of the incident, stage of observed activity, threat actor characterization, and recoverability" (NIST SP 800-61r3). Automation can propose a severity from a rubric. A human owns it.
Anything irreversible. Deleting an OIDC provider. Terminating an instance before evidence capture. Wiping a device. Killing a pod — AWS states it plainly: "Gather forensic evidence before removing the node — an attacker might attempt to destroy evidence through termination" (EKS Best Practices).
Anything that touches production availability. Draining a Kubernetes node is the perfect example of a step that looks automatable and is not: kubectl drain respects PodDisruptionBudgets, meaning a PDB can block your containment drain entirely, or the drain can succeed and evict the very pod holding your evidence (Kubernetes).
Anything whose blast radius scales with a false positive. Microsoft recommends containing no more than 100 devices at a time in Defender for Endpoint for performance reasons (Microsoft Learn). An automation with no cap will find that limit for you, in production, at 04:00.
Actionable takeaway: list every automation you currently run and sort each one into three columns — read-only, reversible, irreversible. Anything sitting in the irreversible column with no human on it comes out of production this week. And if enrichment is not your largest category by volume, you built the exciting automations before the profitable ones.
A gate is not a speed bump. It is a place where the automation stops, hands a human a decision it cannot legitimately make, and waits — and the whole design problem is that this happens at 03:00 to someone who was asleep four minutes ago.
Chapter 13 sets out the fatigue evidence. Its consequence for gate design is narrow and specific: a tired approver can still follow a rule, but they cannot improvise and they cannot reconstruct missing context. So the gate must supply the context, not request it.
Everything below goes on one screen, in the tool the approver is actually holding — the paging app, not a dashboard behind SSO they cannot reach from a phone.
| Field | Content | Why it is there |
|---|---|---|
| Proposed action | The literal operation and its target: "Isolate LAPTOP-4471 (full isolation, Defender for Endpoint)." | Removes ambiguity about what "contain" means for this tool. |
| Trigger | The detection name and its validated true-positive rate over the last 90 days. | Lets the approver weight the evidence without opening the SIEM. |
| Blast radius | Who and what stops working. Named owner, business service, user count. | This is the decision. Everything else is input. |
| Reversibility | "Reversible: release-from-isolation, effective ~1 min" or "IRREVERSIBLE." | The single strongest predictor of how careful the approver should be. |
| Default on timeout | "No action at T+10 min; escalates to on-call IC." Or the reverse, if the safe default is to act. | A gate with no timeout default is a gate that hangs. |
| Two buttons | Approve / Decline. A third option is a research project. | Choice architecture. Three options at 03:00 is a conversation. |
Treat the workflow engine as a Scribe that never gets tired, and hold it to the same standard you hold a human Scribe to. For every gated action, the record must contain:
Actionable takeaway: take your most-fired automated action and try to reconstruct, from logs alone, what a specific approver saw at a specific moment three weeks ago. If you cannot, you do not have an audit trail — you have a status field.
Severity determines how much autonomy the machine gets. Chapter 13 defines SEV-1 through SEV-4; this is the automation ladder bolted onto it. Note that autonomy goes down as severity goes up — which is the opposite of what most teams build, because the high-severity cases are the ones where speed feels most valuable and where a wrong action is most expensive.
| Severity | Automation posture | Machine may | Machine must not |
|---|---|---|---|
| SEV-4 | Fully autonomous | Enrich, correlate, deduplicate, create and close the case with attached evidence | Act on any production system |
| SEV-3 | Autonomous with notification | All of the above, plus single-entity reversible containment (one host, one session, one key) and evidence capture | Contain more than one entity; act on a tier-0 or critical asset |
| SEV-2 | Human-on-the-loop | Prepare and stage every containment action, run all evidence collection, draft the timeline and comms | Execute containment without an approval; touch identity infrastructure |
| SEV-1 | Human-in-the-loop, IC-directed | Collect evidence, generate timeline, distribute status, hold the staged actions ready | Execute anything not individually directed by the IC |
Three rules govern movement on this ladder.
Round up under uncertainty. PagerDuty's rule generalizes cleanly: "If you are unsure which level an incident is… treat it as the higher one," reassessed at the postmortem, never during (PagerDuty). For automation that means low classifier confidence escalates the severity, which lowers the autonomy. Uncertainty should cost the machine authority, not grant it.
Critical-asset location overrides everything. CISA's NCISS scores "Location of Observed Activity" on a modified Purdue model where level 3 is Business Network Management — admin workstations, Active Directory, trust stores — and levels 6 and 7 are Critical Systems and Safety Systems (CISA NCISS). Encode that as a hard gate: any proposed automated action whose target sits at level 3 or above requires a human, regardless of how confident the detection is. This is the defensible, non-arbitrary reason your automation may isolate a laptop and may not isolate a domain controller.
Aggregation escalates. NCISS's campaign rule — "if three or more component incidents have the same high water mark, the overall campaign's priority level is raised to the next level" — has no equivalent in most SOAR platforms. Implement it. Three autonomously-closed SEV-4s on three hosts in the same subnet within an hour is not three SEV-4s.
Actionable takeaway: write your own autonomy ladder against your own severity levels, then check it against one question — does the machine get less authority as severity goes up? If any row grants more, you have built a speed setting and called it a safety control. Encode the critical-asset list as a hard exclusion before you enable the next autonomous rule, not after the first one hits a domain controller.
Draw this once, put it on a wall, and mark every arrow with what happens when it breaks. Most architecture diagrams show the arrows working. The useful diagram shows them failing.
[ Telemetry ] identity · endpoint · cloud control plane · network · SaaS · email
|
| (1) ingest — normalized, timestamped UTC, schema-versioned
v
[ SIEM / data platform ] --(2) detection fires--> [ SOAR / orchestrator ]
^ | | | | |
| | | | | |
| (7) enrichment + action results written back | | | | |
+------------------------------------------------+ | | | |
| | | |
(3) query/act --> [ EDR ] isolate · collect package · scan
(4) query/act --> [ IAM / IdP ] revoke sessions · disable · block CA
(5) create/update --> [ Ticketing / case ] case of record, SLA clock
(6) notify --> [ Comms ] war-room channel · paging · status page
|
v
[ Evidence store ] WORM, separate trust
domain, out-of-band credentialsTwo structural rules before the failure modes.
The SOAR must not authenticate through the identity plane it may be asked to contain. This is the same principle that governs backup credentials: if your orchestrator signs in with SSO against the IdP, and the incident is an IdP compromise, your response tooling is inside the blast radius of your own containment action. CISA's playbook says the general version explicitly — segment and manage SOC systems separately from broader enterprise IT so that "IR and defensive systems and processes will be operational during an attack" (CISA Federal Playbooks).
Every integration needs a break-glass manual path, and it needs to be printed. CISA's advice on the plan applies with more force to the automation: "Print these documents and the associated contact list and give a copy to everyone you expect to play a role in an incident. During an incident, your internal email, chat, and document storage services may be down or inaccessible" (CISA IRP Basics). The paradox of playbooks-as-code is that the artefact must survive the loss of the systems that host it.
| Link down | What you observe | What actually happens | Required design |
|---|---|---|---|
| (1) Ingest | Dashboards look calm | Detections cannot fire. Silence reads identically to safety. | Heartbeat monitoring per log source with an alert on absence; a daily "sources that stopped reporting" report |
| (2) SIEM → SOAR | Alerts in the SIEM, no cases created | Queue builds silently; SLA clocks never start | Queue-depth alarm and a manual triage rota that activates on orchestrator outage |
| (3) SOAR → EDR | Containment action shows as submitted | If the device is offline, Defender for Endpoint retries for up to three days, then you must reissue (Microsoft Learn) | Never treat "submitted" as "contained." Poll for the end state and alert on pending actions older than the window |
| (4) SOAR → IAM | Session revocation returns success | Access tokens live until expiry — and in CAE sessions token lifetime increases to long-lived, up to 28 hours, with propagation latency of up to 15 minutes (Microsoft Learn) | Verify containment by observing that no new tokens are issued and no new sign-ins occur, not by the API return code |
| (5) SOAR → Ticketing | Actions taken, no case | The response has no record of authority, no timeline, no chain of custody | Ticketing is the case of record; if it is down, the automation must halt gated actions and fall back to the printed log |
| (6) SOAR → Comms | No one is paged | The 22-second window becomes a morning discovery | Two independent paging paths, tested monthly. Test the distribution list itself — Equifax's own account of its breach records that "the recipient list for the notice was out-of-date and, as a result, the notice was not received by the individuals who would have been responsible for installing the necessary patch" (GAO-18-559) |
| (7) Write-back | Analysts re-run enrichment by hand | Duplicate work, contradictory findings in the same case | Enrichment results are written to the case, once, with the timestamp and the source |
Actionable takeaway: draw your own version of that diagram on one page and write, beside every arrow, the name of the alert that fires when it stops. The arrows with nothing written next to them are your silent failures — build those alerts first. Then answer one question in writing: if the identity provider is the incident, can your orchestrator still sign in? If the answer is no, that is the project for this quarter.
I use AI every day, and I will tell you exactly where it earns its place and exactly where it will hurt you.
Triage — the clearest production use case today: pulling context from six systems, comparing an alert to prior instances of the same detection, and producing a ranked recommendation with the evidence attached (Panther). Summarization — turning ninety log lines and four tool outputs into a paragraph a responder reads in fifteen seconds; high value, low risk, because the source material is right there to check. Enrichment reasoning — not just fetching the reputation score, but noticing the same ASN appeared in an alert eleven days ago on a different host. And drafting: first-pass incident narratives, customer notifications, detection logic. Draft is the operative word.
Overconfident closure backed by weak proof, and hallucinated detail in investigation narratives, are the two that recur in production, alongside failure on ambiguous alerts, blindness to novel attack patterns, and missing organizational context. The sharpest statement of the risk is that "the agent acts on a confident hallucination before a human sees it" (Panther; UnderDefense; Kaspersky).
A hallucinated narrative is worse than a hallucinated answer, because a narrative is exactly the artefact that gets pasted into the incident record, read by the IC, and eventually handed to a regulator. Wrong facts in a timeline are wrong facts under legal privilege review.
Here is the part almost nobody has designed for. Your triage agent reads attacker-controlled text as part of its job. Phishing email bodies. HTTP user-agent strings. Filenames. Process command lines. Registry values. User-submitted ticket bodies. Web page content the agent fetched to enrich a domain. Every one of those is a field an adversary can write into, and every one lands in the agent's context window.
The structural cause is not a bug you can patch: LLMs process instructions and data on the same channel, so there is no reliable in-band separation between content and command. Treat every model-adjacent data source — email, ticket, wiki page, web fetch, PDF, tool description — as untrusted input to a privileged executor. This is LLM01 Prompt Injection, which has held the top slot for two consecutive editions of the OWASP Top 10 for LLM Applications, alongside LLM06 Excessive Agency (OWASP GenAI); in the agentic taxonomy it is ASI01 Agent Goal Hijack and ASI02 Tool Misuse (OWASP Top 10 for Agentic Applications 2026).
Two documented facts should end any argument that this is theoretical.
EchoLeak (CVE-2025-32711) was a zero-click indirect prompt injection in Microsoft 365 Copilot, CVSS 9.3, disclosed June 2025. A single crafted email with instructions hidden in HTML comments and white text was ingested into RAG context; when the user later asked Copilot an unrelated question, the hidden instructions caused it to retrieve sensitive tenant data and encode it into an auto-fetched URL — evading Microsoft's cross-prompt-injection classifier, defeating link redaction using reference-style Markdown, and abusing a Teams proxy. No user interaction. Microsoft patched server-side (arXiv analysis).
And the tell in the second confirmed agentic intrusion is the one that should worry a SOC specifically: Sysdig observed an LLM-driven actor parse and act on a canary directive hidden in a JSON error response (Sysdig). An agent read text in a tool output and followed it. That is the same mechanism as your triage agent reading an attacker's email body. The attacker's version does not say "ignore previous instructions"; it says something that looks like an internal note explaining that this alert class is a known false positive and should be closed.
Use AI to augment your analysts, not to replace their judgement — especially on escalations. The deployment sequence that teams report working is enrichment first, then summaries, then autonomous closure of a narrow set of known-good alert classes, with each phase gated on measured analyst confidence in the previous one, and autonomy configured per action class rather than globally (Panther). And keep this counterweight in the assumptions section of your program: Mandiant's conclusion from over 500,000 hours of 2025 incident response is that 2025 was not the year breaches directly resulted from AI, and most intrusions still stem from human and systemic failures (M-Trends 2026). AI is a force multiplier on both sides of the wire. It is not yet the wire.
Actionable takeaway: before your agent goes near a live queue, run a red-team pass where the team writes injection payloads into the fields the agent actually reads — subject lines, filenames, user-agent strings, ticket bodies — and measure how often it changes its recommendation. If nobody on your team has tried to talk your agent into closing a true positive, your agent has not been tested. It has been demoed.
This is the automation with the best ratio of value to risk in the entire building, and it is the one teams build last.
The timeline. Google's incident-management guidance is unambiguous that "the incident commander's most important responsibility is to keep a living incident document" (Google SRE Book), and PagerDuty assigns a dedicated Scribe to capture an accurate record of what happened, when, and what decisions were made (PagerDuty). Both are correct, and both are the first thing that degrades at hour six of a SEV-1.
So write the mechanical half automatically. Every orchestrator action, every gate decision, every tool response, every state change, appended to one immutable ordered record with UTC ISO 8601 timestamps. Then have the human Scribe add the half a machine cannot produce: what the IC decided and why, what was considered and rejected, what the room believed at the time. A machine-generated timeline is a record of actions; an incident timeline is a record of reasoning. The machine writing the first is what frees the Scribe to write the second.
The order matters for evidence too. Automated collection should follow the order of volatility from RFC 3227 — registers and cache; routing table, ARP cache, process table, kernel statistics, memory; temporary file systems; disk; remote logging and monitoring data; physical configuration and network topology; archival media (RFC 3227) — with the cloud amendment that the "remote logging" tier is frequently the most important evidence and the shortest-lived. Export before you contain.
Status updates. The Internal Liaison delivers executive updates on roughly a 30-minute cadence, kept short and to the point (PagerDuty). Automate the assembly — current severity, systems affected, actions taken and verified, actions pending approval, next update time — and let a human send it. Never let automation publish externally. NCSC's rules on what to say exist because retraction is expensive: "avoid saying anything that may have to be retracted later," and avoid compromising future investigations through "speculation or premature conclusions about the cause or extent of the incident, or who is behind it" (NCSC). No template engine has judgement about that. Chapter 15 owns the external notification decision entirely; automation's job there is to start the clock and name the owner, not to draft the statement.
Actionable takeaway: turn on automatic timeline capture for the next incident you declare, whatever its severity, and hand the Scribe the machine's record instead of a blank page. Then ask one question at the after-action: does the timeline say what was decided and why, or only what was clicked? The first is an incident record. The second is a log with nicer formatting.
Four ways to hurt yourself, each with a real mechanism.
Auto-containment that takes down production. The blast radius of an automated isolate is whatever the target turns out to be. Isolating a Hyper-V host blocks network traffic to all its child VMs. Web proxies configured by PAC or WPAD can prevent a device recovering from isolation at all, which is why Microsoft recommends selective isolation in those environments, and a device behind a full VPN tunnel cannot reach the Defender cloud service once isolated — you need split tunnelling for the management traffic or the device is simply gone. On Linux, an isolated device is released from isolation if an administrator modifies or adds an iptables rule (Microsoft Learn). All of that is in the vendor documentation, and all of it will surprise a team that automated the happy path.
Auto-blocking that an adversary weaponises. Any automation that takes an action based on an attacker-controllable signal is an availability weapon pointed at you. If reporting a sender auto-blocks that sender, a phisher spoofs your payroll provider and reports it. If N failed logins auto-disables an account, an adversary disables your executives on a Friday afternoon — and, worse, generates the noise that hides the one account they actually took. The cheap mitigations are the same in every case: rate limits per rule and per hour, a hard daily cap, an allowlist of never-auto-actioned identities and assets (executives, break-glass accounts, service principals, DCs, DNS and DHCP), and a required second signal from an independent telemetry source before any action fires. Microsoft's own guidance carries the same shape of caution in a different context — it explicitly recommends against turning off integrated applications tenant-wide as a response to OAuth consent abuse (Microsoft Learn). The broad lever is available. It is still the wrong lever.
Runaway loops. A containment action generates telemetry. Telemetry fires a detection. The detection triggers the containment workflow. Congratulations, you have built a machine that isolates your fleet one host at a time until someone notices. Every workflow needs a maximum execution count per window, a maximum affected-entity count per run, a loop-detection guard on its own generated events, and a global kill switch that one on-call person can reach in under a minute without SSO. Test the kill switch quarterly. An untested kill switch is a comment in a runbook.
Tipping your hand. The subtlest one, and the failure mode that turns a contained incident into a nine-month one. Mandiant's articulation is the clearest published statement: "incident responders must recognize that each defensive action may prompt the adversary to react: organizations should delay implementing actions that will directly disrupt the attacker until they are ready to eradicate the threat completely." The documented chain when you contain piecemeal is that responders remove known compromised systems, feel accomplished, tip their hand — and the adversary, using backdoors on systems the responders do not know about, abandons the burned infrastructure and takes steps to ensure continued access, leaving the responders "blind, and unaware" until an outside party notifies them again (Aldridge, Black Hat USA 2012). CISA states the same tension in playbook language: develop as complete a picture as possible of the adversary's capabilities and reactions to "avoid 'tipping off' the adversary" (CISA Federal Playbooks).
Automation is a whack-a-mole machine by default. It sees one mole and hits it, at machine speed, before anyone has asked whether there are eleven others. The design answer is a campaign flag: when the case system holds an open investigation into a suspected intrusion — as opposed to a discrete alert — automated containment for related entities switches from execute to stage, and the IC releases the staged actions together as a single remediation event. Aldridge is clear that whack-a-mole is still correct in some cases, such as cash being stolen in near real time. It is a decision, and it belongs to the IC, not to a workflow.
State this as policy, in one sentence, and enforce it in code review of the workflow:
No automated action ships without a tested, documented, single-command rollback that does not depend on the connectivity or credentials the action itself removed.
Test it against these questions. If your automation isolates an endpoint, what releases it, who can run that, and does it work when the device is unreachable? Microsoft provides a downloadable force-release script from the device page, but only for Windows (Microsoft Learn). If your automation contains a Falcon host, the reverse action is lift_containment on the Hosts API, requiring hosts write scope (CrowdStrike Developer Center). If it attaches a quarantine SCP, who detaches it, and is that person's access dependent on the account you just quarantined? And if it disables an Entra account, know the cost of undoing it first: re-enabling has a documented delay of 15 minutes for SharePoint and Teams and 35–40 minutes for Exchange Online (Microsoft Learn).
Actionable takeaway: pick your three highest-volume automated actions and, for each, write down the rate limit, the never-auto-action allowlist, the loop guard, and the one-command rollback — then test that rollback this month against an offline host. Any action on that list you cannot undo in one command does not ship until it can be.
The number everyone reports is the fraction of alerts closed without human touch. Report it. Then never let it stand alone, because it is the easiest metric in security to game and the consequences are invisible for months.
Here is the failure: an automation that closes 80% of alerts looks identical, on that dashboard, to an automation that closes 80% of alerts including the four true positives it misclassified. The number goes up as the SOC gets worse — the same defect that makes MTTR dangerous, since a falling detection time with rising false negatives is a worse SOC that looks better. Chapter 16 owns the board-level metric set; this is the operational one.
Measure it as a set, or do not measure it:
| Metric | Definition | Why it is in the set |
|---|---|---|
| Autonomous closure rate | % of alerts closed with no human interaction, by detection | The headline. Meaningless alone. |
| Spot-check accuracy | % of a random sample of autonomously closed alerts, re-reviewed by a human, judged correctly closed | The honesty control. Sample weekly, blind, and publish the number next to the closure rate. |
| Escalation precision | Of alerts the automation escalated to a human, % that were genuinely worth escalating | Detects an over-cautious agent that has quietly become a routing layer |
| Gate response time | Median and p95 time from approval request to decision, by hour of day | If p95 at 03:00 is 40 minutes, your gate design is broken, not your people |
| Gate timeout rate | % of approvals that hit their default because nobody answered | The leading indicator of approval fatigue |
| Rollback rate | % of automated actions subsequently reversed, by action type | A rising rollback rate on one action is a defect report on that automation |
| Automation availability | % of time the orchestrator and each integration were healthy | Nobody measures this, and then nobody knows the SOAR was down for six hours on a Sunday |
| Time-to-verified-containment | Detection to observed containment (no new tokens, no new sessions, no new API calls), not to API success | The only containment number that is not a self-report |
Actionable takeaway: stand up the blind weekly spot-check before you turn on a single autonomous closure rule. Not after. Not once you have volume. Before. If you cannot staff the spot-check, you cannot staff the autonomy.
Every automation you build is a decision you made in daylight and will execute at 3am without waking you. That is the entire point, and it is also the entire risk. Write the step so a machine can run it, gate the step so a human owns it, log the step so the after-action can read it, and test the undo before you ever need it. Stay scripted, stay reversible, and never let a robot close a ticket it cannot show its work on.
AUTO, AUTO+GATE, or HUMAN, and the classification is recorded in the playbook itself. [IG1] [RS.MA] [CIS 17][IG1] [RS.MA] [A.5.26][IG2] [RS.MA][IG2] [RS.MI][IG2] [RS.MI] [CIS 17][IG1] [GV.RR] [A.5.24][IG2] [RS.MA][IG2] [RS.AN] [A.5.28][IG2] [RS.AN][IG2] [RS.MA][IG1] [RS.MI] [CIS 1][IG2] [RS.MI][IG2] [RS.MI][IG2] [RS.MI][IG3] [RS.MI] [RS.MA][IG2] [PR.AA] [A.5.24][IG2] [DE.CM] [CIS 8][IG2] [RS.MI] [A.8.16][IG1] [RS.AN] [A.5.28] [CIS 8][IG2] [RS.AN] [A.5.28][IG2] [PR.AA] [RS.MI][IG3] [ID.IM] [DE.AE][IG1] [RS.CO][IG2] [ID.IM][IG3] [ID.IM]