How to configure a cloud control plane so it produces evidence, detect the identity and misconfiguration attacks that actually happen there, and contain a compromised account, instance, cluster or workload without destroying the only proof you will ever get.
Who needs this: Cloud platform engineers, SREs, security engineers, detection engineers, incident responders, CISOs signing the log-retention budget | Read time: 27 min | Maps to: CSF 2.0 IDENTIFY, PROTECT, DETECT, RESPOND | CIS Controls 3, 4, 5, 6, 8, 13 | ISO 27001 A.8.9, A.8.15, A.8.16, A.5.28
Welcome back, cyber warriors. Pour the coffee, because this is the chapter where the abstractions stop and the commands start.
In May 2026, Sysdig's threat research team watched an LLM-driven attacker work a cloud environment hands-on-keyboard. It exploited a vulnerability in a marimo notebook, enumerated its own escape options, found an exposed Docker socket, launched a privileged container with the host filesystem bind-mounted at /:/host, read /etc/shadow and the SSH keys, then replayed a projected Kubernetes service-account token against the API server and dumped the cluster's entire Secret store — database credentials, AWS keys, OpenAI API keys. The tell that it was an agent and not a person: it parsed a canary directive hidden inside a JSON error response and acted on it, and it unit-tested its own payload delivery with "hello" before running the escape scripts (Sysdig).
Read that chain again and notice what is missing. No IMDS call. No zero-day in Kubernetes. No malware. A misconfigured socket, a mounted token, and standing permission did the whole job. That is the shape of cloud compromise in 2026: cloud-conscious intrusions are up 37% overall and 266% among state-nexus actors, and 35% of cloud incidents involve valid account abuse (CrowdStrike 2026 Global Threat Report). Meanwhile 82% of CrowdStrike's detections in the period were malware-free. Your EDR has nothing to say about any of this. The evidence lives entirely in the control plane, and the control plane only remembers what you paid it to remember.
That last point is the one that costs organizations their investigations. Nearly every major cloud breach of recent years landed on the customer's side of the shared-responsibility line — misconfiguration, identity, exposed data — not on the provider's. And nearly every failed cloud investigation failed for the same banal reason: the logs that would have answered the question had a default retention of seven days, thirty days, or ninety, and the question got asked on day ninety-one.
This chapter is about closing both gaps before you need them closed, and about what to do in the first hour when you did not.
Every vendor slide about shared responsibility shows the same two-color stack, and every one of them is technically correct and operationally useless. The useful version is the one in the providers' own words.
AWS frames it as **security of the cloud versus security in the cloud. AWS protects "the infrastructure that runs all of the services offered in the AWS Cloud." You own "the guest operating system (including updates and security patches), other associated application software," and the configuration of firewalls and security groups. The sentence people skip is the one that matters most: "Customer responsibility will be determined by the AWS Cloud services that a customer selects"** (AWS shared responsibility model). Run EC2 and you carry nearly everything above the hypervisor. Use S3 or DynamoDB and AWS operates deeper into the stack, leaving you managing data, encryption options, classification and IAM. Two services, same account, completely different obligations. Your responsibility is not a property of "the cloud" — it is a property of each service you turned on, and it changes every time an engineer adopts a new one.
Microsoft mirrors the model with explicit IaaS / PaaS / SaaS boundaries, and adds the constant that a lot of teams get wrong: data, endpoints, account and access management are always the customer's, in every service model (Microsoft shared responsibility). There is no tier of service you can buy where identity becomes somebody else's problem.
Google states shared responsibility and then argues past it, framing the relationship as "shared fate" — the position being that a clean boundary leaves customers standing alone on the wrong side of it, so Google pairs it with secure-by-default foundations, blueprints and risk-transfer programs (Google Cloud). Whatever you think of the framing, it points at something real: a boundary is not a control.
| Layer | Provider owns | You own | Where teams get it wrong |
|---|---|---|---|
| Facilities, hardware, hypervisor, provider network backbone | Yes | No | Assuming this coverage extends upward into your VMs |
| Guest OS, patching, agents | No | Yes (IaaS) | "It's managed, so it's patched" — true for PaaS, false for EC2/GCE/Azure VMs |
| Application code, dependencies, container images | No | Yes | Base-image CVEs treated as the registry's problem |
| Network controls (security groups, NSGs, firewall rules, NetworkPolicy) | No | Yes | Believing a default VPC is a secure VPC |
| Identity, accounts, roles, keys, tokens, consent grants | No | Yes, in every service model | Expecting the IdP to be secure because the vendor is |
| Data, classification, encryption choices, key custody | No | Yes, in every service model | Server-side encryption treated as a data-governance answer |
| Control-plane log generation | Provider generates | You must enable, route, retain and pay for it | Assuming logging is on because the service exists |
| Regulatory notification when your data is breached | No | Yes | The single most expensive misunderstanding in the table |
That last row is the whole point. Shared responsibility is a responsibility boundary, not a liability boundary. When a provider has an incident, your regulator does not send the provider a letter. It sends you one. Chapter 15 covers what the clocks look like; Chapter 11 covers the contractual clauses that make a provider tell you in time to meet them.
The practical consequence for this chapter is narrower and more urgent: the boundary determines evidence availability. Your forensic capability stops where the provider's plane begins. You cannot subpoena a hypervisor. Everything you will ever know about an incident in your tenant has to have been logged, routed and retained by decisions you made before the incident started.
Actionable takeaway: Build a one-page responsibility matrix per service, not per provider, and make "who owns the logs, and for how long" a mandatory row. Any service in production without an owner named in that row is an unowned service — assign it this week or turn it off.
Every cloud attack you will investigate ends up as a question about API calls: who called what, from where, with which credential, and what did it return. The control-plane log is the only witness. So the first design decision in cloud security is not a tool — it is a retention policy with a budget attached.
The international logging guidance is blunt about the default: "Default log retention periods are often insufficient." The same document notes that "in some cases, it can take up to 18 months to discover a cyber security incident and some malware can dwell on the network from 70 to 200 days before causing overt harm," and it tells you specifically to log "all control plane operations, including API calls and end user logins… configured to capture read and write activities, administrative changes, and authentication events" (Best Practices for Event Logging and Threat Detection, PDF). Note deliberately what it does not do: it sets no single numeric minimum. Anyone telling you "CISA requires twelve months" is quoting OMB M-21-31, a US federal memo binding on federal civilian agencies, not this guidance.
So you have to pick your own number. Here are the defaults you are picking against.
| Log source | Default retention | The trap |
|---|---|---|
| CloudTrail Event history (console) | 90 days of management events in a Region, immutable (docs) | It is not a trail. No S3 object-level visibility, hard 90-day wall |
| CloudTrail trails → S3 | Whatever the bucket lifecycle policy says | A lifecycle rule written by a cost engineer silently sets your evidence window |
| CloudTrail Lake event data store | Up to 3,653 days (~10 yrs) on one-year extendable pricing, or 2,557 days (~7 yrs) on seven-year retention pricing; query results viewable 7 days | Not on by default; costs money; must exist before the incident |
| CloudTrail data / Insights events | Off. "Trails and event data stores log management events, but not data or Insights events" | S3 object reads and Lambda invocations are invisible until you opt in |
| Azure Activity log (subscription control plane) | 90 days, collected by default, then deleted; entries cannot be changed or deleted (Activity log) | The Azure answer to CloudTrail Event history, with the same hard wall. A diagnostic setting to Log Analytics, Storage or an Event Hub is the only way past 90 days |
| Azure resource (diagnostic) logs | Not collected at all. "Resource logs aren't collected by default. To collect them, you must create a diagnostic setting for each Azure resource" (resource logs) | Per resource, not per subscription. Key Vault access, storage data-plane reads, database queries — all invisible until somebody configures each one |
| Entra ID audit + sign-in logs | 7 days Free / 30 days P1 / 30 days P2 (Entra data retention) | Thirty days is shorter than the time it takes most organizations to notice |
| Entra risky sign-ins | 7 days Free / 30 days P1 / 90 days P2 | The one place P2 buys real retention |
| Microsoft Graph activity logs | P1/P2 only, and not retained at all unless routed to storage/analytics | Licensed but empty is the worst of both worlds |
| Microsoft Purview Audit (Standard) | 180 days (raised from 90; records generated on/after 2023-10-17) (audit retention policies) | Separate system from Entra logs, separate licensing |
| Purview Audit (Premium) | 1 year; 10 years requires the add-on plus a custom retention policy that is actually created and targeted | Buying the add-on and never creating the policy retains nothing extra |
| GCP Admin Activity + System Event | 400 days, _Required bucket, not configurable and not deletable (Cloud Logging retention) | The longest non-configurable default in the table — and the reason people forget the next row |
| GCP Data Access + Policy Denied | 30 days in _Default, and Data Access is off by default except BigQuery | "We're on GCP, we have 400 days" is half true and the wrong half |
| Google Workspace admin/login/OAuth/Drive | 6 months; email log search 30 days (data retention and lag) | OAuth token events lag by a couple of hours — a consent-grant hunt run immediately returns a false negative |
Two sentences from that table should end up on a wall somewhere.
The first is Microsoft's, and it is the single most expensive fact in cloud IR: "Log retention changes aren't retroactive. When you upgrade from Free to P1 or P2, only data still within the free retention period (up to seven days) is available. Data that has already expired can't be recovered unless it was previously archived." You cannot buy your way out of this on day one of an incident. Upgrading a license mid-investigation gets you the logs from that moment forward, and nothing before it.
The second is Google's, and it cuts the other way: "Administrators cannot delete log event data or change the length of time that the data is available." In Workspace, that is an evidence-integrity feature — an attacker with admin cannot shorten your window. In AWS and Azure, they very much can, which is why Stealth:IAMUser/CloudTrailLoggingDisabled is a GuardDuty finding type in the first place.
SearchQueryInitiatedExchange and SearchQueryInitiatedSharePoint, which tell you what an intruder searched for, arguably the highest-signal record of intent you can get. CISA gives the command shape: Set-Mailbox <identity> -<sign-in type> @{Add="SearchQueryInitiated"} (CISA Microsoft Expanded Cloud Logs Implementation Playbook).The cheap version. If you cannot fund a full SIEM ingest of every cloud log, do this instead and you will still be able to investigate. Send the control plane only — CloudTrail management events, Entra sign-in and audit logs, the Azure Activity log, GCP Admin Activity — to cheap object storage with a lifecycle that goes to a cold tier at 30 days and expires at 12 to 18 months, with Object Lock or the platform equivalent turned on. Query it with the provider's own query engine when you need it: CloudTrail Lake takes SELECT-only Trino-dialect SQL with the event data store ID as the FROM value, driven from the CLI with start-query, describe-query, get-query-results, and --delivery-s3-uri to write results to S3 (Lake queries with the CLI). Hot search is a luxury. Having the data at all is not.
Actionable takeaway: This week, run one query per provider — "what is our oldest retained control-plane event?" — and write the answer on the risk register. If the answer is under twelve months, you have an evidence gap, not a logging strategy. Fix the retention before you buy another detection tool, because a detection you cannot investigate is a notification you cannot scope.
You do not need every service on this list. You need to know what each one is actually good at, so you stop paying for overlap and start covering gaps.
| Service | What it is genuinely good at | What it is not |
|---|---|---|
| Amazon GuardDuty | Managed threat detection over CloudTrail, DNS and flow data. Its IAM finding types are the fastest signal that credentials have left the building | Not a config scanner. Not a source of truth for activity volume (see the ML caveat below) |
| AWS Security Hub | Aggregation and normalization to ASFF, standards-based posture checks, single pane across accounts | Not an investigation tool; it tells you that, not how |
| Amazon Detective | Builds a behavior graph from CloudTrail, VPC Flow Logs and GuardDuty findings using ML, statistics and graph theory; finding groups correlate related findings and entities, severity-scored on ASFF (finding groups) | Not a detector. It answers "what else did this principal touch," after something else has alerted |
| IAM Access Analyzer | Five analyzer types; the IR-relevant ones are external access (what is shared outside your zone of trust), internal access, unused access, and policy generation from CloudTrail activity (overview) | External access analyzers are Region-scoped — one per Region or you have blind Regions |
| Microsoft Defender for Cloud | The Azure resource-plane equivalent of GuardDuty plus Security Hub in one product: CNAPP combining CSPM posture with CWPP workload alerts across subscriptions, and across AWS and GCP once connected. Defender for Resource Manager is the one to enable first for IR — it monitors control-plane operations for unusual and potentially harmful activity (Defender for Cloud) | Posture (Foundational CSPM) is free; the threat detection is not. Workload alerts arrive only for the specific plans you enabled, so "Defender for Cloud is on" says nothing about whether storage, containers or Key Vault are actually covered |
| Microsoft Defender XDR advanced hunting | KQL across identity, endpoint, mail and cloud-app tables; CloudAppEvents carries OAuthAppId, ActionType, AccountObjectId, IPAddress, UserAgent, IsAdminOperation, RawEventData, plus LastSeenForUser and UncommonForUser anomaly columns (CloudAppEvents) | CloudAppEvents is populated only if Defender for Cloud Apps is deployed and the Microsoft 365 activities connector is enabled. Otherwise your queries return nothing, silently |
| Google Security Command Center — Event Threat Detection | Near-real-time matching over Cloud Logging streams against known IoCs, adversarial techniques and behavioral anomalies; at org level it can also monitor Google Workspace streams (ETD overview) | Only sees what Cloud Logging carries — Data Access logs off means Data Access detections blind |
| SCC Container Threat Detection | Findings from low-level observed behavior in the container guest kernel (threat detection in SCC) | Runtime behavior, not image or manifest posture |
Four operational notes that will save you an embarrassing status update.
Security Hub is the normalization layer, and that is worth more than its dashboard. Most teams enable it, look at the compliance score, and never wire it into anything. The IR value is in three places. First, ASFF: Security Hub "processes finding data using the AWS Security Finding Format (ASFF), a standard finding format," which "eliminates the need to manage findings from myriad sources in multiple formats." It receives findings from GuardDuty, Inspector, Macie and the other integrated services, which means your SOAR writes one parser instead of five, and a playbook trigger written against ASFF fields keeps working when you turn on a new detection service. Second, cross-account aggregation is your first scoping question. Security Hub "consolidates your security findings across accounts and provider products" — so "is this confined to one account, or is the same finding type live in six?" is a filter, not an investigation. Ask it before you decide on a per-account or org-level containment. Third, workflow status is your case-tracking hook. Findings carry NEW, NOTIFIED, SUPPRESSED and RESOLVED, settable through aws securityhub batch-update-findings and automation rules (workflow status, Security Hub CSPM). Two traps come with it: Security Hub "only detects and consolidates findings that are generated after you enable" it, so it is worthless for a retrospective question, and marking a finding RESOLVED or SUPPRESSED "doesn't prevent Security Hub CSPM from generating a new finding for the same issue" — suppression is a triage note, not a mute button. Note also that AWS now brands the service Security Hub CSPM; if your runbooks say "Security Hub," check you are pointing at the right product page.
GuardDuty goes quiet on sustained activity. AWS documents it plainly: "If GuardDuty observes continued activity from a remote host, its ML model will identify this as an expected behavior. Therefore, GuardDuty will stop generating this finding" (GuardDuty IAM finding types). Persistent exfiltration eventually stops producing new findings. Never treat finding volume as a proxy for activity volume, and never close an incident because the alerts stopped.
Two GuardDuty families deserve dedicated routing. The UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.* and .../ResourceCredentialExfiltration.* findings mean credentials are demonstrably outside your control — the Resource variants cover Lambda functions and ECS tasks, not just EC2, and on the .InsideAWS variants you pivot on service.action.awsApiCallAction.remoteAccountDetails.accountId and .affiliated. The behavioral families (Persistence:, PrivilegeEscalation:, Exfiltration:IAMUser/AnomalousBehavior) mean escalation or staging is in progress, and Stealth:IAMUser/CloudTrailLoggingDisabled means somebody is turning off the witness. Chapter 14.3 lists the full trigger set for the account-takeover playbook.
Plan around the SCC tiering change. The Security Command Center Enterprise service tier shuts down on 21 May 2027, and organizations on Enterprise move automatically to Premium on or after that date (SCC release notes). If your GCP detection design assumes Enterprise-tier features, put the migration on the roadmap now rather than discovering the gap in a renewal cycle.
Actionable takeaway: For every cloud detection you own, record three separate facts — do we have the telemetry, does the logic exist and is it enabled, and has it fired on a validated test within the last 90 days. Chapter 9 covers the coverage model in full. Anything not green on all three is a named gap with a named owner, not a covered technique.
Cloud security spending skews toward threat detection, and cloud breaches skew toward misconfiguration and standing permission. That mismatch is the whole reason these two acronyms exist.
CSPM — Cloud Security Posture Management — answers "is anything configured wrongly." Public buckets, unencrypted volumes, open management ports, disabled logging, unrestricted security groups, missing IMDSv2 enforcement. It is a continuous config audit, mapping to CIS Control 4 and CSF's PR.PS.
CIEM — Cloud Infrastructure Entitlement Management — answers "who could do what if they wanted to." This is the harder and more valuable question, because permission is invisible until it is used. A role with * on s3 looks identical in a console to a role with three scoped actions, right up until the morning it is used to copy a database.
CIEM is the more urgent of the two because non-human identities now dominate cloud estates. CI runners, service accounts, app registrations and workload identities vastly outnumber human accounts and carry standing privilege that no MFA prompt ever guards; Mandiant records the theft of hard-coded keys and personal access tokens as a routine MFA-bypass path (M-Trends 2026), and the Sysdig case above ended in a Secret dump with no human credential involved at any point. Chapter 4 owns machine identity lifecycle; what belongs here is the cloud-specific measurement: for each principal, what could it reach, and when did it last actually use that reach?
The expensive version is a commercial CSPM/CIEM platform with graph-based blast-radius analysis across accounts and providers. On a large multi-cloud estate it earns its keep, mostly by making "who could reach this data" a query instead of a project.
The cheap version works, and you can start it this quarter:
One prioritization rule beats any vendor's severity score: fix the misconfigurations that grant identity first. A public S3 bucket is a data-exposure incident. An over-permissive role trust policy is every incident, forever, because it is the machine that manufactures the next compromise.
Actionable takeaway: Stand up an unused-access analyzer in every account this month and delete the top 20 unused privileged grants it finds. It is free, it is reversible, and it is the highest-yield security work available to a team with no budget.
The instance metadata service exists so a workload can get credentials without an engineer embedding a key. It is a genuinely good design. It is also a credential vending machine reachable at a fixed link-local address from anything running on the host — which means any server-side request forgery in your application is, potentially, a credential theft primitive. Shai-Hulud, the self-replicating npm worm, specifically harvests from cloud metadata endpoints alongside CI pipelines (Unit 42, CISA alert). This is not an edge case any more; it is a standard step in commodity tooling.
AWS's own CloudTrail investigation guidance gives you the pivots (Part 1, Part 2). Learn these fields; they are the difference between "we think something happened" and a defensible timeline.
| Field | What it tells you |
|---|---|
ec2RoleDelivery | A value of "1.0" explicitly confirms IMDSv1 was used to obtain the credential. This is the single most load-bearing field for answering "was this SSRF-to-IMDS?" |
userIdentity.type | AssumedRole vs IAMUser |
userIdentity.principalId | Role ID plus session name — the session name is attacker-chosen and frequently masquerades as something plausible like a migration script |
userIdentity.sessionContext.attributes.mfaAuthenticated | Whether MFA was present on the session |
userIdentity.sessionContext.sessionIssuer | The role-assumption chain |
sessionCredentialFromConsole | Console-derived versus programmatic credential |
readOnly | Separates reconnaissance (true) from modification (false) |
awsRegion | Cross-Region evasion — query every Region, not just the one you got paged about |
| Key prefix | AKIA = long-term IAM user key; ASIA = STS short-term credential (compromised credentials) |
The classic signature is an ASIA credential belonging to an instance role, calling from a source IP that is not in AWS. Add ec2RoleDelivery: "1.0" and you have both the theft and the mechanism in one record.
AWS's investigation checklist from Part 2 is worth following literally: query all Regions for that role's session activity, correlate CloudTrail timestamps against VPC Flow Logs for the actor's source IP, and then hunt IAM write events for persistence — CreateUser, CreateAccessKey.
IMDSv2 requires a session token obtained via a PUT request, which defeats the naive SSRF pattern. The commands are documented (modify instance metadata options):
# Require IMDSv2 (session token required) on an existing instance.
# --http-endpoint must be set whenever --http-tokens is set.
aws ec2 modify-instance-metadata-options \
--instance-id i-1234567890abcdef0 \
--http-tokens required \
--http-endpoint enabled
# Restrict how many network hops the PUT response may travel.
# A limit of 1 blocks container-to-IMDS in many topologies — that is the point,
# and also the reason it can break things. Test before fleet-wide rollout.
aws ec2 modify-instance-metadata-options \
--instance-id i-1234567890abcdef0 \
--http-put-response-hop-limit 3 \
--http-endpoint enabled
# Turn IMDS off entirely on an instance that does not need it.
aws ec2 modify-instance-metadata-options \
--instance-id i-1234567890abcdef0 \
--http-endpoint disabledDo the pre-flight check or you will cause an outage. AWS documents it: the MetadataNoToken CloudWatch metric tracks IMDSv1 calls, and "when MetadataNoToken records zero IMDSv1 usage for an instance, the instance is then ready to require IMDSv2" (configure IMDS options). Watch the metric until it is flat at zero, then enforce. Reversing that order is how a well-intentioned hardening sprint takes down a payments service.
Precedence matters when you roll this out at scale: launch parameter beats account-level default beats the AMI's ImdsSupport: v2.0 setting. Account-level enforcement is HttpTokensEnforced via ModifyInstanceMetadataDefaults; once it is enabled, a launch specifying HttpTokens=optional fails. That is the control you want in a production account — it makes the insecure configuration unlaunchable rather than merely discouraged. Note also that a hop limit of 1 "can cause issues" in container environments, which is exactly where you most want it; treat container topologies as a per-cluster test, not a fleet-wide flag flip.
Part 2's containment line, for an instance you already believe is compromised, is aws ec2 modify-instance-metadata-options --http-tokens required --http-put-response-hop-limit 1.
Actionable takeaway: Enable account-level IMDSv2 enforcement (HttpTokensEnforced) in every non-production account today and every production account after MetadataNoToken sits at zero. Enforcement at the account default is worth ten times the same setting applied instance-by-instance, because it survives the next Terraform module somebody copies from a blog post.
This is the section to bookmark. Everything below is plain, sequenced and boring on purpose — a responder reading it at 03:00 should find no jokes and no ambiguity.
The governing principle: preserve, then scope, then contain in one burst, then verify. The order exists because cloud evidence is short-lived and cloud containment is loud. A containment action taken before preservation can permanently remove the only record of what happened. A containment action taken piecemeal hands the adversary a window between each step.
| # | Action | Who | Destroys evidence? | Done when |
|---|---|---|---|---|
| 1 | Start control-plane log export for the affected accounts/tenants to a write-once location; place legal hold | Operations Lead (Cloud) | No | Export job running and hold confirmed by Legal Liaison |
| 2 | Snapshot affected EBS/persistent volumes; capture live memory and runtime state on any instance you will later stop | Operations Lead (Cloud) | No | Snapshot IDs recorded in the evidence register |
| 3 | Enumerate scope: role sessions across all Regions, created IAM users/keys, OAuth grants, service accounts, trust-policy changes | Operations Lead (Cloud) + Identity | No | Scope list handed to IC, time-boxed |
| 4 | Attach a quarantine SCP at the org level (AWS); on Azure, remove the principal's role assignments at management-group or subscription scope and assign a deny-effect Azure Policy; on GCP, remove the IAM binding at the org or folder | Operations Lead (Cloud) | No | attach-policy returns success; denied calls appear in CloudTrail / the Azure Activity log |
| 5 | Revoke role sessions and change permissions in the same action (see below — one is not enough) | Operations Lead (Cloud) | No | New API calls from the principal return AccessDenied (AWS) or 403 Forbidden (Azure) |
| 6 | Deactivate compromised access keys (Inactive, do not delete yet); on Azure, delete the compromised service-principal secret or certificate and disable the service principal | Operations Lead (Cloud) | Deleting does — Inactive does not. Azure has no inactive state, so record the credential's key ID before deleting | get-access-key-last-used shows no activity after the change |
| 7 | Revoke identity sessions and remove attacker-created persistence in one burst (see Chapter 14.3) | Operations Lead (Identity) | No | No new token issuance observed for the principal |
| 8 | Apply a block Conditional Access policy / IdP-level block for the affected identities | Operations Lead (Identity) | No | Sign-in logs show blocked attempts |
| 9 | Move the instance to an isolation security group with no 0.0.0.0/0 (0-65535) rule in either direction, remove all other SG associations; on Azure, swap the VM's NIC to an isolation NSG | Operations Lead (Cloud) | No — but see the tracked-connection caveat | Instance reachable only from the forensic path |
| 10 | Add NACL denies for confirmed C2 IPs | Operations Lead (Cloud) | No | Established C2 sessions drop |
| 11 | Stop or terminate the instance | Operations Lead (Cloud) | YES — memory is gone permanently | Only after steps 2 and 9 are complete and verified |
| 12 | Delete an OIDC provider or federation trust | Operations Lead (Cloud) | No, but causes an outage — every role trusting it fails to assume | Executive Sponsor has approved the outage |
Revoking IAM role sessions is not the same as removing permissions. AWS states it directly: "Temporary security credentials are valid until they expire… You can revoke these credentials, but you must also change permissions for the IAM user or role" (disabling permissions for temporary credentials). Session duration ranges from 900 seconds to 129,600 seconds (36 hours), default 43,200 seconds (12 hours) — so a session you fail to kill can outlive your entire first shift.
The console's "Revoke active sessions" attaches an inline policy named AWSRevokeOlderSessions to the role (requiring PutRolePolicy), denying all access to sessions assumed in the past and approximately 30 seconds into the future to absorb propagation delay. "Any user who assumes the role more than approximately 30 seconds after you choose Revoke active sessions is not affected" — which is why step 4's SCP and step 5's permission change both matter. The policy AWS attaches looks like this (revoke IAM role sessions):
{
"Version": "2012-10-17",
"Statement": {
"Effect": "Deny",
"Action": "*",
"Resource": "*",
"Condition": {
"DateLessThan": {"aws:TokenIssueTime": "2014-05-07T23:47:00Z"}
}
}
}Three exceptions that will bite you mid-incident:
Deny on the resource, keyed on aws:PrincipalArn or aws:SourceIdentity.For surgical denies that do not nuke a role every other workload depends on, condition on aws:SourceIdentity (immutable once set, and it survives role chaining), aws:PrincipalArn, or aws:userId — AROAXROLE1:* denies every session for a role, AROAXROLE2:<session-name> denies exactly one. The AWS-managed AWSDenyAll policy is the blunt instrument when you want the whole principal dead. And tell your responders to clear their own client caches (rm -r ~/.aws/cli/cache on Linux/macOS, del /s /q %UserProfile%\.aws\cli\cache on Windows) or they will spend twenty minutes debugging a credential that no longer exists.
Quarantine SCPs beat in-account denies during an active incident.
# Attach a quarantine policy to a root, OU, or 12-digit account ID.
aws organizations attach-policy \
--policy-id p-examplepolicyid111 \
--target-id ou-examplerootid111-exampleouid111(attach-policy, SCP concepts) The reason this is the better containment lever is structural: the SCP lives in the management account, outside the compromised account's control, so a principal holding admin in the member account cannot detach it. An inline deny on a role can be removed by the attacker and is subject to IAM eventual consistency. The AWS CIRT playbook documents exactly this pattern — a deny-all conditioned on the offending identitystore:userId or aws:TokenIssueTime, attached at the Root or a target OU (Compromised IAM Credentials playbook).
Three exclusions, and they are the difference between contained and only feeling contained:
If the compromised principal lives in the management account, the SCP is not your lever. Nothing you attach at the root will touch it. Contain on the identity side instead: attach an explicit deny to the principal, deactivate its access keys, revoke its role sessions, and — for an Identity Center user — revoke the permission-set session in Identity Center. That is the case where the blunt AWSDenyAll policy and the aws:PrincipalArn conditions above are doing the actual work, and the SCP is doing none.
Key rotation runs backwards during an incident. AWS's no-downtime rotation sequence is create → update applications → verify with get-access-key-last-used → set Inactive → confirm → delete (update access keys). For a compromised key, invert it: deactivate first, then create the replacement. Set it to Inactive rather than deleting it — an inactive key still tells you it existed, who created it and when it was last used; a deleted one tells you nothing.
Instance isolation, and the caveat that breaks naive playbooks. AWS's documented procedure is: create a dedicated Isolation security group with no rule permitting 0.0.0.0/0 (0-65535) in either direction, associate it with the instance, then remove all other security group associations (remediating a compromised EC2 instance).
# Replaces the instance's security groups with the isolation group.
# You must specify at least one security group ID.
aws ec2 modify-instance-attribute \
--instance-id i-1234567890abcdef0 \
--groups sg-0isolationNow the caveat, quoted: "The existing tracked connections won't be terminated as a result of changing security groups — only future traffic will be effectively blocked by the new security group." An established C2 channel survives your isolation. For that you need NACLs based on the network IoCs, which AWS's own ransomware response playbook covers in its "Enforce NACLs based on network IoCs" section (Ransom_Response_EC2_Linux). Google documents the identical trap on its side: "Adding firewall rules doesn't close existing connections."
Evidence handling across accounts. Snapshots are Region-scoped, so copy to move Regions. If the snapshot is encrypted, you must also share the customer-managed KMS key that encrypted it, or the forensic account receives an unreadable blob. The forensic role should have read-only access to collected artefacts (forensic investigation environment strategies, SEC10-BP03, capture backups and snapshots).
Federation containment is a demolition tool. There is no disable operation for an OIDC provider — only delete: aws iam delete-open-id-connect-provider --open-id-connect-provider-arn <arn>. It is idempotent, and AWS is explicit about the consequence: "Deleting an OIDC provider does not update roles that reference it. Any attempt to assume such roles will fail" (delete-open-id-connect-provider). That failure is the containment effect, and it is also an outage across every CI pipeline and workload that federated through it. The surgical alternative is remove-client-id-from-open-id-connect-provider, which drops one audience rather than the whole trust.
GCP has its own version of the "revocation is not enough" trap, and it is the most important sentence in a GCP containment playbook: "Disabling a service account key does not revoke short-lived credentials that were issued based on the key." The documented remedy is to disable or delete the service account itself, which immediately stops any workload using it (disable and enable service account keys).
# Disable a suspect key. NOTE: tokens already minted from this key remain valid.
gcloud iam service-accounts keys disable KEY_ID \
--iam-account=SA_NAME@PROJECT_ID.iam.gserviceaccount.com \
--project=PROJECT_IDAzure has no SCP, and pretending otherwise will cost you an hour. There is no policy object that sits above a subscription and denies arbitrary actions to a compromised principal the way an SCP does. Two things are commonly mistaken for one. Azure deny assignments look exactly right — they attach deny actions to a principal at a scope and beat any role assignment — but Microsoft is blunt: "You can't directly create your own deny assignments. Deny assignments are created and managed by Azure" (deny assignments). They arrive via deployment stacks and managed resources, not via your incident. Azure Policy with the deny effect is assignable by you at management-group scope, and it is the closest analogue — but read what it actually does: it "prevent[s] a resource request that doesn't match defined standards… The request is returned as a 403 (Forbidden)" (deny effect). That blocks resource creation and update. It does not block reads, and it does not block data-plane actions. It will stop an attacker deploying crypto-mining VMs. It will not stop them reading your storage accounts.
So on Azure the containment lever is identity-side, and it is removal rather than denial. Enumerate before you delete — az role assignment delete removes every assignment matching the query:
# ALWAYS run list first. delete removes every assignment matching these arguments.
az role assignment list \
--assignee 00000000-0000-0000-0000-000000000000 \
--scope /subscriptions/<subscription-id> \
--include-inherited
# Remove the compromised principal's assignments at the subscription scope.
az role assignment delete \
--assignee 00000000-0000-0000-0000-000000000000 \
--scope /subscriptions/<subscription-id>
# Delete a compromised service-principal secret. Record --key-id in the evidence
# register first: Azure has no "inactive" state, so the credential is simply gone.
az ad sp credential delete \
--id 00000000-0000-0000-0000-000000000000 \
--key-id <key-id>
# Isolate a VM by swapping its NIC to a pre-built isolation NSG.
# Build the isolation NSG in advance, in every VNet, like the AWS one in step 9.
az network nic update \
--resource-group <resource-group> --name <nic-name> \
--network-security-group <isolation-nsg>And the Azure trap that mirrors the GCP one: removing a role assignment or deleting a credential does not invalidate an access token the attacker already holds. Entra access tokens stay valid until they expire, and Entra "can't directly revoke a session token issued by an application." Removing the assignment stops the next token; it does not stop the one in flight. This is the same shape as the GCP short-lived-credential trap above and the AWS session trap above it — three providers, three different commands, one identical failure. Pair every removal with session revocation and a Conditional Access block on the identity side (Chapter 14.3), and treat the token lifetime as your real containment clock.
For the identity half of this — Entra session revocation, Conditional Access blocks, OAuth grant removal, Google Workspace signOut and token deletion — see Chapter 14.3, which owns the full account-takeover playbook, and Chapter 4 for the standing controls. The short version you need here: revocation is the control that matters and expiry is not, because in Continuous Access Evaluation sessions token lifetime increases to long-lived, up to 28 hours, and CAE propagation can take up to 15 minutes (continuous access evaluation).
Actionable takeaway: Rehearse this table as a drill in a non-production account, timed, with the snapshot and export steps actually executed. Every step you have never run will take three times as long during an incident, and step 11 — stopping the instance — is the one people run first and regret permanently. Preserve. Then contain. In that order, on every incident, without a debate about it.
Kubernetes is where all of the above compounds, because a cluster is simultaneously a compute platform, an identity provider and a secret store, and its default settings favor developer velocity over your investigation.
| Platform | Default | What you must do |
|---|---|---|
| EKS | Control-plane audit logging is off by default, per log type; the audit type defaults to Metadata level | Enable it explicitly; it publishes to a CloudWatch log group (EKS control plane logging) |
| GKE | Admin Activity audit logging on by default at Metadata level; Data Access logs off by default | Enable Data Access logs; both land in Cloud Logging with the retention from §2 |
| AKS | Audit categories ship to Log Analytics only when a diagnostic setting is configured | Configure the diagnostic setting — no setting means no audit evidence at all |
There is a tuning rule here that most teams miss and every privilege-escalation investigation depends on. Metadata level on all verbs is not enough. You need Request level on Secrets, ServiceAccounts and RBAC objects, because the request body is what shows the escalation — which role, which subject, which secret. Metadata tells you a RoleBinding was created; Request tells you it bound cluster-admin to the attacker's service account. That is the entire finding.
The EKS best-practices guide gives verbatim one-liners for the three questions you will ask first (EKS incident response and forensics):
# Which node is the suspect pod running on?
kubectl get pods <name> --namespace <namespace> -o=jsonpath='{.spec.nodeName}{"\n"}'
# Every pod using a given service account, with its node.
kubectl get pods -o json --namespace <namespace> \
| jq -r '.items[] | select(.spec.serviceAccount == "<service account name>") | "\(.metadata.name) \(.spec.nodeName)"'
# Every pod running a compromised image, cluster-wide.
IMAGE=<malicious image>
kubectl get pods -o json --all-namespaces \
| jq -r --arg image "$IMAGE" '.items[] | select(.spec.containers[] | .image == $image) | "\(.metadata.name) \(.metadata.namespace) \(.spec.nodeName)"'Do not delete the pod.
AWS states it plainly: "Gather forensic evidence before removing the node — an attacker might attempt to destroy evidence through termination." Pods are ephemeral by design. Deleting one destroys the container's writable layer and all in-memory state, and if it is managed by a Deployment, the controller helpfully schedules a replacement — which may re-run the attacker's payload from the same compromised image, restarting the incident with your only evidence already gone.
Capture first, in this order:
netstat for connections and open ports.docker top, docker logs, docker inspect, docker diff, docker checkpoint; for containerd and CRI-O runtimes, the crictl equivalents.Kubernetes gives you two non-destructive live-triage moves, and they should be your reflex (debug running pods):
# Attach an ephemeral debug container to the RUNNING pod. Does not restart it.
kubectl debug -it POD_NAME --image=busybox --target=CONTAINER_NAME
# Take a copy of the pod to examine, leaving the original running and observable.
kubectl debug POD_NAME --copy-to=POD_NAME-debug --image=DEBUG_IMAGEOnly after capture: kubectl delete pods POD_NAME --grace-period=10, or delete the Deployment so that no replacement is scheduled — which is GKE's documented sequence, and the correct one when the image itself is the problem.
Network-policy quarantine. A deny-all policy scoped to the compromised pod's labels:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
spec:
podSelector:
matchLabels:
app: web
policyTypes:
- Ingress
- EgressVerify enforcement; do not assume it. NetworkPolicy is enforced by the CNI, not by Kubernetes itself. On EKS it requires the VPC CNI network-policy feature, or Calico or Cilium. A cluster without a policy-enforcing CNI will accept this object, report success, and enforce absolutely nothing. Test this in a drill, on every cluster, before you depend on it in an incident. An object that applies cleanly and does nothing is worse than no control at all, because it produces confident status updates that are false.
Node isolation.
kubectl cordon <node-name> # marks unschedulable; does NOT evict anything
kubectl drain --ignore-daemonsets <node> # evicts, respecting PDBs and grace periods
kubectl uncordon <node-name> # reverse itdrain "respect[s] the desired graceful termination period, and respect[s] the PodDisruptionBudget you have defined" (safely drain a node) — meaning a PodDisruptionBudget can block your containment drain. Kubernetes recommends the AlwaysAllow unhealthy-pod eviction policy for exactly this reason. Find out which of your PDBs would block a drain before you need to drain.
GKE's documented quarantine pattern is the elegant one: pin the compromised pod in place while moving every healthy workload off the node (mitigate security incidents in GKE):
kubectl cordon NODE_NAME
kubectl label pods POD_NAME quarantine=true
kubectl drain NODE_NAME --pod-selector='!quarantine'Then cut egress at the VPC layer:
gcloud compute instances add-tags NODE_NAME --zone COMPUTE_ZONE --tags quarantine
gcloud compute firewall-rules create quarantine-egress-deny \
--network NETWORK_NAME --action deny --direction egress \
--rules tcp --destination-ranges 0.0.0.0/0 --priority 0 --target-tags quarantineRemember Google's caveat: adding firewall rules does not close existing connections. On EKS, additionally detach IAM roles from the compromised worker node and remove IAM policies from pod-assigned roles, which is what stops the cluster compromise from becoming a cloud control-plane compromise.
This is the part almost every pre-2025 playbook gets wrong. Modern Kubernetes service-account tokens are bound: their validity is tied to an API object — a Pod, a Secret, or a Node (Node binding GA in v1.33) — and private JWT claims carry that object's metadata.name and metadata.uid. "If a referenced object is deleted or doesn't exist (or its metadata.uid doesn't match), authentication with that token fails immediately." For objects pending deletion with finalizers, tokens fail 60 seconds after the deletionTimestamp (managing service accounts).
That gives you a revocation decision tree:
| Token type | What revokes it |
|---|---|
| Legacy long-lived token in a Secret | kubectl delete secret <secret> -n <ns> (the controller creates a replacement for ServiceAccount-owned secrets) |
| Pod-bound token | kubectl delete pod <pod> — after evidence capture |
| Node-bound token | kubectl delete node <node> |
| All tokens for a service account | kubectl delete serviceaccount <sa> -n <ns> |
Verify what a captured token is bound to before you decide, using a TokenReview — kubectl create -o yaml -f tokenreview.yaml with an authentication.k8s.io/v1 TokenReview carrying spec.token. The status returns authentication.kubernetes.io/pod-name, pod-uid, node-name and node-uid. And when you mint a replacement, bind it deliberately: kubectl create token my-sa --bound-object-kind="Pod" --bound-object-name="test-pod".
Strip the RBAC too. Deleting a ServiceAccount without removing its RoleBindings and ClusterRoleBindings leaves the grant sitting there, waiting for a recreated ServiceAccount of the same name to inherit it. That is not eradication; that is a scheduled re-compromise.
hostNetwork: true pod, or a hop limit of 2 that lets a container reach the metadata service. IRSA exchanges projected service-account tokens for IAM roles, so an over-broad IRSA trust policy, or an sts:AssumeRoleWithWebIdentity condition that does not pin sub to a specific namespace and service account, lets any pod in the cluster assume that role (privilege escalation in EKS via worker node instance roles, Wiz EKS best practices).Actionable takeaway: Audit automountServiceAccountToken across every namespace and set it to false wherever the workload does not call the API server, then enable Request-level audit logging on Secrets, ServiceAccounts and RBAC objects. Those two changes remove the most common escalation primitive and give you the evidence to see the next one. Chapter 14.10 carries the complete Kubernetes compromise playbook.
Serverless shrinks your patching obligation and expands your identity obligation, which is a trade most teams accept without noticing the second half. Three things change materially.
Your invocation record is opt-in. Lambda invocations are CloudTrail data events, and data events are off by default. Without them you have management-plane visibility into who deployed the function and nothing whatsoever about who called it. Enable them for functions handling regulated data or holding privileged roles.
The credential-theft finding is a different one. GuardDuty's UnauthorizedAccess:IAMUser/ResourceCredentialExfiltration.OutsideAWS and .InsideAWS cover Lambda functions and ECS tasks, not just EC2. If your alerting routes only the InstanceCredentialExfiltration variants, you are blind to exactly the compute model you adopted partly for security reasons.
Containment is permission-shaped, not host-shaped. There is no instance to isolate and no security group to swap. The levers are the ones in §6: deny the execution role's permissions, revoke its sessions, remove event-source mappings and triggers, and — if the function itself is the malicious artefact — remove the deployment. Preservation still comes first: capture the function's code, configuration, environment variables and layer versions before you change anything, because a redeploy overwrites the evidence of what was running.
One more, easy to miss: IAM Access Analyzer's external-access analyzers cover Lambda alongside S3, IAM roles, KMS keys, SQS, Secrets Manager, SNS, EBS volume snapshots, RDS snapshots, ECR, EFS and DynamoDB. A snapshot shared to an unknown account is an exfiltration channel that leaves almost no other trace.
Multi-cloud is not three times the work. It is three times the work plus the integration cost of reconciling three incompatible mental models, which is the part nobody budgets for.
The specific failure is that containment semantics differ per provider, in ways that are individually documented and collectively lethal:
| Provider / plane | The thing that is not enough | What you must also do |
|---|---|---|
| AWS — resource | Revoking role sessions | Change permissions as well; sessions run to 36 hours. And if the principal is in the management account, the quarantine SCP does nothing — deny on the identity instead |
| GCP — resource | Disabling a service-account key | Disable or delete the service account itself — short-lived credentials minted from the key survive |
| Azure — resource | Removing role assignments, or an Azure Policy deny | Policy deny blocks creates and updates only, not reads or data-plane calls. Delete the service-principal credential, disable the principal, and revoke sessions — there is no user-creatable deny assignment and no SCP equivalent |
| Microsoft Entra — identity | Resetting the password | Revoke sessions, and separately remove OAuth grants; Entra "can't directly revoke a session token issued by an application" |
A responder who has internalized the AWS model and applies it to GCP will disable the key, watch the API calls continue, and lose twenty minutes deciding whether their tooling is broken. That is a training problem with a documentation answer: write the per-provider revocation semantics into one card and put it in the war room.
Four rules that make multi-cloud tractable:
2024-07-25T20:54:59.649Z), millisecond granularity where available, from a validated time source — exactly what the allied logging guidance calls for. It is the difference between a timeline and a pile of files.Actionable takeaway: Write a one-page per-provider revocation card — for AWS, Azure/Entra and GCP, what kills a session, what kills a credential, and what each one does not reach — and laminate it into the incident war-room kit. The five minutes a responder spends reading it is the cheapest control in this chapter.
Cloud security is not really about the cloud. It is about whether you configured a machine that keeps receipts, whether you know which of your thousands of standing permissions actually get used, and whether the person who gets paged at 03:00 knows to take the snapshot before they kill the pod. None of that requires an enterprise budget. All of it requires deciding, in advance and in writing, what you will do — because the control plane will absolutely do what you told it to, exactly as fast as an attacker can ask.
Log everything that grants power, revoke before you reset, and never, ever delete the pod first.
[IG1] [GV.RR] [ID.AM][IG1] [DE.CM] [CIS 8] [A.8.15][IG2] [PR.DS] [CIS 8] [A.5.28][IG2] [DE.CM] [CIS 8][IG2] [DE.CM] [CIS 8]_Default. [IG2] [DE.CM] [CIS 8][IG2] [DE.CM]SearchQueryInitiatedExchange and SearchQueryInitiatedSharePoint are activated for privileged and high-risk mailboxes. [IG3] [DE.CM][IG2] [DE.AE]HttpTokensEnforced) in all production accounts, and the MetadataNoToken metric reads zero for the fleet. [IG2] [PR.PS] [CIS 4] [A.8.9]ec2RoleDelivery is "1.0", and for ASIA instance-role credentials used from a source IP outside AWS. [IG2] [DE.CM][IG2] [PR.AA] [CIS 5] [CIS 6][IG2] [ID.AM] [PR.AA][IG1] [PR.PS] [CIS 4] [A.8.9][IG2] [RS.MI] [A.5.26]0.0.0.0/0 (0-65535) rule in either direction, and the runbook documents that changing security groups does not terminate established connections. [IG2] [RS.MI][IG3] [RS.AN] [A.5.28][IG1] [RS.MA] [A.5.24][IG2] [DE.CM] [CIS 8]automountServiceAccountToken is set to false for every workload that does not call the API server, verified by policy rather than by convention. [IG2] [PR.AA] [CIS 4][IG2] [PR.IR] [CIS 13][IG2] [RS.AN] [A.5.28][IG3] [RS.MI]sub claim to a specific namespace and service account, with no cluster-wide assumable roles. [IG3] [PR.AA] [CIS 6][IG2] [RS.AN][IG2] [DE.CM] [CIS 8] [A.8.15]