#14.10 Kubernetes and Container Compromise

Playbook ID: PB-K8S | Default severity: SEV-2 (escalate to SEV-1 if a container escape to the node is confirmed, the cluster Secret store was read, node or cloud credentials were used outside the cluster, or the compromised image is running in more than one cluster) | Owner: Operations Lead (Platform)

#When to run this

• Runtime detection on a container doing something the image never did: a shell spawned in a distroless container, curl or wget in a workload with no egress requirement, a crypto-miner process, or an outbound connection to a mining pool. GKE's Container Threat Detection reports these from the guest kernel (SCC threat detection).
• GuardDuty UnauthorizedAccess:IAMUser/ResourceCredentialExfiltration.OutsideAWS or .InsideAWS naming an ECS task or Lambda, or UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.OutsideAWS naming a worker node's instance role (finding types).
• Kubernetes audit events showing a service account reading Secrets it has never read, creating a privileged or hostNetwork pod, creating a ClusterRoleBinding, or hitting pods/exec.
• An admission controller or image scanner blocking, or belatedly flagging, a running image — including an image pulled from a registry whose publishing credentials were compromised.
• A worker node with unexplained CPU saturation, an unexplained /host bind mount, or an exposed Docker socket found in a pod spec.
• A vendor or package advisory that names a container image, base image, GitHub Action or CI runner you consume.

Not for: the compromise of the CI/CD system, registry or upstream package itself — that is Playbook 14.5, which owns the vendor-side and build-chain work; run it in parallel and come back here for the cluster-side eviction. Human user or SaaS account takeover is 14.3; identity provider compromise is 14.4. Encryption of cluster storage with a ransom demand escalates to 14.1. Compromise of a model-serving or agent workload's behavior rather than its container is 14.11. The preventative controls — CSPM/CIEM, admission policy, IMDS hardening, cluster hardening baselines — belong to Chapter 6; this playbook assumes they were insufficient.

#What you are dealing with

The defining mistake in this scenario takes one keystroke. A responder sees a bad process in a pod, runs kubectl delete pod, and feels like they contained something. What they actually did was delete the container's writable layer, throw away every byte of process memory, and — because it was a Deployment — hand the scheduler an instruction to start a fresh copy of the same compromised image, possibly on a different node, almost certainly with the same service-account token mounted. The attacker gets a new pod for free and you get nothing. AWS states the rule flatly in its own EKS guidance: gather forensic evidence before removing the node, because an attacker may attempt to destroy evidence through termination (EKS Best Practices — Incident Response and Forensics). Pods are cattle right up until one of them is the crime scene.

What the adversary is after is almost never the container. It is the identity mounted inside it. The May 2026 Sysdig case is the cleanest illustration on record: an exposed Docker socket let the actor start a privileged container with the host filesystem bind-mounted at /host, read host credentials including /etc/shadow and SSH keys, and then replay the pod's own projected service-account token against the API server to dump the entire cluster Secret store — database credentials, AWS keys, third-party API keys. That chain made no IMDS call at all; the mounted token was sufficient (Sysdig). The lesson for your triage order is unambiguous: what could this token reach comes before what did they run.

It moves at control-plane speed, which is to say instantly. Cloud-conscious intrusions rose 37% overall and 266% among state-nexus actors, and 35% of cloud incidents involved valid account abuse (CrowdStrike 2026 GTR). The escape half is not theoretical either — three critical runC vulnerabilities disclosed in November 2025 affect Docker, Kubernetes, containerd and CRI-O, and CVE-2025-23266 in the NVIDIA Container Toolkit carries CVSS 9.0 (Wiz). And when a poisoned build reaches you, it arrives already knowing how to move: the March 2026 LiteLLM compromise shipped a payload that harvested credentials, moved laterally across Kubernetes clusters, and dropped a persistent systemd backdoor (Resecurity).

One more thing, because teams get it wrong every time. If what you found is a cryptominer, you have not found a nuisance. Cryptomining is the most common payload in compromised container environments, but the durable pattern established by Sysdig's SCARLETEEL research and repeated since is that the mining foothold and the credential-theft path are the same access (Dark Reading). The miner is the part they did not bother to hide. Treat it as proof of control-plane access, not as commodity noise.

#Roles for this incident

Marking used below: `TIP-OFF = the adversary can observe this action. EVIDENCE` = this destroys or degrades evidence and must not run before capture.

#Phase 1 — Detection and Triage

# Which node is the pod on, and which pods share the compromised identity or image.
# Verbatim from the EKS Best Practices Guide (Incident Response and Forensics).
kubectl get pods <name> --namespace <namespace> -o=jsonpath='{.spec.nodeName}{"\n"}'

kubectl get pods -o json --namespace <namespace> \
  | jq -r '.items[] | select(.spec.serviceAccount == "<service account name>") | "\(.metadata.name) \(.spec.nodeName)"'

IMAGE=<malicious image>
kubectl get pods -o json --all-namespaces \
  | jq -r --arg image "$IMAGE" '.items[] | select(.spec.containers[] | .image == $image) | "\(.metadata.name) \(.metadata.namespace) \(.spec.nodeName)"'

#Phase 2 — Containment

Order matters more here than in any other playbook in this chapter. Capture, then cut the network, then cut the identity, then move the node. Reverse any two of those and you lose either the evidence or the adversary.

# Deny-all quarantine for a labelled pod. Enforced by the CNI, not by the API server:
# on a cluster without a policy-enforcing CNI this object is accepted and does nothing.
# Test egress from the pod after applying it. Do not assume.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
spec:
  podSelector:
    matchLabels:
      app: web
  policyTypes:
  - Ingress
  - Egress

# Non-destructive live triage. Neither of these restarts the target pod.
# Ephemeral debug container attached to the running pod:
kubectl debug -it POD_NAME --image=busybox --target=CONTAINER_NAME
# Copy of the pod to work on, original left running:
kubectl debug POD_NAME --copy-to=POD_NAME-debug --image=DEBUG_IMAGE

# Node quarantine, GKE's documented pattern: pin the compromised pod, move the rest.
kubectl cordon NODE_NAME
kubectl label pods POD_NAME quarantine=true
kubectl drain NODE_NAME --pod-selector='!quarantine'

# VPC-level egress cut. Reminder: this does not close connections already established.
gcloud compute instances add-tags NODE_NAME --zone COMPUTE_ZONE --tags quarantine
gcloud compute firewall-rules create quarantine-egress-deny \
  --network NETWORK_NAME --action deny --direction egress \
  --rules tcp --destination-ranges 0.0.0.0/0 --priority 0 --target-tags quarantine

# Require IMDSv2 and block container-to-IMDS via the hop limit.
# AWS documents that a hop limit of 1 "can cause issues" in container environments.
aws ec2 modify-instance-metadata-options \
    --instance-id i-1234567890abcdef0 \
    --http-tokens required \
    --http-put-response-hop-limit 1 \
    --http-endpoint enabled

# Pod deletion is a Phase 2 step 8 action, not a Phase 1 reflex.
# Run it only after steps 1-4 have captured memory, runtime state and volumes.
kubectl delete pods POD_NAME --grace-period=10
# If a controller will simply reschedule the payload, delete the workload object instead.

#Phase 3 — Eradication

#Phase 4 — Recovery

#Phase 5 — Post-Incident

#Decision points

#Communications and notification triggers

Nothing in this scenario starts a regulatory clock by itself. Containers being compromised is an operational event; what starts a clock is confirmed unauthorized access to data, which in this scenario almost always arrives through the Secret store rather than through the application. The trigger to watch for is Phase 3 step 1: the moment you can say a specific secret was read, and that secret unlocked a system holding personal or regulated data, brief Legal Liaison — do not wait until you can quantify records. If that path is confirmed, hand off to Playbook 14.7.

Two other notification paths matter here and are easy to miss. First, machine credentials cross organizational boundaries: if a rotated secret belonged to a partner, a customer, or a vendor's API, someone outside your organization needs to rotate too, and that is a contractual notice, not a courtesy call. Second, if the entry vector was a poisoned image, action or package, you may be one of many consumers — coordinate the disclosure through Playbook 14.5 rather than publishing independently. Chapter 15 holds the notification decision tree and every regulatory deadline. Do not reconstruct them here and never commit to a deadline from memory.

#Automation notes

Automate freely. Everything in Phase 1 that gathers and preserves: on a runtime detection, automatically resolve the pod's node, record the image digest, run the service-account and image blast-radius queries, export the audit-log window, snapshot the node volumes, and open the legal hold. All of it is reversible, all of it produces evidence, and all of it is verifiable after the fact. Automating the snapshot is one of the highest-value SOAR plays available, because it removes the pressure that makes responders reach for kubectl delete in the first place.

Automate behind a human gate. The deny-all NetworkPolicy and kubectl cordon are strong one-click actions — reversible, non-evicting, and a false positive costs latency rather than an outage. Gate them on a named approver and rate-limit them, and have the automation prove CNI enforcement with an egress test rather than reporting success on the API response.

Never automate. Node drain, service-account or RBAC deletion, IAM role detachment, node termination, and cluster-wide image bans. Every one of them is either irreversible or scales its blast radius with your false-positive rate, and auto-draining a node is specifically the case where the automation either gets blocked by a PDB or evicts the evidence. The documented failure modes of agentic triage are overconfident closure backed by weak proof and hallucinated detail in the investigation narrative, so make the rule structural: automation may gather, enrich, correlate and recommend without approval; it may act only where the action is reversible, scoped and rate-limited; irreversible or cluster-wide actions require a named human approver, and every automated action carries the artefact that justified it. Chapter 17 has the gate design in full.

#Pitfalls

Takeaway: treat every cluster compromise as a credential compromise until you have proved otherwise. Scope rotation by what the service account's RBAC could reach, not by what namespace the pod was in, and prove the quarantine with an egress test from inside the pod rather than with a green object in kubectl get netpol. The miner is the noise. The Secret store is the incident.