Runtime container security and isolation

Why this matters

A perfectly built image still runs as a process on a shared kernel. Objective 1.4 established that a container is isolated by kernel features rather than by a hypervisor, and this lesson is the practical consequence: the defaults are convenient rather than safe, and the work is in narrowing them.

The threat model is specific. Assume an attacker gets code execution inside one container — through an application vulnerability, a compromised dependency, or a bad input. The questions that decide how bad that is are: what can that process do inside the container, what can it reach across the network, and can it get out onto the host. Each of the controls below closes one of those.

The lesson

Running as non-root, dropping capabilities, and read-only root filesystems

Three hardening measures that cost almost nothing and remove most of what an attacker would do next.

Run as a non-root user. By default many images run as root inside the container. That is not root on the host — user namespaces and other protections sit between — but it is a much better starting position for an attacker, and if any other isolation is weak or misconfigured it becomes far more dangerous. Specify a non-root user in the image and enforce it at the platform, so a workload that tries to run as root is refused. Also prevent privilege escalation within the container, which stops a process gaining more privileges than its parent.

Drop capabilities. Linux splits root's powers into discrete capabilities — binding low ports, changing file ownership, loading kernel modules, manipulating network configuration. Containers get a default set, most of which typical applications never use. Drop them all and add back only what is needed; the common legitimate case is binding to a privileged port, and the better answer there is usually to listen on a high port and map it.

Read-only root filesystem. Mount the container's filesystem read-only and provide writable volumes only where genuinely needed — a temporary directory, a data mount. This defeats a large class of post-exploitation behaviour: the attacker cannot write a tool to disk, cannot modify the application, cannot establish persistence in the filesystem. It also surfaces applications writing where they should not, which is usually worth knowing.

The one to avoid entirely: privileged mode. A privileged container has essentially full access to the host's devices and kernel interfaces, and compromising one is effectively compromising the node. The legitimate uses are narrow — certain monitoring and storage agents — and each should be justified individually. The same goes for mounting the host filesystem or the container runtime's socket into a container: a workload that can talk to the runtime can start a privileged container and own the node, which makes it a complete escape path.

Resource limits as a security control, not only a performance one

Requests and limits are usually presented as scheduling and cost controls (objective 1.5). They are also a security control, because without them one container can consume a node's CPU or memory and take down everything on it — a denial of service that needs no exploit, only a bug or a loop.

Set both:

• Requests reserve capacity and drive scheduling.
• Limits cap consumption. A container exceeding its memory limit is terminated; one exceeding its CPU limit is throttled.

Other limits worth setting for the same reason: process and thread counts, so a fork bomb cannot exhaust the node's process table; ephemeral storage, so runaway logging cannot fill the node's disk and evict every workload on it; and rate limits at the application or ingress layer.

The related control is quotas per namespace or tenant, capping what one team or application can consume of the cluster as a whole. In a shared cluster this is what stops one workload's incident becoming everyone's.

The failure this prevents is not exotic. A memory leak in one service, with no limit, gradually consumes a node until unrelated workloads are evicted. The cause looks like a platform problem for as long as nobody checks the limits.

Network policy between workloads, because a flat cluster network is the default

This is the most consequential default in the lesson.

In most orchestrators, every pod can reach every other pod by default. There is no segmentation inside the cluster. An attacker who compromises a public web front end can, without any further exploitation, attempt to connect to the database, the internal APIs, the metrics endpoints and every other workload in the cluster.

Network policies fix this by defining which workloads may talk to which, selected by label, on which ports, in which direction. The model to adopt:

1. Default deny for both ingress and egress, per namespace.
2. Allow explicitly what each workload legitimately needs — usually a small list: its own dependencies, DNS, and whatever calls it.
3. Restrict egress too, which is the half people skip. Egress control is what stops a compromised container reaching out to download a second stage or exfiltrate data, and it also blocks access to the instance metadata service — the credential-theft path from objective 2.4, which is reachable from inside a container by default and is one of the most important single things to block.

Points to be aware of:

• Policies require a network plugin that enforces them. On some platforms policies can be created and silently do nothing. Verify enforcement by testing a connection that should be blocked — the same "test what must not work" discipline as objective 2.4.
• DNS must be allowed, or everything breaks in a way that looks unrelated.
• A service mesh offers the stronger version: mutual TLS between workloads, identity-based authorisation rather than address-based, and encryption of internal traffic. More capability and more operational weight; CV0-004 does not require operating one, but recognising what it provides is fair.

Admission control that refuses a workload which breaks policy

Everything above describes how a workload should be configured. Admission control is what makes it so, by evaluating every workload as it is submitted and rejecting what does not comply.

Typical rules:

• No privileged containers
• No running as root, and no privilege escalation
• No host filesystem, host network or host process namespace
• Read-only root filesystem required
• Resource limits mandatory
• Images only from approved registries
• Image signatures verified (objective 4.3's signing control)
• Required labels present, so ownership and network policy selection work

Two distinctions worth carrying:

• Validating admission accepts or rejects. Mutating admission modifies the request — adding a default security context, injecting a sidecar, applying labels. Mutation is convenient and should be used carefully, because what runs then differs from what was submitted.
• Enforce, warn, or audit. Introduce policies in audit mode first to discover what would break, then move to enforcement. Turning on strict enforcement across an existing cluster blocks deployments immediately and gets the whole mechanism disabled.

This is the same idea as the guardrails in objective 4.2 and the policy scans in objective 2.2: the strongest control is the one that makes the unsafe configuration impossible rather than detectable.

Runtime detection, and the process that should never start inside a container

The last layer assumes something got through, and it exploits a property containers have that traditional servers do not: they are predictable.

A container runs one application, from an immutable image, doing the same things every time. So a behavioural baseline is genuinely achievable, and deviation from it is a strong signal rather than noise.

What runtime detection watches for:

• A shell starting inside a container. In production this is almost never legitimate, and it is the classic indicator of interactive access.
• Unexpected processes — a package manager, a compiler, a network scanner, a cryptocurrency miner.
• Writes to the root filesystem, particularly to binary directories, which a read-only filesystem would have prevented.
• Unexpected outbound connections, especially to unfamiliar destinations.
• Attempts to read the metadata service or cloud credentials.
• Attempts to mount filesystems, load kernel modules, or access the container runtime socket — all escape-shaped behaviour.
• Privilege escalation attempts.

The response should be automatic where it can be, and containers make that unusually safe: kill the container. The orchestrator replaces it, the attacker's foothold is gone, and the disruption is minimal because the workload was designed to be replaceable. That is a materially better position than the equivalent on a long-lived server, and it is worth recognising as a security benefit of the whole immutable model.

Two supporting practices: preserve evidence before killing where the incident warrants investigation (objective 6.3), and make sure container logs and runtime events reach central logging immediately (objective 3.3) — because the container that gets killed takes everything local with it.

What to take into the exam

• Non-root, dropped capabilities, no privilege escalation, read-only root filesystem. Avoid privileged containers and never mount the container runtime socket — both are host compromise.
• Resource limits are a security control: without them one workload's leak or loop evicts everything on the node.
• Cluster networking is flat by default. Adopt default-deny ingress and egress; egress control blocks second-stage downloads, exfiltration and the metadata service.
• Verify that network policy is actually enforced — some plugins accept policies and ignore them.
• Admission control enforces all of the above. Roll it out in audit mode first. Mutating admission changes what runs.
• Containers are predictable, so runtime detection works well. A shell starting in a production container is the classic indicator, and killing the container is a safe automatic response.