Patching and updating without taking the service down

Why this matters

Patching is the operational task most likely to be deferred and most likely to be the reason an incident happened. It sits directly between two domains: it is lifecycle management here, and it is the remediation half of vulnerability management in objective 4.1.

The cloud changes the problem in a genuinely useful way. On owned hardware, patching means changing a machine and hoping it comes back. With elastic infrastructure you can replace rather than change, and the replacement can be tested before it takes traffic. That is a better answer to the same problem, and knowing why is the point of this lesson.

The lesson

Patch classes and the urgency each one actually justifies

Not all patches deserve the same response, and treating them alike produces either constant disruption or a backlog.

• Critical security patches for a vulnerability that is remotely exploitable, on something exposed, with known exploitation. Days, sometimes hours. This is the only class that legitimately bypasses the normal cycle, and doing so needs an emergency change process rather than no process.
• Routine security patches. The regular cycle — a monthly or fortnightly cadence, through the normal rings.
• Bug fixes. Applied when they fix something you have, or on the routine cycle. A patch for a bug you do not experience carries risk without benefit.
• Feature updates and version upgrades. Deliberate projects with testing and a rollback plan, not patches.
• Firmware and hypervisor updates. Usually the provider's responsibility under shared responsibility — but note they may require an instance stop/start or a scheduled retirement event, which you must act on.

Urgency should come from exposure and exploitability, not from the severity score alone. An internet-facing service with a known exploited vulnerability outranks an internal system with a theoretically higher CVSS. Objective 4.1 develops this properly; it is the same judgement.

Keeping a patch baseline — the stated level everything should be at — turns "are we patched?" into a measurable question with a number attached.

Test, staging and production rings, and the soak time between them

A ring (or wave) deployment moves a change through progressively more important populations.

A typical progression:

1. Test / canary ring. Non-production, or a small slice of production such as the IT team's own systems. Catches the patch that breaks on contact.
2. Early adopter ring. A larger, tolerant population.
3. Broad ring. The bulk of production.
4. Critical ring. The systems with the least tolerance, patched last, when the patch has been proven elsewhere.

Between rings sits soak time — a deliberate wait, long enough for problems to surface. The right length is a business-cycle question rather than a technical one: a fault might only appear at the nightly batch, the weekly report or the month-end run, so a 24-hour soak catches daily problems and nothing rarer. For critical systems, soaking across a full week is a reasonable standard.

Two points the exam tests:

• The rings must be representative. A test environment on different versions, different data or different configuration will not reveal the problem. This is what the "one template, parameters only" discipline in objective 2.2 was protecting.
• Automate promotion between rings, with a gate. A human decision to promote, executed automatically, gives control without the inconsistency of manual patching.

Pair this with maintenance windows: agreed periods when disruptive change is allowed, published, and honoured. And note that a window is a permission to disrupt, not a technique for avoiding disruption — that is what the strategies below are for.

Rolling, blue-green and canary deployments as three answers to the same risk

Three strategies for changing a running service, with different trade-offs.

Rolling. Replace instances in batches. Cheap — no extra capacity beyond the surge — and gradual. The risks are the ones from objective 1.5: two versions run simultaneously, so the change must be compatible in both directions; rollback means rolling back through the same process, which is slow; and maxUnavailable set too high drops capacity below demand.

Blue-green. Stand up a complete second environment on the new version, test it while it takes no traffic, then switch traffic at once. Rollback is switching back and is nearly instantaneous, which is the main attraction. It costs double the infrastructure during the change, and the switch is all-or-nothing so every user meets the new version simultaneously. The shared database is the usual complication: both environments normally use the same data, so schema compatibility is still required.

Canary. Route a small fraction of real traffic to the new version, watch real metrics, and widen gradually. The best early warning because it uses real traffic and real data, with the smallest blast radius. It needs traffic splitting and good observability (objective 3.3) to decide whether the canary is healthy — and an automatic rollback on error-rate or latency regression is what makes it genuinely strong.

Choosing between them, which is how the question is usually framed:

Immutable infrastructure: replacing an instance instead of patching it

The cloud-native answer, and the one that fits everything else in this course.

Rather than patching a running instance, build a new image with the patch applied, and replace the instances. Nothing is modified in place; instances are cattle rather than pets.

The benefits are substantial:

• No configuration drift. Every instance came from the same image, so "patched" is a property of the image, not a hope about each machine.
• The patch is tested once, on the image, rather than applied independently to every machine with independent chances of failing.
• Rollback is deploying the previous image, which is fast and reliable.
• Failed patching cannot leave a half-updated machine, because nothing was updated — the old instance is intact until the new one is healthy.
• It composes with everything else: the image pipeline is CI/CD (objective 5.3), the replacement is a rolling update, and the whole thing is in code.

What it requires:

• An image pipeline, building and testing images on a schedule and on demand.
• Genuine statelessness, since instances are destroyed routinely.
• Discipline about not logging in to fix things, which is the same cultural rule as the console-versus-code rule in objective 2.2.

Where it does not apply — a stateful database primary, a licensed appliance — in-place patching remains necessary, and those systems need the snapshot and rollback plan below.

Maintenance windows, change records, and the rollback that must exist first

The process wrapper, which the exam treats as part of the skill.

Change records. Every change documented before it happens: what, why, who, when, expected impact, how it will be verified, and how it will be reversed. In a regulated environment this is mandatory; everywhere else it is what makes incident investigation possible, because the first question after any incident is "what changed?" and an estate with good change records answers it in minutes.

Distinguish the three standard types: standard changes (pre-approved, low-risk, repeatable), normal changes (assessed and approved on the usual cycle) and emergency changes (expedited, with retrospective review). A critical security patch is an emergency change — expedited, not undocumented.

The rollback plan must exist before the change starts, and must be specific enough to execute under pressure:

• What to do, precisely. "Redeploy the previous image version" is a plan; "revert the patch" is not.
• How long it takes, because a rollback longer than the remaining window is not available to you.
• The trigger, agreed in advance, so the decision is not made by whoever is most optimistic at 2am.
• What was captured first — a snapshot for in-place changes, the previous image version for immutable ones, and a database backup verified restorable where schema changes are involved.

The irreversible case deserves naming: a schema migration is usually not reversible by rolling back code. Once data has been transformed, redeploying the old version leaves it reading data it does not understand. The answer is expand-then-contract — deploy a schema both versions can use, migrate, then remove the old shape in a later change — and it is the reason a rollback plan must be thought about rather than assumed.

What to take into the exam

• Urgency comes from exposure and exploitability, not the severity score alone.
• Rings plus soak time, where soak is long enough to cover the business cycle that would reveal the fault. Rings must be representative.
• Rolling = cheap, gradual, two versions at once, slow rollback. Blue-green = instant rollback, double cost, all-at-once switch. Canary = real-traffic evidence, smallest blast radius, needs observability.
• Immutable infrastructure replaces rather than patches: no drift, tested once, rollback by previous image.
• A critical patch is an emergency change — expedited, not undocumented.
• The rollback plan exists before the change, with a named trigger and a known duration. A schema migration is not undone by redeploying old code.