Disaster recovery, redundancy and continuity

Why this matters

This is the domain 3 capstone, and it is the objective where precise vocabulary earns marks directly. RPO and RTO sound similar, are constantly confused, and mean entirely different things — one is about data, the other about time. MTTR and MTBF are the same trap.

The concepts also decide how money gets spent. The difference between a warm site and a hot site is a very large number, and the right answer depends on metrics the business sets, not on what engineers would prefer.

The lesson

High-availability approaches

High availability means designing so that a single failure does not become an outage. The two arrangements:

Active-passive. One system handles all the work while a standby waits, monitoring it and ready to take over. On failure, the standby becomes active — a failover.

• Simpler, because only one system is serving and there is no question of coordination.
• The standby is idle, so you pay for capacity that produces nothing.
• There is a failover interval, short but non-zero, during which service is interrupted.

The FHRP protocols from the routing lesson — HSRP and VRRP — are active-passive.

Active-active. Both systems handle traffic simultaneously, with load shared between them.

• No capacity is wasted, and aggregate throughput is higher.
• Failover is faster, because the survivor is already working.
• More complex: state has to be shared or the design has to tolerate its loss, and both sides must be able to carry the full load alone — otherwise losing one leaves you overloaded rather than degraded, which is a worse outcome than a clean failover.

GLBP is the active-active FHRP, and most load-balanced server pools are active-active.

The metrics, and the distinction that matters

RPO (Recovery Point Objective) — how much data you can afford to lose, expressed as time. An RPO of one hour means losing up to one hour of work is acceptable, which in turn means backing up or replicating at least hourly.

RTO (Recovery Time Objective) — how long you can afford to be down before service is restored. An RTO of four hours means service must be back within four hours of the failure.

The reliable way to hold them apart: RPO looks backwards from the failure at the data; RTO looks forwards from the failure at the clock. Both are set by the business based on what the outage costs, not by IT based on what is convenient — and both drive spending, because tightening either is expensive.

MTTR (Mean Time To Repair) — the average time taken to restore service after a failure. It measures your response: detection, diagnosis, parts, and the fix. You improve it with monitoring, spares, documentation and practice.

MTBF (Mean Time Between Failures) — the average operating time between failures of a repairable system. It measures the equipment's reliability and is usually a vendor figure.

The pairing to remember: MTBF is how often it breaks, MTTR is how long it takes to fix. Availability rises by increasing the first or decreasing the second, and the second is usually the one you can actually influence.

DR sites

If a site is lost entirely, recovery happens somewhere else. Three options, differing in how much is ready in advance — and the choice is a direct trade between cost and RTO.

Cold site. A facility with space, power and connectivity, and no equipment installed. Cheapest by a wide margin. Recovery means procuring or shipping hardware, building it, restoring data and testing — days to weeks. Appropriate where a long outage is survivable.

Warm site. Hardware is installed and configured, and data is periodically replicated — so it is current to the last sync rather than to the moment. It must be brought up to date and brought online: hours to a day. The common middle choice.

Hot site. A full duplicate, running, with data replicated continuously. It can take over in minutes or less, and with active-active operation the takeover may be seamless. The most expensive by far, because you are running a second data centre.

Also worth naming: a cloud DR site, where the recovery environment is defined as code and instantiated on demand, which can give warm-site recovery at close to cold-site cost — and is increasingly the default answer.

Map them to the metrics and the exam question answers itself. A stated RTO of fifteen minutes cannot be met by a warm site. An RTO of a week does not justify a hot one.

Testing

A DR plan that has never been tested is a document, not a capability. Untested plans fail on the mundane: backups that were not running, documentation naming people who left, a dependency nobody mapped, credentials only one person had.

Tabletop exercises are discussion-based. The team walks through a scenario verbally — "the primary data centre is unreachable, what happens now?" — checking the plan's logic, roles and decision points.

• Cheap, quick, no disruption, and can involve senior people who would never be available for a live test.
• Finds gaps in the plan, and cannot prove the technology works. Everything is assumed to behave as documented.

Validation tests actually exercise the recovery: restore a backup and verify the data, fail over to the secondary and confirm service, run the failback.

• Proves the capability rather than the intention, and produces real timings to compare against the RTO.
• Costs time, risks disruption, and needs a maintenance window.

They are complementary, and a mature programme runs tabletops often and validation tests periodically. The point the exam wants is that only a validation test tells you whether recovery works — and that the results should feed back into the plan, because a test that finds nothing usually means the test was too gentle.