Rehearse a failover and a failback, and time both

capstone · 140 min · Objective 3.2

Task

Pull the Operations domain together by proving a recovery rather than documenting one. Build a primary and a recovery environment, fail over, measure the actual RPO and RTO achieved, then fail back -- which is the half nobody rehearses.

Steps

  1. Build the primary: an application, a database and a client generating steady writes. Build the recovery environment as a pilot light -- the database replicating, everything else defined in code but not running.
  2. Write lab/cap3/runbook.md BEFORE any failure: the detection criterion, who decides, the failover steps in order, the verification, and the failback procedure. Store it outside both environments.
  3. Fail the primary abruptly -- stop it without warning. Execute the runbook, timing each step. Measure the ACHIEVED RTO (time to service restored) and the ACHIEVED RPO (how many of the client's writes were lost). Record in lab/cap3/failover.md.
  4. Record every place the runbook was wrong, missing a step, or assumed something unavailable during the outage. This list is the real output of a DR test.
  5. Bring the primary back and FAIL BACK: reverse the replication, choose a cutover moment, and restore normal operation -- then record what it cost, whether reverse replication existed before you needed it, and whether promoting the recovery site would have been better. Compare achieved figures against the objectives you set.

Verify

test -s lab/cap3/runbook.md && echo "runbook present"
grep -Eci 'achieved rto|rto[^a-z]*[0-9]' lab/cap3/failover.md
grep -Eci 'achieved rpo|rpo[^a-z]*[0-9]' lab/cap3/failover.md
grep -Eci 'failback|fail back' lab/cap3/failover.md
grep -Eci 'runbook (was )?(wrong|missing|incorrect)|gap' lab/cap3/failover.md

Measured RPO and RTO, a completed failback, and at least one recorded runbook gap. A DR test that found nothing wrong with the runbook was almost certainly run by the person who wrote it, following it from memory.

Notes

Writing the runbook before the failure, and storing it outside both environments, is the whole design of this exercise. A runbook readable only from the environment that is down is the sharpest version of the problem this domain keeps returning to.

This is an independent study companion for CompTIA Cloud+ CV0-004 and is not produced by or endorsed by CompTIA.