Run a recovery exercise against a stated RTO and RPO

capstone · 140 min · Objective 3.4

Task

Pull Domain 3 together: state recovery objectives, build to them, then run a real recovery and find out whether you met them. The gap between the objective and the measurement is the output of this exercise, and it is the finding a business impact analysis exists to produce.

Steps

  1. Define the service: a small application on one VM with data it writes continuously — a script appending timestamped records to a database or file is enough.
  2. State the objectives BEFORE building anything: write /tmp/objectives.md with an RTO and an RPO you have chosen, and the reasoning for each in business terms rather than technical ones.
  3. Build to them. The RPO decides backup or replication frequency; the RTO decides whether the second VM is warm, cold, or already running.
  4. Document the recovery procedure so that somebody else could follow it at 3am: exact commands, in order, including how to confirm success.
  5. Now destroy the primary — stop it abruptly and delete its data directory — and start the clock.
  6. Recover by following your own written procedure exactly, without improvising. Where the procedure is wrong or incomplete, note it and keep going.
  7. Stop the clock when the service is genuinely usable again. Measure two things: elapsed time, against your RTO; and how many records were lost, against your RPO.
  8. Write /tmp/exercise.md: both objectives, both measurements, whether each was met, every gap in the written procedure, and one change you would make to close the largest gap.

Verify

python3 - <<'PY'
import re
t=open('/tmp/exercise.md').read()
def num(pat):
    m=re.search(pat,t,re.I)
    return float(m.group(1)) if m else None
rto_t=num(r'rto[^0-9]{0,40}([0-9.]+)'); rto_a=num(r'actual[^0-9]{0,40}([0-9.]+)')
rpo_t=num(r'rpo[^0-9]{0,40}([0-9.]+)'); rpo_a=num(r'lost[^0-9]{0,40}([0-9.]+)')
for name,target,actual in (('RTO',rto_t,rto_a),('RPO',rpo_t,rpo_a)):
    assert target is not None, name+' target not found in the write-up'
    assert actual is not None, name+' measurement not found in the write-up'
    print('%s target %s | measured %s | %s' %
          (name,target,actual,'MET' if actual<=target else 'NOT MET'))
PY
grep -ciE "gap|missing step|would change" /tmp/exercise.md
diff <(sort /tmp/records-before.txt) <(sort /tmp/records-after.txt) | grep -c '^<'

The Python block requires all four numbers to be present and prints whether each objective was met. Not meeting them is a perfectly good result and is what the exercise is for — an unmet objective discovered in a lab is a finding, and the same objective unmet during a real incident is an outage nobody planned for. The final diff counts the records that did not survive, which is your RPO in the only units that matter.

Notes

The step that produces the most value is following your own procedure without improvising. Every recovery procedure has gaps, and the only way to find them is to be unable to improvise past them — which is exactly the position the person on call at 3am is in, and the reason the Domain 3 lesson insists that failover is the only test that proves capability.

This is an independent study companion for CompTIA Security+ SY0-701 and is not produced by or endorsed by CompTIA.