Run a recovery exercise against a stated RTO and RPO

capstone · 140 min · Objective 3.4

Task

Pull Domain 3 together: state recovery objectives, build to them, then run a real recovery and find out whether you met them. The gap between the objective and the measurement is the output of this exercise, and it is the finding a business impact analysis exists to produce.

Steps

  1. Define the service: a small application on one VM with data it writes continuously — a script appending timestamped records to a database or file is enough.
  2. State the objectives BEFORE building anything: write /tmp/objectives.md with an RTO and an RPO you have chosen, and the reasoning for each in business terms rather than technical ones.
  3. Build to them. The RPO decides backup or replication frequency; the RTO decides whether the second VM is warm, cold, or already running.
  4. Document the recovery procedure so that somebody else could follow it at 3am: exact commands, in order, including how to confirm success.
  5. Now destroy the primary. In one command, copy its records to /tmp/records-before.txt (the lab's ground truth -- in a real incident you would not have it), stop it abruptly and delete its data directory: cp <data file> /tmp/records-before.txt && kill -9 <pid> && rm -rf <data dir>. Note the time: that is the incident, and the clock starts now.
  6. Recover by following your own written procedure exactly, without improvising. Where the procedure is wrong or incomplete, note it and keep going.
  7. Stop the clock when the service is genuinely usable again, and save its records to /tmp/records-after.txt. Measure two things, both in minutes: elapsed time, against your RTO; and the data-loss window, against your RPO -- the time of the incident minus the timestamp of the newest record that survived. An RPO is a span of time, so the count of lost records is not the measurement; it is the evidence for it.
  8. Write /tmp/exercise.md: both objectives, both measurements, whether each was met, every gap in the written procedure, and one change you would make to close the largest gap. Put the four numbers on lines of their own, in minutes, exactly like this so the Verify can read them:
RTO target: 30
RTO actual: 41
RPO target: 15
RPO actual: 6

Files the Verify reads

The Verify block reads these by name, so save them exactly here:

  • /tmp/records-before.txt -- the data set's records before the incident, one per line.
  • /tmp/records-after.txt -- the same after recovery, same format, so the diff shows exactly what was lost.

Verify

python3 - <<'PY'
import re
t=open('/tmp/exercise.md').read()
def val(label):
    m=re.search(r'^\s*'+label+r'\s*:\s*([0-9.]+)',t,re.I|re.M)
    return float(m.group(1)) if m else None
for name in ('RTO','RPO'):
    target,actual=val(name+' target'),val(name+' actual')
    assert target is not None, 'no "%s target: <minutes>" line in the write-up' % name
    assert actual is not None, 'no "%s actual: <minutes>" line in the write-up' % name
    print('%s target %g min | measured %g min | %s' %
          (name,target,actual,'MET' if actual<=target else 'NOT MET'))
PY
grep -ciE "gap|missing step|would change" /tmp/exercise.md
diff <(sort /tmp/records-before.txt) <(sort /tmp/records-after.txt) | grep -c '^<'

The Python block requires all four numbers to be present and prints whether each objective was met. Not meeting them is a perfectly good result and is what the exercise is for — an unmet objective discovered in a lab is a finding, and the same objective unmet during a real incident is an outage nobody planned for. The final diff counts the records that did not survive: the evidence behind your RPO measurement. Check the two agree -- the lost records should all carry timestamps inside the window you reported.

Notes

The step that produces the most value is following your own procedure without improvising. Every recovery procedure has gaps, and the only way to find them is to be unable to improvise past them — which is exactly the position the person on call at 3am is in, and the reason the Domain 3 lesson insists that failover is the only test that proves capability.

This is an independent study companion for CompTIA Security+ SY0-801 and is not produced by or endorsed by CompTIA.