Measure a failover instead of assuming it

short · 40 min · Objective 3.4

Task

Build a two-node service with a health check, fail one node, and measure how long the service was actually unavailable. The number is almost never the one people expect, and an RTO that has not been measured is a target rather than a capability.

Steps

  1. Run the same simple service on both VMs, each returning its own hostname so you can tell which answered.
  2. Put a load balancer or a health-checked proxy in front of them. nginx with two upstreams and a short health check interval is enough, or haproxy.
  3. Start a client loop that requests the service every 200ms and logs the timestamp and which node answered, to /tmp/failover.log.
  4. Let it run for a minute, then stop the service on the active node abruptly — kill the process rather than shutting it down gracefully, because a graceful stop measures the wrong thing.
  5. Let the loop continue for another minute, then stop it.
  6. Compute the gap: the time between the last successful response and the next one. Compare that with the health check interval and the failure threshold you configured, and write both into /tmp/rto.md.

Verify

wc -l < /tmp/failover.log
python3 - <<'PY'
from datetime import datetime
rows=[l.split() for l in open('/tmp/failover.log') if l.strip()]
ok=[datetime.fromisoformat(r[0]) for r in rows if 'OK' in r[-1] or 'node' in r[-1]]
gaps=[(b-a).total_seconds() for a,b in zip(ok,ok[1:])]
worst=max(gaps) if gaps else 0
print('requests logged: %d | longest outage: %.1fs' % (len(rows),worst))
assert len(rows)>200, 'the loop did not run long enough'
assert worst>0.5, 'no outage was observed - the node was not actually killed'
print('measured failover gap: %.1f seconds' % worst)
PY
grep -ciE "health check|interval|threshold" /tmp/rto.md

The assertion requires a real measurable gap: if the longest outage is under half a second, the node was not genuinely killed and the failover was never exercised. The number you get is your actual recovery time for this failure mode, and it is usually several times the health check interval because the threshold requires consecutive failures.

Notes

Now compare that measured number with the RTO you would have written down. This is the MTTR-versus-RTO gap from the Domain 5 lesson, and it is the most actionable output a business impact analysis produces — a stated objective the measured capability does not meet.

This is an independent study companion for CompTIA Security+ SY0-701 and is not produced by or endorsed by CompTIA.