Prove a backup by restoring it

applied · 45 min · Objective 1.6

Task

Take a backup of a running service's data, then destroy the data and restore it from the backup alone. The point is the second half: a backup nobody has ever restored is a hope, not a backup. You will measure the one number that matters -- whether the restored data is byte-for-byte what you backed up.

Steps

Record the truth: hash the data set before backup (find /srv/data -type f -exec sha256sum {} + | sort > /tmp/before.sha).
Take the backup with the tool the service actually recommends -- a database dump for a database, tar/rsync for files. Note whether the service must be quiesced first, and do so if it must.
Copy the backup OFF the machine, or at least off the disk holding the live data. A backup on the same disk dies with it.
Destroy the live data for real: stop the service and rm -rf the data directory (this is why the VM is disposable).
Restore from the backup alone, following the documented restore procedure -- not your memory of it.
Start the service and re-hash: find ... -exec sha256sum {} + | sort > /tmp/after.sha.
Time the whole restore. That number is your recovery time; write it down, because "we have backups" without it answers the wrong question.

Verify

diff /tmp/before.sha /tmp/after.sha && echo "restore is bit-identical"
systemctl is-active the-service          # service came back up

diff returning nothing is the proof: every file present before is present after with the same hash. If the service also has application-level integrity (a database consistency check), run it -- matching file hashes prove the bytes returned, an integrity check proves they are usable.

Notes

The failure this lab prevents is the backup that runs nightly for a year and cannot be restored -- wrong flags, a missing pre-quiesce step, a dump that excluded a table, a compression the restore host cannot read. None of that shows up until the restore, so the restore is the test. Schedule restore drills the way you schedule the backups.