Prove a backup by restoring it

applied · 45 min · Objective 1.6

Task

Take a backup of a running service's data, then destroy the data and restore it from the backup alone. The point is the second half: a backup nobody has ever restored is a hope, not a backup. You will measure the one number that matters -- whether the restored data is byte-for-byte what you backed up.

Steps

  1. Record the truth, twice, before the backup. The contents: find /srv/data -type f -exec sha256sum {} + | sort > /tmp/before.sha. And the shape -- every file, directory and symlink with its type, mode and owner, which the content hashes cannot see: find /srv/data -printf '%y %m %u:%g %p\n' | sort > /tmp/before.meta.
  2. Take the backup with the tool the service actually recommends -- a database dump for a database, tar/rsync for files. Note whether the service must be quiesced first, and do so if it must.
  3. Copy the backup OFF the machine, or at least off the disk holding the live data. A backup on the same disk dies with it.
  4. Destroy the live data for real: stop the service and rm -rf the data directory (this is why the VM is disposable).
  5. Restore from the backup alone, following the documented restore procedure -- not your memory of it.
  6. Start the service and record both again, into /tmp/after.sha and /tmp/after.meta, with the same two commands.
  7. Time the whole restore. That number is your recovery time; write it down, because "we have backups" without it answers the wrong question.

Verify

diff /tmp/before.sha /tmp/after.sha && echo "every file's contents came back"
diff /tmp/before.meta /tmp/after.meta && echo "and every owner, mode, directory and link"
systemctl is-active the-service          # service came back up

Both diffs returning nothing is the proof, and it takes both. The content hashes cover regular files only: a restore that lost an empty spool directory, flattened a symlink, or brought the data back owned by root instead of the service account passes them completely -- and the last of those is the one that stops a database starting. The metadata manifest is what catches it. If the service also has application-level integrity (a database consistency check), run it -- matching file hashes prove the bytes returned, an integrity check proves they are usable.

Notes

The failure this lab prevents is the backup that runs nightly for a year and cannot be restored -- wrong flags, a missing pre-quiesce step, a dump that excluded a table, a compression the restore host cannot read. None of that shows up until the restore, so the restore is the test. Schedule restore drills the way you schedule the backups.

This is an independent study companion for CompTIA Linux+ and is not produced by or endorsed by CompTIA.