Restore testing: the only proof a backup works

Why this matters

A backup is a claim. A restore is evidence. Until a restore has been performed, an organisation does not have backups — it has jobs that report success.

This is not a rhetorical point. Backup jobs fail silently in specific, common ways: they back up the wrong thing, they back up an encrypted volume without the key, they succeed against a database that was never quiesced, they exclude a path added six months ago, or they write to a destination that has been full for a week while the job reports completion.

Every one of those produces a green dashboard and no recoverable data, and every one is discovered at the worst possible moment unless somebody tests.

The lesson

The backup job that reports success and restores nothing, and how it happens

The specific failure modes, because recognising them is most of the skill:

• Backing up the wrong scope. A new volume attached to an instance is not in the backup selection, because the selection names volumes rather than instances or tags. Everything on it is unprotected and nothing reports an error. Tag-based selection — back up everything tagged backup:true, and alert on resources with no backup tag — is the structural fix.
• Crash-consistent captures of a transactional system, per the previous lesson. The job succeeds; the restore produces a database that will not open.
• Missing encryption keys. Data restored without the key that decrypts it is not recovered. This is especially easy to miss with customer-managed keys, cross-account copies and cross-region restores, where the key may not exist in the destination.
• A destination that silently stopped accepting data — full storage, expired credentials, a changed permission — where the job's own status is reported before the write is confirmed.
• Retention deleting faster than anyone realised, so the backup from the date you need has already expired.
• Backup of the data but not of what makes it usable — the schema, the configuration, the certificates, the secrets. A restored database nobody can authenticate to is not a restored service.

The common thread: the job's definition of success is not the same as yours. A job succeeds when it finishes writing. You succeed when a working system comes back.

Restoring to a clean environment rather than over the thing you still need

The first rule of restore testing: never test by restoring over production. A test that can cause the incident it protects against is not an acceptable test.

Restore to an isolated environment instead — a separate account, a separate network, or at minimum a separate instance with no production connectivity. This is one of the clearest advantages the cloud has over traditional infrastructure: standing up a parallel environment for an afternoon is cheap and requires nobody to find spare hardware.

The isolation matters for more than safety. A restored copy with production connectivity can do real damage: send real emails to real customers, process real payments, write to real integrations, or register itself with real monitoring. Restore into a network with no outbound path to anything that matters, and check what the restored system tries to do when it starts.

Two further points:

• Restore from the backup, not from a replica or the live system. It is surprisingly easy to build a test that never touches the backup at all.
• Test the oldest retained backup occasionally, not only yesterday's. A format or tooling change can make old backups unreadable, and you want to learn that before you need one.

Measuring restore TIME, not just restore success, against the promise you made

A restore that works and takes eighteen hours has still failed if the recovery objective was four.

So every restore test should produce a duration, broken into its parts:

• Time to locate and select the right backup — which is longer than people expect when nobody has practised.
• Time to provision the target environment.
• Time to transfer the data, which for archive tiers includes the retrieval delay before transfer even begins.
• Time to restore and recover — replaying logs, rebuilding indexes, checking consistency. For large databases this frequently dominates.
• Time to validate and cut over — reconfiguring, repointing DNS, restarting dependencies.

The last two are routinely omitted from estimates and routinely dominate the actual number.

Compare the measured total against the RTO from the next lesson. If it does not meet it, you have a concrete, evidenced problem with named options: warmer standby, more frequent fulls, faster storage tiers, parallel restores, or a renegotiated objective. Without the measurement, the conversation is opinion against opinion.

Measure RPO too — the age of the data you recovered. A nightly backup restored at midday means losing a morning's work, and whether that is acceptable is a business question that should be answered before the incident rather than during it.

Partial restores: one file, one table, one instance, which are the common real requests

Full disaster recovery is the dramatic case and the rare one. In practice, almost every restore request is small:

• "Someone deleted a folder."
• "A bad deployment corrupted one table."
• "We need the version of this document from Tuesday."
• "One instance failed and needs rebuilding."

These are what your backups will actually be used for, so they are what to practise. They also expose design problems the full-restore test does not:

• Granularity. A volume snapshot can restore a volume. Extracting one file from it means restoring the whole volume, attaching it somewhere, and copying the file out — which works and takes far longer than people assume. If single-file recovery is a real requirement, a file-level backup is needed.
• Table-level restore usually means restoring a full database copy to a temporary instance and extracting from it. Expensive and slow, and worth knowing in advance.
• Object versioning is the neat answer for the document case: enabled on a bucket, it keeps previous versions and makes single-object recovery trivial. It is a different control from backup and complements it — and note that versioning alone does not protect against deletion of the bucket itself.
• Self-service restore for common cases removes the operations team from the loop, and is worth building where the requests are frequent.

Practise the small restores, because they are the ones you will be doing.

A restore test schedule, its evidence, and who signs it

Testing must be a schedule with an owner, or it becomes something that happened once.

A workable programme:

• Automated restore verification, continuously. Restore something every night, automatically, and assert on the result — a row count, a checksum, a service responding. This is the strongest control available and it needs no human. Some platforms offer this as a feature; otherwise it is a pipeline.
• A monthly partial restore, performed by a person, rotating through systems and through the request types above.
• A quarterly or annual full-system recovery exercise, as part of the disaster-recovery testing in the next lesson.
• A restore test after any material change to a system, its backup configuration, or its encryption.

The evidence is what makes it auditable, and objective 4.4 will ask for exactly this. Each test should record: what was restored, from which backup of which date, to where, how long each phase took, what was validated and how, what failed, and what was changed as a result.

Who signs it matters as much as who runs it. The system owner should acknowledge that the measured recovery time and data loss are acceptable to the business — or state that they are not, which converts a technical measurement into a decision with a budget attached. That signature is the mechanism by which "we cannot meet the stated RTO" becomes a business problem rather than a private worry, and getting it in writing is the professional move.

What to take into the exam

• A backup job reporting success is not evidence. Common silent failures: wrong scope, crash-consistent databases, missing keys, full destinations, retention already expired.
• Tag-based backup selection, plus alerting on untagged resources, fixes the scope problem structurally.
• Never test by restoring over production. Restore into an isolated environment with no path to real systems.
• Measure duration, including retrieval, recovery and validation, and compare against RTO. Measure the recovered data's age against RPO.
• Most real restores are partial. Single-file recovery from a volume snapshot is slow; object versioning is the right control for document recovery.
• Automate nightly restore verification where possible, and keep written evidence signed off by the system owner.