Let me say it bluntly: a backup you have never restored is not a backup — it is a gamble. I have seen environments where admins religiously check backup job success status every morning, yet nobody has ever actually restored a single file from those backups. The result is always the same: when disaster hits, the "successful" backup turns out to be corrupted, incomplete, or missing a critical component. If you are not doing regular restore testing, you are storing data you cannot recover.
This is something I touched on before in my post about silent data loss scenarios (https://furkanikkan.com/urun/geri-yuklemede-basarisiz-olan-yedekler-sessiz-veri-kaybi-senaryosu-27), but this time I want to go deeper into the actual drill process — not just the theory.
Why Backup Success Reports Lie to You
Most backup tools are generous with their success reporting. Veeam says "Success" if the snapshot was taken and the data was moved to the repository. Bacula says the job finished with 0 errors. But here is what those reports do not tell you:
- Whether the application can actually start from the restored data
- Whether the backup captured the transaction logs or just the data files
- Whether the restore will work on a different host with different hardware
- Whether the encryption keys are still accessible and not expired
- Whether the backup chain is actually complete or missing an incremental
I learned this the hard way with a PostgreSQL server. The backup job reported success for 14 straight days. When we needed to restore, it turned out the WAL archiving had silently stopped after a config change. The base backup was fine, but we could only recover up to the moment the base backup was taken — losing a full week of transactions. The success report was technically accurate. It just was not telling us what we actually needed to know.
How I Structure Restore Testing in Production
I run restore tests on a schedule that matches the risk level of each system. Here is the rough breakdown I use in my environment:
- Critical databases (PostgreSQL, MSSQL): Weekly full restore to a test instance, plus automated daily transaction log replay
- File servers: Monthly restore of random sample files, size check against source
- VM backups: Monthly full VM restore to an isolated network, boot test, service check
- Active Directory / DNS: Quarterly full restore to a lab domain controller, verify replication metadata
- Web servers (Nginx configs, app files): Monthly restore + config syntax check before reload
The key word here is "isolated." You never restore to the production network. I use a dedicated VLAN with no route to production subnets. If the restored VM has old network config or tries to contact a domain controller, it should fail safely in isolation — not cause chaos on the real network.
A Practical Restore Drill: PostgreSQL Example
Here is what a real weekly PostgreSQL restore drill looks like for me. I keep it simple and scripted so it runs the same way every time.
# On the test restore host
export PGPASSWORD=$RESTORE_TEST_PASS
# Drop and recreate the test database
dropdb --if-exists -h localhost restore_test_db
createdb -h localhost restore_test_db
# Restore from the latest base backup
pg_restore -h localhost -d restore_test_db \
-j 4 --no-owner --no-privileges \
/backups/pg/base/latest.dump
# Replay WAL up to a specific timestamp
pg_ctl -D /var/lib/postgresql/restore_test \
-o "-c restore_command='cp /backups/pg/wal/%f %p'" \
-o "-c recovery_target_time='2024-01-15 14:30:00+03'" start
# Verify: count rows in a critical table
psql -h localhost -d restore_test_db -c \
"SELECT count(*) FROM orders WHERE created_at >= now() - interval '7 days';"If the row count matches what production has for that same period, the restore is good. If it does not match, I have a problem to investigate before it becomes a real emergency. I log every drill result — date, database, row count, time taken — in a simple text file. It is not fancy, but it gives me a history I can grep when someone asks "are we sure backups work?"
The Three Questions Every Restore Test Must Answer
A restore test is not just about copying files back. You need to answer three concrete questions:
- Can the application start? Restored files are useless if the app cannot read them. Boot the service, check the logs, and make sure it actually starts without errors.
- Is the data point-in-time consistent? If you are doing point-in-time recovery, verify that the data reflects the exact moment you targeted. Check timestamps on recent records.
- Can someone else do it? If the only person who knows the restore process is on vacation, you do not have a restore plan. Document it, test the documentation by having a colleague follow it blindly.
What Happens When You Skip Restore Testing
I will keep this short because I have seen the outcome more than once. A company loses a critical server. The admin is confident because backups have been running for months. The restore starts, and then:
- The backup repository is on a volume that was silently running out of space, so the last 3 backups were truncated
- The encryption key is stored on the same server that just died
- The backup software version was upgraded 2 months ago and the old backup format is not compatible with the new restore tool
- Nobody knows the restore command because the person who set it up left the company
Every one of these is preventable. But you only find them through restore testing.
Note: If you are using a managed backup service, do not assume the provider is doing restore tests for you. Most do not. Read the SLA carefully and run your own tests against the data they store.
Building a Restore Testing Culture
The technical part is easy. The hard part is making restore testing a habit that survives deadlines and staffing changes. My approach:
- Calendar it as a recurring event, not a "when I have time" task
- Automate as much as possible — manual testing gets skipped when people are busy
- Share results with the team so everyone sees when a test fails
- Treat a failed restore test with the same urgency as a production incident
As I mentioned in the ransomware response checklist (https://furkanikkan.com/urun/ransomware-mudahale-plani-ilk-60-dakika-kontrol-listesi-32), the first 60 minutes of a ransomware incident are about containment and assessment. But if your backups have never been tested, that 60-minute window becomes a guessing game about whether you can recover at all.
Backups without restore tests are theater. They make you feel safe until the moment you actually need them. Run the drill. Find the broken backup before the attacker or the disk failure does.
Cover image: cogdogblog · CC0 (Openverse / kamu malı) · https://www.flickr.com/photos/37996646802@N01/4158729317