A backup that cannot be restored is not a backup — it is just a pile of files taking up disk space. I learned this the hard way after believing for months that my backups were running fine. The logs said success, the cron jobs fired every night, the storage kept growing. Then a customer needed a restore, and I discovered the hard way that every single backup was useless.
This is the scenario nobody talks about: backups that look healthy but fail silently when you actually try to recover from them. Here is what went wrong in my environment, how I fixed it, and what I test now to make sure a restore actually works.
What Happens When You Trust Backup Logs Blindly
The setup looked solid on paper. A nightly script dumped databases, compressed them, and shipped them to a secondary storage box. Every morning I checked the logs and saw Backup completed successfully. I even had monitoring alerts wired up — if the script failed, I would get a notification.
What I did not have was a verification step. The script was producing empty gzip files because the database dump command had silently failed weeks earlier. A credentials rotation broke the connection string, the dump command exited with a non-zero code, but the script kept going and compressed the empty stdout output anyway. The log line Backup completed successfully was printed at the end of the script regardless of whether the actual data dump worked.
That is the trap. Logs tell you the script ran. They do not tell you the backup is usable.
Why Restore Testing Is the Only Real Backup Test
I used to think checking file sizes was enough. If today's backup file is roughly the same size as yesterday's, the backup must be fine, right? Wrong. A corrupted database can still produce a dump file of similar size. A half-failed backup can miss entire tables but still look normal in a directory listing.
The only way to know a backup works is to restore it. Not glance at it, not check the file size — actually restore it to a test server and verify the data is there.
Here is what I test now on a regular schedule:
- File integrity: Can I decompress the archive without errors?
- Database restore: Does the dump import cleanly into a fresh database instance?
- Row counts: Do critical tables have the expected number of rows after restore?
- Application boot: Can the application actually start against the restored data?
- Point-in-time accuracy: Does the restored data match what production looked like at the backup timestamp?
If any of those checks fail, the backup is broken. Full stop.
Common Silent Backup Failures I Have Seen
Over the years I have hit several failure modes that do not trigger obvious errors. Here are the ones that caught me off guard:
Empty or partial dumps. As I mentioned, a credentials change or a silent connection failure can produce empty backup files that still compress and transfer successfully.
Encoding mismatches. A database dump restored into a server with a different default encoding can corrupt text data. The restore runs without errors, but Turkish characters turn into garbage.
Missing dependencies. The backup script works on Server A, but when you try to restore on Server B, a required tool is missing or the version is wrong. The backup was fine all along, but the restore environment was never prepared.
Incremental backup chain breaks. If one incremental in the chain is corrupted or missing, every subsequent incremental becomes unrestorable. You might have 30 days of incremental backups and only the first full backup actually works.
Storage full on the backup target. The backup script writes successfully to local disk, but the transfer to remote storage fails silently because the remote disk is full. Old backups pile up, new ones never arrive.
How I Verify Backups Now
After that incident, I built a verification routine that runs automatically. The principle is simple: if a human is not checking the backup, a script should be.
For database backups, I run a restore test on a disposable container every week. The script looks something like this:
#!/bin/bash
BACKUP_FILE=$(ls -t /backups/db/*.sql.gz | head -1)
tmpdir=$(mktemp -d)
gunzip -c "$BACKUP_FILE" > "$tmpdir/restore.sql"
docker run --rm \
-e POSTGRES_PASSWORD=test \
-v "$tmpdir/restore.sql":/restore.sql \
postgres:15 \
bash -c "pg_restore -U postgres -d postgres /restore.sql && \
psql -U postgres -d postgres -c 'SELECT count(*) FROM users;'"
if [ $? -ne 0 ]; then
echo "RESTORE FAILED for $BACKUP_FILE"
exit 1
fiIf that script exits non-zero, my monitoring system alerts me. No alert means the backup restored and a critical table returned a row count. Is it a full data validation? No. But it catches the failure mode that bit me — empty or corrupted dumps that look fine until you try to use them.
For file-level backups, I do a periodic test restore of a random file and compare checksums against the source. If the checksums do not match, the backup is flagged as suspect.
The Backup Rule I Actually Follow Now
I used to follow the classic 3-2-1 rule: three copies, two media, one offsite. I still do, but I added one more requirement that matters more than the rest:
Every backup must be restorable and verified.
A backup you have never restored is a hypothesis, not a backup. You are guessing it works. I was guessing for months, and I lost data because of it.
If you take one thing from this post, let it be this: pick one backup, any backup from this week, and try to restore it right now. Not tomorrow, not after you write a script. Right now, manually, in a terminal. If it works, great — now automate that test. If it does not work, you just found a problem before a real disaster did.
That is the whole point. Find the broken backup before someone asks you to restore it.
Cover image: Lenharth Systems · CC0 (Openverse / kamu malı) · https://stocksnap.io/photo/computer-hard-2J3PLNMO9M