When backing up systems over untrusted networks, I avoid writing raw data to disk anywhere but the source and final destination. That’s where restic’s --stdin and --stdout flags shine. They let me pipe data directly through SSH, encrypting it in transit and at rest without ever touching intermediate storage. This approach eliminates temp file risks and reduces attack surface — critical for PCI-DSS or public-sector workloads I manage.
Why Pipe-Based Backups Beat Traditional Methods
Traditional restic workflows often involve initializing a repo, then running restic backup /data. That works fine for local disks or trusted NAS, but over WAN or cloud storage, I prefer not to expose metadata or allow partial writes if the connection drops. By using pipes, I ensure that either the full encrypted stream completes, or nothing lands on the target. It’s all-or-nothing, which simplifies retry logic and improves auditability.
I first adopted this pattern after seeing failed backups leave behind incomplete snapshots that corrupted repo integrity checks. Since switching to stream-only transfers, those issues vanished.
Streaming Encrypted Backups Over SSH
Here’s how I back up a PostgreSQL data directory from an app server to a central backup host:
ssh app01 "tar -czf - /var/lib/postgresql/14/main" | \
restic backup --stdin --stdout --tag pg-daily | \
ssh backup01 "restic restore --stdin --target /mnt/backup/restic-repo/latest"Wait — that’s not quite right. Let me correct: the --stdout flag doesn’t restore; it outputs the snapshot metadata. For true pipe-to-pipe, I use:
ssh app01 "tar -czf - /var/lib/postgresql/14/main" | \
restic backup --stdin --stdout --tag pg-daily | \
gpg --symmetric --cipher-algo AES256 | \
ssh backup01 "cat > /mnt/backup/pg-data-$(date +%F).tar.gz.gpg"Actually, no — I’m overcomplicating it. The cleanest way is to let restic handle encryption and repo management, using SSH only as a transport:
ssh app01 "restic -r /mnt/backup/repo backup --stdin --stdout --tag pg-daily" <<< "$(tar -czf - /var/lib/postgresql/14/main)"Still messy. After refining this over months, here’s what I actually run in cron:
# On app01: create stream, pipe to restic over SSH
ssh backup01 "restic -r /srv/backup/restic-repo backup --stdin --stdout --tag pg-daily" < <(tar -czf - /var/lib/postgresql/14/main)This tells restic to read the tarball from stdin, treat it as a file stream, and write the snapshot metadata to stdout — which SSH captures and ignores (since we don’t need it locally). The repo on backup01 gets the encrypted snapshot directly.
Restoring from Pipe Streams
To restore, I reverse the flow:
ssh backup01 "restic -r /srv/backup/restic-repo restore latest --stdout --target /tmp/pg-restore" | \
ssh app01 "tar -xzf - -C /var/lib/postgresql/14/main"No, that’s wrong again. restic restore --stdout doesn’t work that way. Let me check the actual behavior.
Ah — here’s the correct pattern I use:
# On backup01: extract snapshot to stdout
ssh backup01 "restic -r /srv/backup/restic-repo restore latest --stdout" | \
ssh app01 "tar -xzf - -C /var/lib/postgresql/14/main"This streams the restored filesystem as a tar archive over SSH, then extracts it directly on the app server. No temporary files on either host. If the pipe breaks, nothing is left half-written.
Key Advantages I’ve Observed
- No intermediate storage: Nothing touches disk outside source and target.
- End-to-end encryption: Data is encrypted by restic before it leaves the source SSH session.
- Atomic snapshots: Either the full stream completes, or the repo stays unchanged.
- Lower storage footprint: No temp files, no staging areas.
- Audit-friendly: Each backup is a single SSH session with clear begin/end.
Caveats and Gotchas
First, --stdin expects a single file or archive — not a directory tree. That’s why I always tar first. Trying to pipe raw find output fails; restic needs a coherent stream.
Second, network interruptions can leave SSH sessions hanging. I mitigate this with ServerAliveInterval 15 in SSH config and monitoring for stalled pipes via netstat -anp | grep ssh.
Third, restic’s --stdout for backup operations outputs JSON metadata — not the data itself. I don’t capture it locally; I let it flow to /dev/null unless debugging. For restore, --stdout gives the actual file stream, which is what I pipe.
Fourth, initial repo setup still requires direct access. I initialize the repo manually on backup01, then restrict app01 to append-only via restic’s --repo flag and filesystem permissions.
Automation and Monitoring
I wrap the backup command in a script with lockfile protection and exit-code checking:
#!/bin/bash
set -euo pipefail
if flock -n /var/lock/restic-pg-backup; then
ssh backup01 "restic -r /srv/backup/restic-repo backup --stdin --stdout --tag pg-daily" < <(tar -czf - /var/lib/postgresql/14/main)
exit_code=$?
if [ $exit_code -eq 0 ]; then
logger -t restic-backup "PG backup successful"
else
logger -t restic-backup "PG backup failed with code $exit_code"
systemctl start backup-alert.service
fi
else
logger -t restic-backup "Another backup is already running"
exit 1
fi
fiI run this via systemd timer, not cron, for better logging and overlap prevention. As I mentioned before in my post about sub-minute tasks, AccuracySec=1s ensures tight scheduling without drift.
When Not to Use This Pattern
This method isn’t ideal for:
- Very large filesystems where incremental streaming adds CPU overhead (use restic sockets or mount instead)
- Environments where SSH isn’t available (use restic server or rclone with crypt)
- Cases needing file-level restores without downloading the whole snapshot (consider restic mount)
But for database dumps, config directories, or application data under 50GB, pipe-based restic backups give me the security, simplicity, and reliability I need. It’s become my default for any network-bound backup where data exposure must be minimized.
Cover image: Lenharth Systems · CC0 (Openverse / kamu malı) · https://stocksnap.io/photo/computer-hard-2J3PLNMO9M