Visualizing Borg Backup Success and Failures in ELK with Structured JSON Logs

When I first started using BorgBackup for our infrastructure backups, I relied on email notifications and manual log checks to confirm success. That worked until backup jobs began silently failing due to network timeouts or repository locks, and I wouldn’t notice until days later. The turning point came when I enabled Borg’s structured JSON logging and forwarded it to our ELK stack. Suddenly, I could see every backup run as a searchable event — success, duration, bytes transferred, and crucially, the exact reason for any failure. In this post, I’ll show you how I configured BorgBackup with --log-json, shipped logs to Logstash, indexed them in Elasticsearch, and built visualizations and alerts in Kibana to monitor backup health at a glance.

Why Structured Logging Matters for Backups

Plain text logs from Borg are readable but hard to parse at scale. When you’re managing dozens of backup targets across different schedules, grepping for 'ERROR' or 'archive created' becomes unreliable. Borg’s --log-json flag changes that by emitting each log line as a valid JSON object with consistent fields like event, path, repository, archive, duration, and success. This makes it trivial to ship to a centralized log system where you can filter, aggregate, and alert based on actual backup outcomes rather than text patterns.

In my setup, each backup job runs via a systemd timer and includes --log-json in the borg create command. The output goes straight to stdout, which journald captures and forwards to our Loki/Promtail stack — though for ELK, I use Filebeat to read the journal or a dedicated log file. The key is ensuring the JSON remains intact so Logstash can parse it without breaking on malformed lines.

Configuring Borg for JSON Logging

Here’s an example of how I invoke Borg in a backup script:

#!/bin/bash
REPOSITORY=ssh://backup@backup-server:/mnt/borg/{hostname}
SOURCE_DIRS="/etc /var/www /home"

borg create \
    --log-json \
    --compression lz4 \
    --stats \
    $REPOSITORY::'{hostname}-{now:%Y-%m-%d_%H:%M}' \
    $SOURCE_DIRS

# Exit code check still matters — JSON logs don’t replace it
if [ $? -eq 0 ]; then
    logger -t borg-backup "Backup completed successfully"
else
    logger -t borg-backup "Backup failed with exit code $?"
fi

The --log-json flag ensures every significant event — start, checkpoint, archive completion, errors — is output as JSON. I still keep a traditional logger call for immediate syslog visibility, but the real power comes from the structured stream.

Shipping JSON Logs to ELK with Filebeat

I use Filebeat to read the systemd journal (or a log file if you redirect output) and forward JSON lines to Logstash. The critical part is telling Filebeat to treat each line as JSON and preserve its structure.

In my Filebeat module configuration:

filebeat.inputs:
- type: filestream
  enabled: true
  paths:
    - /var/log/borg-backup.log
  json.keys_under_root: true
  json.add_error_key: true

# If reading from journal instead:
# - type: journald
#   enabled: true
#   journal.paths: ['/var/log/journal/']
#   json.keys_under_root: true

The json.keys_under_root: true setting flattens the JSON fields directly into the Elasticsearch document, so event, success, duration, and repository become top-level fields you can query and visualize without pain.

Parsing and Enriching in Logstash

Although Filebeat can send JSON directly to Elasticsearch, I route it through Logstash to add context and handle edge cases. For example, I enrich each backup event with the source host name and backup type (daily, weekly, etc.) using tags from the filename or environment.

A simple Logstash filter looks like this:

filter {
  if [event] == "archive_create" {
    mutate {
      add_field => { "backup_type" => "%{[@metadata][backup_type]}" }
      add_tag => [ "borg-backup" ]
    }
    if [success] == false {
      mutate { add_tag => [ "backup-failed" ] }
    }
  }
}

This lets me create Kibana filters like tag:backup-failed to instantly see all failed runs, or chart average duration by backup_type over time.

Visualizing Backup Success in Kibana

Once the data is in Elasticsearch, I built a few core visualizations in Kibana:

• A time series chart showing daily backup success rate (count of success:true vs success:false)
• A bar chart of average backup duration by host, to spot slow outliers
• A table listing recent failed runs with repository, error message (from the message field), and duration
• A pie chart of failure reasons — I extract common patterns from the message field using a Painless script or Logstash grok (e.g., "connection timeout", "repository locked", "checkpoint failed")

These live on a dedicated "Backup Monitoring" dashboard that our ops team reviews weekly. Because the data is structured, I can drill down from a failed bar in the chart to the exact log entries showing what went wrong.

Setting Up Alerts for Backup Failures

Visualization helps with trend analysis, but I needed proactive alerts. In Kibana, I created a rule that triggers when:

• tag:backup-failed appears
• And the timestamp is within the last 15 minutes
• And the host is in our critical infrastructure group

The action sends a message to our Mattermost channel and creates a ticket in our internal tracking system via webhook. Because the alert is based on a tagged field rather than log text matching, it’s reliable — no false positives from grep-like patterns.

Tips and Gotchas

• Always test your --log-json output with a dry run: borg create --log-json --dry-run ...
• Ensure your log collector doesn’t truncate or merge lines — JSON logging depends on intact lines
• If you see _jsonparsefailure tags in Elasticsearch, check for non-JSON output (e.g., password prompts going to stdout)
• Consider separating logs by severity: use --log-json for operational events and keep verbose/debug logs to a separate stream if needed

Final Thoughts

Switching to structured JSON logging with BorgBackup transformed how I monitor backups. Instead of hoping notifications work, I now have a searchable, auditable record of every run — complete with context that helps me diagnose issues fast. Combined with ELK, it turns backup monitoring from a reactive chore into a transparent, measurable part of our infrastructure health.

If you’re using Borg and still parsing text logs for success/failure, try --log-json. It’s a small change that pays off immediately when something goes wrong — and you’ll wonder how you ever managed without it.

Cover image: Lenharth Systems · CC0 (Openverse / kamu malı) · https://stocksnap.io/photo/computer-hard-2J3PLNMO9M