Prevent journald Disk Spikes with RateLimitInterval and Burst Settings

I’ve seen it too many times: a misbehaving service starts spamming the journal, and suddenly your root partition hits 100%. The system slows, logs stop being written, and you’re scrambling to free space before something critical breaks. In my environment, I rely on systemd-journald’s built-in rate limiting to keep log volume under control. By adjusting RateLimitInterval and Burst in /etc/systemd/journald.conf, you can prevent a single service from flooding the journal and eating up disk space.

Understanding journald Rate Limiting

systemd-journald includes two key settings for controlling log influx: RateLimitInterval and Burst. Together, they define how many log entries are allowed within a specific time window before excess messages are dropped. RateLimitInterval sets the time window in seconds, and Burst defines the maximum number of messages allowed in that window. If a service exceeds this limit, journald stops processing further messages until the interval resets, preventing runaway logging from consuming disk I/O and storage.

By default, these values are set conservatively to allow normal operation while still offering protection. But on busy systems or when debugging noisy applications, the defaults might be too permissive. I’ve adjusted them on several production servers after noticing journal growth spikes during cron runs or application restarts.

Checking Current Journal Usage

Before tuning, I always check how much space the journal is currently using and whether rate limiting is already active. You can see disk usage with journalctl --disk-usage. This shows the total space consumed by archived and active journals.

To see if messages are being dropped due to rate limiting, look for entries like "Journal file /var/log/journal/... is full" or check the rate limit counters with:

journalctl --since "1 hour ago" | grep -i "rate limit"

If you see frequent drops, it’s a sign that your current settings are being hit regularly. That doesn’t always mean you need to increase limits—sometimes it means a service is logging too verbosely and should be fixed at the source.

Setting RateLimitInterval and Burst

Edit /etc/systemd/journald.conf as root. I usually set RateLimitInterval to 30s and Burst to 5000 for most servers. This allows up to 5,000 messages every 30 seconds—enough for bursts during startup or log rotation, but not enough for a runaway process to fill the disk quickly.

# /etc/systemd/journald.conf
RateLimitInterval=30
RateLimitBurst=5000

After changing the file, restart systemd-journald:

systemctl restart systemd-journald

Then verify the settings are active:

systemctl show systemd-journald --property=RateLimitInterval --property=RateLimitBurst

You should see the values you just set. If not, double-check for typos or syntax errors in the config file.

Monitoring for Log Floods

Even with rate limiting in place, I monitor journal growth to catch issues early. I set up a simple check using journalctl --disk-usage in a cron script that alerts if usage exceeds 80% of the allocated limit. The max journal size is controlled by SystemMaxUse, which defaults to 10% of the filesystem but can be adjusted.

Here’s a snippet I use in a monitoring script:

#!/bin/bash
USAGE=$(journalctl --disk-usage | awk '/total/ {print $3}' | tr -d 'MB')
LIMIT=$(grep SystemMaxUse /etc/systemd/journald.conf | cut -d'=' -f2 | tr -d 'M')
if [ -z "$LIMIT" ]; then
  LIMIT=1024  # default 10% of 10G disk as example
fi
PERCENT=$((USAGE * 100 / LIMIT))
if [ $PERCENT -gt 80 ]; then
  echo "WARNING: Journal usage at ${PERCENT}%" | systemd-cat -t journald-check -p warning
fi

This runs hourly and logs a warning to the journal itself if usage gets high. You could extend it to send email or trigger a monitoring alert.

When to Adjust vs. Fix the Source

Rate limiting is a safety net, not a fix for noisy logging. If you’re constantly hitting the burst limit, ask why. Is a service debugging at INFO level in production? Is a cron job failing and logging the same error every minute? In those cases, reducing log verbosity or fixing the underlying issue is better than just raising the limits.

I’ve seen teams increase Burst to 20000 just to silence alerts, only to fill their disk days later when the real problem worsened. Treat rate limiting as a buffer, not a license to ignore log noise.

Conclusion

Tuning RateLimitInterval and Burst in systemd-journald.conf gives you a practical way to absorb log spikes without risking disk exhaustion. Combined with monitoring and a habit of checking journalctl --disk-usage, it’s a low-effort, high-impact control for any Linux server running systemd.

As I mentioned before in my post on auditd for /etc/passwd monitoring, proactive log hygiene prevents bigger issues down the line. Start with conservative limits, monitor usage, and adjust only when you understand the source of the log flood.

Cover image: Unknown · CC0 (Openverse / kamu malı) · https://www.rawpixel.com/image/5906903/photo-image-background-public-domain-technology