I’ve seen it too many times: a misbehaving service starts spamming the journal, and suddenly the root partition hits 100%. Services crash, monitoring alerts flood in, and you’re scrambling to free space before users notice. The good news is systemd-journald includes built-in rate limiting to stop this exact scenario. By configuring how much a single service can log over time, you protect disk space without losing visibility into genuine issues.
Why Rate Limiting Matters in journald
Journald collects logs from all services and stores them in binary format under /var/log/journal/. By default, it will keep writing until it hits a disk usage limit (usually 10% of the partition or 4GB, whichever is smaller). A single faulty service—think a tight loop in a cron job or a misconfigured application—can blow through that allocation in minutes. I once had a DNS resolver logging every query at debug level fill 20GB in under an hour on a busy recursive server. The system didn’t crash immediately, but log rotation stalled, backups failed, and I nearly missed a real security alert buried in the noise.
Rate limiting isn’t about hiding problems; it’s about preventing the logging mechanism itself from becoming a denial-of-service vector. You still get the first N messages to diagnose the issue, but after that, journald starts dropping excess entries and periodically summarizes what was skipped.
Configuring Rate Limits per Service
The key settings live in /etc/systemd/journald.conf or drop-in files under /etc/systemd/journald.conf.d/. You don’t need to edit the main file; I prefer creating a custom drop-in to avoid overriding vendor defaults.
First, check current settings:
systemd-analyze cat-config systemd/journald.confLook for RateLimitInterval and RateLimitBurst. These define the window (in seconds) and maximum number of messages allowed in that window before rate limiting kicks in.
To apply strict limits globally, create a drop-in:
mkdir -p /etc/systemd/journald.conf.dThen create /etc/systemd/journald.conf.d/rate-limit.conf with:
[Journal]
RateLimitInterval=30s
RateLimitBurst=1000This means: no more than 1000 log entries per 30 seconds from any single source. If a service exceeds that, journald will suppress further messages until the interval resets, then emit a line like:
Message suppressed, or lost due to rate limitingYou’ll see this in the journal when you query it, so you’re not flying blind.
Tuning for High-Volume but Legitimate Services
Some services naturally generate high log volume—think proxies, DNS resolvers, or batch processors. Applying a global 1000/30s limit might hide useful data there. Instead, you can use journalctl filters with _SYSTEMD_UNIT to set per-unit overrides via drop-ins, but journald doesn’t support per-unit rate limits directly in its config.
What I do instead is adjust logging verbosity at the service level where possible. For example, if nginx is too chatty, I lower its access_log or error_log level. If that’s not enough, I rely on global rate limiting as a safety net and increase the burst slightly for known busy units—say 5000/30s for a caching proxy—while keeping the global default lower.
After changing journald.conf, reload the daemon:
systemctl restart systemd-journaldVerify it’s active:
systemctl status systemd-journaldThen test with a logger spam command (in a screen or tmux):
for i in {1..2000}; do logger -t ratelimit-test "This is test message $i"; doneCheck the journal afterward:
journalctl -t ratelimit-test | tail -5You should see the first burst of messages, then suppression notices.
Monitoring Rate Limit Events
You don’t want to discover rate limiting only when something breaks. I include a simple check in my monitoring:
journalctl _TRANSPORT=journal | grep -c "suppressed" || echo 0If that counter starts rising, it means something is hitting the limits. Pair it with a per-unit breakdown to find the culprit:
journalctl | grep "suppressed" | awk -F'[' '{print $2}' | awk -F']' '{print $1}' | sort | uniq -c | sort -nr | head -5This shows which units are generating the most suppressed messages. From there, you can decide: fix the service, adjust its logging, or tune the rate limit if the volume is legitimate but bursty.
Balancing Safety and Visibility
Rate limiting is a last line of defense, not a substitute for proper logging hygiene. I still advocate for:
- Setting appropriate log levels in services (avoid debug in production)
- Using logrotate for traditional logs where applicable
- Centralizing logs to a remote syslog or Loki/Graylog for retention
But as a safety net, journald’s rate limiting has saved me more than once. It’s lightweight, requires no extra agents, and works even if the system is otherwise unresponsive due to I/O pressure.
If you’re running Linux servers—especially those handling public traffic or automated workflows—spend ten minutes checking your journald config. Set a sensible burst and interval, monitor for suppression events, and let the kernel do the throttling so you don’t have to.
As I mentioned before in my guide on kernel CPU troubleshooting, sometimes the best fix is preventing the symptom from overwhelming the system in the first place.