Furkan İkkan Türkçe
Back to posts
Post

· Network & System Administrator

Prevent journald Disk Full with RateLimitInterval and Burst Settings

Learn how to control systemd-journald disk usage using RateLimitInterval and Burst to avoid log floods filling your root filesystem.

Sistemsystemd-journaldlog managementdisk spaceRateLimitInterval

I’ve hit the wall more than once where a misbehaving service starts spamming the journal and suddenly /var/log/journal is eating gigabytes in minutes. Left unchecked, this can fill your root filesystem and take down services. In my environment, I rely on systemd-journald’s built-in rate limiting to keep log volumes predictable. It’s not about silencing logs — it’s about preventing denial-of-service from your own logging subsystem.

Understanding journald Rate Limiting

systemd-journald includes two key directives for throttling excessive log entries: RateLimitInterval and RateLimitBurst. Together, they define how many messages are allowed within a time window before further messages are dropped (not logged, but counted). This prevents runaway logging from consuming disk space while still giving you visibility into the fact that throttling occurred.

By default, journald allows up to 1000 messages per 30 seconds. If a service exceeds that, excess messages are ignored until the interval resets. You’ll see a notice like "journal has been rotated due to exceeding rate limit" when this happens — a signal that something is misbehaving.

Configuring RateLimitInterval and Burst

You adjust these settings in /etc/systemd/journald.conf or via a drop-in file under /etc/systemd/journald.conf.d/. I prefer drop-ins so upgrades don’t overwrite custom values.

Example: limiting to 200 messages every 60 seconds:

# /etc/systemd/journald.conf.d/rate-limit.conf
[Journal]
RateLimitInterval=60s
RateLimitBurst=200

After changing, reload systemd-journald:

sudo systemctl restart systemd-journald

Then verify the settings are active:

sudo systemctl show systemd-journald --property=RateLimitInterval --property=RateLimitBurst

Output should reflect your new values:

RateLimitInterval=60
RateLimitBurst=200

When to Tighten or Relax Limits

I start conservative on production systems — especially containers or VMs with limited /var space. For noisy applications (like debug-heavy Java apps or misconfigured cron jobs), I’ll first investigate the source. But if fixing the app isn’t immediate, rate limiting buys time.

Useful scenarios for tuning:

  • Systems with small /var partitions (common in cloud instances)
  • Ephemeral hosts where logs aren’t preserved long-term
  • High-frequency cron jobs that log to stdout/stderr
  • Containerd or Docker daemons generating repetitive health-check logs

Conversely, on debug servers or when chasing intermittent issues, I may temporarily disable rate limiting:

RateLimitInterval=0

Setting interval to 0 disables throttling — but only do this short-term and with monitoring.

Monitoring for Rate Limiting Events

You won’t see dropped logs in the journal, but journald logs its own activity. Check for rate limit hits with:

sudo journalctl -u systemd-journald | grep -i "rate limit"

Look for lines like:

systemd-journald[123]: Journal started
systemd-journald[456]: Suppressed 1245 messages from /system.slice/myapp.service

That "Suppressed" count tells you how many messages were dropped in the current boot. If this number keeps growing, your rate limit may be too tight or the underlying issue needs fixing.

Pair this with disk usage checks:

sudo journalctl --disk-usage

If usage is creeping up despite rate limiting, consider adjusting Storage= or SystemMaxUse= in journald.conf to cap total journal size.

Combining with Other Protections

Rate limiting isn’t a substitute for proper log rotation or remote shipping — but it’s a critical first line of defense. In my setup, I combine it with:

  • Forwarding critical logs to a central syslog server via ForwardToSyslog=yes
  • Setting SystemMaxUse=50M to bound journal size
  • Using systemd slices (as I covered earlier) to isolate IO for logging-heavy services

As I mentioned before in my systemd slice guide, isolating logging pressure helps prevent one noisy service from starving others of disk bandwidth.

Final Thoughts

You don’t need complex tools to stop journald from filling your disk. RateLimitInterval and RateLimitBurst are simple, effective knobs built into systemd. Tune them based on your service’s normal log volume, monitor for suppression events, and pair them with size caps.

It’s one of those small config tweaks that saves you from a 3 a.m. pager — because the last thing you want is your monitoring system failing because /var/log/journal took over the root partition.

Cover image: Anita Peeples · CC0 (Openverse / kamu malı) · https://stocksnap.io/photo/guy-man-DO1KKRFQMB