Catch systemd Service Memory Leaks Early with MemoryUsage and WatchdogSec

I’ve seen too many production incidents where a slow memory leak in a systemd service goes unnoticed until the OOM killer finally slams it down. By then, users are already affected, logs are flooded, and you’re scrambling to restart the service. In my environment, I rely on two simple but powerful systemd directives: MemoryUsage= and WatchdogSec=. Together, they give you early visibility into creeping memory growth and let you act before the kernel steps in.

The core idea is straightforward: use MemoryUsage= to define a threshold that triggers a warning when exceeded, and pair it with WatchdogSec= to automatically restart the service if it doesn’t check in within a set interval. This isn’t about replacing monitoring tools — it’s about adding a lightweight, kernel-enforced safety net directly in the unit file.

How MemoryUsage= Works for Early Detection

MemoryUsage= doesn’t just limit memory — it can also generate a journal entry when a service crosses a defined threshold. This is perfect for spotting leaks early. For example, if a service normally uses 200MB but starts creeping toward 500MB, you can set a warning at 400MB.

Here’s what I typically add to a service unit file:

[Service]
MemoryUsage=400M

When the service exceeds 400MB, systemd logs a message like:

systemd[1]: myservice.service: Memory usage exceeded limit: 450.3M/400M

This appears in journalctl -u myservice and can be picked up by your log aggregator. I route these to a dedicated alerting rule in Grafana or Loki so I get notified before performance degrades. It’s not a hard limit — the service keeps running — but it gives you a heads-up.

Pairing with WatchdogSec= for Automatic Recovery

Now, combine that with WatchdogSec=. This directive expects the service to periodically notify systemd that it’s alive. If it doesn’t check in within the specified time, systemd assumes it’s hung and restarts it.

But here’s the trick: I don’t use WatchdogSec= just for hung services. I use it as a reset mechanism when memory grows too high. Even if the service is still responsive, a leak means it’s unhealthy. So I set a tight watchdog interval and rely on the service to reset its own watchdog flag only after garbage collection or cleanup cycles.

For example:

[Service]
WatchdogSec=30

The service must call sd_notify("WATCHDOG=1") at least every 30 seconds. If memory grows and the service starts slowing down or skipping checks, the watchdog triggers a restart.

Practical Example: A Java Service with a Slow Leak

I had a Java-based API service that leaked about 5MB per hour due to unresolved thread locals. Normally resting at 300MB, it would hit 1.2GB over four days — close to the OOM threshold.

I configured:

[Service]
MemoryUsage=800M
WatchdogSec=60

At 800MB, I got an early journal warning. But more importantly, as the leak worsened, the JVM’s garbage collection took longer, causing watchdog misses. After two missed check-ins, systemd restarted the service automatically — clearing the leak and restoring normal operation.

I verified the behavior with:

# Trigger a manual warning test (if supported)
systemctl kill -s SIGWARNING myservice

# Check journal for memory events
journalctl -u myservice -b | grep -i "memory usage exceeded"

# Verify watchdog is active
systemctl show myservice -p WatchdogTimestamp

Tips for Safe Rollout

Start with monitoring-only mode. Set MemoryUsage= to a high value (e.g., 150% of normal baseline) and watch for triggers in journals over a few days. Once you’re confident, lower it to a true early-warning threshold.

For WatchdogSec=, ensure your service actually implements watchdog notifications. Many modern languages have libraries: sd_notify in C, systemd-daemon in Go, or third-party jars for Java. If your service doesn’t support it, consider a sidecar adapter — but don’t fake it.

Also, avoid setting MemoryUsage= too low. You want to catch leaks, not restart services during normal spikes. I usually baseline with:

# Get peak memory over last 24h
journalctl -u myservice | grep -i "memory\|rss\|vmrss" | awk '{print $NF}' | sort -n | tail -5

When Not to Use This Combo

This approach works best for long-running services where memory growth is gradual. It’s less useful for short-lived or batch jobs. And it’s not a substitute for proper profiling or fixing the root cause — it’s a safety net.

I still use systemd-oomd and adjust OOMScoreAdjust for critical services, as I covered in my post on OOM scoring. But for services where a restart is acceptable and leaks are slow, MemoryUsage= and WatchdogSec= give you visibility and autonomy you don’t get from external monitors alone.

Conclusion

You don’t need complex tools to catch memory leaks early. By leveraging built-in systemd features, you get journal-based warnings and automatic recovery with just two lines in a unit file. It’s low-overhead, reliable, and deeply integrated into the init system.

If you’re running services on Linux and want to reduce surprise OOM kills, start here. Add MemoryUsage= for early detection, pair it with WatchdogSec= for automated cleanup, and let systemd do the heavy lifting — before the OOM killer has to.

Cover image: USDAgov · PDM (Openverse / kamu malı) · https://www.flickr.com/photos/41284017@N08/7644752188