Using systemd WatchdogSec to Auto-Recover Hung Services

I’ve seen too many production systems where a critical service hangs silently—no crash, no log spam, just stuck. Users start complaining, monitoring shows normal CPU, but the service isn’t doing its job. In my environment, I rely on systemd’s built-in WatchdogSec to catch these silent failures and auto-restart the service before impact spreads.

How WatchdogSec Works in systemd

When you enable WatchdogSec in a service unit, systemd expects the service to periodically notify it via sd_notify() with the WATCHDOG=1 signal. If the service fails to send this ping within the configured interval, systemd assumes it’s hung and takes action—usually restarting it based on Restart= settings.

This isn’t just a timeout; it’s an active liveness check. The service must be instrumented to call sd_notify()—many modern daemons like nginx, Redis, or custom Go/Python apps already support it via libraries. If yours doesn’t, you can use a wrapper like watchdog from the systemd package.

Configuring WatchdogSec: A Practical Example

Here’s how I set it up for a Python-based API service that occasionally deadlocks under load:

[Unit]
Description=Internal API Service
After=network.target

[Service]
ExecStart=/usr/local/bin/api-server
WatchdogSec=30
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

I placed this in /etc/systemd/system/api.service. The WatchdogSec=30 means systemd waits 30 seconds for a watchdog ping. If the service doesn’t check in, it’s considered unresponsive and restarted after 5 seconds.

To make the service actually send the ping, I modified the Python code to include:

import systemd.daemon

while True:
    # do work
    systemd.daemon.notify('WATCHDOG=1')
    time.sleep(25)  # ping every 25s, well under 30s limit

This keeps the watchdog happy. If the loop blocks—say, on a deadlocked DB query—the ping stops, and systemd restarts the service.

Verifying Watchdog Status

After reloading (sudo systemctl daemon-reload) and restarting the service, I check watchdog state with:

$ systemctl status api.service
● api.service - Internal API Service
   Loaded: loaded (/etc/systemd/system/api.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2024-05-20 10:15:00 UTC; 2h 30min ago
 Main PID: 1234 (api-server)
    Tasks: 8 (limit: 4915)
   Memory: 120.5M
   CGroup: /system.slice/api.service
           └─1234 /usr/local/bin/api-server

May 20 10:15:00 server systemd[1]: Started Internal API Service.
May 20 10:15:00 server systemd[1]: Watchdog enabled on api.service.

The line Watchdog enabled on api.service confirms it’s active. If the service hangs, you’ll see:

May 20 12:45:22 server systemd[1]: api.service: Watchdog timeout!
May 20 12:45:22 server systemd[1]: api.service: Main process exited, code=killed, status=5/TRAP
May 20 12:45:22 server systemd[1]: api.service: Unit entered failed state.
May 20 12:45:27 server systemd[1]: api.service: Started Internal API Service.

That automatic recovery is what makes WatchdogSec valuable—it reduces MTTR without manual intervention.

When Not to Use WatchdogSec

I’ve seen teams enable WatchdogSec on services that fork or spawn short-lived children (like some cron-like workers). If the parent exits quickly but children keep working, the watchdog may trigger falsely. In those cases, either refactor the service to keep the parent alive or use a different monitoring approach.

Also, avoid setting WatchdogSec too low. I start with 2-3x the expected longest normal operation cycle. For a service that usually responds in under 10 seconds, 30 seconds is safe. Too short, and you get needless restarts during temporary spikes.

Combining with Other systemd Features

I often pair WatchdogSec with:

• Restart=on-failure to ensure restarts only happen on actual failures
• StartLimitIntervalSec and StartLimitBurst to prevent restart loops if the service crashes immediately on start
• FailureAction=reboot in extreme cases for critical infrastructure (use with caution)

As I mentioned before in my post about isolating dangerous commands with systemd-nspawn and seccomp, layering these mechanisms builds resilience.

Real-World Impact

After adding WatchdogSec to our payment gateway service, we saw a 70% drop in PagerDuty alerts for "service unresponsive" incidents. The service still had bugs causing hangs, but systemd brought it back within seconds—often before users noticed.

It’s not a fix for the root cause, but it’s a solid safety net. Think of it like a circuit breaker: it doesn’t prevent the overload, but it stops the cascade.

If you’re running systemd services and want better self-healing, start with WatchdogSec on your most critical ones. Test it by sending SIGSTOP to the process and watch systemd bring it back.