When memory runs low on a Linux server, the OOM killer steps in to free RAM by terminating processes. By default, it picks victims based on their OOM score, which reflects memory usage and runtime behavior. But not all processes are equal — killing a database or a web frontend can cause more harm than stopping a batch job. I’ve seen this happen on monitoring stacks where Prometheus got killed while a low-priority log collector survived, making post-mortem analysis harder.
The good news is you can influence this decision. Each process has an oom_score_adj value that shifts its likelihood of being killed. The range is from -1000 (least likely) to +1000 (most likely). Setting a negative value protects a process; a positive one makes it more expendable.
To view a process’s current OOM score and adjustment, check:
cat /proc/<PID>/oom_score
cat /proc/<PID>/oom_score_adjFor example, if nginx (PID 1234) has an oom_score of 300 and oom_score_adj of 0, its effective score is 300. If I set oom_score_adj to -500, the kernel treats it as if the score were -200 — making it very unlikely to be killed.
You can adjust this value in two ways. For a running process:
echo -500 > /proc/1234/oom_score_adjThis requires root and takes effect immediately. But it doesn’t survive a reboot. For persistence, the best method is via systemd. In your service file, add:
[Service]
OOMScoreAdjust=-500Then reload and restart:
systemctl daemon-reload
systemctl restart nginxI use this on my load balancers and API gateways — services where downtime cascades fast. Meanwhile, I give background workers or log shippers a positive adjustment, like +200, so they’re preferred targets if memory gets tight.
Important: Avoid setting oom_score_adj to -1000 unless you’re absolutely sure. That tells the kernel to never kill the process, which could lock up the system if it truly misbehaves. I once set a debug agent to -1000 and had to pull the plug when it leaked memory and left no escape hatch.
Also note: oom_score_adj doesn’t change the actual memory usage — it only changes the kill priority. A process with high RSS and a negative adjustment can still contribute to pressure; it just won’t be the first to go.
If you’re tuning a busy system, start by observing which processes get killed during stress tests. Use dmesg | grep -i kill after triggering memory pressure to see what the OOM killer picked. Then adjust accordingly.
As I mentioned before in my guide on kernel CPU debugging, understanding low-level behavior like this helps you build more resilient systems — not just react when things break.
For most production services, I start with -500 for critical paths and +100 to +300 for expendable ones. Monitor, adjust, and validate under load. It’s a small tweak, but it gives you control when the kernel starts making life-or-death decisions.
Cover image: USDAgov · PDM (Openverse / kamu malı) · https://www.flickr.com/photos/41284017@N08/7644752188