Linux High CPU Usage: Kernel Space Troubleshooting Guide

When top shows 90% CPU usage but no user process is responsible, you're dealing with kernel-space CPU consumption. In my environment, this happens more often than people think — softirq storms, lock contention, I/O wait, or a driver bug silently burning cycles. Linux high CPU usage kernel troubleshooting is about looking past the process list and profiling where the kernel itself is spending time. Here's how I approach it when the usual suspects come back clean.

Why top and htop Can Mislead You

The classic mistake: you open top, sort by CPU, and see a bunch of processes at 1-2%. But the system is at 95% load and SSH feels like dial-up. The problem is that top shows user-space CPU by default. Kernel time — the time spent in syscalls, interrupt handling, softirq processing, and scheduler overhead — often doesn't get attributed to a single process cleanly.

Look at the %Cpu(s) line in top. If you see sy (system) at 60% and us (user) at 5%, that's your red flag. The kernel is doing the heavy lifting, not your application.

%Cpu(s):  3.2 us, 71.5 sy,  0.0 ni, 12.1 id, 13.2 wa,  0.0 hi,  0.0 si,  0.0 st

In this output, sy at 71.5% tells me the kernel is burning CPU. wa at 13.2% means I/O wait is also contributing. No single process will explain this.

Check softirq and Hardware Interrupts First

Softirqs are deferred interrupt handlers. Network packet processing (NET_RX), timer ticks, and block I/O completion all run here. On busy e-commerce nodes with high network throughput, I've seen ksoftirqd eating an entire core because the NIC wasn't distributing interrupts across CPUs.

Start with:

cat /proc/interrupts
watch -n1 "cat /proc/softirqs"

If you see NET_RX climbing fast on a single CPU while others sit idle, you've got an interrupt affinity problem. Check whether RSS (Receive Side Scaling) is enabled on your NIC:

ethtool -l eth0
ethtool -L eth0 combined 4

This spreads receive interrupts across multiple queues. On physical servers, I also pin IRQs using irqbalance or manual /proc/irq/*/smp_affinity masks. As I mentioned before in my network troubleshooting post (https://furkanikkan.com/urun/ag-yavasliginin-gercek-nedenini-bulmanin-7-yolu-50), the root cause often isn't where you expect — and interrupt distribution is one of those silent killers.

Profile the Kernel with perf When Nothing Else Talks

When softirq and interrupt checks come back normal, I go straight to perf. This is the tool that tells you exactly which kernel function is eating CPU. No guessing.

perf record -a -g -- sleep 30
perf report

The -g flag captures call graphs so you can trace from the top-level kernel function down to the leaf. What you're looking for:

• Functions like __do_softirq, net_rx_action, or tcp_v4_rcv → network stack pressure
• __blk_run_queue, blk_update_request → block I/O layer
• __schedule, try_to_wake_up → scheduler contention or excessive context switching
• futex_wait_queue_me, rwsem_down_read_slowpath → lock contention

A quick one-liner I use when I don't want the full report:

perf top

This gives a live, updating view of kernel functions by CPU usage — like top but for kernel internals.

I/O Wait Disguised as Kernel CPU

Here's one that bit me last year. A Proxmox node was showing 40% sy and 30% wa in top. Every process looked fine. Turns out the ZFS ARC was thrashing because a backup job was reading cold data from spinning disks, and the kernel was spending all its time in the block layer waiting for I/O completion.

The key diagnostic:

iostat -xz 1

Look for %util near 100 on a specific device and high await values. If await is above 20-30ms on SSDs or above 100ms on HDDs, the kernel is blocking on I/O and burning CPU cycles managing that wait queue.

Other tools I check:

• vmstat 1 — if the r column (runnable processes) is high but CPU isn't 100% user, the kernel is context-switching heavily
• sar -w 1 — context switch rate; anything above 50,000/sec on a single CPU is worth investigating
• pidstat -w 1 — shows per-process context switches, helps identify the offender

Warning: I/O wait can show up as both wa and sy depending on kernel version and how the scheduler accounts time. Don't assume wa is the only indicator.

Lock Contention and Scheduler Overhead

Sometimes the kernel burns CPU just managing itself. Lock contention happens when multiple CPUs fight for the same spinlock or rwsem. I've seen this on busy database servers with high syscall rates and on systems running older kernels with known scheduler bugs.

perf will show functions like _raw_spin_lock, queued_spin_lock_slowpath, or rwsem_down_write_slowpath near the top. When that happens:

1. Check your kernel version — older 4.x kernels had known spinlock issues under high network load
2. Look at sysctl kernel.sched_migration_cost_ns — lowering it can reduce scheduler overhead on NUMA systems
3. Use perf sched to profile scheduler behavior specifically:

perf sched record -- sleep 10
perf sched latency --max

This shows you which tasks have the highest scheduling latency and which CPUs are overloaded from a scheduler perspective.

Ftrace for Targeted Kernel Tracing

When perf gives you the function name but you need to see when and how often it's called, ftrace is the next step. It's built into the kernel — no package needed.

echo 1 > /sys/kernel/debug/tracing/events/sched/sched_switch/enable
cat /sys/kernel/debug/tracing/trace_pipe | head -50

This streams context switch events in real time. If you see thousands of switches per second between two specific processes, you've found a scheduling storm.

For function-level tracing:

echo function > /sys/kernel/debug/tracing/current_tracer
echo tcp_v4_rcv > /sys/kernel/debug/tracing/set_ftrace_filter
echo 1 > /sys/kernel/debug/tracing/tracing_on

Now every call to tcp_v4_rcv gets logged with a timestamp. I used this once to track down a SYN flood that was causing kernel CPU spikes without any user-space process involvement.

Quick Diagnostic Checklist

When I get paged for high CPU and the process list looks clean, here's my sequence:

• Check top %Cpu(s) line — is sy or wa high?
• Run cat /proc/softirqs and cat /proc/interrupts — any single CPU overloaded?
• perf top for 30 seconds — what kernel function dominates?
• iostat -xz 1 — is a disk at 100% util?
• vmstat 1 — high context switches or runnable processes?
• dmesg | tail — any kernel warnings or errors in the last hour?

That last one sounds obvious, but I've solved mysteries with a single dmesg output showing a driver repeatedly resetting a failed NIC. The kernel was spending CPU on error handling and recovery loops that never showed up in the process list.

Kernel-space CPU issues are tricky because the standard tools are built around processes. Once you shift to profiling the kernel itself with perf, ftrace, and interrupt analysis, the root cause usually surfaces within minutes. The hard part is remembering to look there in the first place.

Cover image: Operate, Defend, Attack, Influence! · PDM (Openverse / kamu malı) · https://www.flickr.com/photos/131622585@N06/48986472741