Furkan İkkan Türkçe
Back to posts
Post

· Network & System Administrator

How to Sample User and Kernel Functions Separately with perf top

Learn to use perf top -u and -k to isolate user-space and kernel-space hotspots for precise performance tuning.

Sistemperfperformance tuningLinux kerneluser space

When I need to understand where CPU time is really going on a busy Linux server, perf top is one of the first tools I reach for. It gives a live, function-level view of where time is spent, but by default it mixes user-space and kernel-space samples together. That can make it hard to tell whether a slowdown is coming from my application code or from something deeper in the kernel. Using the -u and -k flags lets me separate those views and focus on exactly what I need to investigate.

Sampling Only User-Space Functions with -u

Sometimes I suspect the bottleneck is in the application itself — maybe a Java service, a Python script, or a custom binary. In those cases, I run perf top with the -u flag to sample only user-space functions. This filters out kernel activity and shows me which functions in my executable or shared libraries are consuming the most CPU.

For example, on a server running a high-traffic web app, I might see:

# perf top -u
Samples: 12K of event 'cpu-clock', Event count (approx): 12000000
Overhead  Command          Shared Object          Symbol
..      .                .                      .
5.23%   java             libjvm.so              [.] InterpretBytecode
4.87%   java             libjava.so             [.] Java_java_lang_String_indexOf
3.91%   java             [unknown]              [.] 0x00007f8d12345678
2.15%   mysqld           mysqld                 [.] sql_parse
1.89%   nginx            nginx                  [.] ngx_http_header_filter

This tells me that over 10% of user-space CPU is spent in JVM bytecode interpretation and string operations — a clue that I might need to look at application logic or JVM tuning. I’ve used this approach before when investigating high CPU in Java services, as I mentioned in my post about JVM monitoring.

Isolating Kernel-Space Activity with -k

On other occasions, the issue feels more systemic — high interrupt rates, slow disk I/O, or unexplained system load that doesn’t correlate with any specific process. That’s when I switch to perf top -k to look only at kernel functions. This removes user-space noise and highlights where the kernel itself is burning cycles.

A typical kernel-only view might look like:

# perf top -k
Samples: 8K of event 'cpu-clock', Event count (approx): 8000000
Overhead  Command          Shared Object          Symbol
..      .                .                      .
6.42%   [kernel]         [k]                    [.] _raw_spin_lock_irqsave
5.18%   [kernel]         [k]                    [.] __alloc_pages_nodemask
4.03%   [kernel]         [k]                    [.] ext4_journal_start_sb
3.71%   [kernel]         [k]                    [.] tcp_v4_rcv
2.89%   [kernel]         [k]                    [.] __x64_sys_write

Here, I see significant time spent in spin locking and memory allocation — potential signs of contention under high concurrency. If I see a lot of time in _raw_spin_lock_irqsave, I know I need to look at locking patterns in drivers or subsystems under stress. Similarly, high __alloc_pages_nodemask might point to memory pressure or fragmentation.

Combining Views for Deeper Insight

The real power comes from using both views together. I’ll often run perf top -u first to see if the hotspot is in user space. If it’s flat (no single function dominates), I’ll switch to -k to check for kernel-side issues. Sometimes the problem is a interaction — like a user-space process making frequent system calls that trigger expensive kernel paths.

For instance, if perf top -u shows high CPU in a database user process, but perf top -k shows a lot of time in ext4_journal_start_sb, I know the DB is spending time waiting for journal commits — which might lead me to check mount options, disk speed, or workload characteristics.

Practical Tips and Gotchas

  • perf top requires root or CAP_SYS_ADMIN to access kernel symbols and counters.
  • On systems with KASLR enabled, symbol resolution might be less accurate unless you have /proc/kallsyms accessible.
  • If you see a lot of [unknown] in the output, consider installing debug symbols (e.g., linux-image-*-dbg) for better resolution.
  • The default sample rate is 4000 Hz; you can adjust it with -c or -f if you need higher resolution or lower overhead.
  • Always correlate perf top findings with other tools like vmstat, iostat, or pidstat to build a full picture.

perf top -u and -k are simple flags, but they change the tool from a general CPU viewer into a focused diagnostic instrument. Whether I’m tuning a latency-sensitive service or tracking down a mysterious system load spike, being able to isolate user and kernel space at the function level has saved me hours of guesswork.

Cover image: Kevin Buehner · CC0 (Openverse / kamu malı) · https://www.thingiverse.com/thing:2655875