When You Need CPU-Level Profiling
eBPF tools are excellent for event-based tracing — "every time X happens, record Y." But sometimes you need a different question answered: "Where is my CPU actually spending its time?" For this, perf (Linux Performance Events) is the standard tool.
perf works by sampling: it interrupts the CPU thousands of times per second and records what code was executing at that moment. After collecting thousands of samples, you get a statistical picture of where CPU time is being spent. Because it's sampling rather than tracing every event, the overhead is configurable and typically low (1–5%) — governed by the sampling frequency, conventionally set to a deliberately odd number like 99 Hz to avoid accidentally synchronizing with periodic timers in the application. Samples are captured with perf record and then examined as a call-graph report with perf report.
Flame Graphs: Making perf Output Human-Readable
Raw perf report output is a text table that's hard to interpret for complex call stacks. Flame graphs, invented by Brendan Gregg, transform this data into an intuitive visualization:
- The x-axis represents the proportion of time spent (wider = more time).
- The y-axis represents the call stack (bottom = kernel, top = leaf functions).
- Each rectangle is a function. Its width shows how much CPU time was spent in that function and all functions it called.
A "hot" function — one consuming lots of CPU — appears as a wide plateau near the top of the flame graph. You can immediately see which code paths are expensive.