The Wall of Fire
Raw profiling data — millions of stack samples — is useless without a good visualization. The standard visualization for profiling data is the Flamegraph, invented by Brendan Gregg at Netflix. Once you know how to read one, you will never look at performance data the same way.
A flamegraph is built by aggregating all the stack samples collected during a time window. If the profiler took 10,000 samples and 6,000 of them had processRequest somewhere in the stack, then processRequest gets 60% of the horizontal width.
The three rules of flamegraphs:
-
X-Axis (Width) = Resource consumption. This is the most important rule and the most commonly misunderstood. The x-axis does NOT represent time passing left to right. It represents the proportion of total samples in which that function appeared. A wide bar means that function was on the CPU a lot. A narrow bar means it was fast. Width = cost.
-
Y-Axis (Height) = Call stack depth. The bottom of the graph is the entry point (e.g.,
mainor the thread start function). Each layer above represents a function called by the layer below. IfAcallsBwhich callsC, you see A at the bottom, B in the middle, C at the top. -
Color = Category, not severity. In most tools, color is used to distinguish packages or languages (e.g., Go runtime functions in blue, your application code in green, kernel functions in red). A red bar is not necessarily bad — it just means "kernel code."
The diagnostic rule: Find the widest bar that has no wide children. That is your bottleneck. It is spending time doing work itself (high "self-time") rather than delegating to sub-functions. This is where you focus your optimization effort.