How It Works: Sampling and Stack Traces

We Don't Watch Everything

Tracking every single instruction a CPU executes would make your application orders of magnitude slower — completely impractical for production. Instead, continuous profilers use statistical sampling, which trades perfect accuracy for negligible overhead while still producing statistically reliable results.

Here is how the sampling loop works:

1. The interrupt: The profiler wakes up at a fixed frequency — typically 100 times per second (100 Hz).
2. The snapshot: It interrupts the CPU and asks, "What function are you running right now?" It captures the full call stack (also called a stack trace): the chain of function calls that led to the current instruction, from the entry point of the program down to the currently executing function. For example: main → handleRequest → parseJSON → json.Unmarshal.
3. The aggregation: It records this stack trace and goes back to sleep. After thousands of snapshots, it counts how often each function appeared.

If the profiler takes 1,000 snapshots over 10 seconds, and 600 of them show the CPU was inside encrypt_password(), you can statistically conclude that encrypt_password() consumed approximately 60% of your CPU time during that window. The more samples you collect, the more accurate the estimate — this is the same principle behind opinion polling.

What a Stack Trace Looks Like

A raw stack trace captured during a snapshot might look like this:

main.main
  main.handleRequest
    main.processUser
      crypto.bcrypt.GenerateFromPassword
        crypto.bcrypt.expensiveHash   <-- CPU was HERE

The profiler captures thousands of these. Functions that appear frequently across many snapshots are the ones consuming the most CPU.

The Overhead Trade-off

At 100 Hz, the profiler interrupts the CPU 100 times per second. Each interrupt takes microseconds. The total overhead is typically 0.1–1% of CPU, which is why continuous profiling is safe to run permanently in production. Some profilers let you tune the sampling frequency — higher frequency means more accurate data but slightly more overhead.