Profiling: X-Raying the CPU with pprof

The Problem with "It's Slow"

When a service is slow or consuming excessive CPU, logs are nearly useless. The application isn't crashing — it's working, just badly. You need to answer a specific question: which function in the code is responsible for this resource consumption?

This is what profiling does. A profiler periodically samples the call stack of a running process — essentially taking thousands of snapshots per second of "what is the CPU doing right now?" — and aggregates those snapshots into a statistical picture of where time is being spent.

pprof is the standard profiling tool for Go programs. Since Kubernetes itself (and most of its ecosystem) is written in Go, pprof is the primary tool for profiling both your applications and Kubernetes control plane components.

How pprof Works

A Go application that imports net/http/pprof automatically exposes HTTP endpoints that return profiling data on demand:

These endpoints are not enabled by default in production applications for security reasons — exposing them leaks internal code structure. However, Kubernetes control plane components (API server, scheduler, controller manager) expose them on their metrics/health ports, protected by RBAC.

Profiling a Kubernetes Component

Profiling the API server when it's consuming excessive CPU follows three steps: port-forward to the component's port on its static Pod, use curl (with the cluster's admin TLS credentials) to fetch a 30-second CPU profile from its /debug/pprof/profile endpoint into a local file, then feed that file to go tool pprof to open a browser UI with multiple views. The most useful is the Flame Graph.

Reading a Flame Graph

A flame graph is a visualization where:

• The x-axis represents the proportion of total CPU time (wider = more time spent).
• The y-axis represents the call stack depth (bottom = entry point, top = leaf function).
• Each rectangle is a function call. Its width shows how much CPU time it and its callees consumed.

You look for wide rectangles near the top — these are leaf functions consuming significant CPU with no further delegation. That's where your optimization effort should focus.

Profiling Your Own Go Application

To enable pprof in your own Go service:

import (
    _ "net/http/pprof"  // Side-effect import registers handlers
    "net/http"
)

func main() {
    // Start pprof server on a separate port (never expose to public internet)
    go func() {
        log.Println(http.ListenAndServe("localhost:6060", nil))
    }()
    // ... rest of your application
}

Then profile it the same way: port-forward and use go tool pprof.