Introduction: When "Kubectl Logs" Isn't Enough

The Limits of Standard Tools

Think of debugging a Kubernetes application like diagnosing a car problem. Most of the time, the dashboard warning light tells you exactly what's wrong — you read the code, fix the issue, and drive on. That's kubectl logs and kubectl describe. They cover the obvious 90%.

But some failures are more like a car that runs fine in the driveway but stalls on the highway. The dashboard shows nothing. You need a mechanic with specialized diagnostic equipment — an oscilloscope, a fuel pressure gauge, a live data feed from the ECU. Kubernetes has equivalents for each of these.

The remaining 10% of failures fall into three categories that standard tools cannot address:

• Distroless Images: Your container image contains only the application binary — no shell, no ls, no cat, no curl. When you run kubectl exec -it my-pod -- sh, you get executable file not found in $PATH. There is nothing to execute. This is intentional for security, but it makes interactive debugging impossible with standard tools.

• Heisenbugs (Performance Anomalies): The application is running but consuming 100% CPU or leaking memory over hours. Logs show nothing unusual because the application isn't crashing — it's just doing the wrong thing internally. You need to look inside the running process, not at its output.

• Networking Ghosts: DNS resolution is timing out intermittently. ping works. curl works sometimes. The network "seems fine." These failures are often caused by subtle misconfigurations in how DNS forwarding is chained together, and they require deep packet-level inspection to diagnose.

This lesson covers the forensic toolkit for each category: Ephemeral Containers for distroless images, pprof for performance profiling, and CoreDNS loop detection for DNS failures. Each tool requires understanding a different layer of how Kubernetes works internally.