The Ready-to-Use Toolbox
You don't usually write raw eBPF bytecode yourself. The BCC (BPF Compiler Collection) project provides a large library of pre-written, production-tested Python and Lua scripts that use eBPF under the hood to solve specific observability problems. On most modern Linux distributions, they're available via the bpfcc-tools package (Debian/Ubuntu) or bcc-tools (RHEL/Fedora).
These tools are designed to be safe to run in production. They have been used by Netflix, Facebook, Google, and others to debug live production systems.
The Essential BCC Tools
Process and Execution:
- execsnoop: Watches the entire system and prints a line every time any new process is executed. Invaluable for catching short-lived scripts that top never sees because they start and finish between sampling intervals. Great for detecting cryptominers or runaway restart loops.
- runqlat: Measures how long processes spend waiting in the CPU run queue before they get scheduled. High run queue latency means CPU saturation — processes are ready to run but can't get CPU time.
File System:
- opensnoop: Prints a line every time any process opens a file, showing the process name, PID, and filename. Answers "what config files is this application actually reading?" and "why is this process failing to open a file?"
- filetop: Like top, but shows which files are being read and written most frequently, in real-time.
Network:
- tcptop: Shows active TCP connections sorted by bandwidth usage. Answers "which connection is consuming all my bandwidth?"
- tcpretrans: Shows TCP retransmissions with the source/destination IP and port. Retransmissions indicate network packet loss — a common cause of latency that's invisible to application-level monitoring.
- tcpconnect / tcpaccept: Log every new outbound TCP connection or inbound accepted connection. Essential for security auditing and understanding service dependencies.
Disk I/O:
- biolatency: Measures block device I/O latency and presents it as a histogram. A histogram showing most I/Os completing in 1–5ms is healthy; one showing a long tail at 100ms+ indicates a slow disk or I/O saturation.
- biotop: Like top, but for disk I/O — shows which processes are doing the most disk reads and writes.
Each of these ships as a standalone -bpfcc-suffixed executable once the package is installed, and most accept an optional runtime duration or a filter (like a process name) to scope what they report on.
A common SRE interview scenario: "A service is slow, but CPU and memory look fine. How do you investigate?" The answer that impresses: "I'd check for I/O wait with iostat, then use biolatency to see if disk latency has a long tail, tcpretrans to check for network packet loss, and runqlat to rule out CPU scheduling delays. These BCC tools give me per-event histograms with near-zero overhead, so I can safely run them in production." This demonstrates you know which tool answers which question.