Seeing Inside a Black Box
Imagine you have a sealed appliance that is making a strange noise. You can't open it, but you can hold a stethoscope to it and hear exactly what's happening inside. strace is that stethoscope for processes. It intercepts and records every system call — every request the application makes to the operating system kernel — in real time.
System calls are the boundary between user-space code and the kernel. Every file open, network connection, memory allocation, and thread synchronization goes through this boundary. By watching system calls, you can understand exactly what a process is doing even without access to its source code or debug symbols.
Attaching to a Running Process
strace can attach to an already-running process by PID, optionally timestamp each call to spot slow ones, or filter the output down to just network-related or file-related syscalls — narrowing a wall of syscall noise to the category you actually suspect.
The Three Patterns That Appear in Every Debugging Interview
Pattern 1: Stuck on connect() — Network Timeout
connect(5, {sa_family=AF_INET, sin_port=htons(5432), sin_addr=inet_addr("10.0.1.50")}, 16
The call has no return value yet — it is hanging. The application is trying to reach a database at 10.0.1.50:5432 (PostgreSQL's default port) and getting no response. The database is down, unreachable, or a firewall is silently dropping packets.
Pattern 2: Looping on stat() or openat() — Missing Config File
openat(AT_FDCWD, "/etc/myapp/config.yaml", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/myapp/config.yaml", O_RDONLY) = -1 ENOENT (No such file or directory)
The app is retrying a missing file in a tight loop. ENOENT means "Error: No Entry" — the file does not exist. The fix is to create the file or correct the path.
Pattern 3: Stuck on futex() — Deadlock
futex(0x7f8a4c001234, FUTEX_WAIT_PRIVATE, 1, NULL
futex is the kernel primitive underlying mutexes and condition variables. A process stuck here is waiting for a lock that another thread holds — and that other thread may be waiting for a lock this thread holds. This is a deadlock. No amount of waiting will resolve it.
A common interview question is: "An application is consuming 0% CPU but is completely unresponsive. How do you debug it?" The answer that impresses: "I'd attach strace -p <PID> and look at the blocking syscall. If it's futex, I suspect a deadlock and would use gdb or /proc/<PID>/wchan to confirm. If it's connect() or read() on a socket, I suspect a network dependency that's unreachable. The syscall tells me which layer to investigate next."