Asking the Kernel Directly
Every time a program wants to do something outside of its own memory — read a file, send network data, allocate more RAM, create a new process — it cannot do so directly. It must ask the kernel for permission via a system call (syscall). System calls are the formal API between user-space programs and the kernel. There are roughly 300–400 of them in a modern Linux kernel, covering everything from read() and write() to clone() (create a process) and mmap() (map memory).
strace is a classic tool that intercepts and logs every single system call a process makes. It works by using the ptrace kernel facility — the same mechanism used by debuggers like gdb. Think of it as attaching a wiretap to the phone line between an application and the kernel: every request the application makes, and every response the kernel sends back, is logged.
What strace Reveals
If you run strace ls, you won't just see a file listing. You'll see hundreds of lines showing ls opening shared libraries from /lib, checking locale settings, calling mmap() to load those libraries into memory, calling getdents64() to read directory entries, and finally calling write() to send the output to your terminal. Each line shows the syscall name, its arguments, and its return value — for example, an openat() call resolving to a file descriptor: openat(AT_FDCWD, "/etc/hostname", O_RDONLY) = 3. strace can also be filtered to a single syscall (e.g., only openat calls) to cut the noise down to the events you actually care about.
This is invaluable for debugging: "Why is my app failing to start?" — run strace and look for ENOENT (file not found) or EACCES (permission denied) return codes. The answer is usually in the first few lines.
The Critical Limitation: Performance Overhead
strace works by using ptrace, which forces the kernel to pause the target process for every single system call — once before the call executes (so strace can log the arguments) and once after (so strace can log the return value). On a process making thousands of syscalls per second, this constant context-switching can slow the application down by 10x to 100x. This makes strace dangerous to use on busy production processes. It is best reserved for development, staging, or attaching briefly to a low-traffic process.
Interviewers frequently ask: "How would you debug a process that's behaving unexpectedly in production?" If you say strace, a good interviewer will push back: "What's the risk?" The correct answer is that strace uses ptrace, which introduces significant overhead by pausing the process on every syscall — potentially causing latency spikes or cascading failures in a production system. The safer production answer is eBPF-based tools (covered later in this lesson), which have near-zero overhead.