The "Distroless" Problem
A traditional Docker image is built on a base like ubuntu:22.04 or debian:slim. These include a shell (/bin/sh), standard Unix utilities (ls, cat, ps, curl), and a package manager. This is convenient for debugging but also means your production container ships with hundreds of tools an attacker could use if they gain code execution.
Distroless images (pioneered by Google, now widely adopted) strip all of that out. The image contains only the application binary and its runtime dependencies — nothing else. A Go binary compiled with CGO_ENABLED=0 might produce a container with literally one file in it. There is no shell to exec into.
This is excellent security hygiene. It is also a debugging nightmare.
The Solution: Ephemeral Containers
Kubernetes 1.23+ (stable) introduced Ephemeral Containers — a mechanism to inject a temporary, diagnostic container into a running Pod without restarting it or modifying its spec.
An ephemeral container is fundamentally different from a regular container:
| Property | Regular Container | Ephemeral Container |
|---|---|---|
| Defined in Pod spec | Yes | No |
| Persists across restarts | Yes | No |
| Can share PID namespace | Configurable | Yes (via --target) |
| Has resource limits | Yes | Optional |
Appears in kubectl get pods |
Yes | No (use kubectl describe) |
The key capability is namespace sharing. Linux containers are isolated using kernel namespaces — separate views of the process tree (PID), network stack (NET), filesystem (MNT), and more. When you use --target=<container-name>, the ephemeral container joins the PID namespace of the target container. This means:
ps auxinside the debug container shows the target container's processes./proc/<pid>/rootgives you a view into the target container's filesystem, even though the debug container has no access to those files directly./proc/<pid>/environshows the target container's environment variables./proc/<pid>/fdshows open file descriptors.
Running kubectl debug
kubectl debug attaches an ephemeral container to a running Pod, targeting a specific container by name so its PID namespace is shared — a lightweight busybox image works for basic inspection, while a richer image like nicolaka/netshoot adds full network tooling. A --copy-to variant instead creates a full copy of the Pod with a modified image, useful when you need to change the entrypoint entirely.
Once inside the ephemeral container, you can inspect the target process: list its running processes, browse its open file descriptors and filesystem via /proc/<pid>/, and read its environment variables — all without the target image needing a shell of its own.
Debugging Nodes Directly
kubectl debug also works on nodes, not just pods. Targeting a node creates a privileged pod on it with access to the host filesystem, mounted at /host. You can inspect kubelet logs, container runtime state, and kernel parameters without SSH access to the node.
Interviewers frequently ask: "How do you debug a pod running a distroless image?" A weak answer is "rebuild the image with debugging tools." The correct answer demonstrates knowledge of ephemeral containers: use kubectl debug -it <pod> --image=busybox --target=<container> to attach a debug sidecar that shares the PID namespace of the target. This avoids redeployment, preserves the running state of the failing process, and doesn't compromise the security posture of the production image. Follow up by explaining that --target enables PID namespace sharing, which is what makes /proc/<pid>/root filesystem access possible.