When You Need to See the Actual Packets
Sometimes you need to go deeper than DNS configuration — you need to see the actual bytes on the wire. This is useful for:
- Confirming whether a DNS query is actually being sent (versus failing before it leaves the pod).
- Inspecting TLS handshake failures.
- Verifying that traffic is reaching the correct destination IP.
- Debugging service mesh mTLS issues.
tcpdump is the standard tool for packet capture. The challenge in Kubernetes is that your application container likely doesn't have it installed, and even if it did, you'd need elevated privileges to capture packets.
Using an Ephemeral Container for Packet Capture
The ephemeral container approach works here too, but you need a container with tcpdump and the NET_ADMIN capability — attaching a debug container built on an image like nicolaka/netshoot, which is a purpose-built network debugging image that includes tcpdump, dig, nslookup, curl, netstat, ss, iperf3, and dozens of other tools.
Once inside, tcpdump can filter by port (to capture DNS traffic on UDP 53), by host (to capture traffic to a specific service IP), or write the capture to a file for later analysis in Wireshark.
Because the ephemeral container shares the network namespace of the pod (not just the PID namespace), tcpdump sees the same network interface as the application container. You are capturing the actual traffic the application sends and receives.
Interpreting DNS Capture Output
A healthy DNS exchange shows a query going out and a matching answer coming back with a resolved IP. A DNS loop instead shows the query going out and a SERVFAIL response coming back with no answer. A SERVFAIL response from CoreDNS with no upstream answer is a strong indicator of a forwarding loop or unreachable upstream resolver.