Tracing a Packet End-to-End
Understanding the full packet journey is the single most useful mental model for debugging Kubernetes networking. Let's trace a request from Pod A (on Node 1) to Service B, whose backing Pod B lives on Node 2.
Step 1 — Pod A sends a packet to the Service IP
Pod A's application calls http://service-b:80. DNS resolves this to the ClusterIP, say 10.96.5.10. Pod A's kernel sends a TCP SYN packet with destination 10.96.5.10:80.
Step 2 — The packet leaves Pod A's network namespace
Pod A lives in its own network namespace with a virtual ethernet interface (eth0). This is connected to a veth pair — think of it as a virtual patch cable. One end is inside the Pod namespace (eth0), the other end (vethXXXXXX) is in the Node's root network namespace. The packet crosses this boundary.
Step 3 — The kernel intercepts the packet (Netfilter PREROUTING)
The packet enters the Node's networking stack. Before the kernel decides where to route it, it passes through the PREROUTING hook in Netfilter. kube-proxy has placed a rule here: "If destination is 10.96.5.10:80, jump to chain KUBE-SVC-XXXXXXXX."
Step 4 — DNAT rewrites the destination
Inside the Service chain, a rule fires and rewrites the destination IP from 10.96.5.10 to 10.244.2.5 (Pod B's real IP on Node 2). The source IP remains Pod A's IP. The packet is now destined for a real Pod.
Step 5 — The routing table decides the next hop
The kernel consults its routing table. 10.244.2.5 is not local — it belongs to Node 2's Pod CIDR. The routing table points this traffic at the CNI interface (e.g., cni0 or flannel.1).
Step 6 — CNI encapsulates the packet
The CNI plugin (Flannel, Calico, Cilium, etc.) takes the packet and wraps it for transit across the physical network. With Flannel using VXLAN, the original packet becomes the payload of a UDP packet sent to Node 2's physical IP on port 4789. This is called encapsulation — hiding the Pod-network packet inside a node-network packet.
Step 7 — Node 2 receives and decapsulates
Node 2's CNI interface receives the UDP packet, strips the outer wrapper, and recovers the original packet destined for 10.244.2.5.
Step 8 — Delivery to Pod B
Node 2's routing table knows that 10.244.2.5 is reachable via the veth pair connected to Pod B's namespace. The packet is delivered. Pod B sees a connection from Pod A's IP.
The return path is symmetric: Pod B responds to Pod A's IP, the packet traverses the CNI tunnel back to Node 1, and Netfilter's connection tracking (conntrack) automatically reverses the DNAT — rewriting the source IP from 10.244.2.5 back to 10.96.5.10 so Pod A sees the reply as coming from the Service IP it originally contacted.
A common interview question is: "Why does Pod A see the reply as coming from the Service IP rather than the Pod IP?" The answer is conntrack (connection tracking). Linux Netfilter tracks every NATted connection in a state table. When the reply packet arrives at Node 1, Netfilter looks up the connection in the conntrack table and automatically reverses the DNAT — rewriting the source IP from the Pod's IP back to the Service's ClusterIP. This is transparent to the application. If conntrack entries are exhausted (a real production failure mode), NAT breaks silently.