How Packets Actually Travel Between Nodes
When a Pod on Node A sends a packet to a Pod on Node B, the CNI plugin must get that packet across the physical network. There are two fundamentally different strategies for doing this.
Overlay Networks (Encapsulation)
In an overlay network, the CNI plugin wraps the original Pod-to-Pod packet inside a new packet addressed to the destination node. The physical network only ever sees node IP addresses — it has no idea Pod IPs exist.
The most common encapsulation protocols are:
- VXLAN (used by Flannel, Calico in overlay mode): Wraps packets in UDP. Works on almost any network.
- IP-in-IP (used by Calico): A lighter-weight encapsulation that wraps IP packets inside other IP packets.
The trade-off: encapsulation adds CPU overhead (wrapping and unwrapping every packet) and reduces the effective MTU (Maximum Transmission Unit — the largest packet size the network can carry). A standard Ethernet MTU is 1500 bytes; VXLAN overhead consumes 50 bytes, leaving 1450 bytes for Pod traffic. Misconfigured MTU is a common source of mysterious packet drops and connection hangs.
Direct Routing (L3 Native)
In direct routing mode, Pod packets are sent as-is. The physical network routers are taught — via BGP or static routes — that Pod subnet 10.244.1.0/24 lives on Node A, and 10.244.2.0/24 lives on Node B.
This is how Calico in BGP mode and AWS VPC CNI work. The result is native network performance with no encapsulation overhead and easy debugging with standard tools like tcpdump and traceroute.
The trade-off: your physical network infrastructure must support this. In a managed cloud VPC, the provider handles it. In a bare-metal data center, you need BGP-capable switches and routers — not always available in legacy environments.