Fitting a Box Inside a Box
This is the single most common cause of broken overlay networks in production, and it produces one of the most frustrating failure modes in networking: large transfers fail silently while small packets work perfectly.
Understanding MTU:
MTU (Maximum Transmission Unit) is the largest single packet that a network link will carry. Standard Ethernet has an MTU of 1500 bytes. If a packet is larger than the MTU, the network either fragments it (breaks it into smaller pieces) or drops it — depending on whether the "Don't Fragment" (DF) bit is set.
The encapsulation overhead:
Every layer of VXLAN wrapping adds bytes to the packet:
Outer Ethernet header: 14 bytes
Outer IP header: 20 bytes
Outer UDP header: 8 bytes
VXLAN header: 8 bytes
─────────────────────────────────
Total overhead: 50 bytes
GENEVE adds slightly more due to its base header (8 bytes) plus any TLV options — typically 54–70 bytes of overhead depending on options used.
The failure scenario:
VM sends packet: 1500 bytes (fills the standard MTU)
VTEP adds VXLAN headers: + 50 bytes
Resulting packet: 1550 bytes
Physical network MTU: 1500 bytes
Result: PACKET DROPPED ✗
Why ping works but downloads don't:
A standard ICMP ping sends 64-byte packets. Even with 50 bytes of overhead, the result is 114 bytes — well under 1500. The ping succeeds. But when an application sends a full-sized TCP segment (1460 bytes of payload + 40 bytes of TCP/IP headers = 1500 bytes), the encapsulated result is 1550 bytes and gets dropped.
TCP has a mechanism called Path MTU Discovery (PMTUD) that is supposed to detect this and reduce packet sizes automatically. However, PMTUD relies on ICMP "Fragmentation Needed" messages being returned to the sender. In many cloud environments, these ICMP messages are blocked by firewalls, causing PMTUD to fail silently. The connection appears to hang — the TCP handshake completes (small packets), but data transfer never starts (large packets dropped).
Key Point: MTU mismatch from encapsulation overhead causes silent packet loss for large transfers while small packets (like pings) succeed. This is the classic "it works but it doesn't" overlay networking failure mode. Always verify MTU configuration when deploying any overlay network.