Two Worlds, One Network
To troubleshoot modern cloud and Kubernetes networking, you must be able to mentally switch between two completely separate network realities that coexist on the same physical hardware.
The Underlay (Physical World)
The underlay is the physical infrastructure: routers, switches, cables, and the IP addresses assigned to physical server NICs. It speaks standard routing protocols — typically OSPF for smaller environments or BGP for large-scale data centers. The underlay's only job is to move IP packets from one physical server to another as efficiently as possible.
Critically, the underlay is completely unaware of virtual machines, pods, or overlay networks. From the underlay's perspective, a VXLAN-encapsulated packet is just a large UDP packet. It doesn't know or care what's inside.
The Overlay (Virtual World)
The overlay is the virtual network created by VXLAN or GENEVE tunnels. Virtual machines and containers live in the overlay. They have their own IP addresses (e.g., 10.0.0.5) and MAC addresses that are entirely separate from the physical server's addresses. From a VM's perspective, it's connected to a virtual Ethernet switch — it has no idea it's actually communicating through a UDP tunnel across a physical data center.
Side-by-side comparison:
| Dimension | Underlay | Overlay |
|---|---|---|
| Devices | Physical servers, switches, routers | Virtual machines, containers, pods |
| Addresses | Physical server IPs (e.g., 192.168.1.x) | Virtual VM/pod IPs (e.g., 10.0.0.x) |
| Protocol | OSPF, BGP, standard IP routing | Virtual Ethernet (VXLAN/GENEVE tunnels) |
| Awareness | Sees only outer UDP packets | Sees only inner Ethernet frames |
| Troubleshooting tool | ip route, traceroute, BGP table |
arp, bridge fdb, CNI plugin logs |
Why this matters for troubleshooting:
When a pod can't reach another pod, you need to determine which layer is broken:
- Can the physical servers reach each other? (Underlay problem)
- Are the VTEP mappings correct? (Control plane problem)
- Is the encapsulated packet being dropped? (MTU or firewall problem)
- Is the inner packet being delivered correctly? (Overlay routing problem)
Each layer requires different tools and different mental models.
Key Point: The underlay and overlay operate independently. A healthy underlay does not guarantee a healthy overlay — you must be able to diagnose each layer separately using the appropriate tools.