The "Overlay Network" Approach (Not EKS)
In standard Kubernetes deployments using plugins like Flannel or Calico in VXLAN mode, pods live on a virtual "overlay network" — a fake address space (e.g., 10.244.0.0/16) that is tunneled over the real network. When Pod A on Node 1 talks to Pod B on Node 2, the packet is encapsulated, sent across the real network, and decapsulated at the destination. This adds latency and makes debugging harder because your network monitoring tools see the node IPs, not the pod IPs.
The EKS Reality: Real VPC IPs
EKS uses the Amazon VPC CNI Plugin, which takes a fundamentally different approach: every pod gets a real, routable IP address from your VPC subnet. There is no overlay, no encapsulation, no tunneling.
How it works mechanically:
- When a worker node starts, the VPC CNI plugin attaches one or more Elastic Network Interfaces (ENIs) to the EC2 instance.
- Each ENI can hold multiple Secondary IP addresses (in addition to its primary IP).
- When a pod is scheduled onto the node, the CNI plugin assigns one of these pre-allocated Secondary IPs to the pod's network namespace.
- The pod's IP is a real VPC IP — it can be reached directly from anywhere in the VPC, including other services, RDS databases, and on-premises networks connected via Direct Connect.
Pros and Cons of the VPC CNI Approach
| Detail | |
|---|---|
| ✅ Performance | No encapsulation overhead. Pod-to-pod traffic is native VPC routing — as fast as EC2-to-EC2. |
| ✅ Visibility | VPC Flow Logs capture the actual pod IP, not just the node IP. Security teams can trace traffic to individual pods. |
| ✅ Native Integration | Security Groups can be applied directly to pods (not just nodes), enabling fine-grained network policy at the AWS layer. |
| ❌ IP Exhaustion | 1,000 pods = 1,000 VPC IPs consumed. Small subnets (e.g., /24 = 254 IPs) fill up fast. |
| ❌ Subnet Planning Required | You must design your VPC with large enough subnets for your expected pod density, or use separate subnets for pods. |
Senior Insight: IP exhaustion is one of the most common production incidents in EKS clusters. The fix is either to use larger subnets from the start, enable Prefix Delegation (covered next), or use a custom networking mode where pods use a separate CIDR from nodes. Plan your VPC CIDR allocation before you deploy your first cluster — retrofitting is painful.