What CNI Is and Why It Exists
When Kubernetes creates a Pod, it needs to give that Pod a network interface, assign it an IP address, and ensure it can communicate with every other Pod in the cluster — even on different nodes. Kubernetes itself does not implement any of this. Instead, it defines a standard interface called the Container Network Interface (CNI), and delegates all networking setup to a plugin that implements that interface.
CNI is a specification, not a product. It defines two operations: ADD (set up networking for a new container) and DEL (tear it down). When the kubelet creates a Pod, it calls the CNI plugin's ADD command with the container's details. The plugin is responsible for:
- Creating a network interface inside the Pod's namespace (usually
eth0) - Assigning an IP address from the node's allocated CIDR range (IPAM — IP Address Management)
- Setting up routes so the Pod can reach other Pods and the outside world
- Configuring the node side of the connection (the veth pair, bridge, or overlay)
The Three CNI Architectures
Different CNI plugins use fundamentally different approaches to connect Pods across nodes:
Overlay Networks (Flannel VXLAN, Calico VXLAN)
Pod traffic is encapsulated inside UDP packets and tunneled between nodes. The physical network only sees node-to-node traffic; it has no knowledge of Pod IPs. This is the easiest to set up because it requires no changes to the underlying network. The cost is encapsulation overhead (typically 50–100 bytes per packet) and slightly higher CPU usage for encap/decap.
Underlay / BGP Routing (Calico BGP, Cilium BGP)
Pod IPs are advertised directly to the physical network using the Border Gateway Protocol (BGP). Routers in the datacenter learn that 10.244.2.0/24 is reachable via Node 2's IP. Packets travel natively without encapsulation. This is faster and easier to debug with standard network tools, but requires a network infrastructure that supports BGP peering — not always available in cloud environments.
eBPF Dataplane (Cilium)
Cilium replaces both kube-proxy and traditional CNI packet forwarding with eBPF (extended Berkeley Packet Filter) programs loaded directly into the kernel. Instead of iptables rules or IPVS tables, eBPF programs intercept packets at the lowest possible level and make forwarding decisions with minimal overhead. Cilium also enables network policy enforcement, observability (via Hubble), and service mesh features without a sidecar proxy.
CNI Configuration Files
The kubelet discovers which CNI plugin to use by reading files from /etc/cni/net.d/ on the node. A typical Flannel config names the network, declares the CNI spec version, and lists a chain of plugins to run in order — the flannel plugin itself (configured to act as the default gateway with hairpin mode enabled), followed by a portmap plugin that handles hostPort-to-podIP port mappings.
If this directory is empty or the file is malformed, every new Pod will fail to start with a CNI error — a common failure mode after node reboots or failed CNI upgrades.