The "ContainerCreating" Nightmare
The most common symptom of a CNI failure is a Pod stuck in ContainerCreating status indefinitely. The Pod's Events describe a sandbox creation failure — the networking plugin (e.g. flannel) could not be found in the node's CNI binary path. That specific error means the CNI binary is missing from /opt/cni/bin/ on the node — a common issue after node replacement or OS upgrades that don't reinstall CNI binaries.
Systematic Troubleshooting Steps
1. Check the CNI DaemonSet Pods
CNI plugins run as DaemonSets in kube-system (labeled e.g. calico-node, flannel, or cilium depending on the plugin). If a CNI Pod is crashing or not scheduled on a node, that node cannot set up Pod networking — check the DaemonSet's Pod status and, for anything in CrashLoopBackOff, its previous-container logs.
2. Check for IP Address Exhaustion
Each node is assigned a Pod CIDR (e.g., 10.244.1.0/24 = 254 usable IPs). If all IPs in that range are allocated, new Pods cannot start. CNI logs will contain messages like no IP addresses available in range set. Each node's assigned CIDR is visible on the Node object itself. A /24 gives 254 Pod IPs per node. If you're running large nodes with many Pods, consider using /23 or larger CIDRs, configured at cluster creation time.
3. Check CNI Config Files on the Node
An empty /etc/cni/net.d/ on the node means kubelet has no CNI configuration. An empty /opt/cni/bin/ means the CNI binaries were never installed or were deleted.
4. Check Node-to-Node Connectivity
If Pods on the same node can communicate but cross-node communication fails, the physical network is blocking CNI traffic. Each CNI plugin has its own required ports:
| CNI Plugin | Protocol | Port | Purpose |
|---|---|---|---|
| Flannel VXLAN | UDP | 4789 | Overlay tunnel |
| Calico IPIP | IP Protocol | 4 | Overlay tunnel |
| Calico VXLAN | UDP | 4789 | Overlay tunnel |
| Calico BGP | TCP | 179 | BGP peering |
| Cilium VXLAN | UDP | 8472 | Overlay tunnel |
| Cilium Geneve | UDP | 6081 | Overlay tunnel |
Testing whether the relevant port is reachable between two nodes directly confirms whether the physical network is the blocker.
5. Check kube-proxy Rules
If Pods can communicate directly by IP but Service IPs don't work, kube-proxy rules are the problem. Confirm kube-proxy itself is running, then inspect the kernel-level rules for the specific Service's ClusterIP — the relevant iptables NAT table entries, or the IPVS table if running in that mode. If no rules exist for a Service IP, kube-proxy either hasn't synced yet or is failing, which its logs will show.
Interviewers testing SRE depth often ask: "A Pod can reach other Pods by IP, but Service IPs don't work. Where do you look?" The answer is kube-proxy. Pod-to-Pod connectivity is the CNI's job; Service IP routing is kube-proxy's job. These are independent systems. If direct Pod IPs work but ClusterIPs don't, kube-proxy rules are missing or stale — check the kube-proxy DaemonSet, its logs, and then iptables-save | grep <service-ip> or ipvsadm -Ln depending on the mode.