Debugging DNS: Tracing the Infinite Loop

Why DNS Failures Are Invisible

DNS is the phone book of your cluster. Every service call — http://payment-service/charge, https://api.stripe.com/v1/charges — starts with a DNS lookup. When DNS breaks, everything breaks, but the error messages are misleading. You see connection refused or timeout on the application layer, not "DNS failed." This makes DNS the most common source of mysterious, hard-to-diagnose outages.

Kubernetes uses CoreDNS as its cluster DNS server. CoreDNS runs as a Deployment in kube-system and handles two types of queries:

1. Cluster-internal: payment-service.default.svc.cluster.local → resolves to a ClusterIP.
2. External: api.stripe.com → CoreDNS cannot answer this, so it forwards the query to an upstream resolver.

The forwarding configuration is where loops are born.

The DNS Forwarding Loop

A DNS forwarding loop occurs when DNS server A forwards to server B, and server B forwards back to server A. The query bounces forever, consuming CPU and never resolving.

The most common trigger in Kubernetes is systemd-resolved, a DNS caching daemon that runs on many Linux distributions (Ubuntu 18.04+, Debian 10+). It listens on 127.0.0.53 — a loopback address — and acts as a local DNS stub resolver.

Here is how the loop forms:

1. Kubelet reads /etc/resolv.conf from the node.
2. That file contains nameserver 127.0.0.53 — the systemd-resolved stub.
3. Kubelet copies this into the Pod's own /etc/resolv.conf.
4. The Pod queries CoreDNS for an external name like api.stripe.com.
5. CoreDNS cannot answer directly, so it forwards to upstream.
6. CoreDNS reads its own forwarders config and sees 127.0.0.53.
7. Inside the container network, 127.0.0.53 resolves back to CoreDNS itself.
8. CoreDNS forwards the query to itself — an infinite loop.

Diagnosing the Loop

Step 1: Check CoreDNS logs for loop detection. CoreDNS's own loop plugin actively detects this condition and logs a warning naming the zone where the loop was found.

Step 2: Check what DNS server the node is using. Inspecting the node's /etc/resolv.conf — if it names 127.0.0.53 or 127.0.0.1, systemd-resolved is the stub resolver in play.

Step 3: Find the real upstream DNS. On the node, systemd-resolved's own status command reveals which DNS servers it's actually forwarding to underneath the stub.

Step 4: Check CoreDNS ConfigMap. The forward plugin line in the Corefile should point to a real DNS server, not a loopback address — forward . /etc/resolv.conf is the bad pattern that causes the loop, while naming explicit upstream IPs (like 8.8.8.8 8.8.4.4) avoids it.

The Fix

There are two approaches:

Option A: Configure Kubelet to use the real DNS

Tell Kubelet, in its config, to point resolvConf at the real upstream file (/run/systemd/resolve/resolv.conf, which contains the actual upstream DNS servers) instead of the default /etc/resolv.conf that resolves through the 127.0.0.53 stub, alongside its clusterDNS setting naming CoreDNS's own ClusterIP.

Option B: Override CoreDNS forwarding explicitly

Edit the CoreDNS ConfigMap and change its forward line from reading /etc/resolv.conf to naming explicit upstream IPs directly (e.g. 8.8.8.8 1.1.1.1), then restart the CoreDNS Deployment so the change takes effect.

Option B is simpler but hardcodes public DNS servers. Option A is more correct for enterprise environments with private DNS zones.