Free — no signup required

The Controller Manager: The State Reconciler

2 min read

Desired vs. Current State

Kubernetes is built on a declarative model. Instead of issuing imperative commands ("start this container now"), you declare your desired end state ("I want 3 replicas of this container running") and Kubernetes figures out how to get there and how to stay there.

The component that continuously drives the cluster toward desired state is the kube-controller-manager. It is a single binary that packages dozens of independent controller processes, each responsible for a specific type of Kubernetes object.

Controller Responsibility
Deployment Controller Creates and manages ReplicaSets when Deployments change
ReplicaSet Controller Ensures the correct number of Pod replicas exist
Node Controller Monitors node health; marks nodes as unreachable after timeout
Job Controller Tracks Jobs to completion; creates Pods to run batch tasks
ServiceAccount Controller Creates default ServiceAccounts in new namespaces
Namespace Controller Cleans up resources when a namespace is deleted
EndpointSlice Controller Populates EndpointSlice objects (linking Services to Pods)

Running them as a single binary rather than separate processes simplifies deployment and reduces the number of processes that need leader election coordination.

The Reconciliation Loop

Every controller runs the same fundamental pattern: an infinite reconciliation loop (also called a control loop or watch loop).

while true:
    desired_state = watch(API Server, "what should exist?")
    current_state = watch(API Server, "what actually exists?")

    if current_state != desired_state:
        take_action_to_close_the_gap()

In practice, controllers use the API Server's watch mechanism rather than polling. The API Server maintains a long-lived HTTP connection to each watcher and pushes events (ADDED, MODIFIED, DELETED) as objects change. This makes reconciliation event-driven and efficient rather than requiring constant polling.

A concrete example — ReplicaSet Controller:

  1. You create a Deployment with replicas: 3. The Deployment Controller creates a ReplicaSet with replicas: 3.
  2. The ReplicaSet Controller observes: desired=3, current=0. It creates 3 Pod objects via the API Server.
  3. One of the worker nodes crashes. The Node Controller marks it unreachable. The Pod on that node is marked as Failed.
  4. The ReplicaSet Controller observes: desired=3, current=2. It creates 1 new Pod object.
  5. The Scheduler assigns the new Pod to a healthy node. The kubelet starts it.
  6. The ReplicaSet Controller observes: desired=3, current=3. No action needed.

This loop runs continuously. The cluster is always trying to heal itself.

This is one of 18 chapters

Every chapter is free — Kubernetes, Terraform, SRE, distributed systems, and more — with fast daily review built in.

Sign up free