Finding a Home for Pods
When a Pod object is created in etcd — whether by a controller, a user running kubectl run, or any other mechanism — it initially has no Node assigned to it. Its spec.nodeName field is empty and its status is Pending.
The kube-scheduler watches for exactly these Pods: newly created Pods with no Node assignment. For each one, it runs a decision process to find the most suitable Node, then writes that assignment back to the API Server (which persists it to etcd). The scheduler does not start the Pod itself — it only decides where the Pod should run.
This separation of concerns is important: the scheduler is a pure decision-maker. The actual work of starting containers is done by the kubelet on the chosen worker node, which notices the assignment and acts on it.
The Two-Phase Scheduling Decision
The scheduler uses a two-phase process to select a Node for each Pod.
Phase 1: Filtering (finding feasible nodes)
The scheduler eliminates all nodes that cannot run the Pod. A node is eliminated if any of the following are true:
- Insufficient resources: The node doesn't have enough free CPU or memory to satisfy the Pod's
requests. - Node affinity mismatch: The Pod requires nodes with specific labels (e.g.,
disktype=ssd) and this node doesn't have them. - Taint/Toleration mismatch: The node has a taint (e.g.,
dedicated=gpu:NoSchedule) and the Pod doesn't have a matching toleration. - Port conflicts: The Pod requires a specific host port that is already in use on this node.
- Volume zone mismatch: The Pod requires a PersistentVolume that is only available in
us-east-1a, but this node is inus-east-1b.
Nodes that pass all filters are called feasible nodes. If no nodes are feasible, the Pod remains Pending and the scheduler retries periodically.
Phase 2: Scoring (ranking feasible nodes)
The scheduler scores each feasible node using a set of priority functions. Each function assigns a score from 0 to 10, and the scores are weighted and summed. Common scoring criteria:
- LeastAllocated: Prefer nodes with more free resources (spreads load across the cluster).
- ImageLocality: Prefer nodes that already have the container image cached (faster startup).
- InterPodAffinity: Prefer nodes where related Pods are already running (or avoid them, for anti-affinity).
- TopologySpreadConstraints: Prefer nodes that improve distribution across failure domains (zones, racks).
The node with the highest total score wins. If there's a tie, the scheduler picks randomly among the tied nodes.
Interviewers often ask: "A Pod is stuck in Pending state. How do you diagnose it?" The answer: run kubectl describe pod <pod-name> and look at the Events section at the bottom. The scheduler emits events explaining why it couldn't schedule the Pod — common messages include 0/3 nodes are available: 3 Insufficient memory or 0/3 nodes are available: 3 node(s) had untolerated taint. This immediately tells you whether it's a resource, taint, or affinity issue.