Taints and Tolerations: The Bouncer

Repelling Pods from Nodes

Think of a Node as a nightclub and a Taint as the bouncer at the door. The bouncer's job is to turn people away by default — only guests on the VIP list get in. Taints are applied to Nodes. They repel Pods. If a Node is tainted, no Pod can schedule onto it unless that Pod explicitly carries a matching Toleration.

This is the primary mechanism for dedicated node pools: GPU nodes, high-memory nodes, nodes reserved for system components, or nodes with specialized hardware licenses.

Applying a Taint

A taint is applied to a node as a key, a value, and an effect — for example, key gpu, value true, effect NoSchedule. Removing a taint is the same command with a trailing marker indicating removal rather than addition.

Taint Effects

When you taint a node, you choose an Effect that defines how strictly the rule is enforced:

• NoSchedule is the most common. Use it to dedicate nodes to specific workloads. Existing Pods that were scheduled before the taint was added are not disturbed.
• PreferNoSchedule is a soft hint. The scheduler avoids the node but will use it as a last resort. Useful for gradual migrations.
• NoExecute is the nuclear option. It's used when a node is unhealthy or being decommissioned — it evicts all non-tolerating Pods immediately. Kubernetes itself uses NoExecute taints internally when a node becomes NotReady.

Writing a Toleration

A Toleration lives in the Pod spec. It says: "I acknowledge this taint and am allowed to run here." — matched against a taint by the same key, value, and effect. The operator field can be:
- Equal — key, value, and effect must all match exactly.
- Exists — only the key needs to match; value is ignored. Useful for tolerating any taint with a given key regardless of value.

A Toleration is a permission slip, not a destination request. A Pod with a toleration for gpu=true:NoSchedule is allowed on GPU nodes, but it won't necessarily go there. For that, you need Node Affinity.