The Node Coverage Problem
Some infrastructure components don't have a fixed replica count — they need to run on every node in the cluster. Consider:
- Log collectors (Fluentd, Filebeat): Must run on every node to read that node's container logs from
/var/log/containers/. A log collector on node A cannot read logs from node B. - Monitoring agents (Prometheus Node Exporter): Must run on every node to expose that node's CPU, memory, and disk metrics.
- Network plugins (Calico, Cilium): Must run on every node to configure the node's network interfaces and routing rules.
- Storage daemons (Ceph, GlusterFS clients): Must run on every node to mount distributed storage.
You could try to manage this with a Deployment set to replicas: N where N equals your node count, but this breaks immediately when you add or remove nodes. You'd have to manually update the replica count every time the cluster scales.
How DaemonSets Work
A DaemonSet ensures exactly one copy of a Pod runs on every node (or a selected subset of nodes). The Kubernetes scheduler handles the rest automatically:
- New node joins the cluster: The DaemonSet controller immediately schedules the Pod onto it.
- Node is removed from the cluster: The Pod is garbage collected.
- Node is cordoned (marked unschedulable): By default, DaemonSet Pods are still scheduled on cordoned nodes because they use a special toleration for
node.kubernetes.io/unschedulable. This ensures monitoring and logging continue even on nodes being drained.
Targeting a Subset of Nodes
You don't always want a DaemonSet on every node. Using nodeSelector or nodeAffinity in the Pod template restricts it to nodes matching a given label — for example, only nodes labeled as having GPU hardware. This is useful for specialized hardware monitoring — you only want the GPU monitoring agent on nodes that actually have GPUs.