Why controllers are robust by design
Kubernetes controllers — both built-in and custom ones — use level-based triggering, not edge-based triggering. Understanding this distinction is key to understanding why Kubernetes is so resilient.
- Edge-based: "An event happened — react to it." If your controller crashes at the exact moment an event fires, you miss it. You'd need complex event replay or journaling to recover.
- Level-based: "The current state is X. The desired state is Y. They differ, so I must act." It doesn't matter when or how the state changed — only that it is currently different from what's desired.
The thermostat analogy: A thermostat doesn't care that you turned the dial 5 minutes ago (an edge event). It wakes up, reads the dial (desired: 72°F) and the thermometer (current: 65°F), sees a difference, and turns on the heater. If the thermostat was unplugged for an hour, it doesn't need a log of what happened while it was off — it just looks at the current levels and acts.
This approach makes Kubernetes controllers extremely robust. If a controller crashes and restarts, it simply reads the current state of the cluster and continues working toward the desired state. No events were "missed" because the controller never relied on events for correctness — only for efficiency (events wake the controller up faster than polling).
The practical implication for Operator authors: your Reconcile() function must be idempotent. It will be called many times for the same object. Calling it 10 times in a row with the same desired state must produce the same result as calling it once.