Webhooks: Intercepting Requests

The bouncers at the door

Before the API Server saves a new or updated object to etcd, it can call out to external HTTPS web services to inspect, modify, or reject the request. These are called Admission Webhooks, and they are one of the most powerful policy enforcement tools in Kubernetes.

They come in two flavors, executed in a specific order:

1. Mutating Webhooks (The "Fixers")

Mutating webhooks run first. They receive the incoming object and can return a modified version of it — a JSON Patch — before it proceeds to validation.

Common use cases:
- Sidecar injection: Service meshes like Istio and Linkerd use mutating webhooks to automatically inject their proxy container into every Pod in labeled namespaces. The developer submits a simple Pod YAML with one container; the webhook adds the sidecar before the object is saved. The developer never has to know.
- Default injection: Setting default values for fields the user didn't specify (e.g., defaulting resources.requests.memory to "128Mi" if not set).
- Label stamping: Automatically adding audit labels like created-by or cost-center to every object.

2. Validating Webhooks (The "Gatekeepers")

Validating webhooks run second, after mutation and schema validation. They can only say "yes" or "no" — they cannot modify the object. This makes them simpler and safer to reason about.

Common use cases:
- Image policy enforcement: Reject any Deployment that uses an image with the latest tag (because latest is not reproducible and makes rollbacks unreliable).
- Port restrictions: Reject any Service that tries to expose port 22 (SSH) to prevent accidental exposure.
- Quota enforcement: Reject resource requests that exceed team-specific limits that standard ResourceQuotas can't express.
- Security policy: Reject Pods that request privileged: true unless they're in the kube-system namespace.

Webhook configuration

Webhooks are registered with the API server using MutatingWebhookConfiguration or ValidatingWebhookConfiguration objects. These tell the API server which URL to call, which resource types to intercept, and what to do if the webhook server is unreachable (Fail or Ignore).

A ValidatingWebhookConfiguration names which operations and resource types to intercept (create/update on Deployments, say), where to send the request (an in-cluster service and path), and — critically — a failurePolicy of Fail or Ignore controlling what happens if the webhook server is unreachable. The failurePolicy: Fail setting is critical for security-sensitive webhooks. If your webhook server is down and failurePolicy is Ignore, objects bypass your policy entirely. For security policies, always use Fail — and make your webhook server highly available.