Don't Route to a Corpse
A load balancer that routes traffic to a crashed server is worse than no load balancer at all — at least without an LB, clients would get a connection refused error immediately. With a broken LB, they get a timeout, which is far more painful.
Load balancers use health checks to continuously probe backend servers and remove unhealthy ones from the rotation. There are three levels of health check sophistication:
TCP Health Check (L4): The LB attempts to open a TCP connection to the backend's port. If the three-way handshake completes, the server is considered alive. This is the fastest and cheapest check, but it only verifies that the process is listening — not that it's actually serving requests correctly. A web server with a broken database connection will pass a TCP health check.
HTTP Health Check (L7): The LB sends an HTTP GET request to a specific endpoint (e.g., /healthz or /health) and expects a 200 OK response within a timeout window. The application is responsible for implementing this endpoint to check its own dependencies (database connectivity, cache availability, etc.) and return a non-200 status if anything is wrong. This is the standard for modern web services.
Custom/Deep Health Check: Some LBs support sending a specific request and validating the response body — for example, checking that a JSON field {"status": "ok"} is present. This catches cases where the server returns 200 but with corrupted or empty responses.
Key parameters every health check has:
- Interval: How often to probe (e.g., every 10 seconds).
- Timeout: How long to wait for a response before declaring the probe failed (e.g., 5 seconds).
- Healthy threshold: How many consecutive successful probes before marking a server healthy (e.g., 2).
- Unhealthy threshold: How many consecutive failed probes before removing a server from rotation (e.g., 3).
The unhealthy threshold prevents a single transient network hiccup from yanking a healthy server out of rotation. The healthy threshold prevents a server that just restarted from receiving traffic before it's fully warmed up.