The Pattern: Circuit Breaking Bad Queries

The Death Spiral

Imagine a developer pushes a bug at 2 PM on a Tuesday: a query that is missing a WHERE clause index and takes 5 seconds to run instead of 5 milliseconds.

Here's what happens next, in order:

1. Users visit the page. It loads slowly. They wait.
2. Impatient users hit "Refresh." Now the database has two copies of the slow query running simultaneously.
3. More users arrive. The connection pool — typically capped at 100–200 connections — fills up entirely.
4. New, fast queries (login checks, health endpoints, homepage loads) are blocked waiting for a free connection. They time out.
5. The database CPU hits 100% trying to run 200 concurrent 5-second queries.
6. The entire application goes down. Not because of the bad query alone, but because the bad query consumed all shared resources.

This is called a cascading failure or death spiral. One bad query takes down everything.

The Circuit Breaker Pattern

A Circuit Breaker is a pattern borrowed from electrical engineering. In your home, if a circuit is overloaded, a breaker trips and cuts power to that circuit — protecting the rest of the house. The database equivalent works the same way.

The circuit breaker is implemented in the application layer or in a connection proxy (like PgBouncer, ProxySQL, or Envoy). It has three states:

Closed (Normal Operation)
All traffic flows through to the database. The breaker monitors error rates and latency. This is the default state.

Open (Tripped)
When a threshold is exceeded — for example, "more than 30% of queries in the last 10 seconds took longer than 2 seconds" — the breaker trips. In the Open state, the application immediately rejects new database requests without even attempting to connect. It returns a fast error (e.g., HTTP 503) to the caller. This gives the database breathing room to drain its queue of in-flight queries and recover.

Half-Open (Testing Recovery)
After a configurable timeout (e.g., 15 seconds), the breaker allows exactly one request through. If that request succeeds within the latency threshold, the breaker closes and normal traffic resumes. If it fails, the breaker returns to Open and the timeout resets.

The key insight is that failing fast is better than failing slow. A user who gets an immediate "Service temporarily unavailable" error has a better experience than one who waits 30 seconds for a timeout. And the database gets the silence it needs to recover.