The Token Bucket: Bursts Allowed

The Analogy

Imagine a bucket that holds tokens — think of them as permission slips.

• Refill: A dispenser adds tokens to the bucket at a fixed rate (e.g., 1 token per second).
• Consumption: To send a request, a client must take one token from the bucket. No token, no request.
• Capacity: The bucket has a maximum size (e.g., 10 tokens). If the dispenser tries to add a token to a full bucket, the token is discarded — you can't accumulate unlimited credit.

How It Actually Works

The Token Bucket is implemented as a counter with a refill rate and a capacity ceiling:

1. Tokens accumulate in the bucket at rate r tokens/second, up to a maximum of capacity tokens.
2. When a request arrives, check if tokens >= cost (usually cost = 1).
3. If yes: decrement tokens, allow the request.
4. If no: reject the request (or queue it, depending on implementation).

The critical property: if tokens have accumulated, multiple requests can be served simultaneously. A client that has been idle for 10 seconds with a refill rate of 1/sec and a capacity of 10 will have a full bucket — and can fire 10 requests instantly.

Bucket state after 10 seconds idle: [●●●●●●●●●●] (10 tokens)

10 requests arrive simultaneously:
  → All 10 consume 1 token each
  → All 10 are ALLOWED immediately
  → Bucket is now empty: []

11th request arrives 0.5 seconds later:
  → 0.5 tokens refilled (not enough for 1)
  → Request REJECTED

Use Case: User-facing APIs, SDKs, and any system where clients exhibit natural "clumpiness." A user clicking "Refresh" five times quickly should succeed — that's normal behavior. You only want to block them if they sustain a high rate over time.

Variable token cost: Advanced implementations allow requests to consume multiple tokens based on their weight. A lightweight GET /status might cost 1 token; a heavy POST /export-report might cost 10. This lets you enforce rate limits based on resource consumption rather than raw request count.