The Standard Building Blocks
Every system design interview draws from the same toolkit of scaling patterns. You need to know not just what each component is, but when to use it and what trade-offs it introduces.
1. Load Balancing
A load balancer sits in front of a pool of servers and distributes incoming requests across them. Without it, all traffic hits one server, which becomes a bottleneck and a SPOF.
Common algorithms:
- Round Robin: Requests go to servers in rotation (1→2→3→1→2→3). Simple, but ignores server load.
- Least Connections: New requests go to the server with the fewest active connections. Better for variable-length requests.
- Consistent Hashing: A request is always routed to the same server based on a hash of a key (e.g., UserID). Critical for stateful services or caches — ensures a user's session always hits the same backend.
Trade-off: Load balancers themselves can become a SPOF. Production systems use active-passive or active-active pairs of load balancers.
2. Caching
Caching stores the result of an expensive operation (a database query, an API call, a computation) in fast memory (RAM) so subsequent requests can be served without repeating the work.
Cache-Aside (Lazy Loading):
1. Application receives request.
2. Check cache. If HIT → return cached value.
3. If MISS → query database, store result in cache, return value.
Pro: Only caches data that is actually requested. Con: First request after a cache miss is slow (cache stampede risk under high load).
Write-Through:
1. Application writes data.
2. Write to cache AND database simultaneously.
3. Cache is always consistent with the database.
Pro: Cache is never stale. Con: Every write is slower (two writes instead of one). Rarely-read data wastes cache space.
Key cache metrics to discuss: TTL (Time To Live), eviction policy (LRU, LFU), cache hit rate (target >90% for hot-path data).
3. Database Sharding
Sharding splits a single large database into multiple smaller databases (shards), each holding a subset of the data. A single machine has physical limits — typically 5–10 TB of fast SSD storage and a ceiling on concurrent connections. Sharding breaks through those limits by distributing data across many machines.
Shard by UserID example:
Shard = UserID % N (where N = number of shards)
UserID 1001 → 1001 % 4 = 1 → Shard 1
UserID 1002 → 1002 % 4 = 2 → Shard 2
UserID 1003 → 1003 % 4 = 3 → Shard 3
UserID 1004 → 1004 % 4 = 0 → Shard 0
Trade-offs of sharding:
- Hot shards: If one UserID range is disproportionately active (e.g., a celebrity's account), one shard gets overloaded. Mitigate with consistent hashing or virtual nodes.
- Cross-shard queries: A query like "find all users who signed up this week" must fan out to all shards and aggregate results. This is expensive. Design your shard key to match your most common query patterns.
- Resharding: Adding a new shard requires redistributing data. Consistent hashing minimizes the data movement required.
4. Message Queues
A message queue (Kafka, RabbitMQ, SQS) decouples producers from consumers. Instead of Service A calling Service B directly (synchronous, tight coupling), A writes a message to a queue and B reads from it when ready (asynchronous, loose coupling).
When to use: Any time a task can be done asynchronously — sending emails, processing video uploads, generating thumbnails, updating search indexes. If the task takes more than ~100ms, consider making it async.
A common interview question: "Your API endpoint is slow because it does too much work synchronously. How do you fix it?" The answer pattern is: identify which parts of the work are not required to return a response to the user, move those parts to an async worker via a message queue, and return a 202 Accepted immediately. For example, when a user uploads a video, you don't need to transcode it before confirming the upload succeeded — accept the raw file, enqueue a transcoding job, and process it in the background.