Understanding the Scale
Nanoseconds are impossible to intuit. To make the differences visceral, let's scale time so that one CPU cycle (roughly 0.3 nanoseconds on a 3 GHz processor) takes 1 second of human time.
Here is how long it takes to get data from different places in this scaled analogy:
| Storage Level | Analogy | Scaled Time | Actual Latency |
|---|---|---|---|
| L1 Cache | Grab a pen from your desk | ~1–4 seconds | ~1 ns (~3–4 cycles) |
| L2 Cache | Open your desk drawer | ~4–12 seconds | ~4 ns (~10–12 cycles) |
| L3 Cache | Walk to a bookshelf across the room | ~15–50 seconds | ~15 ns (~40–50 cycles) |
| Main Memory (RAM) | Walk to the library down the hall | ~1.5–3 minutes | ~50–100 ns (~150–300 cycles) |
| NVMe SSD (random read) | Drive to a warehouse across town | ~3–8 hours | ~10–25 μs |
| Spinning Disk (HDD) | Fly to another country | ~3–14 days | ~1–5 ms |
| Network Round-Trip (same datacenter) | Send a letter across the country | ~5–17 days | ~500 μs (0.5 ms) |
| Network Round-Trip (cross-continent) | Send a letter to another continent | ~4–5 months | ~50–150 ms |
Notice the pattern: each step down the hierarchy is roughly 10x–100x slower than the one above it. The jump from L1 cache to RAM is about 100x. The jump from RAM to SSD is about 200x. The jump from SSD to network is another 20–200x.
Key Takeaway: The primary goal of high-performance programming is to keep the data you are actively using in the fast L1/L2 caches and avoid the long, slow walk to main memory. For platform engineers, this same principle scales up: keep hot data in RAM (not on disk), keep frequently accessed services in the same datacenter (not across the ocean), and size your caches (Redis, page cache, buffer pool) to hold the working set.
A common interview question is: "Walk me through what happens when a CPU needs to read a value from memory." The strong answer describes the cache hierarchy lookup: L1 → L2 → L3 → RAM, explaining that each miss cascades to the next level. Mention that L1 is split into instruction cache (L1i) and data cache (L1d), that L2 is typically per-core, and L3 is shared across cores. Finish by noting that a cache miss to RAM costs ~100 ns—enough time for ~300 wasted CPU cycles on a modern processor. Candidates who can recite actual latency numbers stand out immediately.