Every clue on this board points to the same suspect: doing the same expensive work twice. Follow the red string to see how caching solves it - and where the trail goes cold.
Every piece of data lives somewhere, and reading it costs time proportional to where it physically sits. That cost isn't arbitrary - it's a hardware fact: memory that's fast to access is expensive and small (RAM), memory that's cheap and large is slow (disk, network storage). You can't build something simultaneously fast, large, and cheap. That gap is called the memory hierarchy.
Caching is the general strategy of copying data from a slow, authoritative location to a fast, temporary one - so repeat requests pay the fast cost instead of the slow one. It only works because of two real properties of access patterns:
Temporal locality - data accessed recently is likely to be accessed again soon.
Spatial locality - data near something accessed is likely to be accessed too.
Tap any card to open the evidence.
Dominant in-memory app cache. Configurable eviction (LRU/LFU variants), native per-key TTL. Used across most large web companies as cache, session store, and rate-limiter backend.
Simpler, no persistence. Facebook's 2013 paper describes thousands of nodes behind a proxy (mcrouter), with a "leases" mechanism to prevent stampedes.exact figures unverified
Cloudflare, Akamai, Fastly, CloudFront cache static (and sometimes dynamic) content at edge locations near the user - same hit/miss mechanism, applied geographically.
Heavy edge caching for video metadata and assets. Believed to use an internal layer (EVCache) built on Memcached for cross-region data.details unverified
Buffer pool caches disk pages in RAM automatically - caching happening even when nobody explicitly designed a caching layer.
Cache-Control, ETag, and If-None-Match headers drive the freshness/invalidation logic for the browser cache layer.
A hot key expires; every concurrent miss hammers the database at once. Fix: request coalescing (one fetch, everyone else waits on it) + TTL jitter.
Cache confirms the write, then crashes before flushing to the database. That write is gone - a real trade-off, not a bug.
Each app server keeps its own local cache; one updates, others still serve stale data. Needs a shared tier or broadcast invalidation.
Cache node unreachable - bypass it and hit the DB directly (usually right), or return an error? Availability usually wins.
TTL correctness assumes synced clocks across nodes. Drift in a cross-region cluster causes early or late expiry.
One viral key overwhelms the single node responsible for it, even though the cache works exactly as designed. Fix: replicate that key, or add a local L1 in front.
Database handles 5,000 QPS comfortably. Traffic grows.
The design decision that flips at 100×: caching alone stops being enough. You now need a higher hit rate, a sharded cache, or a sharded database - horizontal scaling of the cache layer itself, not just a bigger single cache.
| Decision | Options | Pick based on |
|---|---|---|
| Read pattern | Cache-aside / read-through | Cache-aside for app-level control; read-through for cleaner code |
| Write pattern | Write-through / write-back / write-around | Consistency-critical → through. Latency-critical → back. Write-heavy, rarely read → around |
| Eviction | LRU / LFU / FIFO / ARC / W-TinyLFU | Default LRU. Permanently-hot subset → LFU. Max simplicity → FIFO |
| Invalidation | TTL / explicit / event-driven / versioned keys | Can tolerate staleness → TTL. Need near-zero staleness → explicit/event-driven |
| Placement | Browser / CDN / reverse proxy / app cache / DB buffer | Static, shared → CDN. Per-user computed → app cache |
| Topology | Single node / sharded / replicated | Fits in memory → single. Outgrew it → sharded. Read-heavy redundancy → replicated |
| Stampede | Coalescing / TTL jitter / early recompute | Very hot single keys → coalescing. Many keys expiring together → jitter |