CASE FILE NO. 03 - SYSTEM DESIGN SERIES

The Case of the
Overworked Database

Every clue on this board points to the same suspect: doing the same expensive work twice. Follow the red string to see how caching solves it - and where the trail goes cold.

1 First principles

Why does this problem even exist?

Every piece of data lives somewhere, and reading it costs time proportional to where it physically sits. That cost isn't arbitrary - it's a hardware fact: memory that's fast to access is expensive and small (RAM), memory that's cheap and large is slow (disk, network storage). You can't build something simultaneously fast, large, and cheap. That gap is called the memory hierarchy.

Caching is the general strategy of copying data from a slow, authoritative location to a fast, temporary one - so repeat requests pay the fast cost instead of the slow one. It only works because of two real properties of access patterns:

Temporal locality - data accessed recently is likely to be accessed again soon.
Spatial locality - data near something accessed is likely to be accessed too.

"A cache is a sticky note on your desk with today's most-needed numbers. Check the note before you go dig through the phone book."

The memory hierarchy, to scale

CPU register
~0.3 ns
RAM
~100 ns
SSD
~50–150 μs
Spinning disk
~5–10 ms
Cross-region network
~50–150 ms
2 Core mechanism - run it yourself
TTL6s
Hits: 0Misses: 0
Client Cache empty Database $19.99
Click "Send request" to see a cache miss, then click again to see a hit.
3 The full decision space

Tap any card to open the evidence.

4 Witness statements - real systems

Redis

Dominant in-memory app cache. Configurable eviction (LRU/LFU variants), native per-key TTL. Used across most large web companies as cache, session store, and rate-limiter backend.

Memcached

Simpler, no persistence. Facebook's 2013 paper describes thousands of nodes behind a proxy (mcrouter), with a "leases" mechanism to prevent stampedes.exact figures unverified

CDNs

Cloudflare, Akamai, Fastly, CloudFront cache static (and sometimes dynamic) content at edge locations near the user - same hit/miss mechanism, applied geographically.

Netflix

Heavy edge caching for video metadata and assets. Believed to use an internal layer (EVCache) built on Memcached for cross-region data.details unverified

MySQL InnoDB

Buffer pool caches disk pages in RAM automatically - caching happening even when nobody explicitly designed a caching layer.

Browsers

Cache-Control, ETag, and If-None-Match headers drive the freshness/invalidation logic for the browser cache layer.

5 Red flags - what breaks this

Cache stampede

A hot key expires; every concurrent miss hammers the database at once. Fix: request coalescing (one fetch, everyone else waits on it) + TTL jitter.

Write-back durability gap

Cache confirms the write, then crashes before flushing to the database. That write is gone - a real trade-off, not a bug.

Multi-node inconsistency

Each app server keeps its own local cache; one updates, others still serve stale data. Needs a shared tier or broadcast invalidation.

Fail-open vs fail-closed

Cache node unreachable - bypass it and hit the DB directly (usually right), or return an error? Availability usually wins.

Clock skew

TTL correctness assumes synced clocks across nodes. Drift in a cross-region cluster causes early or late expiry.

Hot key

One viral key overwhelms the single node responsible for it, even though the cache works exactly as designed. Fix: replicate that key, or add a local L1 in front.

6 Scale & numbers

Database handles 5,000 QPS comfortably. Traffic grows.

10×
50,000 QPS total.
90% hit rate → only 5,000 QPS reaches the DB. Same load it started with.
100×
500,000 QPS total.
Same 90% hit rate → 50,000 QPS now hits the DB - 10× its design limit. Hit rate that worked before no longer does.

The design decision that flips at 100×: caching alone stops being enough. You now need a higher hit rate, a sharded cache, or a sharded database - horizontal scaling of the cache layer itself, not just a bigger single cache.

7 Reference sheet
DecisionOptionsPick based on
Read patternCache-aside / read-throughCache-aside for app-level control; read-through for cleaner code
Write patternWrite-through / write-back / write-aroundConsistency-critical → through. Latency-critical → back. Write-heavy, rarely read → around
EvictionLRU / LFU / FIFO / ARC / W-TinyLFUDefault LRU. Permanently-hot subset → LFU. Max simplicity → FIFO
InvalidationTTL / explicit / event-driven / versioned keysCan tolerate staleness → TTL. Need near-zero staleness → explicit/event-driven
PlacementBrowser / CDN / reverse proxy / app cache / DB bufferStatic, shared → CDN. Per-user computed → app cache
TopologySingle node / sharded / replicatedFits in memory → single. Outgrew it → sharded. Read-heavy redundancy → replicated
StampedeCoalescing / TTL jitter / early recomputeVery hot single keys → coalescing. Many keys expiring together → jitter
8 The interrogation