Caching

SYSTEM DESIGN SERIES · TOPIC 07 OF 23+ · part of the 06 → 07 → 08 chain

1. Why Cache At All?

Trade a little staleness for a lot of speed

Removes repeated, expensive work - a DB query, a computation, an API call
Directly reduces the CPU / disk I/O a server burns through (topic 03)
Cache hit → fast path. Cache miss → falls back to the real, slow path
Even a 90% hit rate means the slow path only runs 1 in 10 times
The best cache is invisible - until it's serving stale data

2. The Cache Hierarchy - Caches All the Way Down

Every layer between the user and the disk caches something

Browser - client keeps its own local copy
CDN / edge - a geographically close copy (topic 08)
App-local cache - fastest, but not shared (topic 05's "sneaky" stateful cache)
Distributed cache (Redis/Memcached) - shared across every instance
DB buffer pool - the database caches its own hot pages in memory

3. Read Strategies

Cache-Aside (Lazy Loading)

App checks the cache first, itself
On a miss, app fetches from DB and writes to cache
Most common - explicit, easy to reason about

Read-Through

Cache sits in front of the DB as a proxy
On a miss, the cache itself fetches and fills
App only ever talks to the cache - simpler app code

🧊 Analogy: Cache-aside - you check the fridge; if it's empty, you go to the store and restock it. Read-through - you ask a stocked pantry service, and it goes to the store for you.

4. Write Strategies

The real question: where does the write go, and in what order?

Write-Through

Every write hits cache AND DB, synchronously. Safe and consistent - write latency = DB write cost.

Write-Back (Write-Behind)

Write hits cache immediately; DB is updated later, async. Fast writes - but risks loss if the cache dies first.

Write-Around

Write goes straight to the DB, skipping cache entirely. Avoids filling the cache with rarely-read writes.

5. Eviction Policies - When the Cache Is Full

Something has to be thrown out to make room

LRU

Evicts whatever hasn't been touched in the longest time. Most common default.

LFU

Evicts whatever has been accessed the fewest times, ever.

FIFO

Evicts the oldest inserted item - usage pattern doesn't matter at all.

TTL

Every item has an expiry clock. Evicted on schedule, used or not.

6. The Cache Stampede Problem

A hot key expires, and every waiting request hammers the DB at once

The Stampede

Locking / Coalescing

Serve Stale, Refresh Behind

Jittered Expiry

Warm It Before It Expires

This is the failure mode that takes down otherwise-healthy systems. The cache did its job for months - then one popular key expired at the wrong moment.

7. Invalidation & What NOT to Cache

Invalidation approaches:

✔ TTL - just let it expire

✔ Explicit - clear the key on every write

✔ Event-driven - pub/sub tells caches what changed

✔ Negative caching - cache "not found" too

Stale Data Is a Choice - Decide How Stale Is Acceptable

Poor candidates for caching:

✘ Highly personalized, one-shot data

✘ Data that changes every second

✘ Anything needing strong consistency (topic 12)

✘ Write-heavy, rarely-read data

💡 CDN / Edge (topic 08) is caching taken to its geographic extreme - same ideas, applied at the edge of the network.