Distributed Locks

SYSTEM DESIGN SERIES  ·  TOPIC 20 OF 23+  ·  part of the 19 → 20 → 21 chain
1. The Lock That Expired While You Weren't Looking
A pauses too long, its lock expires, B steps in - who wins the write? (looping)
No fencing: A gets the lock, then stalls (GC pause, slow disk, doesn't matter why)
No fencing: the lock's TTL expires - A doesn't know, B grabs it and writes safely
No fencing: A finally wakes up and writes too - stale data silently wins
With fencing: every lock grant carries a token that only ever goes up
With fencing: the resource remembers the highest token it's ever accepted
With fencing: A's late, lower-numbered write gets flatly rejected - not silently accepted
2. Why You Need a Lock At All
There's no shared memory to protect you here
  • On one machine, a mutex is free - the OS just handles it
  • Across machines, nothing stops two processes from both starting at once
  • "Only one worker processes this job" needs something everyone can check
  • Same lost-update risk as topic 13 - just stretched across a network
🔑 One key, one door. Except now the door and the people wanting in are on different machines, with no shared clock between them.
3. Pessimistic vs Optimistic Locking
Pessimistic
  • Acquire the lock before touching anything
  • Everyone else simply waits
  • Simple to reason about, can stall the line
VS
Optimistic
  • No lock - check a version number at write time
  • Changed since you read it? Retry (topic 13's MVCC)
  • No waiting, but conflicts cost a retry
🎫 Analogy: pessimistic is taking a numbered ticket and standing in line. Optimistic is just walking up and hoping - and checking the receipt before you commit to it.
4. The Slow-Process Problem
"I still have the lock" is a belief, not a fact
  • A GC pause, a slow disk, a starved CPU (topic 03) - any of them can freeze a process
  • The lock service doesn't know that - it just knows the TTL ran out
  • The paused process wakes up with no idea time has passed at all
  • A TTL protects availability. It does nothing to protect correctness by itself
⏸️ The scariest bugs aren't crashes. They're processes that pause, get declared dead, and then keep going anyway.
5. Fencing Tokens - The Actual Fix
The lock service isn't enough. The resource has to check too.
  • Every time the lock is granted, hand out a number that only increases
  • The client attaches that number to every write it makes
  • The resource itself remembers the highest number it's accepted
  • A lower number arriving late gets rejected - no matter how confident the sender is
6. Where Locks Actually Live
Borrowed infrastructure, mostly
Redis
(SETNX / Redlock)
ZooKeeper
(ephemeral znodes)
etcd
(leases)
Database Row Lock
(unique constraint)
Almost always a consensus system (topic 19) doing the hard part underneath
Redlock is controversial for a reason. Without fencing, even a "correct" lock service can't stop a paused client from writing late.
7. When to Reach for a Distributed Lock
Use one when:
✔ Only one worker may touch this at a time, genuinely
✔ Correctness truly depends on mutual exclusion
✔ You can add fencing tokens where it's enforced
A Lock Without Fencing Is a Suggestion, Not a Guarantee
Consider alternatives when:
✔ Idempotency (topic 17) already makes retries harmless
✔ A single-consumer queue (topic 16) sidesteps the need
✔ You can't enforce the token where it matters
💡 None of this helps if you can't see what your system is actually doing when it goes wrong. That's Observability - topic 21, next.