What Actually Happens During a Write

SYSTEM DESIGN SERIES  ·  TOPIC 09 OF 23+  ·  part of the 08 → 09 → 10 chain
1. The Write Path - Step by Step
Watch the request move through the system, one stage at a time (looping)
1. Client sends the write request
2. App server receives it, begins the write
3. Appends to the Write-Ahead Log first - durable, sequential
4. Applies the change to the in-memory structure
5. Sends the write onward to the replica
6. Replica acknowledges it has the write too
7. App responds to the client - write confirmed
2. Why the WAL Comes First
The log gets written before the "real" data structure does
  • Every write is appended to a durability log before anything else changes
  • Sequential appends are fast - no seeking, no fragmentation
  • Crash right after? Replay the log on restart, nothing is lost
  • The "real" structure (B-tree, memtable) is updated after, sometimes lazily
📝 Analogy: Like jotting a note before starting a task. If you get interrupted mid-task, the note tells you exactly what you meant to finish.
3. Sync vs Async Replication Ack
Synchronous
  • Wait for the replica to confirm before telling the client "done"
  • Safer - survives the primary dying right after
  • Slower - pays the network round trip too
VS
Asynchronous
  • Tell the client "done" right after the local write
  • Faster - replica catches up in the background
  • A crash before it catches up can lose the write
⚖️ This is a durability-vs-latency knob, not a free lunch - topic 11 (Replication) goes much deeper here.
4. What Can Go Wrong, Mid-Write
Every gap in the pipeline is a place a crash can land
  • Before the WAL append: client got no ack - safe to just retry
  • After WAL, before replica ack: primary has it, replica might not (topic 02)
  • Ack strategy from zone 3 is exactly what decides which of these is possible
5. Sequential vs Random I/O
Why appending to a log is so much cheaper than updating in place
  • WAL append - one sequential write, disk head never moves
  • Updating a B-tree page - scattered, random I/O, all over the disk
  • Sequential can be 10–100x faster, especially on spinning disks
6. A Write Touches Several Systems
Not one atomic instant - a small pipeline, every time
The Log
Memory
Disk Page
Replica
The Network
Every one of these can be the thing that's slow, or the thing that fails
"Just write it to the database" quietly means all five of these - that's the whole point of this topic.
7. Why This Matters Going Forward
This write path is why:
✔ Indexes (topic 10) add extra writes per write
✔ Replication (topic 11) has a lag to reason about
✔ Consistency models (topic 12) even need to exist
A Write Is a Pipeline, Not an Instant
And also why:
✔ Transactions (topic 13) need real isolation rules
✔ "It's slow" always has a specific stage to blame
✔ Durability is a choice you make, not a default
💡 Every one of these writes might also have to update every index on the table. That's Indexing - topic 10, next.