CASE FILE NO. 04 - SYSTEM DESIGN SERIES

The Case of the
Data That Wouldn't Fit

One machine ran out of room. Now it's copied, split, and scattered across a dozen more - and every clue on this board is about keeping the story straight once it is.

1 · First principles 2 · Mechanism 3 · Decision space 4 · Real systems 5 · Failure modes 6 · Scale 7 · Reference table 8 · Interrogation
1 First principles

Why does this problem even exist?

A single database on a single machine has the same ceiling as any single machine - finite CPU, RAM, disk. Past that ceiling you split the data: copy it (replication) and/or divide it (sharding). The moment data lives on more than one machine talking over a network, you inherit a fact you can't engineer away: networks partition. Every other topic on this board - CAP, quorums, consistent hashing - is an answer to the same question forced by that one fact:

"When a machine can't confirm the data is current, does it answer anyway, or refuse to answer at all?"

SQL vs. NoSQL is a downstream decision, not a first principle - it's about which guarantees you need enough to justify a rigid schema, versus which flexibility you need enough to give some of those guarantees up.

2 Core mechanism - run it yourself

A single DB writes to a log before applying anything (durability) and reads through an index (a B-tree or LSM-tree) instead of scanning everything. Everything below is what happens once one machine isn't enough.

Mode
Synchronous Asynchronous
Leader Replica A value: - Replica B value: -
Try "Write value = 42" in Synchronous mode, then again in Asynchronous mode - then immediately read Replica B.
The ring holds 4 shards and 6 keys. Add or remove a shard and watch how few keys actually move.
Region link
Connected Partitioned
Region B chooses
CP AP
Region A balance = $100 Region B balance = $100
Set the link to "Partitioned," pick CP or AP for Region B, then try reading from it.
3 The full decision space

Tap any card to open the evidence.

4 Witness statements - real systems

PostgreSQL / MySQL

Single-leader relational, strong ACID guarantees, read replicas for read scaling. Sharding is a manual/add-on concern (e.g. Citus for Postgres), not built in.

MongoDB

Document model, flexible schema. Multi-document transaction guarantees have strengthened over the years.current specifics unverified

Cassandra

Wide-column, leaderless replication, tunable N/W/R quorum, consistent hashing for partitioning - direct descendant of Amazon's Dynamo paper. AP-leaning by default.

DynamoDB

Also Dynamo-lineage: consistent hashing, quorum reads/writes, eventual by default with a strongly-consistent read option.current defaults unverified

Google Spanner

Strong consistency across a globally distributed DB using tightly synchronized clocks (TrueTime). Still bound by CAP during an actual partition.

Redis

Key-value, typically single-leader with async replicas - simpler replication model than the quorum-based systems above. (Full profile: Caching case file.)

5 Red flags - what breaks this

Replication lag

Write, then immediately read a replica that hasn't caught up - the read-your-own-writes problem. Fix: route a user's own reads to the leader right after they write.

Split-brain leader election

A partition makes replicas think the leader is dead when it isn't - two nodes both accept writes as "the leader." Both sets need reconciling once it heals.

Hot shard

One key gets disproportionate traffic, overwhelming its shard even though sharding is working exactly as designed. Same fix as a hot key on a hash ring: split it or replicate it specifically.

Live resharding risk

Moving data while staying online usually needs a dual-write window. A bug there can silently drop or duplicate writes.

Clock skew

Breaks naive last-write-wins conflict resolution, and is the exact uncertainty Spanner's TrueTime is built to bound.

Cross-shard transactions

Needs distributed coordination (two-phase commit or a saga) - both add latency and new failure modes a single shard never had.

Denormalization write amplification

Duplicated data for read speed means every write must update every copy. Miss one and it's a silent correctness bug.

6 Scale & numbers

A single-leader database handles roughly 5,000 writes/second comfortably.

10×
50,000 writes/sec needed. Read replicas help zero - they only serve reads. The write path is still one leader.
100×
500,000 writes/sec. Even 10 shards from the step above are back at 50,000 each - their own ceiling. Shard count must keep growing, and cross-shard cost stops being an edge case.

The decision this forces: past a certain write volume, replication doesn't help the bottleneck at all - only sharding does, because replicas copy data, they don't divide the write load.

7 Reference sheet
DecisionOptionsPick based on
SQL vs NoSQLRelational / document / key-value / wide-column / graphMulti-row transactions, known schema → SQL. Flexible schema → document. Fast key lookup → key-value. Huge write volume → wide-column. Connected data → graph
ACID vs BASEStrict transactions / eventual consistencyCorrectness-critical → ACID. Must stay available through partitions → BASE
Isolation levelRead uncommitted → serializableDefault read committed; go stricter only for a specific anomaly you must prevent
ReplicationSingle-leader / multi-leader / leaderlessSimplicity → single-leader. Multi-region writes → multi-leader. No write SPOF → leaderless + quorum
Quorum (N,W,R)Tune W+R vs NGuaranteed-fresh reads → W+R > N. Speed/availability over freshness → W+R ≤ N
ShardingRange / hash / directory / consistent hashingRange queries → range. Even load → hash. Max flexibility → directory. Frequent resharding → consistent hashing
CAP choiceCP / APCan't tolerate stale data → CP. Must stay available through partitions → AP
NormalizationNormalized / denormalizedWrite correctness → normalize. Read speed at scale → denormalize (update every copy)
8 The interrogation