Consensus & Leader Election

SYSTEM DESIGN SERIES · TOPIC 19 OF 23+ · part of the 18 → 19 → 20 chain

1. Why Split-Brain Doesn't Happen

Same 5-node cluster, same network split - watch who's allowed to lead (looping)

Crash + election: the leader dies, a follower notices, becomes a candidate

Crash + election: it asks the others for a vote - 3 of 5 is a majority

Crash + election: majority reached, it becomes leader - clean, no ambiguity

Partition: the network splits into a group of 3 and a group of 2

Partition: only the group of 3 can ever reach a majority of 5

Partition: the group of 2 tries, can't get there, and safely stays leaderless

2. Why Consensus Is Hard

Agreeing sounds simple until the network gets involved

No shared clock - nodes can't just agree on "now"
A silent node might be dead, slow, or just unreachable (topic 02) - you can't tell which
Waiting for everyone to reply means one dead node blocks the whole system forever
The real question: how do you decide safely without hearing from everyone?

🗳️ You can't poll a room where some people might have quietly left. You need a rule that works anyway.

3. Quorums - The Practical Trick

You don't need everyone. You need more than half.

A majority is more than half of all nodes - 3 of 5, not 5 of 5
Any two majorities must share at least one node - that's the whole trick
Two conflicting decisions can never both get a majority at once
Same math behind topic 11's W + R > N

4. Leader Election, Step by Step

The Raft-style version, in four beats

Follower

Quietly trusts the current leader's heartbeats. Default state, most of the time.

Timeout

No heartbeat arrives in time. Could mean the leader is gone - or just slow.

Candidate

Bumps the term number, votes for itself, asks everyone else for theirs.

Leader

Got a majority. Starts sending heartbeats - until it doesn't, and this repeats.

5. Split-Brain - Why Quorums Prevent It

The direct answer to topic 11's open question

A partition can split a cluster, but it can't split a majority in two
With an odd cluster size, at most one side of any split can have > half
The minority side keeps trying, keeps failing to get enough votes - and that's the point
Refusing to elect anyone is the safe outcome, not a failure

6. Real Consensus Systems

Nobody sane builds this from scratch

ZooKeeper
(Zab)

etcd
(Raft)

Consul
(Raft)

Paxos
(the original)

Paxos is famously correct and famously hard to explain. Raft was designed on purpose to be teachable - which is why most new systems reach for it.

7. When You Need This

You need consensus when:

✔ Multiple nodes must agree on one source of truth

✔ You're doing leader election for real (topic 11)

✔ Losing agreement even briefly is unacceptable

Consensus Turns "Everyone Guesses" Into "Everyone Agrees, Safely"

Don't build it yourself when:

✔ ZooKeeper, etcd, or Consul already solve it

✔ Getting Paxos/Raft subtly wrong is worse than not having it

✔ A simpler guarantee (topic 12) already covers your case

💡 Once nodes can agree on who's in charge, they can agree on who's allowed to touch something right now, too. That's Distributed Locks - topic 20, next.