Load Balancers

SYSTEM DESIGN SERIES  ·  TOPIC 06 OF 23+  ·  part of the 05 → 06 → 07 chain
1. What a Load Balancer Actually Does
Sits between clients and a pool of servers, spreading the work
  • Single entry point clients always talk to
  • Decides which backend handles each request
  • Makes horizontal scaling (topic 04) actually usable
  • Only routes cleanly when the backend is stateless (topic 05)
  • Skips servers that fail health checks - never sends traffic to a dead node
2. Health Checks - Never Route to the Dead
The LB only sends traffic where it knows something will answer
  • Active checks - LB pings a /health endpoint on a timer
  • Passive checks - LB notices real requests failing and backs off
  • A server that fails checks is pulled from rotation automatically
  • Put back in once it starts passing again - no human required
  • This is failure detection (topic 02) doing real work
3. L4 vs L7 - How Deep Does It Look?
L4 (Transport)
  • Routes by IP + port only
  • Never reads the actual content
  • Very fast - "dumb" by design
VS
L7 (Application)
  • Reads method, URL, headers, cookies
  • Can route by path or hostname
  • Smarter, but costs more CPU per request
📬 Analogy: L4 is a mail sorter reading only the zip code. L7 opens the envelope - reads the subject line, the sender, the contents - to route it precisely.
4. Load Balancing Algorithms
Different rules for picking "who's next"
Round Robin
Cycles through servers in order. Simple - assumes every request costs about the same.
Least Connections
Sends to whichever server has the fewest active requests right now.
Weighted
Bigger servers get proportionally more traffic - weight reflects real capacity.
IP Hash
Same client IP always maps to the same server - a crude form of affinity.
5. Sticky Sessions, Revisited
The same trap from topic 05, seen from the load balancer's side
  • LB pins a client to one server via cookie or IP hash
  • Still used where legacy apps keep sessions in memory
  • Uneven load - "busy" clients overload their pinned server
  • Lose that server and the pinned clients still lose their session
🔓 The real fix: push session data to a shared store. Once any server can answer any client, stickiness stops being necessary at all.
6. Where Load Balancers Live
DNS-Level
Hardware Appliance
Software (Nginx, HAProxy)
Cloud (ALB / ELB)
CDN / Edge
Different layers, same job spread the load before it breaks something
Most real stacks use several of these at once - a CDN edge in front of a cloud LB, in front of software load balancing between app pods.
7. The Load Balancer's Own Achilles Heel
Problem:
✘ One LB = one more single point of failure
✘ Restarting it drops in-flight connections
✘ Manual failover is slow and error-prone
✘ It's now guarding the door for everything behind it
The Load Balancer Must Be as Available as What It Protects
Solution:
✔ Run multiple LBs behind DNS or anycast
✔ Active-passive or active-active LB pairs
✔ Cloud-managed LBs (ALB/ELB) are HA by default
✔ Health-check the load balancer itself, too
💡 A fast backend behind a slow, uncached path doesn't help much. That's exactly where Caching (topic 07) comes in next.