Deploys & Rollback

SYSTEM DESIGN SERIES  ·  TOPIC 22 OF 23+  ·  part of the 21 → 22 → 23 chain
1. Same Bad Deploy, Very Different Blast Radius
10 users, same buggy release - watch how many actually hit it (looping)
Blue-Green: every user switches to the new version at once
Blue-Green: the bug is live for 100% of traffic before anyone can react
Blue-Green: rollback is instant - but the damage is already done to everyone
Canary: only a small slice of traffic sees the new version first
Canary: the bug shows up in that slice, monitored (topic 21), and gets caught
Canary: rollback happens before the other 90% were ever exposed
2. Why Deploys Are Risky
Every change is a bet against yesterday's working code
  • Tests catch what you thought to test for - not what you didn't
  • Production traffic, real data, and real scale find the rest
  • The goal was never "never deploy" - it's "know within seconds if this one's bad"
  • That "knowing quickly" is entirely topic 21's job
🎢 You're not trying to avoid every bump. You're trying to make sure nobody's still on the ride when you find one.
3. Deployment Strategies
Three answers to "how does new code become live code?"
Blue-Green
Two full environments. Switch all traffic at once. Instant rollback - double the infrastructure.
Canary
Small slice first, watched closely. Grows if healthy. Small blast radius, slower rollout.
Rolling
Replace instances gradually, no extra infra. Old and new run side by side, mid-rollout.
4. The Mixed-Version Problem
Canary and rolling both mean old and new code run at once
  • Old instances and new instances often hit the exact same database
  • A schema change has to work for both versions, mid-rollout
  • Don't drop a column the old code still reads. Don't rename what it doesn't expect
  • "Deploy the code" and "migrate the schema" are two separate, careful steps (topic 10)
5. Feature Flags - Deploy ≠ Release
Shipping the code and turning it on don't have to be the same moment
  • Ship the new code, dark - the flag keeps it off for everyone
  • Turn it on for 1%, then more, independent of how the code itself was deployed
  • A bad feature gets "rolled back" by flipping a flag - no new deploy required
  • Fastest rollback available: nothing to redeploy, nothing to undo
6. Automating the Safety Net
The best rollback is the one nobody had to trigger by hand
Health Checks
(topic 06)
Automated Rollback Triggers
(topic 21)
Migration Compatibility Checks
Connection Draining
(topic 06)
A human watching a dashboard is slower than a metric that reacts itself
The whole point of topic 21 was to make this possible. An automatic rollback still needs something worth watching, watched.
7. Choosing Your Deploy Strategy
Blue-Green when:
✔ You can afford to run two full environments
✔ Rollback speed matters more than blast radius
✔ The change is well-tested and low-risk already
The Best Deploy Is the One You Can Undo in Seconds, for Almost Nobody
Canary / Rolling when:
✔ Blast radius matters more than rollout speed
✔ Doubling infrastructure isn't worth it
✔ You have real monitoring (topic 21) to watch the canary
💡 Storage, communication, coordination, safety - every piece is on the table now. Topic 23 puts them together into real case studies.