Two machines, a thousand miles apart, need to understand each other perfectly, over a wire that drops things, delays things, and occasionally forgets you exist. Everything on this board is how that still works.
Two computers exchanging data need to agree on an enormous number of things at once: how bits become voltages on a wire, how those bits are grouped, how to find the other machine at all, how to notice and fix lost data, and finally what the data actually means. Solving all of that in one giant tangled step would be unmaintainable - nobody could change how packets are routed without also risking how web pages render.
The fix is the same idea behind every well-designed system: split it into layers, each solving exactly one problem, each only trusting the layer directly below it. That's the entire reason the OSI and TCP/IP models exist. Everything else in this file - DNS, HTTP, proxies, retries - is a specific answer to a specific layer's problem.
OSI has 7 layers, mostly used as a teaching/reference model. Real-world networking runs on the simpler 4-layer TCP/IP model - most engineers think in this one day to day.
Mental model: it's an envelope inside an envelope. Application data gets wrapped with a TCP header, that gets wrapped with an IP header, that gets wrapped with a link-layer header. Each layer on the receiving end only opens its own envelope and hands the contents up.
An API gateway is a reverse proxy that also understands the application: it routes by API path/version, checks auth tokens, applies rate limits, and sometimes combines several backend calls into one response - a plain reverse proxy just forwards bytes.
Tap any card to open the evidence.
The default reverse proxies and load balancers for a huge share of the web - SSL termination, caching, request routing, all in one process in front of your app servers.
Reverse proxy + CDN combined at global scale - terminate TLS, cache static content, and absorb DDoS traffic before it ever reaches origin servers.
Purpose-built API gateways: auth, rate limiting, routing to backend services, request/response transformation, all managed as configuration rather than code.
Netflix's own edge/API gateway service, built specifically to handle routing, monitoring, and resilience at their scale.current status unverified
Public recursive DNS resolvers built for speed and privacy, used as alternatives to an ISP's default (often slower, sometimes less private) resolver.
A long-standing, classic forward proxy - often used by organizations for content filtering, caching, and monitoring outbound traffic.
Change a DNS record and it can take up to its TTL for every cached copy worldwide to notice. Deploy-day surprise if not planned for.
Every client retrying a failing service at the same instant, with no backoff, doubles or triples the load on a service that's already struggling - often what actually causes an outage, not the original blip.
Too short: healthy-but-slow requests get killed and retried needlessly. Too long: a hung downstream call ties up resources until the whole service starves.
An expired cert doesn't degrade gracefully - it hard-fails every HTTPS connection the moment it lapses. A famous, entirely preventable class of outage.
Put one reverse proxy in front of everything and you've built a new single point of failure - same lesson as the load balancer needing its own HA story.
No Keep-Alive (or a pool sized too small) means connections are opened faster than they're released, until new requests simply can't get one.
In HTTP/1.1, one slow request on a connection blocks everything queued behind it. HTTP/2 fixes this at the HTTP layer but can still stall at the TCP layer - HTTP/3 (QUIC, over UDP) is the fix for that remaining case.
A cross-region TCP handshake costs roughly one round trip - say 100ms. HTTPS adds a TLS handshake on top - another 100ms (modern TLS 1.3; older TLS 1.2 could cost double that). Without Keep-Alive, that 200ms tax is paid again on every single request.
The design decision this forces: at low request volume the handshake cost barely matters. Past a certain request rate, connection reuse stops being a nice-to-have and becomes the majority of your latency budget if skipped.
| Decision | Options | Pick based on |
|---|---|---|
| Transport protocol | TCP / UDP | Need guaranteed, ordered delivery → TCP. Need raw speed, can tolerate loss → UDP |
| HTTP vs HTTPS | Plaintext / TLS-encrypted | Always HTTPS in production - the "trade-off" is a small fixed handshake cost, not a real choice anymore |
| HTTP version | HTTP/1.1 / HTTP/2 / HTTP/3 | Legacy compatibility → 1.1. Multiplexed streams over TCP → 2. Avoid TCP-level head-of-line blocking too → 3 (QUIC/UDP) |
| Proxy type | Forward / reverse | Hide/control the client's outbound traffic → forward. Hide/balance across your own servers → reverse |
| DNS TTL | Short / long | Need fast failover or frequent IP changes → short TTL. Stability, fewer lookups → long TTL |
| Retry strategy | None / fixed / exponential backoff + jitter | Idempotent operation, flaky dependency → backoff + jitter. Non-idempotent → be very careful, or don't retry at all |
| Gateway vs plain proxy | API gateway / reverse proxy | Need auth, rate limiting, routing by API semantics → gateway. Just need to forward/balance traffic → plain reverse proxy |
| Connection reuse | Keep-Alive on / off | Almost always on in production - off only for deliberately one-shot, rare connections |