Load Balancer & API Gateway - The Complete Guide

A STANDALONE REFERENCE  ·  WHAT ACTUALLY SITS BETWEEN A CLIENT AND YOUR SERVICES
Two different problems, often solved by boxes sitting in the same place in your architecture. One: a single server has a ceiling - CPU, memory, open connections - and traffic keeps growing past it, and a single server is also a single thing that can die. Two: a modern system usually isn't one program, it's dozens of small services, and no client outside should have to know that, or re-prove who they are to each one separately. A load balancer solves the first problem. An API gateway solves the second - and often the first too, on the way in. This page covers both, and exactly where they stop being the same thing.
1. First Principles - Why Something Has to Stand in Between
Neither of these exists because someone liked extra infrastructure
  • The load problem: one machine can only hold so many open connections and do so much work per second - and traffic doesn't stop growing to be polite
  • The failure problem: one machine is also one thing that can crash, get overloaded, or get rebooted - taking the whole service down with it if it's the only copy
  • The many-services problem: a real system is rarely one program - it's often dozens of small services, and a client shouldn't need a map of all of them
  • The repeated-work problem: authentication, rate limiting, logging - reimplementing these in every service separately is exactly how inconsistent bugs multiply
  • The split: a load balancer answers "which healthy server handles this?" An API gateway answers "what should even happen to this request, and which of my services does it actually belong to?" - a bigger question, asked earlier
2. How a Load Balancer Actually Works
One job: pick a healthy server, and stop picking the ones that stopped being healthy (looping)
  • Every incoming request gets handed to one of several identical backend servers, spreading the load
  • A health check quietly pings every server on a schedule - miss enough checks, and you're pulled out of rotation
  • Nothing about "which algorithm picked you" matters if you're not healthy in the first place - health checks come first, algorithm second
3. How an API Gateway Actually Works
The questions get asked before anyone picks a server at all (looping)
  • First: is this caller even allowed in? Then: are they within their rate limit (topic 18 of the main series)?
  • Then: which of my many backend services does this request actually belong to - routing, not load balancing
  • The thing a plain load balancer never does: a gateway can call two or three services for one client request and hand back a single combined answer
4. Load Balancer vs API Gateway - What's Actually Different
The question people mix up most often
Load Balancer
  • Decides which server - that's the whole job
  • Doesn't look inside the request much, if at all
  • Usually one kind of backend behind it
VS
API Gateway
  • Decides what happens - auth, limits, routing, shaping
  • Reads the request closely: path, headers, token, body
  • Often fronts many different backend services
🚪 Analogy: a load balancer is a host seating you at whichever open table fits your party. A gateway is the front desk that checks your reservation, figures out which restaurant in the building you actually meant, and sometimes brings dishes from two different kitchens to your table.
5. L4 vs L7 Load Balancing
How much of the request the load balancer is even allowed to look at
Layer 4 (Transport)
  • Sees IP addresses and ports - nothing else
  • Extremely fast, very cheap to run
  • Can't route by URL path, header, or cookie
Layer 7 (Application)
  • Reads the actual HTTP request - path, headers, cookies
  • Can route /api/orders one way, /api/users another
  • More work per request - a real, if usually small, cost
✉️ Analogy: L4 is a mail sorter reading only the ZIP code on the envelope. L7 opens the envelope and reads the actual letter before deciding where it goes.
6. Load Balancing Algorithms
Four different rules for the same question: which server, this time?
Round Robin
Take turns, in order. Dead simple - assumes every server and every request costs about the same, which isn't always true.
Least Connections
Send it to whoever's currently doing the least work. Adapts to slow requests better than round robin does.
Weighted
Some servers are just bigger. Give the beefier boxes a proportionally bigger share of traffic.
IP Hash / Consistent Hashing
Same client, same server, every time - useful for sticky sessions, and ties directly to topic 15's hash ring.
7. API Gateway Patterns
Where the gateway logic actually lives - this has changed a lot over the years
Single / Centralized Gateway
One gateway in front of everything. Simple, one place to enforce every rule - but a big blast radius if it slows down, and every team's release now shares one bottleneck.
Backend-for-Frontend (BFF)
A separate, smaller gateway per client type - mobile, web, partner API. Each gets exactly the shape of response it needs, at the cost of some duplicated logic across them.
Service Mesh / Sidecar
Every service gets its own tiny proxy (Envoy, Istio) sitting right next to it. Routing, retries, and auth happen at every hop, not just at the edge - more moving parts, but no single choke point.
8. Real-World Architecture - Who Actually Builds This
Names worth recognizing, not just concepts
Nginx / HAProxy
AWS ALB / NLB
Envoy
Kong / Apigee /
AWS API Gateway
Istio
(service mesh)
  • Nginx / HAProxy - the classic software load balancers, L4 and L7, still everywhere
  • AWS ALB / NLB (and equivalents on other clouds) - managed L7 / L4 load balancers, no server to patch yourself
  • Envoy - a modern proxy built for exactly this job, the data plane most service meshes are actually built on
  • Kong, Apigee, AWS API Gateway - dedicated API gateway products: auth, rate limiting, and routing as a managed layer
  • Istio - a full service mesh control plane, usually running Envoy as its sidecar in every pod
9. Failure Scenarios & Edge Cases
The load balancer needs its own redundancy - or you just moved the single point of failure (looping)
  • Health-check flapping: a server that's borderline-slow can get marked unhealthy, healthy, unhealthy again - repeatedly yanked in and out of rotation
  • Thundering herd on recovery: a server rejoins after being down, and the load balancer immediately floods it with a full share of traffic before it's actually warmed up
  • Cascading failures: without a circuit breaker, one slow backend can back up connections at the gateway until it takes healthy backends down with it
  • Sticky sessions vs horizontal scaling: pin a client to one server for session state, and you've quietly reintroduced topic 05's stateful-server problem
10. Scale & Numbers
The math behind why huge systems have layers of load balancers, not just one
📊 When one load balancer stops being enough
A single well-tuned software load balancer can realistically sustain somewhere around 50,000-100,000 concurrent connections and tens of thousands of requests per second for typical HTTP traffic. Say your peak load is 500,000 requests per second. Even at a generous 50,000 RPS per instance, that's 10 load balancer instances just to absorb the traffic - and now those 10 need their own upstream distribution too, because a single DNS record pointed at 10 IPs doesn't guarantee even spread once client-side DNS caching gets involved. This is exactly why huge systems run tiers: a cloud L4 load balancer (handling millions of raw connections) in front of a fleet of L7 gateways (handling the actual routing logic) in front of the real service instances.
11. Trade-offs at a Glance
Every decision above, on one page, for whenever you actually need to choose
DecisionOptionWhat you getWhat it costs you
LB LayerL4Extremely fast, very cheapCan't route by path, header, or cookie
L7Smart, content-aware routingMore work per request
LB AlgorithmRound RobinDead simpleIgnores real server load
Least ConnectionsAdapts to slow requestsNeeds live connection tracking
WeightedMatches uneven hardwareWeights need manual tuning
IP Hash / Consistent HashingSticky, predictable routingUneven load if traffic is skewed
Gateway PatternCentralizedOne place to enforce every ruleSingle choke point, shared blast radius
Backend-for-FrontendTailored response per client typeDuplicated logic across gateways
Service Mesh / SidecarNo single choke point, per-hop controlReal operational complexity
12. Interview Framing
The questions that actually get asked, and the sharpest honest answer to each
Q: What's the actual difference between a load balancer and an API gateway?
A load balancer decides which server; an API gateway decides what should even happen to the request - auth, rate limiting, routing to the right service, sometimes combining several services into one response - often before a load balancer's question is even asked.
Q: How do you make the load balancer itself highly available?
Don't run just one. Use an active-passive pair with a floating/virtual IP that fails over automatically, or DNS-based failover across multiple LB endpoints. A single load balancer without its own redundancy is still a single point of failure - you've just moved it.
Q: When would you choose a service mesh over a centralized API gateway?
When most of the traffic is service-to-service, internal, and high-volume - a central gateway becomes a bottleneck and a shared point of failure for that traffic. A mesh distributes routing and retries to a sidecar next to every service instead, at the cost of real operational complexity.
Q: What happens when a recovering server gets slammed right after coming back online?
Thundering herd on recovery - the load balancer treats it as instantly healthy and sends a full share of traffic before caches or connection pools are warm. The fix is a slow-start ramp: gradually increasing the share of traffic a recovered server receives instead of switching it on at 100%.
Q: Why can sticky sessions be a problem?
Pinning a client to one server for session state quietly reintroduces the stateful-server problem - that server becomes special, harder to scale horizontally, and a bigger deal to lose. The more robust fix is usually externalizing session state to a shared store, so any server can serve any request.
13. Putting It All Together - A Decision Guide
Three real situations, and what actually belongs in front of them
🔌 A simple app, one backend service, moderate traffic
Just a load balancer. Round robin or least connections, L7 if you ever need path-based routing, health checks non-negotiable. No gateway needed yet - you don't have enough services to justify one.
🌐 A public API for third-party developers
An API gateway in front, doing real work: API keys, rate limiting per customer (topic 18), request/response transformation, versioning - with a load balancer behind it distributing to the actual service instances.
🕸️ Dozens of internal microservices, mostly talking to each other
A service mesh (sidecars) for the internal, service-to-service traffic - that volume shouldn't funnel through one central gateway. Keep a lighter gateway only at the actual edge, for the traffic that comes from outside.
14. Quick Reference - The Whole Thing, One Place
For whenever you just need the short version
Choosing an LB algorithm:
✔ Requests cost about the same → Round Robin
✔ Request cost varies a lot → Least Connections
✔ Servers aren't identical → Weighted
✔ Need the same client hitting the same server → IP Hash
Choosing a gateway pattern:
✔ One consistent client type → Centralized gateway
✔ Very different client needs (mobile/web/partner) → BFF
✔ Heavy internal service-to-service traffic → Service mesh
✔ Not many services yet → Skip the gateway entirely
🏁 A load balancer answers "which server?" An API gateway answers "what should even happen here?" Most real systems eventually need an honest answer to both - just not necessarily from the same box.