How Things Work

Load Balancers & Reverse Proxies

How one address fans traffic across a fleet — L4 vs L7, the distribution algorithms, and the health checks that keep a crashed server from ever serving a failed request.

How It Works

A load balancer is the front door to a horizontally scaled system. It accepts every client request and forwards it to one healthy backend, distributing load by an algorithm and removing failed nodes automatically. It's what makes a pool of stateless servers look like one reliable service.

A Single Front Door

Clients connect to one stable address (the load balancer's VIP). The LB sits between users and a pool of backends, forwarding each request to one of them. Backends can be added, removed, or restarted without clients ever noticing.

L4 vs L7

Layer-4 LBs route on IP/port (TCP/UDP) — fast and protocol-agnostic but blind to content. Layer-7 LBs understand HTTP, so they can route by path/host/header, terminate TLS, retry failed requests, and rewrite responses. Most app traffic uses L7.

Choosing a Backend

The algorithm decides who gets the request: Round Robin (in turn), Least Connections (fewest active — great for uneven workloads), Weighted (proportional to capacity), or IP/consistent Hash (same client → same server). The right choice depends on whether requests are uniform.

Health Checks & Failover

The LB continuously probes each backend (active checks) or watches for errors (passive checks). An unhealthy node is pulled from rotation within seconds and re-added once it recovers — so one crashed instance never serves a single failed request.

Key Concepts

🚪Reverse Proxy

A server that accepts client requests and forwards them to backends, hiding the pool behind one address. A load balancer is a reverse proxy that also distributes load.

🧱L4 vs L7

Layer-4 routes on TCP/IP (fast, content-blind). Layer-7 understands HTTP and can route by path, terminate TLS, and retry.

🩺Health Check

Active probes (GET /health) or passive error-watching that keep unhealthy backends out of rotation.

📌Sticky Session

Pinning a client to one backend (via cookie or IP hash) so server-local state survives. Reduces rebalancing freedom.

🔐TLS Termination

Decrypting HTTPS at the LB so backends handle plain HTTP — offloads crypto work and centralizes certificates.

🎯Consistent Hashing

A hashing scheme where adding/removing a server reshuffles only a small fraction of keys — used for sticky routing and sharded caches.

NGINX upstream config

tsx

1# NGINX as an L7 reverse proxy + load balancer
2upstream api_backend {
3    least_conn;                 # algorithm: fewest active connections
4    server 10.0.0.11:8080 weight=3;   # bigger box → more traffic
5    server 10.0.0.12:8080;
6    server 10.0.0.13:8080 backup;     # only used if others are down
7}
8
9server {
10    listen 443 ssl;             # TLS terminates HERE (offloaded from app)
11    location / {
12        proxy_pass http://api_backend;
13        proxy_next_upstream error timeout http_502;  # retry next server
14        # active health check (NGINX Plus) drops unhealthy nodes:
15        # health_check interval=5s fails=3 passes=2;
16    }
17}

💡

Why This Matters

Every scaled-out system needs load balancing. The LB is where availability, rolling deploys, TLS, and traffic shaping all live. Understanding L4 vs L7 and the algorithm trade-offs is core to system-design interviews and to running services that don't fall over when one node dies.

Common Pitfalls

⚠Picking Round Robin for highly variable request durations — it creates hot spots. Use Least Connections instead.

⚠No health checks (or checking a port instead of a real /health endpoint) — the LB keeps routing to dead apps.

⚠Over-relying on sticky sessions, which defeats even distribution and breaks when a server is removed.

⚠Forgetting the LB itself is a single point of failure — run it redundantly (active/passive or anycast).

⚠Terminating TLS at the LB but leaving the LB→backend hop unencrypted on an untrusted network.

Real-World Use Cases

1Slow Requests Pile Up on One Server

Scenario

An API mixes fast reads (~10ms) with occasional heavy report endpoints (~8s). Under Round Robin, some servers get stuck behind long reports while others sit idle, and p99 latency spikes.

Problem

Round Robin assumes every request costs the same. When durations vary wildly, it keeps handing new work to a server already busy with a slow request, creating hot spots.

Solution

Switch to Least Connections so the LB routes to whichever backend currently has the fewest in-flight requests. Slow endpoints naturally accumulate on fewer servers while fast traffic flows to idle ones.

💡

Takeaway: Match the algorithm to your traffic shape. Round Robin is fine for uniform requests; Least Connections wins whenever request durations are uneven.

2Zero-Downtime Deploys

Scenario

A team needs to ship 10× a day to a fleet of 8 instances without dropping user requests during each rollout.

Problem

Restarting an instance mid-request would return errors. Naively redeploying all at once causes a brief outage.

Solution

They use the LB's health checks plus connection draining: before redeploying an instance, mark it unhealthy so the LB stops sending new requests, let in-flight requests finish (drain), deploy, then wait for the health check to pass before re-adding it. Rolling through the fleet keeps the service fully available.

💡

Takeaway: Health checks plus connection draining turn the load balancer into the backbone of safe rolling deploys — no request hits a server that's going down.

Join the discussion

Comments

Loading comments...

Cookie preferences

We use essential cookies for sign-in and preferences. With your permission, we also use basic analytics to improve lessons.