# Load Balancer Internals: L4 vs L7 and the Algorithms Between

> A 'load balancer' is anywhere from a NAT rewrite to a full HTTP proxy with retry logic — and the algorithm matters less than people think while the health checking matters more. The mechanics, from ECMP and Maglev down to power-of-two-choices.

URL: https://harshith.in/blog/load-balancer-internals/
Author: Harshith Sunku (Staff Engineer) — https://harshith.in/about/
Tags: distributed-systems, networking, systems

"Put a load balancer in front" hides a stack of very different
machines: DNS answers, ECMP router hashes, kernel-level packet
rewriters, and full userspace HTTP proxies — usually several at once
(a request to a large service commonly crosses *four* balancing tiers
before touching application code). Knowing which layer does what — and
where the real failure modes live — turns "the LB is acting up" into
an actual diagnosis.

## The Two Species

**L4 balancers** operate on connections: they see 5-tuples, pick a
backend per new flow, and rewrite/encapsulate packets — NAT-style
(rewrite destination, must sit on the return path), tunnel/DSR-style
(encapsulate to the backend, which answers *directly* to the client —
the asymmetry that lets one balancer front massive egress, since
responses skip it entirely; IPVS speaks all three modes). They're
fast, cheap per connection, and blind: no headers, no requests, no
retries — one connection in, one connection out. Google's Maglev and
its descendants (Katran) are this species scaled out: many identical
L4 boxes behind router **ECMP**, using consistent hashing per flow so
that when a balancer node dies, most flows still land on the same
backend via its siblings — connection state made largely stateless.

**L7 balancers** (nginx, Envoy, HAProxy in HTTP mode, ALBs) terminate
the client's connection, parse the protocol, and originate a *new*
connection to a backend. That word "terminate" carries everything:
per-request (not per-connection) routing — which HTTP/2 made
mandatory, since one client connection multiplexes hundreds of
requests that would otherwise all pin one backend — plus retries,
timeouts, TLS offload, header-based routing, connection pooling. The
bill: the balancer is now a full proxy with buffer memory, TLS CPU,
its own connection limits, and *its own failure domain* — plus two
observability gotchas that eat debugging days: backends see the
proxy's IP (hence X-Forwarded-For / PROXY protocol), and client
idle-timeout mismatches between the tiers produce those intermittent
502s where the proxy reuses a connection the backend just closed.

## Algorithms: Less Important Than Advertised, Except When Not

- **Round robin** is fine when requests are uniform and backends
  identical — i.e., in demos. Variance is the enemy: one slow request
  pins a backend that RR keeps feeding.
- **Least connections** adapts to variance — outstanding connections
  proxy for load — and is the sane default for L7 pools with mixed
  request costs.
- **Power of two choices** (pick two random backends, send to the
  less loaded) is the one with a beautiful theorem behind it:
  exponential improvement in max load over random, at O(1) cost, and
  — crucially for distributed balancers — it degrades gracefully when
  load information is *stale*, where global least-connections
  herd-stampedes the momentarily-idlest node. It's Envoy's default
  for a reason.
- **Hashing** (client IP, cookie, header) is for stickiness/affinity
  — session state, cache locality — and is a *constraint*, not a
  balancer: hash-based schemes inherit hot-key imbalance and break
  sessions on pool resize unless consistent/ring hashing (Maglev,
  ketama) bounds the churn.

The dirty secret of the algorithm wars: in most real outages the
algorithm was irrelevant and **health checking** was everything. The
questions that matter: active checks (probe /healthz) vs passive
(watch real failures — outlier ejection); how fast a dead backend is
ejected vs how fast a flapping one oscillates; whether the check
verifies *dependencies* (a backend that's up but its database isn't
— should it fail the check and risk cascading removal of the whole
pool? — every serious LB has a "panic threshold"/fail-open mode
because someone learned this live); and **draining** — removing a
backend from rotation while letting in-flight requests finish, the
difference between deploys being invisible and deploys being a 502
spike. Slow-start/warmup ramps (new backend gets traffic gradually —
cold caches, JIT warmup) round out the list of things that page you.

The composed picture, one request deep: DNS/anycast picks a region →
ECMP sprays across L4 tier → consistent-hash to an L7 proxy → 
least-loaded/P2C to a backend — four decisions, three algorithms, and
health state at every layer. When "the LB is broken," start by asking
*which one*.

## Takeaways

- L4 moves connections (NAT/tunnel/DSR — mind the return path); L7
  terminates and re-originates, buying per-request routing (mandatory
  under H2) at proxy-failure-domain prices.
- Scaled L4 = ECMP + consistent hashing (Maglev/Katran): flow
  stability without shared state.
- Least-connections beats RR under variance; power-of-two-choices
  wins under stale/distributed load info; hashing is affinity, not
  balancing.
- Health checks, ejection thresholds, panic/fail-open behavior, and
  connection draining cause (and prevent) more incidents than any
  algorithm choice.
- Requests traverse several balancing tiers — diagnose by naming the
  tier before naming the symptom.