Why Distributed Consensus Needs a Quorum
Replicate data across five machines and you'd think any one of them could answer. But to stay consistent through failures, a majority has to agree on every decision. Here's the arithmetic behind quorums.
Replicate data across five machines and you'd think any one of them could answer. But to stay consistent through failures, a majority has to agree on every decision. Here's the arithmetic behind quorums.
Raft won because you can hold the whole protocol in your head: terms, a leader, and one log-matching rule. The mechanics of election and replication — and the operational corners (fsync, membership changes, pre-vote) where deployments actually bleed.
Wall clocks on different machines disagree by milliseconds — eternity for a fast system. Lamport's insight: you don't need time, you need causality. Lamport clocks, vector clocks, and the hybrid compromise running inside your database.
Fixed windows let 2x through at the boundary, token buckets are three lines of arithmetic, and the distributed version is a consistency problem wearing an ops costume. The algorithms, their failure modes, and what to return when you say no.
Between your server and each user sit browser caches, CDN tiers, and shared proxies you don't operate — all obeying Cache-Control headers you may have set by accident. The freshness/validation model, and the headers that cause incidents.
A 'load balancer' is anywhere from a NAT rewrite to a full HTTP proxy with retry logic — and the algorithm matters less than people think while the health checking matters more. The mechanics, from ECMP and Maglev down to power-of-two-choices.
Every queue advertises delivery guarantees; every distributed-systems engineer eventually learns the ack-crash window makes duplicates inevitable. Why exactly-once is really exactly-once *processing*, and the idempotency patterns that make it true.
hash(key) % N reshuffles nearly everything when N changes — a cache stampede with a math degree. Consistent hashing moves only 1/N of keys per node change, and virtual nodes fix the balance problem the elegant version hides.