Serve ten thousand connections with a thread each and you buy ten thousand stacks and a scheduler groaning under wakeup storms. The alternative that won — one thread, many sockets, act only on the ready ones — needs the kernel to answer a question efficiently: which of my 10,000 fds can make progress right now? The history of that question (Dan Kegel’s “C10K problem” essay named it in 1999) is the history of select → poll → epoll, and the details still leak into every async runtime you use today.

Why select and poll Lost

select/poll are stateless: every call hands the kernel the entire fd list, the kernel walks it all, sets up wait entries, tears them down, and returns — O(n) per call, at every wakeup, forever (select adds insult: a 1024-fd bitmap cap compiled into everything). The observation that kills them at scale: between two calls, your interest set barely changed — you watch the same 10,000 sockets — yet you re-describe it every time. At 10k connections with 100 active, you pay for 10,000 to harvest 100.

epoll makes the interest set kernel-resident state:

c
int ep = epoll_create1(0);
struct epoll_event ev = { .events = EPOLLIN, .data.fd = sock };
epoll_ctl(ep, EPOLL_CTL_ADD, sock, &ev);        /* once per fd */

for (;;) {
    int n = epoll_wait(ep, events, 64, -1);      /* O(ready), not O(watched) */
    for (int i = 0; i < n; i++) handle(events[i]);
}

Registration hooks each file’s wait queue; when data arrives, the wakeup callback moves that fd onto the epoll instance’s ready list. epoll_wait just drains the ready list — cost proportional to activity, independent of the watch count. That inversion is the whole trick, and it’s why one nginx worker holds 100k idle connections for approximately free. (BSD’s kqueue got there first and more generally; io_uring is the next generation — completion- rather than readiness-based; different post.)

The Trap Everyone Falls Into: LT vs ET

epoll has two notification modes, and the difference has eaten weeks of engineer-time industry-wide:

  • Level-triggered (default): “fd is readable” — reported as long as data remains. Forgiving: read some, get notified again.
  • Edge-triggered (EPOLLET): “fd became readable” — reported once per transition. Read only part of the buffer and the rest waits silently forever; no new bytes, no new edge. The ET contract is non-negotiable: loop until EAGAIN, on every event, with non-blocking fds. The classic symptom — connections that hang until an unrelated byte arrives — is an ET loop that stopped early.

Why suffer ET at all? Fewer wakeups under load (no re-reporting of half-drained buffers), and it composes with EPOLLONESHOT for multi-threaded designs where exactly one worker should own an event. Runtimes split honestly here: nginx runs ET; libuv (Node) runs LT and eats the extra wakeups as the price of simplicity.

Two more sharp edges from the same drawer. Accept storms/thundering herd: N workers epolling one listen socket all wake per connection; one accepts, N−1 burned a scheduler trip. EPOLLEXCLUSIVE (4.5) wakes one; SO_REUSEPORT shards the listen queue itself — the modern default for prefork servers. And stale events: epoll reports on the open file description, not the fd number — close an fd while events are in flight (or after dup) and you can process an event for a connection you already replaced; the data.ptr-to-connection-object pattern plus deferred frees exists precisely because of this.

The Shape It Imposes

The syscall’s economics dictate the architecture above it: since one thread multiplexes everything, nothing may block — every fd non-blocking, every slow operation (disk I/O, DNS, sleep) pushed to a side pool or its own async path, callbacks/coroutines to re-linearize the logic. When people say “Node is single-threaded” or ask why Redis commands must be fast, they’re describing epoll’s contract: the loop is only as responsive as its slowest callback. The event loop is not a framework fashion; it’s the shape the readiness model carved into software.

Takeaways

  • select/poll re-scan the world per call; epoll keeps interest in the kernel and returns only the ready — O(activity) beats O(watched).
  • Level-triggered is forgiving; edge-triggered demands drain-to-EAGAIN on non-blocking fds — half-read hangs are the signature failure.
  • EPOLLEXCLUSIVE and SO_REUSEPORT are the herd-taming tools for shared listeners; ONESHOT hands events to exactly one thread.
  • Events bind to the file description: close/dup carelessly and you’ll process ghosts — track connections by pointer and defer frees.
  • The no-blocking rule isn’t style, it’s arithmetic: one stalled callback stalls every connection the loop owns.