# epoll: The Syscall Under Every Event Loop

> nginx, Node, Redis, and every async runtime you use sit on one Linux primitive. Why select died at scale, how epoll inverts the cost model, the level- vs edge-triggered trap, and the thundering herd that took years to fix.

URL: https://harshith.in/blog/epoll-and-event-loops/
Author: Harshith Sunku (Staff Engineer) — https://harshith.in/about/
Tags: linux, networking, performance, systems

Serve ten thousand connections with a thread each and you buy ten thousand
stacks and a scheduler groaning under wakeup storms. The alternative that
won — one thread, many sockets, act only on the *ready* ones — needs the
kernel to answer a question efficiently: *which of my 10,000 fds can make
progress right now?* The history of that question (Dan Kegel's "C10K
problem" essay named it in 1999) is the history of select → poll → epoll,
and the details still leak into every async runtime you use today.

## Why select and poll Lost

`select`/`poll` are stateless: every call hands the kernel the entire fd
list, the kernel walks it all, sets up wait entries, tears them down, and
returns — O(n) *per call*, at every wakeup, forever (select adds insult:
a 1024-fd bitmap cap compiled into everything). The observation that kills
them at scale: between two calls, your interest set barely changed — you
watch the same 10,000 sockets — yet you re-describe it every time. At 10k
connections with 100 active, you pay for 10,000 to harvest 100.

epoll makes the interest set **kernel-resident state**:

```c
int ep = epoll_create1(0);
struct epoll_event ev = { .events = EPOLLIN, .data.fd = sock };
epoll_ctl(ep, EPOLL_CTL_ADD, sock, &ev);        /* once per fd */

for (;;) {
    int n = epoll_wait(ep, events, 64, -1);      /* O(ready), not O(watched) */
    for (int i = 0; i < n; i++) handle(events[i]);
}
```

Registration hooks each file's wait queue; when data arrives, the wakeup
callback moves that fd onto the epoll instance's ready list. `epoll_wait`
just drains the ready list — cost proportional to *activity*, independent
of the watch count. That inversion is the whole trick, and it's why one
nginx worker holds 100k idle connections for approximately free. (BSD's
kqueue got there first and more generally; io_uring is the next
generation — completion- rather than readiness-based; different post.)

## The Trap Everyone Falls Into: LT vs ET

epoll has two notification modes, and the difference has eaten weeks of
engineer-time industry-wide:

- **Level-triggered** (default): "fd is readable" — reported *as long as*
  data remains. Forgiving: read some, get notified again.
- **Edge-triggered** (`EPOLLET`): "fd *became* readable" — reported once
  per transition. Read only part of the buffer and the rest waits
  silently forever; no new bytes, no new edge. **The ET contract is
  non-negotiable: loop until EAGAIN, on every event, with non-blocking
  fds.** The classic symptom — connections that hang until an unrelated
  byte arrives — is an ET loop that stopped early.

Why suffer ET at all? Fewer wakeups under load (no re-reporting of
half-drained buffers), and it composes with `EPOLLONESHOT` for
multi-threaded designs where exactly one worker should own an event.
Runtimes split honestly here: nginx runs ET; libuv (Node) runs LT and
eats the extra wakeups as the price of simplicity.

Two more sharp edges from the same drawer. **Accept storms/thundering
herd**: N workers epolling one listen socket all wake per connection;
one accepts, N−1 burned a scheduler trip. `EPOLLEXCLUSIVE` (4.5) wakes
one; `SO_REUSEPORT` shards the listen queue itself — the modern default
for prefork servers. And **stale events**: epoll reports on the *open
file description*, not the fd number — close an fd while events are in
flight (or after dup) and you can process an event for a connection you
already replaced; the `data.ptr`-to-connection-object pattern plus
deferred frees exists precisely because of this.

## The Shape It Imposes

The syscall's economics dictate the architecture above it: since one
thread multiplexes everything, **nothing may block** — every fd
non-blocking, every slow operation (disk I/O, DNS, sleep) pushed to a
side pool or its own async path, callbacks/coroutines to re-linearize
the logic. When people say "Node is single-threaded" or ask why Redis
commands must be fast, they're describing epoll's contract: the loop is
only as responsive as its slowest callback. The event loop is not a
framework fashion; it's the shape the readiness model carved into
software.

## Takeaways

- select/poll re-scan the world per call; epoll keeps interest in the
  kernel and returns only the ready — O(activity) beats O(watched).
- Level-triggered is forgiving; edge-triggered demands drain-to-EAGAIN
  on non-blocking fds — half-read hangs are the signature failure.
- EPOLLEXCLUSIVE and SO_REUSEPORT are the herd-taming tools for shared
  listeners; ONESHOT hands events to exactly one thread.
- Events bind to the file description: close/dup carelessly and you'll
  process ghosts — track connections by pointer and defer frees.
- The no-blocking rule isn't style, it's arithmetic: one stalled
  callback stalls every connection the loop owns.