The standard way to serve a file is a loop of read(file, buf) and write(sock, buf) — and it’s worth counting what that loop actually does to each byte. Disk to page cache (DMA), page cache to your buffer (CPU copy), your buffer back into a kernel socket buffer (CPU copy), socket buffer to NIC (DMA). Four movements, two of them the CPU shoveling bytes your code never inspects, plus two syscalls per chunk. At 10 Gbit/s that shoveling is a meaningful slice of a core, and memory bandwidth stolen from work that matters. Zero-copy is the family of techniques that deletes the middle.

sendfile: The Canonical Fix

c
off_t off = 0;
sendfile(sock_fd, file_fd, &off, count);   /* file → socket, in-kernel */

One syscall; the kernel wires the page-cache pages holding the file directly into the socket’s send path. With NIC scatter-gather, not even a kernel-internal copy happens — the driver DMAs straight from page cache, and the “copy” that remains is a few bytes of descriptor bookkeeping:

text
  read/write:  disk ─DMA▶ page cache ─copy▶ user buf ─copy▶ skb ─DMA▶ NIC
  sendfile:    disk ─DMA▶ page cache ────────────────────────DMA▶ NIC

This is nginx’s sendfile on;, how Kafka streams log segments to consumers at wire speed (their docs credit the design by name), and the right answer whenever bytes flow through your process untouched. The constraint is the same as the benefit: your code never sees the data — so TLS breaks the trick (bytes must be transformed… unless the kernel does that too: ktls + sendfile re-enables it, which is precisely why kernel TLS exists), and so does any filtering/compression/inspection.

splice generalizes sendfile: move pages between any fd and a pipe — the pipe serving as a kernel-space conveyor of page references — and vmsplice/tee round out plumbing arbitrary fd-to-fd flows without surfacing. (copy_file_range covers file-to-file, letting filesystems go further: on reflink filesystems like Btrfs/XFS it can “copy” a gigabyte by sharing extents — zero copy in an even stronger sense.)

The Send Side: MSG_ZEROCOPY and Its Fine Print

For data your process generates (so sendfile can’t apply), send(..., MSG_ZEROCOPY) (4.14+) pins your user pages and lets the NIC DMA from them directly. The fine print teaches the real economics:

  • Completion is asynchronous — you must not touch the buffer until the kernel signals (via the error queue) that transmission finished. Buffer lifecycle management just entered your application design.
  • Pinning costs: page-pinning and the completion machinery have fixed overhead — the kernel documentation itself says copies win below roughly 10 KB. Zero-copy is a large-transfer optimization, full stop.

That trade — bookkeeping overhead vs copy bandwidth — is the honest frame for the whole topic. A 64-byte RPC response is cheaper to copy than to pin, track, and complete. A 4 MB video segment is not. Measure with the crossover in mind, not the slogan. (mmap-then-write, the folk zero-copy, sits in the same ledger: it trades copies for page-fault and TLB traffic — I’ve written about that trade separately.)

At the far end of the spectrum sit the frameworks that remove the kernel from the data path entirely — DPDK polling NICs from userspace, AF_XDP mapping NIC rings into a process, RDMA letting a remote NIC write your memory — where “zero-copy” is table stakes and the design cost is absorbing driver-level responsibility. And orthogonally, io_uring attacks the other tax from the counting exercise: the per-operation syscall. Shared submission/completion rings mean batches of I/O with amortized-to-zero crossings, plus registered buffers to make repeated zero-copy sends cheap. Modern fast-path design is those two multiplied: copies eliminated where bytes are big, crossings eliminated where operations are many.

Takeaways

  • Count the copies: naive relay moves each byte 4× with 2 CPU copies; sendfile deletes both CPU copies for pass-through data.
  • Kernel TLS exists to keep sendfile viable under HTTPS; splice/tee/ copy_file_range extend in-kernel movement to arbitrary plumbing.
  • MSG_ZEROCOPY trades copies for pinning + async completion — wins only above ~10 KB; small messages should just copy.
  • Bypass stacks (DPDK/AF_XDP/RDMA) and io_uring attack the two remaining taxes: the kernel data path and the per-op syscall.
  • Zero-copy applies to bytes you don’t inspect; the moment code must see data, you’re choosing where the one necessary copy lives.