# Zero-Copy I/O: Stop Paying to Move Bytes You Never Touch

> A naive file-to-socket server copies every byte four times and crosses the kernel boundary twice per chunk. sendfile collapses it to zero user-space copies — and the lineage from there (splice, MSG_ZEROCOPY, io_uring) is the story of modern fast I/O.

URL: https://harshith.in/blog/zero-copy-io/
Author: Harshith Sunku (Staff Engineer) — https://harshith.in/about/
Tags: performance, linux, networking, systems

The standard way to serve a file is a loop of `read(file, buf)` and
`write(sock, buf)` — and it's worth counting what that loop actually does
to each byte. Disk to page cache (DMA), page cache to your buffer (CPU
copy), your buffer back into a kernel socket buffer (CPU copy), socket
buffer to NIC (DMA). Four movements, two of them the CPU shoveling bytes
your code never inspects, plus two syscalls per chunk. At 10 Gbit/s that
shoveling is a meaningful slice of a core, and memory bandwidth stolen
from work that matters. Zero-copy is the family of techniques that
deletes the middle.

## sendfile: The Canonical Fix

```c
off_t off = 0;
sendfile(sock_fd, file_fd, &off, count);   /* file → socket, in-kernel */
```

One syscall; the kernel wires the page-cache pages holding the file
directly into the socket's send path. With NIC scatter-gather, not even a
kernel-internal copy happens — the driver DMAs straight from page cache,
and the "copy" that remains is a few bytes of descriptor bookkeeping:

```text
  read/write:  disk ─DMA▶ page cache ─copy▶ user buf ─copy▶ skb ─DMA▶ NIC
  sendfile:    disk ─DMA▶ page cache ────────────────────────DMA▶ NIC
```

This is nginx's `sendfile on;`, how Kafka streams log segments to
consumers at wire speed (their docs credit the design by name), and the
right answer whenever bytes flow *through* your process untouched. The
constraint is the same as the benefit: your code never sees the data —
so TLS breaks the trick (bytes must be transformed... unless the kernel
does that too: `ktls` + sendfile re-enables it, which is precisely why
kernel TLS exists), and so does any filtering/compression/inspection.

`splice` generalizes sendfile: move pages between any fd and a pipe —
the pipe serving as a kernel-space conveyor of page *references* — and
`vmsplice`/`tee` round out plumbing arbitrary fd-to-fd flows without
surfacing. (`copy_file_range` covers file-to-file, letting filesystems
go further: on reflink filesystems like Btrfs/XFS it can "copy" a
gigabyte by sharing extents — zero copy in an even stronger sense.)

## The Send Side: MSG_ZEROCOPY and Its Fine Print

For data your process *generates* (so sendfile can't apply),
`send(..., MSG_ZEROCOPY)` (4.14+) pins your user pages and lets the NIC
DMA from them directly. The fine print teaches the real economics:

- **Completion is asynchronous** — you must not touch the buffer until
  the kernel signals (via the error queue) that transmission finished.
  Buffer lifecycle management just entered your application design.
- **Pinning costs**: page-pinning and the completion machinery have fixed
  overhead — the kernel documentation itself says copies win below
  roughly 10 KB. Zero-copy is a *large-transfer* optimization, full stop.

That trade — bookkeeping overhead vs copy bandwidth — is the honest frame
for the whole topic. A 64-byte RPC response is cheaper to copy than to
pin, track, and complete. A 4 MB video segment is not. Measure with the
crossover in mind, not the slogan. (mmap-then-write, the folk zero-copy,
sits in the same ledger: it trades copies for page-fault and TLB traffic
— I've written about that trade separately.)

At the far end of the spectrum sit the frameworks that remove the kernel
from the data path entirely — DPDK polling NICs from userspace,
`AF_XDP` mapping NIC rings into a process, RDMA letting a remote NIC
write your memory — where "zero-copy" is table stakes and the design
cost is absorbing driver-level responsibility. And orthogonally,
**io_uring** attacks the *other* tax from the counting exercise: the
per-operation syscall. Shared submission/completion rings mean batches
of I/O with amortized-to-zero crossings, plus registered buffers to make
repeated zero-copy sends cheap. Modern fast-path design is those two
multiplied: copies eliminated where bytes are big, crossings eliminated
where operations are many.

## Takeaways

- Count the copies: naive relay moves each byte 4× with 2 CPU copies;
  sendfile deletes both CPU copies for pass-through data.
- Kernel TLS exists to keep sendfile viable under HTTPS; splice/tee/
  copy_file_range extend in-kernel movement to arbitrary plumbing.
- MSG_ZEROCOPY trades copies for pinning + async completion — wins only
  above ~10 KB; small messages should just copy.
- Bypass stacks (DPDK/AF_XDP/RDMA) and io_uring attack the two remaining
  taxes: the kernel data path and the per-op syscall.
- Zero-copy applies to bytes you don't inspect; the moment code must see
  data, you're choosing where the one necessary copy lives.