# Anatomy of a System Call: What read() Really Costs

> A function call costs ~2ns. A syscall costs ~100ns before doing any work — and that's after decades of optimization plus a Spectre-era tax. Where the time goes, why vDSO exists, and what that means for API design.

URL: https://harshith.in/blog/anatomy-of-a-syscall/
Author: Harshith Sunku (Staff Engineer) — https://harshith.in/about/
Tags: linux, kernel, performance, systems

`read(fd, buf, 4096)` looks like a function call. It is not. It's a controlled
CPU-mode transition: the one legitimate doorway from your unprivileged code into
the kernel, with hardware-enforced credentials checked at the threshold. That
doorway has a toll — small enough to ignore for a 4 KB read, large enough that
"how many times do we cross" quietly shapes the design of every fast I/O API of
the past two decades.

## The Doorway

On x86-64, glibc's `read` wrapper loads the syscall number (0 for read) into
`rax`, arguments into `rdi/rsi/rdx`, and executes the `syscall` instruction.
That single instruction: switches the CPU to ring 0, swaps in the kernel's
stack via per-CPU state, saves the return address, and lands at one fixed,
kernel-chosen entry point. Userspace doesn't get to choose *where* it enters —
that's the whole security model.

```text
  user: read(fd, buf, n)
    └─ syscall            ── ring 3 → ring 0, ~fixed cost
        └─ entry_SYSCALL_64: save regs, switch stack (+KPTI page-table swap)
            └─ sys_call_table[rax] → ksys_read()
                └─ fd table lookup → file->f_op->read_iter()
                    └─ copy_to_user(buf, ...)   ── SMAP-guarded copy
        ┌─ sysret          ── ring 0 → ring 3
  user: (continues)
```

Inside, the kernel validates *everything*: `fd` against the process's file
table, `buf` against the user address range (`copy_to_user` — the kernel never
dereferences user pointers raw; SMAP even makes accidental dereference fault).
This is why a syscall can't be inlined, trusted, or skipped: the parameters are
adversarial by definition.

`strace ./app` shows you this layer verbatim — it's the ground truth of what a
process asks of the OS, which is why it's the first tool out of the bag for
"what is this binary actually doing."

## The Bill

Measure a null-ish syscall (`getppid` in a loop, or lmbench's `lat_syscall`) and
you get **~50–100 ns** on current hardware — versus ~1–2 ns for a function call.
Where it goes: the mode switch itself (tens of cycles each way), register
save/restore, and — since 2018 — the Meltdown/Spectre tax: KPTI swaps page
tables on every entry/exit and mitigations fence speculation, which roughly
doubled syscall latency overnight on affected CPUs (`spectre_v2=off mitigations=off`
benchmarks made the point vividly; don't run them in production).

There's also an *indirect* cost that often exceeds the direct one: you return
to userspace with polluted branch predictors, evicted cache lines and TLB
entries. FlexSC-era research put this at up to half the total damage.

100 ns sounds free. Now multiply: a server doing 1M small packet ops/sec
through per-packet syscalls burns a visible fraction of a core on crossings
alone. That arithmetic — crossings × rate — explains a whole lineage of Linux
APIs:

- `writev`/`sendmmsg`/`recvmmsg`, `pwritev2`: batch N operations per crossing.
- `epoll`: one syscall reports readiness of thousands of fds.
- `sendfile`/`splice`: move data fd-to-fd without round-tripping through
  userspace buffers.
- `io_uring`: the logical endpoint — submission and completion rings in shared
  memory; in polled mode, I/O with *zero* syscalls in steady state.

## The Calls That Aren't Calls

Some "syscalls" never cross at all. `gettimeofday`, `clock_gettime`, `getcpu`:
the kernel maps a small region of code+data into every process — the **vDSO**
— and glibc calls *that*, reading a kernel-maintained timestamp page in pure
userspace. That's why timing loops don't show up in `strace` (a perennial
source of "strace says it makes no syscalls!?" confusion) and why
`clock_gettime` costs ~20 ns, not ~200. `ltrace` vs `strace` on the same binary
makes the layering visible: library call ≠ system call.

The general lesson generalizes to your own APIs: **if a boundary is expensive,
the API evolves toward batching, shared memory, or moving the common case to
the cheap side.** The syscall interface just got there decades before your
microservice did.

## Takeaways

- A syscall is a hardware privilege transition to one fixed entry point, with
  all arguments treated as hostile — ~50–100 ns before any real work.
- Post-2018 mitigations (KPTI etc.) roughly doubled the crossing cost; the
  indirect cache/predictor pollution can cost as much again.
- The cost×rate product drove epoll, sendmmsg, splice, and ultimately
  io_uring's shared-ring, zero-crossing design.
- vDSO calls (`clock_gettime`) are syscall-shaped but run in userspace — strace
  won't see them.
- `strace` for the crossing-level truth, `perf stat` for the cost; batch the
  boundary when the product gets big.