Schedulers feel like magic until you profile a thread-heavy workload and watch real time disappear into __schedule. A context switch is not free, and the cost is not where most people think. Let’s trace exactly what happens between the instant one thread stops running and the next one starts.

What Counts as a Context Switch

Two very different things wear the same name:

  • Voluntary — a thread blocks (I/O, a lock, sleep) and hands the CPU back.
  • Involuntary — the scheduler preempts a thread that still wanted to run.

Both end up in the same kernel routine, but the trigger matters: involuntary switches cluster under contention and are the ones that wreck tail latency.

The Three Things the Kernel Saves

When schedule() decides thread B should run instead of thread A, it has to make A resumable later and B runnable now. Three pieces of state move:

c
// Simplified from kernel/sched/core.c
static void context_switch(struct rq *rq,
                           struct task_struct *prev,
                           struct task_struct *next)
{
    prepare_task_switch(rq, prev, next);
    switch_mm_irqs_off(prev->active_mm, next->mm, next); // address space
    switch_to(prev, next, prev);                         // registers + stack
    finish_task_switch(prev);
}
  1. CPU registers — general-purpose registers, the stack pointer, and the instruction pointer get parked in thread A’s kernel stack.
  2. The address space — if B belongs to a different process, switch_mm installs B’s page tables by writing CR3 on x86.
  3. Bookkeeping — runqueue accounting, scheduler statistics, and FPU state (saved lazily on most CPUs).

Where the Cycles Actually Go

The register save/restore is cheap — tens of nanoseconds. The expensive part is the address-space change:

text
        thread A                 thread B
   ┌──────────────┐         ┌──────────────┐
   │ registers    │  fast   │ registers    │
   │ kernel stack │ ──────▶ │ kernel stack │
   └──────────────┘         └──────────────┘
          │  write CR3 (only if mm differs)
          ▼
   ┌─────────────────────────────────────────┐
   │  TLB entries for A may now be invalid    │  ← the real tax
   └─────────────────────────────────────────┘

Writing CR3 can flush the TLB. The next thread then takes a burst of TLB misses as it re-walks page tables for code and data it touches. That refill — not the register copy — is what you see in a flame graph.

This is why thread switches within the same process are cheaper than process switches: same mm, no CR3 write, TLB stays warm. It’s also why PCID (process-context identifiers) exists — tagging TLB entries so a switch doesn’t have to flush everything.

The Path, End to End

flowchart LR
    A[Thread A running] --> B{schedule called}
    B --> C[pick next: Thread B]
    C --> D[switch_mm: write CR3]
    D --> E[switch_to: swap registers + stack]
    E --> F[Thread B running]
    F --> G[TLB refills on first accesses]

Measuring It Yourself

perf makes the cost visible without guesswork:

bash
# Count context switches for a command
perf stat -e context-switches,cpu-migrations ./your_workload

# See who is switching and why
perf sched record ./your_workload
perf sched latency --sort max

If context-switches scales with load while useful work stalls, you’re paying the scheduler tax — usually from lock contention or oversubscribed threads.

Takeaways

  • A context switch saves registers, may swap the address space, and updates scheduler bookkeeping — in that order of cost.
  • The register copy is cheap; the TLB refill after a CR3 write is the real expense. Same-process thread switches avoid it.
  • PCID/tagged TLBs exist specifically to soften that cost.
  • Reach for perf stat -e context-switches and perf sched latency before blaming the scheduler — most “scheduler overhead” is contention in disguise.