“I trace, profile, and prove it” is the tagline of this whole site, and the flame graph is the single most efficient tool for the prove it part. I lean on it enough that I built a profiler around live flame graphs streamed to a browser. But the tool only helps if you can read it correctly — and the common misreadings (mistaking depth for cost, missing that the profiler only sees on-CPU time) send people optimizing the wrong function. Here’s how sampling works and how to read the picture in ten seconds flat.

Sampling: Statistics, Not Surveillance

A sampling profiler like perf doesn’t trace every function — that would be ruinously slow and would distort the thing it measures. Instead it interrupts the program at a fixed frequency (say 99 Hz — 99, not 100, to avoid lockstep with periodic timers) and records the call stack at that instant:

text
  99 times/sec:  freeze → walk the stack → log it → resume
  after 30s:  ~3000 stacks, each a snapshot of "what was running"

The statistical bet is simple and sound: a function that appears in 50% of samples was on-CPU ~50% of the time. You don’t need to catch every call; you need enough samples that the proportions are trustworthy — which is why short profiles of rare events are unreliable and why 99 Hz over 30 seconds beats 4000 Hz over one. Low overhead (a few percent) is the whole reason sampling beats instrumentation for production profiling.

Reading the Graph in Ten Seconds

Thousands of stacks are unreadable as a list. The flame graph (Brendan Gregg’s invention, ~2011) aggregates them: identical stacks merge, and the result is drawn as stacked boxes.

text
   ┌──────────────────────────────────────────────┐  ← full profile
   │ main                                          │
   ├───────────────────────┬──────────────────────┤
   │ handle_request        │ background_flush      │
   ├───────────┬───────────┼──────────────────────┤
   │ parse     │ db_query ▓▓▓▓▓▓ (wide = HOT)      │
   └───────────┴───────────┴──────────────────────┘

The two rules that are the entire skill:

  • Width = time. A box’s width is the fraction of samples containing that frame. Wide boxes are where the CPU time goes. This is the only thing you’re hunting for.
  • Y-axis = stack depth, NOT importance. Boxes stack because one function called another. A tall tower is just deep call nesting; it says nothing about cost. The #1 beginner error is chasing tall spires — look for wide plateaus, especially wide boxes near the top (leaf functions doing actual work, not just calling down).

That’s it: scan across the top for the widest boxes, and those are your targets. Colors are usually just for visual separation (warm random palette); ignore them unless the graph specifically uses color to encode something (some tools tint by language or by whether frames are kernel vs user). Interactive versions let you click a box to zoom into that subtree — the fast path to “what is db_query spending its time on.”

The Traps That Mislead

A flame graph is honest about what it measures and silent about what it doesn’t — and the gaps are exactly where people get fooled:

  • It only shows on-CPU time. A program blocked on disk, network, a lock, or sleep is not running, so it generates no samples — time spent waiting is invisible on a standard CPU flame graph. If your service is slow but the flame graph looks idle or unremarkable, your bottleneck is off-CPU (I/O or lock contention), and you need an off-CPU flame graph (from scheduler tracing) instead. This single misunderstanding accounts for most “the profiler says nothing’s wrong but it’s slow” confusion.
  • Inlining hides functions. At -O2 the compiler inlines aggressively (see my optimization post), so a hot function may not appear as its own frame — its time is attributed to the caller it was folded into. Confusing until you remember the optimizer erased the boundary.
  • Broken stacks need frame pointers. Reliable stack walking wants frame pointers, which optimized builds often omit (-fno-omit-frame-pointer to keep them), or DWARF/LBR-based unwinding as the fallback. Symptoms are truncated towers or [unknown] frames — a collection problem, not a code problem. (Fedora and others flipped frame pointers back on by default in 2023 largely to make production profiling trustworthy again.)
  • Sampling misses the rare. A function called once for 5 ms in a 30 s profile may not appear at all. Flame graphs answer “where does the bulk go,” not “did this specific thing happen.”

Used with those caveats in mind, it’s the fastest path from “the system is slow” to a specific function and line — capture with perf record -F 99 -g -- ./app, fold, and render, or use any of the tools that do it live. The discipline is always the same: measure, read width not height, and check whether your time is even on-CPU before you trust the picture.

Takeaways

  • Sampling profilers snapshot the stack at fixed frequency; frame frequency ≈ time spent — low overhead, statistically sound over enough samples.
  • Read width, not height: wide boxes (especially wide leaves at the top) are the cost; tall towers are just deep call chains.
  • Standard flame graphs show only on-CPU time — I/O, lock, and sleep waits are invisible; use off-CPU graphs when the CPU view looks innocent but the service is slow.
  • Inlining folds hot functions into callers; missing/[unknown] frames mean you need frame pointers or DWARF unwinding, not a code fix.
  • Capture with perf record -F 99 -g; the workflow turns “it’s slow” into “it’s this function” in minutes — the proof step behind trace-and-prove.