“It’s always DNS” is a joke with a mechanism behind it: name resolution is the rare system where a single lookup crosses six administrative domains — your process, your OS, your router, your ISP or chosen resolver, a TLD, and the target’s DNS host — each with its own cache and failure modes. When it breaks, people flail because they’re debugging a path they’ve never actually traced. Trace it once, and every DNS incident becomes a binary-search over layers.

Layer 0: Before Any Network — Your Own Box

getaddrinfo() doesn’t do DNS; it does name service switching. /etc/nsswitch.conf’s hosts: line decides the order — typically files dns or, on systemd distros, mymachines resolve [!UNAVAIL=return] files ... dns. Consequences people rediscover weekly:

  • /etc/hosts wins over everything (which is both the oldest debugging trick and the oldest “why does this box resolve differently” answer).
  • dig and your application can disagree, by design: dig speaks straight DNS to whatever’s in resolv.conf, while the app went through nsswitch — through systemd-resolved’s cache, mDNS for .local, container-runtime magic for service names. “dig works but curl doesn’t” is a nsswitch-layer statement, not a paradox.
  • systemd-resolved adds a stub listener at 127.0.0.53 with its own cache and per-interface DNS routing (resolvectl status shows the real upstreams; resolvectl query asks the same path apps use). In containers, /etc/resolv.conf is injected by the runtime and points at more middleware (kube-dns/CoreDNS — with its own famous quirks like ndots:5 making every external lookup try five cluster suffixes first).

The Wire: Recursion Downward

Assuming the query escapes your machine, it hits a recursive resolver (ISP, corporate, or 1.1.1.1/8.8.8.8), whose job is the actual walk:

text
  you ─▶ recursive ─▶ root: "com. is at these 13 letter-servers"    (NS)
                  ─▶ .com: "example.com's NS is ns1.provider.net"   (NS+glue)
                  ─▶ ns1.provider.net: "www.example.com A 93.184.216.34"
      ◀─ answer, cached for TTL

dig +trace www.example.com performs exactly this walk in front of you — the single best DNS-education command. In practice the resolver rarely starts at the root: every intermediate NS referral is cached, so the walk usually begins at the deepest cached delegation. Also on the wire: UDP with TCP fallback on truncation (large answers, DNSSEC), EDNS0 buffer-size negotiation, and — increasingly — the whole first hop wrapped in DoT/DoH, which moves the privacy boundary but changes nothing else in the model.

Caching: The Feature That Is Also the Incident

Every layer caches by the record’s TTL, and every classic DNS incident is a cache-coherence story:

  • Propagation is a myth — nothing propagates; caches expire. A migration cut over “an hour ago” still serves old IPs to whoever cached the record 3,599 seconds ago with a 3600 TTL. The professional move is lowering TTL to 60 a day before the change (you must wait out the old TTL before the new one applies), then raising it after.
  • Negative caching: NXDOMAIN answers carry their own TTL (from the zone’s SOA). Ask for a record before creating it and you’ve cached its nonexistence — the deploy-then-it-still-404s classic.
  • Resolvers clamp: some ignore very low TTLs, some (and some apps — the JVM’s networkaddress.cache.ttl defaulting to forever with a security manager was legendary) cache far longer than told. End-to-end behavior is min/max over layers you don’t control.

The debugging ladder, then, follows the path: resolvectl query (what apps see) → dig @127.0.0.53, dig @<upstream> (whose cache is stale?) → dig +trace (is the authoritative data even right?) → check serials with dig SOA when primaries and secondaries disagree. Ninety percent of “it’s always DNS” resolves to one layer’s cache or one file on the local box — the mystique is just the path being invisible.

Takeaways

  • Resolution = nsswitch (files/resolved/mDNS) → stub → recursive → root/TLD /authoritative; dig bypasses the first two layers, which is why it can “work” while apps fail.
  • dig +trace shows the authoritative walk; resolvectl shows the local truth on systemd machines; containers add their own resolv.conf layer.
  • Nothing propagates — caches expire. Lower TTLs a day ahead of changes, and mind negative caching before first-time lookups.
  • Layers clamp and override TTLs (resolvers, JVMs); end-to-end freshness is the worst cache on the path.
  • Debug in path order, one layer per step; DNS mystique is just an untraced path.