IP routing gets your packet to the right subnet; it says nothing about delivering it the last meter. Ethernet doesn’t speak IP — the NIC delivers frames to MAC addresses — so every single packet’s final hop depends on a translation: who, on this wire, is 192.168.1.7? The protocol answering that (ARP, RFC 826, sixteen lines of logic from 1982) has no security, subtle caching behavior, and a starring role in an outsized share of networking mysteries. Layer 2½ deserves more respect than it gets.

The Mechanism, Complete

Host A (192.168.1.5) wants to send to B (192.168.1.7) on the same subnet:

text
  A ──▶ broadcast (ff:ff:ff:ff:ff:ff): "who has 192.168.1.7? tell 192.168.1.5"
  B ──▶ unicast to A:                  "192.168.1.7 is at 52:54:00:ab:cd:ef"
  A caches the mapping, sends the real frame

Off-subnet destinations never get ARPed — the routing decision picks a next-hop gateway, and A ARPs for the gateway’s IP instead. That one sentence is the debugging fork: on-link problems are ARP problems; everything else is routing.

The cache (ip neigh — the neighbor table, shared conceptually with IPv6’s ND) is what makes this survivable: entries go REACHABLE for a configurable lifetime, decay to STALE, and get re-verified on next use (DELAY → PROBE, a unicast re-ask) rather than re-broadcast. Reading those states is diagnostic: a neighbor stuck FAILED means no reply on the wire (wrong VLAN, dead host, L2 filtering); INCOMPLETE means the request itself is vanishing. And the table is finite — hosts sitting on huge flat L2 networks (/16 container networks, big VM subnets) can thrash the neighbor table’s garbage-collection thresholds (gc_thresh1/2/3), producing the classic “random timeouts on a big subnet” incident whose dmesg signature is neighbour: arp_cache: table overflow.

The Tricks Built on Top

The protocol’s flexibility comes from an omission: anyone can answer, and answers are believed. Three legitimate systems exploit it:

  • Gratuitous ARP — announcing your own IP unasked. This is the beating heart of every IP-failover scheme: keepalived/VRRP takes over a virtual IP and broadcasts gratuitous ARPs so every switch and host on the segment repoints the MAC immediately, instead of after cache expiry. Failover that “takes 30 seconds for some clients” is a GARP not being sent — or being ignored.
  • Proxy ARP — a router answering for addresses behind it, letting hosts believe a remote destination is on-link. Historically how dial-up and some VPN/DSL setups stitched segments; today mostly a surprise you discover enabled (/proc/sys/net/ipv4/conf/*/proxy_arp) while debugging why a misconfigured /8 netmask works anyway.
  • ARP announcement on address config — the duplicate-address detection ping; two boxes claiming one IP produce alternating cache flaps visible as ip neigh MAC changes and intermittent 50% packet loss — one of networking’s most recognizable symptom patterns once you’ve seen it.

The same omission is the attack: ARP spoofing — answer faster and more often than the real host, become the man in the middle for a whole segment (arpspoof is thirty years old and still works on flat unprotected LANs). Which is why enterprise gear ships Dynamic ARP Inspection (validate replies against DHCP-snooping state), why hypervisors filter guest ARP by policy, and why “the segment is the trust boundary” is a rule that predates and outlives most network security fashion.

Field kit for the last hop: ip neigh (states, and whether the MAC is the one you expect), tcpdump -e arp (watch requests go unanswered — the -e prints MACs and settles who’s really talking), arping -I eth0 <ip> (targeted ask, bypasses cache), and clearing a poisoned/stale entry with ip neigh flush dev eth0. Ninety percent of “can ping the gateway but not my neighbor” resolves inside those four commands.

Takeaways

  • ARP answers the only question Ethernet cares about: which MAC owns this on-link IP. Off-link traffic ARPs for the gateway — the fork in every L2-vs-L3 debug.
  • The neighbor cache’s states (REACHABLE/STALE/PROBE/FAILED) are readable diagnostics; big flat subnets overflow it — raise gc_thresh or shrink the L2 domain.
  • Gratuitous ARP makes IP failover instant; its absence is why failovers half-work.
  • Anyone can answer: spoofing is trivial on unprotected segments — DAI/ snooping where it matters, and never treat a LAN as friendly by default.
  • tcpdump -e arp + ip neigh + arping localize nearly every last-hop mystery.