# The TCP State Machine: Where Connections Actually Live

> Eleven states, two of which cause most production incidents. Implementing TCP taught me the state diagram isn't trivia — TIME_WAIT port exhaustion, SYN backlogs, and orphaned FIN_WAIT_2 are all right there in the RFC 793 figure.

URL: https://harshith.in/blog/tcp-state-machine/
Author: Harshith Sunku (Staff Engineer) — https://harshith.in/about/
Tags: networking, systems, linux

`ss -tan | awk '{print $1}' | sort | uniq -c` is one of my first commands on
any misbehaving box, and it only means something if the state names do. When
I implemented a [TCP stack in software](https://harshith.in/projects/simple-software-based-tcp-ip-stack/), the state machine went
from an exam diagram to the actual data structure at the heart of the
protocol — and it turns out the production folklore (port exhaustion,
connection leaks, half-closed hangs) is all legible directly from the
transitions. Eleven states; here's the tour, weighted by operational pain.

## Setup: Three Packets, Two Queues

```text
  client                          server
  CLOSED                          LISTEN
    │ ── SYN ──────────────────▶  SYN_RECV
  SYN_SENT                          │
    │ ◀────────────── SYN+ACK ──   │
    │ ── ACK ──────────────────▶  ESTABLISHED
  ESTABLISHED
```

The handshake exists to synchronize *sequence numbers* — each side picks an
ISN and acknowledges the other's; both directions of the byte stream get
independent numbering. Two operational facts hide in `SYN_RECV`: the kernel
holds embryonic connections in a **SYN queue** until the final ACK promotes
them to the **accept queue** (where they wait for your `accept()` call). SYN
floods target the first queue — the SYN cookies countermeasure encodes the
connection state *into* the ISN so nothing is stored. A slow application
targets the second: when the accept queue overflows (`somaxconn`, the
`listen()` backlog), Linux silently drops handshake-completing ACKs, and
clients see mysteriously hanging connects. `ss -lt` shows both queue depths
per listener; `nstat -az TcpExtListenDrops` counts the bodies.

## Teardown: Four Packets and the Two Problem States

Close is two independent half-closes — each direction's FIN/ACK pair —
because each byte stream ends separately:

```text
  active closer                   passive closer
  ESTABLISHED                     ESTABLISHED
    │ ── FIN ──────────────────▶  CLOSE_WAIT   (app must close()!)
  FIN_WAIT_1                        │
    │ ◀──────────────────  ACK ──  │
  FIN_WAIT_2                        │
    │ ◀──────────────────  FIN ──  LAST_ACK
  TIME_WAIT ── ACK ────────────▶  CLOSED
    │ (2×MSL, then CLOSED)
```

**CLOSE_WAIT is an application bug wearing a network costume.** The remote
side closed; the kernel is waiting for *your process* to call `close()`.
CLOSE_WAIT sockets accumulating means a code path that stops reading without
closing — a leaked fd, an exception past the cleanup. No kernel tunable
fixes it; `lsof` finds the process, the fix is in your code.

**TIME_WAIT is by design — and still bites.** The active closer holds the
socket for 2×MSL (60 s on Linux) to (a) retransmit the final ACK if the
peer's FIN re-arrives and (b) let stray delayed segments die before the
4-tuple is reused, where they could corrupt a successor connection. The
famous incident shape: a service making rapid short-lived *outbound*
connections to one backend accumulates tens of thousands of TIME_WAITs, and
since the 4-tuple space to one destination is bounded by ~28k ephemeral
ports, connect() starts failing with EADDRNOTAVAIL at a few hundred new
connections/second. Real fixes, in order of correctness: **reuse connections**
(keep-alive/pooling — the actual answer), `net.ipv4.tcp_tw_reuse=1` (safe:
timestamps prove segment freshness for outbound reuse), more ephemeral ports
/ more source IPs. The folklore fix `tcp_tw_recycle` was so unsafe behind
NAT that the kernel removed it entirely in 4.12 — its ghost still haunts old
runbooks.

One more: **FIN_WAIT_2 with a dead peer.** You closed, they ACKed, their FIN
never comes (crashed, or a middlebox ate it). For orphaned sockets the
kernel times this out (`tcp_fin_timeout`); for sockets your process still
holds, the half-open connection sits until keepalive — which defaults to
*two hours idle* before the first probe. Any long-lived-connection service
that doesn't set `TCP_KEEPIDLE`/`TCP_KEEPINTVL` (or application-level
heartbeats) is betting its fd table on peers always dying politely.

## Watching the Machine Run

The whole post is checkable live: `ss -tan state time-wait`,
`ss -o state fin-wait-2` (the `-o` shows the running timers),
`watch 'ss -s'` during a load test, and `nstat` deltas for drops and
overflows. Implementing the machine yourself is the deep version — my
enduring takeaway was that the states aren't bureaucracy; every one exists
because some packet can arrive late, twice, or never, and the diagram is
the complete answer to "then what?"

## Takeaways

- Handshake problems are queue problems: SYN queue (floods, cookies) vs
  accept queue (slow apps, silent ACK drops) — ss -lt shows both.
- CLOSE_WAIT piling up = your code didn't close(); no sysctl will save you.
- TIME_WAIT is protection, not garbage; exhaustion is an outbound-churn
  problem solved by pooling, tcp_tw_reuse, or more 4-tuple space — never
  recycle.
- Half-open connections outlive dead peers for hours unless you configure
  keepalive or heartbeat at the application layer.
- ss with state filters and timers makes every claim here verifiable on a
  live box in seconds.