“Just store the text” is a sentence that has caused more quiet data-corruption bugs than almost any other assumption in programming. Text isn’t bytes, isn’t characters, isn’t what-the-user-sees — it’s all three, at different layers, and every layer boundary is a place data gets mangled. The history of how we got here isn’t trivia; it’s the debugging manual for every é, every ?????, every “why won’t this filename delete” you will ever encounter. (I’ve written separately on UTF-8’s bit-level design; this is the wider map.)

The Layers, Built Up Historically

ASCII (1963): 128 characters in 7 bits — English letters, digits, punctuation, control codes. Clean, tiny, and monolingual. The trouble began with the 8th bit: 128 spare slots, and everyone filled them differently.

Code pages (the mojibake era): Latin-1 for Western Europe, Windows- 1251 for Cyrillic, KOI8-R, Shift-JIS for Japanese, and dozens more — each redefining bytes 128–255. A byte was meaningless without knowing which code page, and that context was almost never stored with the text. Byte 0xE9 was é in Latin-1, Ú in another page, half a character in a third. Email, filenames, and databases carried bytes with no encoding tag, and every cross-context transfer was a guess. This is the root of mojibake — not a bug, a design gap: text without its encoding is undecodable, and for decades text traveled without it.

Unicode (1991→): one code point per character, for every script, aiming to end the code-page wars. The foundational split people still trip over: Unicode is a character set (a map from characters to numbers, U+0000…U+10FFFF, organized into 17 “planes” — the Basic Multilingual Plane holds most living scripts; the Supplementary planes hold emoji, historic scripts, rare CJK), while UTF-8/UTF-16/UTF-32 are encodings — different ways to turn those numbers into bytes. “It’s Unicode” is not an encoding statement, and conflating the two is exactly the confusion that made UTF-16’s wchar_t/String a recurring surprise (variable-length after all, once emoji arrived, via surrogate pairs).

The Hard Parts That Remain

Unicode ended the encoding ambiguity and introduced semantic complexity that no byte-level fix can remove — because human writing systems are genuinely this complicated:

  • One character, many byte-sequences. é is either U+00E9 (precomposed) or e + U+0301 (combining accent) — visually identical, different bytes, == says false. Normalization (NFC/NFD) exists to canonicalize before comparison; skip it and “identical” user input fails to match, dedupe misses, and logins break. Filesystems disagree at this exact layer: macOS historically normalizes filenames (NFD-ish), Linux stores raw bytes — so a file created on one is unfindable-by-name on the other, the classic “the file is there but I can’t delete it” cross-platform bug.
  • Character ≠ what users see. A grapheme — one “character” to a human — can be many code points: 👨‍👩‍👧‍👦 is a family emoji built from several people joined by zero-width joiners; flag emoji are pairs of regional-indicators; skin-tone modifiers combine. Cursor movement, truncation, and “length” must operate on grapheme clusters, which needs a Unicode segmentation library — DIY string slicing splits families in half and truncates mid-glyph.
  • Case and collation are locale-dependent. Turkish dotless ı breaks naive toUpperCase; German ß uppercases to SS; sort order differs by language. There is no locale-free “correct” casing or ordering — which is why security code should never lowercase-compare identifiers naively (the Turkish-i bug has produced real auth bypasses).
  • Confusables and direction. Cyrillic а looks identical to Latin a (homograph phishing); right-to-left override characters can make exe.txt display as txt.exe. Text rendering is an attack surface.

The through-line for practitioners is a discipline, not a trick: decode at your input boundary, work in code points/graphemes internally, encode at your output boundary — and know your encoding at every boundary. Store UTF-8, tag it, normalize before comparing, use libraries for casing/collation/segmentation, and treat “text is simple” as the assumption that will page you.

Takeaways

  • Text is layered: bytes → encoding → code points → grapheme clusters; most bugs happen when a boundary drops the encoding context.
  • Mojibake is historical: code pages redefined bytes 128–255 and text traveled without saying which — undecodable by design.
  • Unicode is a character set; UTF-8/16/32 are encodings — “it’s Unicode” says nothing about bytes, and UTF-16 is variable-length too.
  • Normalize before comparing (NFC/NFD), operate on graphemes for length/cursor/truncation, and never case-fold or collate without locale awareness.
  • Decode at input, encode at output, know the encoding at every boundary — text is never “just” anything.