# From ASCII to Unicode: Why Text Is Harder Than You Think

> Text feels like a solved problem until an emoji breaks your database, a filename won't delete, or 'MÜLLER' and 'müller' won't compare equal. The layered history of character encoding explains every mojibake bug you'll ever hit.

URL: https://harshith.in/blog/character-encodings-history/
Author: Harshith Sunku (Staff Engineer) — https://harshith.in/about/
Tags: fundamentals, systems

"Just store the text" is a sentence that has caused more quiet
data-corruption bugs than almost any other assumption in programming.
Text isn't bytes, isn't characters, isn't what-the-user-sees — it's
all three, at different layers, and every layer boundary is a place
data gets mangled. The history of how we got here isn't trivia; it's
the debugging manual for every `é`, every `?????`, every "why won't
this filename delete" you will ever encounter. (I've written
separately on UTF-8's bit-level design; this is the wider map.)

## The Layers, Built Up Historically

**ASCII (1963)**: 128 characters in 7 bits — English letters, digits,
punctuation, control codes. Clean, tiny, and monolingual. The trouble
began with the 8th bit: 128 spare slots, and *everyone filled them
differently*.

**Code pages (the mojibake era)**: Latin-1 for Western Europe, Windows-
1251 for Cyrillic, KOI8-R, Shift-JIS for Japanese, and dozens more —
each redefining bytes 128–255. A byte was meaningless without knowing
*which* code page, and that context was almost never stored with the
text. Byte 0xE9 was `é` in Latin-1, `Ú` in another page, half a
character in a third. Email, filenames, and databases carried bytes
with no encoding tag, and every cross-context transfer was a guess.
This is the *root* of mojibake — not a bug, a design gap: text without
its encoding is undecodable, and for decades text traveled without it.

**Unicode (1991→)**: one code point per character, for every script,
aiming to end the code-page wars. The foundational split people still
trip over: Unicode is a **character set** (a map from characters to
numbers, U+0000…U+10FFFF, organized into 17 "planes" — the Basic
Multilingual Plane holds most living scripts; the Supplementary planes
hold emoji, historic scripts, rare CJK), while UTF-8/UTF-16/UTF-32 are
**encodings** — different ways to turn those numbers into bytes. "It's
Unicode" is not an encoding statement, and conflating the two is
exactly the confusion that made UTF-16's `wchar_t`/`String` a
recurring surprise (variable-length after all, once emoji arrived, via
surrogate pairs).

## The Hard Parts That Remain

Unicode ended the encoding ambiguity and introduced *semantic*
complexity that no byte-level fix can remove — because human writing
systems are genuinely this complicated:

- **One character, many byte-sequences.** `é` is either U+00E9
  (precomposed) or `e` + U+0301 (combining accent) — visually
  identical, different bytes, `==` says false. **Normalization**
  (NFC/NFD) exists to canonicalize before comparison; skip it and
  "identical" user input fails to match, dedupe misses, and logins
  break. Filesystems disagree at this exact layer: macOS historically
  normalizes filenames (NFD-ish), Linux stores raw bytes — so a file
  created on one is unfindable-by-name on the other, the classic
  "the file is there but I can't delete it" cross-platform bug.
- **Character ≠ what users see.** A grapheme — one "character" to a
  human — can be many code points: `👨‍👩‍👧‍👦` is a family emoji built
  from several people joined by zero-width joiners; flag emoji are
  pairs of regional-indicators; skin-tone modifiers combine. Cursor
  movement, truncation, and "length" must operate on **grapheme
  clusters**, which needs a Unicode segmentation library — DIY string
  slicing splits families in half and truncates mid-glyph.
- **Case and collation are locale-dependent.** Turkish dotless ı
  breaks naive `toUpperCase`; German ß uppercases to SS; sort order
  differs by language. There is no locale-free "correct" casing or
  ordering — which is why security code should never lowercase-compare
  identifiers naively (the Turkish-i bug has produced real auth
  bypasses).
- **Confusables and direction.** Cyrillic `а` looks identical to Latin
  `a` (homograph phishing); right-to-left override characters can make
  `exe.txt` display as `txt.exe`. Text rendering is an attack surface.

The through-line for practitioners is a discipline, not a trick:
**decode at your input boundary, work in code points/graphemes
internally, encode at your output boundary — and know your encoding
at every boundary.** Store UTF-8, tag it, normalize before comparing,
use libraries for casing/collation/segmentation, and treat "text is
simple" as the assumption that will page you.

## Takeaways

- Text is layered: bytes → encoding → code points → grapheme clusters;
  most bugs happen when a boundary drops the encoding context.
- Mojibake is historical: code pages redefined bytes 128–255 and text
  traveled without saying which — undecodable by design.
- Unicode is a character set; UTF-8/16/32 are encodings — "it's
  Unicode" says nothing about bytes, and UTF-16 is variable-length too.
- Normalize before comparing (NFC/NFD), operate on graphemes for
  length/cursor/truncation, and never case-fold or collate without
  locale awareness.
- Decode at input, encode at output, know the encoding at every
  boundary — text is never "just" anything.