Unicode

A character set, not an encoding — and the distinction between the two is exactly where a lot of confusion about text starts.

What it is

Unicode is a standard that assigns a unique number — called a code point — to essentially every character used in written human language, plus symbols, emoji, and technical characters. The Latin letter "A" is code point U+0041; the emoji 😀 is U+1F600. Unicode currently defines well over 140,000 characters, maintained by the nonprofit Unicode Consortium, and is what makes it possible for a single document to mix English, Japanese, Arabic, and emoji without conflict.

What it replaced

Before Unicode, different systems used different, incompatible character sets — ASCII covered only the basic unaccented English alphabet and a handful of symbols in 128 slots; other regions used their own extended or entirely separate encodings. A file created under one system's character set could display as garbage on another. Unicode's goal was one universal set that every system could agree on.

The distinction that trips people up

Unicode itself is not a way of storing text as bytes — it's just the mapping from character to code point. How those code points get stored as actual bytes on disk or in memory is a separate decision, called an encoding. UTF-8 is the most common encoding of Unicode, but it isn't the only one — UTF-16 and UTF-32 also encode the same Unicode code points, just using different byte layouts. Saying a file "uses Unicode" is technically incomplete without also saying which encoding it uses.

Why this matters for text processing

Most of the invisible-character problems this site's tools deal with — a stray BOM, a zero-width space, mismatched line-ending characters — are Unicode code points that exist and are valid, but aren't visible and often aren't wanted. Understanding that these are real characters with real code points, not rendering glitches, is the first step in reliably finding and removing them with the invisible character finder.

Source

Maintained by the Unicode Consortium, publisher of the Unicode Standard.

Back to the full glossary — 200 terms covering case conversion, style guides, and text tools.