Tokenization

Splitting a stream of text — or an identifier like parseXMLHttpRequest — into its correct component pieces before doing anything else with it.

What it is

Tokenization is the process of breaking a stream of text into smaller meaningful units, called tokens, before further processing happens. In natural-language contexts, that usually means splitting a sentence into words. In code, it means splitting an identifier — a variable, function, or class name — into the individual words that were joined together to form it.

Why it's harder than it looks

Converting between case formats is mechanically simple once you know where the word boundaries are. The actual difficulty is finding those boundaries correctly in the first place. Take parseXMLHttpRequest: the correct tokenization is four words — parse, XML, Http, Request. A naive tokenizer that simply splits at every capital letter produces five broken fragments instead, mangling the acronym: converting to snake_case would wrongly yield parse_x_m_l_http_request instead of the correct parse_xml_http_request.

What a smart tokenizer checks for

• ALLCAPS acronym runs — consecutive capital letters (XML, HTTP, API) are recognized as one unit rather than split letter-by-letter.
• Numeric suffixes — a trailing number like the "2" in user2FA is kept attached to its word rather than treated as its own token.
• Existing separators — underscores, hyphens, and periods already present in the input are respected as boundaries rather than re-guessed.
• Known brand-name exceptions — names like iPhone or eBay that don't follow standard capitalization are recognized rather than forcibly re-split.

Where this shows up

Every one of the programming naming conventions on this site — camelCase, PascalCase, snake_case, kebab-case, and the rest — runs through the same tokenizer before conversion, so the acronym-handling behavior is consistent no matter which case you're converting into.

Back to the full glossary — 200 terms covering case conversion, style guides, and text tools.