Tokenization

Breaking text into the tokens a model can process.

What changes when an atlas becomes atlases?

A catalogue editor adds two letters to a label. Both versions contain two words and a period. Counting the pieces with the same tokenizer gives two different totals.

Singular label

“Blue atlas.” takes three tokens. The space and atlas belong to one piece.

Plural label

“Blue atlases.” takes four. The tokenizer splits the second word across two pieces.

Exact pieces and token IDs in cl100k_base
Blue atlas.
Blue (10544) · ␠atlas (62032) · . (13)
Blue atlases.
Blue (10544) · ␠atl (71554) · ases (2315) · . (13)

The open-box symbol marks one ordinary space. It is shown here so you can see which piece owns it. Decoding either sequence restores its original sentence, space and period included.

Which count applies to your text?

Use the tokenizer associated with the model and preserve the exact input. Capitalization, spaces and punctuation can change the result. A token ID identifies a piece within that tokenizer’s vocabulary.

Why Tokenization Matters

Many practical questions in modern AI depend on tokens rather than characters or pages. Usage limits, context size, latency, and price are often measured in tokens. That is why a document that seems short to a person may still consume a large amount of model context if it contains dense formatting, code, tables, or complex terminology.

Tokenization also affects behavior. Different token boundaries can influence how well a model handles names, rare words, code snippets, long numbers, or different languages. It is one of the reasons that the same idea can behave differently depending on phrasing.

Follow the idea

Token · Read a token split from the model’s point of view.

Context Window · See how the input and reply share a token budget.

Source · OpenAI’s tiktoken implementation

Return to Tokenization in the A–Z glossary