01 What a model reads

1 / 12

Goal

Tokenize a sentence of your own and look at how it splits.

A language model never sees letters or words. Before anything else happens, text is cut into tokens — common chunks of characters — and each token becomes a number. The model reads and writes those numbers.

This is the real tokenizer used by GPT-4-era models (cl100k_base), with about 100,000 tokens in its vocabulary. Common words are one token each, usually including the space before them. Rarer words are split into pieces.

Try it

Tokenize anything you like. The dots in the strip are spaces — notice how they belong to the start of the next token.

Go to the terminal

Hint

tokenize "Hello, world! Tokenization isn't always intuitive."

Show a solution

tokenize "Hello, world! Tokenization isn't always intuitive."

This interactive environment needs JavaScript. Everything you need to read is on this page, and the solution works the same way for real.

LLMs

Starting LLMs…