04 Not every language costs the same

4 / 12

Goal

Find text of 10 characters or fewer that costs at least 10 tokens.

The vocabulary was learned mostly from English text, so English is cheap: "hello world" is two tokens. Other scripts often fall back to raw bytes — "नमस्ते दुनिया" (Hindi for "hello world") is 13 tokens, nearly one per character.

Emoji are worse: a single flag is two characters and up to six tokens. When a token is just part of a character's bytes, the strip shows it as <f0><9f>.

Since you pay per token and the context window is measured in tokens, the same message genuinely costs more, and fits less, in some languages than others.

Try it

Find something tiny that's expensive.

Go to the terminal

Hint

Emoji and many non-Latin scripts take several tokens per character. Try flags: tokenize 🇬🇧🇫🇷🇩🇪 is 6 characters and 18 tokens.

Show a solution

tokenize "नमस्ते दुनिया"
tokenize 🇬🇧🇫🇷🇩🇪

This interactive environment needs JavaScript. Everything you need to read is on this page, and the solution works the same way for real.

LLMs

Starting LLMs…