Course 06 — LLMs, under the hood

How a languagemodel actually writes.

Strip away the chat window and a language model does one thing: pick the next token, again and again. Here you tokenize text with a real GPT tokenizer, watch probabilities reshape as you turn the temperature dial, and see exactly what falls out of the context window.

Start lesson one Open the sandbox

12 lessons · 4 chapters · free

LLMs — tokenize, then sample
  1. 01 · Read

    Each lesson is one idea, explained in a few short paragraphs — no walls of theory.

  2. 02 · Do

    Type into a live environment and watch it react. Tab completes, ↑ recalls, undo takes back any step.

  3. 03 · Match

    Reach the goal any way you like — you're judged on the result, not the keystrokes. Par is the shortest route.

01

Tokens

What a model actually reads, and why it's counted in tokens, not words.

  1. 01 What a model reads Text goes in as tokens, not letters or words.
  2. 02 Words aren't tokens Rare words split into pieces — and that's why counts surprise you.
  3. 03 Numbers are chunks of digits Why language models stumble on arithmetic.
  4. 04 Not every language costs the same Emoji, scripts and the real cost of a character.
02

Prediction

A model is a next-token probability machine. See the distribution.

  1. 05 One token at a time A language model is a next-token probability machine.
  2. 06 Always take the top pick Greedy decoding — and why it loops.
03

Sampling

Greedy, temperature, top-k and top-p — choosing from the distribution.

  1. 07 Turn up the temperature Temperature reshapes the distribution before sampling.
  2. 08 Keep only the best k top-k cuts the long tail of unlikely tokens.
  3. 09 Nucleus sampling top-p keeps just enough candidates to cover a probability.
  4. 10 Make it reproducible Random, but repeatable — the seed.
04

Context

The window, what falls out of it, and why the system prompt matters.

  1. 11 When the window fills up The oldest text falls out — including your instructions.
  2. 12 Leave room for the answer max_tokens is reserved from the same window.

Tokenization uses OpenAI's real cl100k_base vocabulary. Generation uses a small word-level model trained in your browser on Alice's Adventures in Wonderland — far smaller than a real LLM, but sampling, temperature, top-k, top-p and context limits work exactly the same way.