08 Keep only the best k

8 / 12

Goal

Generate at least 15 tokens from Alice was with temperature 1.5 and top_k 5.

High temperature makes text lively, but it also gives a chance to thousands of tokens that make no sense at all. One wrong pick and the sentence derails.

top-k keeps only the k most likely candidates and throws the rest away before sampling. With top_k 5 there are only ever five options, however hot the temperature.

Try it

Set both and look at the bars: everything past the fifth is cut. Generate and compare with the last lesson's output.

Go to the terminal

Hint

set temperature 1.5, set top_k 5, then generate 15.

Show a solution

set temperature 1.5
set top_k 5
predict
generate 15

This interactive environment needs JavaScript. Everything you need to read is on this page, and the solution works the same way for real.

LLMs

Starting LLMs…