Softmax sampling: spin the temperature, roll the next token
Five candidate words, adjustable logits, a temperature slider, and weighted dice. This is how language models pick the next token at decode time.
Last week A* pathfinding expanded a maze with a frontier you could watch. This week the machine does the opposite of planning: it guesses the next word and lives with uncertainty.
After a model (see BPE tokens and attention weights) emits logits, one score per candidate token, a softmax turns those scores into probabilities that sum to 100%. Temperature divides the logits before that step: turn it down and the top token almost always wins (greedy, safe, repetitive); turn it up and rare words get a fairer lottery (creative, risky, sometimes nonsense). That knob is part of everyday inference for every LLM from ChatGPT onward.
How to play:
- Read the sentence stub; five next-token candidates sit below with logit sliders (raw scores, not percentages yet).
- Drag Temperature and watch the Chance bars reshape without changing the underlying logits.
- Press Sample once or Sample ×10 and watch Recent picks fill; the highlighted word is the latest roll.
- Try presets (Clear leader, Close race, Flat scores), then set temperature to 0.15 vs 1.80 on the same preset.
Try this:
- Clear leader at temperature 0.20: the top bar should sit near 90%; ten samples should almost always pick the same word.
- Same preset at 1.80: the top bar might fall toward 35–45%; ten samples should mix words.
- Flat scores at 1.00: each bar near 20%; long runs look like a fair five-sided die.
- Close race at 0.25: one tiny logit bump should dominate; at 2.00 the race should look almost even.
Why this matters
The Transformer revolution made generation cheap; GPT-3 showed what fluent sampling could look like at scale. Product teams expose temperature (and top-p, not simulated here) because the same weights can sound like a cautious clerk or a drunken poet. Training teaches what is likely; decoding chooses how much surprise you allow. That is not magic, it is arithmetic you can see.
Missed the earlier editions? Train a perceptron, watch a Markov chain babble, lose to unbeatable tic-tac-toe, teach a filter to see edges, watch an agent learn from reward, see how a model reads text in tokens, watch one roll downhill to learn, let a machine find groups on its own, see a line learn to bend, watch words decide who to listen to, explore vs exploit on slot machines, chat with ELIZA, scribble Hopfield memory, count words for spam, or paint walls for A*.
The cat sat on the …