Markov Babbler: the original next-word machine
Long before ChatGPT, machines guessed the next word by counting. Feed this tiny model any text and watch it babble, one dice-roll at a time.
This week the AI keeps it humble. Today’s chatbots feel like magic, so it is worth meeting their oldest, dumbest ancestor: a machine that writes by counting words.
The gadget below is a Markov chain text generator. It does not understand a single thing. It reads your training text, remembers which word tends to follow which word, and then generates new text by rolling weighted dice, one word at a time.
How to play:
- The box already has some training text. Edit it, or paste your own (song lyrics, a recipe, a news article, anything).
- Pick the memory: with “1 word” the model only looks at the last word before guessing; with “2 words” it looks at the last two, so the output reads more fluently but repeats the source more.
- Press Generate and watch it write. The panel underneath shows the candidate next words and their probabilities, and highlights the one the dice picked.
Why this is the ancestor of language models
The idea goes back to Claude Shannon, who in 1948 used letter and word statistics to generate text that almost made sense. That trick, predicting the next token from the ones before it, is exactly what a modern large language model does. The difference is scale and cleverness, not the core move.
A Markov babbler has a memory of one or two words. A modern model effectively considers thousands, and it learns rich patterns instead of raw counts. But both are doing next-word prediction, and neither “knows” what it is saying. That is why the same statistical shortcut also powered older natural language processing and the autocomplete on your phone.
Watch it break
Set the memory to “1 word” and the output turns into cheerful nonsense: grammatical fragments that wander off a cliff. Switch to “2 words” and it suddenly sounds smarter, because it is leaning harder on your text, sometimes quoting it word for word. That tension, between originality and just copying the source, never really went away; it is one of the arguments people have about big models today.
For another program that felt clever while doing something very simple, meet ELIZA (1966). And if you missed it, our first edition let you train a Perceptron in the browser.