Watson on Jeopardy!: when a machine won the quiz (2011)
In February 2011 IBM's Watson defeated the two most successful Jeopardy! champions in a televised match. The system showed how far natural language processing had come, and IBM then set out to sell the technology to medicine and business.
Over three evenings in February 2011, February 14 through 16, IBM’s Watson faced the two most successful champions in Jeopardy! history in two matches. Ken Jennings had won 74 games in a row, a record. Brad Rutter held the show’s highest lifetime winnings, about $3 million. The competition had been recorded in January 2011 at IBM’s research center in Yorktown Heights, New York. The system was named after IBM’s first CEO, Thomas J. Watson Sr., and developed by a research team led by David Ferrucci.
Why Jeopardy!
After Deep Blue beat Garry Kasparov in 1997 (article), IBM looked for a new human-versus-machine challenge. In 2006, David Ferrucci proposed a Jeopardy! system. More than two dozen scientists, engineers, and programmers in Yorktown Heights worked on it for five years.
Jeopardy! was considered hard for computers because of its fast format and clues that rely on subtle meanings, wordplay, and riddles, unlike the fixed rules of chess. That made it a test of natural language processing. Ferrucci put the goal plainly: “The goal is not to model the human brain. The goal is to build a computer that can be more effective in understanding and interacting in natural language, but not necessarily the same way humans do it.”
Hundreds of algorithms, one buzzer
Watson ran on DeepQA, software designed for question answering. The original machine filled a room: 10 racks with 90 servers and 2,880 processor cores. Its knowledge came from Wikipedia, encyclopedias, dictionaries, religious texts, novels, plays, and books from Project Gutenberg, among other sources. During the match it had no internet connection.
Hundreds of algorithms analyzed each clue at once, gathered candidate answers and evidence, and scored them. The more algorithms that independently agreed, the higher Watson’s confidence. It pressed the buzzer only when confidence was high enough, all within about three seconds. Unlike a search engine that returns a list of documents, Watson was built to give a direct answer, the defining idea of question answering.
Three evenings in February
Watson was not flawless. In the first match’s Final Jeopardy!, category “US Cities”, it answered “What is Toronto?????” The five question marks signaled low confidence; the correct city was Chicago. An IBM engineer wore a Toronto Blue Jays jacket to the recording of the second match. Watson went into that match tied with Rutter.
The final totals were $77,147 for Watson, $24,000 for Jennings, and $21,600 for Rutter. IBM donated Watson’s $1 million first-place prize to World Vision and World Community Grid. Jennings received $300,000 and Rutter $200,000.
Jennings wrote on his own correct Final Jeopardy! response: “I for one welcome our new computer overlords.” He later said: “‘Quiz show contestant’ may be the first job made redundant by Watson, but I’m sure it won’t be the last.” Paul Miller of Engadget, quoted by the BBC, argued that the Jeopardy! questions were not especially hard and that Watson was mainly faster on the buzzer.
Why it matters
Where Deep Blue searched a closed game world, Watson tried to answer open questions in language. That shift pointed toward broader AI ambitions beyond chess.
On the evening of the final, IBM announced a research agreement with Nuance Communications to explore and market Watson’s analytical capabilities in healthcare. In January 2022, IBM and Francisco Partners signed an agreement for Francisco Partners to acquire data and analytics assets from IBM’s Watson Health business; financial details were not disclosed.
Eleven years later, ChatGPT answered questions in natural language for anyone, using a large language model instead of an ensemble of specialist algorithms. Watson showed that a spectacular demo and a viable business are two different things. The next great human-versus-machine moment was AlphaGo, which learned Go rather than retrieving facts from a knowledge base.