Deep belief networks: when deep nets became trainable again (2006)
In July 2006 Geoffrey Hinton and colleagues showed that stacking restricted Boltzmann machines layer by layer, then fine-tuning with backpropagation, could train deep networks. The result helped restart the deep-learning revival.
In July 2006, Geoffrey Hinton, Simon Osindero, and Yee-Whye Teh published A Fast Learning Algorithm for Deep Belief Nets in Neural Computation (volume 18, issue 7). Later the same month, Hinton and Ruslan Salakhutdinov showed in Science (28 July 2006) that a deep autoencoder could compress high-dimensional data far better than shallow methods. Together they introduced deep belief networks (DBN): neural networks built by greedily stacking restricted Boltzmann machines (RBMs) and then fine-tuning with backpropagation.
Layer by layer, without vanishing gradients
Training all layers of a deep net at once had long been difficult. The 2006 recipe trained one RBM at a time: each new layer learned to model the patterns produced by the layer below. Only after this unsupervised pretraining did supervised learning adjust the whole stack. The Science paper demonstrated the payoff on image patches and document vectors, reducing dimensionality with far lower reconstruction error than principal components analysis.
A revival, not the last word
The results arrived as ImageNet was taking shape (article) and helped convince researchers that depth was workable again. Other groups, including Yoshua Bengio and colleagues, explored related greedy layer-wise training around the same period. Within a few years, much of the field moved on: ReLU activations, better weight initialization, GPUs, and large labelled datasets made end-to-end backpropagation enough for models such as AlexNet, which won the ImageNet challenge in 2012.
Why it matters
Deep belief networks were a bridge: they reused Boltzmann machine building blocks and Markov chain ideas from the 1980s, then handed a trained stack to the same gradient descent machinery that powers most deep learning today. Hinton’s later Nobel Prize citation reaches back to that earlier probabilistic line. The pretraining fad faded, but the proof that deep nets could be trained at all helped open the era this site documents from AlexNet onward.