GPUs for deep learning: Raina, Madhavan and Ng at ICML 2009
At ICML 2009 in Montreal, Andrew Ng's group reported up to 70-fold speedups training deep belief networks on graphics processors. A 100-million-parameter model trained in about a day instead of weeks on CPUs.
At ICML 2009 in Montreal (14 June 2009 on the conference program), Rajat Raina, Anand Madhavan, and Andrew Ng presented Large-scale Deep Unsupervised Learning using Graphics Processors. The paper showed that training deep belief networks (DBNs) on a GPU could run up to about 70 times faster than on a dual-core CPU for their benchmarks. A four-layer network with on the order of 100 million parameters could be trained in about a day on a GPU instead of weeks on CPUs, making large-scale unsupervised learning experiments practical.
Between DBNs and AlexNet
The result built directly on Hinton’s 2006 deep belief network recipe: stack restricted Boltzmann machines (RBMs), then fine-tune. Consumer graphics cards exposed parallel matrix operations through CUDA and similar APIs, so the same math that rendered video games could accelerate neural network training. The authors also highlighted sparse coding as a workload that benefited from the same hardware.
The paper arrived as the ImageNet benchmark was maturing and three years before AlexNet would win the ImageNet challenge on two GPUs. It did not replace later end-to-end convolutional network training, but it helped establish GPUs as the default accelerator for deep learning research.
Why it matters
Before affordable GPU clusters, many groups could not iterate on deep models at scale. Raina, Madhavan, and Ng quantified the gap in wall-clock time and parameter count, not only asymptotic complexity. That engineering message mattered as much as the algorithmic revival that began with DBNs in 2006 and culminated in industrial vision pipelines after 2012.