GPU

Graphics processing unit; parallel processor originally built for rendering images, later repurposed to accelerate matrix operations in neural network training.

A graphics processing unit (GPU) is a processor designed to render images and video by running many small calculations in parallel. Consumer graphics cards built for video games pack thousands of simple cores that happen to excel at the matrix multiplications that dominate neural network training. Before GPUs, training deep models on CPUs was often impractically slow.

By the late 2000s researchers including Geoffrey Hinton’s group were exploring GPU acceleration for stacked models such as deep belief networks (article). The breakthrough came with AlexNet in 2012: Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton trained a deep convolutional network on two gaming GPUs and crushed the ImageNet benchmark (article). That result helped turn GPU clusters into standard infrastructure for deep learning and large-scale machine learning.