CUDA
NVIDIA's parallel computing platform and API for programming GPUs for general-purpose workloads such as neural network training, introduced in 2007.
CUDA (Compute Unified Device Architecture) is NVIDIA’s platform for running general-purpose code on graphics processing units (GPUs). Developers write kernels in C-like languages that execute thousands of threads in parallel, which maps well to the matrix multiplications inside neural network training.
At ICML 2009, Andrew Ng’s group used CUDA-capable hardware to speed up deep belief network training (article). Later work such as AlexNet relied on the same class of consumer GPUs, turning gaming cards into standard deep learning infrastructure.