Extending Tensor Contractions for Deep Neural Networks

Introduction During a conversation with OpenAI CTO Greg Brockman [1], Geoffrey Hinton recounted how his lab’s seminal work on AlexNet [2] wouldn’t have been possible without Alex Krizhevsky’s strong CUDA programming skills. This observation demonstrates the importance of custom compute kernels for neural network research, which since then have also enabled the use of faster…