hotchipsvideos's video: HC30-T2: Architectures for Accelerating Deep Neural Nets

Tutorial 2, Hot Chips 30 (2018), Sunday, August 19, 2018. Organizers: Kurt Keutzer, UC Berkeley, Geoffrey Burr, IBM, Bill Dally, Nvidia, and Ralph Wittig, Xilinx In the first portion of this tutorial, we provide a very brief introduction to Deep Neural Nets and their applications in computer vision, speech recognition, and other areas. We review the two key computational elements of Deep Neural Nets: inference and training in regards to their compute and memory requirements. Finally, we review popular target architectures for supporting these applications, including CPUs, GPUs, and custom DNN accelerators, including a discussion around common micro-architectures for acceleration of typical computational patterns and computational considerations around batch sizes, quantization and pruning.In the second portion of this tutorial we turn our focus to the problem of accelerating inference in edge devices. The devices range from autonomous vehicles, through mobile phones to very low power IOT devices. We consider both the real-time speed requirements of these applications as well as power and energy constraints. We consider effective design principles for reducing the computational requirements of DNNs and useful techniques for quantization/compression of DNN computations. We then go into depth on accelerator architectures for meeting these constraints.In the last portion of this tutorial we consider the problem of training Deep Neural Nets, particularly in the cloud. We briefly examine the ubiquitous synchronous stochastic gradient descent and asynchronous variants. We look at the problem of scaling DNN training on distributed multiprocessors and its attendant problems of increasing batch size and balancing computation and communication. We then broadly survey the architectures to support training, including special purpose accelerators. Overview of Deep Learning and Computer Architectures for Accelerating DNNs Michaela Blott, Xilinx Research Accelerating Inference at the Edge Song Han, MIT Accelerating Training in the Cloud William L. Lynch and Ardavan Pedram, Cerebras Systems