John Danskin (NVIDIA)

The Physics of Computation and GPU Architecture

In 1974, Dennard et al argued that power dissipation of integrated circuits would be independent of circuit density, and that furthermore, same-circuit gate delays would scale with transistor feature size. Moore’s law implied that circuit density would double every two years. Together, these implied that computational power per square inch and computational power per watt should double every 18 months. The rate of feature scaling, and feature dependent scaling of capacitance, resistance, and voltage have all fallen off of Dennard’s projections. Despite this, by using massively parallel computation, GPUs have mostly been able to stay on the Moore’s law overall performance curve. I’ll explain how we’ve done this, what this means for programming models, and make some projections for the future of computing.