- 15th May, 2018
GPUs in Data Science: What’s all the hype about?
By Phil Massie
There is a lot of hype in the world of AI and deep learning about Graphics Processing Units or GPUs. What's the hype about and are they worth it?
Most operations on your computer are carried out by your central processing unit or CPU, and your CPU is very good at most of them. Some types of operations are however not very well suited to CPUs.
According to Nvidia: “Architecturally, the CPU is composed of just few cores with lots of cache memory that can handle a few software threads at a time. In contrast, a GPU is composed of hundreds of cores that can handle thousands of threads simultaneously. The ability of a GPU with 100+ cores to process thousands of threads can accelerate some software by 100x over a CPU alone. What’s more, the GPU achieves this acceleration while being more power - and cost-efficient than a CPU.”
What that equates to is CPUs being very good at doing a few things at a time while GPUs can potentially do thousands of things at the same time. Naturally there is a lot more to it. For instance, although GPUs can execute many more operations simultaneously than CPUs, individually their cores are slower than those of CPUs. Additionally they aren’t built with the same sets of features as CPUs and simply can’t perform all of the same operations. It quickly becomes a fairly dense technical topic and an interesting discussion can be found here.
Deep neural nets rely heavily on dense linear algebra, a type of operation very well suited to GPUs. Powerful GPUs therefore result in substantially faster model training and inference than CPUs.
There are numerous deep learning frameworks out in the wild including Tensorflow, PyTorch and Theano. For the most part, these frameworks leverage the CUDA parallel processing platform exposed by Nvidia GPUs ‘out the box’ although Theano can also use OpenCL, the platform exposed by AMD GPUs (and possibly Intel at some point). For what it’s worth, Tensorflow and PyTorch can be compiled to use OpenCL too.
Pretty much every PC or laptop on the market ships with some sort of GPU integrated into the CPU. GPUs that expose their parallel processing capabilities and have their own resources are usually discrete units and use their own resources. This means there are never conflicts between bandwidth or memory usage. At work I use a moderate laptop with an Intel core i5 processor and a discrete Nvidia 940MX GPU. It is by no means a powerful unit but it’s useful for testing ideas nevertheless. Training enterprise ready models takes place in the cloud or on remote clusters where we have access to extremely powerful GPU systems.
Last week I put together a small script to quantify the capabilities of my laptop’s CPU vs GPU. The test itself is very simple. It solves the dot product, a linear algebra operation critical to deep learning, of increasingly large matrices on both processors and then plots the results. The code uses Tensorflow and CUDA and can be found in my Github repo.
I’m pretty happy with the result shown above. It clearly shows how the CPU takes much longer to solve increasingly large dot products than the GPU. I’m not suggesting that large scale neural nets should be trained or deployed on small laptops, but I thought this curve illustrated the capabilities of my GPU vs. my CPU rather well.
Neural network algorithms are nothing new. Implementing them the way we do today however has only become possible fairly recently thanks to increased computational power. GPU Computations are taking these possibilities to new levels and the opportunities seem endless. So, are GPUs worth the hype? I say yes absolutely!