Graphics processing units (GPUs) are far more than graphics chips. They have been the heart of the artificial intelligence revolution for many years. This is due in great part to the fact that the computational substrate for high-fidelity 3D image processing lends itself beautifully to the mathematics that underpin the neural networks powering today’s most sophisticated AI applications.
GPUs seem to have been engineered for AI from the start, but that would misrepresent the historical development of this technology. Nvidia, AMD, and other chipmakers have made a lot of money providing GPUs for PC graphics, interactive gaming, image postprocessing, and virtual desktop infrastructure for many years.
Nevertheless, the affinity between graphical processing and AI is undeniable. Convolutional neural networks (CNNs), for example, are at the forefront of AI and are principally for image analysis, classification, rendering, and manipulation. It almost goes without saying that GPUs are one of the primary hardware workhorses for CNN processing in many applications.
What AI and image processing have in common
From a technical standpoint, what image processing and AI have in common is a reliance on highly parallel matrix and vector operations, which is where GPUs shine. Essentially, a matrix (aka “tensor graph”) in AI terminology is equivalent to a matrix of pixels—or rows and columns of dots—in a computer-generated image frame. A GPU’s embedded memory structures process an entire graphic image as a matrix—perhaps enriched through the adaptive intelligent that comes from concurrent execution of deep learning and other AI matrices. This architecture enables GPU-powered systems to use inline AI to dynamically and selectively accelerate processing of image updates and modifications.
The symbiotic relationship between these workloads is also evident at the application level, which explains why GPUs are often the preferred hardware accelerator technology for many intelligent, graphics-rich applications. Increasingly, we’re seeing AI being embedded in mass-market image-processing products, such as smart cameras that automatically stabilize images, adjust color and exposure, select focal points, and otherwise tailor the image in real time in the field to the scenes being captured, thereby reducing the likelihood that any of us might take a technically inept photograph.
Likewise, we’re seeing growing adoption of GPU-powered AI for such sophisticated image-processing capabilities as denoising, resolution scaling, variable rate shading, object detection, edge detection, motion detection, and interpolation of missing elements being embedded into smart cameras everywhere. Much of the growing adoption of browser-based machine learning focuses on GPU-boosted graphics-intensive applications such as image detection, recognition, classification, and manipulation. And GPUs are powering generative adversarial networks (GANs), a leading-edge AI approach whose graphical applications include generating photorealistic, high-resolution images for gaming, artistic, entertainment, medical, and other applications.
There is even a new generation of GPU-enabled smartphones that, among other applications, apply dynamic image manipulations designed to render realistic-looking (but essentially touched-up) enhancements to images of the human face.
Although there’s a symbiotic relationship between GPUs and CPUs in most deployments, it’s clear where GPUs shine: Compared to CPU cores, GPUs’ execution units, called stream multiprocessors, can perform more operations in parallel. GPUs can store more reusable cache data in local memory and registers for fast vectorization and matrix multiplication. GPUs also have higher memory bandwidth, greater thread parallelism, and can hold more data in short-path register memory than CPUs.
How Nvidia’s Turing architecture will enable AI-driven immersive experiences
Nvidia’s recent announcement of its new Turing architecture underlined the longstanding advantages of GPUs for applications that integrate 3D image processing with artificial neural networking. Turing is Nvidia’s eighth-generation GPU architecture, which has been incorporated into several new GPUs: the Nvidia Quadro RTX 8000, Quadro RTX 6000, and Quadro RTX 5000. In addition to enhancements in the graphics-optimized CUDA Cores and AI-boosting Tensor Cores that have been integral to prior versions of the vendor’s GPUs, Nvidia has incorporated a new graphics-optimization technology into Turing called RT Cores, which is designed to make Nvidia’s GPUs much faster and more efficient at performing ray-tracing operations in real-time scenarios.
Ray tracing—which has only been available heretofore in high-performance computer-generated imaging applications—models the physics of light-ray propagation. Although Nvidia is promoting Turing’s real-time ray tracing primarily for “lifelike gaming”—which also helps to explain Microsoft’s investment in the technology—the feature is clearly well-suited for any real-world application that relies on realistic visual processing to deliver an immersive experience. What it does is model how light-rays are reflected, diffracted, diffused, or blocked by objects in the surrounding environment. Dynamically ray-traced scenes can look much more realistic than traditional rasterized computer visuals. Real-time ray tracing can simulate with high fidelity the shifting patterns of glaring, shadowing, color shifting, and other light-ray propagation effects.
Although computationally intensive, real-time ray tracing can make all the difference in gaming, collaboration, and other immersive applications where users expect a seamless blend of virtual and physical worlds. As a consequence, Nvidia’s Turing GPU is geared toward any AI-enhanced graphic applications that immerse users in a fully artificial digital environment, overlay virtual objects on the real-world environment, or anchor virtual objects to the real world and let users interact with the virtual objects.
In other words, GPUs—as represented by Nvidia’s Turing architecture—are an essential component of the virtual, augmented, and mixed reality devices and applications. So it was no surprise that Nvidia’s Turing launch included native support for VirtualLink, the new USB-C alternate mode that standardizes virtual-reality connections around a single high-bandwidth USB-C cable. And it’s no surprise that Intel is planning to launch its own GPU in the next two years—though it seems awfully late to the game, oddly mistargeted purely at gaming, and seemingly designed to cannibalize interest in its newly announced next generation of graphics multicore CPUs optimized for graphics workloads in gaming and content creation.
Regardless, Nvidia seems likely to extend its GPU market lead with the Turing chips. Its new generation of GPUs are well-suited to leading-edge immersive applications that are driving AI deeply into mobile, embedded, edge, and internet of things (IoT) applications around the world.
Where you can expect to see the immersive technology applied
It’s easy to imagine the sorts of graphically rich AI applications where the Turing GPU will be deployed:
- Collaboration: AI-generated graphical avatars might support immersive blend of virtual and physical collaboration, in which every person has access to headsets and noise-canceling headphones and can enter a collaborative, immersive virtual environment.
- Learning: AI-generated simulations could provide an interactive, realistic, 3D experiential learning environment for individual and groups.
- Marketing: AI-generated product catalogs could enable realistically rendered 3D items to be virtually overlaid, placed, customized, and fitted on a customer’s body or into their houses, cars, and other environments.
- Design: AI-generated prototypes could be used by developers and engineers to interact with perfectly rendered 3D designs for possible future creations before they are committed to physical prototypes or send to the factory floor.
- Navigation: AI-generated graphic auto-completion or “inpainting,” when implemented in head’s-up displays, could help drivers to see what’s in their vehicles’ blind spots or to highlight salient objects in their real-time surroundings that they may have otherwise overlooked.
- Medicine: AI-generated graphic resolution enhancement could help doctors, when using smart goggles, to see fine details in bodily tissues more readily when performing surgeries, while medical mannequins can be virtually overlaid on actual patients to assist in diagnosis, training, and other healthcare scenarios.
- Industrial: AI-augmented digital-twin environments could support graphical modeling, optimization, and maintenance of manufacturing, logistics, and other industrial real assets in blended physical and virtual environments.
Where GPUs don’t do that well
Bear in mind, though, that GPUs are not a one-size-fits-all hardware accelerator. GPUs generally have less memory capacity than CPUs, GPUs must use CPUs to ingest data, and GPU clock speeds tend max out at one third of high-end CPUs, a limitation that limits GPUs’ ability to process sequential task rapidly.
Just as important, GPUs are not necessarily the optimal hardware-accelerator technology for every AI workload that breathes intelligence into graphics-infused mobile, edge, IoT, and other applications. In the systems-on-a-chip (SoC) at the heart of more modern products, GPUs are taking up residence alongside CPUs and various specialized neural-network processors, such as tensor processing units (TPUs), field programming gate arrays (FPGAs), and application-specific integrated circuits (ASICs).
In the evolving ecosystem of intelligent edge devices, the true symbiosis will depend on how well all these embedded AI chips play together to create visual experiences that astonish us with their lifelike realism.