Getting Started with CUDA and Parallel Programming | NVIDIA GTC 2025 Session

Getting Started with CUDA and Parallel Programming

Speakers: Stephen Jones, CUDA Architect, NVIDIA

Conference: NVIDIA GTC

YouTube: https://www.youtube.com/watch?v=GmNkYayuaA4

Overview

In this NVIDIA GTC session, Stephen Jones, a seasoned CUDA Architect at NVIDIA, delivered a compelling and insightful talk titled "Getting Started with CUDA and Parallel Programming." Far from being a traditional "how-to" guide for writing low-level GPU kernels, Jones's presentation fundamentally reframed the approach to parallel programming on NVIDIA GPUs. His core message, delivered with a touch of humor and pragmatism, was that for the vast majority of developers, the goal should be to avoid direct parallel programming as much as possible, leveraging the extensive and increasingly sophisticated CUDA software stack.

Jones, who has been working with CUDA since 2008, shared his mental model for how the GPU operates and how NVIDIA designs its programming interfaces to be intuitive and productive. He emphasized that parallel programming is inherently difficult, prone to bugs, and time-consuming. Therefore, NVIDIA has invested heavily in building layers of abstraction – from high-level frameworks to highly optimized libraries and new programming models – that abstract away the complexities of GPU parallelism, allowing developers to achieve high performance with minimal effort.

This talk is particularly relevant in the current landscape of AI and high-performance computing, where GPUs are indispensable. It matters because it addresses a critical bottleneck: developer productivity. By providing tools and methodologies that enable developers to tap into the immense power of GPUs without becoming low-level parallel programming experts, NVIDIA aims to accelerate innovation across numerous domains, from AI model development to scientific simulations and graphics. The session ultimately introduced CuTile, a new programming model designed to bridge the gap between highly optimized libraries and custom kernel development, further simplifying the creation of performant GPU applications.

Background

▶ Watch: Why you should avoid parallel programming (1:17)

The challenge of parallel programming has long been a formidable hurdle in computer science. While multi-core CPUs and, more recently, GPUs offer immense computational power through parallel execution, harnessing this power efficiently is notoriously difficult. Developers face complexities such as thread synchronization, data races, memory management across different memory spaces (host vs. device), load balancing, and optimizing for specific hardware architectures. The debugging process for parallel code is also significantly more intricate than for serial code, often leading to non-deterministic bugs that are hard to reproduce.

In the machine learning and systems space, this problem is exacerbated by the sheer scale of modern datasets and models. Training large neural networks or running complex simulations requires processing billions of operations, often on data that doesn't fit into a single memory space. GPUs, with their thousands of processing cores, are uniquely suited for these tasks, but their architectural nuances demand a specialized programming approach.

NVIDIA's CUDA platform was introduced to provide a robust framework for general-purpose computing on GPUs. Over the years, CUDA has evolved into a comprehensive ecosystem, encompassing not just the low-level programming model but also a vast array of libraries, tools, and higher-level abstractions. Early CUDA programming often involved writing explicit GPU kernels, where developers meticulously managed threads, shared memory, and global memory accesses. While this offered maximum control and potential for extreme optimization, it demanded deep expertise in GPU architecture and parallel programming paradigms.

The problem persists because while high-level frameworks like PyTorch and TensorFlow abstract much of this away for common AI tasks, there are still scenarios where custom operations or novel algorithms are needed. When these custom components cannot be expressed using existing library functions, developers are pushed into the realm of kernel authoring, where the aforementioned complexities resurface. The central theme of Jones's talk is to acknowledge this inherent difficulty and present NVIDIA's strategy to minimize the instances where developers must engage in direct, low-level parallel programming, thereby shifting the "effort-to-performance" curve significantly to the left, meaning faster results with less effort.

Key Findings

▶ Watch: CUDA's secret: avoiding parallel programming (2:07)

The talk's core message revolves around a multi-layered approach to GPU programming, with a strong emphasis on abstraction and productivity over explicit parallel coding. Stephen Jones presented several key findings and principles:

1. Parallel Programming is Fundamentally Hard, and Should Be Avoided When Possible: This is the overarching principle. Jones argued that direct parallel programming is needed for a tiny fraction (perhaps 1%) of application code, and efforts should be focused on leveraging existing, highly optimized solutions for the remaining 99%.
2. CUDA is a Full Stack of Abstractions: CUDA is not just a low-level programming language but a comprehensive platform with multiple layers. These layers include high-level frameworks (e.g., PyTorch, TensorFlow, Houdini, LAMMPS), highly optimized math libraries (e.g., cuBLAS, cuFFT, cuRAND), and mid-level programming constructs (e.g., Cub, CUDA C++ Cooperative Libraries (CCCL)).
3. The "Effort vs. Performance" Curve Can Be Shifted Left: By using higher-level abstractions, developers can achieve "good enough" performance much faster, avoiding the long, difficult tail of optimization that yields diminishing returns. Frameworks and libraries provide rapid development and strong baseline performance.
4. GPU Architecture Relies on Oversubscription for Scaling: NVIDIA GPUs are designed to be oversaturated with work (more blocks than can fit on the Streaming Multiprocessors (SMs)). This ensures continuous execution, high throughput, and seamless scaling across different generations of hardware with varying SM counts.
5. Fixed Thread-per-SM Design for Future-Proofing: While the number of SMs increases with new GPU architectures, the number of threads per SM generally remains constant (around 2000). This design choice ensures that existing CUDA code continues to run efficiently on new hardware without requiring significant rewrites, promoting long-term code stability.
6. Introduction of CuTile for Simplified Custom Kernel Development: For scenarios where existing libraries don't suffice, Jones introduced CuTile (tile programming for CUDA). This new programming model bridges the gap between high-level libraries and low-level thread programming. It allows developers to define operations on "tiles" (arrays or tensors) rather than individual threads, with the compiler handling the complex thread management and optimization.
7. CuTile Achieves Near State-of-the-Art Performance with Less Effort: Early results for CuTile-generated code, such as a Llama 8B inference network, demonstrated performance within 10% of highly optimized, hand-tuned cuDNN kernels. This indicates a significant leap in productivity for custom kernel development without sacrificing much performance.

These findings collectively underscore NVIDIA's commitment to making GPU computing more accessible and productive, allowing developers to focus on algorithm design and application logic rather than the intricate details of parallel hardware management.

Technical Deep Dive

▶ Watch: Where parallel programming is truly needed in CUDA (2:47)

Stephen Jones meticulously detailed the underlying principles of CUDA and the design philosophy behind its programming models, from fundamental execution to advanced abstractions.

At its core, CUDA operates on a hierarchical execution model. A GPU program, or kernel, is launched as a grid of thread blocks. Each thread block is a group of threads that can cooperate and synchronize. Within a thread block, individual threads execute the kernel function. For example, in a simple AXPY (A\*X + Y) operation on a million elements, a typical CUDA approach might launch a million threads, with each thread processing one element.

However, the GPU hardware doesn't have a million physical cores. Instead, it features multiple Streaming Multiprocessors (SMs). Each SM is capable of running multiple thread blocks concurrently and manages thousands of threads. For instance, an older Kepler GPU might have 15 SMs, while a modern Hopper GPU boasts 132 SMs. A crucial concept Jones highlighted is oversubscription: launching significantly more thread blocks than the GPU has SMs. As one block completes execution on an SM, another block from the queue is immediately dispatched. This deep pipeline ensures that the GPU remains constantly busy, maximizing throughput and minimizing idle time. This strategy also inherently facilitates scaling: code written for an older GPU with fewer SMs will automatically utilize more SMs on newer, more powerful hardware without modification.

Jones initially presented a seemingly straightforward one-to-one mapping of threads to data elements. However, he quickly demonstrated that this isn't always optimal. By having each thread process multiple data elements (e.g., four elements per thread instead of one), several benefits arise:

• Reduced Overhead: Fewer blocks and threads launched means less scheduling overhead.
• Memory Coalescing/Batching: Accessing contiguous memory regions in larger chunks improves memory bandwidth utilization.
• Reduced Jitter: Smoother memory access patterns.

This simple optimization alone could yield a 2x speedup on a laptop GPU. This highlights a fundamental tension: while GPUs offer massive parallelism, intelligent management of thread-to-data mapping is critical for peak performance. Notably, while the number of SMs (and thus total cores) has increased dramatically across GPU generations, the number of threads per SM has remained relatively constant (around 2,000 threads). This design decision ensures backward compatibility and prevents requiring developers to constantly refactor their code for wider parallelism, which involves significant structural changes.

Jones then delved into the two fundamental types of parallelism: task parallelism and data parallelism.

• Task Parallelism: Independent programs or instances of a program running concurrently. In CUDA, this maps to independent thread blocks. Even if they run the same kernel, each block operates on a distinct portion of data and can progress independently.
• Data Parallelism: The same operation applied to multiple data elements simultaneously. Within a CUDA thread block, the threads (e.g., 2048 threads on an SM) perform data-parallel operations.

He illustrated data parallelism with the classic reduction algorithm (e.g., summing all elements in an array). A serial reduction takes linear time (N steps for N elements). A multi-core CPU can speed this up by dividing the array into chunks. However, the true power of parallel reduction, especially on a GPU, comes from a divide-and-conquer approach, where pairs of elements are summed, then pairs of those sums, and so on, leading to a logarithmic time complexity (log N steps). This is a monumental speedup for large N. The ability of GPUs to deploy hundreds of thousands of threads simultaneously makes them exceptionally well-suited for such algorithms.

However, implementing highly optimized parallel algorithms like reduction manually is incredibly complex. Jones referenced a paper by his colleague Mark Harris, which demonstrated a 30x speedup between a basic parallel reduction and a "ninja-level" optimized version involving intricate shared memory usage, template programming, and synchronization. Such optimizations require deep architectural understanding and are prohibitively time-consuming for most developers.

This led to the introduction of higher-level abstractions:

• Cub: A C++ template library for CUDA C++ Cooperative Libraries (CCCL), providing highly optimized building blocks for common parallel algorithms like sort, reduction, and prefix sum, operating at block-wide, sub-block (warp), and device-wide granularities. Its Pythonic counterpart is CUDA Cooperative.
• Device-Enabled Math Libraries: Functions from cuBLAS (linear algebra), cuFFT (Fast Fourier Transforms), and cuRAND (random number generation) can now be invoked directly within GPU kernels, further reducing the need for custom low-level implementations.

The culmination of this strategy is CuTile, a new programming model for CUDA. CuTile aims to bridge the gap between fixed-function, highly optimized libraries and the complex, low-level thread-based kernel programming. Instead of managing individual threads, developers program operations at the granularity of arrays or tensors (referred to as "tiles"). The developer is responsible for the task-parallel decomposition – how to break the problem into independent tiles (e.g., row-wise, column-wise, square tiles for image processing). However, the CuTile compiler then intelligently handles the data-parallel mapping of these tiles onto the GPU's Tensor Cores and SMs, optimizing thread management, memory access patterns, and synchronization automatically. This approach simplifies custom kernel writing significantly, especially for regular, structured data common in domains like image processing and linear algebra. By communicating just enough information (e.g., dense block, dimensions), the compiler can make highly informed optimization decisions that would be impossible for a human to manage across diverse hardware.

Experimental Setup & Results

▶ Watch: The tiny fraction of code needing parallel programming (4:50)

The talk primarily focused on conceptual models and programming paradigms rather than presenting extensive new experimental results from a specific research project. Stephen Jones's presentation was more about guiding developers on how to approach parallel programming with CUDA and showcasing NVIDIA's ongoing efforts to simplify this process.

However, he did provide a crucial anecdotal result regarding the effectiveness of the new CuTile programming model. He mentioned that a Llama 3 inference network (specifically a Llama 8B model) implemented using CuTile achieved performance within 10% of a corresponding implementation using cuDNN. cuDNN is NVIDIA's highly optimized, state-of-the-art deep neural network library, whose kernels are meticulously hand-tuned by NVIDIA's "ninjas" over many years. Achieving such close performance with a programming model that significantly reduces developer effort (weeks for CuTile vs. months for hand-tuning) is a compelling indicator of CuTile's potential. Jones also expressed optimism that as the CuTile compiler matures and gains more optimization intelligence over the next few years, it will asymptotically approach or even match cuDNN's performance.

To illustrate the potential for optimization in low-level kernels, Jones referenced a well-known example from his colleague Mark Harris's past work on parallel reductions. This work demonstrated that a basic parallel reduction implementation could be improved by a factor of 30x through a sequence of seven detailed optimizations, involving techniques like shared memory usage, loop unrolling, and careful synchronization. While this specific result wasn't a new experiment from the talk, it served to underscore the immense complexity and effort required for manual, "ninja-level" kernel optimization, thereby reinforcing the value proposition of higher-level abstractions like CuTile.

The talk did not detail specific hardware configurations (beyond mentioning Kepler and Hopper GPUs in general terms), specific datasets used for the Llama 8B example, or other baselines beyond the implicit comparison to cuDNN. The focus was on the architectural and programming model implications rather than a rigorous performance benchmark study.

Practical Implications

▶ Watch: Leveraging high-level GPU-accelerated frameworks (5:30)

The insights shared by Stephen Jones carry significant practical implications for a wide range of professionals interacting with AI/ML and HPC systems.

For Practitioners (Data Scientists, ML Engineers, Researchers):

• Prioritize Abstraction: The most important takeaway is to always start at the highest level of abstraction possible. Before considering writing custom CUDA kernels, first explore existing frameworks (PyTorch, TensorFlow, JAX), then specialized libraries (cuBLAS, cuFFT, cuDNN, GraphBLAS, etc.), and finally mid-level utilities like Cub or CCCL. This approach dramatically reduces development time and leverages years of NVIDIA's optimization efforts.
• Increased Productivity: By offloading complex parallel programming details to optimized libraries and compilers, practitioners can focus more on their core domain problems, algorithm design, and application logic. This accelerates the pace of experimentation and innovation.
• Accessible Performance: High GPU performance is no longer exclusively the domain of low-level experts. Even developers with less parallel programming experience can achieve highly competitive performance by correctly utilizing the CUDA stack.
• Future-Proofing: Code written using higher-level abstractions or the new CuTile model is more likely to run efficiently on future GPU architectures without requiring extensive refactoring, thanks to NVIDIA's commitment to scaling through oversubscription and fixed thread-per-SM design.

For Infrastructure Teams and System Architects:

• Stack Rationalization: Infrastructure teams should educate their users on the full breadth of the CUDA software stack, encouraging the use of higher-level components rather than direct kernel programming unless absolutely necessary.
• Resource Allocation: Understanding the GPU's oversubscription model can help in efficient job scheduling and resource allocation, ensuring GPUs are kept busy.
• Tooling Investment: Investing in tools and training that support CuTile and other high-level CUDA features will empower developers to build more performant and maintainable applications.

For Model Builders and Deployers:

• Faster Iteration: Model builders can rapidly prototype and iterate on novel architectures or custom operations by leveraging CuTile, which provides a balance of flexibility and performance.
• Deployment Efficiency: Models deployed with components built on optimized CUDA libraries or CuTile will inherently run more efficiently, reducing inference latency and increasing throughput, which is critical for real-time applications.
• Customization without Compromise: For unique model layers or data processing steps not covered by standard frameworks, CuTile offers a path to implement custom kernels with performance close to highly optimized libraries, avoiding the "performance cliff" often associated with bespoke code.

Tradeoffs and Limitations:

• Not a Universal Solution: While CuTile significantly simplifies kernel development for "regular chunks of data" (arrays, tensors), it is not a panacea. Irregular data structures, highly divergent control flows, or algorithms that don't map well to tile-based operations might still necessitate traditional, low-level thread programming.
• Learning Curve: While simpler than raw CUDA C++, CuTile still represents a new programming model with its own concepts (e.g., defining tile shapes, mapping strategies). Developers will need to invest some time in learning its paradigms.
• Compiler Maturity: As a relatively new model, the CuTile compiler is still under active development. While initial results are promising, its full optimization potential and coverage for all edge cases will evolve over time. Developers might encounter scenarios where hand-tuned kernels still offer marginal advantages.
• Debugging Complexity: While CuTile aims to simplify, debugging parallel code (even at a higher abstraction) can still be more complex than debugging serial code, especially for subtle issues related to data dependencies or synchronization.

Ultimately, NVIDIA's strategy, as articulated by Jones, aims to empower a broader range of developers to harness GPU power effectively, accelerating the development of cutting-edge AI and HPC applications while minimizing the burdens of low-level parallel programming.

Key Takeaways

• Avoid Direct Parallel Programming When Possible: For the vast majority of application code, leveraging NVIDIA's comprehensive CUDA software stack (frameworks, libraries) is more productive and often yields comparable or superior performance to custom low-level kernels.
• CUDA is a Multi-Layered Ecosystem: The CUDA platform offers a rich hierarchy of abstractions, from high-level AI/HPC frameworks down to specialized math libraries (cuBLAS, cuFFT, cuRAND) and mid-level parallel primitives (Cub, CCCL).
• GPU Scaling Leverages Oversubscription: Modern NVIDIA GPUs achieve high throughput and seamless scaling across generations by continuously executing a deep pipeline of thread blocks, ensuring SMs are rarely idle.
• CuTile Simplifies Custom Kernel Development: The new CuTile programming model allows developers to write custom GPU kernels by defining operations on arrays or tensors (tiles), with the compiler handling complex thread management and optimization, achieving performance close to hand-tuned libraries with significantly less effort.
• Mix and Match Programming Models: Developers are encouraged to combine different CUDA programming models—frameworks for general tasks, libraries for common functions, CuTile for custom tensor operations, and low-level thread programming for highly irregular or specialized cases—to achieve optimal productivity and performance.
• Focus on Problem Solving, Not Low-Level Details: NVIDIA's goal is to enable developers to spend less time on the intricacies of parallel hardware and more time on innovative algorithm design and application development.

About the Speaker(s)

Stephen Jones is a distinguished CUDA Architect at NVIDIA. With a deep and extensive background in GPU computing, he has been actively involved with CUDA since 2008, contributing significantly to its evolution and design. Jones specializes in thinking about how new hardware capabilities can be intuitively represented in programming languages, striving to make CUDA accessible and efficient for developers. His prior experience at NVIDIA includes working on the highly optimized FFT library within the CUDA ecosystem. Through his talks and work, he aims to share his profound understanding of GPU operation to help programmers build effective parallel applications.