Multi-GPU Communication Libraries for Scaling HPC and AI Workloads | NVIDIA GTC 2025

Multi-GPU Communication Libraries for Scaling HPC and AI Workloads

Speakers: Jiri Kraus, Principal Developer Technology, NVIDIA

Conference: NVIDIA GTC

YouTube: https://www.youtube.com/watch?v=kyQtbyR536I

Overview

In this comprehensive talk from NVIDIA GTC, Jiri Kraus, Principal Developer Technology at NVIDIA, delves into the critical role of multi-GPU communication libraries in addressing the escalating demands of High-Performance Computing (HPC) and Artificial Intelligence (AI) workloads. The presentation highlights how Message Passing Interface (MPI), the NVIDIA Collective Communications Library (NCCL), and NVIDIA SHMEM (NVSHMEM) serve as foundational technologies for efficiently orchestrating data movement across multiple GPUs, both within single nodes and across distributed systems. Kraus positions these libraries within a broader multi-GPU programming landscape, emphasizing their balance between generality—the ability to express diverse algorithms—and productivity for developers.

The motivation for multi-GPU computing is clear: the imperative to compute faster for quality-of-service requirements, such as real-time weather prediction, and the necessity to handle ever-larger models, like Large Language Models (LLMs) and high-resolution Computational Fluid Dynamics (CFD) simulations. As data continues to explode, leveraging the aggregate compute power of many GPUs becomes indispensable. This talk, the third in a three-part series, focuses on the "advanced multi-GPU scaling with communication libraries," underscoring their role as the underlying data movement infrastructure for higher-level programming models and task-based runtimes. By understanding and effectively utilizing these libraries, developers can unlock the full potential of modern GPU-accelerated systems, pushing the boundaries of scientific discovery and AI innovation.

Kraus meticulously dissects the technical intricacies of MPI, NCCL, and NVSHMEM, demonstrating how they enable CUDA-aware communication, allowing direct data transfers between GPU memories. He illustrates their distinct programming models—two-sided communication for MPI and NCCL, and one-sided communication for NVSHMEM—and their respective advantages in different scenarios. Through detailed performance analysis and real-world application benchmarks, the talk makes a compelling case for optimizing communication patterns, leveraging technologies like GPU Direct RDMA and NVLink, and adopting device-initiated communication paradigms to achieve unprecedented scaling and efficiency in complex HPC and AI applications.

Background

▶ Watch: Motivation for multi-GPU scaling in HPC/AI (0:00)

The relentless pursuit of faster computation and the ability to tackle increasingly complex problems in both HPC and AI have made multi-GPU systems the norm rather than the exception. Whether it's training massive deep learning models, simulating intricate physical phenomena, or performing real-time data analytics, single-GPU performance often hits fundamental limits. The challenge then shifts to effectively coordinating and exchanging data between multiple GPUs, which are inherently separate compute units, to ensure they work cooperatively on a single problem. This data exchange, or communication, often becomes the bottleneck that limits overall application scalability and performance.

Historically, Message Passing Interface (MPI) has been the de facto standard for inter-process communication in distributed computing, providing a robust API for point-to-point and collective operations. However, traditional MPI implementations were CPU-centric, requiring data to be staged through host memory before being sent or received, leading to significant overheads when dealing with GPU-resident data. This problem spurred the development of CUDA-aware communication libraries, which can directly operate on GPU memory, eliminating the need for costly CPU-GPU memory transfers.

Several foundational technologies underpin the efficiency of modern multi-GPU communication. Unified Virtual Addressing (UVA), introduced with CUDA compute capability 2.0, provides a unified virtual address space for all memory allocated in a CUDA program, regardless of whether it resides on the CPU or a GPU. This allows libraries to query the physical location of a pointer and take appropriate action for efficient data movement. Building on UVA, GPU Direct Peer-to-Peer (P2P) enables direct data transfers between GPUs within the same node, bypassing CPU memory and leveraging high-bandwidth interconnects like PCI Express and NVLink. For multi-node communication, GPU Direct RDMA (Remote Direct Memory Access) allows third-party PCIe devices, such as InfiniBand or RoCE network adapters, to directly read from and write to GPU memory. These technologies are crucial for minimizing latency and maximizing bandwidth, making efficient multi-GPU communication possible.

The evolution of multi-GPU programming has seen a spectrum of approaches, from high-level frameworks that abstract away communication details to low-level libraries offering fine-grained control. This talk focuses on the latter, presenting MPI, NCCL, and NVSHMEM as the essential building blocks that enable advanced multi-GPU scaling. These libraries are not just about raw data transfer; they are about orchestrating complex communication patterns, minimizing synchronization overheads, and maximizing the overlap between computation and communication, thereby addressing the core problem of efficient data exchange in the ML/systems space.

Key Findings

▶ Watch: Core topic: Nickel, MPI, Nvshmem for GPU communication (2:10)

The talk presents several key findings that underscore the importance and capabilities of modern multi-GPU communication libraries for HPC and AI workloads:

• CUDA-aware Communication is Paramount: All three libraries—MPI, NCCL, and NVSHMEM—are CUDA-aware, meaning they can directly operate on GPU memory. This capability is fundamental to achieving high performance by eliminating intermediate CPU memory staging, which is a significant bottleneck in traditional CPU-centric communication.
• GPU Direct Technologies Drive Performance: GPU Direct RDMA for multi-node communication and GPU Direct Peer-to-Peer (P2P) for intra-node transfers (especially with NVLink) are critical enablers. When properly configured and utilized, these technologies allow network adapters and GPUs to move data directly, bypassing the CPU and saturating network and interconnect bandwidths. Performance comparisons with and without GPU Direct show dramatic improvements, with up to 1.4x-1.6x speedups in real-world applications like ICON.
• NCCL Excels in GPU-Accelerated Collectives and Overlap: The NVIDIA Collective Communications Library (NCCL) is highly optimized for GPU-accelerated collective operations, essential for data-parallel deep learning. Its CUDA stream-aware interface allows communication calls to be submitted to the same GPU stream as compute kernels, making them non-blocking with respect to the host. This enables significant overlap between computation and communication, minimizes CPU-GPU synchronization, and leads to denser GPU timelines. For the VASP application, NCCL delivered a 2.6x speedup over CUDA MPI by leveraging this overlap and GPU-accelerated reductions.
• NVSHMEM Enables Maximal Overlap and Kernel Fusion via Device-Initiated Communication: NVSHMEM, an implementation of the OpenSHMEM standard, introduces a Partitioned Global Address Space (PGAS) model with a symmetric heap and supports device-initiated communication. This allows GPU kernels to directly issue communication operations (puts, gets, atomics) without CPU involvement. This paradigm drastically reduces kernel launch overheads, enables natural communication-computation overlap through threading, and facilitates kernel fusion, where multiple operations (compute, pack, communicate) are combined into a single GPU kernel. For the QUDA Lattice QCD application, device-initiated NVSHMEM achieved up to a 1.7x speedup over MPI in strong scaling scenarios, particularly at lower precision and extreme scales (512 GPUs), by overcoming latency barriers and eliminating gaps between small kernel launches.
• Interoperability for Flexible Optimization: MPI, NCCL, and NVSHMEM are fully interoperable. This means developers can use NCCL or NVSHMEM within an existing MPI application, selectively optimizing performance-critical communication paths without a full rewrite. This flexibility is crucial for accelerating legacy HPC codes and building complex AI workflows.
• Significant Application-Level Speedups: The benefits of these libraries and underlying hardware technologies are directly translated into substantial performance gains for real-world applications:
• ICON (Numerical Weather Prediction): 1.4x speedup on 8-12 nodes, up to 1.6x on 32 nodes with CUDA MPI and GPU Direct technologies.
• VASP (Molecular Dynamics): 2.6x speedup on 32 nodes using NCCL's GPU-accelerated collectives and computation-communication overlap.
• QUDA (Lattice QCD): Up to 1.7x speedup on 512 GPUs with NVSHMEM's device-initiated communication and kernel fusion.
• Future Enhancements: Kernel-initiated communication, currently a key feature of NVSHMEM, is actively being developed and will soon be available in NCCL, further extending the possibilities for optimal overlap and efficiency across the ecosystem.

Technical Deep Dive

▶ Watch: Launching multi-GPU applications with launchers (4:15)

The talk provides a deep dive into the technical underpinnings and programming models of MPI, NCCL, and NVSHMEM, highlighting their unique strengths and how they leverage NVIDIA's hardware and software stack.

MPI (Message Passing Interface)

MPI is a venerable standard for inter-process communication, offering a rich API for point-to-point communication (MPI_Send, MPI_Recv) and collective operations (MPI_Allreduce, MPI_Gather). Its strength lies in its generality and widespread adoption across diverse HPC applications and languages. The key to its efficiency in multi-GPU environments is its CUDA-aware implementations.

1. Foundational Technologies:

• Unified Virtual Addressing (UVA): Introduced with CUDA compute capability 2.0, UVA maps all GPU and host memory into a single virtual address space. This allows a CUDA-aware MPI library to inspect any pointer passed to MPI_Send or MPI_Recv, determine if it points to CPU or GPU memory, and on which GPU, then take the most efficient data transfer path.
• GPU Direct Peer-to-Peer (P2P): For intra-node communication, P2P enables direct data transfers between GPUs via PCI Express or NVLink. NVLink, in particular, offers significantly higher bandwidth than PCIe, making P2P crucial for high-speed local communication.
• GPU Direct RDMA: For multi-node communication, GPUDirect RDMA allows third-party PCIe devices, such as InfiniBand or RoCE (RDMA over Converged Ethernet) network adapters, to directly read data from or write data to GPU memory. This bypasses the CPU and host memory, eliminating intermediate staging and associated overheads.

1. Communication Model: MPI primarily uses a two-sided communication model, where both the sender (MPI_Send) and receiver (MPI_Recv) explicitly participate in the data transfer. In CUDA-aware MPI, these calls can take device pointers directly.

• Optimal Path (GPUDirect RDMA): When GPUDirect RDMA is available and properly configured, the network adapter directly pulls data from the source GPU memory and pushes it to the destination GPU memory. This is highly efficient, often showing no GPU activity in profilers, as the network adapter masters the transfer, only revealing activity on the PCIe bus.
• Staging Path: If GPUDirect RDMA is not available (e.g., due to system topology or network limitations), CUDA-aware MPI will still accept device pointers but internally stages data through CPU buffers. This involves cudaMemcpyDeviceToHost and cudaMemcpyHostToDevice operations, but these transfers are pipelined with network transfers to mitigate performance loss.
• Non-CUDA MPI: Without CUDA-awareness, developers must manually copy data from GPU to host, then use CPU-MPI, and then copy from host to GPU. This serializes operations, introduces significant stalls in the pipeline, and results in substantial performance degradation due to lack of overlap.

NCCL (NVIDIA Collective Communications Library)

NCCL is purpose-built for efficient GPU-accelerated communication, particularly excelling at collective operations crucial for data-parallel deep learning (e.g., all-reduce, all-gather). Since version 2.8, it also supports point-to-point send/receive operations, allowing it to express arbitrary communication patterns.

1. GPU-Centric Design: NCCL operations are translated into GPU kernels that run on CUDA streams. This design has several implications:

• CUDA Stream-Aware: Communication operations are submitted to a specific CUDA stream. They are blocking with respect to that stream (i.e., they won't execute until prior work in the stream completes) but non-blocking with respect to the host CPU. This allows the CPU to immediately enqueue more work or proceed with other tasks, fostering greater overlap between computation and communication.
• Minimizing CPU-GPU Synchronization: By executing communication directly on the GPU, NCCL reduces the need for explicit CPU-GPU synchronization points (like cudaStreamSynchronize), which can introduce significant latency.
• Leveraging Compute Throughput: NCCL can utilize the GPU's compute capabilities to accelerate collective operations, achieving very high bandwidth, especially when NVLink and SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) are employed.

1. Communication Model: NCCL primarily offers collective operations, but its point-to-point send/receive also follows a two-sided model, similar to MPI, where sender and receiver explicitly specify the communication. The key distinction is its GPU-native execution.

• Computation-Communication Overlap: The stream-aware nature of NCCL allows developers to easily overlap computation and communication. Instead of synchronizing the CPU after a compute kernel to initiate communication, the communication operation can be enqueued in the same stream, ensuring it starts as soon as the compute kernel finishes, without CPU intervention. This leads to significantly denser GPU timelines and improved overall throughput.

NVSHMEM (NVIDIA SHMEM)

NVSHMEM is an implementation of the OpenSHMEM standard, designed for high-performance, one-sided communication within a Partitioned Global Address Space (PGAS) model.

1. Symmetric Heap and One-Sided Communication:

• Symmetric Heap: The core principle is the allocation of memory from a symmetric heap using nvshmem_malloc. This ensures that all communicating processes (called Processing Elements or PEs) allocate a chunk of the same size at the same relative offset in their respective memory spaces. This simplifies addressing, as a PE can calculate the address of its peer's data without explicit communication about offsets, enabling one-sided operations.
• One-Sided Model: NVSHMEM supports one-sided communication via put (write to a remote PE's memory) and get (read from a remote PE's memory) operations, as well as atomics for synchronization. In this model, only one PE explicitly initiates the communication; the target PE does not need to explicitly participate.
• Requirements: NVSHMEM critically relies on GPU Direct Peer-to-Peer and GPU Direct RDMA to set up direct mappings and enable one-sided access without involving the target CPU or GPU. If these are not available, NVSHMEM cannot function in its intended high-performance manner.

1. Device-Initiated Communication: This is a hallmark feature of NVSHMEM, offering unparalleled opportunities for communication-computation overlap and kernel fusion.

• CPU-Initiated vs. Device-Initiated: Traditionally, even with CUDA-aware libraries, communication is initiated by the CPU. This means the CPU launches a compute kernel, waits for it (implicitly or explicitly), then launches a communication operation, and then waits again before launching the next compute kernel. This introduces offload latencies and synchronization points on the critical path, limiting overlap.
• GPU-Native Communication: With device-initiated communication, GPU kernels themselves issue put, get, and atomic operations. As soon as a piece of data is computed by a thread, that thread (or warp) can immediately initiate its transfer to a remote GPU.
• Benefits:
• Reduced Offload Latencies: Fewer kernel launches and API calls from the CPU.
• Natural Overlap: Communication can be interwoven with computation at the thread or warp level, leveraging the GPU's parallel execution model to hide communication latency.
• Kernel Fusion: The ability to issue communication from within a kernel allows for fusing multiple kernels (e.g., compute, pack, communicate, unpack) into a single, larger kernel. This keeps more state in fast on-chip memory (registers, shared memory), avoids costly global memory round trips, and eliminates gaps between sequential kernel launches.
• Fine-Grained vs. Coarse-Grained: NVSHMEM supports both fine-grained, thread-level communication (nvshmem_put_p) for maximum overlap on memory fabrics like NVLink, and larger-group communication for efficiency over network fabrics like InfiniBand.

1. Memory Consistency APIs: NVSHMEM provides explicit memory consistency APIs, giving developers precise control over when communication operations complete and memory changes become visible to other PEs, which can further limit the amount of synchronization needed.

Experimental Setup & Results

▶ Watch: Introduction to the Message Passing Interface (MPI) (5:59)

The talk provides compelling evidence of the performance benefits of these communication libraries through a combination of standard benchmarks and real-world application use cases, run on advanced NVIDIA hardware.

OSU Benchmarks (CUDA MPI)

• Setup: The OSU (Ohio State University) benchmarks were used to measure point-to-point latency and bandwidth. The system used for comparison was Jules Booster in ULIC, equipped with an HDR200 InfiniBand network.
• Intra-Node Performance:
• CUDA MPI with GPUDirect RDMA: This configuration demonstrated the highest performance, saturating the HDR200 InfiniBand network bandwidth. The profiling showed minimal GPU activity, indicating that the network adapter directly managed data transfers to and from GPU memory, with activity primarily visible on the PCIe bus.
• CUDA MPI with Staging: When GPUDirect RDMA was not available, CUDA MPI performed internal staging, moving data through CPU memory. While it still achieved reasonable performance by pipelining cudaMemcpyDeviceToHost and network transfers, it showed a noticeable performance drop compared to GPUDirect RDMA.
• Non-CUDA MPI (Manual Staging): Manually copying data from GPU to CPU, performing CPU MPI, and then copying back to GPU resulted in significantly worse performance. The profiling timeline showed large gaps and stalls, with no overlap between cudaMemcpy operations and network transfers, highlighting the inefficiencies of a CPU-centric approach.
• Inter-Node Performance:
• NVLink vs. PCIe: When comparing intra-node communication, the performance differences between using NVLink (with GPUDirect P2P) and PCIe were amplified. NVLink's higher bandwidth capabilities provided even greater benefits, making GPUDirect P2P essential for maximizing intra-node GPU-to-GPU communication.

ICON (Numerical Weather Prediction)

• Application: ICON (ICOsahedral Nonhydrostatic Model) is a widely used numerical weather prediction and climate simulation model, developed in Germany and Switzerland.
• Workload: A cubic workload at 10 km horizontal resolution with 190 vertical edges, a typical operational setup.
• Comparison: CPU MPI vs. CUDA MPI (leveraging GPU Direct P2P for single-node analysis).
• Results:
• Profiling: The CPU MPI version showed distinct gaps in the GPU timeline, characterized by a cudaMemcpyDeviceToHost operation, followed by a long period of no GPU activity (while CPU MPI executed), and then a cudaMemcpyHostToDevice. In contrast, the CUDA MPI profile for a single node showed direct peer-to-peer transfers and a much denser timeline with fewer gaps, indicating more efficient execution and better overlap.
• Scaling Experiment: Running on the EOS supercomputer with H100 GPUs and an NDR400 network:
• At 8 to 12 nodes, a 1.4x speedup was observed for CUDA MPI over CPU MPI, demonstrating its relevance for production-scale workloads.
• For academic strong scaling exercises, at 32 nodes, the speedup increased to 1.6x. This illustrates how efficient GPU communication becomes even more critical as the number of GPUs increases.

VASP (First-principles Molecular Dynamics)

• Application: VASP (Vienna Ab initio Simulation Package) is a versatile package for ab initio calculations based on first-principles molecular dynamics, commonly used in materials research.
• Workload: A Hafnium simulation with 216 atoms.
• Comparison: An existing CUDA MPI implementation versus a newer NCCL communication option. The NCCL version specifically leveraged GPU-accelerated collectives and the CUDA stream-aware interface to implement computation-communication overlap, which was not present in the MPI version.
• Results:
• Profiling: The CUDA MPI profile showed significant "white space" and serial execution, particularly during reduction operations which were performed on the CPU. The NCCL version, however, displayed kernels running in parallel (two rows of kernels) with NCCL kernels overlapping compute kernels. The GPU remained busy almost continuously, demonstrating effective communication-computation overlap and efficient GPU-accelerated reductions.
• Scaling Experiment: On the EOS supercomputer, scaling from 4 to 32 nodes, the NCCL version achieved a 2.6x speedup over CUDA MPI. This significant gain highlights NCCL's strength in optimizing collective operations and enabling effective overlap for computationally intensive scientific applications.

QUDA (Lattice QCD)

• Application: QUDA (QCD on CUDA) is an open-source library providing building blocks for Lattice QCD simulations on GPUs.
• Workload: Strong scaling the Wilson D-slash operator.
• Comparison: CUDA MPI versus NVSHMEM utilizing device-initiated communication.
• Results:
• Profiling (Single Node): The MPI version showed gaps between packing kernels, interior compute kernels, and data copies. These gaps were attributed to the latency of launching multiple small kernels and communication taking longer than the interior compute kernel.
• NVSHMEM with Fused Kernels: The NVSHMEM approach enabled a fully fused kernel, where packing, computation, and communication were integrated. This eliminated all the "white space" on the GPU timeline. Furthermore, the ability to launch kernels for the next iteration while the current one was running (kernel pre-launch) contributed to continuous GPU utilization.
• Strong Scaling Experiment: On the EOS supercomputer with H100 GPUs and an NDR400 network, scaling up to 512 GPUs:
• NVSHMEM showed increasingly larger performance differences at higher scales, particularly with lower precision workloads (where compute intensity is lower, making communication a relatively larger bottleneck).
• At 512 GPUs, NVSHMEM achieved an up to 1.7x speedup over MPI for lower precision.
• The MPI version struggled to scale effectively beyond 128 GPUs, with a noticeable performance "bump" from 128 to 256 GPUs due to partition changes requiring the launch of more boundary kernels. NVSHMEM's device-initiated, fused approach effectively bypassed these latency barriers, demonstrating superior strong scaling capabilities.

Practical Implications

▶ Watch: Simple CUDA MPI send/receive example walkthrough (8:00)

The insights presented in this talk have profound practical implications for various stakeholders involved in developing, deploying, and optimizing HPC and AI applications on multi-GPU systems.

For Practitioners (Application Developers & Researchers)

• Strategic Library Selection: The choice of communication library should be driven by the application's specific needs:
• For data-parallel deep learning with heavy use of collective operations (e.g., all-reduce), NCCL is the clear choice due to its GPU-accelerated collectives and stream-aware interface for overlap.
• For legacy HPC codes that heavily rely on MPI, migrating to CUDA-aware MPI is the first step. Then, identify performance-critical sections and selectively integrate NCCL for collectives or NVSHMEM for one-sided, device-initiated communication to achieve targeted optimizations.
• For latency-sensitive HPC applications that can benefit from fine-grained, one-sided communication and kernel fusion, NVSHMEM with its device-initiated communication model offers the most significant gains, especially for strong scaling.
• Embrace Overlap: Actively design algorithms to overlap computation and communication. Libraries like NCCL and NVSHMEM facilitate this by being non-blocking with respect to the host and supporting device-initiated operations. This means avoiding explicit CPU synchronizations wherever possible.
• Leverage CUDA Streams and Graphs: Utilize CUDA streams to manage dependencies and allow for asynchronous execution. For even lower overhead, especially with recurring communication patterns, CUDA graphs can capture entire sequences of kernel launches and communication operations, reducing CPU overheads significantly. All three libraries are CUDA stream-aware and can be captured in graphs.

For Infrastructure Teams & System Administrators

• Hardware Configuration is Key: Ensure that multi-GPU systems are properly configured to enable GPU Direct Peer-to-Peer (P2P) and GPU Direct RDMA. This includes using NVLink for intra-node GPU communication and InfiniBand or RoCE networks with RDMA support for multi-node communication. The performance benefits of these libraries are maximized when the underlying hardware allows for direct GPU-to-GPU and GPU-to-NIC transfers, bypassing the CPU.
• Monitoring and Profiling: Use tools like NVIDIA Nsight Systems to profile applications. The visual timelines help identify communication bottlenecks, white spaces, and opportunities for overlap, guiding optimization efforts. Understanding whether GPUDirect RDMA is active (by observing PCIe bus activity without GPU compute activity during communication) is crucial for diagnostics.

For Model Builders & Deployers (AI/ML)

• Scaling LLMs and Large Models: As LLMs and other large AI models grow, efficient multi-GPU communication is non-negotiable for both training and inference. NCCL is foundational for data-parallel training, and its evolving support for point-to-point and future kernel-initiated communication will be vital for more complex parallelization strategies like tensor or pipeline parallelism.
• Performance Tradeoffs: While NVSHMEM offers extreme performance for specific patterns, its requirement for a symmetric heap and one-sided model might require more significant code restructuring compared to integrating NCCL into an existing MPI application. Developers need to weigh the potential performance gains against the development effort.

Tradeoffs and Limitations

• MPI: Most general and widely supported, but can be the slowest if not CUDA-aware or if GPUDirect RDMA/P2P are unavailable, forcing CPU staging. Interoperability with NCCL/NVSHMEM provides a good path for optimization.
• NCCL: Highly optimized for GPU collectives and increasingly capable for point-to-point, offering excellent overlap. However, it requires GPU-resident data and is primarily GPU-centric. Its future support for kernel-initiated communication will further enhance its capabilities.
• NVSHMEM: Provides the most control for one-sided, device-initiated communication, leading to maximal overlap and kernel fusion. However, it strictly requires GPUDirect P2P/RDMA to function efficiently due to its one-sided model and symmetric heap. It might also demand a more specific programming style and memory allocation model.
• Complexity: As one moves down the "productivity" axis towards more "general" and low-level libraries, the complexity of programming increases. While higher performance is achievable, it often comes with a steeper learning curve and more intricate code management.

In summary, the practical implication is clear: to effectively scale HPC and AI workloads on modern multi-GPU systems, developers and infrastructure teams must deeply understand and strategically employ these communication libraries, ensuring the underlying hardware is optimally configured to support their advanced features. The continuous evolution of these libraries, particularly towards device-initiated communication, promises even greater efficiency in the future.

Key Takeaways

• CUDA-aware communication is essential: Modern multi-GPU/multi-node applications must leverage CUDA-aware MPI, NCCL, and NVSHMEM to directly operate on GPU memory, eliminating costly CPU staging.
• GPU Direct technologies are foundational: GPU Direct Peer-to-Peer (P2P) (with NVLink) and GPU Direct RDMA are critical for high-bandwidth, low-latency communication by enabling direct data transfers between GPUs and network adapters, bypassing the CPU.
• NCCL excels in collectives and overlap: The NVIDIA Collective Communications Library (NCCL) is highly optimized for GPU-accelerated collective operations and, with its stream-aware interface, enables significant computation-communication overlap, crucial for deep learning workloads.
• NVSHMEM offers maximal efficiency via device-initiated communication: NVSHMEM's one-sided communication model, symmetric heap, and unique device-initiated communication capability allow GPU kernels to directly manage communication, leading to unparalleled overlap, kernel fusion, and superior strong scaling, especially in latency-sensitive HPC applications.
• Interoperability allows flexible optimization: MPI, NCCL, and NVSHMEM can be used together in the same application, allowing developers to selectively apply the most appropriate library for specific communication patterns or to incrementally optimize existing MPI codebases.
• Configuration and profiling are vital: Achieving peak performance requires careful system configuration to enable GPU Direct technologies and diligent profiling (e.g., with Nsight Systems) to identify and address communication bottlenecks and maximize overlap.

About the Speaker(s)

Jiri Kraus is a Principal Developer Technology at NVIDIA. He is deeply involved with the GPU Communications (GPU Comms) team, focusing on the development and optimization of multi-GPU communication libraries and programming models for scaling HPC and AI workloads. His expertise lies in helping developers understand and effectively utilize technologies like MPI, NCCL, NVSHMEM, GPU Direct, and CUDA graphs to achieve high performance on NVIDIA's GPU platforms. Kraus frequently shares his knowledge at conferences like NVIDIA GTC, providing detailed technical insights and practical guidance on advanced multi-GPU programming. He has also presented similar talks in previous years, demonstrating his ongoing commitment to advancing the state-of-the-art in GPU communication.