Kubernetes WG Device Management - GPUs, TPUs, NICs and More With DRA - Kevin Klues & Patrick Ohly

Kubernetes WG Device Management - GPUs, TPUs, NICs and More With DRA - Kevin Klues & Patrick Ohly

Speakers: Kevin Klues; Patrick Ohly

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=Z_15EyXOnhU

Overview

This talk provides a comprehensive update on the Kubernetes Working Group Device Management, primarily focusing on the significant advancements and ongoing development of Dynamic Resource Allocation (DRA). Presented by co-chairs Patrick Ohly and Kevin Klues, who were instrumental in initiating DRA within Kubernetes, the session highlights the project's evolution from an early effort to a beta-level feature poised for General Availability (GA). The core mission of DRA is to enable simple, efficient configuration, sharing, and allocation of accelerators and other specialized devices, extending Kubernetes' capabilities beyond traditional microservice management to support modern workloads like AI inference and training.

The speakers emphasize that DRA addresses critical limitations of Kubernetes' original device management model, offering a more flexible and expressive way to describe and consume hardware resources such as GPUs, TPUs, and advanced NICs. This is particularly crucial as compute-intensive applications increasingly rely on diverse and specialized hardware. The session details the architectural changes, new API objects, and a host of recently introduced features designed to enhance resource description, scheduling, and lifecycle management, making Kubernetes a more robust platform for complex, hardware-accelerated workloads.

The importance of DRA is underscored by its wide-reaching impact across Kubernetes' architecture, affecting autoscaling, networking, node interaction, and especially scheduling decisions. The working group's collaborative efforts across various SIGs (Special Interest Groups) demonstrate a concerted push to mature DRA, with a clear roadmap towards GA and continuous feature expansion. This talk serves as a vital update for anyone involved in deploying or managing hardware-accelerated applications on Kubernetes, offering insights into current capabilities and future directions.

Background

▶ Watch: Introduction to Kubernetes WG Device Management and mission (0:00)

Before the advent of Dynamic Resource Allocation (DRA), Kubernetes managed specialized hardware through the Device Plugin mechanism and the extended resource API. While functional for basic allocation, this model presented significant limitations as hardware capabilities grew more complex and user requirements became more nuanced. The extended resource API essentially treated devices as opaque, countable resources (e.g., "nvidia.com/gpu: 1"). This approach was insufficient for scenarios requiring specific device attributes, shared access, or multi-node coordination.

The limitations of the traditional model included:

• Lack of Device Description: Users could only request a quantity of a resource, not specific attributes like GPU memory size, vendor, product ID, or compute capabilities. This meant a user requesting "one GPU" might receive a device unsuitable for their specific workload.
• Inflexible Allocation: Device plugins offered limited flexibility. They typically allocated entire devices, making it challenging to implement fractionalization or to share parts of a device among multiple containers in a controlled manner.
• No Attribute-Based Scheduling: The Kubernetes scheduler could only match based on resource counts, not on detailed device characteristics. This often led to suboptimal placement or the inability to fulfill requests requiring specific hardware configurations.
• Complex Sharing: Sharing a single device among multiple pods or containers was cumbersome and often required custom, out-of-band solutions.
• Limited Lifecycle Management: The existing model lacked native mechanisms for managing device health, maintenance, or dynamic changes in device availability.
• Multi-Node Challenges: Coordinating resource allocation across multiple nodes, especially for interconnected hardware like NVIDIA's NVLink, was virtually impossible within the standard framework.

Recognizing these shortcomings, the Kubernetes community, spearheaded by individuals like Patrick Ohly and Kevin Klues, initiated the development of DRA. The goal was to overcome these limitations by introducing a more expressive API and a redesigned resource management pipeline. This new approach sought to provide granular control over device allocation, enable attribute-based scheduling, facilitate complex sharing patterns, and integrate device lifecycle management directly into Kubernetes, moving beyond its original roots as solely a microservices orchestrator. The formation of the Working Group Device Management nearly a year prior to the talk formalized this effort, bringing together diverse stakeholders from various SIGs to drive DRA's development and adoption.

Key Findings

▶ Watch: Resource Slice: Detailed hardware description for drivers (3:15)

The talk highlights several pivotal advancements and milestones for Dynamic Resource Allocation (DRA), signaling its maturity and growing adoption within the Kubernetes ecosystem.

1. Beta Release in Kubernetes 1.32: A significant achievement, DRA reached beta in Kubernetes 1.32. This marked a critical step, indicating that the core API and functionality are stable enough for broader testing and initial production use. The speakers emphatically state, "DRA is here to stay," reinforcing its commitment to the project's longevity. This was made possible by a key architectural shift, moving all resource matching logic into the Scheduler, simplifying the overall design and unblocking progress.

1. Rapid Feature Expansion in Kubernetes 1.33: Following the beta release, an impressive array of new features were introduced or promoted to alpha/beta in Kubernetes 1.33, demonstrating active development and community engagement:

• Driver-owned Resource Claim Status: Promoted to beta, this feature is crucial for network devices to publish dynamic information like IP addresses within the Resource Claim status.
• Partitionable Devices: Introduced as alpha, this allows for defining overlapping hardware partitions within a single physical device, enabling more granular resource allocation where picking one partition might consume resources needed by another.
• Device Taints and Tolerations: An alpha feature, this provides a management mechanism similar to node taints, allowing administrators to mark specific devices as unhealthy or under maintenance, preventing new pods from using them and potentially evicting existing ones.
• Prioritized Alternatives in Device Requests: Also alpha, this enhances user flexibility by allowing them to express preferences for a list of acceptable devices (e.g., "vendor A preferred, but vendor B is acceptable"), with the scheduler attempting to satisfy one of the alternatives.
• Admin Access Enhancements: Improvements were made to the privileged mode for administrators, standardizing a required namespace label to ensure secure, built-in access control for overriding device usage.
• Core DRA API Simplification (v1beta2): Structural changes were introduced in v1beta2 to simplify the API, making it easier to promote to GA. Importantly, v1beta1 remains available and interoperable.

1. Reliability and Deployment Improvements:

• Predefined RBAC Rules: Standardized Role-Based Access Control rules are now provided, simplifying secure deployment.
• Seamless Driver Upgrade Mode: A groundbreaking feature for Kubelet plugins, this enables rolling updates for DRA drivers where two driver containers can run in parallel, ensuring zero downtime during upgrades.

1. GA Target for Kubernetes 1.34: The core DRA is targeted for General Availability in Kubernetes 1.34, along with the claim status feature. This signifies a major milestone, making DRA a fully supported and recommended component of Kubernetes. Other alpha features from 1.33 are expected to move to beta in the same cycle.

1. NVIDIA's Comprehensive DRA Adoption: Kevin Klues highlights NVIDIA's success in leveraging DRA. By Kubernetes 1.33, NVIDIA's GPU driver supports 11 out of 12 previously outlined use cases for GPUs, a significant improvement from 6/12 in 1.30 and 9/12 in 1.31. This demonstrates DRA's effectiveness in addressing real-world, complex GPU allocation scenarios.

1. Emergence of Multi-Node Resource Management: A new and critical use case identified is the support for cross-node resources, specifically multi-node NVLink for systems like the NVIDIA GB200 NVL72. This introduces a new API server object called a compute domain, which abstracts multi-node NVLink configurations, allowing DRA to allocate and manage these high-bandwidth, interconnected GPU systems across multiple Kubernetes nodes.

These findings collectively illustrate DRA's rapid maturation, its ability to tackle complex hardware challenges, and its increasing relevance for advanced AI/ML and high-performance computing workloads within Kubernetes.

Technical Deep Dive

▶ Watch: DRA reached Beta in Kubernetes 1.32 (5:45)

Dynamic Resource Allocation (DRA) fundamentally rearchitects how Kubernetes interacts with specialized hardware, moving beyond the limitations of the extended resource API and Device Plugins. At its core, DRA introduces a more expressive, attribute-driven model for resource description and consumption.

The DRA framework revolves around four key components:

1. New API Objects: DRA introduces two primary API objects:

• Resource Slice: This object, published by DRA drivers, provides a comprehensive, structured description of the hardware available on a node. Unlike the opaque resource counts of old, a Resource Slice lists individual devices, each with a rich set of attributes. These attributes can include vendor, product ID, amount of GPU RAM, number of compute units, or any other relevant characteristic. This structured format, referred to as Structured Parameters, allows for logical conclusions and simulations by the scheduler.
• Resource Claim: Users define their hardware requirements using Resource Claims. This standalone object specifies the desired attributes a device must possess. It typically references a Device Class, which categorizes devices from a specific vendor. Within a Resource Claim, users can employ cell expressions to define flexible requirements, such as "at least 16GB of GPU memory" or "a device from vendor A or vendor B." A crucial feature is control sharing, where multiple pods or containers can reference the same Resource Claim, ensuring they are allocated the same physical hardware instance.

1. Updated Scheduler: A key architectural change in DRA is the relocation of resource matching logic to the Kubernetes Scheduler. Previously, device plugins would reserve resources, but DRA empowers the scheduler to directly evaluate Resource Claims against available Resource Slices. The scheduler performs logical matching, considering all specified attributes and expressions, to find the optimal device(s) for a pod. This shift was critical for unblocking DRA's progress to beta.

1. Kubelet and DRA Driver Interaction: Once a pod is scheduled, the Kubelet interacts with a DRA driver running on the node. This driver, similar in concept to a device plugin but using a different API, is responsible for setting up the hardware and making it available within the container's environment. This typically involves actions like mounting device files, setting environment variables, or configuring specific device capabilities.

Advanced Features and API Evolution

DRA has seen rapid feature development since its beta release in Kubernetes 1.32, with many enhancements targeting Kubernetes 1.33 and 1.34:

• Partitionable Devices (Alpha in 1.33): This feature allows a single physical device to be logically partitioned. For instance, a high-end GPU might be divided into smaller, isolated compute units. The challenge addressed here is ensuring that selecting one partition correctly accounts for shared underlying resources, preventing oversubscription or conflicts.
• Device Taints and Tolerations (Alpha in 1.33): Inspired by node taints, this mechanism allows administrators to mark specific devices with "taints" (e.g., unhealthy, maintenance). Pods that require these devices must explicitly declare "tolerations" for the taint to be scheduled. This provides granular control over device lifecycle management and prevents workloads from being scheduled on faulty or unavailable hardware.
• Prioritized Alternatives in Device Requests (Alpha in 1.33): Users can now specify a list of alternative device requirements within a Resource Claim. The scheduler will attempt to fulfill the first preference, falling back to subsequent alternatives if the primary choice is unavailable. This enhances workload flexibility and resilience.
• Admin Access Enhancements (1.33): The privileged mode, allowing administrators to access devices even if they are in use, was made more secure. Kubernetes now includes a standardized label that must be set in a namespace to enable the creation of privileged Resource Claims, preventing misuse.
• Core API Simplification (v1beta2): To prepare for GA, the core DRA API underwent structural changes in v1beta2 to simplify its design. This ensures a cleaner, more stable API for long-term use, while maintaining backward compatibility with v1beta1.
• Reliability Features (1.33):
• Predefined RBAC Rules: Standardized RBAC rules are now provided, simplifying the secure deployment and operation of DRA drivers and related components.
• Seamless Driver Upgrade Mode: This innovative feature enables rolling updates for DRA drivers. Unlike traditional Kubelet plugins that require downtime, DRA drivers can now have two instances running concurrently, allowing for zero-downtime upgrades. The Kubelet intelligently communicates with either instance, ensuring continuous service.

Multi-Node Resource Management with NVIDIA

NVIDIA has been a significant driver and beneficiary of DRA's capabilities. A standout new use case is the support for cross-node resources, specifically for multi-node NVLink systems like the GB200 NVL72. These advanced systems feature multiple compute trays and NV switches, creating high-bandwidth interconnections between GPUs across different physical nodes.

To manage this, NVIDIA introduced a new API server object called a compute domain. Instantiating a compute domain behind the scenes creates the necessary DRA abstractions to allocate and utilize these multi-node NVLinks. This allows users to launch workloads that span multiple nodes, with GPUs communicating at speeds traditionally only achievable within a single node, leveraging the high-bandwidth NVLink fabric rather than slower Infiniband or Ethernet. This represents a significant leap forward for distributed AI/ML training and other high-performance computing applications within Kubernetes. The NVIDIA GPU operator and its DRA driver are being updated to support these complex topologies, using the standard NVML library to enumerate device capabilities.

Demo / Proof of Concept

▶ Watch: New Alpha feature: Partitionable devices (6:20)

Kevin Klues provided a compelling demonstration of DRA's advanced capabilities, specifically showcasing the multi-node NVLink feature developed by NVIDIA. This demo highlighted how DRA enables high-bandwidth GPU communication across different Kubernetes nodes, a significant advancement for distributed computing workloads.

The demonstration environment was a "mini GB200 cluster," a scaled-down version of the full GB200 NVL72 system. While a full GB200 cluster can comprise 18 compute nodes, the demo utilized a smaller setup:

• Hardware: Four Kubernetes nodes, each equipped with four NVIDIA GPUs. This totaled 16 GPUs across the cluster.
• Connectivity: Crucially, these 16 GPUs had full NVLink connectivity between them, meaning all GPUs across all four nodes could communicate directly via the high-speed NVLink fabric, not just within a single node.

The workload executed during the demo was designed to stress and measure this cross-node GPU communication:

• Workload Type: Four worker MPI jobs. MPI (Message Passing Interface) is a standard for parallel computing, often used in high-performance computing to manage communication between processes.
• Functionality: These MPI jobs were configured to measure the throughput of memory reads and writes between every GPU in the entire 16-GPU mesh. This comprehensive test ensures that all possible inter-GPU communication paths are exercised and measured.

The output of the demo visually confirmed the success of the multi-node NVLink integration:

• Results: The demonstration showed that all GPUs, regardless of whether they were on the same node or different nodes, communicated at approximately the same high speed. This is a stark contrast to traditional Kubernetes deployments where cross-node GPU communication would be significantly slower due to reliance on network fabrics like Infiniband or Ethernet. The uniform high bandwidth achieved across the entire mesh validated DRA's ability to abstract and manage complex, high-performance, multi-node hardware topologies.

This proof of concept effectively illustrated how DRA, combined with NVIDIA's compute domain API, allows Kubernetes to orchestrate distributed GPU workloads that leverage cutting-edge, high-bandwidth interconnects, making it a powerful platform for next-generation AI and HPC applications.

Defensive Implications

▶ Watch: Prioritized alternatives for device requests (7:40)

Dynamic Resource Allocation (DRA) introduces several critical capabilities that significantly enhance the defensive posture and operational efficiency of Kubernetes clusters managing specialized hardware. These implications span resource isolation, reliability, security, and simplified management.

1. Enhanced Resource Isolation and Control:

• Granular Device Description: By moving to Resource Slices that provide detailed attributes of each device, administrators gain unprecedented visibility into their hardware inventory. This allows for more intelligent placement decisions and prevents workloads from inadvertently consuming resources that are suboptimal or reserved for specific purposes.
• Attribute-Based Scheduling: The Scheduler's ability to match Resource Claims based on detailed attributes (e.g., vendor, memory, compute capabilities) ensures that sensitive or critical workloads can be precisely placed on certified or high-performance hardware, while less demanding tasks can use other devices. This reduces the risk of resource contention or performance degradation on critical infrastructure.
• Partitionable Devices: For devices that can be logically divided, such as powerful GPUs, DRA's partitionable devices feature allows for stricter isolation. Different partitions can be allocated to different workloads or tenants, preventing one workload from monopolizing or interfering with another's performance on the same physical device.

1. Improved Reliability and Maintenance:

• Device Taints and Tolerations: This feature is a game-changer for maintaining device health and managing hardware lifecycle. Administrators can proactively "taint" a device that is showing signs of degradation, undergoing maintenance, or being taken offline. This prevents new pods from being scheduled on the affected device and can trigger eviction of existing pods, ensuring workloads only run on healthy, available hardware. This reduces silent failures and improves overall cluster stability.
• Seamless Driver Upgrade Mode: A common point of failure or downtime in hardware-accelerated clusters is driver upgrades. DRA's seamless upgrade mode for drivers allows for zero-downtime updates. By enabling two driver instances to run in parallel during a rolling update, the Kubelet can seamlessly switch between them, ensuring continuous availability of device services. This is crucial for maintaining high uptime in production environments.

1. Strengthened Security Posture:

• Standardized Admin Access Controls: The enhanced admin access feature, requiring a standardized namespace label for privileged Resource Claims, provides a built-in, secure mechanism for administrative overrides. This ensures that only authorized users in designated namespaces can request privileged access to devices, preventing unauthorized or accidental disruption of production workloads. This moves away from potentially less secure cluster-level validation admission policies to a more granular, Kubernetes-native control.
• Predefined RBAC Rules: The inclusion of predefined RBAC rules simplifies the secure deployment and operation of DRA drivers. Administrators can apply these roles to ensure that DRA components operate with the minimum necessary privileges, adhering to the principle of least privilege and reducing the attack surface.

1. Simplified Management and Adoption:

• Interoperability and Gradual Migration: The plan to map extended resource API requests to DRA drivers allows for a gradual migration from the older device plugin model to DRA. This means existing applications don't need to be rewritten immediately, reducing friction for adoption and allowing administrators to transition their clusters incrementally. A single DRA driver can service both old and new request types, simplifying deployment.
• Standardized Driver Development: The availability of an example DRA driver and readily available container images simplifies the development and deployment of new DRA drivers by hardware vendors. This standardization fosters a healthier ecosystem and reduces the operational burden on cluster administrators.

In summary, DRA moves Kubernetes device management from a reactive, opaque model to a proactive, transparent, and highly configurable system. This empowers defenders with better tools for ensuring reliability, security, and efficient utilization of specialized hardware resources.

Key Takeaways

• DRA is Maturing Rapidly: Dynamic Resource Allocation (DRA) has made significant strides, reaching beta in Kubernetes 1.32 and targeting General Availability (GA) in Kubernetes 1.34 for its core functionality, demonstrating its stability and commitment as a core Kubernetes feature.
• Flexible Resource Description and Consumption: DRA introduces new API objects like Resource Slice (for detailed device attributes) and Resource Claim (for flexible, attribute-based user requests with cell expressions and sharing), moving beyond the limitations of opaque device counts.
• Enhanced Scheduler Intelligence: A key architectural shift places resource matching logic directly within the Kubernetes Scheduler, enabling it to make intelligent, attribute-aware placement decisions for hardware-accelerated workloads.
• Critical New Features for Complex Workloads: Recent additions like partitionable devices, device taints and tolerations, and prioritized alternatives provide unprecedented control over device allocation, lifecycle management, and user flexibility.
• Multi-Node Hardware Orchestration: DRA now supports advanced use cases such as multi-node NVLink systems (e.g., NVIDIA GB200 NVL72) through new concepts like compute domains, enabling high-bandwidth GPU communication across Kubernetes nodes for distributed AI/ML.
• Improved Reliability and Security for Operations: Features like seamless driver upgrade mode (for zero-downtime updates) and standardized admin access controls with RBAC rules significantly enhance the operational reliability and security of hardware-accelerated Kubernetes clusters.

About the Speaker(s)

Patrick Ohly is one of the co-chairs of the Kubernetes Working Group Device Management. He was instrumental in initiating the Dynamic Resource Allocation (DRA) effort in Kubernetes, along with a few other individuals, long before it formally became a working group. His deep involvement stems from the early days of the project, focusing on driving dynamic resource allocation forward within the Kubernetes ecosystem.

Kevin Klues is also one of the co-chairs of the Kubernetes Working Group Device Management and was an early contributor to the Dynamic Resource Allocation (DRA) project. Representing NVIDIA, Kevin provides crucial insights into real-world use cases and the practical application of DRA for GPUs. He actively tracks and drives the adoption of DRA within NVIDIA's GPU operator, demonstrating how DRA addresses complex challenges like multi-node GPU communication with features such as multi-node NVLink and compute domains.