An Introduction to Building Humanoid Robots | NVIDIA GTC 2025

An Introduction to Building Humanoid Robots | NVIDIA GTC 2025

Speakers: Jim Fan (NVIDIA Gear Research Lab), Yuka (NVIDIA Gear Research Lab), Yen (NVIDIA Engineering), Leela (NVIDIA Engineering)

Conference: NVIDIA GTC

YouTube: https://www.youtube.com/watch?v=Oyon1QDpU6g

Overview

This talk, delivered by a team of NVIDIA researchers and engineers including Jim Fan, Yuka, Yen, and Leela, provides a comprehensive introduction to Project Groot and NVIDIA's overarching strategy for developing humanoid robots. It delves into the motivations, foundational research principles, and the three-computer architecture—OVX, DGX, and AGX—that underpins this ambitious initiative. The core of the presentation centers on Groot N1, hailed as the world's first open humanoid foundation model, designed to be a generalist brain for a diverse range of physical robotic embodiments.

The significance of this work is multi-faceted. Economically, it addresses a growing global problem of job vacancies in physical labor sectors, which current AI models like large language models (LLMs) cannot solve. By bridging this gap, humanoid robots could unlock trillions of dollars in economic value. Technologically, NVIDIA positions robotics as the "last standing challenge of AI," drawing parallels to the transformative impact of foundation models in natural language processing and computer vision. The talk outlines a strategic roadmap that combines advanced simulation, massive data generation, and cutting-edge hardware to accelerate the development and deployment of intelligent, adaptable, and safe humanoid robots, ultimately aiming to bring physical AI into the real world.

Background

▶ Watch: Introduction to humanoid robots and Project Groot (0:00)

The motivation for Project Groot stems from three compelling statistics. First, job vacancies in the 30 most advanced economies have increased 4.2x over the last decade, with sectors like healthcare, construction, manufacturing, and hospitality facing critical labor shortages that require physical presence and manipulation, not just cognitive work. Second, this deficit translates to an estimated $1.4 trillion annual economic gap, equivalent to 1.5% of the entire US GDP, representing a significant lost opportunity. Third, the cost of humanoid robots has drastically decreased from $1.5 million for NASA's Robonaut in 2001 (over $2 million today with inflation) to approximately $40,000, comparable to a car, with expectations for further reductions as production scales. This trend signals a maturing market ripe for disruption.

Historically, robotics research has often focused on specialized systems. Early approaches involved highly modular pipelines for perception, planning, and control, pieced together after individual sub-problems were solved. While this offered modularity, it created information bottlenecks preventing full-stack optimization. Deep learning introduced monolithic "pixel-to-torque" models, but these often sacrificed modularity and compositionality. NVIDIA's approach seeks a middle ground, integrating compositional structures within a framework that allows for end-to-end optimization.

A key challenge in robotics, distinct from fields like natural language processing, is the "data hunger" problem. Unlike LLMs that can leverage the vast "internet of human text," there is no equivalent "internet of human control" for robot actions. This necessitates creative data sourcing strategies. NVIDIA champions the development of generalist foundation models for robotics, akin to the paradigm shift seen with models like GPT, Llama, and Claude in NLP. These generalist models, once trained on massive, open-ended data, can then be specialized for various downstream tasks, offering significant advantages over building specialist systems from scratch.

The choice of humanoid form factor is central to Project Groot. Humanoid robots offer a general-purpose, versatile body capable of performing tasks in environments built for humans (a "brownfield project" argument), reusing existing tools and infrastructure. Furthermore, the physical similarity to humans is a crucial data argument: it allows tapping into the immense wealth of human-centric data available online, such as motion capture databases, YouTube videos, and other internet content depicting human actions, which can be leveraged to train robots. This strategy is also inspired by discoveries in human cognition, specifically the "thinking fast and slow" concept, distinguishing between a fast, intuitive "System 1" (unconscious, reactive, closed-loop control) and a slow, deliberate "System 2" (conscious, reasoning, planning). The goal is to integrate these two thinking systems into a coherent learning framework for robots.

Key Findings

▶ Watch: The physical AI life cycle for robotics development (4:40)

Project Groot is not merely a single model or product, but a comprehensive strategy for NVIDIA's future in physical AI computing platforms. It’s defined by three core components: the OVX computer for data generation (simulation and teleoperation), the DGX computer for training large foundation models, and the AGX computer for deploying these models on edge devices.

The central contribution is Groot N1, introduced as the world's first open humanoid foundation model. This 2-billion-parameter model punches above its weight, designed to be small enough for efficient deployment on edge computing devices. A significant architectural innovation of Groot N1 is its dual-system approach, combining the strengths of discrete-token-based Large Language Models (LLMs) for high-level reasoning (System 2) and continuous-value-generating Diffusion Models for real-time motor action control (System 1).

A critical aspect of Groot N1's design is its cross-embodiment capability. It was trained on diverse robot configurations, ranging from single robot arms with grippers to two arms with hands, demonstrating its adaptability across different physical forms. This is achieved through embodiment-specific adapters that share core learned representations.

To address the data scarcity in robotics, NVIDIA has developed a Data Pyramid strategy. This involves pragmatically leveraging real-world data (high quality, low volume), web data (unstructured, human-centric), and crucially, synthetic data generated from advanced simulators like NVIDIA Omniverse and Isaac Lab. Key innovations in synthetic data generation include:

• Generative AI tools: For automating the creation of 3D assets, reward functions, task goals, and environment randomization (textures, lighting).
• Digital Cousin: A concept where simulation prioritizes diversity and quantity over exact fidelity to a "digital twin," enabling orders of magnitude more synthetic data generation.
• Dex Mimigen: A trajectory multiplier that can expand a small set of human demonstrations (e.g., 5 demos) into thousands of diverse synthetic trajectories (e.g., 3,000), yielding significant performance improvements (65% policy performance increase).
• Latent action codebooks: Techniques to recover action information from action-less video data (human videos, neuro-generated videos like those from Sora-like models), treating them as additional "embodiments" for training.

The talk also highlighted NVIDIA Thor, a new edge computing platform specifically designed for humanoid robotics. Thor delivers 2 Petaflops of FP4 performance, a 10x improvement in I/O with 425 GbE, and incorporates advanced safety (up to ASIL D ready functional safety island) and security features (model-level encryption, firmware-based TPMs). This platform is crucial for running large transformer models like Groot N1 directly on the robot.

Finally, the presentation showcased impressive results, including zero-shot sim-to-real transfer for mobility policies trained entirely on synthetic data, and a 10,000x acceleration in physics simulation within Isaac Lab, allowing a year's worth of training in less than an hour on a single GPU.

Technical Deep Dive

▶ Watch: NVIDIA's three computer solution for Project Groot (5:20)

The technical core of Project Groot revolves around the Groot N1 foundation model and its supporting ecosystem. Groot N1 is designed as a neuro-network that operates directly on hardware, interacting with the physical world. Its high-level function involves taking visual tokens (egocentric view from the robot's cameras) and language instructions as input, and generating motor action tokens as output. These motor actions are continuous floating-point values that control the robot's actuators.

The architecture of Groot N1 ingeniously combines two powerful AI paradigms to implement the dual-system cognition model:

1. System 2 (Reasoning): This slow, deliberate system is handled by a visual language model (VLM). It uses discrete-style tokens and an auto-regressive transformer model for vision-language reasoning, allowing the robot to understand the scene and instructions. The Ego VLM, part of NVIDIA's broader VLM family, is integrated here.
2. System 1 (Reactive Control): This fast, intuitive system is implemented by a diffusion transformer. It takes encoded robot states and noisy action tokens, then performs a series of diffusion steps to generate the actual continuous motor action outputs. This diffusion transformer runs at a high frequency of 120 Hz, essential for smooth, reactive robot motions.

For cross-embodiment support, Groot N1 uses embodiment-specific adapters. These adapters process robot states and actions into tokens, which then feed into a shared set of intermediate layers. These shared layers capture general learned representations across various robot types (e.g., single arm, dual arm, different humanoid models), while the adapters handle the specific kinematic and dynamic properties of each embodiment.

The Data Pyramid strategy is crucial for training Groot N1. It categorizes data into three tiers:

• Real-world data: Collected via teleoperation (e.g., using Apple Vision Pro to track human hand movements and map them to robot hands). This data is high-quality but limited in volume and expensive to acquire.
• Web data: Unstructured, human-centered multimodal data from the internet (YouTube videos, motion capture databases). While abundant, it lacks direct robot action labels.
• Synthetic data: Generated using NVIDIA Omniverse and Isaac Lab. The key is to overcome the traditional "human-in-the-loop" bottleneck and the sim-to-real gap. This is achieved through generative AI:
• Generative Simulations: Large language models generate reward functions and task goals. Image generation models randomize textures and lighting. 3D asset generation from language prompts creates diverse environments. This enables the creation of "digital cousins" – simulations that are diverse and plentiful, even if not perfect digital twins.
• Dex Mimigen: This "trajectory multiplier" generates thousands of new trajectories from a handful of human demonstrations. It leverages the principle of SE3 equivariance, preserving the relative pose between the robot's end-effector and the object under spatial transformations. It can also manage complex multi-hand coordination tasks by identifying different operational modes and state machine logic.
• Data Mixing: A sophisticated strategy combines the unbalanced real-world and synthetic datasets, using smart sampling and rebalancing techniques to maximize the benefits of both.

Beyond traditional physics engines, NVIDIA is exploring neural simulation with video generation models (like Sora). The challenge here is to recover action information from action-less videos. This is done by learning latent action codebooks that align across different embodiments, allowing these neuro-generated videos to be annotated with actions and used as training data.

The NVIDIA Isaac Lab platform, built on Omniverse, provides a robust environment for robot learning, supporting imitation learning and reinforcement learning. It includes APIs, assets, environments, sensors, and utilities for synthetic data generation. NVIDIA Cosmos, a world model, converts these simulated environments into photorealistic videos, helping to close the sim-to-real gap.

Specific workflows include:

• Isaac Groot synthetic motion generation: Combining human teleoperation (or digital twin teleoperation) with Dex Mimigen and Cosmos for large-scale, photorealistic data.
• Mobility Gen/Control: The REMEMBER workflow (VLM + LLM + vector databases) provides System 2 long-term memory and high-level goal generation. This feeds into System 1 mobility policies (end-to-end pixel-in, action-out). The Mobility Gen pipeline uses Isaac Lab and Cosmos to augment data, allowing users to import USD scenes and robot assets and customize sensor configurations and control methods.
• Campus (Cross-Embodiment Mobility Policy): This workflow starts with an imitation learning-based generalist policy (Isaac Mobility), then uses residual reinforcement learning (RL) in Isaac Lab to train specialist "experts" for different embodiments (e.g., quadrupeds, humanoids). Finally, cross-embodiment distillation combines these specialists into a more powerful generalist policy, achieving zero-shot sim-to-real transfer for mobility.
• Neural Whole Body Controller (Hover): For humanoids, this high-frequency controller (100s-1000s Hz) is trained using large-scale human motion data (retargeted for specific robot kinematics). It involves an RL policy trained with privileged information, followed by student distillation via imitation learning, incorporating random masking for different control modes (e.g., Apple Vision Pro, autonomous policies, root velocity commands). This process is massively accelerated by Isaac Lab's physics engine.

The deployment platform, NVIDIA Thor, is an edge AI supercomputer featuring a Blackwell-based GPU architecture with a new transformer engine natively supporting FP4 and FP8 precisions, crucial for large transformer models. It also supports multi-instance GPU (MIG) for mixed-criticality workloads. The module includes a 14-core Poseidon AE ARM CPU (2.6x performance increase), 128 GB of LPDDR5X memory (double capacity, 273 GB/s bandwidth), and various accelerators (PVA, OFA, ISP, video encode/decode). Its Hollow Scan Sensor Bridge simplifies sensor integration via Ethernet, providing ultra-low latency data transfer directly to GPU memory, with support for functional safety up to ASIL D.

Experimental Setup & Results

▶ Watch: Research principles guiding Groot's development: generalist, dual system (6:30)

The effectiveness of Project Groot's approach is demonstrated through several key experimental findings:

• Groot N1 Model Size and Efficiency: The foundation model is relatively compact with 2 billion parameters, a conscious design choice to ensure it can fit and run efficiently on edge computing devices like NVIDIA Thor, overcoming the computational constraints typical for physical AI.
• Synthetic Data Augmentation: In the Groot N1 paper, training leveraged 100 times more synthetic data than real-world data. This massive synthetic data volume, generated through tools like Dex Mimigen, proved critical. For instance, using 3,000 automatically generated trajectories (derived from just 5 human demonstrations) resulted in a 65% improvement in policy performance compared to training solely on the limited human demonstrations.
• Task Success Rates: Groot N1 was shown to outperform prior state-of-the-art methods by over 30% on success rate for certain tasks involving grasping and pick-and-place operations. In specific scenarios, the model achieved very high success rates, often exceeding 80% or 90%.
• Cross-Embodiment Demonstration: Groot N1 was fine-tuned and demonstrated on various robot platforms, including industrial robot arms, the 1x Neo Gamma humanoid (a collaboration with the Norwegian company 1x), and even the $110 open-source Hugging Face Le Robot. This showcases its versatility across different hardware.
• Sim-to-Real Transfer: The Campus workflow, which trains mobility policies entirely on synthetic data, achieved zero-shot sim-to-real transfer, meaning the models could directly operate in the real world without any real-world fine-tuning. This is a significant breakthrough for reducing development costs and time.
• Simulation Acceleration: The NVIDIA Isaac Lab environment provides 10,000x acceleration for physics simulations. This translates to being able to run a year's worth of intense robot training in less than an hour of clock time on a single GPU, dramatically speeding up the iterative development and refinement of robot policies.
• Deployment Hardware: The NVIDIA Thor platform provides 2 Petaflops of FP4 performance, making it capable of running large transformer models at the edge. Its enhanced I/O (425 GbE, 10x improvement) and powerful CPU (14-core Poseidon AE ARM, 2.6x faster) are designed to meet the real-time control and sensor processing demands of complex humanoids.

These results collectively validate NVIDIA's strategy of combining advanced foundation models, sophisticated data generation techniques, and powerful computing platforms to tackle the grand challenges of robotics.

Practical Implications

▶ Watch: The 'Data Pyramid' principle for robot training data (8:10)

Project Groot and NVIDIA's comprehensive ecosystem have profound practical implications for various stakeholders in the AI and robotics fields:

For Practitioners and Model Builders:

• Accelerated Development: The open-source Groot N1 foundation model, coupled with provided training data, evaluation protocols, and benchmarks, significantly lowers the barrier to entry for developing humanoid robot applications. Developers can quickly fine-tune and adapt the model for specific tasks and environments.
• Generalist Capabilities: The emphasis on generalist models means practitioners can develop robots with core competencies that can be adapted to a wide variety of tasks, rather than building bespoke solutions for each. This promises greater flexibility and broader applicability in diverse settings like homes, hospitals, and factories.
• Simulation-First Paradigm: The robust Isaac Lab, Omniverse, and Cosmos platforms enable a simulation-first approach, which is critical for cost-effective and safe development. Generating vast quantities of high-quality synthetic data, augmented by generative AI and tools like Dex Mimigen, mitigates the high cost and logistical challenges of real-world data collection. This also allows for rapid iteration and testing in diverse, controlled virtual environments before deployment.
• Zero-Shot Sim-to-Real: The demonstrated ability for zero-shot sim-to-real transfer for mobility models is a game-changer, potentially eliminating the need for extensive real-world fine-tuning, thus saving significant time and resources.

For Infrastructure and Deployment Teams:

• Full-Stack Solution: NVIDIA provides a cohesive three-computer problem solution (OVX for generation, DGX for training, AGX/Thor for deployment), offering a vertically integrated stack that simplifies the development and deployment pipeline for physical AI.
• Edge Computing Power: The NVIDIA Thor platform is specifically designed for the demanding computational needs of humanoids at the edge. Its 2 Petaflops of FP4 performance, high-speed I/O (425 GbE), and specialized accelerators enable complex transformer models like Groot N1 to run autonomously in real-time on the robot itself, crucial for reactive and intelligent behavior.
• Simplified Sensor Integration: The Hollow Scan Sensor Bridge streamlines the integration of diverse sensor types (cameras, IMUs, motors) via Ethernet, reducing hardware and software development complexity and offering low-latency data transfer, scalability, and built-in safety features.
• Safety and Security: Thor's integrated functional safety island (up to ASIL D ready) and security features (model-level encryption, secure boot, TPMs) are vital for deploying robots in safety-critical human environments and protecting proprietary IP.

Tradeoffs and Limitations:

• No "Final Answer" Architecture: The speakers acknowledge that the optimal architecture for embodied AI is still evolving. While the dual-system approach of Groot N1 is promising, continuous innovation and adaptation to new breakthroughs (e.g., in computer vision, reasoning, or novel architectures beyond transformers) will be necessary.
• Sim-to-Real Gap Challenges: While significant progress has been made, the sim-to-real gap remains an open problem, particularly for complex physical phenomena like liquids or smoke.
• Safety for High-Speed Operation: Current humanoid robots are often demonstrated at slower speeds due to safety concerns (a 100lb robot falling could be dangerous). Achieving true human-like speed and agility in shared human environments will require further advancements in compliant robotics and robust safety mechanisms beyond software.
• Scaling Laws for Robotics: Unlike LLMs, the precise scaling laws for robotics (how performance scales with compute, data, and parameters) are still being systematically explored. This uncertainty can impact long-term investment and development planning.
• Grounded Reasoning: While LLMs excel at reasoning with text tokens, robotic reasoning needs to be deeply integrated and grounded in the sensorimotor space, a complex research challenge that requires connecting latent reasoning with real-time physical interaction.

Overall, NVIDIA's Project Groot offers a compelling vision and a robust set of tools and platforms to drive the next generation of robotics, moving towards a future where intelligent humanoids can address pressing societal and economic needs.

Key Takeaways

• Project Groot: NVIDIA's Physical AI Strategy: Groot is a comprehensive strategy encompassing the OVX (simulation), DGX (training), and AGX/Thor (deployment) computing platforms, aiming to solve the "last standing challenge of AI" by building generalist humanoid robots.
• Groot N1: Open Humanoid Foundation Model: The world's first open humanoid foundation model, Groot N1, is a 2-billion-parameter cross-embodiment model that integrates a VLM for high-level reasoning (System 2) and a diffusion transformer for real-time continuous motor action generation (System 1 at 120 Hz).
• Data Pyramid for Robotic Learning: NVIDIA leverages a "Data Pyramid" strategy combining real-world data, web data, and massive synthetic data generated using advanced tools like Dex Mimigen (trajectory multiplier) and generative AI, achieving significant performance gains (e.g., 65% policy improvement from synthetic augmentation).
• Isaac Lab & Omniverse Ecosystem: The NVIDIA Isaac Lab (built on Omniverse) provides a 10,000x accelerated physics simulation environment for robot learning, enabling rapid iteration and zero-shot sim-to-real transfer for mobility policies, with Cosmos providing photorealistic rendering.
• NVIDIA Thor: Edge AI Supercomputer for Humanoids: The new Thor platform delivers 2 Petaflops of FP4 performance with a Blackwell-based GPU, 425 GbE I/O, and integrated safety/security features, designed for deploying large transformer models and real-time control on edge robotics.
• Vision for Open-Ended Instruction Following: The long-term goal is to develop highly capable, general-purpose humanoid robots that can understand and execute open-ended language instructions in diverse real-world environments, addressing the Moravec paradox and bridging critical labor gaps.

About the Speaker(s)

The talk featured a diverse team from NVIDIA, bringing together both cutting-edge research and robust engineering expertise.

• Jim Fan is from NVIDIA's Gear Research Lab, leading the discussion on the foundational motivations and the technical architecture of Groot N1.
• Yuka is also from NVIDIA's Gear Research Lab, detailing the research principles behind Groot, including the generalist model approach, dual-system architecture, and the innovative data pyramid strategy.
• Yen is a leader on NVIDIA's engineering team, sharing insights into the engineering side of the Groot initiative, including the simulation and robot learning frameworks within Isaac Lab, Omniverse, and Cosmos, and various workflows for mobility and manipulation.
• Leela is another leader on NVIDIA's engineering team, who closed the session by introducing NVIDIA Thor, the edge computing platform designed to deploy these advanced AI models on real-world robots, highlighting its architecture, performance, safety, and security features.