National Labs Use of XR

National Labs Use of XR

Speakers: Martin Pratt, Software Engineer, Pacific Northwest National Lab

Conference: DEF CON 32

YouTube: https://www.youtube.com/watch?v=Qua92jLf2fE

Overview

In this DEF CON 32 talk, Martin Pratt, a software engineer at the Pacific Northwest National Lab (PNNL), provided a comprehensive overview of how Extended Reality (XR) technologies – encompassing Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) – are being leveraged across various critical initiatives at the lab. The presentation delved into PNNL's eight-year journey with immersive computing, highlighting its evolution from early consumer-grade headsets to advanced professional systems. Pratt emphasized the versatility of XR in addressing complex challenges faced by national labs, from nuclear remediation and renewable energy research to critical infrastructure cybersecurity.

Pratt's talk underscored the strategic importance of XR as a tool for scientific visualization, training, and public communication, demonstrating its capacity to provide novel insights and enhance situational awareness. A significant portion of the presentation was dedicated to the Celer XR project, a collaborative effort with the Cybersecurity and Infrastructure Security Agency (CISA), a component of the Department of Homeland Security (DHS). This project specifically utilizes XR to create realistic testbeds for cybersecurity threat analysis and red/blue teaming exercises targeting Industrial Control Systems (ICS), which are vital for national security and critical infrastructure. The work presented by Pratt is crucial for understanding how cutting-edge immersive technologies are being applied to safeguard essential systems against modern cyber threats.

Background

▶ Watch: Introduction to PNNL's immersive computing work (0:00)

The Pacific Northwest National Lab (PNNL) has been at the forefront of immersive computing applications for approximately eight years, beginning around the "big push" into VR in 2015-2016. As a National Lab, PNNL's overarching mission is to enhance efficiency in the U.S. economy for the government and the public. Its work spans a wide array of scientific and engineering disciplines, with a strong historical connection to the Hanford site in Eastern Washington, which involves extensive nuclear remediation and nuclear energy research. Beyond this, PNNL is deeply involved in renewable energy initiatives, including geothermal, wind, and tidal power, with a dedicated marine sciences campus in Western Washington focusing on marine energy and its environmental impacts.

The diverse nature of PNNL's research and development naturally lends itself to the application of XR technologies. The lab's immersive computing team functions as a critical enabler, working directly with sponsors to design and envision solutions. This process is highly interactive, starting with early-stage engagement to ensure that the chosen immersive platform is the most appropriate tool for the specific need. For instance, creating entirely virtual environments often favors VR, while integrating digital twins or overlaying data onto real-world settings is better suited for AR. A core capability of PNNL's XR team is extensive 3D asset development, as many of the complex systems and environments they seek to visualize or simulate are highly specialized and do not exist as readily available digital models. These custom 3D assets are then meticulously implemented into virtual, augmented, or mixed reality environments, enabling exploration, analysis, and communication. This foundational work in content creation and platform adaptation underpins all of PNNL's XR applications, providing the necessary infrastructure for advanced scientific and security endeavors.

Key Findings

▶ Watch: PNNL's XR development process: design, assets, implementation (2:20)

PNNL's extensive experience with XR technologies has yielded several key findings regarding their utility across diverse applications. The lab has successfully deployed immersive solutions in four primary areas:

1. Training Platforms: XR environments offer an unparalleled capability to simulate hazardous, complex, or otherwise inaccessible scenarios. This allows individuals to repeatedly practice procedures and build critical skills in a safe, controlled digital space before encountering real-life situations. This is particularly valuable for high-stakes environments such as nuclear facilities or critical infrastructure operations.

1. Scientific Visualizations: For data that is inherently three-dimensional, VR and AR systems provide a powerful medium for visualization. Scientists can immerse themselves within their data, enabling a deeper understanding and revealing new insights that might be missed with traditional two-dimensional representations. This has proven particularly effective in fields ranging from material science to environmental modeling.

1. Situational Awareness: XR enhances situational awareness by allowing users to visualize complex data in context. Whether it's overlaying real-time sensor data onto physical equipment (AR) or creating comprehensive virtual command centers (VR), these systems provide an intuitive and immersive way to monitor and react to dynamic situations, improving decision-making in critical scenarios.

1. Communication and Outreach: Beyond internal scientific and operational uses, XR serves as a highly engaging tool for communicating complex projects and research to a broader audience. This includes sponsors, academic partners, industry stakeholders, and the general public. By transporting viewers into unique or difficult-to-access environments, XR fosters a novel and impactful way to convey information and garner support for PNNL's initiatives.

A standout application demonstrating these findings is the Celer XR project. This initiative, undertaken in partnership with the Cyber Security and Infrastructure Security Agency (CISA), directly applies XR to the pressing challenge of critical infrastructure cybersecurity. The project focuses on creating realistic test platforms for Industrial Control Systems (ICS). By replicating real-world hardware and software configurations within a safe, observable environment, Celer XR facilitates advanced threat analysis, red/blue teaming, and the development of robust defensive strategies against cyberattacks on vital infrastructure. The ability to interact with these simulated or augmented systems provides an invaluable resource for enhancing national security.

Technical Deep Dive

▶ Watch: Key XR applications: training, scientific visualization, communication (3:50)

The technical foundation of PNNL's immersive computing efforts is built on a strategy of continuous adaptation to evolving hardware and a robust methodology for content creation and platform deployment. The lab's journey began with early mainstream VR and AR devices, including the HoloLens 1, Oculus Rift, and the original HTC Vive. This early adoption allowed PNNL to develop foundational expertise in building immersive experiences.

As the technology matured, PNNL transitioned to more advanced, higher-fidelity systems. A notable current piece of equipment is the Vario XR3, a high-end mixed reality headset known for its exceptional visual fidelity and integrated passthrough capabilities. The speaker mentioned that the even newer Vario XR4 was released shortly after PNNL acquired their XR3 units, indicating the lab's commitment to utilizing state-of-the-art technology to meet demanding sponsor requirements for rendering capability and realism. The use of advanced VR passthrough is particularly emphasized, suggesting a strong focus on mixed reality applications where digital content seamlessly integrates with the physical environment.

Central to PNNL's technical approach is its comprehensive 3D asset development capability. Because many of the systems and environments PNNL works with are highly specialized—ranging from intricate nuclear facility components to complex marine energy devices—off-the-shelf 3D models are often insufficient or nonexistent. The team therefore dedicates significant resources to creating high-fidelity, custom 3D models. This meticulous content creation ensures that the virtual and augmented representations are accurate and detailed enough for scientific analysis, precise training, and realistic threat simulation.

The choice of immersive platform is carefully considered based on the specific project requirements. For scenarios demanding full immersion in a fabricated environment, Virtual Reality is typically employed. This is ideal for training exercises in hazardous zones or for visualizing purely theoretical constructs. Conversely, for applications requiring the integration of digital information with real-world objects, Augmented Reality or Mixed Reality is preferred. This approach is particularly effective for creating digital twins of physical systems, where virtual overlays provide real-time data, diagnostic information, or interactive controls directly on the actual equipment. The lab's ability to be "adaptable" and "change with what those different sponsors may need" highlights a platform-agnostic approach, prioritizing the most effective immersive tool for each unique challenge.

The Celer XR project exemplifies PNNL's technical prowess in applying these capabilities to cybersecurity. Developed in partnership with CISA, Celer XR aims to create realistic test platforms for Industrial Control Systems (ICS). The core technical innovation here lies in building environments that utilize "the same hardware, they use the same software as real systems that exist out in the real world." This commitment to authenticity is paramount for effective cybersecurity testing. The project involves not only replicating the software interfaces but also creating physical, scaled-down models of these ICS, referred to as "skids." These tabletop-sized skids are "completely three-dimensional physical models" that accurately represent the functionality and architecture of full-scale industrial systems. By combining these physical models with XR overlays, Celer XR provides an environment where cybersecurity researchers and red teams can interact with highly realistic representations of critical infrastructure, testing vulnerabilities and defensive measures in a controlled and safe manner.

Demo / Proof of Concept

▶ Watch: Introduction to the Celer XR project with CISA (6:00)

The talk specifically highlighted a live demonstration of the Celer XR project at the DEF CON XR Village. This demonstration served as a tangible proof of concept for PNNL's work in applying immersive technologies to critical infrastructure cybersecurity. Attendees had the opportunity to experience firsthand the capabilities of the system.

The core of the demo involved interaction with the "skids" – the physical, scaled-down models of Industrial Control Systems (ICS) that represent real-world industrial hardware and software. These skids, designed to be tabletop size, offer a high-fidelity, three-dimensional physical representation of critical infrastructure components.

Using a high-end Vario XR3 headset, participants were able to engage in a mixed reality experience. While the speaker did not detail the exact nature of the interaction, the context of Celer XR being for "finding threats, to do red blue teaming" strongly implies that the demo showcased how virtual overlays and interactive elements within the XR environment could be used to:

• Visualize system states: Displaying real-time operational data, network traffic, or sensor readings directly onto the physical skid.
• Simulate attacks: Allowing users to initiate virtual cyberattacks against the physical/virtual hybrid system.
• Observe defensive reactions: Demonstrating how defensive mechanisms respond to simulated threats, perhaps by visualizing data flow changes or system alerts.
• Conduct training: Providing an immersive platform for operators and cybersecurity professionals to practice incident response and threat mitigation in a realistic, yet safe, setting.

The use of the Vario XR3, with its advanced passthrough capabilities, would have allowed participants to see the physical "skid" in their real environment while simultaneously experiencing augmented digital information and interactive elements overlaid onto it. This blend of physical and digital interaction is crucial for the realism and effectiveness of the Celer XR platform for cybersecurity testing and training. The demo served to illustrate the practical application of PNNL's technical development in creating realistic, interactive testbeds for safeguarding vital industrial systems.

Defensive Implications

▶ Watch: Celer XR: realistic test platforms and physical models (6:50)

The work presented by Martin Pratt, particularly the Celer XR project, carries significant defensive implications for safeguarding critical infrastructure and national security. The ability to create realistic test platforms for Industrial Control Systems (ICS) using XR technologies directly addresses several key challenges faced by cybersecurity defenders:

1. Enhanced Training for Critical Infrastructure Operators: Traditional training for ICS often involves classroom learning or limited simulations. XR-based platforms, like those developed by PNNL, allow operators to undergo immersive, hands-on training in simulated hazardous environments or complex operational scenarios without risk. This repeated exposure to realistic situations helps build muscle memory and critical decision-making skills, which are vital during actual cyber incidents or operational failures. The ability to visualize and interact with digital twins of real systems means training can be highly specific and relevant.

1. Realistic Red/Blue Teaming and Threat Emulation: The Celer XR project's commitment to using "the same hardware, they use the same software as real systems" is a game-changer for cybersecurity exercises. Defenders (blue teams) can test their incident response plans, detection capabilities, and mitigation strategies against attacks mounted by red teams within an environment that closely mimics their actual operational technology (OT) networks. This realism helps identify blind spots, validate security controls, and refine defensive playbooks in ways that purely virtual simulations or abstract tabletop exercises cannot. The physical "skids" further bridge the gap between virtual and physical interaction, allowing for testing of attacks that might have physical consequences.

1. Vulnerability Research and Exploit Development (Controlled Environment): Researchers can use Celer XR's realistic testbeds to safely conduct vulnerability research and develop exploits without endangering live critical infrastructure. Understanding how specific vulnerabilities manifest in real-world ICS hardware and software is essential for proactive defense. This controlled environment allows for comprehensive analysis of attack vectors, impact assessment, and the development of tailored countermeasures.

1. Situational Awareness for Incident Response: XR can provide advanced situational awareness tools during a live incident. By overlaying real-time network telemetry, sensor data, and threat intelligence onto virtual or augmented representations of the affected ICS, incident responders can gain a clearer, more intuitive understanding of the attack's progression and its potential physical impacts. This visual context can accelerate decision-making and improve coordination during high-stress events.

1. Secure Design and Validation: Before new ICS components or systems are deployed, they can be integrated into XR testbeds for security validation. This allows for security-by-design principles to be rigorously tested in a dynamic, interactive environment, helping to identify and remediate architectural or configuration weaknesses early in the lifecycle.

In essence, PNNL's work with Celer XR empowers defenders by providing a high-fidelity, safe, and dynamic platform to understand, simulate, and mitigate cyber threats against the critical infrastructure that underpins modern society. It moves beyond abstract models to provide concrete, interactive environments for robust cybersecurity defense.

Key Takeaways

• XR as a Strategic Tool for National Labs: PNNL leverages Virtual, Augmented, and Mixed Reality across diverse domains, including nuclear energy, renewable energy, and critical infrastructure, demonstrating its versatility beyond entertainment.
• Celer XR Project for ICS Cybersecurity: In partnership with CISA, PNNL developed Celer XR to create realistic, hardware- and software-accurate test platforms for Industrial Control Systems (ICS) cybersecurity threat analysis and red/blue teaming.
• Importance of 3D Asset Development: Due to the specialized nature of PNNL's work, extensive custom 3D asset creation is a core capability, ensuring high-fidelity and accurate representations of complex systems.
• Advanced Hardware and Adaptive Platform Use: PNNL utilizes state-of-the-art XR hardware like the Vario XR3 and XR4, adapting platform choices (VR vs. AR) based on specific sponsor needs for optimal visualization and interaction.
• Enhanced Training and Situational Awareness: XR provides invaluable platforms for training personnel in hazardous or complex environments and for improving situational awareness through immersive data visualization.
• Realistic Threat Simulation for Critical Infrastructure: The physical "skids" combined with XR in the Celer XR project offer an unprecedented level of realism for simulating cyberattacks on ICS, enabling robust defensive strategy development and validation.

About the Speaker(s)

Martin Pratt is a Software Engineer at the Pacific Northwest National Lab (PNNL), located in Washington State. He specializes in the application of immersive technologies, including virtual and augmented realities, to address a wide range of scientific and engineering challenges. His work at PNNL involves the design, 3D asset development, and implementation of XR solutions for various projects, including critical infrastructure cybersecurity initiatives like the Celer XR project. Pratt's expertise lies in adapting cutting-edge immersive computing to practical, high-impact applications for government sponsors and the public.