Bridging Space and Medicine

Bridging Space and Medicine

Speakers: Fernando De La Peña Llaca

Conference: DEF CON 32

YouTube: https://www.youtube.com/watch?v=8BC0oHu5QGQ

Overview

This talk, presented by Fernando De La Peña Llaca at DEF CON 32, delves into the groundbreaking application of holographic teleportation, or volumetric video, within the extreme and constrained environment of space, specifically the International Space Station (ISS). The presentation highlights the journey from initial conceptualization to the successful implementation of live holographic interaction, enabling advanced telemedicine capabilities for astronauts. It addresses the unique challenges of deploying such sophisticated technology in a low-bandwidth, high-radiation, and computationally limited setting, ultimately showcasing how these hurdles were overcome to facilitate real-time, interactive medical assistance and communication across vast distances.

The significance of this work extends beyond mere technological achievement; it represents a crucial step towards ensuring the health and safety of astronauts on long-duration space missions, where immediate physical presence of medical experts is impossible. By enabling virtual presence through holograms, this project empowers medical professionals on Earth to guide complex procedures, conduct evaluations, and provide psychological support in a manner far more immersive and effective than traditional video conferencing. De La Peña Llaca's talk not only illustrates the technical ingenuity required but also paints a compelling picture of a future where geographical and environmental barriers are significantly diminished through mixed reality technologies.

Background

▶ Watch: HoloLens assisting astronaut surgery with Samantha Christopher (0:00)

The genesis of this ambitious project can be traced back to an earlier success involving the Microsoft HoloLens, a mixed reality computer headset. In an initial application, a NASA surgeon in Houston successfully assisted astronaut Samantha Christopher on the ISS with a medical procedure. Through the HoloLens, the surgeon could project holograms directly into Christopher's field of view, providing visual guidance for incisions, measurements, and displaying X-rays and other critical information. This pioneering effort, led by NASA surgeon and Air Force General Dr. George Smith, proved to be a resounding success, demonstrating the immense potential of mixed reality in remote medical contexts.

Inspired by this achievement, Dr. Smith challenged Fernando De La Peña Llaca to explore further possibilities, specifically inquiring about "holographic teleportation." At the time, Microsoft offered a studio-based solution where individuals were captured by 109 cameras, lasers, and lidar systems. This process generated a massive file that, after several weeks of processing, could be viewed as a recorded hologram of oneself through a HoloLens – a non-live, pre-rendered experience. Dr. Smith’s vision, however, was far more ambitious: he wanted this technology deployed live on the ISS. This request, made in 2018, set the stage for a monumental engineering and communication challenge, transforming a complex, offline, studio-dependent technology into a real-time, space-hardened solution. The transition from recorded, high-fidelity capture to live, low-resource streaming was the core problem that needed to be solved, laying the groundwork for the project discussed in the talk.

Key Findings

▶ Watch: Explaining holographic teleportation as volumetric video (2:00)

The central finding and primary contribution of this project is the successful adaptation and deployment of live holographic teleportation, or volumetric video, to the International Space Station. This achievement fundamentally transforms the paradigm of remote interaction, particularly for critical applications like telemedicine in space. The team demonstrated that despite severe technical and environmental limitations inherent to space operations, it is possible to create an immersive, interactive holographic presence that offers "six axes of freedom," far exceeding the capabilities of traditional video conferencing.

Key findings include:

1. Feasibility of Live Volumetric Video in Extreme Environments: The project proved that live capture and rendering of volumetric video could be achieved with significantly reduced camera counts (from 109 to just one on the ISS) and on severely underpowered, radiation-hardened computers.
2. Overcoming Bandwidth and Communication Constraints: The team successfully engineered a system to operate within an allocated 5 megabits per second (Mbps) bandwidth, a fraction of the total 20 Mbps available on the ISS, and to contend with intermittent communication due to frequent satellite handovers.
3. Enhanced Telemedicine Capabilities: The demonstration of medical procedures, such as stroke evaluations and physical therapy exercises, through live holograms showcased a new level of remote medical support. This enables medical professionals on Earth to guide astronauts with unprecedented visual and spatial fidelity, potentially improving diagnostic accuracy and procedural success rates in critical situations.
4. Resilience of Legacy Hardware: The project highlighted the surprising capability of older, radiation-resistant SSC computers (Space Station Computers) to handle complex, real-time volumetric video processing, albeit with significant optimization. This underscores the trade-offs between computational power and environmental hardening in space systems.

These findings collectively underscore a pivotal advancement in mixed reality technology, demonstrating its potential to bridge vast distances and overcome severe operational challenges, thereby opening new frontiers for human exploration and remote assistance.

Technical Deep Dive

▶ Watch: Overview of ISS deployment technical challenges (2:40)

The technical implementation of live holographic teleportation on the International Space Station presented a formidable array of challenges, requiring ingenious solutions to adapt a demanding technology to an unforgiving environment. The speaker used the analogy of 3D printing to explain volumetric video: just as 3D printing builds a solid object from thousands of material layers, volumetric video constructs a solid, interactive hologram from thousands of video layers. This technology offers six axes of freedom, allowing for a much richer, spatially aware interaction compared to flat, two-dimensional video calls.

The core technical hurdles stemmed from the stark contrast between the Earth-based development environment and the operational constraints of the ISS:

1. Camera Infrastructure: On Earth, Microsoft's dedicated studios for holographic capture leverage an array of 109 cameras complemented by laser and lidar systems to meticulously capture a subject's three-dimensional form. This extensive setup generates highly detailed, static holographic files. The ISS, however, could only accommodate a single camera for this purpose. The team had to develop algorithms and processing techniques capable of generating a convincing live volumetric video stream from this severely limited input. This likely involved sophisticated real-time depth estimation, subject segmentation, and perhaps predictive modeling based on limited data to reconstruct a full 3D representation.

1. Computational Power: Earth-side holographic processing, particularly for live streaming, typically relies on powerful gaming computers or high-performance workstations. These systems are designed for intensive graphical rendering and data processing. In contrast, the ISS is equipped with SSC computers (Space Station Computers), which are intentionally "very old computers." The rationale for using antiquated hardware is its superior resilience to the harsh space environment, specifically resistance to radiation and atomic oxygen damage. These older processors, while robust, possess significantly less computational power than modern consumer-grade systems. The team had to optimize the volumetric video encoding, compression, and rendering pipelines to run efficiently on these legacy systems, likely involving custom codecs, highly optimized software, and potentially offloading some processing to Earth where possible, or prioritizing key visual information.

1. Communication Infrastructure: Maintaining constant, high-bandwidth communication with a rapidly moving orbital platform like the ISS is inherently difficult.

• Satellite Handovers: The ISS orbits Earth every 90 minutes, necessitating frequent adjustments of its communication antenna. This antenna typically aims downwards to connect with ground stations, but its rapid movement means it must be re-aimed to a new satellite every 15 to 20 minutes. During these re-aiming periods, communication is entirely lost, creating intermittent "satellite handovers" that disrupt continuous data flow. The system had to be designed to buffer data, gracefully handle disconnections, and quickly re-establish and synchronize streams upon re-connection, minimizing perceived interruptions for the users.
• Network Segmentation and Firewalls: The ISS network environment is highly segmented and protected by multiple firewalls. This is a critical security measure to protect vital station operations from external threats and to manage internal network traffic. However, it adds complexity to establishing and maintaining real-time, low-latency data streams required for live holography. The team would have needed to navigate these network policies, ensuring necessary ports were open and data flows were prioritized without compromising the station's security posture.
• Bandwidth Limitations: Perhaps the most significant constraint was bandwidth. The ISS's total internet capacity, managed through systems referred to as Skynet (not related to the movie) and Cloudnet, is a mere 20 megabits per second (Mbps). This limited bandwidth is shared across all station operations, including docking spacecraft, scientific experiments, and general station maintenance. For the holographic teleportation project, the team was initially allocated only 5 Mbps. This extremely tight bandwidth budget necessitated aggressive data compression techniques for the volumetric video, prioritizing visual fidelity in critical areas (e.g., facial expressions, hand gestures for medical procedures) while potentially reducing detail elsewhere. The goal was to transmit "thousands of layers of video" within this minimal allocation, a testament to advanced compression algorithms.

1. Haptic Feedback: The talk also mentions the potential for users to "feel the hologram with something we call haptics." While the primary focus was on visual and interactive presence, the inclusion of haptic feedback would add another layer of immersion, allowing for tactile interaction with the holographic environment or remote user. This would introduce further technical challenges related to sensor input, data transmission, and the deployment of haptic devices in space.

In essence, the project successfully engineered a highly optimized, resilient, and adaptive volumetric video system that could function within extreme constraints of hardware, bandwidth, and communication reliability. This involved a deep understanding of data compression, real-time rendering, network protocols, and the unique operational requirements of space.

Demo / Proof of Concept

▶ Watch: Introducing telemedicine stroke evaluation demonstration (5:20)

The talk presented several compelling demonstrations and proofs of concept that validated the efficacy and potential of live holographic teleportation in both terrestrial and extraterrestrial contexts.

The initial success that spurred the project involved the Microsoft HoloLens assisting astronaut Samantha Christopher on the ISS. A NASA surgeon, Dr. George Smith, guided Christopher through a medical procedure by projecting holograms of instructions, measurements, and X-rays directly into her field of view. This demonstration, which proceeded "with no issues at all," established the foundational utility of mixed reality for remote medical support in space.

The core of the live holographic teleportation demonstration involved showing how individuals could interact in real-time across significant distances. The speaker highlighted a personal experience: "one of my employees in Alabama holoporting next to me in Texas." This anecdote visually illustrated the capability of the system to create a shared virtual space where a holographic representation of a person appears as if they are physically present, allowing for natural communication and interaction.

The most detailed demonstration focused on telemedicine applications through a video clip played during the presentation. This video showcased a simulated medical consultation, specifically a stroke evaluation test and physical therapy exercises. The setup involved two individuals: Mark (the observer/doctor) and Nathan Reem (the patient), physically located in different places (Mark in one location, Nathan in Alabama). Through the holographic system, Mark perceived Nathan's hologram as being "right there" in front of him, and vice-versa. The video depicted various steps of the stroke evaluation, such as Nathan being asked to "put your hands up, palms up" to check for pronator drift. This interactive procedure highlighted how a remote medical expert could visually assess a patient's motor functions and guide them through diagnostic steps with a level of spatial awareness and detail far superior to a standard video call. The ability to observe subtle movements and provide precise instructions to a holographic patient underscores the transformative potential for remote diagnostics and rehabilitation in space or other geographically isolated environments.

These demonstrations collectively served as powerful evidence that live holographic teleportation could overcome its significant technical challenges to deliver practical, high-impact applications, particularly in the critical domain of space medicine.

Defensive Implications

▶ Watch: Demonstrating immersive holographic remote interaction (6:00)

While Fernando De La Peña Llaca's talk focuses on the application and feasibility of holographic teleportation rather than explicit cybersecurity vulnerabilities, the context of deploying such a critical system on the International Space Station inherently brings forth significant defensive implications. The security and resilience of this technology are paramount, especially given its role in medical procedures and astronaut well-being.

The speaker explicitly mentioned the presence of "multiple firewalls" on the ISS network, referred to as Skynet and Cloudnet. This indicates a layered defense strategy designed to segment the network, control traffic flow, and protect sensitive operational systems from unauthorized access or malicious activity. For the holographic teleportation system, these firewalls would be crucial in:

1. Protecting Data Integrity: Ensuring that the volumetric video streams and associated medical data (such as X-rays, diagnostic information, and procedural guidance) are not tampered with during transmission or processing. Any compromise of data integrity could lead to incorrect diagnoses or flawed medical procedures, with potentially catastrophic consequences for an astronaut.
2. Maintaining System Availability: Given the intermittent communication through satellite handovers and limited bandwidth, the system must be resilient to network disruptions. Firewalls and network segmentation can help isolate potential issues, preventing a localized problem from cascading and affecting the entire holographic communication channel or broader ISS operations.
3. Preventing Unauthorized Access: The holographic system, by its nature, creates a virtual presence. Unauthorized injection of malicious holograms, or unauthorized access to the system controlling the HoloLens, could pose significant threats. This could range from psychological distress caused by unwanted or distorted holographic presences to the compromise of sensitive medical or operational information.
4. Securing Legacy Hardware: The reliance on "very old computers" (SSC computers) for their radiation resistance presents a unique security challenge. While robust against physical environmental factors, legacy hardware often lacks modern security features, patches, and frequent updates, making them potentially more susceptible to software vulnerabilities. This necessitates stringent network isolation, intrusion detection, and continuous monitoring to compensate for inherent hardware limitations.
5. Ensuring Procedural Integrity: In a medical context, the integrity of the holographic display is critical. Defenders must ensure that the holograms guiding medical procedures are accurate, untampered, and correctly aligned. A malicious actor could, for example, subtly alter the projected incision line or misrepresent vital signs, leading to medical errors. This implies a need for robust authentication of the source of holographic content and cryptographic integrity checks.

From a defensive standpoint, the successful deployment of this system suggests a strong emphasis on resilience engineering, secure communication protocols, and stringent access controls. While not explicitly a cybersecurity talk, the very nature of "Bridging Space and Medicine" with advanced technology in a high-stakes environment demands a proactive and comprehensive security posture to ensure the reliability, privacy, and safety of all operations.

Key Takeaways

• Live holographic teleportation (volumetric video) has been successfully adapted for use on the International Space Station, overcoming severe technical limitations.
• The technology enables highly immersive and interactive remote presence with "six axes of freedom," far surpassing traditional video conferencing for applications like telemedicine.
• Significant engineering challenges were overcome, including reducing camera requirements from 109 to one, processing on "very old" radiation-hardened SSC computers, and operating within a strict 5 Mbps bandwidth limit with intermittent satellite handovers.
• Telemedicine demonstrations, such as stroke evaluations and physical therapy exercises, highlight the potential to provide critical, visually guided medical assistance to astronauts in isolated environments.
• The project underscores the importance of robust network security, including "multiple firewalls," and resilience engineering when deploying critical systems in extreme, constrained, and high-stakes environments like space.
• This advancement represents a crucial step towards enhancing astronaut well-being and supporting longer-duration space missions by bridging geographical and environmental divides through mixed reality.

About the Speaker(s)

Fernando De La Peña Llaca is the speaker who presented "Bridging Space and Medicine" at DEF CON 32. His talk highlights his direct involvement in developing and deploying live holographic teleportation technology for the International Space Station. He was approached by NASA surgeon and Air Force General Dr. George Smith to make live holographic teleportation a reality on the ISS after an initial success with the HoloLens. De La Peña Llaca's work demonstrates his expertise in adapting advanced mixed reality and communication technologies to highly constrained and critical environments, particularly within the domain of space exploration and telemedicine. His presentation reflects a deep understanding of the technical challenges and innovative solutions required to achieve such ambitious goals.