Simulating attacks against hydroelectric power plants

Simulating Attacks Against Hydroelectric Power Plants

Speakers: Julia Dewitz-Würzelberger

Conference: DEF CON 32

YouTube: https://www.youtube.com/watch?v=O48DBFT02eU

Overview

This talk, presented by Julia Dewitz-Würzelberger from Verbund, in collaboration with colleagues Sarah and Bernhard from Enviso, delves into the critical need for robust operational technology (OT) cybersecurity training through the development of a unique Industrial Control System (ICS) firing range. The focus of this innovative project is a prototype of a run-of-river hydroelectric power plant, designed to simulate cyber-physical attacks and provide both foundational awareness and advanced forensic training. As a major hydropower energy supplier in Austria, Verbund recognizes the paramount importance of securing critical infrastructure, particularly in the face of an escalating global OT threat landscape.

The motivation behind this joint venture between Verbund, a utility company, and Enviso, a security consultancy, stems from a dual objective. Firstly, to raise general awareness about the potential impact of cyberattacks on physical systems among non-specialist personnel. Secondly, to offer in-depth forensic training for OT cybersecurity specialists, equipping them with the skills to detect, analyze, and respond to real-world threats. This prototype serves as a tangible and immersive learning environment, bridging the gap between theoretical knowledge and practical application, and represents a crucial step towards Verbund's larger goal of establishing a comprehensive OT cyber security range in Austria.

The project highlights the growing urgency for critical infrastructure operators to not only implement defensive measures but also to actively train their workforce in incident response and forensic analysis. By creating a scaled-down yet highly realistic model of a hydropower plant, the team has provided an invaluable tool for understanding the cascading effects of cyberattacks, from network intrusion to physical disruption, such as simulating an "immense flood" over the plant. This initiative underscores a proactive approach to cybersecurity resilience, emphasizing that effective defense relies as much on human preparedness as it does on technological safeguards.

Background

▶ Watch: Introduction to the run-of-river hydropower plant prototype (0:00)

The increasing sophistication and frequency of cyberattacks targeting critical infrastructure, particularly Operational Technology (OT) environments, has made robust cybersecurity a top priority for entities like Verbund. As a large power supplier in Austria, operating over 130 power plants, Verbund is acutely aware of its role as critical infrastructure and the catastrophic potential of disruptions to energy supply. The speakers emphasized that the current OT threat landscape is "getting worse," necessitating advanced capabilities for detection and response. This context provided the primary impetus for Verbund to invest in an innovative project aimed at strengthening its cyber resilience.

Verbund's long-term vision includes building a substantial, dedicated OT cyber security range in Styria, Austria, to train both internal specialists and external partners. The run-of-river hydroelectric power plant prototype discussed in this talk serves as a foundational step and proof-of-concept for this ambitious undertaking. The objective was not merely to create a theoretical training module but a highly visual and practical approach that makes the abstract concepts of cyber threats tangible. This hands-on methodology is considered vital for effective learning and for raising awareness across different levels of technical understanding within an organization.

Enviso, as a security consultant company specializing in critical infrastructure, played a pivotal role in the construction and design of the prototype. Their motivation aligns with strengthening the security resilience of their customers by developing realistic attack scenarios that mirror actual OT cyber threats. This collaboration facilitated the integration of deep cybersecurity expertise with the operational knowledge of hydropower plant engineers. Crucially, the design process involved extensive input from Verbund's internal operational colleagues and equipment manufacturers, ensuring the abstracted model was "as close as possible" to a real plant, including the crucial element of a control room Human-Machine Interface (HMI) where all operational data converges. This iterative and collaborative design approach was essential to creating a training environment that truly reflects the complexities and challenges of real-world industrial control systems.

Key Findings

▶ Watch: Summarizing project objectives: awareness and forensic training (4:00)

The primary finding of this project is the successful development and implementation of a highly effective ICS firing range prototype specifically tailored for a run-of-river hydroelectric power plant. This prototype demonstrates that complex, real-world OT environments can be accurately abstracted into a functional training model, offering significant benefits for cybersecurity preparedness.

Key findings and contributions include:

1. Dual-Purpose Training Efficacy: The range effectively serves two distinct but equally important objectives:

• Awareness Building: For individuals with limited cybersecurity knowledge, the visual and tangible nature of the prototype makes the impact of cyberattacks on physical infrastructure immediately apparent and understandable.
• Forensic Training: For specialists, it provides a deep-dive environment to investigate the intricacies of an attack, understand attacker methodologies, and practice incident response.

1. Realistic Scenario Simulation: The prototype integrates both physical and virtual components to create a convincing simulation. This includes a physical model of the power plant with operational elements like wears and turbines, alongside a sophisticated network infrastructure mirroring both IT and OT environments. The ability to project simulated water levels onto the physical model further enhances realism, allowing trainees to visually observe the consequences of their actions or simulated attacks.

1. Comprehensive Forensic Curriculum: The team developed a robust forensic training curriculum consisting of 15 distinct labs, categorized into three main parts:

• Analyzing: Focusing on initial data collection and assessment.
• Attacker Methods and Tools: Exploring common techniques and tools used by adversaries in OT environments.
• Siemens Toolbox Analysis: A deeper examination of specific industrial software and its role in potential attack vectors and forensic investigation.

1. Addressing Operational Challenges: The project highlighted and successfully navigated several practical challenges inherent in building such a complex training environment:

• Logistics of Physical Infrastructure: The "bulky and heavy" nature of the ICS firing range underscored the physical demands of setting up and transporting such a system, a crucial lesson for future large-scale deployments.
• Robust Backup and Restore Solutions: A significant effort was dedicated to developing a "easy to use and also good resilient backup and restore solution." This was critical for quickly resetting training scenarios, involving consistent ESXi backups combined with graphics and Raspberry Pi configurations. This finding emphasizes the operational necessity of rapid recovery in training environments, mirroring the urgency in real-world incident response.
• Tool Adaptability for OT Operators: The team discovered that standard security tools, like T-Shark, were not always familiar to OT operators. Consequently, they adapted their training to use more user-friendly alternatives, such as Wireshark, to ensure accessibility and effectiveness for the target audience. This highlights the importance of tailoring training approaches and tools to the specific skillset and background of OT personnel.

1. Integration of Real-World Feedback: The project's success was heavily reliant on incorporating knowledge from internal Verbund colleagues (OT operators) and external manufacturers. This collaborative approach ensured the prototype's design and functionality accurately reflected real-world operational requirements, particularly the central role of the HMI as a control and monitoring hub.

In essence, the project not only delivered a functional training prototype but also generated valuable insights into the practicalities of developing, maintaining, and deploying effective OT cybersecurity training infrastructure, setting a precedent for future critical infrastructure resilience initiatives.

Technical Deep Dive

▶ Watch: Incorporating operational knowledge and HMI control room (5:15)

The ICS firing range prototype for the hydroelectric power plant is architecturally divided into two primary layers: the Scenario and Visualization Layer and the underlying Electric Cabinet containing the technical components. This layered approach ensures both realistic physical representation and robust technical functionality.

The Scenario and Visualization Layer provides the physical, tangible representation of the power plant. It includes a modulated hydro power plant model featuring:

• A simulated river, complete with head and tail water sections.
• A turbine building housing the critical power generation components.
• Two types of weirs (or wears): a twin weir for normal operations and an emergency weir for critical situations. These physical elements are central to demonstrating water flow control and potential attack vectors.
• To enhance interaction and realism, a Human-Machine Interface (HMI) is integrated directly into the cyber range. This HMI acts as a control room interface, allowing trainees to monitor various operational data streams from the plant model and actively control elements like the weirs, including their opening levels, and even simulate raising water levels.
• A detailed model of the turbine, complete with shutters that physically control the water flow impacting the turbine blades.

A crucial innovation for visualizing dynamic processes on the static model is the Visualization Machine. This system comprises a projector mounted above the model, working in conjunction with a thin client. The thin client consumes simulated water level data from an MQTT broker – a lightweight messaging protocol often used in IoT and industrial contexts – and the projector then dynamically projects water flow animations onto the physical model. This ingenious solution brings the "water" to life, allowing trainees to visually track water levels and flow changes in response to simulated operational adjustments or cyberattacks, making the abstract concept of water management tangible.

Beneath this visualization layer lies the sophisticated Network Infrastructure, which is logically separated into IT (Information Technology) and OT (Operational Technology) networks, reflecting real-world industrial environments. This segregation is fundamental for simulating typical attack paths that often bridge these two domains.

The IT Network is entirely virtualized, hosting multiple virtual machines that emulate a standard office environment. This allows for the simulation of initial attack vectors that might originate in the corporate IT network before pivoting to the more sensitive OT systems.

The OT Network is further divided into physical and virtual components to accurately represent industrial control systems:

• The physical OT network hosts the actual Programmable Logic Controllers (PLCs). These are the workhorse components of industrial automation, directly controlling physical processes like weir gates and turbine shutters. In the prototype, these PLCs are responsible for translating commands from the HMI and other systems into physical actions on the model.
• The virtual OT network hosts a virtualized Active Directory, which is common in many modern industrial environments for user authentication and management. This provides another layer for simulating realistic cyberattack scenarios, such as credential theft or privilege escalation within the OT domain.

Communication between these diverse components relies on industry-standard protocols and interfaces:

• The PLCs and Raspberry Pis (likely used for sensor/actuator interfacing and potentially gateway functions) communicate via Modbus. Modbus is a widely adopted serial communication protocol in industrial automation, making its inclusion essential for realism.
• The HMI communicates with the PLCs using S7comm. This is a proprietary Siemens protocol commonly used for communication with Siemens PLCs, highlighting the specific industrial context of the training.
• Sensors and Actuators, such as the opening level sensors for the weirs and the control mechanisms for the turbine shutters, are connected to the Raspberry Pis via I2C (Inter-Integrated Circuit) and GPIOs (General Purpose Input/Output). These low-level interfaces are typical for embedded systems and provide direct control and feedback from the physical model components.

This intricate technical architecture allows the ICS firing range to simulate a wide array of cyber-physical interactions, from network-based intrusions in the IT domain to direct manipulation of industrial processes at the PLC level, culminating in visible physical effects on the hydroelectric power plant model.

Demo / Proof of Concept

▶ Watch: Technical overview: scenario, visualization, and physical layers (7:45)

While the transcript does not provide a step-by-step walkthrough of a live demonstration, the core of the presentation itself serves as a comprehensive proof of concept for the ICS firing range. The speakers described the full functionality and capabilities of the prototype, which inherently demonstrates its viability as a training tool.

The primary demonstration involves the operational hydroelectric power plant model, where the HMI acts as the central control panel. Trainees can interact with this HMI to perform typical operational tasks, such as adjusting the weirs or monitoring water levels. This interactive capability, coupled with the visualization machine projecting dynamic water flow onto the model, makes the effects of control actions immediately apparent. For instance, manipulating the weir controls on the HMI would result in a visual change in water levels and flow on the physical model, showcasing the direct link between digital commands and physical outcomes.

Crucially, the range is designed to simulate cyberattacks and their resulting physical impacts. Although the detailed attack scenario was not fully elaborated in the transcript, the speakers alluded to a dramatic outcome: "immense flood is running over the water power plant and also the environment." This statement confirms the ability of the range to demonstrate catastrophic physical consequences stemming from cyber intrusions, making the abstract threat of a cyberattack tangible and impactful for trainees.

The forensic training modules, consisting of 15 labs, also serve as a demonstration of the range's analytical capabilities. Trainees are immersed in scenarios where they must investigate how such an attack worked, utilizing tools and techniques to analyze network traffic (PCAP files) and artifacts from industrial software, such as the Siemens toolbox. This hands-on investigation, facilitated by the realistic network and control system setup, proves the range's utility for developing critical incident response and forensic skills.

In essence, the entire ICS firing range, with its integrated physical model, HMI, visualization, and network infrastructure, stands as a robust proof of concept for effective, realistic OT cybersecurity training. It moves beyond theoretical discussions to provide a dynamic environment where the consequences of cyberattacks on critical infrastructure can be experienced and analyzed firsthand.

Defensive Implications

▶ Watch: Integrating visualization machine with projector for realism (8:45)

The development and insights from this hydroelectric power plant ICS firing range offer several critical defensive implications for organizations operating critical infrastructure and OT environments:

1. Prioritize Realistic, Hands-On Training: The most prominent implication is the undeniable need for realistic and tangible training for OT personnel. The project explicitly aims to strengthen "security resilience" through "realistic attack scenarios." This moves beyond theoretical cybersecurity education to practical, experiential learning, which is crucial for building muscle memory and effective decision-making during a real incident. Organizations should invest in or develop similar cyber ranges to provide this invaluable hands-on experience.

1. Foster Cross-Domain Awareness: The range's dual objective of "awareness building" for general staff and "forensic training" for specialists highlights the importance of a multi-tiered training approach. Defenders must ensure that not only their cybersecurity experts but also their operational staff understand the potential cyber threats and their physical consequences. This broad awareness can improve early detection and communication during an incident.

1. Strengthen Incident Response and Forensic Capabilities: The 15 forensic training modules underscore the necessity of robust incident response (IR) and digital forensics and incident response (DFIR) capabilities within OT. Defenders must be proficient in analyzing network traffic (PCAP files), understanding attacker methodologies, and investigating artifacts from proprietary industrial software like the Siemens toolbox. Training should focus on the unique challenges of OT forensics, including the analysis of industrial protocols such as Modbus and S7comm.

1. Develop Robust Backup and Restore Strategies: The significant effort required to create a "good resilient backup and restore solution" for the complex range directly translates to real-world OT environments. Defenders must prioritize the development and testing of rapid, consistent backup and restore procedures for their critical industrial control systems, encompassing not just software but also configurations for PLCs, HMIs, and associated network infrastructure. The ability to quickly revert to a known good state is paramount for operational continuity after an attack.

1. Tailor Tools and Training to OT Operators: The experience of switching from T-Shark to Wireshark for forensic analysis due to operator familiarity is a crucial lesson. Defensive strategies and training programs must be tailored to the existing skill sets and preferred tools of OT personnel. Forcing unfamiliar tools can hinder effective incident response. Organizations should identify and standardize on tools that are both effective and user-friendly for their operational teams.

1. Integrate Operational Knowledge into Security Design: The success of the prototype relied heavily on input from internal operational colleagues and manufacturers. This emphasizes that effective OT security solutions and training must be co-designed with operational staff who possess deep knowledge of the physical processes and system intricacies. Security professionals must collaborate closely with OT engineers to understand the true impact of vulnerabilities and design appropriate controls.

1. Emphasize Physical-Cyber Interdependencies: The ability of the range to simulate an "immense flood" due to a cyberattack vividly demonstrates the direct link between cyber intrusions and physical consequences. Defenders must always consider the cyber-physical attack surface and understand how digital vulnerabilities can manifest as real-world operational disruptions, safety hazards, or environmental damage. Security assessments should explicitly evaluate these interdependencies.

By adopting these defensive implications, organizations can move towards a more proactive, integrated, and resilient cybersecurity posture for their critical OT infrastructure, better preparing them to detect, respond to, and recover from sophisticated cyber threats.

Key Takeaways

• Realistic ICS firing ranges are indispensable for effective OT cybersecurity training, bridging the gap between theoretical knowledge and practical application for critical infrastructure.
• Comprehensive OT training programs should cater to both general awareness building for broader personnel and deep forensic analysis for specialized incident responders.
• Successful development of OT training environments requires significant integration of operational knowledge from internal engineers and expertise from equipment manufacturers to ensure realism and relevance.
• Robust backup and restore solutions are paramount for complex OT training ranges, mirroring the critical need for rapid recovery capabilities in real-world industrial control systems.
• Effective incident response in OT environments necessitates the use of tools familiar to OT operators (e.g., Wireshark over T-Shark) to maximize training efficacy and real-world applicability.
• Simulating the physical consequences of cyberattacks, such as an "immense flood" on a hydroelectric plant, makes abstract threats tangible, highlighting the cyber-physical impact and urgency of OT security.

About the Speaker(s)

Julia Dewitz-Würzelberger is a cybersecurity professional from Verbund, a prominent hydropower energy supplier based in Austria. Her work focuses on Operational Technology (OT) cyber security, and she was a key figure in initiating and leading the project to develop the ICS firing range prototype discussed in this talk. Her involvement underscores Verbund's commitment to enhancing its cyber resilience as a critical infrastructure provider.

Sarah is a security consultant from Enviso, a European security consulting company. Enviso specializes in providing security services, particularly for the financial, technology, governmental, and critical infrastructure sectors. Sarah played a significant role in the creation process of the run-of-river power plant prototype, contributing Enviso's expertise in building realistic attack scenarios and tailored training approaches.

Bernhard, also from Enviso, is involved in OT cyber security. He contributed to the project's technical and training aspects, particularly in the development of the detailed forensic training modules within the ICS firing range. His expertise was instrumental in designing the practical investigative components of the training.