To Patch or Not to Patch OT— A Risk Management Decision

To Patch or Not to Patch OT— A Risk Management Decision

Speakers: Omak Hayman, Rockwell Automation

Conference: RSAC 2024

YouTube: https://www.youtube.com/watch?v=kRitLHA_msI

Overview

In the critical realm of Operational Technology (OT), the decision to implement security patches is far more complex than in traditional IT environments. This talk, delivered by Omak Hayman of Rockwell Automation at RSAC 2024, delves into the unique challenges and imperative considerations surrounding patch management in industrial control systems (ICS). Hayman argues that patching in OT must be approached not merely as a technical task, but as a comprehensive risk management decision, carefully weighing the potential impacts of vulnerabilities against the risks of system downtime or instability introduced by patching.

Hayman draws upon his 28 years of experience with Rockwell Automation, focusing on securing ICS for the past five years, to illuminate the stark differences between IT and OT priorities. While IT often prioritizes confidentiality, integrity, and availability (CIA), OT environments are overwhelmingly driven by availability and integrity (AI). This fundamental difference shapes every aspect of security, particularly when it comes to applying updates to systems that operate 24/7/365 and underpin critical infrastructure and manufacturing processes. The discussion emphasizes the need for tailored strategies, automated tools, and a deep understanding of operational context to navigate the complexities of OT patching effectively.

The significance of this topic is underscored by the current threat landscape. Manufacturing, which constitutes approximately 11% of global GDP ($2.9 trillion), is increasingly targeted by sophisticated cyberattacks. Hayman highlights a 2023 Dragos study reporting a 50% increase in ransomware attacks targeting the manufacturing sector. With the majority of OT facilities and their control systems being decades old—some as ancient as 35-40 years—and not designed with modern cybersecurity threats in mind, the urgency for robust, risk-informed patch management strategies is paramount to protect essential services and economic stability.

Background

▶ Watch: Introduction to OT patch management challenges and leaky pipe analogy (0:00)

The landscape of Operational Technology is characterized by a unique set of challenges that fundamentally differentiate its security requirements from those of Information Technology. A primary concern is the pervasive presence of aging infrastructure. Hayman notes that the majority of manufacturing facilities are around 25 years old, housing control systems that are typically 10 to 25 years old, or even older. He provides a vivid example of a customer still relying on a Rockwell Automation PLC 5, a controller that is 35-40 years old, running a critical processing facility. These legacy systems were designed in an era when the concept of inserting malicious packets or sophisticated cyberattacks was virtually non-existent, leaving them inherently vulnerable.

This foundational lack of security by design is compounded by several operational realities. Many OT environments suffer from insecure networks and a critical lack of segmentation, allowing potential attackers lateral movement once inside. Endpoint protection is often absent, and crucially, many systems have never been patched since their initial deployment. The primary driver for this neglect is the demand for continuous operation: OT systems run 24/7/365, with any downtime being extremely limited and meticulously scheduled, sometimes for as little as an hour. The interdependencies between various control systems are often poorly documented, creating significant risks when attempting to apply patches, as unforeseen compatibility issues can lead to widespread operational disruptions. Furthermore, traditional IT security measures like Multi-Factor Authentication (MFA) are largely non-existent in older OT setups, although modern control systems are beginning to incorporate capabilities like Single Sign-On (SSO) and Role-Based Access Control (RBAC) that can tie into MFA.

Hayman references his previous RSAC talk, "U2 Can Secure OT," which focused on defense-in-depth controls, emphasizing that patching is one critical layer within a broader security strategy. The core problem lies in the inherent conflict of priorities: IT typically adheres to the Confidentiality, Integrity, Availability (CIA) triad, while OT prioritizes Availability above all else, followed by Integrity (AI). Downtime in an OT environment can translate directly into lost production, financial penalties, and even safety hazards, making any security measure that risks availability a hard sell for operational stakeholders. This context necessitates a specialized, risk-centric approach to patch management that acknowledges and mitigates these unique operational constraints.

Key Findings

▶ Watch: Specific challenges in securing operational technology environments (3:00)

Omak Hayman's presentation underscores several critical findings that challenge conventional IT-centric views on patch management when applied to OT environments. Foremost among these is the assertion that patching in OT is fundamentally a risk management decision, not solely a technical one. This decision must transcend simple vulnerability scoring, integrating a deep understanding of operational impact, interdependencies, and the organization's unique risk appetite.

A significant finding is the inadequacy of manual patch management processes for meeting regulatory compliance and effectively mitigating modern threats. Hayman cites examples like NERC CIP for power generation, which mandates security evaluations every 35 days, and TSA guidelines for gas pipelines, requiring a documented process. He contends that manually validating against "non-exploited vulnerabilities" for such frequent cycles is practically impossible, leading to the conclusion that automated tools are critical for generating asset visibility, correlating vulnerabilities, and ensuring consistency.

Another key finding highlights the limitations of traditional vulnerability scoring systems like CVSS in OT. Hayman stresses that CVSS alone is insufficient for prioritizing OT vulnerabilities. Instead, organizations must tailor their risk assessments by considering additional variables such as the specific manufacturing process, potential operational impacts, and the often-complex interdependencies between systems. This necessitates the involvement of OT experts and suppliers, who possess invaluable knowledge of system behaviors and potential integration pitfalls.

Furthermore, the talk emphasizes the necessity of OT-specific change management and disaster recovery tools. These tools must be capable of granular differentiation, down to the program and rung levels of control systems, to accurately track changes and facilitate rapid rollbacks. Hayman provides a cautionary tale of a customer who, despite having an OT-specific change management system, failed to automate the backup of its SQL database, leading to significant data loss after a ransomware attack. This underscores that a plan, however well-designed, is useless if not regularly exercised and automated to ensure accuracy and repeatability.

Finally, the presentation highlights the need for compensating controls for legacy systems that are inherently unpatchable. For instance, a 35-40 year-old PLC 5 cannot receive security updates, necessitating external protective measures like a next-generation firewall to safeguard the asset. These findings collectively advocate for a holistic, automated, and context-aware approach to OT security that moves beyond IT paradigms.

Technical Deep Dive

▶ Watch: Outline of a structured OT patch management process (4:10)

The technical core of Hayman's presentation centers on the unique architectural and operational characteristics of Industrial Control Systems (ICS) that dictate a specialized approach to cybersecurity, particularly patch management. He frames the entire discussion using a relatable "leaky pipe" analogy, where each leak represents a vulnerability requiring careful assessment before patching.

A fundamental distinction drawn is the OT vs. IT priority matrix. In IT, the CIA triad (Confidentiality, Integrity, Availability) typically guides security decisions. However, in OT, Availability is paramount, followed closely by Integrity. Confidentiality, while possible with modern systems (even at the IO level), is often a distant third priority, as system uptime directly correlates to production, safety, and revenue. This means any patch that risks availability, even for a severe vulnerability, requires extensive justification and pre-validation.

Hayman strongly advocates for the adoption of established frameworks and standards tailored for ICS. He highlights the ISA/IEC 62443 series as a "holistic group of standards" for industrial control system security. He encourages organizations without a formal patch management process to use 62443 as a starting point, adapting its requirements—such as inventory, documentation, regular checks, supplier validation, and routine patch procedures—to their specific environments. He also references NIST Special Publication 800-82, "Guide to Industrial Control Systems Security," which provides practical guidance on documentation, industry-specific procedures, leveraging OT experts, and implementing segmentation (e.g., using Industrial Demilitarized Zones - IDZs) to protect critical systems. NIST 800-82 also emphasizes the importance of backup plans and compensating controls for unpatchable systems, such as placing a next-generation firewall in front of a legacy PLC 5.

The discussion delves into the nature of OT assets and their vulnerabilities. Legacy systems like the 35-40-year-old PLC 5 have no inherent security features, requiring external protections. Modern control systems, however, are increasingly capable, offering features like SSO, RBAC, and MFA integration, as well as the ability to enforce integrity and confidentiality at the IO level. Connectivity between controllers can range from simple "message instructions" to more sophisticated, vendor-agnostic protocols like OPC UA. These diverse technologies necessitate a nuanced approach to vulnerability assessment and patching.

A critical technical detail involves firmware updates. Hayman illustrates this with a Rockwell drive example: older drives used unsigned firmware, while newer versions require signed firmware. Flashing a drive with signed firmware often makes it impossible to revert to unsigned firmware, a crucial detail that must be understood and tested as part of the patch plan to avoid irreversible changes.

Finally, the talk emphasizes OT-specific change management and disaster recovery tools. Unlike generic IT solutions, these tools must differentiate changes down to the "program and rung levels" of control logic. They should automate the upload of configurations on a schedule, validate system integrity, and provide an accurate, current copy for disaster recovery. Hayman recounts a scenario where a customer's OT change management system used a SQL backend but lacked automation for its own database backup, leading to significant data loss after a ransomware attack. This highlights the need for end-to-end automation and validation within these critical systems.

Demo / Proof of Concept

▶ Watch: Importance of risk management and industry compliance standards (5:00)

While Omak Hayman's presentation was rich with practical insights and technical guidance, it did not feature a live technical demonstration or a software proof of concept. Instead, the speaker effectively utilized an extended analogy of a "leaky pipe" at a paper mill to illustrate the complexities and decision-making processes involved in patching Operational Technology systems. He also mentioned leveraging Chat GPT 4.0 to generate the visual graphics that accompanied his narrative, bringing the story to life. The focus remained on the strategic and procedural aspects of OT patch management rather than showcasing specific tools in action.

Defensive Implications

▶ Watch: Regulatory compliance examples for industrial patch management (6:00)

The defensive implications of Omak Hayman's talk are profound, urging a fundamental shift in how organizations approach security in Operational Technology environments. The central message is to move beyond IT-centric patching paradigms and adopt a risk-based, OT-specific strategy that prioritizes availability and operational integrity.

Firstly, automation is non-negotiable. Defenders must invest in and implement automated tools for asset inventory and vulnerability correlation. Manual processes are prone to human error, lack consistency, and cannot keep pace with the compliance requirements (e.g., NERC CIP's 35-day evaluation cycle) or the sheer volume of potential vulnerabilities. Automated solutions reduce the administrative burden and provide the necessary visibility to make informed decisions.

Secondly, vulnerability prioritization must be tailored. Relying solely on CVSS scores is insufficient for OT. Defenders must incorporate operational context, potential impact on production, safety, and environmental factors into their risk assessments. This requires close collaboration between IT, OT, and business stakeholders to develop a stakeholder-specific vulnerability categorization process that reflects the true risk to the organization.

Thirdly, supplier and vendor engagement is critical. OT environments often comprise a heterogeneous mix of systems from various manufacturers. Defenders must actively engage these suppliers, leveraging their deep knowledge of product interdependencies, known vulnerabilities, and recommended patching procedures. This collaborative approach can prevent unforeseen compatibility issues and ensure patches are applied correctly.

Fourthly, robust, OT-specific change management and disaster recovery programs are essential. These programs must go beyond simply backing up files; they need the capability to differentiate changes down to the "program and rung levels" of control logic. Automated, scheduled uploads of configurations to a secure database are vital, coupled with validation mechanisms to detect unauthorized changes. This ensures that accurate, current system configurations are available for rapid recovery in the event of a patch failure or a cyberattack like ransomware. The speaker's anecdote about the SQL database not being backed up highlights that the recovery plan itself needs to be comprehensively secured and automated.

Fifthly, compensating controls are a must for legacy systems. For unpatchable assets like 35-40-year-old PLC 5s that lack inherent security, defenders must implement external protective measures. This could involve network segmentation with Industrial Demilitarized Zones (IDZs), deploying next-generation firewalls in front of critical legacy assets, or implementing other network-based protections to isolate and monitor vulnerable systems.

Finally, plans must be regularly exercised and ingrained into organizational culture. A well-documented patch management or disaster recovery plan is ineffective if it's not practiced. Defenders need to schedule regular drills, training, and reviews to ensure that personnel are familiar with procedures, tools are functioning correctly, and the organization can respond effectively when a real incident occurs. This ensures repeatability and builds confidence in the organization's ability to manage OT security risks.

Key Takeaways

• Automate OT Inventory and Vulnerability Correlation: Implement automated tools to gain comprehensive visibility into OT assets and efficiently correlate them with identified vulnerabilities. Manual processes are insufficient for compliance and effective risk management in complex OT environments.
• Develop OT-Specific Change Management and Disaster Recovery: Establish a robust program that includes OT-specific tools capable of differentiating changes down to program and rung levels, automating configuration uploads, and ensuring accurate backups for rapid recovery from patch failures or cyberattacks.
• Tailor Vulnerability Prioritization Beyond CVSS: Move beyond generic CVSS scores by incorporating operational impact, interdependencies, and business-specific risk tolerance. Involve OT experts and manufacturers in this decision-making process to account for unique system behaviors and constraints.
• Regularly Exercise All Plans: A patch management and disaster recovery plan is only as good as its last exercise. Conduct frequent drills, training, and reviews (e.g., within a six-month period) to ensure the plan is repeatable, effective, and understood by all relevant personnel.
• Prioritize Availability with Compensating Controls: Acknowledge that availability is paramount in OT. For unpatchable legacy systems, implement compensating controls such as network segmentation and next-generation firewalls, rather than attempting risky or impossible patches.
• Engage Manufacturers and OT Experts: Leverage the specialized knowledge of system manufacturers and internal OT personnel to understand system interdependencies, known vulnerabilities, and the safest patching methodologies.

About the Speaker(s)

Omak Hayman is a seasoned expert in industrial control systems with an extensive career at Rockwell Automation. He has been with the company for 28 years, dedicating his focus to the intricate world of ICS. For the last five years, Hayman has specialized in assisting Rockwell's customers in securing these critical systems. His deep understanding of both operational technology and cybersecurity challenges makes him a credible voice in addressing the complexities of patch management in industrial environments. He has also previously presented at RSAC, giving a talk titled "U2 Can Secure OT," which focused on broader defense-in-depth strategies.