Software Update Frameworks and the CI Supply Chain

Software Update Frameworks and the CI Supply Chain

Speakers: Brian Romansky

Conference: S4

YouTube: https://www.youtube.com/watch?v=Eq6Vc0WKQ3Y

Overview

In his S4 conference talk, "Software Update Frameworks and the CI Supply Chain," Brian Romansky of Convolution Solutions and George Washington University delves into the complex and often contradictory nature of software updates, particularly within Operational Technology (OT) environments. The presentation highlights a critical paradox: while software updates are essential for patching vulnerabilities and improving functionality in Information Technology (IT), they simultaneously introduce significant attack surface and operational risks in critical infrastructure. Romansky argues that the traditional approaches to software updates, often designed for IT, are fundamentally inadequate for the unique demands and stringent security requirements of OT systems.

Romansky's research, conducted in cooperation with Dr. Thomas Mazuki and Sharam Sakani, focuses on developing next-generation software update solutions that bridge this gap. The talk underscores the devastating impact of compromised software supply chains, citing prominent examples like SolarWinds and ASUS Shadow Hammer. A central theme is the need to move beyond simplistic digital signatures to more robust frameworks that can withstand sophisticated attacks while accommodating the "trust but verify" ethos prevalent in critical infrastructure. The proposed solution involves extending existing frameworks like The Update Framework (TUF) with a new "deployer role" to facilitate secure, validated updates in complex OT networks.

The importance of this work cannot be overstated in an era where supply chain attacks are increasingly common and impactful. As industrial control systems (ICS) become more interconnected and reliant on software, securing the update mechanism becomes paramount to national security and economic stability. Romansky's insights provide a crucial roadmap for organizations to re-evaluate their software update strategies, ensuring resilience against advanced persistent threats and maintaining operational integrity in critical infrastructure environments.

Background

▶ Watch: Introduction: Software update contradiction in IT vs. OT (0:00)

The core premise of Romansky's talk revolves around what he terms the "contradiction or conundrum" of software updates. In the IT world, frequent updates are a best practice, crucial for blocking new attacks, supporting defense-in-depth, and maintaining system health. Neglecting updates is often considered negligent. However, in the OT world, the calculus changes dramatically. Software updates become a highly attractive target for attackers, offering a direct conduit for injecting malicious code. They can have unintended consequences, especially for low-powered devices running at their operational limits, potentially causing control system instability even if a security vulnerability is patched. Furthermore, updates can inadvertently highlight vulnerabilities by revealing what was fixed, enabling attackers to target unpatched systems by analyzing binary diffs.

This dichotomy is vividly illustrated by comparing a consumer Nest thermostat with an industrial temperature controller. While both perform similar functions, the Nest device benefits from automated, signed updates, assuming the user has limited defense-in-depth. An industrial controller, however, demands a "trust but verify" approach: rigorous testing on an identically configured internal system, multiple validation steps, and careful deployment across potentially thousands of devices. This process is expensive, complex, and requires significant investment in hardware and time.

The history of software supply chain compromises underscores the urgency of this problem. Romansky cites several high-profile incidents:

• Stuxnet: A relatively simplistic attack leveraging USB sticks to update devices with malicious code.
• ASUS Shadow Hammer: Attackers compromised a software distribution hub, gaining credentials to post their own fake code, which was then distributed to millions of ASUS devices.
• SolarWinds: Attackers infiltrated the development and release cycle, injecting malicious code upstream that was subsequently distributed through legitimate updates, leading to widespread compromise.
• Kaseya VSA: A software installation tool used by many Managed Service Providers (MSPs) was compromised, allowing attackers to inject and automatically distribute malicious code to numerous client systems.

These examples highlight that attacks can occur at various stages of the supply chain, from development to distribution.

A critical challenge in OT environments is the inherent complexity of multi-layer networks, designed with extensive defense-in-depth. This architecture makes it difficult for edge devices to perform necessary security checks, such as accessing Certificate Revocation Lists (CRLs) or obtaining time updates, due to their limited external visibility and computational capabilities. Even the "gold standard" of digital signatures presents limitations. A simple digital signature on code doesn't account for key compromise, expiration, or the need for Public Key Infrastructure (PKI) updates, making it insufficient for long-term security in dynamic threat landscapes. Romansky stresses the crucial distinction between source integrity (e.g., relying solely on HTTPS connections) and code integrity (verifying the content of the binary itself with a digital signature), noting that source integrity alone is easily compromised.

Even with digital signatures, various sophisticated attacks remain possible. These include rollback attacks (deploying older, vulnerable but legitimately signed software), freeze attacks (preventing updates to prolong an exploitation window), and extraneous dependency attacks (compromising linked libraries or components that an update dynamically pulls). These threats necessitate a more comprehensive and resilient framework for managing software updates.

Key Findings

▶ Watch: Real-world examples of software supply chain attacks (2:25)

Romansky's talk identifies several key findings regarding the state of software update security, particularly in the context of critical infrastructure:

1. The Update Contradiction: Software updates are simultaneously a critical security measure (in IT) and a significant attack vector (especially in OT). This fundamental tension necessitates distinct approaches for different operational environments.
2. Inadequacy of Basic Digital Signatures: While digital signatures are a foundational security measure, relying solely on them is insufficient. They do not inherently address issues like key compromise, expiration, revocation, or the need for secure key rollover, leaving systems vulnerable to sophisticated attacks.
3. PKI Maturity is Paramount: A robust and mature Public Key Infrastructure (PKI) is essential. This includes having clear certificate policies and practice statements, documented processes for certificate revocation, and well-defined root and Intermediate Certificate Authority (ICA) rollover policies to manage the lifecycle of cryptographic keys effectively.
4. Source vs. Code Integrity: There is a critical distinction between ensuring source integrity (e.g., using HTTPS for secure transport) and code integrity (cryptographically verifying the actual content of the software package). Relying only on source integrity is a common and dangerous misconception, as demonstrated by attacks like ASUS Shadow Hammer.
5. Sophisticated Update Attacks: Even with digital signatures, systems are vulnerable to specific attack types, including arbitrary software installation, rollback attacks (deploying older, vulnerable signed versions), freeze attacks (preventing updates to extend an exploit window), and extraneous dependency attacks (compromising components pulled dynamically by an update).
6. TUF as a Foundation, but Lacking OT Specificity: The Update Framework (TUF) is recognized as a robust, industry-standard framework designed to protect against a wide range of software supply chain attacks through the use of multiple keys and signatures. However, its original design is primarily for IT environments (e.g., Linux repositories, Docker containers) and lacks specific mechanisms for the rigorous on-premises validation required in OT.
7. The Need for an OT-Specific "Deployer Role": The most significant contribution of Romansky's work is the proposal to extend TUF with a new "deployer role." This addition allows for local, on-premises validation and multi-signature approval of updates after they have been received from the vendor but before they are deployed to target OT devices. This addresses the "trust but verify" requirement of critical infrastructure.

These findings collectively underscore the complexity of securing software updates in OT and provide a clear direction for developing more resilient and context-appropriate solutions.

Technical Deep Dive

▶ Watch: Contrasting update needs: Nest thermostat vs. industrial controller (4:00)

The core of Romansky's technical discussion centers on the limitations of conventional software update mechanisms and the architecture of more robust solutions, culminating in his proposed extension for OT.

Firstly, Romansky critiques the common reliance on simple digital signatures for verifying software updates. While a good starting point, a basic signature on code is insufficient because it doesn't address the lifecycle management of cryptographic keys. Keys can be compromised, expire, or need to be replaced. A truly secure system requires a robust Public Key Infrastructure (PKI) that includes:

• Certificate Policy (CP) and Certificate Practice Statement (CPS): Documented definitions of what certificates are signed, why, and how the PKI operates. Operators should be able to review these to understand the vendor's security posture.
• Certificate Revocation Process: A clear, documented mechanism for recognizing when a key or certificate has been compromised and effectively revoking its validity across all relying systems.
• Root and ICA Rollover Policy: Since keys and certificates have finite lifetimes, there must be a plan for securely updating root and intermediate certificates on devices in the field, ensuring continuity of trust as old keys expire. Without these, devices can be locked into trusting potentially compromised or outdated keys indefinitely.

Romansky also emphasizes the critical difference between source integrity and code integrity. Many vendors mistakenly believe that simply delivering updates over HTTPS guarantees security. However, HTTPS only ensures the integrity of the connection, not the content itself. Attackers can compromise the distribution hub upstream, as seen with ASUS Shadow Hammer, and inject malicious code that is then served over a legitimate HTTPS connection. True security requires the device to independently check the digital signature on the binary content of the update package, verifying its authenticity and integrity directly.

The talk then details specific threats that even digitally signed updates can face:

• Arbitrary Software Installation: This is the most basic threat, where a device accepts and runs any code without verification. Digital signatures mitigate this, but other attacks still exist.
• Rollback Attacks: An attacker might attempt to deploy an older version of software known to contain a vulnerability. If the update mechanism only checks for a valid digital signature and not for version numbers or incremental counters, it might accept the older, vulnerable code, allowing the attacker to exploit known flaws.
• Freeze Attacks: Instead of injecting malicious code, an attacker might try to prevent a crucial security update from reaching a device. This could involve compromising the delivery channel to send junk files or corrupting the update package so it fails validation. The goal is to extend the window during which a known vulnerability in the system can be exploited.
• Extraneous Dependencies: Many update mechanisms, particularly in repository-based systems, allow updates to reference external dependencies (e.g., "to make this work, you also need the latest version of this other library"). If these dependencies are not part of the signed package or are compromised, they can introduce vulnerabilities. Updates should ideally be atomic and fully packaged, avoiding dynamic fetching of additional components.

To counter these sophisticated threats, Romansky introduces The Update Framework (TUF). TUF is a robust, open-source framework designed to secure software update systems by using multiple keys and signatures, thereby increasing the complexity for attackers. Key components of TUF include:

• Root Keys: These are the most trusted keys, shared with the system at manufacturing. TUF includes mechanisms to securely update these root keys over time, allowing for root rollover and management.
• Targets.txt: This is a digitally signed manifest that specifies the hash of all the code components that should be installed. It is designed to be fully inclusive, meaning there are no hidden or external dependencies. This protects against extraneous dependency attacks and ensures the integrity of the entire package.
• Release.txt: This is another digitally signed file that points to the most recent valid version of the targets.txt file. It ensures that devices are always directed to the latest authorized release.
• Timestamp.txt: This file is periodically digitally signed and delivered to devices. It indicates the current valid release and serves as a heartbeat. If a device fails to receive an expected timestamp.txt file, it immediately knows something is wrong (e.g., a freeze attack is in progress), even if it cannot recover from the attack. This provides crucial detection capabilities.

While TUF is highly robust for IT, Romansky highlights its limitations for OT environments. OT requires an additional layer of on-premises validation due to the "trust but verify" principle. To address this, Romansky proposes adding a "deployer role" to TUF. This new role introduces a deploy.text file, which is a locally signed file. After an OT operator receives a software update from a vendor (which would be secured by the existing TUF mechanisms), they would perform their rigorous internal testing and validation on a nearly identical system. If the tests are successful and the code is deemed safe for deployment, the operator would then create and sign the deploy.text file. This file references the version number and hash of the targets.txt manifest, effectively attesting that the code has been locally validated and approved. Romansky further suggests that a best practice would involve at least two signatures on the deploy.text file (e.g., from two different operators or departments) to prevent collusion and ensure a higher level of internal approval before deployment to critical devices. This deployer role extends TUF's strong cryptographic guarantees with the necessary operational checks for OT.

Demo / Proof of Concept

▶ Watch: Challenges of updates in complex critical infrastructure networks (5:00)

The talk by Brian Romansky at S4 was presented as a high-level overview of his research. As stated in the transcript, "this talk is fairly high level. There is a more detailed technical representation of this work that's coming out in IEEE Spectrum." Consequently, the presentation did not include a live demonstration or a detailed proof of concept of the proposed TUF extension with the deployer role. The focus was on outlining the problem, existing solutions like TUF, and the conceptual framework for adapting it to OT.

Defensive Implications

▶ Watch: Limitations of digital signatures in OT environments (5:30)

The insights presented by Brian Romansky offer critical guidance for both OT operators and software vendors to enhance the security of software update processes.

For OT Operators:

1. Adopt "Trust But Verify" Rigorously: Do not simply apply updates based on vendor assurance or digital signatures alone. Implement a comprehensive internal testing and validation pipeline. This means running updates on nearly identically configured internal systems under various operating conditions before deploying them to production. This is an expensive but necessary investment.
2. Demand PKI Maturity from Vendors: When evaluating vendors, ask specific questions about their Public Key Infrastructure (PKI). Inquire about their Certificate Policy (CP) and Certificate Practice Statement (CPS), their documented processes for certificate revocation, and their plans for root and ICA rollover. A mature PKI is a hallmark of a responsible vendor.
3. Understand Code Integrity vs. Source Integrity: Recognize that a secure connection (e.g., HTTPS) does not guarantee the integrity of the software content. Always ensure that devices or intermediate systems verify the digital signature on the actual binary content of the update package.
4. Be Aware of Advanced Attack Vectors: Educate teams on threats beyond simple code injection, such as rollback attacks (ensuring update mechanisms check version numbers and prevent downgrades), freeze attacks (monitoring for expected timestamp.txt or similar "heartbeat" files), and extraneous dependency attacks (preferring atomic, fully packaged updates).
5. Implement a Deployer Role or Similar Internal Approval: Integrate a formal, cryptographically enforced approval step for updates within your operational environment. The proposed TUF deployer role with multi-signature approval (e.g., two operators signing off) provides a robust model for this. This creates a strong internal control gate, ensuring that only validated and approved code reaches critical systems.
6. Invest in Monitoring and Alerting: Monitor update delivery channels for anomalies, missing updates, or failed validations, which could indicate a freeze attack or other compromise.

For Software Vendors:

1. Build Mature PKI Infrastructures: Go beyond basic digital signatures. Develop and maintain a comprehensive PKI with clear policies, robust revocation mechanisms, and planned key rollover procedures. This demonstrates a commitment to long-term security.
2. Implement Robust Update Frameworks: Consider adopting or adapting frameworks like The Update Framework (TUF) in your software distribution pipeline. TUF's multi-key, multi-signature approach offers significantly stronger protection against a wide array of supply chain attacks compared to simpler methods.
3. Ensure Atomic and Comprehensive Updates: Package updates atomically, including all necessary components and dependencies within the signed manifest (like targets.txt in TUF). Avoid dynamic fetching of external, unsigned, or unverified components.
4. Provide Transparency and Documentation: Be transparent about your update security mechanisms, providing clear documentation of your PKI, signing processes, and threat mitigations to your customers. This builds trust and enables operators to perform their due diligence.
5. Design for OT Constraints: Acknowledge the unique constraints of OT environments (limited device capabilities, isolated networks, "trust but verify" ethos). Design update mechanisms that can be adapted for these environments, potentially incorporating features like the proposed deployer role for local validation.

By adhering to these defensive implications, both operators and vendors can collectively strengthen the software supply chain, making critical infrastructure more resilient against sophisticated cyber threats.

Key Takeaways

• Software updates present a fundamental paradox: essential for security and functionality in IT, but a major threat vector and operational risk in OT.
• Basic digital signatures and relying solely on HTTPS are insufficient for securing software updates against modern supply chain attacks.
• A mature Public Key Infrastructure (PKI), with documented policies, revocation processes, and key rollover plans, is crucial for robust update security.
• The Update Framework (TUF) provides a strong foundation against sophisticated attacks like rollback, freeze, and dependency compromise through its multi-key, multi-signature architecture.
• OT environments require an extension to frameworks like TUF, specifically a "deployer role" that facilitates local, on-premises validation and multi-signature approval before updates are deployed to critical systems.
• OT operators must adopt a "trust but verify" approach, implementing rigorous internal testing and validation for all software updates, even from trusted vendors.

About the Speaker(s)

Brian Romansky is a researcher at George Washington University, where he conducts work on next-generation software update solutions. He also represents Convolution Solutions, a company actively developing in this field. His research, including the work presented in this talk, is carried out in cooperation with his advisors, Dr. Thomas Mazuki and Sharam Sakani. Romansky's expertise lies in addressing the complex security challenges associated with software supply chains, particularly in critical infrastructure and Operational Technology (OT) environments.