RSAC Tech Talk - Open Source Security by Aeva Black

RSAC Tech Talk - Open Source Security by Aeva Black

Speakers: Aeva Black, Section Chief for Open Source Security, CISA

Conference: RSAC 2024

YouTube: https://www.youtube.com/watch?v=La_E69gCEpc

Overview

In this compelling RSAC 2024 presentation, Aeva Black, Section Chief for Open Source Security at the Cybersecurity and Infrastructure Security Agency (CISA), delivered a critical address on contextualizing open source software security within CISA's broader mission. The talk meticulously outlined the pervasive yet often invisible role of open source software (OSS) in modern critical infrastructure and federal government operations, highlighting the inherent risks stemming from its unique development model. Black underscored CISA's mandate to understand, manage, and reduce these risks, not only for the federal government and critical infrastructure but also as a responsible consumer of open source itself.

Black's presentation served as a stark reminder of the foundational dependence on OSS, illustrating its ubiquity with striking statistics: Black Duck's survey estimates that 96% of all enterprise codebases include open source, with an average of over 500 discrete components per product. This reliance extends across every sector and national critical function, from everyday devices like light bulbs and toasters to complex systems like the Mars Rover and critical security products. The talk emphasized that despite its immense value—a Harvard study estimated an $8 trillion demand-side value for OSS, excluding the Linux kernel due to its overwhelming presence—the security and sustainability of this ecosystem are often precariously balanced on the efforts of a few dedicated volunteers, creating a significant systemic risk.

The core message revolved around CISA's strategic roadmap to foster a secure, sustainable, and resilient open source ecosystem. This involves not only identifying and assessing risks but also actively hardening the ecosystem through coordinated efforts, community partnerships, and the adoption of modern security practices. Black detailed CISA's proactive measures, including the establishment of an Open Source Program Office (OSPO), the promotion of memory-safe programming languages, and the critical role of Software Bill of Materials (S-bombs) in enhancing supply chain transparency. The talk provided essential insights for both government agencies and private enterprises on navigating the complex landscape of open source security, advocating for a shift towards responsible consumption and sustainable contribution to the open source commons.

Background

▶ Watch: CISA's mission and the pervasive scale of open source (0:00)

The journey into understanding open source software security begins with its foundational definition, rooted in the 1976 Copyright Act. Open source software is characterized by four essential freedoms: the freedom to use it as desired, to study its source code, to modify the source code, and to redistribute it in original or modified forms. This broad definition encompasses everything from highly sophisticated commercial products to a college student's GitHub project, making it a diverse and often disparate landscape. The responsibility for safe usage, Black stressed, primarily rests with the consumer, not the producer, given the diverse nature of its creators.

Historically, the open source community, active since the 1980s, developed norms around transparency, often summarized as the "four opens": open source code, open design process (auditable by third parties), open development (including code review, build process, CI systems, and logs), and an open community for participants. This ethos led to the widely circulated aphorism, "given enough eyeballs, all bugs are shallow," suggesting that open source, due to its public nature, was inherently more trustworthy than proprietary alternatives. While this might have held true for large, well-resourced projects like the Linux kernel 25 years ago, Black argued that it is no longer universally applicable. The sheer volume of new code, particularly with the acceleration of AI-driven authorship tools like Co-pilot, far outstrips the human capacity for review. This leads to a contemporary inverse: "given a few eyes, it contains vulnerabilities."

A critical cultural difference highlights the core problem: open source projects and their communities often have "time but no money," relying heavily on volunteers, even if some are paid by industry but contribute in their spare time. Conversely, commercial products and the companies building them possess "money but no time," driven by market competition. This inherent tension is vividly illustrated by a now-famous XKCD comic, which became particularly resonant after the Log4Shell vulnerability. The comic depicts complex, critical infrastructure layers—representing projects like OpenSSL, Nginx, Curl, and LibreSSL—all supported by the tireless, often uncompensated, efforts of one or two individual volunteers. This dependence of modern digital infrastructure on unpaid individuals forms the heart of the systemic risk.

The historical trajectory saw early adopters like Google, Facebook, and Amazon gain a competitive advantage by securely utilizing open source and, critically, contributing back to the projects they depended on. However, this model of reciprocal support did not universally scale. The precariousness of the current situation became acutely evident with major incidents like Log4Shell and, more recently, the XZ Utils backdoor. The XZ incident, discovered just a month before the talk, involved a sophisticated, multi-year attack, potentially by a nation-state actor, targeting a small but foundational library. Its discovery by a single volunteer, not as part of their job, underscored both the fragility of the ecosystem and the inherent benefit of open source transparency when "eyes are on the project." These events highlight the urgent need for a more structured, resilient approach to open source security, moving beyond the reliance on individual altruism.

Key Findings

▶ Watch: Understanding open source and consumer's responsibility for safety (3:00)

CISA's engagement with open source security is directly aligned with its overarching mission to understand, manage, and reduce risks to federal government and critical infrastructure. Recognizing OSS as the invisible foundation for nearly all critical functions and businesses, CISA has established a clear roadmap with four primary goals: to establish its role in securing open source, identify and assess usage risks, reduce those identified risks, and ultimately harden the entire open source ecosystem.

To achieve these goals, CISA is implementing a multi-faceted strategy. First, it is coordinating its internal efforts across the agency and actively fostering partnerships with the broader open source community. This includes collaborating with centralized entities such as foundations (e.g., Linux Foundation's OpenSSF), non-profits, package managers, and coding platforms like GitHub and GitLab. A significant step in this direction was CISA's first summit with these stakeholders, held two months prior to the talk, which included a tabletop exercise to threat model attacks against open source. Black noted the fortuitous timing of this exercise, as the XZ Utils backdoor incident occurred just a month later, validating the foresight of their proactive planning.

Secondly, CISA is focused on identifying the prevalence of open source software across federal civilian networks and critical infrastructure sectors. This involves developing a robust framework for assessing systemic risk and prioritizing actions to mitigate those risks. As part of this effort, CISA is actively evaluating shared services that could be offered to other federal agencies to assist them in their own open source security endeavors. Furthermore, the agency is developing policies to encourage secure open source usage and contract fulfillment across multiple agencies. A concrete outcome of this initiative is the piloting of an Open Source Program Office (OSPO) within CISA, set to launch approximately a month after the talk. This OSPO will be the second within the federal government, following the lead of the Health and Human Services (HHS) OSPO, signifying a growing federal commitment to structured open source engagement.

Finally, CISA is committed to hardening the open source ecosystem as a whole through targeted interventions. A key focus is driving the adoption of memory-safe programming languages. Following a recent White House announcement on the subject, CISA, drawing on research by colleagues like Bob Lord, advocates for rewriting older code written in memory-unsafe languages (such as C/C++) into languages like Rust. While not a simple or free undertaking, this transition is deemed essential for significantly reducing the prevalence of memory-related vulnerabilities. Another critical initiative is the widespread adoption of Software Bill of Materials (S-bombs). These are considered vital for understanding open source supply chains, enabling organizations to quickly identify components and their dependencies, particularly in the wake of incidents like Log4Shell. CISA is also actively working with open source communities to enhance the vulnerability disclosure process, supporting projects like the Python Foundation and kernel.org in becoming CVE Numbering Authorities (CNAs). This empowers projects closer to the code to enrich and publish CVEs more effectively, leveraging their intimate knowledge of the software.

Technical Deep Dive

▶ Watch: Reframing 'enough eyeballs' for modern open source security (4:00)

The technical strategies CISA is championing to secure the open source ecosystem are multifaceted, addressing fundamental aspects of software development, supply chain visibility, and vulnerability management. A cornerstone of their approach is the aggressive promotion of memory-safe programming languages. As highlighted by the White House and CISA's own research, a significant percentage of critical vulnerabilities stem from memory safety issues in languages like C and C++. Black emphasized the long-term benefits of migrating legacy codebases from these memory-unsafe languages to modern alternatives such as Rust. While acknowledging that this is a non-trivial and costly endeavor, the reduction in potential vulnerabilities and the increased effort required to secure older code against memory exploits make it a worthwhile investment for critical infrastructure components. This shift is not just a recommendation but a strategic imperative to build more secure software from the ground up.

Another critical technical intervention is the widespread adoption and utilization of Software Bill of Materials (S-bombs). S-bombs provide a comprehensive, machine-readable inventory of all software components, including open source dependencies, within a given product. Black explicitly linked the necessity of S-bombs to incidents like Log4Shell, stating that they are "really critical for open source to help understand open source supply chains. And what is in a product that is open source, so the next time something like Log4Shell happens, you all can we all can know what's in the box." This transparency allows organizations to rapidly identify their exposure to newly discovered vulnerabilities, facilitating quicker and more targeted remediation efforts, thereby significantly reducing the response time and impact of supply chain attacks.

CISA is also actively engaging with open source communities to improve the vulnerability disclosure process, specifically by encouraging and supporting open source projects to become CVE Numbering Authorities (CNAs). This initiative empowers projects like the Python Foundation and kernel.org to directly assign, enrich, and publish CVEs (Common Vulnerabilities and Exposures) for vulnerabilities found within their own codebases. The rationale is that the project maintainers and core contributors possess the deepest understanding of their software, enabling more accurate and timely vulnerability reporting than external security researchers might achieve. This decentralization of the CVE process to the project level is intended to streamline disclosure, reduce latency, and improve the quality of vulnerability information available to the broader security community.

Furthermore, CISA, in conjunction with the Open Source Security Foundation's (OpenSSF) Securing Software Repositories working group, has developed and announced principles for package repository security. This framework outlines a set of best practices that any open source repository should follow to enhance its security posture. Black noted that five major package repositories have already committed to these principles, with others considering adoption. These principles aim to establish a baseline of security for the critical infrastructure of the open source supply chain—the repositories themselves—reducing risks such as package tampering, account compromise, and malicious package injection. As part of this collaboration, CISA's inaugural open source security summit also established a dedicated channel for real-time information sharing. This channel is designed to facilitate rapid communication and coordinated response should open source projects come under direct threat or if a vulnerability requires specialized handling beyond the capacity of an individual project or community.

Demo / Proof of Concept

▶ Watch: XKCD comic illustrates critical infrastructure relying on volunteers (5:00)

The presentation by Aeva Black was a strategic and informational overview of CISA's initiatives in open source security, rather than a technical demonstration. No live demo or proof of concept was presented during the talk. The focus remained on outlining CISA's mission, identifying the systemic risks, and detailing the agency's roadmap and collaborative efforts to secure the open source ecosystem.

Defensive Implications

▶ Watch: Log4Shell and XZ vulnerabilities highlight precarious open source security (6:40)

For organizations consuming open source software, the defensive implications of CISA's insights are profound and necessitate a proactive, multi-pronged approach. First and foremost, companies must transition from passive consumers to responsible consumers of open source. This entails establishing robust internal processes for ingesting, staging, reviewing, and continuously scanning open source components. Black emphasized the importance of verifying the provenance and security of open source dependencies before integrating them into products or systems. This initial vetting process is crucial for mitigating risks introduced by malicious or vulnerable components.

Beyond internal processes, responsible consumption extends to sustainable contribution. Companies heavily reliant on open source have a direct business risk if the underlying projects are not sustained. To mitigate this, organizations should actively contribute back to the open source commons. This contribution can take various forms: time, where employees dedicate work hours to contribute code, documentation, or project management to open source projects; treasure, involving direct financial donations to foundations supporting critical projects; or talent, where companies donate in-kind resources or services (e.g., infrastructure, storage, tooling) to help communities thrive. This reciprocal relationship is vital for ensuring the long-term health and security of the open source ecosystem that underpins their own operations.

Furthermore, organizations should prioritize the adoption of specific technical controls and practices:

• Implement Software Bill of Materials (S-bombs): S-bombs are no longer optional but a critical tool for supply chain transparency. Companies should generate and maintain S-bombs for all their products, enabling rapid identification of affected components during a vulnerability disclosure event (e.g., Log4Shell). This visibility is key to efficient incident response and risk management.
• Embrace Memory-Safe Programming Languages: For new development and strategic refactoring of critical components, organizations should prioritize the use of memory-safe languages like Rust. For existing codebases written in languages prone to memory vulnerabilities (e.g., C/C++), a strategic assessment should be undertaken to identify high-risk areas for potential rewriting, significantly reducing a common class of exploits.
• Engage with Vulnerability Disclosure Processes: Companies should support and, where applicable, participate in efforts to improve vulnerability disclosure, including encouraging open source projects they rely on to become CVE Numbering Authorities (CNAs). This fosters a more efficient and accurate flow of vulnerability information, benefiting the entire ecosystem.
• Leverage CISA Resources and Guidance: As CISA develops frameworks for systemic risk assessment, shared services, and policy guidance for secure open source usage, organizations should actively monitor and integrate these recommendations into their security strategies. Participating in CISA-led initiatives, such as tabletop exercises or information-sharing channels, can provide invaluable insights and collaborative defense opportunities.

In essence, the defensive posture must shift from merely patching vulnerabilities to proactively building a more secure and sustainable open source supply chain, recognizing that the security of one's own products is inextricably linked to the health of the open source projects they depend upon.

Key Takeaways

• Ubiquitous Dependency: Open source software forms the invisible foundation of 96% of enterprise codebases and virtually all critical infrastructure, with an estimated $8 trillion demand-side value.
• Systemic Risk from Volunteer Reliance: Modern digital infrastructure precariously depends on the uncompensated efforts of a few volunteers, creating significant systemic risk, as highlighted by incidents like Log4Shell and XZ Utils.
• CISA's Strategic Role: CISA is actively establishing its role in securing the open source ecosystem by identifying risks, developing assessment frameworks, piloting an OSPO, and fostering broad community partnerships.
• Memory Safety is Critical: A key hardening strategy involves promoting the adoption of memory-safe programming languages (e.g., Rust) to mitigate a vast class of common vulnerabilities found in older codebases.
• S-bombs for Supply Chain Transparency: Software Bill of Materials (S-bombs) are essential for understanding open source dependencies and enabling rapid response to supply chain vulnerabilities.
• Responsible Consumption and Sustainable Contribution: Organizations must act as responsible consumers by vetting open source components and, crucially, contribute back to the projects they rely on through time, treasure, or talent to ensure the ecosystem's sustainability.

About the Speaker(s)

Aeva Black is the Section Chief for Open Source Security at the Cybersecurity and Infrastructure Security Agency (CISA). In this role, she leads CISA's efforts to understand, manage, and reduce risks associated with open source software across federal government networks and critical infrastructure. Her work involves coordinating agency-wide initiatives, partnering with the open source community, and developing strategic frameworks to enhance the security and resilience of the open source ecosystem.