The National Vulnerability Database (NVD) – Where It Is and Where It’s Going

The National Vulnerability Database (NVD) – Where It Is and Where It’s Going

Speakers: Tanya Brewer, Program Manager, NVD; Matt Schul, Oversees security standards, metrics and measurements, NIST

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=BLu9ebR88uQ

Overview

The National Vulnerability Database (NVD), maintained by the National Institute of Standards and Technology (NIST) under the Department of Commerce, serves as the United States government's repository of standards-based vulnerability management data. This talk at VulnCon, delivered by NVD Program Manager Tanya Brewer and NIST's Matt Schul, provided a crucial update on the NVD's current state, recent operational overhauls, and strategic direction. It addressed significant challenges faced in the past year, including a processing "pause" and a surge in vulnerability disclosures, outlining how NIST is adapting to ensure the NVD remains a reliable and efficient resource for the global security community.

The presentation underscored the NVD's critical role in the cybersecurity ecosystem, serving not only U.S. government agencies but also an international audience spanning approximately 160 countries monthly. It detailed extensive internal restructuring, technological upgrades, and forward-looking initiatives aimed at improving data quality, processing speed, and community engagement. The speakers emphasized NIST's reaffirmed commitment to the NVD as a mission priority, highlighting efforts to enhance its foundational Common Platform Enumeration (CPE) specifications and explore advanced automation techniques, including artificial intelligence and machine learning, to meet the escalating demands of vulnerability management.

Ultimately, this talk provided transparency into the NVD's operational challenges and strategic vision, reassuring stakeholders of its continued evolution. It laid out a roadmap for significant improvements in data ingestion, enrichment, and accessibility, demonstrating NIST's proactive approach to maintaining the NVD's relevance and utility in a rapidly changing threat landscape. The proposed changes, particularly the overhaul of CPE specifications and the move towards greater automation, are poised to have a profound impact on how vulnerabilities are identified, categorized, and managed across the industry.

Background

▶ Watch: Introduction of NVD program manager and NIST oversight (0:00)

The NVD is a cornerstone of global cybersecurity, providing a comprehensive database of Common Vulnerabilities and Exposures (CVEs). Its primary mission is to enrich CVE records with additional context, such as Common Platform Enumeration (CPE) for affected products, Common Vulnerability Scoring System (CVSS) scores for severity, and Common Weakness Enumeration (CWE) for vulnerability types. This enriched data is vital for a wide array of users, from security scanning tool manufacturers to government agencies and individual researchers, enabling them to accurately identify, assess, and mitigate software vulnerabilities. The NVD's data sources primarily include the official CVE List, but also incorporates translations from Incay and critical Known Exploited Vulnerability (KEV) data from CISA's catalog, which helps prioritize processing.

The NVD has faced significant challenges in recent times, prompting the need for the updates discussed in this talk. In early 2023, the NVD experienced an operational "pause" that led to a substantial backlog in processing new CVEs. This disruption, coupled with an alarming 32% increase in incoming CVEs in 2024 compared to 2023, placed immense pressure on the NVD's capabilities. Compounding these issues was the recognition that the existing CPE specifications were "woefully out of date," hindering accurate product identification and vulnerability mapping. Furthermore, the NVD's internal systems, including its API and search functionalities, required modernization to handle the growing volume and complexity of data, and to better integrate with the evolving vulnerability disclosure ecosystem, including the rise of Authored Data Publishers (ADPs). These factors necessitated a strategic re-evaluation and a comprehensive overhaul of the NVD's operational model and underlying infrastructure.

Key Findings

▶ Watch: Decision against CRADAs for more flexible community engagement (2:40)

The talk revealed several critical updates and strategic shifts for the NVD, demonstrating NIST's commitment to addressing past challenges and future demands:

• Reaffirmed Mission Priority: NIST leadership has unequivocally reaffirmed the NVD as a mission priority, ensuring continued resourcing, staffing, and capability growth.
• Team Reconstitution and System Overhauls: Following a "pause" in early 2023, the NVD successfully established entirely new enrichment and development teams. Concurrently, internal systems were updated and upgraded to fully ingest and utilize ADP data, a process that required two attempts to ensure robust future compatibility.
• Enhanced API Performance and Stability: A significant overhaul of the NVD's APIs addressed previous issues, including frequent 503 errors, resulting in the "best performance and stability" the APIs have ever achieved.
• CPE Automation and Specification Overhaul: A proof of concept (PoC) tool was developed for automating CPE creation using applicability statements. More critically, a major project has commenced to overhaul the outdated CPE specifications, involving extensive community outreach and a virtual workshop to gather consensus.
• Temporary Gap Filling for Backlog Reduction: To accelerate the processing of the CVE backlog, the NVD will temporarily adopt a "gap filling" approach. If a CNA (CVE Numbering Authority) provides CVSS or CWE scores, the NVD will utilize these and only fill in missing data, rather than creating its own from scratch. This is expected to significantly speed up throughput.
• Upcoming User-Facing Improvements: An overhaul of the NVD search engine is planned within 2-3 months, introducing advanced search capabilities by CNA and ADP. Later in the year, the Fonttology (a tool for vulnerability context and relationships) will be released for public testing and feedback, alongside an externally facing CPE console aimed at allowing organizations to manage their own CPEs.
• Exploration of AI/ML for Enrichment: A dedicated NIST team will explore the application of Artificial Intelligence (AI) and Machine Learning (ML) to improve and expedite the vulnerability enrichment process.
• Shift from Formal Consortia to Open Engagement: While initial plans involved formal Cooperative Research and Development Agreements (CRADAs) or consortia for external engagement, NIST determined these mechanisms were too heavy. The NVD will instead pursue a more flexible, open, and informal approach to stakeholder collaboration through workshops, conferences, and direct outreach.

Technical Deep Dive

▶ Watch: NIST leadership reaffirms NVD as a critical mission priority (5:50)

The technical advancements and strategic shifts outlined for the NVD are substantial, focusing on data ingestion, processing efficiency, and the foundational elements of vulnerability description.

One of the most critical developments is the NVD's ability to fully ingest and utilize Authored Data Publisher (ADP) data. This involved a two-phase internal system upgrade. Initially, the focus was on consuming enrichment data directly from CISA in the aftermath of the previous year's operational hiccup. However, recognizing the broader trend of ADPs, NIST pivoted to ensure their systems could universally handle ADP data from any source, including the CVE ADP. This move is crucial for integrating diverse data streams and reducing the NVD's sole reliance on manual enrichment for all CVEs. By leveraging pre-enriched data from authoritative sources, the NVD can significantly enhance the speed and coverage of its database.

Central to the NVD's data quality and automation efforts is the overhaul of Common Platform Enumeration (CPE). The current CPE specifications are acknowledged as "woefully out of date" and unrealistic for modern usage. NIST is embarking on a major project to update these specifications, initiating a community-wide workshop with a virtual component to ensure broad participation and consensus. The goal is to create more robust and realistic CPEs, which are fundamental for accurate product identification in vulnerability scanning and asset management. Complementing this, a proof of concept (PoC) tool has been developed to automate CPE creation. This tool is designed to leverage CPE applicability statements that are expected to be integrated into the CVE list. Automating CPE generation will dramatically reduce manual effort and improve consistency, leading to more reliable vulnerability matching.

Internal operational efficiency is being addressed through significant overhauls of the NVD's consoles. The internal Vong console, used by the enrichment team, has undergone a "significant overhaul," leading to "a lot of efficiencies" and increased "flow through." This improvement directly impacts the speed at which CVEs are processed and enriched. A similar overhaul is planned for the internal CPE console, which is expected to yield further efficiencies for the enrichment team. Critically, this internal improvement will pave the way for an externally facing CPE console later this year. The long-term goal for this external console is to enable organizations, particularly CNAs, to directly manage and submit their own CPEs into the CPE dictionary. This distributed management approach could drastically improve the accuracy and timeliness of CPE data.

The NVD's public-facing interfaces have also received significant attention. The NVD APIs, which previously suffered from "a lot of problems" and "503 errors," have undergone an overhaul, resulting in "the best performance and stability" they've ever had. This stability is vital for tool manufacturers and other automated consumers of NVD data. Furthermore, the search engine is being overhauled to include an "advanced button" that will allow users to search by CNA and ADP, greatly improving the discoverability of specific vulnerability information.

Looking ahead, the NVD is actively developing its Fonttology, a framework for understanding vulnerability context and relationships. A public announcement and a user interface for the Fonttology are expected by summer, inviting community feedback to ensure its usability and completeness. This initiative aims to provide a richer, more interconnected understanding of vulnerabilities beyond simple enumerations.

To tackle the existing backlog and future influx of CVEs, the NVD is implementing a temporary "gap filling" strategy. Historically, the NVD would generate its own CVSS and CWE scores. Under the new approach, if a CNA provides these scores in a CVE record, the NVD will utilize them and only fill in any missing information, rather than re-evaluating or re-creating all enrichment data. This strategic shift, effective from late April/early May, is designed to expedite the processing pipeline.

Finally, NIST is making a concerted effort to explore advanced automation. A proof of concept tool is under development specifically for automating the enrichment of Linux kernel CVEs, an area known for its volume and complexity. Beyond this, a dedicated NIST team will lead the charge in investigating how Artificial Intelligence (AI) and Machine Learning (ML) can be leveraged to "improve our enrichment process and make it faster." This signals a long-term vision for transforming the NVD's operational model from primarily manual to increasingly AI-driven.

Demo / Proof of Concept

▶ Watch: New NVD enrichment and development teams now fully operational (7:00)

While the talk did not feature live demonstrations of new tools, it highlighted two significant proof of concept (PoC) initiatives that represent the NVD's future direction in automation and efficiency.

Firstly, a PoC tool has been developed for automating CPE creation. This tool is designed to leverage CPE applicability statements, which are anticipated to be integrated into the CVE list. The vision is that these statements, embedded within CVE records by CNAs, will allow the NVD's tool to automatically generate accurate CPE strings. This would be a significant leap forward, as manual CPE creation is a labor-intensive and error-prone process. The PoC demonstrates the feasibility of this automation, suggesting a future where the initial, manual burden of identifying affected products can be substantially reduced, leading to faster and more consistent enrichment.

Secondly, a separate PoC tool is under development to automate the enrichment process specifically for Linux kernel CVEs. The Linux kernel is a highly complex and frequently updated software component, resulting in a large volume of associated CVEs. Automating the enrichment of these particular vulnerabilities, which often require deep technical understanding to correctly categorize and score, would significantly alleviate the NVD's manual workload. The success of this PoC could pave the way for similar automated enrichment efforts across other high-volume or complex software ecosystems, demonstrating the potential for AI and machine learning to revolutionize the NVD's operational model.

These PoCs, though not extensively detailed in their internal workings during the talk, illustrate NIST's commitment to leveraging technology to address the escalating volume of vulnerabilities and enhance the NVD's processing capabilities. They represent tangible steps towards a more automated and efficient future for vulnerability data enrichment.

Defensive Implications

▶ Watch: Internal systems upgraded for ADP data and CPE automation (8:40)

The NVD's ongoing evolution has significant defensive implications for security professionals, tool vendors, and organizations worldwide. The improvements discussed aim to enhance the timeliness, accuracy, and accessibility of vulnerability data, directly benefiting those tasked with identifying, assessing, and mitigating risks.

Firstly, the strategic shift towards ingesting and utilizing ADP data and the temporary gap filling mechanism directly contribute to a faster processing of CVEs. This means that enriched vulnerability data, including CVSS scores and CWEs, will become available more quickly after a CVE is published. For defenders, this translates to reduced windows of exposure, as they can integrate more complete vulnerability information into their security tools and processes sooner. Early access to accurate severity ratings and vulnerability types allows for more informed prioritization of patching and mitigation efforts.

The planned overhaul of CPE specifications is perhaps one of the most impactful changes for defenders. Outdated CPEs hinder accurate asset inventory and vulnerability mapping. Modernized CPEs will enable security tools (e.g., vulnerability scanners, asset management systems) to more precisely identify affected software and hardware versions. This improved accuracy will reduce false positives and false negatives, leading to more reliable vulnerability assessments and more effective patch management strategies. The long-term goal of an externally facing CPE console will further empower organizations to contribute and manage their own CPEs, fostering a more current and comprehensive CPE dictionary.

Improvements to the NVD APIs and the search engine overhaul (including search by CNA and ADP) are crucial for automated and manual data consumption. Stable, high-performing APIs ensure that security tools relying on NVD data for their vulnerability intelligence feeds operate reliably. Enhanced search capabilities will allow security analysts and researchers to more efficiently pinpoint specific vulnerabilities, understand their context, and track disclosures from particular CNAs or ADPs, streamlining incident response and threat intelligence gathering.

The exploration of AI and machine learning for enrichment and the PoC for automating Linux kernel CVE enrichment signal a future where the NVD can scale its operations to meet the ever-increasing volume of vulnerabilities without sacrificing quality. For defenders, this means a sustained supply of high-quality, enriched vulnerability data, even as the threat landscape expands. This proactive approach ensures the NVD remains a relevant and essential resource for situational awareness and risk management.

Finally, the NVD's reaffirmed status as a NIST mission priority, coupled with its commitment to open engagement through workshops and direct outreach, provides assurance to the defensive community. It signifies that NIST is dedicated to evolving the NVD in alignment with stakeholder needs, ensuring that the database continues to serve as a foundational element for national and international cybersecurity defense strategies. Defenders should actively participate in these community discussions, particularly regarding CPE specifications, to ensure their operational needs are reflected in future standards.

Key Takeaways

• The NVD is undergoing a significant strategic and operational overhaul, driven by a past processing "pause" and a 32% increase in CVEs in 2024.
• NIST has reaffirmed the NVD as a mission priority, ensuring continued investment in staffing and capabilities.
• New teams, internal system upgrades, and API overhauls have drastically improved the NVD's ability to ingest ADP data and deliver stable services.
• A major project is underway to overhaul the outdated CPE specifications, with community input via workshops, to improve the accuracy of vulnerability identification.
• To address the backlog, the NVD is temporarily implementing "gap filling" using CNA-provided CVSS/CWE data to expedite processing.
• Future enhancements include an overhauled search engine (with CNA/ADP search), a public Fonttology release, an external CPE console for organizational management, and the exploration of AI/ML for automated enrichment.

About the Speaker(s)

Tanya Brewer is the current Program Manager of the National Vulnerability Database (NVD) at the National Institute of Standards and Technology (NIST). She plays a pivotal role in overseeing the NVD's operations, strategic direction, and its ongoing modernization efforts. Her leadership is crucial in navigating the NVD through significant challenges and implementing new initiatives to enhance its service to the global cybersecurity community.

Matt Schul is a representative of NIST and oversees security standards, metrics, and measurements for the U.S. government, with the NVD being one of his key responsibilities. He brings a strategic and bureaucratic perspective to the NVD's development, focusing on long-term planning, resource allocation, and ensuring that the NVD aligns with NIST's broader mission as the national metrology institute. His role involves ensuring the NVD continues to provide essential measurement reference data for software and software security.