Sharing cyber threat intelligence: Does it really help?

Sharing cyber threat intelligence: Does it really help?

Speakers: Beomjin Jin

Conference: NDSS Symposium

YouTube: (no public video)

Overview

In an era of relentlessly escalating cyber threats, the effective sharing of Cyber Threat Intelligence (CTI) is paramount for robust security postures. Beomjin Jin's talk, "Sharing cyber threat intelligence: Does it really help?", delves into a critical examination of this practice, specifically focusing on the Structured Threat Information eXpression (STIX) standard. The research addresses a significant gap in understanding how high-level CTI, beyond basic Indicators of Compromise (IoCs), is actually disseminated and utilized in the wild.

The presentation introduces CTI-Lense, a novel framework designed for a comprehensive, large-scale empirical analysis of STIX data sharing. This framework systematically evaluates STIX data across four crucial dimensions: volume, timeliness, coverage, and quality. By collecting and analyzing nearly a decade of publicly available STIX data, the work aims to uncover the efficacy and limitations of current CTI sharing mechanisms, providing essential insights for both producers and consumers of threat intelligence.

This article details the methodologies, key findings, and implications of Jin's research, highlighting the challenges that hinder the effective utilization of STIX and proposing concrete recommendations for enhancing its role in global cybersecurity defense. The findings underscore that while STIX adoption is growing, significant improvements are needed in the consistency, completeness, and promptness of shared intelligence to truly help organizations counter sophisticated cyber attacks.

Background

[▶ Watch: Introduction, motivation, and the problem with CTI sharing (0:00)]()

Cyber Threat Intelligence (CTI) serves as evidence-based knowledge that empowers security analysts to make informed decisions regarding cybersecurity threats. It encompasses a spectrum of information, ranging from low-level Indicators of Compromise (IoCs) like malware hashes and IP addresses, to high-level strategic intelligence such as Tactics, Techniques, and Procedures (TTPs), threat actors, and campaigns. While much CTI initially exists in unstructured formats—like narrative reports or blog posts—its utility for automated decision-making is limited without significant preprocessing, including deduplication and structurization.

To address this challenge, standardized, machine-readable formats for CTI have emerged, with Structured Threat Information eXpression (STIX) becoming the de facto standard. Developed through a collaboration between the US Department of Homeland Security (DHS) and MITRE, STIX provides a common language for expressing diverse cyber threat information. The STIX ecosystem involves standardization bodies (like OASIS), producers (security analysts generating STIX data), service providers (managing repositories or TAXII servers), and consumers (fetching and utilizing STIX data). STIX data comprises various objects and attributes, including observables, indicators, incidents, TTPs, exploit targets, courses of action, campaigns, and threat actors, allowing for a rich representation of threat landscapes.

The evolution of STIX is noteworthy. STIX 1, introduced in 2012, utilized an XML format, often leading to complex, nested structures and redundant representations. To overcome these limitations, STIX 2 was standardized by OASIS in 2017, adopting a more lightweight, programmer-friendly JSON format and a flatter, relational data model. Despite the ecosystem's transition towards STIX 2.x (with version 2.1 released in March 2020), many providers continue to share STIX 1 data, necessitating analysis of both versions. Complementing STIX, the Trusted Automated eXchange of Indicator Information (TAXII) protocol facilitates the automated exchange of STIX data over HTTPS, defining APIs for sharing between producers and consumers. A crucial limitation is that TAXII does not support backward compatibility between STIX 1 and STIX 2.

Previous research into CTI has predominantly focused on quantitative evaluations of basic IoC data types, such as IP addresses and malware hashes, assessing their coverage, accuracy, and timeliness. While these studies have highlighted the utility of public CTI blocklists against specific attack vectors like phishing, they often overlooked advanced threat data like TTPs or threat actors, and rarely examined the actual sharing patterns of STIX objects. Efforts to automate CTI extraction from unstructured data, such as CyTIME, CTIMiner, and INTIME, have also largely concentrated on simple IoC information or achieved limited F1 scores for more advanced data. This talk addresses this critical knowledge gap by providing a large-scale empirical analysis of the sharing of STIX data, including both basic IoCs and high-level threat information, across its volume, timeliness, coverage, and quality.

Key Findings

[▶ Watch: Defining Cyber Threat Intelligence (CTI) and STIX (2:09)]()

The comprehensive analysis conducted using the CTI-Lense framework yielded several critical findings regarding the current state of STIX data sharing:

1. Insufficient Volume of Unique Data: While STIX data sharing has shown a steady increase since 2016, indicating growing adoption, the actual volume of unique STIX objects shared daily remains critically low, averaging just 2,063 objects. This quantity is vastly insufficient to cover the estimated 450,000 new malware samples and potentially unwanted applications reported daily by sources like the AV-TEST Institute. The analysis also revealed substantial intra-source duplication (e.g., JamesBrine showed a 68.74% duplication rate), though inter-source duplication across different providers was relatively low at 4.10%. This suggests that while individual providers frequently share redundant data, the unique content they provide is less likely to be replicated by others.

1. Variable Timeliness of Sharing: The promptness of STIX data sharing varies significantly by object type. For URLs, STIX data is disseminated quite promptly, with approximately 72% shared either before or on the same day as their appearance on VirusTotal, and 88% before or on the same day as HybridAnalysis. This makes STIX effective for detecting new attack websites. However, the sharing of file hashes (malware signatures) is notably slower than commercial scanning services like VirusTotal and MetaDefender. This delay severely limits STIX's efficacy in preventing malware attacks, especially given that many malware samples have very short active lifespans (e.g., 24 hours). Overall, STIX data is generally shared within 2-4 days after a security incident is reported. The study found weak evidence of a causal relationship between security incidents reported on Malpedia and security news and STIX data sharing, but no significant evidence for CVE publications.

1. Underutilization of STIX Standard's Expressive Power: The STIX standard, with its rich set of objects and attributes, is significantly underutilized in practice. Only 75% of the defined object types are used in both STIX 1 (6 out of 8) and STIX 2 (15 out of 20) datasets. The Indicator object type overwhelmingly dominates, accounting for over 90% of the data in both STIX 1 (98.77%) and STIX 2 (94.93%) datasets. This dominance means that STIX data is largely focused on simple IoCs. A critical finding is that only a mere 0.09% of STIX 1 Indicator objects contain Test_Mechanisms (e.g., Snort rules) for automated detection, severely hindering the ability to automatically detect and respond to cyber attacks based on shared intelligence.

1. Significant Quality Issues: STIX data quality suffers from both incorrect values and improper usage of attributes.

• Correctness: 19% of Threat actor objects contained incorrect information, such as TTP or malware names instead of actual threat actor names. Furthermore, a substantial portion of malware family information (62% in STIX 1 TTPs and 58% in STIX 2 Malware objects) did not match entries in the ground truth dataset, often due to producers' custom naming conventions or misspellings. While FileObjectType and URIObjectType data generally exhibited high accuracy (over 90% when validated against VirusTotal), AddressObjectType and DomainNameObjectType data showed low detection rates (below 50.62%), indicating lower reliability. Producer identity was found to correlate with data accuracy, with known producers like phishtank.com demonstrating very high accuracy (99.79% for URIObjectType).
• Completeness/Usage: Producers frequently describe specific threat information in narrative forms within generic attributes like Description, rather than utilizing the precise, structured STIX objects and attributes designed for that purpose. For instance, malware names and threat actors were found embedded in Description attributes instead of being properly structured within TTP or Threat Actor objects. This "improper usage" undermines machine readability and the primary goal of STIX, forcing security analysts to manually extract information, which is time-consuming and prone to human error.

In essence, the study concludes that while STIX data sharing is increasing, its current implementation falls short in volume, timeliness for critical threat types, comprehensive utilization of the standard, and overall data quality, thereby limiting its potential to effectively counter the rapidly evolving cyber threat landscape.

Technical Deep Dive

[▶ Watch: Overview of key STIX objects and attributes (3:30)]()

To conduct this extensive analysis, the researchers developed CTI-Lense, a comprehensive framework designed to aggregate, process, and analyze STIX data from multiple open sources across four dimensions: volume, timeliness, coverage, and quality. The architecture of CTI-Lense is modular, comprising several key components:

1. CTI Collector: This module is responsible for gathering raw CTI data from various external sources.

• STIX data: Collected in both XML (STIX 1.x) and JSON (STIX 2.x) formats from publicly available TAXII servers and repositories. This formed the primary dataset for the study.
• Incident data: Text-based data from sources like CVE, Malpedia, and security news websites, including titles, descriptions, and publication times, which were crucial for timeliness analysis.
• APT report data: Textual reports from security companies such as Kaspersky and TrendMicro, providing insights into malicious IoCs, threat actors, and malware types for quality assessment.
• AV report data: JSON-formatted scan results and first submission times from popular online scanning services like VirusTotal, MetaDefender, and HybridAnalysis, used for both timeliness and quality validation of observables.

1. Preprocessor: This component handles the initial preparation and standardization of the collected data. A critical function was the conversion of all STIX 1 XML data into JSON format using the python-stix library, ensuring consistency across the dataset for easier processing. It also normalized Observable objects, ensuring they existed as independent entities with unique IDs, which was necessary to resolve variations in how references are handled across different STIX versions and sources.

1. Data Manager: After preprocessing, the CTI data is stored in a MongoDB database, providing a flexible and scalable storage solution for the large volume of diverse data.

1. CTI Analyzer: This is the core analytical engine, equipped with specialized sub-analyzers tailored for each research question:

• Volume Analyzer: Quantified the daily and monthly data volume, as well as the proportion of duplicated data both within and across sources.
• Coverage Analyzer: Assessed the usage patterns of STIX objects and attributes, tracking monthly trends to understand the extent to which the STIX standard's full potential is being leveraged.
• Timeliness Analyzer: Compared STIX data release times with security incidents from Malpedia and CVEs, and with submission times to scanning services like VirusTotal, HybridAnalysis, and MetaDefender. This involved Granger causality analysis to determine predictive relationships between incident reports and STIX sharing.
• Quality Analyzer: Identified "improper values" by validating attributes like Threat actor, TTP, and Malware against ground truth datasets, and detected "improper usage" where information was described narratively instead of using precise STIX attributes.

Implementation Details and Dataset:

The research collected data over an extensive period of nearly nine years, from October 31, 2014, to April 10, 2023. Initially identifying 84 STIX-related services, the team filtered down to 10 active, publicly accessible sources (e.g., Hail a TAXII, AlienVault OTX, IBM X-Force Exchange for STIX 1; JamesBrine, DigitalSide, MITRE ATT&CK for STIX 2). A total of 10,392,889 raw STIX objects were collected.

A rigorous, two-step deduplication process was crucial:

1. Objects with identical identifiers from the same source were removed.
2. Duplicate objects based on unique attribute values (e.g., malware hashes, IP addresses) were eliminated across different sources.

This process reduced the dataset to 6,362,065 unique STIX objects, comprising 3,894,595 STIX 1 objects and 2,467,470 STIX 2 objects.

For ground truth evaluation of data quality:

• Threat actor and malware family names were validated against Malpedia, MITRE ATT&CK, and MISP GitHub, yielding 2,111 threat actor names and 4,071 malware family names.
• Observable attributes (malware hashes, domains, URLs, IP addresses) were validated using scanning reports from VirusTotal (70,997 file hashes, 374,973 URLs), HybridAnalysis (14,924 domains, 10,179 file hashes, 836 IPs, 8,055 URLs), and MetaDefender (31,290 file hashes).

The experiments were conducted on a virtual machine with at least 4 cores and 32 GiB memory, and the entire framework, including dependencies and dataset, is made available as a Docker image (jinbumjin/cti-lense:artifactv1.1) to ensure reproducibility of the results.

Demo / Proof of Concept

[▶ Watch: Differentiating between STIX 1 and STIX 2 versions (3:45)]()

While the talk did not feature a live, interactive demonstration in the traditional sense, the researchers provided a robust and reproducible artifact that serves as a proof of concept for their entire analytical framework, CTI-Lense. This artifact, detailed in Appendix A of their paper and mentioned in the presentation, is a Docker image (jinbumjin/cti-lense:artifactv1.1) containing all necessary software dependencies, source code, and the dataset used for their analysis.

This demonstration of reproducibility allows other researchers and interested parties to independently verify the findings and explore the dataset and methodologies. Users can install Docker, pull the provided image, start the MongoDB service within the container, and then execute the CTI_Lense.py script with specific arguments (e.g., python3 CTI_Lense.py -e timeliness) to reproduce the results for timeliness analysis or any other dimension investigated in the study. This commitment to open science and reproducibility is a crucial aspect of validating the technical integrity and reliability of the research, effectively demonstrating the operational capability of the CTI-Lense framework.

Defensive Implications

[▶ Watch: Unique contribution compared to related CTI research (4:20)]()

The findings from this comprehensive analysis of STIX data sharing carry significant implications for cybersecurity defenders, highlighting areas where current practices fall short and suggesting pathways for improvement.

Firstly, defenders should acknowledge the inherent limitations in the volume and timeliness of publicly shared STIX data. With only an average of 2,063 unique STIX objects shared daily against hundreds of thousands of new threats, relying solely on public STIX feeds for comprehensive threat coverage, especially for rapidly evolving malware, is insufficient. For file hashes and IP addresses, STIX data is often generated too slowly to be effective against malware with short lifespans. Defenders must therefore complement STIX feeds with faster, more dynamic IoC sources, such as commercial threat intelligence platforms, honeypots, or internal detection systems that can identify new malware signatures and malicious IPs more rapidly. However, for URL and URI-based threats, STIX appears to be a more timely and effective source, suggesting its utility in defending against phishing or malicious domain activity.

Secondly, the study reveals a critical underutilization of the STIX standard's expressive capabilities, particularly the scarcity of Test_Mechanisms (e.g., Snort rules) in Indicator objects (only 0.09%). This means that STIX, as currently shared, offers limited direct support for automated detection and response systems. Defenders should not expect robust, ready-to-deploy security rules directly from public STIX feeds. Instead, they will likely need to manually develop or integrate their own detection rules based on the IoCs provided, or leverage other intelligence sources specifically designed for rule generation.

Thirdly, the quality issues – incorrect values in threat actor and malware names, and the prevalent use of narrative descriptions instead of structured attributes – pose significant challenges. Defenders consuming STIX data must implement robust validation and parsing mechanisms. This includes:

• Data sanitization and normalization: Developing internal processes to cross-reference threat actor and malware names against trusted internal or commercial ground truth datasets to correct misspellings or inconsistent naming conventions.
• Advanced text processing: Employing natural language processing (NLP) techniques to extract structured information from narrative Description fields, although this adds complexity and potential for error. This manual or semi-automated extraction undermines the primary benefit of STIX as a machine-readable format.
• Producer accountability: Prioritizing STIX data from known, reputable producers (e.g., phishtank.com showed high accuracy for URI objects) and exercising caution with data from less reliable or anonymous sources. Defenders should consider implementing internal systems to track the historical accuracy and reliability of different STIX producers.

Finally, the high intra-source duplication observed (e.g., 68.74% for JamesBrine) necessitates robust deduplication processes on the consumer side. While inter-source duplication is low, internal redundancy from single providers can inflate data volume without adding unique intelligence, wasting storage and processing resources.

In summary, defenders should approach public STIX data with a critical eye, understanding its strengths (URL timeliness, structured format potential) and weaknesses (volume, malware timeliness, rule scarcity, quality inconsistencies). It is a valuable piece of the CTI puzzle but requires significant supplementary intelligence, internal processing, and validation to be truly effective in a proactive defense strategy.

Key Takeaways

• Insufficient Unique Volume: Despite increasing adoption, the daily volume of unique STIX objects (average 2,063) is critically low and inadequate to cover the vast number of new cyber threats emerging daily.
• Variable Timeliness: STIX data for URLs is often shared promptly (72% before/same day as VirusTotal), making it effective for web-based threats, but sharing of malware hashes and IP addresses is significantly slower, limiting its utility for rapid malware prevention.
• Underutilized Standard: The STIX standard's expressive power is largely untapped; only 75% of object types are used, and Indicator objects dominate. Crucially, less than 0.1% of Indicator objects include security rules, hindering automated threat detection.
• Significant Quality Issues: STIX data frequently contains incorrect values (e.g., 19% of threat actor data, 62% of STIX 1 malware names mismatch ground truth) and relies on narrative descriptions instead of precise, structured attributes, undermining machine readability and automation.
• Producer Accountability Matters: The accuracy of STIX data, particularly for observables, correlates with the identity of the producer, emphasizing the need for trust and validation in CTI sourcing.
• Recommendations for Improvement: To enhance STIX effectiveness, the community needs a common, standardized vocabulary for CTI, comprehensive training for producers, automated tools for verification and deduplication, and mechanisms to promote producer accountability.

About the Speaker(s)

Beomjin Jin is the speaker for this presentation, "Sharing cyber threat intelligence: Does it really help?". The transcript and metadata provided do not contain further biographical details about Beomjin Jin's title or affiliation.