Using AI Computer Vision in Your OSINT Data Analysis

Using AI Computer Vision in Your OSINT Data Analysis

Speakers: Mike Raggo

Conference: DEF CON 32

YouTube: https://www.youtube.com/watch?v=TK3A510jArc

Overview

In this DEF CON 32 talk, Mike Raggo, a seasoned veteran of the information security community, delves into the transformative potential of artificial intelligence (AI) and computer vision for enhancing Open-Source Intelligence (OSINT) data analysis. The presentation, building upon years of collaborative research with Chet Hosmer, addresses a critical challenge faced by OSINT practitioners: the overwhelming volume of raw data, often described as a "wasteland," which makes extracting actionable intelligence both time-consuming and inefficient. Raggo's core thesis is that by applying AI-driven computer vision techniques to images and video, analysts can achieve a significantly more automated, efficient, and accurate means of processing vast datasets.

The talk highlights the evolution of OSINT methodologies, moving beyond traditional data collection and basic analysis to a sophisticated approach that leverages advanced AI capabilities. Raggo underscores the necessity of making OSINT data not just collected, but truly actionable. This shift is particularly crucial in the visual domain, where manual analysis of countless images and video frames can quickly become an insurmountable task. By automating the identification and classification of objects, features, and contextual information within visual media, the research presented aims to drastically improve the efficacy and precision of intelligence gathering.

The importance of this work lies in its direct impact on the efficiency and effectiveness of security investigations, threat intelligence, and various other fields reliant on public data. As the volume of publicly available visual data continues to explode across social media, public cameras, and other online sources, the ability to rapidly and accurately analyze this information becomes paramount. Raggo's research offers a pathway to transform this challenge into an opportunity, enabling analysts to uncover critical insights that would otherwise remain buried in the sheer magnitude of data, thereby enhancing situational awareness and decision-making for defenders and investigators alike.

Background

▶ Watch: Welcome and Defcon first-timer advice (0:00)

The landscape of Open-Source Intelligence has long been characterized by a fundamental paradox: an abundance of information juxtaposed with a scarcity of actionable insight. As Mike Raggo and his collaborator Chet Hosmer have frequently highlighted in their numerous previous talks at the Packet Hacking Village, OSINT data collection often results in a "wasteland of data." This refers to the colossal volume of raw, unstructured information—ranging from text and metadata to images and videos—that, once collected, presents a formidable challenge for analysis. Traditional OSINT methodologies, while effective for targeted queries and smaller datasets, quickly become overwhelmed when confronted with modern data scales.

Prior to this presentation, Raggo and Hosmer's research focused extensively on strategies for OSINT data collection, data analysis, reporting, and crucially, pruning down that data. Their objective has consistently been to distill vast quantities of information into highly actionable intelligence, moving away from merely accumulating data. The problem they sought to address was the inherent inefficiency and potential for human error in manually sifting through and interpreting large and diverse datasets. This includes the painstaking process of identifying relevant patterns, anomalies, or specific entities within textual documents, network traffic, or, increasingly, visual media.

The advent and proliferation of digital cameras, smartphones, and ubiquitous online platforms have dramatically amplified the volume of visual data available for OSINT. Images and videos now constitute a significant portion of the "wasteland." Manually reviewing hours of video footage or thousands of images to spot a particular object, person, or detail is not only labor-intensive but also highly susceptible to human fatigue and oversight. This creates a critical gap between the potential intelligence embedded within visual data and the practical capacity of analysts to extract it. This talk specifically aims to bridge that gap by introducing advanced AI techniques, particularly computer vision, as a means to automate and refine the analysis of this visual data, thereby making the OSINT process more robust and reliable in the face of ever-growing data volumes.

Key Findings

▶ Watch: Speaker Mike Raggo's official introduction (2:00)

The central finding of Mike Raggo's presentation is the profound capability of AI computer vision to revolutionize OSINT data analysis, particularly when dealing with image and video datasets. The research, a culmination of years of collaboration with Chet Hosmer, demonstrates that AI can transform the traditional "wasteland of data" into a source of highly actionable intelligence by automating and significantly enhancing the efficiency and accuracy of visual data processing.

Raggo emphasizes that by leveraging AI, analysts can perform a variety of "cool and interesting things" with visual data that were previously impractical or impossible with manual methods. The key contributions and findings revolve around:

1. Automation and Efficiency: AI models can process vast quantities of images and video frames at speeds far exceeding human capabilities. This automation drastically reduces the time and resources required for initial data triage and filtering, allowing human analysts to focus on higher-level interpretation and strategic insights rather than exhaustive manual review. The goal is to move towards a more "automated and efficient manner" of analysis.
2. Improved Accuracy: Raggo asserts that AI-driven computer vision offers "a lot better accuracy" in identifying specific objects and features within visual data compared to human analysts, especially when dealing with subtle cues or complex scenes across massive datasets. This is attributed to the AI's ability to consistently apply learned patterns and detect minute details that might be overlooked by the human eye during prolonged analysis.
3. Refined Object Detection through Bounding Boxes: A critical technical insight highlighted by Raggo is the importance of bounding boxes in training AI models for precise object recognition. Instead of merely identifying a general object like a "car," bounding boxes enable the AI to be trained on and subsequently detect highly specific components, such as "wheels" or "hubcaps," or even the "front left tire." This granular level of detail is crucial for many OSINT scenarios where specific features of an object, rather than the object itself, hold investigative value. The ability to train the AI to focus on "a smaller portion of a picture" allows for a more "refined look at every image," even when the main object (e.g., a car) is presented at different angles.
4. Making Data Actionable: Ultimately, the overarching finding is that AI computer vision provides the tools necessary to convert raw visual data, which often contributes to the "wasteland," into concrete, actionable intelligence. By precisely identifying and categorizing visual elements, the AI facilitates the extraction of meaningful insights that can directly support investigations, threat assessments, and strategic decision-making.

These findings collectively present a compelling case for the integration of advanced AI techniques into the standard OSINT toolkit, promising a future where visual data analysis is not only faster but also significantly more insightful and reliable.

Technical Deep Dive

▶ Watch: Introducing AI computer vision for OSINT data analysis (2:50)

The technical core of Mike Raggo's presentation, while presented at a high level, centers on the application of AI computer vision to overcome the inherent challenges of visual data in OSINT. The primary technical mechanism discussed for achieving enhanced accuracy and efficiency is the strategic use of bounding boxes in the training and application of machine learning models for object detection.

At its essence, computer vision involves enabling computers to "see" and interpret visual information from images and videos. This field encompasses various tasks, including image classification (identifying what an image contains), object detection (identifying where specific objects are located within an image and classifying them), object tracking (following objects across video frames), and semantic segmentation (pixel-level classification of image regions). Raggo's focus on identifying specific features within images, like car wheels or hubcaps, strongly points towards advanced object detection and potentially instance segmentation capabilities.

The speaker highlights a crucial distinction: simply identifying a "car" in an image is often insufficient for detailed OSINT. Investigators frequently need to identify granular details, such as the type of hubcaps, the presence of specific wheel designs, or even the condition of a front left tire. This level of detail presents a significant challenge for generic computer vision models. Raggo explains that the solution lies in the sophisticated application of bounding boxes during the training phase of an AI model.

Bounding boxes are rectangular coordinates drawn around objects of interest within an image during the annotation process. This process is fundamental to supervised learning in computer vision. For example, instead of feeding an AI model thousands of images labeled simply "car," an analyst would meticulously draw bounding boxes around each wheel, each hubcap, or even specific parts of a wheel (e.g., "front-left-tire") in a diverse dataset of car images. These annotated images then become the training data for the AI model.

Raggo elaborates on the benefits of this approach:

• Refined Object Focus: Bounding boxes allow the AI to "focus on a smaller portion of a picture." This is critical because a car image might contain many elements (the entire car, background, other objects), but the OSINT goal might only be to analyze the wheels. By explicitly training the model on bounding boxes around wheels, the AI learns to prioritize and extract features specific to those components.
• Contextual Understanding for Granular Detail: The speaker raises a pertinent question: "How does the AI know like am I looking at the, you know, driver side of the car or the passenger side of the car and those kinds of things?" This implies the need for even more detailed bounding box annotations and potentially more complex model architectures that can infer relative positioning or attributes. For instance, bounding boxes could be labeled not just "wheel" but "driver-side-front-wheel" or "passenger-side-rear-wheel," enabling the model to distinguish between functionally identical objects based on their spatial relationship within the larger context. This requires a robust dataset that captures objects from various angles and perspectives, with precise annotations for each specific feature.
• Improved Accuracy in Specific Tasks: By training the AI with such refined bounding box data, the model develops a highly specialized capability to detect and classify these specific features with greater accuracy. This contrasts with a generic object detector that might correctly identify a car but fail to differentiate between various hubcap designs, which could be crucial for identifying a specific vehicle or fleet in an OSINT investigation. The training process effectively teaches the AI to perform a "more refined look at every image," extracting the exact details required.

While specific tools or frameworks like YOLO (You Only Look Once), Faster R-CNN, or Mask R-CNN were not explicitly named in the transcript, the methodology described by Raggo aligns perfectly with the principles behind these state-of-the-art object detection architectures. These models are designed to efficiently process images, identify objects, and draw bounding boxes around them based on their learned features from annotated datasets. The implication is that Raggo's research likely leverages or proposes the use of such advanced deep learning models, customized with highly specific bounding box annotations, to achieve the desired level of granular analysis for OSINT.

The emphasis on "testing" the models further indicates an iterative development process common in machine learning, where models are trained, evaluated, and refined based on their performance against various test sets. This ensures that the AI's accuracy is not just theoretical but proven in practical OSINT scenarios.

Demo / Proof of Concept

▶ Watch: Talk's specific focus: automated, efficient image/video analysis (3:50)

The transcript does not detail a specific live demonstration or proof of concept that was presented during Mike Raggo's talk. The speaker primarily focused on outlining the theoretical framework, the challenges in traditional OSINT, and the benefits of applying AI computer vision to visual data analysis. While the concepts discussed, particularly the use of bounding boxes for granular object detection (e.g., distinguishing specific car parts like wheels and hubcaps), strongly imply underlying technical implementations, the talk's narrative as captured in the transcript remains at a conceptual and strategic level regarding these capabilities rather than showcasing a working system or specific output.

Defensive Implications

▶ Watch: Speaker's funny origin story: taking down Nasdaq (4:10)

The insights presented by Mike Raggo on leveraging AI computer vision for OSINT data analysis carry significant implications for defenders across various domains, from corporate security to national intelligence and individual privacy. Understanding these capabilities is crucial for both proactive defense and effective counter-intelligence.

1. Enhanced Adversarial Capabilities: Defenders must recognize that the same AI computer vision techniques discussed can be, and likely are, employed by adversaries—whether they are nation-state actors, cybercriminals, or competitive intelligence gatherers. This means that publicly available images and videos, previously considered low-risk or too voluminous for manual analysis, can now be systematically scanned for highly specific and valuable intelligence. For example, an adversary could use AI to identify specific models of security cameras, unique vehicle modifications, employee uniforms, or even subtle changes in physical infrastructure from publicly posted images or satellite imagery.
2. Granular Data Extraction: The emphasis on bounding boxes for detecting minute details like specific hubcaps or the condition of a front left tire highlights the precision with which AI can extract information. This implies that even seemingly innocuous visual details can become critical data points. Defenders need to consider that details like specific equipment serial numbers, unique branding on gear, or even the type of laptop visible in a background could be automatically identified and correlated across vast datasets. This necessitates a re-evaluation of what constitutes "sensitive" visual information in public or semi-public contexts.
3. Operational Security (OPSEC) Re-evaluation: Organizations and individuals need to significantly re-evaluate their OPSEC practices concerning visual content. Any image or video shared online, even if seemingly benign, could be a source of intelligence. This includes photos from company events, social media posts by employees, or even video conference backgrounds. Defenders should educate personnel on the power of AI in visual OSINT and implement stricter guidelines on what visual information is permissible to share, focusing on potential metadata, background details, and specific object identification.
4. Automated Threat Intelligence Gathering: On the proactive side, defenders can adopt similar AI computer vision techniques for their own threat intelligence gathering. This could involve:

• Monitoring Dark Web Forums and Social Media: Automatically scanning images and videos for indicators of compromise, adversarial tools, specific logos of targeted organizations, or the presence of sensitive documents.
• Physical Security Assessments: Analyzing public imagery of facilities to identify potential vulnerabilities, access points, or changes in security posture over time.
• Brand Protection: Identifying unauthorized use of company logos or products in images and videos across the internet.
• Supply Chain Risk Assessment: Analyzing imagery related to supply chain partners to detect anomalies or potential risks.

1. Privacy Implications: For individuals, the widespread application of AI computer vision in OSINT raises significant privacy concerns. Features like facial recognition (though not explicitly detailed in this talk, it's a core CV capability), gait analysis, and object detection can be used to track individuals, identify their possessions, and infer their habits or affiliations from visual data. Defenders, particularly those concerned with civil liberties or human rights, must advocate for ethical AI development and deployment, as well as robust privacy regulations.
2. Data Volume and Speed Advantage: The ability of AI to process "wasteland of data" efficiently means that adversaries can rapidly derive insights from overwhelming data volumes. Defenders must develop strategies to counteract this speed advantage, either by employing similar AI tools for counter-OSINT or by implementing measures to obscure or reduce the discoverability of sensitive visual information.

In essence, Mike Raggo's talk serves as a wake-up call, underscoring that the visual world is no longer just for human eyes. AI, particularly through sophisticated computer vision and precise object detection using bounding boxes, has become a powerful analytical tool that can extract unprecedented levels of detail from images and videos, fundamentally altering the calculus of information security and privacy.

Key Takeaways

• AI Computer Vision Transforms OSINT: AI-driven computer vision provides a crucial advancement for Open-Source Intelligence, moving beyond traditional manual analysis to enable automated, efficient, and significantly more accurate processing of image and video data.
• Addressing the "Wasteland of Data": The research directly tackles the problem of overwhelming data volumes in OSINT, aiming to convert a "wasteland of data" into actionable intelligence through sophisticated visual analysis.
• Bounding Boxes for Granular Detection: The strategic use of bounding boxes during AI model training is paramount for achieving refined, granular object detection, allowing the AI to identify specific components (e.g., car wheels, hubcaps) rather than just the general object.
• Enhanced Accuracy in Visual Analysis: By focusing AI training with precise annotations, the models can achieve superior accuracy in identifying subtle visual cues and specific features that might be overlooked by human analysts.
• Building on Prior OSINT Research: This talk represents a continuation of Mike Raggo's and Chet Hosmer's extensive research into OSINT data collection, analysis, and pruning, specifically extending their methodologies to the visual domain.
• Critical Defensive Implications: Defenders must recognize that adversaries can leverage these same AI computer vision capabilities for highly detailed intelligence gathering, necessitating a re-evaluation of OPSEC, privacy, and proactive threat intelligence strategies.

About the Speaker(s)

Mike Raggo is a highly experienced and respected figure in the cybersecurity community, known for his long-standing contributions to conferences like DEF CON. He has been a regular attendee and speaker at Defcon since the year 2000, frequently presenting at the Packet Hacking Village. Raggo is a dedicated researcher, often collaborating with his good friend Chet Hosmer on a "plethora of talks" focused on various aspects of Open-Source Intelligence (OSINT), including data collection, analysis, reporting, and data pruning to make intelligence actionable.

Raggo shares a memorable origin story regarding his entry into cybersecurity. Back in 1994, while working as a Unix system administrator at the Nasdaq stock market, he encountered an article in Information Week about Dan Farmer's release of Satan—one of the first vulnerability scanners for Unix platforms. In a moment of youthful curiosity, Raggo downloaded and ran Satan wide open on the Nasdaq trading floor during business hours, inadvertently taking down a couple of servers. Although this caused a brief outage, Nasdaq's failover systems worked flawlessly. Instead of being fired, Raggo was offered a promotion, with his CEO tasking him with building Nasdaq's first security team to protect the network he had accidentally demonstrated he could compromise. This foundational experience shaped his career, leading him to become a prominent voice in network and data security.

Chet Hosmer is consistently mentioned by Mike Raggo as a "very good friend" and long-time collaborator on their extensive OSINT research. While Chet Hosmer did not present this specific talk, his contributions are integral to the body of work that Raggo presented, particularly the foundational strategies for OSINT data collection, analysis, and making data actionable that preceded this deep dive into AI computer vision. Their joint efforts have produced numerous presentations at events like DEF CON's Packet Hacking Village, providing valuable insights to the security community.