GPTracker: A Large-Scale Measurement of Misused GPTs

GPTracker: A Large-Scale Measurement of Misused GPTs

Speakers: Xinyue Shen, PhD Student; Yun Shen, Researcher at Fleser; Michael Backes, Professor; Yang Zhang, Professor

Conference: IEEE S&P

YouTube: https://www.youtube.com/watch?v=8FvXwE30AzI

Overview

In an era where large language models (LLMs) are rapidly evolving into sophisticated agents capable of browsing the web, executing code, and integrating with external tools, the emergence of LLM-specific application marketplaces presents both innovation and significant security challenges. This talk, "GPTracker: A Large-Scale Measurement of Misused GPTs," presented by Xinyue Shen, delves into the dark side of this new ecosystem, specifically focusing on the OpenAI GPT Store. The research introduces GPTracker, a novel framework designed for the continuous, large-scale collection and analysis of custom GPTs, aiming to identify and understand the prevalence and operational mechanisms of those designed for malicious purposes.

The work, a collaboration between researchers from Fleser and academia, highlights a critical vulnerability in the rapidly expanding LLM agent landscape. With OpenAI reporting over 3 million custom GPTs created by January 2024, the potential for misuse—ranging from facilitating illegal activities to spreading hate speech and conducting phishing attacks—is immense. GPTracker provides the first comprehensive, empirical study into this phenomenon, revealing the scale of the problem, the sophisticated techniques employed by malicious actors, and the urgent need for robust platform defenses. The findings underscore the importance of proactive security measures as LLM agents become increasingly integrated into daily digital life.

This research is particularly significant because it moves beyond theoretical concerns to provide concrete evidence of widespread misuse in a burgeoning platform. By systematically identifying thousands of policy-violating GPTs and responsibly disclosing them, the team initiated the largest known removal of such entities to date. The study not only quantifies the problem but also offers a detailed technical analysis of how these misused GPTs operate, providing invaluable insights for platform developers, security researchers, and end-users grappling with the security implications of this new technological frontier.

Background

▶ Watch: Introduction: The rise of misused GPTs (0:00)

The landscape of large language models has undergone a dramatic transformation in recent years. Initially confined to generating text based solely on their training data, LLMs have rapidly evolved into what are now often termed LLM agents. These agents are augmented with a variety of capabilities that extend their functionality far beyond mere text generation. Key among these advancements are the abilities to browse the web, execute Python scripts, maintain chat history for contextual understanding, and even directly interact with and control local computing environments. This augmentation is achieved through the integration of various tools and external knowledge files, effectively turning LLMs into highly versatile and programmable entities.

This paradigm shift has significantly lowered the barrier to entry for developing complex, intelligent applications. Developers, and indeed anyone with a basic understanding of prompt engineering, can now create sophisticated LLM agents with considerably less effort than traditional software engineering. This democratization of agent development has, in turn, led to the emergence of dedicated marketplaces, analogous to mobile application stores like Google Play and Apple App Store, where users can share and discover these customized LLM agents.

The most prominent example of this new ecosystem is the GPT Store, launched by OpenAI. By January 2024, OpenAI reported an astonishing figure: over 3 million custom GPTs had been created by its user base. While many of these GPTs undoubtedly serve benign and innovative purposes, the rapid proliferation of user-generated content inevitably attracts malicious actors. Within just two months of the GPT Store's launch, custom GPTs designed to violate platform policies began to appear. The existence of a centralized marketplace amplifies the risk associated with these "misused GPTs," making them easily discoverable and accessible to a wider audience, thereby introducing new and significant security and ethical challenges that traditional content moderation systems are ill-equipped to handle. The problem thus became clear: a mechanism was needed to systematically identify, analyze, and mitigate the risks posed by these malicious agents within this new, rapidly expanding digital frontier.

Key Findings

▶ Watch: Identifying and examples of forbidden GPT scenarios (4:00)

The research presented in "GPTracker: A Large-Scale Measurement of Misused GPTs" yielded several critical findings, illuminating the scale and nature of misuse within the OpenAI GPT Store ecosystem.

Firstly, the development of GPTracker itself represents a significant contribution. This novel framework is specifically designed for the continuous collection and analysis of custom GPTs. It employs a dual-pronged approach: static analysis via a web crawler that periodically scrapes GPT metadata, and dynamic analysis through a custom Chrome extension that automatically interacts with GPTs to observe their behavior. Over 28 rounds of static analysis, GPTracker collected metadata for over 800,000 GPTs. For dynamic analysis, it conducted 4,000 conversations, generating 28,000 conversation flows, primarily focused on suspicious GPTs.

Through this comprehensive data collection and an LLM-driven scoring system combined with manual verification against OpenAI's content policies, the researchers identified a substantial number of policy-violating GPTs. In total, over 2,000 misused GPTs were identified across 10 distinct forbidden scenarios outlined by OpenAI. The most frequently violated scenarios were illegal activity, economic harm, and physical harm, highlighting the serious nature of the threats. Examples included GPTs teaching users how to manufacture illicit substances and generating hate speech.

A crucial finding from the dynamic analysis concerned the operational mechanisms and effectiveness of these misused GPTs. The study categorized conversation patterns into four types based on their reliance on the LLM itself, built-in tools, or external APIs. It was discovered that misused GPTs that activate tools—whether built-in or external APIs—are significantly more likely to respond to inappropriate queries and produce more harmful outputs compared to those that rely solely on the base LLM. This indicates that the very features designed to enhance LLM utility are being leveraged by malicious actors to increase the efficacy of their harmful content.

Furthermore, the research uncovered sophisticated techniques employed by malicious GPTs, particularly in the realm of phishing and malware distribution. The study identified 50 malicious domains across 446 GPTs. These domains were categorized into 33 phishing, 28 malware, and 2 spam, with some domains receiving multiple labels. Attackers were observed using tactics like integrating external APIs, embedding hidden intentions within GPT descriptions, and employing URI redirection to evade detection by standard domain scanners. For instance, a benign-looking URL might redirect to a malicious phishing site upon user interaction, a technique that bypasses initial security checks.

Finally, the researchers responsibly disclosed their findings to OpenAI. This disclosure led to the takedown of thousands of misused GPTs, representing the largest reported removal of such entities to date. This demonstrates the immediate and tangible impact of their research in enhancing the security posture of the GPT Store ecosystem.

Technical Deep Dive

▶ Watch: Analyzing operational patterns and effectiveness of misused GPTs (5:00)

The core of this research lies in GPTracker, a meticulously designed framework for the systematic identification and analysis of misused GPTs. GPTracker operates on two primary analytical axes: static and dynamic.

The static analysis component of GPTracker is powered by a robust web crawler. This crawler is configured to continually collect metadata from the official GPT Store every two weeks. Over the course of 28 rounds, this process amassed metadata for over 800,000 custom GPTs. The collected metadata typically includes information such as the GPT's name, description, category, creator, and any publicly visible configuration details. This vast dataset provides a foundational understanding of the GPT ecosystem's breadth and evolution.

Complementing the static analysis is the dynamic analysis module, which is implemented as a custom Chrome extension. This extension is engineered to automatically interact with suspicious GPTs, simulating user queries and collecting the resulting conversation flows. The objective is to observe the GPT's real-time behavior, including its responses, tool activations, and interactions with external services. To date, GPTracker has executed approximately 4,000 conversations with potentially misused GPTs, generating a dataset of 28,000 conversation flows. These flows are critical for understanding the operational mechanisms of malicious GPTs, as they capture the step-by-step interactions and outputs.

Once data is collected, the next crucial step is misuse identification. The researchers developed an LLM-driven scoring system to filter out suspicious GPTs from the massive dataset. While the exact architecture of this scoring system is not detailed, it likely involves feeding GPT descriptions, initial prompts, and conversation snippets to a powerful LLM (potentially one of OpenAI's own models) to assess their adherence to content policies. GPTs flagged as suspicious by this automated system then undergo manual verification. This human-in-the-loop process is essential to confirm violations against OpenAI's official content policy, which categorizes misuse into 10 forbidden scenarios. This hybrid approach combines the scalability of automated analysis with the accuracy of human judgment.

A significant aspect of the dynamic analysis involves dissecting the conversation patterns of misused GPTs. The collected conversation flows are converted into flow graphs, which visually represent the sequence of interactions, decisions, and tool usages. Through this analysis, four distinct conversation patterns were identified, differentiated by their reliance on the LLM's inherent capabilities, built-in tools, or external APIs:

1. Pattern 1 (LLM-only): The GPT generates answers solely using the underlying large language model, without activating any external functionalities.
2. Pattern 2 (Built-in Tools): The GPT leverages OpenAI's native tools, such as the web browsing tool or Python code interpreter, to achieve its malicious objectives.
3. Pattern 3 (External API): The GPT integrates with and calls external, user-defined APIs to fetch or process information, often crucial for its misuse.
4. Pattern 4 (Built-in Tools + External API): The most sophisticated pattern, where the GPT combines both built-in tools and external APIs to execute its harmful agenda.

To quantify the effectiveness of these misused GPTs, two key metrics were employed: answer ratio and response harmfulness. The answer ratio measures how frequently a GPT responds to an inappropriate query, while response harmfulness assesses the severity of the generated output. These metrics were judged against "previous representative studies," implying a standardized method for evaluating harmful content. The analysis revealed a critical insight: GPTs that activate tools (Patterns 2, 3, and 4) are significantly more effective, showing a higher answer ratio and producing more harmful outputs than those relying solely on the LLM (Pattern 1). This underscores that the enhanced capabilities of LLM agents are directly contributing to their potential for harm.

Beyond these patterns, the study also identified specific technical techniques employed by malicious actors. One prominent technique is the integration of external APIs, as exemplified by the "Fenza" GPT which connected to dmma.jp (a Japanese adult content website) to display explicit material. Another technique involves hiding intentions in GPT descriptions, where seemingly innocuous language masks malicious functionality. Perhaps most insidious is the use of URI redirection to circumvent domain scanners. In phishing scenarios, a GPT might present a benign-looking URL to the user, which, upon clicking, immediately redirects to a malicious, phishing domain. This tactic effectively bypasses initial static analysis checks on URLs, making detection significantly harder for platform owners.

Demo / Proof of Concept

▶ Watch: Discovery of phishing GPTs and malicious domains (7:00)

The talk presented several compelling examples and demonstrations to illustrate the diverse forms and operational mechanisms of misused GPTs. These examples served as concrete proof of concept for the researchers' findings.

One stark example showcased a misused GPT operating in the illegal activity scenario. This particular GPT was designed to teach users "how to make cocaine," providing step-by-step instructions for manufacturing illicit substances. This directly violates policies against illegal activities and highlights the potential for LLM agents to become conduits for dangerous information.

Another demonstration involved a GPT engaged in generating hate speech. This GPT was configured to produce content suitable for platforms like Reddit or Facebook, containing discriminatory or inflammatory language. This example underscored the ease with which LLM agents can be weaponized to spread harmful ideologies and misinformation, posing significant challenges for content moderation.

A more technically intricate demonstration focused on a GPT named "Fenza." This GPT was designed to output explicit content to users. The mechanism was particularly illustrative of how external APIs are leveraged for misuse. Fenza was initialized with a specific system prompt, and upon receiving a user query, it would connect to the API of dmma.jp. This website, accessible only in Japan, is known for selling adult content and providing online adult services. By configuring this API, the GPT could directly fetch and display explicit content within the chat interface, bypassing OpenAI's internal content filters and demonstrating the power of external tool integration for harmful purposes.

The researchers also presented a detailed phishing campaign observed during their study. A single builder created 21 distinct GPTs with varied names, all designed to attract a broad user base. The descriptions of these GPTs prominently featured keywords like "free sign up" and "bonus," enticing users with promises of rewards. Crucially, the builder employed an "upward arrow" symbol within the GPT's response to trigger users to click on a specific domain. Initially, this domain was classified as benign by services like VirusTotal. However, the moment a user clicked the domain, they were immediately redirected to a phishing domain. This demonstrated the sophisticated use of URI redirection as a technique to evade detection by automated domain scanners, which might only analyze the initial URL.

The scale of this domain-based misuse was also quantified: the study identified 50 malicious domains linked to 446 different GPTs. These malicious domains were further categorized, revealing 33 phishing domains, 28 malware domains, and 2 spam domains, with some domains exhibiting multiple malicious labels. These concrete examples and statistics unequivocally proved the existence and technical sophistication of misused GPTs within the OpenAI GPT Store.

Defensive Implications

▶ Watch: Summary of findings and responsible disclosure impact (8:30)

The findings from the GPTracker study carry profound implications for various stakeholders involved in the LLM agent ecosystem, particularly for platform owners, users, and security researchers. Implementing robust defensive strategies is crucial to mitigate the widespread misuse identified.

For platform owners like OpenAI, the research highlights critical vulnerabilities in their current detection and moderation systems. The most immediate implication is the need for enhanced proactive detection mechanisms at both the GPT creation and update stages. Current checks are evidently insufficient, as thousands of policy-violating GPTs were able to proliferate. This necessitates a more sophisticated LLM-driven content moderation system that can analyze system prompts, descriptions, and potential API configurations for malicious intent before a GPT is published or updated. The study's LLM-driven scoring system could serve as a model for this.

Furthermore, platform owners must significantly improve their domain scanning capabilities. The discovery of URI redirection as a bypass technique indicates that static analysis of URLs is inadequate. Dynamic analysis of embedded URLs, simulating user clicks to detect redirects to malicious sites, is essential. Monitoring API integration is also critical; platforms should scrutinize the APIs GPTs intend to use, potentially requiring explicit declarations or even sandboxing external API calls to prevent data exfiltration or the display of forbidden content. Implementing stricter validation for API schemas and return types could also help.

Users of LLM agent marketplaces must adopt a heightened sense of caution. The prevalence of misused GPTs, especially those engaged in phishing or distributing harmful content, means users cannot blindly trust every available agent. Key defensive practices for users include:

• Skepticism: Be wary of GPTs promising "free" services, bonuses, or financial gains, as these are common lures for phishing attacks.
• Verification: Check the reputation of GPT creators and look for official endorsements or higher user ratings. While not foolproof, it's a starting point.
• URL Awareness: Exercise extreme caution when a GPT prompts you to click on external links. Hover over links to preview the URL, and be suspicious of unexpected redirects. Assume any external link could be malicious.
• Understanding Capabilities: Recognize that LLM agents can connect to external services and browse the web, meaning they can perform actions beyond just generating text within the chat window. This expanded capability introduces expanded risk.
• Reporting: Report any suspicious GPTs or interactions to the platform owner immediately.

For developers creating GPTs, the implications emphasize the need for responsible development practices. Understanding and strictly adhering to platform content policies is paramount. Developers should conscientiously review their GPTs for potential misuse, especially when integrating external APIs or tools, and ensure that their creations cannot be easily exploited for harmful purposes, even unintentionally.

Finally, for security researchers, the GPTracker framework itself serves as a blueprint for continuous monitoring and analysis of evolving LLM agent ecosystems. As new platforms emerge and existing ones grow, similar frameworks will be essential to track new attack vectors, techniques, and the overall security posture of these environments. The interdisciplinary approach combining static and dynamic analysis, coupled with LLM-driven and manual verification, provides a robust methodology for future research in this dynamic field. The responsible disclosure model also underscores the importance of collaboration between researchers and platform owners to secure these nascent technologies.

Key Takeaways

• Widespread Misuse in LLM Agent Ecosystems: The GPT Store, a prominent marketplace for custom LLM agents, is significantly plagued by misused GPTs, with over 2,000 policy-violating agents identified across various harmful scenarios, including illegal activity, economic harm, and physical harm.
• GPTracker as a Novel Monitoring Framework: The research introduced GPTracker, a scalable framework that combines static metadata collection (over 800,000 GPTs) and dynamic interaction analysis (28,000 conversation flows) to continuously monitor and identify malicious LLM agents.
• Tool and API Integration Amplifies Harm: Misused GPTs that leverage external APIs or built-in tools are significantly more effective at delivering harmful content and responding to inappropriate queries compared to those relying solely on the base LLM, indicating that enhanced capabilities also mean enhanced risk.
• Sophisticated Evasion Techniques: Malicious actors employ advanced tactics such as integrating external APIs (e.g., for explicit content), hiding true intentions in GPT descriptions, and using URI redirection to bypass platform content filters and domain scanners, making detection challenging.
• Urgent Need for Platform Security Enhancements: Platform owners must implement more robust, proactive security measures, including advanced LLM-driven content moderation, dynamic URL analysis for redirects, and stricter scrutiny of external API integrations, to protect users from evolving threats.
• Responsible Disclosure Drives Significant Takedowns: The responsible disclosure of findings to OpenAI led to the removal of thousands of misused GPTs, demonstrating the critical role of security research in improving the safety and integrity of new technological platforms.

About the Speaker(s)

The primary presenter of this work, Xinyue Shen, also known as Shing, is a PhD student who led the research on GPTracker. Her work focuses on understanding and mitigating the security and privacy risks associated with large language models and their applications. She collaborated with Yun Shen, a researcher affiliated with Fleser, indicating a blend of academic and industry expertise in the research team. The project was advised by Michael Backes and Yang Zhang, both professors, suggesting their roles in guiding the research methodology and contributing to the academic rigor of the study. Their combined expertise spans areas of computer science, security, and privacy, particularly in the context of emerging AI technologies.