AI at the Gates: Combating AI-Driven Assaults on the Customer Experience

AI at the Gates: Combating AI-Driven Assaults on the Customer Experience

Speakers: Nate Carroll, CEO, Spec; Julie Yang, VP of Product, Spec

Conference: RSAC 2024

YouTube: https://www.youtube.com/watch?v=nmVsXn--CK8

Overview

In an era where digital interactions define the customer experience, a new and sophisticated threat is emerging: AI-driven assaults. This talk, delivered by Nate Carroll, CEO, and Julie Yang, VP of Product at Spec, delves into the alarming reality of AI agents manipulating online platforms for illicit gain. Unlike traditional bots or script runners, these AI agents are designed to meticulously impersonate real users, executing complex attacks that can lead to significant financial losses for businesses. The speakers highlight a specific instance where a customer faced a staggering $9 million loss due to such an attack.

The core of the problem lies in the agents' ability to mimic human behavior so effectively that they bypass conventional bot detection and fraud prevention systems. They leverage stolen credentials and payment methods to transform "dirty money" into "clean cash" by abusing the user experience within marketplaces and financial institutions. This presentation serves as a critical wake-up call, urging security professionals to understand the evolution of these threats and adopt multi-layered, collaborative defensive strategies that transcend traditional security silos.

Carroll and Yang emphasize that these aren't theoretical threats but active, evolving attacks that are becoming increasingly accessible. What began with niche applications like sneaker bots has rapidly matured into sophisticated operations targeting high-value assets, tickets, and now, directly impacting financial services. The talk provides a comprehensive look into the mechanics of these AI agents, the tools they use, the patterns they exhibit, and actionable recommendations for organizations to defend against what is essentially a "virtual attack team" operating at scale.

Background

▶ Watch: Introduction and $9M AI-driven attack example (0:00)

The landscape of automated online abuse has undergone a significant transformation, evolving from rudimentary scripts to highly sophisticated AI agents. Julie Yang, VP of Product at Spec, characterizes this shift as an "industrial revolution of bots," tracing its origins back a decade to the emergence of sneakerbots. A prime example, Better Nike Bot, arose in response to the high demand and limited supply of coveted product launches, such as Air Jordans. Tech-savvy customers leveraged automated scripts to gain an unfair advantage, securing inventory ahead of others. This initial "cottage industry" quickly professionalized, establishing a lucrative market where sneakers became akin to unregulated financial securities – easy to buy and sell, fungible for cash, and appreciating significantly in value. The business model of these operations is robust, employing staff and continuously developing tools.

This success spurred competition and diversification. The focus expanded beyond physical goods to digital assets like event tickets, as evidenced by incidents like the Live Nation disaster with Taylor Swift's Eras Tour. From there, the threat naturally extended to two-sided marketplaces, which presented unique vulnerabilities beyond simple inventory stockouts. Marketplaces are susceptible to buyer-seller collusion, where attackers inflate prices to maximize their illicit earnings, degrading the experience for legitimate users. Crucially, marketplaces also involve bidirectional money transfer, making them a logical stepping stone for attackers to target fintechs and traditional financial institutions directly.

The fundamental change underlying this evolution is the shift from command-line interface (CLI) scripts that follow rigid instructions to task-driven AI agents. These agents are given high-level outcomes to achieve and are programmed to act like humans, even collaborating with each other. This qualitative leap in sophistication means attacks can be launched at unprecedented scale and speed. Furthermore, several macro trends contribute to the proliferation of these threats: a broad societal shift from physical to digital transactions, an economic downturn driving individuals to seek alternative income streams, and an explosion in the availability of user-friendly attack tools. These All-in-Ones (AIOs), as they are often called, boast intuitive user interfaces and experiences (UI/UX) that resemble modern SaaS products, making sophisticated fraud accessible even to teenagers and young adults. This ease of use significantly lowers the barrier to entry for attackers, allowing small groups to mount large-scale operations.

Key Findings

▶ Watch: AI attack tools marketed to teenagers, not just pros (1:57)

The core discovery presented in this talk is the profound capabilities of AI agents in orchestrating sophisticated fraud and abuse campaigns that are largely invisible to traditional security measures. These agents do not merely automate tasks; they impersonate real users, executing authentic clicks, keystrokes, and swipes. This enables them to function as a "virtual attack team," allowing small groups of attackers to scale their operations to a level previously requiring dozens of skilled individuals.

A critical finding is the agents' ability to patiently reverse engineer policies and risk models. They systematically test systems, learning how to navigate through defenses and identify vulnerabilities directly within the user experience. This iterative learning process allows them to refine their approach until they can steal millions while appearing to be legitimate customers. Julie Yang highlighted their unwavering discipline: unlike humans, these agents "do not forget to set the proxy, they do not input the wrong data, they do not tire." Their objective is not the success of an individual agent but to collectively increase the overall probability of success by constantly learning and adapting.

The talk revealed a sophisticated collaborative architecture within these attack operations. "Monitor" or "watcher" agents tirelessly scan websites for specific conditions—inventory drops, price glitches, system vulnerabilities. Once an opportunity is identified, they instantly alert other agents, such as "add-to-cart" or "checkout" agents, to complete the transaction. To ensure success, these agents meticulously set up their environment, harvesting cookies, utilizing residential proxies located near shipping addresses, and associating real identity profiles to bypass payment service provider redirects or multi-factor authentication (MFA) step-up challenges during checkout. They are designed to avoid flagging for manual review, ensuring frictionless success.

Perhaps the most alarming finding is the inadequacy of traditional bot detection tools. Because these AI agents are meticulously crafted to replicate human behavior, they are often classified as "no bot" by existing systems. This necessitates a new approach to detection, one that integrates disparate data sources. Spec's analysis showed that while individual fraud, product, or security data points might appear innocuous, their correlation reveals clear attack patterns. For instance, low-and-slow scraping activity, when combined with unusual non-transactional user behavior (e.g., specific wish-listing patterns) and seemingly clean transactional data, exposes the underlying fraud. This cross-functional data integration, using a common identifier like session IDs, is deemed crucial for connecting the dots between network events, application events, and business events.

Finally, the talk underscored the effectiveness of honeypot UX as a defensive strategy. Instead of immediate blocking, which provides valuable training data to the agents, misdirecting attackers with false successes or shadow banning can significantly prolong their retooling time from hours/days to months or more. This not only buys defenders time but also allows for the harvesting of critical intelligence about the attackers' methods and tools.

Technical Deep Dive

▶ Watch: Sneakerbots as unregulated financial securities (3:20)

Spec's ability to detect and analyze these sophisticated AI-driven attacks stems from its unique "NSA style dragnet data collection" methodology. Nate Carroll explained that Spec ingests vast quantities of raw internet traffic from websites and public APIs. This raw data is then processed and distilled into granular real user events that form complete user journeys. This data encompasses three critical layers:

1. Network events: Low-level details such as HTTP requests, endpoints accessed, and IP addresses.
2. Application events: User interactions within the application, including pages visited, searches performed, and specific UI interactions.
3. Business events: High-value actions like authentications, payments, and transfer reversals.

By correlating these diverse data points, Spec gains a holistic view of user activity, enabling the identification of subtle, coordinated patterns indicative of AI agent activity.

The AI agents themselves are highly engineered to mimic human behavior and bypass detection. Their operation involves several sophisticated technical aspects:

• Task-Driven Architecture: Unlike rigid scripts, these agents are given high-level objectives (e.g., "acquire product X," "launder money through account Y"). They then autonomously execute a series of actions, adapting to real-time feedback from the target system.
• Real Browser Emulation: The AIO tools come with native support for specific browsers, ensuring that the agents generate browser fingerprints and network traffic indistinguishable from legitimate human users. They perform genuine clicks, keystrokes, and swipes.
• Environment Preparation: Attackers proactively configure the agents' operating environment to minimize friction during transactions. This includes:
• Cookie Harvesting: Collecting and utilizing valid session cookies to maintain persistent, seemingly legitimate sessions.
• Residential Proxies: Deploying proxies that route traffic through residential IP addresses, often geographically close to the intended shipping or billing address. This evades IP-based blacklisting and location-based fraud detection.
• Identity Profiles: Associating comprehensive, often synthetic or stolen, identity profiles to the agents. This helps them navigate through payment service provider redirects, step-up challenges (like MFA), and other friction points that would typically flag automated activity.
• Collaborative Funnel: The agents operate in a coordinated fashion, often with specialized roles. "Monitor" or "watcher" agents continuously scrape target websites for specific conditions (e.g., inventory availability, price drops, system glitches). Upon detecting an opportunity, they trigger other agents, such as "add-to-cart" or "checkout" agents, to rapidly complete the transaction. This "divide and conquer" approach allows for rapid exploitation of fleeting opportunities.
• Real-time Feedback Loop: The agents are designed for continuous learning. They execute tasks simultaneously against target systems, gather real-time feedback (e.g., success/failure of a transaction, error codes), and then adjust their parameters or strategy for subsequent attempts. This iterative optimization is what allows them to "patiently reverse engineer your policies and risk models."

The speakers illustrated these mechanics with two real-world examples:

1. Fintech Attack Visualization: Spec observed a fraud ring engaged in credential stuffing against one of its fintech customers. A visualization showed agents (red) and their events (blue) along a customer journey. Initially, "peeping Tom" agents logged into compromised accounts, often receiving a HTTP 200 success code. Some proceeded to view the home dashboard or account details. The critical "juicy parts" involved adding new payees (effectively themselves) and attempting express transfers just before close of business on a Friday. This timing suggests an understanding of the target organization's manual review processes and staffing limitations. The agents were "everywhere, feeling for the cracks of your online systems with a seeming knowledge of your online processes."

1. Marketplace Attack (with $9M loss): This attack involved reactivated marketplace sellers and seemingly legitimate buyers.

• Fraud Team's View: Initially, no red flags. Reactivated sellers had correct passwords, passed device/location/biometric checks, and successfully completed MFA. Buyers had clean cards, matching payment methods, and no associated device/location/email risk. Traditional fraud tools saw "squeaky clean" transactions.
• Product Team's View: A deeper look into non-transactional product data revealed anomalies. Buyers exhibited strange behavior, almost exclusively interacting with the listings of the reactivated seller population (e.g., adding to wish lists, adding to cart, page views). This started to form a suspicious network.
• Security Team's View: Independently, security teams observed monitor activity – low-and-slow scraping (a request every few minutes) specifically targeting listings from these reactivated seller accounts. This activity, by itself, was too quiet to trigger alarms.
• The Correlation: The breakthrough came from bringing these three data sets together. When the low-and-slow scraping, the unusual non-transactional buyer behavior, and the seemingly clean transactional data were correlated, the full sophisticated attack pattern became evident, exposing the fraud.

Demo / Proof of Concept

▶ Watch: Evolution: From sneakers to marketplaces and fintech (4:40)

While the talk did not feature a live software demonstration in the traditional sense, the speakers provided compelling conceptual demonstrations and visualizations of both the attacker's tools and the observed attack patterns.

Julie Yang walked the audience through the user interface and workflows of a real-world AIO (All-in-One) tool, specifically a retail-focused distribution. She highlighted its "pretty" and intuitive UI/UX, remarking that it looked as if an "Apple designer" had created it, emphasizing how this design incentivizes usage. The dashboard showed a clear breakdown of agent activities, including "getting a session" and "monitoring." This illustrated the collaborative "divide and conquer" strategy, where monitor/watcher agents look for inventory, price drops, or glitches, and then instruct other agents to "add to cart" or "check out." The demonstration also covered the extensive capabilities of these tools to manage tasks, use synthetic or stolen identity and payment data, and ensure agents appear human by supporting real browsers and setting up environments with harvested cookies and residential proxies. This detailed walkthrough of the AIO tool served as a powerful proof of concept for the sophistication and accessibility of the attacker's arsenal.

Furthermore, Nate Carroll and Julie Yang presented a visualization of actual AI agent activity observed on a fintech customer's system. This graphical representation depicted "agents in red and the events along a customer journey in blue." This visualization clearly showed the "peeping Tom" agents logging in, viewing dashboards and account details, and then the more malicious agents adding new payees and attempting express transfers. This real-world example provided concrete evidence of how AI agents systematically explore and exploit system vulnerabilities, demonstrating their capabilities in a practical, impactful context. The marketplace attack scenario, while not visually demonstrated with a specific tool, was detailed through the lens of how Spec correlates disparate data points from fraud, product, and security teams to expose the hidden patterns of AI agent activity, effectively demonstrating their detection methodology.

Defensive Implications

▶ Watch: Macro trends accelerating AI bot activity (5:45)

Defending against these sophisticated AI-driven assaults requires a fundamental shift in strategy, moving beyond siloed security operations to a more integrated and intelligent approach. The speakers outlined several critical defensive implications:

1. Cross-Functional Data Integration is Paramount: The most significant takeaway is the necessity of bringing together data from disparate teams: fraud, product, and security. Traditional fraud teams focus on payments and returns, product teams on user and conversion events, and security teams on network logs and endpoints. These teams often "speak different languages." To bridge this gap, organizations must establish a common language in their data. Nate Carroll suggested using session IDs as a "lingua franca," a granular concept representing a user visit that can be consistently present across all data sets these three teams work with. This allows for the correlation of seemingly unrelated events—like low-and-slow scraping (security data), unusual non-transactional behavior (product data), and clean transactional outcomes (fraud data)—to expose the full attack pattern.

1. Embrace Delayed Enforcement and Honeypot UX: Immediate blocking (e.g., 403 errors, account locks) provides invaluable training data to AI agents, allowing them to rapidly retool and bypass defenses. Instead, defenders should adopt strategies that provide delayed or no immediate feedback:

• Honeypot UX: This is crucial for turning retooling time from hours/days into months or longer. Examples include:
• Fintech Misdirects: Presenting a false success response for an instant payment, then prompting the attacker to call in, allowing backend cleanup.
• Shadow Banning: For marketplaces, creating listings that appear real to the attacker but are invisible to legitimate buyers. This allows attackers to "succeed" in listing their fraudulent items without actually impacting the marketplace.
• Masking Credential Stuffing Successes: For known credential stuffing attempts, masking the success responses ensures that attackers don't know if their compromised accounts are valid, preventing human follow-up.
• Offline Enforcement: When immediate blocking is not feasible, implement offline enforcement mechanisms that do not provide direct, real-time feedback to the agents. This strategy helps to "harvest an enormous amount of information out of these attackers."

1. Proactive Questioning and Intelligence Gathering: Before implementing any enforcement, security teams should ask critical questions. Can allowing some activity (under strict monitoring) lead to the identification of other bad actors or new exploits? This approach, while requiring careful risk management, can provide deeper insights into attacker methodologies. Looking for linkages between scraping activity in network logs and confirmed fraud outcomes is a direct way to identify if a site is being monitored by these attack tools.

1. Recognize the Limitations of Traditional Bot Detection: Current bot detection tools are largely ineffective against AI agents because these agents are specifically built to replicate human behavior. They will often register as "no bot," highlighting the need for advanced behavioral analytics and cross-data correlation.

1. Anticipate Vertical Expansion: The evolution from sneakerbots to marketplaces and now into fintechs (especially for loan origination and automated money mule operations) indicates a continuous expansion into new sectors. Organizations, particularly in banking and financial services, should be aware that these attacks are "just starting" in their domains and prepare accordingly.

By integrating data, delaying feedback, and intelligently misdirecting attackers, organizations can move from a reactive blocking posture to a proactive, intelligence-led defense that significantly reduces the efficacy and profitability of AI-driven assaults.

Key Takeaways

• AI Agents are a New Class of Threat: These are not traditional bots but sophisticated, task-driven AI agents that meticulously impersonate human behavior, using real clicks and keystrokes to bypass conventional bot detection and fraud prevention systems.
• Cross-Functional Data Integration is Essential: To detect these hidden attacks, security, fraud, and product teams must collaborate by correlating network, application, and business event data, using a common identifier like session IDs to connect disparate observations.
• Immediate Blocking is Counterproductive: Providing immediate feedback (e.g., 403 errors, account locks) serves as valuable training data for AI agents, enabling rapid retooling. Instead, defenders should prioritize delayed enforcement and honeypot UX strategies.
• Honeypot UX Provides Lasting Defense & Intelligence: Implementing misdirects, shadow banning fraudulent listings, or masking success responses can significantly extend attacker retooling time from hours/days to months, while also allowing organizations to harvest critical intelligence.
• Attacker Tools are Sophisticated and Accessible: Modern AIO (All-in-One) tools feature intuitive, consumer-grade UIs, making advanced fraud techniques, including environment setup with residential proxies and identity profiles, accessible to a broader range of attackers.
• Look Beyond Transactional Data: Subtle indicators, such as low-and-slow scraping activity combined with unusual non-transactional user behaviors (e.g., specific wish-listing patterns), are crucial early signals that, when correlated, expose the full extent of AI-driven attacks.

About the Speaker(s)

Nate Carroll is the CEO of Spec, a company dedicated to protecting websites and public APIs from fraud and abuse. His expertise lies in leveraging "NSA style dragnet data collection" to process vast amounts of internet traffic into granular user events and journeys. This comprehensive data analysis allows Spec to identify and combat sophisticated, AI-driven attacks that evade traditional security measures. Carroll's insights focus on the technical mechanisms of detection and the strategic implications for organizations facing these evolving threats.

Julie Yang serves as the VP of Product at Spec. She brings a deep understanding of the product landscape of attack tools, illustrating the "industrial revolution of bots" from their origins as sneakerbots to today's sophisticated AI agents. Yang provides valuable perspectives on the user experience and design of attacker AIO (All-in-One) tools, highlighting how their intuitive interfaces make advanced fraud accessible. Her contributions emphasize the evolution of attack methodologies and the critical need for robust, adaptive defense strategies.