Crush Cred Stuffing: A Method for Neutralizing Credential Stuffing Attacks

Crush Cred Stuffing: A Method for Neutralizing Credential Stuffing Attacks

Speakers: Alex Katz, Senior Security Researcher, ThreatX

Conference: RSAC 2024

YouTube: https://www.youtube.com/watch?v=ao_ip9GzwQ8

Overview

In this insightful talk from RSAC 2024, Alex Katz, a Senior Security Researcher at ThreatX, presented a practical and battle-tested methodology for identifying and neutralizing credential stuffing attacks. Born out of countless urgent pages and sleepless nights in a Security Operations Center (SOC) environment, Katz's approach emphasizes the crucial distinction between credential stuffing and traditional brute force attacks, highlighting the unique challenges posed by widely distributed campaigns. The core of his method revolves around synthesizing multiple data points to paint a comprehensive picture of an ongoing attack, enabling rapid and effective mitigation.

Katz's presentation goes beyond theoretical concepts, offering actionable strategies for security teams grappling with the persistent threat of account takeovers. He underscores the severe consequences of successful credential stuffing, including massive data breaches, erosion of customer trust, significant financial losses, and stringent compliance penalties. The talk provides a detailed technical deep dive into various characteristics of these attacks and how to leverage frequency distribution analysis across different data types—from IP addresses and Autonomous System Numbers (ASNs) to TLS fingerprints and HTTP request components—to detect and block malicious activity.

The significance of this talk lies in its pragmatic approach to a pervasive and evolving threat. By demystifying the anatomy of credential stuffing attacks and providing a structured detection and response framework, Katz empowers security professionals to move beyond reactive measures. He outlines both immediate tactical steps and longer-term strategic implementations, advocating for a proactive stance that leverages automation and continuous analysis to protect user accounts and organizational integrity against sophisticated adversaries.

Background

▶ Watch: Introduction and talk agenda (0:00)

Credential stuffing attacks represent a significant and growing threat in the cybersecurity landscape, distinct from traditional brute force attacks. While a brute force attack involves an attacker attempting to guess random usernames and passwords, often targeting a single account or a small set of accounts, a credential stuffing attack leverages previously compromised username and password pairs. These pairs are typically obtained from data breaches of other services and then "stuffed" into login forms of unrelated applications, banking sites, or e-commerce platforms. The underlying assumption, and unfortunate reality, is that many users reuse their credentials across multiple online services. If an attacker gains access to credentials from one breach, they can then attempt to use those same credentials to compromise accounts on hundreds or thousands of other sites, drastically increasing their odds of success.

The sources of these compromised credentials are varied and readily available to malicious actors. They are frequently found circulating on the dark web in hacker forums, or can be purchased from credential database brokers using cryptocurrencies like Bitcoin. For individuals concerned about their own data, services like Have I Been Pwned allow users to check if their email address or domain has appeared in known data breaches.

Beyond acquiring credentials, attackers also need infrastructure to launch these widespread campaigns. This often comes in the form of botnets, which are networks of compromised devices controlled by the attacker. These devices can be harvested through large-scale scanning for vulnerabilities, leading to the installation of command and control software. Alternatively, attackers can simply rent a botnet as a service on the dark web, providing a list of target hostnames and credentials to initiate an attack. A key characteristic of the attacks discussed by Alex Katz is their widely distributed nature, often spread across thousands, tens of thousands, or even millions of distinct IP addresses, making them challenging to detect and block using conventional rate-limiting or IP-based blocking rules.

The financial and reputational impact of successful credential stuffing attacks is substantial. Katz cited several recent examples: Horizon losing data for 63,000 employees (February 2024), Bank of America having 57,000 customer records exposed via a third party, Xfinity losing 35 million customer records (November 2023), and Dollar Tree/Family Dollar experiencing a breach affecting 1.97 million employees. These incidents lead to an erosion of trust, potential loss of customers to competitors, and significant compliance and legal ramifications under regulations such as GDPR and PCI DSS, often resulting in hefty fines and retroactive security expenditures.

Key Findings

▶ Watch: Brute force vs. credential stuffing attacks explained (4:45)

The central premise of Alex Katz's method for neutralizing credential stuffing attacks is that no single piece of information is sufficient for detection and blocking; instead, a comprehensive approach correlating multiple data points is essential. His experience, stemming from shutting down thousands of these attacks in a SOC, highlights several key findings:

1. Distinct Attack Patterns in Request Per Second (RPS): Credential stuffing attacks often exhibit a characteristic pattern: an enormous spike in traffic, followed by a plateau where the attack is sustained, and then a sharp decrease when the attack concludes. While marketing campaigns or new application rollouts can cause similar spikes, the sustained nature and specific characteristics differentiate credential stuffing. Katz presented two common patterns:

• Low Variability Attacks: Characterized by a stable, high RPS plateau. Example: 1,000 IPs, baseline 80 RPS, peak 3,000 RPS, sustained 1,000 RPS for 6 hours with low variability.
• High Variability Attacks: More recently observed, where attackers introduce fluctuations in RPS, possibly to evade static alerting thresholds. Example: 16,000 IPs, baseline 20 RPS, peak 520 RPS, sustained 380 RPS for 33 hours with high variability.

The critical takeaway is that baseline RPS is significantly lower than attack RPS, meaning the majority of collected data during an attack will pertain to the malicious activity.

1. Frequency Distribution as a Core Analysis Tool: Given the significant volume of attack traffic, frequency distribution analysis becomes a powerful technique. By generating tables of unique values and their counts (e.g., using uniq -c in bash or Pandas in Python), security teams can quickly identify anomalous concentrations across various data points. This allows for a data-driven approach to pinpointing attack characteristics.

1. Two Main Attack Styles:

• Low and Slow Attacks: Highly distributed, with only a single login attempt from a single IP at any one time. This effectively subverts many common counting-based or frequency-based rules designed to detect multiple failed logins from an IP.
• Few and Fast Attacks: Involve a smaller subset of IPs, each making many requests. For instance, in one example, 1,000 IPs made approximately 300 requests per IP.

1. Multi-Dimensional Data Correlation for Effective Blocking: Katz emphasized that relying on a single blocking criterion is insufficient due to attacker evasion tactics. The method advocates for correlating insights from:

• Source IPs: While direct blocking on IP count is often ineffective for widely distributed attacks, understanding the sheer volume of unique IPs is crucial.
• Autonomous System Numbers (ASNs): Often, an entire widely distributed attack originates from a surprisingly small number of ASNs (e.g., 5-10). ASN-based blocking can be highly effective, despite a small risk of false positives, especially when server resources are overwhelmed.
• TLS Fingerprinting (Ja3/Ja3S): These unique hashes derived from the client's (Ja3) or server's (Ja3S) TLS handshake provide a strong indicator of the client's software stack. Malicious bots often use non-standard or easily identifiable TLS fingerprints. However, attackers are adapting through TLS fingerprint rotation.
• User Agents: Similar to TLS fingerprints, user agents can reveal bot activity. However, attackers frequently rotate user agents, and blocking based on common user agents risks false positives (e.g., blocking "Darwin 21.4.0" would block all iPhone users).
• Endpoints and Arguments: Analyzing the sequence of endpoint requests (e.g., GET login page then POST login credentials) helps detect deviations from human behavior. Attackers may attempt to mimic human flow, but frequency analysis of arguments (form fields) can reveal repeated username/password combinations indicative of stuffing.
• Time Zones: A critical, often overlooked detail. In one successful blocking scenario, attackers were lazy and did not rotate time zones across their globally distributed IPs, allowing for targeted blocking of IPs with a specific time zone outside expected geographical regions.

These findings collectively form the foundation of a robust, multi-layered detection strategy, moving beyond simplistic rule-based systems to a more analytical and adaptive defense.

Technical Deep Dive

▶ Watch: Understanding widely distributed credential stuffing attacks (6:00)

The technical core of Alex Katz's methodology for combating widely distributed credential stuffing attacks lies in the meticulous collection and frequency distribution analysis of various metadata points associated with HTTP requests. This section elaborates on the specific data components and their application in identifying malicious traffic.

At the heart of the detection strategy is the concept of frequency distribution. This statistical method involves taking a set of data (e.g., all source IP addresses in a given time window) and counting the occurrences of each unique value. The result is a table or histogram showing unique values alongside their respective counts. Katz explained that this can be achieved with simple command-line tools like sort | uniq -c for basic data, or more sophisticated libraries like Pandas in Python for structured data (e.g., from CSV files or log aggregators). The premise is that during an attack, the malicious traffic will dominate the data, making attack-specific patterns stand out in the frequency distributions.

The key components for analysis include:

1. Source IP Addresses:

• Collection: Ideally collected at the edge of the network (e.g., load balancer, WAF). If not, the X-Forwarded-For or True-Client-IP headers can provide a chain of IPs, with the right-most IP typically being the client's original source.
• Analysis: While frequency distribution of individual IPs might show thousands of unique IPs, making direct blocking by count ineffective for widely distributed attacks, it's crucial for identifying "few and fast" attacks where a smaller number of IPs make many requests (e.g., 300 requests per IP from 1,000 IPs in one observed attack).
• Limitation: For "low and slow" attacks, where only one request comes from an IP, simple IP-based rate limiting fails. Here, analyzing time deltas between requests from the same IP can reveal bot-like consistency or unusual spacing that differentiates them from human behavior.

1. Autonomous System Numbers (ASNs):

• Definition: ASNs are unique identifiers for networks operated by Internet Service Providers (ISPs) or other large entities, assigned by bodies like IANA.
• Analysis: A powerful technique involves rolling up source IPs into their respective ASNs. Katz found that even highly distributed attacks often originate from a surprisingly small number of ASNs (e.g., 5-10).
• Actionable Insight: Identifying these dominant ASNs allows for ASN-based blocking at the WAF or network edge. While this carries a risk of false positives (blocking legitimate users sharing those ASNs), it can be an acceptable trade-off during severe attacks that are overwhelming servers, as the impact on legitimate users is often already high.

1. TLS Fingerprinting (Ja3 and Ja3S):

• Concept: Introduced at a DerbiCon talk in 2015, TLS fingerprints are hashes derived from specific parameters within the TLS handshake. They serve as a unique signature of the client's (or server's) TLS stack.
• Ja3: Focuses on the client hello message, comprising the TLS version, cipher suites, extensions, elliptical curves, and elliptical curve point formats. This information is concatenated and hashed (typically MD5) to produce a unique fingerprint. Bots often exhibit distinct Ja3 fingerprints compared to legitimate browsers.
• Ja3S: A simplified version, derived from the server hello, including the TLS version, cipher, and TLS extension. Useful for client-side validation (e.g., ensuring an app talks to the correct server).
• Caveats:
• Edge Collection: TLS fingerprints must be collected at the network edge; if behind a load balancer, the fingerprint will reflect the load balancer, not the client.
• Mobile Apps: Many mobile applications use a fixed set of TLS versions and cipher suites, resulting in identical fingerprints across many users, potentially masking bot activity or causing false positives.
• Common Browsers/OS: Standard browsers on common operating systems (e.g., Firefox on Ubuntu 24.04 LTS, Chrome on Windows) will share the same TLS fingerprints, making direct blocking risky.
• Attacker Adaptation: Attackers are now employing TLS fingerprint rotation, generating known good fingerprints and cycling through them to evade detection.

1. User Agents:

• Definition: The User-Agent HTTP header identifies the client software originating the request (e.g., browser, OS, device type).
• Analysis: Can correlate with TLS fingerprints, as they both reflect client characteristics. Attackers using the same bot software often share user agents.
• Limitation: Attackers are highly aware of user agent-based detection and frequently employ user agent rotation. Blocking specific user agents can also lead to significant false positives (e.g., blocking "Darwin 21.4.0" would block all iPhone users of a mobile app). Therefore, user agents are best used as a correlating factor rather than a sole blocking criterion.

1. Endpoints and Arguments:

• Endpoint Flow Analysis: By tracking a user's journey through an application (e.g., GET /login followed by POST /login), security teams can identify deviations from expected human behavior. Attackers often attempt to mimic this flow (e.g., 156,000 GET requests to an app's first page, followed by 243,000 POST requests to the login endpoint in one observed attack), making simple flow-based rules insufficient.
• Argument (Form Fields) Analysis: Examining the content of form fields, particularly username and password combinations, is crucial. Frequency distribution on these arguments will quickly reveal repeated username/password pairs characteristic of credential stuffing, as opposed to brute force (which would show unique passwords for a single username or random combinations).

1. Time Zones:

• Unexpected Finding: In one notable case, attackers launching a globally distributed attack failed to rotate the time zone field in their requests. Despite IPs coming from all over the world, all requests showed the same time zone.
• Actionable Insight: This "laziness" allowed the SOC to block all IPs with that specific time zone that were geographically outside the expected region (e.g., blocking all non-Asian IPs reporting an Asian time zone), effectively neutralizing the entire attack. This highlights the value of analyzing all available metadata.

This multi-faceted technical approach, combining statistical analysis with an understanding of attacker tactics, provides a robust framework for detecting and responding to credential stuffing attacks that evade simpler, single-dimension defenses.

Demo / Proof of Concept

▶ Watch: Core method: identifying attack characteristics (10:30)

While Alex Katz's talk did not feature a live, interactive demonstration of a tool or a specific proof of concept exploit, it effectively served as a detailed "method demo" based on his extensive experience in a Security Operations Center (SOC). He presented real-world attack patterns, statistics, and the analytical process his team at ThreatX employs to identify and neutralize credential stuffing.

Katz described how his SOC, protecting hundreds of customers and thousands of applications, uses their product and custom scripts to implement the discussed methodology. He detailed how alerts trigger a process where Python scripts are run to aggregate data from log sources. These scripts collect key pieces of information—such as source IPs, ASNs, TLS fingerprints, user agents, endpoints, arguments, and time zones—from a specific look-back period (e.g., 15-25 minutes around the alert time). This aggregated data is then dumped into a centralized location, pre-analyzed with frequency distribution tables, and presented to a SOC analyst.

This automated data preparation acts as the "proof of concept" for the method's efficiency. It allows an analyst, upon receiving an urgent page, to immediately access and review the critical data points necessary to understand the attack's characteristics and make informed blocking decisions without manual data collation. This significantly reduces response time and the risk of account takeovers. The examples provided throughout the "Technical Deep Dive" section, such as the specific RPS values, IP counts, and the time zone anomaly, illustrate the practical application and effectiveness of this data-driven approach in real-world scenarios.

Defensive Implications

▶ Watch: Identifying the most common attack pattern (Attack One) (12:00)

The detailed methodology presented by Alex Katz offers a comprehensive roadmap for organizations to bolster their defenses against credential stuffing attacks, encompassing both immediate tactical responses and longer-term strategic initiatives.

Immediate Actions:

1. Enforce Multi-Factor Authentication (MFA) on Everything: This is paramount. As Katz emphatically stated, "It's 2024. Everybody should have MFA on everything." Even if an attacker obtains valid credentials, MFA acts as a crucial second line of defense, preventing account takeover.
2. Promote and Enforce Password Manager Usage: Encouraging or mandating the use of password managers ensures that users create unique, strong, randomized passwords for each service. This directly neutralizes the core premise of credential stuffing: that reused credentials will grant access to multiple accounts.
3. Catalog Web-Facing Endpoints: Organizations must have a clear inventory of all login endpoints, API endpoints, and any other public-facing interfaces that require authentication. Security and development teams should collaborate to identify and secure these, addressing any existing backlogs of known vulnerabilities or misconfigurations.

Longer-Term Strategies (3+ Months):

1. Implement Traffic and Frequency Analysis Training: Security teams need to be proficient in analyzing request per second (RPS) metrics, match events per second, and other traffic patterns. This involves training on how to interpret spikes, plateaus, and variability in traffic.
2. Develop Rate-of-Change Based Alarming: Instead of relying on static thresholds (e.g., "alert if RPS > 1000"), which can be evaded by variable attacks or cause false positives during legitimate spikes, organizations should implement alarms based on the rate of change in traffic. This allows for detection of sudden, anomalous increases regardless of absolute volume.
3. Deploy Challenge-Response Tests: Integrating solutions like reCAPTCHA (Google's reCAPTCHA v3 or Enterprise), hCaptcha, or other puzzle-based challenges can effectively block automated bots while minimizing friction for legitimate users. These tests are designed to differentiate between human and automated interactions.
4. Automate Data Aggregation and Analysis: This is a cornerstone of Katz's method. Organizations should develop or acquire tools (like the Python scripts used by ThreatX's SOC) to automatically collect, centralize, and perform frequency distribution analysis on key data points (IPs, ASNs, TLS fingerprints, user agents, endpoints, arguments, time zones) when an alert is triggered. This pre-digested information empowers SOC analysts to take immediate, informed action, reducing response times and mitigating risk.
5. Conduct Security Awareness Training: While often perceived as a basic measure, regular and effective security awareness training for all employees reinforces the importance of strong passwords, MFA, and vigilance against phishing, which can be initial vectors for credential compromise.

Insights from Audience Q&A:

The Q&A session further highlighted practical challenges and advanced defensive considerations:

• Handling NAT IPs: A common issue for rate-limiting is distinguishing between a single attacking IP and a legitimate enterprise using Network Address Translation (NAT), where many users appear to originate from one IP. Katz suggested analyzing time deltas between requests from the same IP (to detect bot-like consistency) and leveraging TLS fingerprints as potential differentiators.
• Dedicated Bot Mitigation Solutions: An audience member from Navy Federal Credit Union emphasized the difficulty of solving this problem in-house and recommended investing in dedicated bot mitigation solutions from vendors. These solutions often provide advanced capabilities like device interrogation, behavioral analysis, and adaptive blocking that are hard to replicate with custom scripts alone, especially given evolving attacker tactics like TLS fingerprint rotation and user agent rotation.
• Secure Password Storage: A fundamental concern raised was how attackers obtain credentials in the first place. The speaker reiterated that weak passwords and lack of password managers are key enablers. The audience member further stressed the unforgivable practice of sites storing passwords in an easily usable form, even if encrypted. Katz acknowledged the trade-offs in hashing complexity (e.g., strong algorithms being computationally expensive for high-volume logins) but agreed on the necessity of proper salting and strong hashing to prevent easy decryption of stolen password databases.
• Physical Security Keys (FIDO2): When asked about the ultimate solution for authentication, Katz strongly endorsed physical security keys (e.g., FIDO2, WebAuthn) as an "extraordinarily more reliable" form of multi-factor authentication, though not 100% infallible.

In essence, a robust defense against credential stuffing requires a multi-layered strategy that combines strong foundational security practices, sophisticated analytical tools, automation, and a willingness to adapt to evolving attacker methodologies, potentially leveraging specialized vendor solutions for advanced bot detection.

Key Takeaways

• Credential Stuffing is Distinct and Dangerous: It leverages previously breached credentials against new targets, differing fundamentally from brute force attacks, and leading to severe consequences including data breaches, financial loss, and compliance penalties.
• Correlation, Not Isolation, is Key: Effective detection and blocking require correlating multiple data points—including source IPs, ASNs, TLS fingerprints, user agents, endpoints, arguments, and even time zones—to form a holistic view of the attack, as no single indicator is sufficient.
• Frequency Distribution is Your Ally: Leveraging frequency distribution analysis on various metadata components helps quickly identify anomalous patterns and concentrations indicative of bot activity during high-volume attacks.
• Automate for Speed and Efficiency: Implementing automated data aggregation and analysis (e.g., using Python scripts) to pre-process attack data for SOC analysts is crucial for rapid response and minimizing the window of opportunity for account takeovers.
• Foundational Security is Non-Negotiable: Enforcing multi-factor authentication (MFA) on all accounts, promoting password manager usage, and maintaining a comprehensive catalog of web-facing endpoints are fundamental defenses that significantly reduce the attack surface.
• Defenses Must Evolve with Attackers: Attackers are constantly adapting, employing tactics like TLS fingerprint rotation, user agent rotation, and mimicking human user flows. This necessitates dynamic detection strategies, rate-of-change based alarming, and potentially dedicated bot mitigation solutions to stay ahead.

About the Speaker(s)

Alex Katz is a Senior Security Researcher at ThreatX. His expertise in combating fraudulent stuffing attacks stems directly from extensive, hands-on experience within a Security Operations Center (SOC). In this role, he was responsible for protecting hundreds of customers and thousands of applications, often facing urgent pages and working late nights to shut down these pervasive threats. His practical, real-world experience forms the foundation of the methodologies and insights shared in his RSAC 2024 talk, demonstrating a deep understanding of attacker tactics and effective defensive strategies.