Pen Testing for NOT Dummies

Pen Testing for NOT Dummies

Speakers: Alex Holden, Nike professional, Cybercrime Hunter

Conference: BSides NYC

YouTube: https://www.youtube.com/watch?v=hk6v6PgkU64

Overview

In his talk, "Pen Testing for NOT Dummies," Alex Holden delivers a compelling and experience-driven critique of conventional penetration testing methodologies, advocating for a more complex, thorough, and intellectually driven approach. Dispelling the notion of "pen testing for dummies," Holden, a seasoned cybersecurity professional with over 10,000 pen tests under his belt, emphasizes the critical role of curiosity, deep technical understanding, and adaptability in uncovering real-world vulnerabilities. The presentation goes beyond surface-level compliance checks or automated vulnerability scans, pushing practitioners to adopt the mindset of a persistent adversary.

Holden's insights are particularly relevant in an era where cyber threats are increasingly sophisticated and diverse. He argues that while compliance and vulnerability scanning have their place, they often fall short of identifying the nuanced attack vectors exploited by cybercriminals. The talk serves as a practical guide for aspiring and experienced pen testers alike, offering actionable advice on scoping, tool customization, information gathering, and ethical considerations. His unique perspective, honed through years of hunting cybercriminals and even recovering stolen assets, imbues the discussion with a sense of urgency and real-world applicability that distinguishes it from more theoretical treatments of the subject.

The significance of Holden's message lies in its call for a paradigm shift in how organizations approach security assessments. By highlighting the limitations of checklist-based testing and the need for continuous learning and creative problem-solving, he challenges the industry to elevate its standards. The talk underscores that effective penetration testing is not merely a technical exercise but an art form demanding an attacker's mindset, a defender's diligence, and a relentless pursuit of every possible weakness to truly secure digital assets against determined adversaries.

Background

▶ Watch: Introduction: Pen Testing for NOT Dummies (0:00)

The landscape of cybersecurity has evolved dramatically since the early days of penetration testing. Alex Holden recounts his first pen test in 1997, a time when tools like nmap were just emerging. His anecdote about discovering a "missing" firewall – which was physically disconnected and stored in a closet – perfectly illustrates the foundational challenge: security measures are only effective if properly implemented and maintained. This initial experience set the stage for a career dedicated to understanding not just the technical vulnerabilities, but also the human and procedural flaws that adversaries exploit.

Holden meticulously distinguishes penetration testing from related, but distinct, security assessments:

• Compliance: While essential for establishing best practices and adherence to regulations (like driving rules), compliance alone does not guarantee security. It defines the lines, but pen testing explores how to bend or break them.
• Vulnerability Testing: This focuses on identifying known weaknesses, often associated with CVEs (Common Vulnerabilities and Exposures). While valuable, it's a reactive approach that doesn't necessarily discover novel attack paths or deeply embedded misconfigurations. It's about "classical vulnerabilities," not comprehensive discovery.
• Red Teaming: Often likened to "capture the flag," red teaming aims to achieve a specific objective using any means necessary. A red team might find one way in and then pivot, demonstrating impact. However, Holden argues that a true pen test demands thoroughness – finding all possible ways in, because "bad guys are like panthers; they just don't share pen test results with you." Defenders, or the blue team, must be 100% right, while attackers only need to find one successful entry point.

This distinction underpins Holden's philosophy for the "ideal pen tester." Beyond technical prowess, he emphasizes Natural Curiosity as paramount. An effective pen tester must possess a deep understanding of technologies (protocols, infrastructure), akin to a developer who can deconstruct systems. System administration experience is crucial for versatility across web applications, Active Directory (AD), and Linux environments. Finally, developer skill sets or scripting capabilities are increasingly vital for automation, allowing testers to move beyond rigid tool-driven approaches and customize their attacks. This holistic skill set is essential for navigating the complexities of modern digital infrastructures and truly uncovering their hidden weaknesses.

Key Findings

▶ Watch: First pen test story: The missing firewall appliance (5:30)

Alex Holden's talk unveils several critical findings that challenge conventional pen testing wisdom and advocate for a more rigorous, intelligent, and adaptable approach:

1. Thoroughness Over Superficiality: The most significant finding is the imperative for thoroughness. Unlike red teaming which might stop after achieving an objective, a pen tester's goal is to discover all possible entry points. This stems from the fundamental asymmetry in cybersecurity: attackers only need one vulnerability, while defenders must secure every single one.
2. The Flawed "Nerd Test Scope": Holden highlights the common mistake of setting up unrealistic test environments. Testing against an "empty system" or a QA environment with disabled modules (as illustrated by the McAfee EPO example where over 100 critical vulnerabilities were missed because modules were not enabled during testing) leads to a false sense of security. Testers must insist on environments that mirror production, complete with data and active features.
3. Human Intelligence Trumps Automated Tools: While tools are essential, relying solely on their default settings or assuming they are comprehensive is a critical error. Holden stresses that the "brain is the biggest tool for pen testing." Customizing open-source tools like DerBuster (which hasn't been updated since 2009 but can be continually improved by the user) and understanding their underlying methodologies is key. He points out that commercial tools often lack transparency in their testing approaches.
4. The Power of Information Gathering (OSINT & Internal Knowledge): Effective pen testing starts long before active scanning. Holden advocates for extensive Open Source Intelligence (OSINT) gathering, including scouring GitHub for exposed code, employee repositories, and even passwords. Crucially, he argues that asking for internal documentation—like routing tables, DNS entries, virtual hosts, and file structures—is not cheating, but rather a way to accelerate and deepen the assessment, mimicking an attacker who would eventually uncover this information.
5. Ethical Use of Stolen Credentials: A nuanced finding concerns the use of credentials found on the Dark Web. While such credentials can offer a quick entry point, Holden strongly advises against paying for them (to avoid empowering cybercriminals) and against using them beyond proof-of-concept. Using stolen credentials for active testing can destroy forensic evidence, doing a "huge disservice to the victim." Instead, the proper course is to inform the client immediately.
6. Uncovering Hidden Data in HTTP 302 Redirects: A specific technical insight is the analysis of HTTP 302 Redirect responses. While often small, indicating a simple redirection, Holden's team found dozens of vulnerabilities by looking for larger 302 responses (10KB, 100KB+). This indicates that the server might be rendering the entire page before redirecting, allowing an attacker to change the 302 status code to 200 OK and access sensitive data that was never intended to be seen.

These findings collectively underscore the need for pen testers to think critically, act creatively, and maintain a relentless focus on realism and thoroughness, mirroring the persistence of real-world adversaries.

Technical Deep Dive

▶ Watch: Distinguishing pen testing from compliance and vulnerability scans (6:00)

Holden's technical deep dive spans the entire pen testing lifecycle, from scoping and tool selection to advanced exploitation techniques across diverse environments.

Types of Penetration Tests

He clarifies the nuances of different testing approaches:

• Black Box: Mimics an external attacker with zero prior knowledge. While realistic, it can be time-consuming and may miss internal-facing vulnerabilities.
• Gray Box: The most common approach, where basic knowledge or user-level access is provided. This is crucial for testing authenticated functionalities, as many modern applications require some form of login. Testers can evaluate if a basic user can escalate privileges or access unauthorized data.
• White Box: Offers full access to both front-end and back-end code and infrastructure. While not reflective of a typical external attacker's initial access, it's invaluable for root cause analysis. Holden suggests deploying a sniffer or logging agent on a SQL server during a white box test to observe the exact impact of SQL injection attempts, revealing whether signals are reaching the database even if the front-end doesn't immediately show an error. This level of insight helps understand how vulnerabilities truly manifest.

Scoping a Pen Test

Proper scoping is paramount. Holden advises:

• External vs. Internal: Conduct external tests first to simulate an outside attacker. Internal tests provide a different perspective.
• Third-Party/Cloud: Be mindful of specific rules and permissions from providers like Azure, Google Cloud, or AWS.
• Avoiding "Nerd Test Scope": Do not accept limitations to "test-only" or "QA" environments if they are empty or have disabled features. The McAfee EPO example perfectly illustrates this, where testing a framework without its active modules led to a severe underestimation of vulnerabilities. Similarly, testing with empty user accounts prevents discovery of data access issues.
• Don't Exclude Similar Environments: Knowledge gained from a less protected Dev or QA environment (e.g., naming conventions) can often be applied to a Production environment. Excluding these unnecessarily limits the tester's advantage.
• Blacklists/Whitelists: While necessary for stability, relying solely on these to exclude problematic IPs during a test is a defensive weakness. Critical systems should be made stable enough to withstand testing, not simply excluded.

Tool Usage and Customization

Holden strongly advocates for intelligent tool use:

• Evaluate Open Source Tools: Understand their function, history, and community support. Tools like DerBuster (last updated 2009) and its Linux equivalent DRB (last maintained 2011) can still be powerful if customized with updated keywords and tweaked code.
• Don't Delegate Control: Commercial tools can be user-friendly, but testers must understand their methodologies rather than blindly trusting their reports.
• Beyond Defaults: Automated tools often use default cookies or follow all redirects. Changing these defaults can bypass Web Application Firewalls (WAFs) that detect automated scans.
• The 302 Redirect Trick: Instead of blindly following HTTP 302 Redirects, analyze their size. A large 302 response (e.g., 10KB, 100KB) suggests the server rendered a full page before redirecting. By modifying the HTTP response code from 302 to 200, an attacker can sometimes force the page to render fully, revealing sensitive data. This is a default DerBuster behavior that needs to be overridden to discover such issues.
• Tool Diversity: Relying on a single tool like Nessus is insufficient. Different tools, libraries, and custom scripts are needed based on speed, return types, and target defenses.

Information Gathering and Reconnaissance

Before any active testing, comprehensive information gathering is critical:

• Ask Smart Questions: Request IP address ranges, routing tables, all DNS entries (which might reveal forgotten subdomains or external assets), virtual hosts, application endpoints, APIs, and even file structures in the web root.
• OSINT Deep Dive: Beyond basic Google searches, scour GitHub for company repositories, employee projects, and forgotten credentials. The speaker mentions finding passwords, the Anthem of Ukraine, family pictures, and links to an internal GitLab instance in a single JavaScript file.
• Dark Web: While controversial, compromised credentials on the dark web can prove a system's vulnerability. However, ethical guidelines dictate not paying for them and reporting them to the client for their own investigation, rather than using them extensively and potentially destroying forensic evidence.
• Application Demos: If internal knowledge is withheld, pretend to be a customer and request a demo to understand the application's functionality and endpoints.
• Web Logs: Analyzing web logs (especially GET requests) can provide an invaluable map of a website's structure and unauthenticated endpoints.

Exploitation Techniques Across Environments

Holden provides a laundry list of common and overlooked attack vectors:

• Web Applications:
• Data Flow Analysis: Understand how data moves between servers and cloud components.
• Authentication/Authorization: Test logout processes, horizontal privilege escalation (user A accessing user B's data), and vertical escalation (normal user making themselves admin). Holden claims 50% of today's applications are vulnerable to simple admin panel call replay for privilege escalation.
• JavaScript Analysis: Parse JavaScript files for hidden endpoints, API keys, and internal infrastructure details.
• SQL Injection: Still prevalent, often requiring intelligent crafting beyond simple tools.
• Active Directory (AD):
• Older Technologies: Often less patched and more vulnerable.
• Writable Network Scripts: Identifying scripts executed by all users (especially as admin) allows for easy privilege escalation.
• Patching and Misconfigurations: Look for missing patches and misconfigured settings.
• WAF Bypass: In some cases, gaining internal access allows for disabling security tools like a WAF for the tester's IP.
• Networking:
• Understand protocols and management platforms.
• Forgotten Management Systems: These are prime targets, often having default credentials or demo licenses that can be bypassed (e.g., blocking communication to a license server to re-enable functionality and access stored AD credentials). The SolarWinds incident serves as a stark reminder of the criticality of securing management platforms.
• End-User Devices: IT devices are often less secured, offering easy entry.
• Silverlight: Even antique technologies like Silverlight (only running in Internet Explorer, now often via Edge's IE mode) can be exploited if still in use.
• IoT Devices:
• Often overlooked, printers can store network passwords.
• Cameras (e.g., in banks) can be disabled.
• Environmental controls and control panels are often left unattended.
• Meeting room tablets (e.g., for room reservations) can sometimes reveal internal Wi-Fi passwords if misconfigured.
• Password Reuse: Common across IoT devices, leading to rapid lateral movement.
• Cloud Pen Testing:
• Map Endpoints: The transient nature of cloud IPs requires diligent mapping.
• Rapid Exposure: Cloud assets, if misconfigured, can be discovered and exploited within hours, not months.
• Management Consoles & Code Repositories: Services like Bitbucket, GitHub, GitLab, Jira, Zendesk, and Confluence hosted in the cloud are frequent sources of exposed data and credentials.
• Advanced Topics:
• Broken Links/Domains: Can be hijacked by attackers.
• Impersonation: Leveraging social engineering or compromised accounts.
• Security Tool Testing: While pen testing, observe if Intrusion Detection Systems (IDS), SIEMs, or honeypots detect the activity. This helps assess the defensive posture.
• Phishing & Ransomware Simulation: Test not just users but also the technology's ability to stop malicious emails and detect data exfiltration.

Demo / Proof of Concept

▶ Watch: Pen testing vs. Red Teaming: Thoroughness is key (8:00)

While Alex Holden's talk did not feature a live, step-by-step technical demonstration in the traditional sense, he effectively presented numerous real-world anecdotes and practical examples that serve as compelling proof-of-concept scenarios. These illustrative stories highlight common vulnerabilities, misconfigurations, and the effectiveness of his recommended methodologies.

Key examples that functioned as "demos" through narrative include:

• The Missing Firewall (1997): His inaugural pen test revealed a firewall physically removed from the network and stored in a closet. This demonstrated a fundamental failure in implementation and asset management, underscoring that even advanced security hardware is useless if not deployed correctly.
• McAfee EPO Vulnerabilities: Holden detailed how a pen test against McAfee's ePolicy Orchestrator (EPO) product uncovered over 100 critical vulnerabilities. The crucial insight here was that the vendor's internal pen test missed these because they had tested only the basic framework with no modules enabled. This illustrated the danger of unrealistic testing environments and the necessity of testing all active features.
• 302 Redirect Data Leakage: Holden explained how analyzing the size of HTTP 302 Redirect responses could reveal sensitive data. If a 302 response is unusually large (e.g., 10KB or 100KB), it indicates the server rendered the full page before redirecting. By manually changing the response code to 200 OK, his team was able to access data that was supposed to be restricted, demonstrating a common web application flaw.
• JavaScript Information Disclosure: He recounted a recent finding where a single JavaScript file contained the Anthem of Ukraine, family pictures, and links to an internal GitLab instance, along with potential passwords. This showcased the rich trove of sensitive information often inadvertently exposed within client-side code.
• Writable Network Scripts in Active Directory: Holden described how, with basic user access to an Active Directory environment, identifying and modifying writable network scripts (especially those executed with administrative privileges) could lead to easy privilege escalation, turning a standard user into an administrator.
• Forgotten Management System Bypass: A particularly inventive "demo" involved a forgotten management system server with an expired demo license. By blocking the server's outbound communication to its licensing server, the application "thought" it was licensed. Combined with default logins that were never changed, this granted access to the company's AD credentials for an administrator account, highlighting the risks of unmaintained and forgotten infrastructure.
• Silverlight Exploitation: He mentioned a recent exploitation of a system using Silverlight, an antique technology. This demonstrated that even deprecated platforms, if still active, remain viable targets and require attention, often requiring specific legacy browser modes (like Edge's Internet Explorer compatibility mode) to interact with.
• Meeting Room Tablet Wi-Fi Disclosure: A simple example involved interacting with a meeting room reservation tablet. Through a few clicks on a misconfigured device, it was possible to reveal the internal Wi-Fi password, showcasing how seemingly innocuous IoT devices can expose critical network access.

These examples, drawn from extensive real-world experience, served as powerful conceptual demonstrations, proving the validity of Holden's "not for dummies" approach to penetration testing.

Defensive Implications

▶ Watch: Qualities of an ideal pen tester: Curiosity and technical skills (8:40)

Alex Holden's talk provides critical insights for defenders looking to strengthen their security posture against sophisticated attacks. The core defensive implications revolve around adopting a more proactive, thorough, and intelligent approach to security, mirroring the mindset of a persistent adversary.

1. Demand Realistic Pen Tests: Organizations must insist on pen tests that accurately reflect their production environment. This means ensuring that all features and modules are enabled, test data is present, and environments are not artificially isolated or "sanitized." If a pen test is scoped to an empty QA system, the results will provide a false sense of security, as demonstrated by the McAfee EPO example.
2. Comprehensive Asset Inventory and Management: Defenders need to have an exhaustive understanding of their digital footprint. This includes not only obvious production systems but also forgotten servers, development environments, cloud endpoints, and all IoT devices. Holden's examples of forgotten management systems or printers storing network passwords underscore that these "out of sight, out of mind" assets are often the easiest entry points. Regular audits for default credentials and unpatched systems are essential.
3. Beyond Automated Scanners: While vulnerability scanners are useful, they are not a substitute for human intelligence and customized testing. Defenders should understand the limitations of their security tools and not rely solely on their default configurations. Implementing advanced logging and monitoring that can detect customized attack patterns, not just known signatures, is crucial.
4. Secure Development Lifecycle (SDL): The prevalence of SQL injection, privilege escalation flaws, and information leakage in JavaScript highlights the need for robust security practices throughout the development process. Developers must be educated on secure coding principles, and security reviews should be integrated into every stage, not just as a final check.
5. Robust Information Security Governance: Review and tighten access controls, especially for internal documentation. While pen testers may request routing tables or file structures, these should not be easily accessible to unauthorized individuals. Implement strong policies around credential management, preventing hardcoded passwords in code repositories like GitHub or GitLab.
6. Ethical Handling of Compromised Data: Establish clear protocols for handling information about stolen credentials found on the Dark Web. Organizations should have a process to verify these credentials and initiate remediation (e.g., forced password resets, multi-factor authentication enforcement) without destroying potential forensic evidence.
7. Continuous Monitoring and Incident Response: Holden emphasizes testing the organization's Intrusion Detection Systems (IDS), SIEMs, and honeypots during a pen test. This validates whether security operations teams can actually detect and respond to real-world attacks. Organizations should foster a culture where security alerts are taken seriously and investigated, rather than being ignored or dismissed as "just a pen test."
8. Proactive OSINT Monitoring: Implement continuous monitoring of public sources, including GitHub, Pastebin, and other developer platforms, for inadvertent exposure of sensitive company information, API keys, or internal configurations.
9. Network Segmentation and Protocol Awareness: Understanding network protocols and implementing proper segmentation can limit lateral movement even if an initial breach occurs. Securing management interfaces for network devices and applications is paramount, as they are often targeted due to default or weak credentials.
10. Patch Management Discipline: Maintain rigorous patch management across all assets, including legacy systems like Silverlight or management platforms. Outdated software is a perennial favorite for attackers.

By embracing these defensive implications, organizations can move beyond a "dummies" approach to security and build a resilient defense capable of withstanding persistent and sophisticated cyber threats.

Key Takeaways

• Penetration testing is about thoroughness and curiosity, not just compliance. True security assessments aim to find all possible entry points, mimicking a persistent attacker, rather than simply checking boxes or demonstrating a single breach.
• Realistic scoping and environment setup are paramount. Testing against empty systems, disabled modules, or artificially segregated environments provides a false sense of security. Pen tests must reflect production conditions to be effective.
• Human intelligence and customization are superior to blind tool usage. While tools like DerBuster are valuable, the "brain is the biggest tool." Testers must understand tool methodologies, customize settings, and think creatively (e.g., analyzing large 302 redirects) to uncover hidden vulnerabilities.
• Comprehensive information gathering (OSINT and internal documentation) is crucial. Leveraging public sources (GitHub, Dark Web) and requesting internal details (routing tables, DNS entries) significantly enhances a pen test's effectiveness and efficiency, accelerating discovery.
• Don't overlook "forgotten" or seemingly innocuous assets. Old systems (e.g., Silverlight), unpatched management platforms (like those involved in SolarWinds), or IoT devices (printers, meeting room tablets) are often easy targets due to default credentials or misconfigurations.
• Ethical conduct is non-negotiable. When encountering stolen credentials or sensitive data, pen testers must prioritize reporting to the client over extensive exploitation, to avoid empowering cybercriminals or destroying forensic evidence.

About the Speaker(s)

Alex Holden is a distinguished cybersecurity professional, who self-identifies as a "Nike professional." Born in Ukraine, he has been a resident of the United States since 1989. Holden emphasizes that he has never been a "hacker" in the malicious sense, but rather a cybersecurity expert driven by an innate curiosity about how technology works. His career trajectory saw him questioning the correctness and bug-free nature of commercial software in the early 2000s. For the past decade and a half, he has dedicated himself to observing the methodologies of hackers and actively hunting cybercriminals. Holden is recognized for significant successes against cybercrime, including an instance highlighted in Forbes Magazine where his efforts led to the recovery of Bitcoin from Russian drug lords, which was then directed to Ukrainian charities. With a wealth of experience, he and his team have conducted over ten thousand penetration tests for companies of all sizes, making him a seasoned authority on the subject.