Maximizing the ROI of Your Pentest

Maximizing the ROI of Your Pentest

Speakers: Xync (Meristem InfoSec, Managing Principal), Brian (CISO, ksl.com), John (Pentester, IBM)

Conference: SAINTCON

YouTube: https://www.youtube.com/watch?v=nOmVlSfR5qg

Overview

In this insightful SAINTCON talk, a unique panel consisting of two brothers who are experienced pentesters (Xync and John) and their younger brother, a CISO (Brian), converge to discuss how organizations can maximize the return on investment (ROI) of their penetration testing efforts. The presentation offers a dual perspective: what pentesters need to be effective, and what clients should consider to get the most value from their security assessments. This convergence of viewpoints from both sides of the pentest engagement provides a comprehensive and practical guide for improving security posture.

The speakers highlight that while many organizations conduct pentests due to compliance mandates or external pressures, the true value lies in proactively understanding and mitigating security risks. They argue that a well-executed pentest, approached strategically with clear goals and effective communication, serves as a crucial summative assessment of an organization's security controls. By dissecting the process into "before, during, and after" phases, the talk provides actionable advice for optimizing every stage of a pentest, ensuring that the investment translates into tangible security improvements rather than just a compliance checkbox.

The core message revolves around the idea that a successful pentest is a collaborative effort, not an adversarial one. It emphasizes transparent communication, realistic expectations, and a blameless approach to findings, ultimately aiming to foster a stronger security culture and a more robust defense against real-world attackers. The speakers leverage their extensive combined experience to offer candid advice, peppered with real-world anecdotes that underscore the importance of their recommendations.

Background

▶ Watch: Introduction and unique speaker dynamic (pentesters/CISO) (0:00)

The necessity of penetration testing often stems from a variety of pressures and motivations. The speakers begin by outlining the common reasons organizations undertake pentests, emphasizing that understanding the underlying "why" is crucial for shaping the test and maximizing its effectiveness.

Compliance: This is frequently cited as the number one driver. Regulations like SOC 2, PCI DSS, HIPAA, or other industry-specific standards often mandate periodic penetration tests. While compliance is a necessary baseline, merely checking this box without deeper engagement can lead to a low ROI.

External Validation: For startups launching new SAS services or companies engaging in mergers and acquisitions, a pentest report provides independent validation of security posture, often a prerequisite for securing large clients or closing deals. This external assurance builds trust and demonstrates due diligence.

Self-Assessment: Mature organizations use pentests to evaluate the efficacy of their existing security strategies and investments. After implementing numerous controls, a pentest acts as a realistic simulation of an attack, revealing how well those defenses hold up against determined adversaries. It helps identify gaps and validate the effectiveness of previous remediation efforts.

Political Elements: Less discussed but equally prevalent are the internal political motivations. Security teams might commission a pentest to demonstrate their efforts to leadership, justifying past investments and showcasing their diligence. Conversely, a CISO or security manager might intentionally seek a "failing pentest" to secure increased budget for tools, personnel, or initiatives, using the findings as leverage. In rare cases, pentests might even be used to expose weaknesses in another team's work. Finally, some organizations face "use-it-or-lose-it" budget scenarios at year-end, leading to rushed or less strategic pentest engagements. The speakers recount an anecdote where a customer requested hacking "industry standard best of breed equipment" not because they expected findings, but because their "C chain" was worried about supply chain attacks and needed to spend budget, highlighting the diverse, sometimes unusual, motivations.

Regardless of the initial impetus, the overarching goal should be to gain a holistic understanding of an organization's security risks. A penetration test, by simulating real-world attacker behaviors, offers a unique and highly valid way to assess an organization's actual defensive capabilities and identify critical vulnerabilities that automated scans might miss.

Key Findings

▶ Watch: The core question: Why get a penetration test? (4:50)

The talk's key findings revolve around optimizing the pentest lifecycle through strategic preparation, proactive communication, and intelligent utilization of results. For both clients and pentesters, maximizing ROI means moving beyond a mere compliance exercise to a collaborative and insightful security assessment.

1. Define Clear Goals and Scope: Before anything else, clients must articulate why they are getting a pentest. This drives all subsequent decisions. The scope must be carefully defined, ensuring that critical inter-component links are tested and avoiding overly narrow boundaries that miss potential attack vectors.
2. Strategic Information Sharing: While a real attacker has no prior knowledge, providing pentesters with relevant information (e.g., source code, network diagrams, business context) significantly increases efficiency and allows them to focus on exploitable weaknesses rather than reconnaissance. This should be annotated in the report to maintain realism.
3. Prioritize Production-Like Testing: Whenever possible, testing in a production environment or a highly accurate staging environment with dummy data yields the most realistic results. While risky, this approach uncovers vulnerabilities that might be masked by different controls in a less representative dev environment.
4. Thorough Access Control Testing: Effective testing of horizontal access control (user A accessing user B's data) and vertical access control (regular user accessing admin functions) requires multiple user IDs per role and per tenant. Restricting this limits the thoroughness of the test.
5. Establish Clear Rules of Engagement: Define expectations regarding testing hours, the handling of critical findings, and sensitive procedures like password cracking. Proactive discussions prevent misunderstandings and potential disruptions.
6. Constant Communication is Paramount: During the test, clients should expect and promptly answer questions from pentesters. Pentesters, in turn, should provide regular updates. This fosters trust, expedites the test, and prevents "surprises" for leadership.
7. Blameless Remediation and Conceptual Fixes: The post-test readout call should focus on productive remediation, not blame. Clients should prepare their teams for findings and encourage fixing the underlying concept of a vulnerability (e.g., all cross-site scripting instances) rather than just the specific reported instances.
8. Leverage External Expertise: Pentesters can serve as external experts to help clients communicate findings effectively to management, amplifying the message and advocating for necessary security investments.

These findings collectively emphasize that a successful pentest is a journey of continuous improvement, demanding active participation and collaboration from both the testing team and the client organization.

Technical Deep Dive

▶ Watch: Common drivers: compliance, new clients, M&A, self-testing (5:15)

While this talk focuses heavily on process and communication, several technical aspects of penetration testing are discussed in detail, offering insights into how these elements contribute to the overall effectiveness and ROI.

Information Disclosure vs. Realism: A recurring theme is the balance between giving pentesters information and simulating a real attacker. Pentesters argue that providing source code or detailed network diagrams significantly speeds up the process. For instance, inspecting specific lines of code related to authentication or cookie generation can reveal vulnerabilities in minutes that would otherwise take hours of black-box experimentation. The key is for pentesters to annotate their reports, clarifying which findings would have been difficult or impossible to discover without the shared information, thus providing a realistic risk assessment. This transparency ensures that the client understands the true exploitability context.

Understanding Business Context for Severity: The speakers emphasize that technical findings must be contextualized by the business impact. An example is given of a federal client moving billions of dollars. A potential vulnerability in the money transfer system might seem critical, but if the user base is so small that any anomalous transfer would immediately trigger a phone call to "Jim" to send the money back, the real-world severity is drastically reduced. Without this business context, pentesters might assign an incorrect severity, leading to misprioritized remediation efforts. This highlights the technical necessity of understanding the "what if" scenarios from a business perspective.

Defining and Testing Scope: Technically, defining the scope is critical. If a client subdivides an environment, such as a single application that communicates with multiple backend services, and only tests individual components in isolation, crucial inter-component vulnerabilities can be missed. The speakers cite an example where a single website might be shared by three different companies, making scope definition challenging. In such cases, the client must communicate these complexities, allowing the pentester to perform thorough reconnaissance and validate assets before proceeding. This can prevent issues, such as a social engineering campaign accidentally targeting the wrong company's employees.

Source Code and Network Diagrams: For application pentesting, providing source code is highly recommended. While it's not a full code review (a more expensive service), it allows pentesters to quickly understand complex logic, especially around areas like custom authentication or data handling. This enables them to craft more precise payloads and identify vulnerabilities faster. Similarly, network diagrams accelerate network pentests by guiding testers to critical infrastructure, though pentesters should still explore to find undocumented assets or out-of-date diagrams.

Testing in Production vs. Development: The debate between testing in production environment versus a dev environment has significant technical implications. While dev environments offer safety, they often lack the exact configurations, data, or supplementary controls present in production. The famous "negative gift card" story illustrates this: a finding in a dev environment was dismissed until a production test confirmed that a negative value gift card could reduce a purchase to $1, bypassing other supposedly protective systems. This demonstrates that for certain critical findings, validating in production (with appropriate risk management and client approval) provides the definitive technical answer. However, the risk of data compromise (e.g., credit card numbers on a tester's laptop) or service disruption makes dummy data in a production-like staging environment the preferred technical approach for many.

Multi-User IDs for Access Control: To effectively test access control vulnerabilities, pentesters require multiple user IDs per role and per tenant. Without these, it's impossible to perform horizontal access control tests (e.g., can User A view User B's documents by changing an ID?) or vertical access control tests (e.g., can a regular user access an admin dashboard by guessing a URL?). The speakers criticize approaches that limit testers to only two IDs, deeming such tests fundamentally flawed as they cannot thoroughly assess these critical security controls. They also highlight that an admin can almost always "hack their own site," so pentesters should focus on what regular users can achieve, using admin accounts primarily for discovery.

Rules of Engagement and Sensitive Tests: Technical rules of engagement include defining when destructive or high-impact tests (e.g., attempting a full database dump) can occur. For instance, scheduling such tests during business hours allows immediate support if a system crashes. The talk also delves into the technical and PR implications of password cracking. While cracking a password list from a compromised customer database can reveal the prevalence of weak passwords (useful for internal awareness), it creates a PR nightmare if clients have to notify users. A technical compromise might be to provide the pentester with a known user ID and password for testing, avoiding the mass cracking and notification issue.

WAF Effectiveness and Bypass: The speakers challenge the common perception of WAFs (Web Application Firewalls). From a pentester's technical perspective, a WAF rarely poses more than a minor inconvenience. The issue is that WAFs are often deployed with generic rules and are "rarely tuned" to a specific application's expected traffic. This means that simple bypasses are often effective. For example, a common SQL injection payload like a=a might be blocked, but a slightly modified, semantically equivalent payload like Z=Z (as demonstrated in the talk) can easily bypass the WAF, rendering it ineffective at preventing the underlying vulnerability. This highlights the technical reality that WAFs are a layer of defense, but not a panacea, and should not be relied upon as the sole protection against application-layer attacks.

These technical considerations, woven into the broader discussion of pentest management, provide a robust framework for approaching security assessments with a deeper understanding of the technical challenges and opportunities.

Demo / Proof of Concept

▶ Watch: Pentesting as a holistic and summative security assessment (6:00)

While the talk primarily focuses on strategic and process-oriented aspects, a compelling real-world example serves as a proof of concept for the importance of testing in environments that accurately reflect production and the iterative nature of vulnerability discovery.

The most vivid demonstration comes from the anecdote of the "negative gift card" vulnerability. During an application pentest for an online e-commerce store that sold gift cards, the pentester discovered a flaw where the system would process a negative value gift card. Initially, this finding was made in a dev environment. When reported, the client's internal teams were skeptical, believing that "other systems" in production would block such a transaction.

However, the pentester was asked to validate this specific finding in the live production environment. The speaker recounted streaming his screen, navigating to the client's website, and successfully purchasing a product. By applying a negative $24 gift card, the total came down to $1 (plus $10 shipping, much to his annoyance). Three weeks later, the product arrived in the mail, serving as a tangible souvenir and undeniable proof of the vulnerability's real-world exploitability.

This example dramatically illustrates several key points:

1. Production Realism: It underscores why testing in production (or a highly realistic staging environment) is sometimes necessary to definitively prove exploitability, as supplementary controls in live environments might behave differently or be absent in development.
2. Challenging Assumptions: It shows how internal teams might have incorrect assumptions about their security controls, and an external pentester can provide objective validation.
3. Specific Findings Validation: It highlights the value of taking specific, high-impact findings discovered in development and validating their exploitability in a live context, even if the full test isn't conducted in production.
4. Tangible Impact: The ability to literally receive a product for $1 demonstrates the direct financial impact a seemingly minor logical flaw can have, making the vulnerability concrete and undeniable for the client.

This "demo" serves as a powerful reminder that security isn't just about theoretical vulnerabilities but about what an attacker can actually do in the real world.

Defensive Implications

▶ Watch: Uncovering political and budgetary motivations for pentests (6:40)

The insights from this talk offer numerous actionable strategies for defenders aiming to enhance their security posture and maximize their pentest ROI.

1. Strategic Pentest Planning:

• Define Clear Objectives: Before engaging a pentester, articulate the primary goal: is it compliance, validating new features, assessing overall risk, or leveraging findings for budget? This guides scope, information sharing, and report interpretation.
• Comprehensive Scope Definition: Work with pentesters to define a scope that includes all relevant components and their interconnections. Be transparent about unknown assets or shared infrastructure. Don't let compliance checklists dictate an artificially narrow scope that misses critical attack paths.
• Proactive Information Sharing: Provide pentesters with relevant internal documentation like network diagrams and, where feasible, source code (for application tests). While this deviates from a black-box approach, it significantly increases the efficiency of the test, allowing pentesters to find more vulnerabilities within the allocated time. Insist that pentesters annotate findings with the level of prior knowledge required, maintaining a realistic risk assessment.

1. Optimizing Test Execution:

• Environment Preparation: Ensure the testing environment (ideally production-like with dummy data) is stable and fully available from day one. Avoid making significant changes or deploying patches during the test, as this disrupts the pentester's work and reduces effectiveness.
• Multi-User Testing: For applications, provision multiple user IDs per role and per tenant to enable thorough horizontal and vertical access control testing. This is crucial for uncovering common privilege escalation and data access flaws.
• Clear Rules of Engagement: Establish clear rules of engagement for sensitive activities like testing hours, handling critical findings (immediate disclosure vs. waiting for the report), and approaches to password cracking. Discuss the pros and cons of password cracking (e.g., internal awareness vs. PR nightmare) and agree on a strategy.
• WAF Tuning and Layered Defense: Understand that WAFs are not a silver bullet. They can be easily bypassed if not meticulously tuned to the specific application's traffic. Defenders should view WAFs as one layer in a broader defense-in-depth strategy, not a primary control against application-layer vulnerabilities. Invest in secure coding practices and robust internal controls rather than solely relying on WAFs.

1. Effective Post-Test Remediation:

• Foster a Blameless Culture: Set the stage for the readout call by communicating to all stakeholders (leadership, development, IT, business intelligence team, fraud team) that findings are expected and are opportunities for improvement, not blame.
• Prioritize Concept Over Instance: When remediating, focus on fixing the underlying conceptual vulnerability (e.g., all instances of cross-site scripting or SQL injection) rather than just the specific examples provided in the report. Educate developers on secure coding patterns to prevent recurrence.
• Strategic Report Utilization: Use the pentest report as a tool for driving security improvements and advocating for resources. Leverage the pentester's external expertise during the readout call to amplify messages to leadership, helping them understand the true risk and necessity of remediation efforts.
• Prepare for Retests: Document the steps to reproduce findings clearly, as this aids both internal development teams in remediation and pentesters during subsequent retests. Ensure that retests validate the conceptual fix, not just the reported instances.
• Address Imposter Syndrome: Recognize that receiving a report with findings can be disheartening for security and development teams. Foster an environment of encouragement and continuous learning, reminding everyone that the goal is collective improvement, not individual failure.

By implementing these defensive strategies, organizations can transform their pentest engagements from transactional compliance checks into powerful, value-driven assessments that significantly enhance their overall security posture.

Key Takeaways

• Strategic Planning is Paramount: Define clear goals for your pentest (compliance, risk assessment, budget justification) to guide scope and maximize ROI.
• Communicate Continuously: Maintain open and frequent communication with your pentesters before, during, and after the engagement to ensure efficiency, build trust, and prevent surprises.
• Share Wisely, Test Realistically: Provide pentesters with relevant internal information (source code, diagrams) to accelerate testing, but ensure findings are contextualized by real-world attacker capabilities. Prioritize testing in production-like environments for accurate vulnerability assessment.
• Thorough Access Control Testing: Insist on providing multiple user IDs per role and tenant to enable comprehensive testing of horizontal and vertical privilege escalation, a critical aspect of application security.
• Fix Concepts, Not Just Instances: When remediating, focus on addressing the root cause and underlying patterns of vulnerabilities (e.g., all cross-site scripting) rather than just patching the specific examples reported, to achieve lasting security improvements.
• Leverage Pentesters as Allies: View pentesters as external experts who can help you understand your risks, advocate for resources, and communicate security needs effectively to leadership, fostering a blameless culture of continuous improvement.

About the Speaker(s)

The talk features three brothers, each bringing a distinct perspective to the topic of penetration testing.

Xync (Meristem InfoSec, Managing Principal): Xync runs his own pentesting company, Meristem InfoSec. He has been an application pentester for about a dozen years, following a diverse career in IT that included project management and major incident management. He describes security as his passion and is happy to have been "brought over to the dark side."

Brian (CISO, ksl.com): Brian serves as the CISO for ksl.com, a large media company in Utah. He started at ksl.com 20 years ago as a software engineer, eventually growing his team and advocating for security attention, which led to his CISO role. His focus extends to security, fraud prevention, trust, and safety. He initiated the security office at ksl.com (Deseret Digital Media).

John (Pentester, IBM): John is the oldest brother and was the first to enter the security field. He began his career as a mainframe operator, where during downtime, he stumbled upon a "how to hack" document, sparking his interest in security. He quickly became deeply involved, eventually moving into a pentesting slot at IBM for four to five years, then consulting, before returning to IBM about eight years prior to the talk. He has been a pentester for nearly 25 years. Outside of work, he was a founding member of the Denhack hacker space and enjoys doing CTFs.