A Pirate's Guide to Snake Oil and Security

A Pirate's Guide to Snake Oil and Security

Speaker: HD Moore (runZero)

Conference: NorthSec 2025 — May 15–16, 2025, Marché Bonsecours, Montreal

Watch on YouTube: https://www.youtube.com/watch?v=yHHImYJ9yKM

Reading time: ~10 minutes

TL;DR

HD Moore, creator of Metasploit and now principal at runZero, dissects the vulnerability management industry with two decades of hard-won credibility. He demonstrates that most commercial vuln scanners cover fewer than half of known vulnerabilities in controlled tests, that anti-benchmarking clauses are universal across vendor EULAs, and that practitioners can build their own comparative benchmarks entirely from public data. The talk ends with concrete evaluation criteria that any organization can apply before signing the next five-year contract.

Introduction

The opening keynote of NorthSec 2025 was always going to land differently when delivered by the person who wrote three to four hundred exploits before most attendees had cleared university. HD Moore, founder of the Metasploit project in 2003 and current member of the team at runZero, stepped onto the Ville-Marie stage with the credibility to say things that would get a vendor representative escorted from the room. His framing was deliberately theatrical — a pirate voyage aboard the "USS Vaughn Secure" navigating the "Island of Vulnerability Management" — but the substance underneath was methodical and data-driven.

Moore's central argument is that the vulnerability management market has drifted from its hacker origins into a vendor-controlled ecosystem where independent benchmarks are legally suppressed, marketing language is indistinguishable across competing products, and practitioners are left making purchasing decisions worth hundreds of thousands of dollars per year on the basis of peer opinion and analyst reports that never actually test whether a scanner finds the vulnerabilities it claims to find.

The talk is a call to arms: not to abandon vulnerability management tooling, but to treat it with the same skepticism practitioners apply to every other claim in their threat landscape.

The Battleship Metaphor and the Reality of Security Debt

Moore opens by comparing a security program to a battleship in hostile seas — not a glamorous metaphor, but an accurate one. A system left unattended on the internet will not survive intact for four years. Vulnerabilities accumulate, software decays, and the default outcome of inaction is compromise. For small and mid-sized organizations, this pressure is compounded by the fact that their threat model rarely resembles the nation-state APT narratives that dominate conference marketing.

▶ Watch: The battleship and the hostile internet (2:00)

The real problem Moore identifies is the gap between what organizations need — a reliable, continuously updated view of their exploitable attack surface — and what the market provides: tools with overlapping and inconsistent scope, purchased once and rarely re-evaluated, from vendors contractually protected against independent scrutiny.

Anti-Benchmarking Clauses and the Absence of Independent Testing

Perhaps the most pointed section of the talk involves intellectual property. Moore states plainly that nearly every security vendor's end-user license agreement contains an explicit anti-benchmarking clause. Vendors prohibit independent testing because comparative results are embarrassing. He describes one EDR vendor whose EULA bars customers from posting screenshots of the product in public forums — including support forums — illustrating how comprehensively these companies can insulate themselves from accountability.

▶ Watch: Anti-benchmarking clauses and the EULA problem (6:00)

Analyst firms such as Gartner, Forrester, and IDC fill the vacuum left by absent independent benchmarks, but their evaluation criteria do not include functional vulnerability detection testing. Their methodology, as Moore describes it, aggregates peer opinion: how does this tool work for you? The result is that no publicly available dataset definitively answers whether a given scanner can perform unauthenticated detection of the CVEs relevant to a specific environment.

Building a DIY Benchmark from Public Data

Rather than accepting this information vacuum, Moore proposes building a comparative benchmark using entirely public data — no reverse engineering, no EULA violations. He references pentest-tools.com's published benchmark, which tested seven common vulnerability scanners against 167 VulHub containers. The headline finding: the top-performing scanner detected fewer than half of the vulnerabilities in the test set. Nmap, paired with Nuclei and OpenVAS (combined cost: zero dollars), produced results comparable to expensive commercial alternatives.

▶ Watch: The VulHub benchmark and what it reveals (8:00)

Moore is clear that synthetic lab tests have limitations — they do not reflect the heterogeneous reality of enterprise environments, with tape libraries, ES400s, and aging Windows endpoints. But the benchmark's value is demonstrative: it shows that the gap between free and paid tools is far smaller than vendor pricing implies, and that meaningful comparative evaluation is possible without proprietary access.

For a more useful real-world benchmark, Moore recommends analyzing public data along four dimensions: response time (how quickly does a vendor add detection for a new CVE?), CVE coverage (how many unique identifiers are supported?), detection methodology (authenticated vs. unauthenticated?), and update cadence.

The CVE Lag Problem and Patch Window Economics

Moore walks through a timing analysis using a Qualys True Risk Report (covering 2023 vulnerabilities, published 2024) combined with Mandiant's exploitation timeline data. The key figures: roughly 2% of published CVEs are actively exploited in the wild, but 25% of those are exploited on day zero. Beyond that, the median lag between a public exploit and CVE assignment is 23 days — which means EDR-based vuln management products, which filter on CVE presence, cannot surface the vulnerability until more than three weeks after exploitation may already be underway.

▶ Watch: CVE assignment lag and the exploitation window (10:00)

The patch compliance picture is equally stark: only 15% of CISA KEV-listed critical vulnerabilities are patched within 30 days. Median patch time across organizations extends past 55 days for half the asset base, and past 180 days for the bottom 20%. This creates an exploitation window that averages close to six months for the majority of systems — a window that the vulnerability management tool is supposed to help close, but often cannot accurately characterize.

Practical Evaluation Criteria for Practitioners

Moore closes with a framework practitioners can use right now. He recommends three concrete steps: First, read the EULA of every security tool in the environment and identify what the vendor prohibits you from testing or disclosing. Second, pull the vendor's CVE coverage data from public sources and cross-reference it against KEV entries, EPSS scores, and the specific OS/software profile of the organization's environment. Third, run a controlled test — even a small one — against a representative subset of internal assets before renewal.

▶ Watch: Building your own benchmark criteria (8:30)

The goal is not to replace every tool in use, but to develop a fact-based understanding of where each tool's coverage ends and where gaps persist. Moore is explicit that combining free tools (Nmap, Nuclei, OpenVAS) with targeted commercial coverage for specific asset classes may outperform a single expensive platform — a conclusion the vendor market is not incentivized to advertise.

Notable Quotes

"If you do nothing at all, your default is to die on the Internet."

"Read the terms of service for every tool you've bought, and there'll be a very clear anti-benchmarking clause in every product you buy in security because they don't like being embarrassed because they suck."

"Even winning only gets you halfway there." — on a benchmark where the top scanner detected 80 of 167 vulnerabilities

"For zero dollars, I can buy Nmap, Nuclei, and OpenVAS, and I'll do a better job combined than anything else I can pay for."

Key Takeaways

• The benchmark vacuum is intentional. Anti-benchmarking EULAs are standard across the security industry. No independent, comprehensive benchmark for vulnerability scanners has survived commercial pressure. Practitioners must build their own.

• CVE-gated tools have a structural blind spot. Products that require CVE assignment before flagging a vulnerability miss the 23-day median window between public exploit and CVE issuance — a period when active exploitation is already occurring.

• Patch compliance timelines are long. Only 15% of critical vulnerabilities make it through patching within 30 days. The average exploitation window extends to roughly six months across most enterprise environments.

• Coverage, not category, is the right metric. "Vulnerability scanner," "EDR-based vuln management," and "web application scanner" are marketing categories, not functional descriptions. The right question is: which CVEs does this tool detect in my environment, authenticated or not?

• Free tools punch above their weight. In available public benchmarks, the combination of Nmap, Nuclei, and OpenVAS rivals commercial products on detection rate while eliminating licensing costs and five-year lock-in contracts.