Software Screws Around, Reverse Engineering Finds Out: How Independent, Adversarial Research Informs Government Regulation

Software Screws Around, Reverse Engineering Finds Out: How Independent, Adversarial Research Informs Government Regulation

Speakers: Andy Sellars, Partner, Albert Sellers LLP; Mike Specter, Assistant Professor, Georgia Tech

Conference: ShmooCon

YouTube: https://www.youtube.com/watch?v=wXbnUm88IJw

Overview

In "Software Screws Around, Reverse Engineering Finds Out," Andy Sellars and Mike Specter deliver a compelling argument about the critical, yet often unacknowledged, role of independent, adversarial security research in shaping government regulation and consumer protection in the software industry. They contend that the inherent complexity and opacity of modern software, coupled with legal doctrines like trade secrets, create a profound information asymmetry between software vendors and consumers. This imbalance leads to a market failure where vendors have little incentive to invest adequately in security, treating it as a credence good—a quality whose value cannot be easily ascertained by the buyer, even after purchase.

The speakers, drawing from both economic theory and a novel empirical study, demonstrate that independent security research acts as a vital mechanism for surfacing otherwise hidden vulnerabilities and deceptive practices. Their research specifically focuses on the Federal Trade Commission (FTC), revealing that a significant portion of the agency's enforcement actions in privacy and security are directly informed by the work of external researchers. This talk not only highlights a systemic problem but also proposes concrete policy changes to better protect and encourage this essential public service, advocating for a legal framework that recognizes reverse engineering as a public good rather than a potentially illicit activity.

Ultimately, Sellars and Specter challenge the prevailing notion that software security can be effectively managed solely through internal vendor mechanisms or limited disclosure programs. They argue that a healthy ecosystem of adversarial research is indispensable for holding software accountable, especially as it permeates critical sectors like voting systems and personal devices. Their work serves as a call to action for policymakers, legal scholars, and the security community to champion transparency and permissionless analysis as cornerstones of a more secure and trustworthy digital future.

Background

▶ Watch: Talk Introduction: Hackers, Lawyers, and Government Regulation (4:00)

The premise of this talk begins with a fundamental challenge: it is exceedingly difficult to ascertain what software is truly doing, especially concerning privacy and security. Modern software is characterized by its immense complexity, often comprising millions of lines of code. Furthermore, key components are frequently cloud-based, making them inaccessible for direct examination, and even on-device software is subject to continuous, often opaque, updates. This inherent opacity has only intensified with the advent of sophisticated technologies like generative AI, making the task of understanding software's behavior daunting for consumers, researchers, and regulators alike.

Compounding this technical challenge is the existing legal framework, which, as Andy Sellars notes, often exacerbates the problem. A long-standing line of legal scholarship points to the doctrine of trade secrets as a primary impediment to software transparency. This legal protection allows software vendors to shield their code from scrutiny, citing proprietary interests. Practically, this means that attempts to learn about government software usage through FOIA (Freedom of Information Act) requests often hit a wall due to trade secret exemptions. Similarly, in civil or criminal discovery, trade secret claims can deny defendants access to crucial software information. Engaging in projects that involve examining software can lead to claims of misappropriation of trade secrets, and vendors can even leverage these property rights to resist calls for greater transparency, framing regulatory demands as an unconstitutional "regulatory taking."

Mike Specter then introduces a critical economic concept: information asymmetry. This occurs when one party in a transaction possesses more or superior information than the other. Economists categorize products into three types based on this asymmetry:

1. Search goods: Products whose value can be known before purchase (e.g., a nail, where melting point and length are evident).
2. Experience goods: Products whose value is only known after purchase and use (e.g., a used car, where hidden issues emerge over time).
3. Credence goods: Products whose value cannot be known even after purchase without expert analysis (e.g., doctor visits or car mechanic services, where the consumer cannot independently verify if the treatment or repair was truly necessary or optimal).

Sellars and Specter argue that software security squarely fits the definition of a credence good. Users, and even many regulators, lack the specialized knowledge to determine the true security posture of software. The critical consequence of a market for credence goods is that if users cannot determine the true quality, the market's dominant strategy for vendors is to underinvest in that quality. This leads to chronic undertreatment, as there's no immediate market pressure or reward for superior security beyond what is minimally perceived.

To address the challenges posed by credence goods, common regulatory methods include:

• Introducing liability: Holding vendors accountable for failures. However, standard software terms of use often disclaim liability, negating this mechanism.
• Labeling and standards: Providing clear, verifiable indications of quality (e.g., "fair trade" labels). The recent White House US Cyber Trust Mark is an attempt to create such a standard for cybersecurity.
• Transparency and verifiability: Allowing external parties to examine the product to confirm its claims.

The core problem, as the speakers highlight, is that without robust enforcement, labeling becomes meaningless ("trust me bro" policies). In the absence of strong liability and verifiable transparency, the market for software security remains fundamentally broken, necessitating external, adversarial scrutiny to expose hidden deficiencies.

Key Findings

▶ Watch: Why Software Transparency is Hard & Law Makes it Worse (5:00)

The central finding of Sellars and Specter's research is a powerful empirical demonstration: adversarial independent research is a necessary precursor to good security regulation and enforcement. They undertook a pioneering study, creating a publicly released dataset available at ftcreverse.engineering, to quantify the impact of independent research on the regulatory activities of the Federal Trade Commission (FTC). The FTC, as the default consumer protection regulator in the United States, enforces specific privacy laws and wields broad authority against "unfair and deceptive commercial practices," which it has historically applied to privacy and security issues.

Their methodology involved analyzing all FTC cases tagged as "privacy and security" between January 2017 and summer 2023. This yielded 102 distinct cases, which were further broken down into 322 individual "counts" — discrete wrongdoings identified by the FTC. After omitting counts that didn't require an investigation (e.g., failure to file paperwork), they focused on 283 investigative counts. For each of these, they painstakingly determined how the FTC initially learned about the problem.

The results were striking:

• 26.5% of all investigated counts across the FTC's privacy and security corpus were attributed to independent research. For approximately 10% of these, a direct link could be drawn, with the FTC citing a specific piece of research (e.g., from Consumer Reports) in its complaint. For others, the link was drawn through prominent news reporting that timed out correctly with the FTC's action.
• The impact is even more profound when considering cases: over a third of the FTC's activity across the entire domain of consumer privacy and security owed at least one count to independent research. This suggests that independent research often triggers an investigation, leading the FTC to uncover additional issues through its own discovery processes.

The influence of independent research was particularly pronounced in cases requiring a deep understanding of software behavior. For instance, in cases involving "deceptive failure to disclose" under Section 5 (the FTC's general unfair and deceptive authority), where companies claimed one thing about data sharing but omitted other uses, two-thirds of these cases were informed by independent research. This highlights that expert analysis is crucial when comparing software's actual operation to vendor claims, a level of scrutiny not typically available to individual consumers.

Furthermore, the data revealed the diverse origins of this critical information. While journalists and media organizations played a significant role, the research also identified contributions from individuals in industry, academia, and notably, unaffiliated researchers. This broad participation underscores the need for policies that protect the activity of adversarial research itself, rather than limiting protections to specific professional categories. The findings unequivocally demonstrate that external, permissionless scrutiny is not merely supplementary but foundational to effective government oversight of software security and privacy.

Technical Deep Dive

▶ Watch: Case Study: Reverse Engineering a Flawed Voting App (7:00)

While this talk does not delve into specific code exploits or network protocols, its "technical deep dive" lies in its rigorous application of economic theory and empirical methodology to illuminate the systemic issues in software security. The core technical framework presented is the concept of credence goods and information asymmetry as applied to software security.

Software security is technically complex to assess. Unlike a "search good" where properties like length or material are immediately obvious, or an "experience good" where flaws might emerge after some use, software security often requires deep, specialized knowledge to evaluate. This means understanding intricate code logic, potential vulnerabilities in communication protocols, the nuances of cryptographic implementations, and the implications of system architecture choices. A typical consumer, or even a general-purpose regulator, simply lacks the technical expertise to verify a vendor's claims about security. This technical barrier creates the information asymmetry that defines software security as a credence good.

The speakers' own prior work exemplifies the technical nature of adversarial research. Mike Specter and his colleagues, as PhD students at MIT, investigated a piece of voting technology intended for the 2020 primaries. This system allowed voting via a smartphone using blockchain technology. From a technical security perspective, the combination of smartphone security (a notoriously complex attack surface), voting systems (demanding extreme integrity and availability), and blockchain (often misunderstood and misapplied) immediately raised red flags for experts. Their investigation involved:

• Reverse engineering: Deconstructing the software to understand its internal workings without access to source code.
• Static analysis: Examining the software's code (or decompiled binaries) without executing it, looking for patterns of vulnerabilities, insecure coding practices, or potential backdoors.
• Dynamic analysis: Observing the software's behavior during execution, monitoring network communications, memory usage, and interactions with the operating system to identify runtime flaws or deviations from stated functionality.

Through these technical methods, they uncovered "serious issues" in the software. This hands-on, low-level technical analysis was crucial because it allowed them to compare the software's actual behavior and underlying design against the vendor's public claims and general security best practices. Their ability to conduct this permissionless analysis was paramount to identifying the vulnerabilities that ultimately led to the suspension of the pilots.

The empirical study itself also involved a rigorous, technically-informed methodology. Analyzing FTC complaints required understanding the specific technical or behavioral issues being cited. For example, identifying "deceptive failure to disclose" often necessitated discerning subtle differences between what a company claimed its software did and what independent technical analysis revealed it was doing. The coding methodology, detailed on their ftcreverse.engineering website, involves a structured approach to categorizing the source of information for each FTC count, distinguishing between direct citations of research, indirect links through media reports, or other sources like consumer complaints. This systematic approach, informed by an understanding of how technical flaws manifest in regulatory actions, underpins their quantitative findings.

Finally, the "doom principle in security economics" articulated in the talk serves as a theoretical framework for understanding the technical consequences of legal and market failures. It posits that when researchers lack a positive right to conduct adversarial, permissionless analysis, software vendors' dominant strategy will be to allow users to suffer from security deficiencies, potentially driving out technically superior but less profitable products. This principle directly links the technical reality of software vulnerabilities to the economic and legal incentives that shape the security landscape.

Demo / Proof of Concept

▶ Watch: Talk's Core Argument: Independent Research Precedes Good Security (8:00)

While the ShmooCon presentation itself did not feature a live technical demonstration of an exploit or a tool, the speakers powerfully presented a real-world "proof of concept" of their core thesis: the indispensable role of independent, adversarial research in informing government action and protecting the public. This demonstration took the form of Mike Specter’s prior work on a smartphone-based voting technology.

Approximately five years prior to the talk, Mike Specter, then a PhD student at MIT, along with his colleagues, initiated an investigation into a new voting system. This system was designed to allow citizens to cast votes via their smartphones, leveraging blockchain technology, and was slated for pilot deployment in the 2020 primaries. As Specter notes, for anyone familiar with voting security, smartphone security, or the practical application of blockchain, this combination immediately raised significant concerns.

The team undertook a comprehensive security assessment, which involved:

• Reverse engineering the proprietary software to understand its internal mechanisms without access to the source code.
• Conducting static analysis to identify potential vulnerabilities within the code structure.
• Performing dynamic analysis to observe the system's behavior during operation and identify runtime flaws.

This rigorous technical scrutiny uncovered "a bunch of serious issues" within the software. These findings were not kept private. Instead, Specter's team, in collaboration with Andy Sellars and his students from the Boston University School of Law clinic, worked to responsibly disclose these vulnerabilities. This disclosure process involved engaging with affected elections officials and the software vendor, facilitated through the then-new Cybersecurity and Infrastructure Security Agency (CISA).

The outcome of this independent, adversarial research was a resounding success for public safety: the planned pilots for the smartphone voting system were suspended before they could be used in a live election, thereby preventing potential harm to the democratic process. This real-world example serves as a potent demonstration of how expert, permissionless technical analysis can bypass the information asymmetry inherent in software, expose critical flaws, and directly inform regulatory bodies and public officials to mitigate risks. It embodies the talk's argument that such research is not merely an academic exercise but a vital public service, capable of driving tangible, positive change in critical infrastructure.

Defensive Implications

▶ Watch: Security as an Economic & Consumer Protection Problem (9:00)

The defensive implications of Sellars and Specter's research extend beyond traditional system hardening to encompass policy, legal frameworks, and the very culture of security disclosure. For defenders—broadly defined to include not just security professionals but also policymakers, regulators, and even the general public—the key takeaway is that relying solely on vendors for software security assurance is inherently flawed.

For Policymakers and Regulators:

• Recognize Reverse Engineering as a Public Good: The data unequivocally shows that independent research is critical for identifying software issues that lead to regulatory action. Laws and policies must explicitly protect and encourage this activity, rather than treating it as a potential legal liability. This means fixing "bad assumptions" in legal literature that narrowly define reverse engineering's purpose (e.g., interoperability) and instead acknowledge its role in consumer protection and public safety.
• Ban Private Contract Restrictions for Crucial Technology: Current contractual terms often prohibit reverse engineering or resale of critical software and hardware (e.g., voting machines, rootable iPhones). These restrictions effectively create impenetrable black boxes, hindering essential public oversight. Policymakers should legislate against such contractual clauses for technologies that impact public welfare, national security, or fundamental rights.
• Promote True Transparency and Verifiability: The US Cyber Trust Mark and similar initiatives are steps in the right direction, but they must be backed by robust, verifiable standards and mechanisms for independent scrutiny. Without the ability for external experts to confirm claims, labels can become mere marketing.

For Security Researchers and the Community:

• Be Wary of Restrictive Disclosure Policies: The speakers strongly advise against participating in bug bounties or disclosure programs that include gag orders or excessive restrictions. Such programs, while offering financial incentives, primarily serve to buy researchers' silence, preventing the information asymmetry problem from being addressed in the broader market. Researchers should prioritize disclosures that genuinely contribute to the public good and market accountability.
• Understand Your Role as a Market Corrector: Independent researchers are not just finding bugs; they are providing critical information that the market, left to its own devices, fails to produce. This adversarial role is essential for driving investment in security quality, acting as an external check on vendor claims and practices.
• Document and Disclose Systematically: Following responsible disclosure practices, working with organizations like CISA, and clearly documenting methodologies (as seen with ftcreverse.engineering) enhance the credibility and impact of research findings, making them more actionable for regulators.

For Software Vendors:

• Proactive Engagement with Independent Research: Instead of viewing independent researchers as adversaries, vendors should recognize them as an essential part of a healthy security ecosystem. Embracing transparency, facilitating responsible disclosure, and even funding non-restrictive research can ultimately lead to more secure products and build greater public trust.
• Re-evaluate Business Models: The "dominant strategy" of underinvestment in security for credence goods is a long-term liability. The FTC's reliance on independent research demonstrates that hidden flaws will eventually be exposed, leading to regulatory enforcement, reputational damage, and potentially costly remediation. Investing in security and transparency from the outset is a more sustainable and responsible approach.

In essence, the defensive implication is a call to shift from a reactive, opaque security posture to a proactive, transparent one, recognizing that robust security for critical software is a public good that necessitates external, adversarial oversight to truly thrive.

Key Takeaways

• Software security is a credence good: Due to its complexity and opacity, consumers and even regulators cannot easily verify software security claims, leading to a market failure where vendors often underinvest in security.
• Independent adversarial research is crucial for accountability: External scrutiny by researchers, journalists, and academics is vital for uncovering hidden vulnerabilities and deceptive practices that vendors might otherwise conceal.
• The FTC heavily relies on independent research: Their empirical study revealed that over a third of the FTC's privacy and security enforcement cases, and 26.5% of specific investigative counts, were directly informed by independent research.
• Current legal and contractual frameworks hinder essential research: Doctrines like trade secrets, restrictive private contracts (e.g., for voting machines or rootable iPhones), and non-disclosure agreements in bug bounties actively suppress the very research needed for public protection.
• Policy changes are urgently needed: To foster a more secure digital environment, governments must recognize reverse engineering as a public good, ban restrictive contract clauses for critical tech, and move away from gag orders in disclosure policies.
• Researchers should critically evaluate bug bounties: Participation in bug bounties that prioritize silence over public disclosure may benefit individual researchers but ultimately perpetuate market information asymmetries, hindering broader societal security improvements.

About the Speaker(s)

Andy Sellars is a partner at Albert Sellers LLP, a public interest technology law firm. For nine years, he taught at Boston University School of Law, where he also ran a law clinic in partnership with BU Law and MIT. This clinic represented students engaged in interesting and provocative technology projects, which is how he initially connected with Mike Specter. His work focuses on the intersection of law and technology, particularly in areas of transparency and accountability.

Mike Specter is an Assistant Professor at Georgia Tech, where he leads the aptly named Specter Lab. His research expertise spans system security, applied cryptography, and tech public policy. Prior to his current role, Mike was a PhD student at MIT, where he conducted the influential research on voting technology that served as a foundational example for this talk. His academic background and practical experience in reverse engineering position him as a leading voice in understanding the technical and policy dimensions of software security.