When the Paper Trail Leads Nowhere

When the Paper Trail Leads Nowhere

Speakers: Ian Patton, Executive Director, Long Beach Reform Coalition

Conference: Voting Village

YouTube: https://www.youtube.com/watch?v=URWjVRUDNiI

Overview

Ian Patton, Executive Director of the Long Beach Reform Coalition, delivers a compelling and often frustrating account of his organization's efforts to audit and recount ballots cast using Los Angeles County's new Voting Solutions for All People (VSAP) system. This $300 million bespoke system, developed by Smartmatic, was intended to modernize elections, promising a voter-verified paper trail. However, Patton's talk meticulously details how this theoretical safeguard proved practically impossible to utilize for a meaningful recount, exposing profound systemic flaws in design, implementation, and transparency.

Patton's presentation serves as a stark warning about the gap between the perceived security of paper ballots and the real-world challenges of auditing complex, large-scale electronic voting systems. By chronicling a grassroots organization's struggle against a bureaucratic and technically opaque system, he highlights how the absence of clear recount procedures, exorbitant costs, and even the apparent suppression of known system capabilities can undermine fundamental democratic rights. The talk underscores the critical need for voting systems that are not only secure but also genuinely auditable and transparent to the public.

This article delves into the specifics of LA County's VSAP system, the recount debacle following the March 2020 Super Tuesday election, the technical and logistical hurdles encountered, and the broader implications for election integrity and accountability. Patton's experience reveals how even with a "paper trail," the path to verifying election results can lead nowhere if the underlying system and administrative processes are not designed for true public scrutiny.

Background

▶ Watch: Introduction to ballot marking devices and uncounted paper trails (0:00)

The concept of a "paper trail" in modern voting systems gained prominence after the controversies surrounding the 2000 U.S. presidential election and the subsequent push for computerized voting. The idea was that while electronic machines would simplify voting and counting, a physical paper record — ideally verifiable by the voter — would always exist as a backup for audits and recounts, ensuring integrity. The Help America Vote Act (HAVA) of 2002 further emphasized the need for accessible and verifiable voting systems.

Los Angeles County, the largest county in the United States, embarked on developing its own custom voting system, VSAP, in a process initiated around 2008 by Registrar-Recorder Dean Logan. This ambitious project, costing an estimated $300 million, was ultimately developed by Smartmatic, a company with a controversial international history. The system was designed to implement a vote center model, allowing any registered voter in the county to cast their ballot at any designated vote center, moving away from traditional precinct-based polling places. While this offered convenience, it fundamentally changed how ballots were collected and aggregated, leading to all 2 million LA County ballots being mixed together rather than sorted by precinct.

The problem, as Patton discovered, lay in a critical oversight: despite the decade-long development and immense investment, the VSAP system, and the county's election administration, seemingly had no practical, cost-effective, or even documented procedure for conducting a recount of the very paper ballots it produced. The Request for Proposal (RFP) for the system, a document spanning over 50 pages, remarkably did not contain the word "recount" even once. This fundamental omission created a significant vulnerability, turning the promised paper trail into an inaccessible illusion when the need for verification arose.

Key Findings

▶ Watch: LA County's $300M Smartmatic VSAP system's disastrous debut (2:20)

Patton's organization, the Long Beach Reform Coalition, became entangled with the VSAP system after a local ballot measure in the March 3, 2020 Super Tuesday election. Initially, the "no" side, supported by Patton's group, appeared to be winning. However, after eight post-election updates, the certified result on March 27th declared the "yes" side victorious by a mere 16 votes out of approximately 100,000 cast – a margin of 0.01%. This razor-thin margin, coupled with the shifting results, prompted a request for a recount, which unveiled a series of critical issues:

1. Lack of Recount Procedures: The Los Angeles County Registrar-Recorder's office had no established policy, procedure, or system to conduct a recount under the new VSAP framework. The manual provided was for the old voting system, and county officials had to "adjust things" due to the nascent COVID-19 pandemic, which Patton suggests was a cover for their unpreparedness.
2. Prohibitive Cost of Physical Recount: Due to the vote center model mixing all 2 million ballots from across LA County, a physical hand recount of 100,000 Long Beach ballots would require 16 county workers to manually sort through 2 million batch boxes for 16 days. This labor-intensive process alone was estimated to cost over $200,000, an amount far beyond the means of a grassroots organization.
3. Flawed Scanned Image Recount: As an alternative, the county offered a recount using scanned images of the ballots. However, this system was "literally built" on the fly, taking a day and a half to create a basic interface that merely pulled up each ballot as a separate PDF. The process was plagued by severe network lag, causing an 8-second delay between screens displaying the same ballot, making synchronized observation and tallying impossible. This rendered the digital recount option impractical and still prohibitively expensive, estimated at over $100,000 for the 100,000 ballots.
4. Consistent Overvote Error: During the limited progress of the scanned image recount (approximately 10,000 ballots), a consistent and significant error pattern was discovered. For hand-marked vote-by-mail ballots, the VSAP machines frequently misidentified valid votes as "overvotes" and discarded them. This occurred when voters made minor marks (a dot, jot, or tick) near the unselected circle, even if their intent to vote for the other option was perfectly clear. Every observer in the room agreed on the clear voter intent, yet the machine's algorithm incorrectly flagged these as overvotes.
5. Failure of the 1% Hand Tally Audit: Despite the discovery of a consistent error pattern, the mandatory 1% hand tally audit, conducted after every election in California, had not reported any such discrepancies. Patton speculates that county workers performing this audit might be instructed to interpret ballots "like the computer," thus overlooking voter intent in favor of machine logic.
6. Undisclosed Sorting Functionality: Years later, Patton discovered that the VSAP ballot scanning machine actually possesses a 21-pocket sorting functionality. This capability, which could have significantly streamlined the recount process, was either unknown to or deliberately withheld by the Registrar-Recorder's office.
7. Litigation Failure: A subsequent lawsuit to compel a proper recount was dismissed. Patton attributes this partly to the contentious national political climate surrounding election integrity following the November 2020 election, suggesting the judge was reluctant to engage with the case.
8. Smartmatic's Controversial History: Patton raised significant concerns about Smartmatic, the company behind VSAP. He highlighted its origins with Hugo Chavez's government in Venezuela, its acquisition and subsequent forced divestment of Sequoia Voting Systems in the U.S. due to national security concerns, and recent indictments of its co-founder, Roger Pinate, for alleged bribery of top election officials in the Philippines and Venezuela.

Technical Deep Dive

▶ Watch: Local ballot measure's suspicious flip after 8 post-election updates (4:15)

The technical heart of the issues discussed by Ian Patton lies in the design, implementation, and operational shortcomings of the Voting Solutions for All People (VSAP) system. VSAP is a bespoke system, meaning it was custom-built for Los Angeles County, a crucial detail given its $300 million price tag and the lack of comparable installations elsewhere. This uniqueness, while perhaps intended to meet LA County's specific needs, also meant a lack of external validation and established best practices for its auditability.

At its core, VSAP utilizes Ballot Marking Devices (BMDs). These machines allow voters to make selections on a touchscreen, which then prints a paper ballot. Critically, these paper ballots often include a QR code that encodes the voter's selections. While the speaker noted that BMD-generated ballots are "always perfect" in terms of how the computer counts them (because the computer generates both the human-readable text and the QR code), this perfection doesn't guarantee the voter's true intent is captured if the BMD itself is compromised or misconfigured. The primary concern, however, focused on hand-marked vote-by-mail ballots, which are scanned and interpreted by the VSAP system.

The shift to a vote center model introduced a significant logistical challenge for recounts. Unlike traditional precinct-based systems where ballots are aggregated and returned by specific polling places, the vote center model meant that all 2 million ballots cast across LA County were physically mixed together. This eliminated the inherent pre-sorting that made recounts in the "old system" more manageable and less expensive. The absence of an automated mechanism to extract a specific subset of ballots (e.g., 100,000 from Long Beach) from this massive, commingled pool was a critical design flaw. The county's proposed solution – 16 human workers manually sifting through 2 million batch boxes for 16 days – speaks volumes about the system's lack of foresight regarding auditability.

The attempted "scanned images recount" revealed further technical deficiencies. The system presented each ballot as a separate PDF file. However, the ad-hoc nature of the interface, "literally built" during the recount attempt, led to severe network lag. This lag caused significant desynchronization across multiple observer screens, with some displays showing a new ballot 8 seconds later than others. Such a delay renders meaningful, simultaneous observation and verification by multiple parties virtually impossible, undermining the very purpose of transparency in a recount. It suggests a lack of robust networking infrastructure, efficient image retrieval mechanisms, and a multi-user interface designed for real-time collaboration and auditing.

Perhaps the most alarming technical finding was the consistent overvote detection algorithm error in the VSAP system. For hand-marked ballots, the system was overly sensitive, misinterpreting minor stray marks (a "dot or a jot or a tick") near an unselected option as an attempt to vote for both options, thus rendering the ballot an "overvote" and discarding it. This indicates a flaw in the optical scan technology or its configuration, where the threshold for detecting a mark was too low, or the algorithm failed to adequately differentiate between clear voter intent and incidental markings. This directly contradicts the "voter intent" standard in California law, highlighting a critical disconnect between legal requirements and technical implementation.

Finally, the revelation that the VSAP ballot scanning machine itself possesses a 21-pocket sorting functionality is a profound technical indictment. This capability, if properly integrated into recount procedures, could have drastically reduced the manual labor and cost associated with extracting specific ballots. The fact that this feature was either unknown to or deliberately withheld by election officials points to a severe breakdown in technical documentation, operational planning, or transparency regarding the system's full capabilities. This implies either gross incompetence in understanding the system's architecture or a deliberate choice to obscure practical audit pathways.

Demo / Proof of Concept

▶ Watch: Requesting a recount and receiving a misleading cost estimate (6:30)

While Ian Patton's talk did not feature a traditional live exploit demonstration or a controlled proof-of-concept in the typical cybersecurity conference sense, the entire narrative of his organization's attempt to conduct a recount served as a real-world, albeit unsuccessful, proof of concept for the VSAP system's auditability.

The "scanned images recount" process, though ultimately abandoned due to its impracticality and cost, functionally demonstrated the system's flaws. The observers witnessed firsthand the severe network lag and desynchronized displays, proving that the digital interface provided by LA County was inadequate for transparent, multi-party verification. More critically, during this limited recount, the consistent overvote detection error was observed on thousands of hand-marked ballots. This direct observation of the machines misinterpreting voter intent, despite universal human agreement on the correct interpretation, effectively served as a live demonstration of a critical flaw in the system's vote-counting logic. The inability to complete a full, accurate, and affordable recount, despite the existence of a "paper trail," was the ultimate, albeit negative, proof of concept that the system, as implemented, failed to deliver on its promise of auditable elections.

Defensive Implications

▶ Watch: Pandemic used as an excuse while county provided wrong recount manual (8:00)

The experience with LA County's VSAP system offers several critical defensive implications for election officials, security professionals, and the public:

1. Prioritize Recountability and Auditability in System Design: Voting system procurement and design must explicitly include robust, cost-effective, and practical recount procedures from the outset. Request for Proposals (RFPs) should mandate specific functionalities for ballot sorting, retrieval, and transparent digital review that align with legal recount requirements. The omission of the word "recount" from a 50-page RFP for a $300 million system is an unacceptable oversight.
2. Ensure Transparency of System Capabilities: Election officials must fully understand and transparently disclose all functionalities of their voting equipment. The discovery of a hidden 21-pocket sorting function on the VSAP scanning machine, years after its deployment and a failed recount attempt, highlights a severe lack of transparency and operational knowledge. All system capabilities that could aid in audits or recounts should be documented and readily available.
3. Implement Robust and Realistic Audit Mechanisms: The failure of the 1% hand tally audit to detect the consistent overvote error demonstrates its limitations. Audit methodologies must be designed to genuinely assess voter intent, not merely validate machine interpretation. Transitioning to Risk-Limiting Audits (RLAs) or similar statistically sound audits, performed by independent parties, is crucial to ensure that errors are detected and corrected. Furthermore, auditors must be trained to prioritize voter intent over strict machine-like interpretations.
4. Calibrate Optical Scan Algorithms for Voter Intent: For systems processing hand-marked ballots, the algorithms used for optical scanning and vote tabulation must be carefully calibrated to accurately interpret voter intent, even with minor imperfections. Overly sensitive overvote detection algorithms can disenfranchise voters by discarding legitimate ballots due to insignificant marks. Regular testing and public review of these algorithms are essential.
5. Conduct Thorough Vendor Due Diligence: The controversial history of Smartmatic, including allegations of bribery and its involvement with authoritarian regimes, underscores the critical need for exhaustive due diligence on voting system vendors. Election officials must not only assess technical capabilities but also the ethical track record, financial transparency, and political independence of potential partners. Continuous monitoring of vendor activities, both domestically and internationally, is paramount.
6. Empower Grassroots Oversight and Legal Avenues: The Long Beach Reform Coalition's efforts demonstrate the vital role grassroots organizations play in holding election systems accountable. Legal frameworks must ensure that the right to a recount is not rendered impossible by prohibitive costs or bureaucratic hurdles. Legal challenges should be assessed on their merits, free from political pressure, to uphold the principle of accessible justice in election disputes.
7. Invest in Open-Source and Standardized Systems: Bespoke, proprietary systems like VSAP often lack transparency and make independent auditing difficult. Exploring open-source voting systems or adopting standardized, well-documented commercial off-the-shelf (COTS) solutions could foster greater transparency, allow for broader security review, and potentially reduce costs and complexity in the long run.

Key Takeaways

• The mere existence of a "paper trail" in modern voting systems does not guarantee effective auditability or the practical ability to conduct recounts if the system's design and administrative procedures are not explicitly built for it.
• Bespoke voting systems, like LA County's $300 million VSAP, can introduce unique, unforeseen logistical and financial barriers to recounts, making them prohibitively expensive and practically impossible for the public.
• Voting system functionalities that could aid in audits (e.g., ballot sorting mechanisms) must be transparently disclosed, understood, and actively utilized by election administrators.
• Automated vote counting systems can exhibit consistent, subtle errors, such as misinterpreting clear voter intent on hand-marked ballots, which may go undetected by standard audit procedures like the 1% hand tally.
• The legal and political environment surrounding election integrity can significantly impede legitimate challenges to voting system accuracy, highlighting the need for a robust, apolitical judicial process for election disputes.
• Thorough and ongoing due diligence on voting system vendors, including their international history and any allegations of misconduct, is essential to mitigate risks to election integrity and public trust.

About the Speaker(s)

Ian Patton is the Executive Director of the Long Beach Reform Coalition, a grassroots organization dedicated to government accountability and transparency in Long Beach, California. Patton describes himself as not possessing technical expertise in election systems, but rather having gained profound knowledge through "bitter experience" leading his organization's efforts to understand and challenge the implementation of LA County's new voting system. Earlier in his career, he had some involvement in election administration, working for a congresswoman who had oversight over the Election Assistance Commission (EAC) and the Help America Vote Act (HAVA), which he notes gave him an initial understanding of election processes before his later "nightmare" awakening to the challenges of modern systems.