Stalking the Wily Hacker: 40 Years Later

Stalking the Wily Hacker: 40 Years Later

Speakers: Cliff Stoll (Astronomer, Author, and Security Pioneer, Acme Klein Bottles)

Conference: DEF CON

YouTube: https://www.youtube.com/watch?v=656058JxTM0

Overview

Cliff Stoll, an accidental cybersecurity pioneer and renowned author of "The Cuckoo's Egg," took the stage at DEF CON to recount his legendary hunt for a German hacker 40 years ago. This talk, "Stalking the Wily Hacker: 40 Years Later," delves into the origins of modern incident response, intrusion detection, and honeypots, all born out of a 75-cent accounting error at Lawrence Berkeley Labs in 1986. Stoll, then an astronomer, found himself thrust into the nascent world of network security, navigating a landscape devoid of established protocols, tools, or even a recognized "cybersecurity industry."

The presentation is a vibrant and passionate journey through a pivotal moment in internet history, illustrating how ingenuity, persistence, and an unwavering sense of outrage transformed a seemingly minor system anomaly into an international espionage investigation. Stoll’s narrative transcends a mere technical recounting; it highlights the profoundly human element of cybersecurity, emphasizing the emotional investment required to confront adversaries and protect digital frontiers. His story serves as a foundational text for understanding the challenges and triumphs that shaped the field, offering timeless lessons for both seasoned professionals and newcomers.

This talk is not just a historical retrospective; it’s a powerful reminder that while technology evolves at a dizzying pace, the core principles of vigilance, creativity, and a moral compass remain indispensable in the ongoing battle against digital threats. Stoll's experience, predating modern cybersecurity frameworks, underscores the importance of fundamental observation and critical thinking, proving that even in the absence of advanced tools, a determined individual can make an indelible impact on the security landscape.

Background

▶ Watch: Introduction and unique presentation style (0:00)

The year was 1986, a nascent era for the internet, then known as the Arpanet. Cliff Stoll, an astronomer at Lawrence Berkeley Labs (LBL), had just completed the optical design for the Keck 10-meter telescope and was reassigned to manage a cluster of UNIX boxes in the basement of Building 50. At this time, the Arpanet was a relatively small, interconnected network, boasting perhaps 400 to 700 nodes and an estimated 4,000 to 8,000 users online. This digital commons was fundamentally built on mutual trust, a stark contrast to today's pervasive zero-trust models. Computing resources were valuable commodities; CPU cycles and compute time were explicitly charged, unlike the ubiquitous, often free, access we have today.

It was into this environment that Stoll stumbled upon an anomaly that would redefine his career and contribute significantly to the nascent field of cybersecurity. On his second day on the job, the accounting program for LBL’s systems crashed. The root cause appeared to be a minor 75-cent accounting error associated with a new user. However, Stoll, as the sole system manager authorized to add users, immediately recognized that something was amiss. The presence of an unauthorized user in the password file, coupled with the accounting system's failure, raised a critical question: "Could somebody be on my system? How could I tell?"

At this time, formal cybersecurity practices, tools, and incident response protocols simply did not exist. There was no "cybersecurity industry," no bug bounties, and certainly no DEF CON. Stoll had no formal training in network security, nor could he consult online forums or specialized experts. The existing organizational structures, including his funding agency (the Department of Energy), were initially hesitant to support an investigation, fearing negative publicity and potential funding cuts if a "hacker in a DOE computer" became public knowledge. This lack of resources, expertise, and institutional mandate forced Stoll to improvise, relying on his scientific training, a DIY ethos, and a healthy dose of resourcefulness to confront an unknown adversary.

Key Findings

▶ Watch: Realizing an unauthorized user was added (4:32)

Cliff Stoll's investigation, which spanned several months, yielded a series of critical findings that not only led to the identification and apprehension of the hacker but also inadvertently laid the groundwork for modern cybersecurity practices.

Initially, Stoll discovered that the unauthorized access wasn't a random occurrence but a systematic intrusion. The hacker utilized a cron tab script to elevate their privileges to root status, subsequently disabling the accounting program to mask their activities. From LBL, the attacker was leveraging the local area network and a Cisco router (one of the first three ever built) to traverse the Arpanet and systematically target military computers on the Milnet. The hacker's primary objective became clear as Stoll observed them using grep commands to search for highly sensitive keywords such as "ballistic missile," "plutonium," and "nuclear" within these military systems, indicating an intent for espionage rather than mere mischief.

A crucial technical breakthrough came from observing the hacker's specific UNIX commands. The attacker consistently typed PS -GXUA to list running processes. Stoll noticed that the -G flag was not a standard option in Berkeley UNIX (BSD), the operating system prevalent in Northern California. This pointed definitively to the hacker likely operating from an AT&T System V environment, providing a vital fingerprint that narrowed the geographical and technical scope of the search.

The investigation further revealed the hacker's capability to crack passwords. After downloading the encrypted /etc/passwd file from LBL, the attacker successfully created new accounts, demonstrating a sophisticated ability to compromise user credentials. Geographically, Stoll traced the hacker's path from Berkeley across North America to a military contractor, MITRE, in Northern Virginia, and subsequently across the Atlantic via X.25 networks (specifically TimeNet) to Europe, landing in a MicroVAX computer at the University of Bremen, and then into the German DEXP network in northwestern Germany.

Perhaps the most significant "findings" were the accidental inventions of foundational cybersecurity concepts. Lacking formal tools, Stoll pioneered intrusion detection by physically monitoring network connections and analyzing logs. His "Operation Showerhead" was an early form of a honeypot, designed to lure and retain the hacker online for long enough to enable a physical phone trace. The entire process also exemplified operational security (OpSec), as Stoll carefully manipulated the system to observe the attacker without tipping them off.

Finally, Stoll highlighted the deeply personal and ethical dimension of cybersecurity. His emotional investment in "chasing the rodent" led him to reject the detached, passive language often used to describe cyber incidents. He argued that cybersecurity is not merely a "technical puzzle" but a "human problem," involving real people making choices, causing harm, and necessitating judgment, a perspective that remains profoundly relevant today.

Technical Deep Dive

▶ Watch: The unconventional printer-based monitoring system setup (6:15)

Cliff Stoll's hunt for the wily hacker was a masterclass in improvisational forensics, conducted without the benefit of modern cybersecurity tools or protocols. His technical approach was a blend of astute observation, physical engineering, and creative problem-solving.

The initial detection mechanism was surprisingly low-tech but effective. Following the 75-cent accounting error and the discovery of an unauthorized user, Stoll transformed the basement of Building 50 into a makeshift monitoring station. He gathered two dozen printers and teletypes from various offices, wired them directly to the incoming modem bank using a soldering iron and Radio Shack clip leads. This setup ensured that every incoming connection and every keystroke typed by the attacker would be immediately printed on paper. Stoll would then sleep behind the router, reviewing the printouts each morning.

Through these printouts, Stoll meticulously pieced together the hacker's modus operandi. The attacker consistently used a cron tab script to execute a set user ID command, elevating their privileges to root. Once root, the hacker would disable the accounting program, effectively going dark. The attacker then used LBL as a pivot point, accessing the Arpanet and subsequently the Milnet via a Cisco router (reportedly the third one ever manufactured). On military systems, the hacker’s primary activity was using the grep command to search for highly sensitive keywords like "ballistic missile," "plutonium," and "nuclear."

A critical technical clue emerged from the hacker's use of the PS (process status) command. While most Berkeley UNIX (BSD) users would type PS -AXU to view all processes, the hacker consistently used PS -GXUA. Stoll recognized that the -G flag was specific to AT&T System V UNIX, an operating system not commonly used in Berkeley. This seemingly minor detail was a significant OS fingerprint, allowing Stoll to infer the hacker's likely technical environment and geographic origin, as it suggested the attacker was not from the BSD-dominated Northern California.

Tracing the hacker's connections involved a combination of primitive network analysis and social engineering. Stoll borrowed a "pullet hacker serial line analyzer" (a data communications analyzer) and programmed it to trigger his rewired Motorola Pageboy pager via Morse code whenever the hacker used specific accounts or passwords. This provided real-time alerts. He also used an old Tektronix oscilloscope to measure packet delay times between IP packets. By averaging several round-trip times, he calculated a delay of 2.84 seconds from Berkeley to the hacker's location. A rough calculation (2.84s / 2 speed of light, approximately 3 10^5 km/s) initially suggested a distance of 400,000 kilometers, humorously implying the hacker was "on the far side of the moon." While this specific calculation was a "dead end," it demonstrates a physicist's approach to network latency.

The geographical trace initially led to a military contractor, MITRE, in Northern Virginia. MITRE initially denied any breach, citing "air gaps" and secure systems. However, Stoll's persistence and the threat of public exposure convinced them to pull the plugs on their modems. Subsequently, the hacker shifted tactics, using X.25 links and traversing international networks like TimeNet to reach Europe, specifically a MicroVAX at the University of Bremen in Germany, and then the German DEXP network.

To facilitate a manual phone trace by German authorities, which required the hacker to stay online for 1-2 hours, Stoll devised "Operation Showerhead." This was an early form of a honeypot and a social engineering tactic. He created a large, fictitious file full of deliberately boring, bureaucratic documents about the "Strategic Defensive Initiative" (SDI), knowing the hacker was interested in military secrets. These documents included fake titles like "37.6 SDI network overview description document" and "41.7 functional requirements document," attributed to a fictitious "Mrs. Barbara Sherwin." The intent was to make the hacker spend considerable time downloading the seemingly valuable but ultimately useless information, thereby keeping the connection active long enough for the trace to complete. Stoll even referenced the era's common UNIX commands, noting the use of more before less became prevalent. This intricate technical and psychological trap ultimately led to the hacker's location.

Demo / Proof of Concept

▶ Watch: Lab director's order: 'catch the rodent' (10:55)

While Cliff Stoll's talk wasn't a live technical demonstration in the contemporary sense, his entire presentation served as a vivid recount of a real-world, pioneering "proof of concept" – the successful tracking and apprehension of an international cyber-spy. He effectively "demonstrated" the historical methods and tools used through his storytelling and visual aids.

Stoll utilized an overhead projector with original viewgraphs from a talk he gave at the NSA in 1987. These physical slides, some illegible due to their age, were themselves artifacts of the demonstration, showcasing the raw data and thought processes from the actual investigation. He explicitly pointed out a slide containing a copy of his lab notebook entry from September 17, 1986, underscoring the vital role of meticulous, written documentation in forensic investigations, especially in an era without digital logging infrastructure.

The core of Stoll's "demo" lay in his detailed explanation of the practical steps taken:

• Physical Monitoring Setup: He described wiring two dozen printers and teletypes to the incoming modem bank using a soldering iron and clip leads. This low-tech yet effective system printed every incoming connection and keystroke, serving as the primary intrusion detection and logging mechanism.
• OS Fingerprinting: His explanation of the PS -GXUA command and the significance of the -G flag in identifying the hacker's AT&T System V environment was a clear demonstration of how specific technical details can be leveraged for attribution.
• Packet Delay Measurement: Stoll recounted using a Tektronix oscilloscope to measure IP packet round-trip times, illustrating an early attempt at network latency analysis to infer geographical distance.
• "Operation Showerhead" Honeypot: This elaborate ruse was the ultimate proof of concept for social engineering and delaying an attacker. Stoll detailed the creation of the fake "Strategic Defensive Initiative" documents, complete with fictitious titles and an imaginary "Mrs. Barbara Sherwin," designed to keep the hacker engaged and online for the duration required for a physical phone trace by German authorities. He even presented a physical copy of one of these fake documents, providing a tangible link to the operation.

In essence, Stoll's talk was a masterclass in historical cybersecurity, demonstrating how foundational principles of observation, data collection, and psychological manipulation were applied in a resource-constrained environment to achieve a groundbreaking outcome. It proved that even without sophisticated software, a determined and creative defender could outwit a persistent adversary.

Defensive Implications

▶ Watch: The strategy: keep the hole open, monitor everything (11:43)

Cliff Stoll's 40-year-old adventure, while set in the early days of the internet, offers remarkably enduring defensive implications for today's cybersecurity landscape. His improvisational approach inadvertently pioneered many concepts that are now industry standards.

1. Proactive Monitoring and Log Review: The entire investigation began with a diligent review of a 75-cent accounting error. This highlights the timeless importance of meticulously monitoring system logs, network traffic, and financial records for anomalies, no matter how small. A minor discrepancy can often be the first indicator of a significant breach. Modern Security Information and Event Management (SIEM) systems and Extended Detection and Response (XDR) platforms automate this, but the principle of vigilance remains.
2. Intrusion Detection Systems (IDS) and Honeypots: Stoll's physical printer setup was a rudimentary, manual IDS, demonstrating the fundamental need to detect unauthorized access in real-time. His "Operation Showerhead" was a classic honeypot – a decoy system or data designed to lure and trap attackers, gaining intelligence and time. Defenders today deploy sophisticated honeypots and honeynets to study attacker tactics, techniques, and procedures (TTPs) and to divert them from production systems.
3. Understanding Attacker Behavior and Fingerprinting: The hacker's unique use of PS -GXUA to identify their AT&T System V environment was a critical piece of threat intelligence. Defenders must continuously analyze attacker tools, commands, and network footprints to build profiles, attribute attacks, and develop targeted countermeasures. This includes monitoring for specific indicators of compromise (IOCs) and indicators of attack (IOAs).
4. Operational Security (OpSec): Stoll’s decision to keep the "hole wide open" and make his systems "look stupid" was a deliberate OpSec strategy. By not immediately patching the vulnerability or alerting the hacker, he gained invaluable time to observe and collect evidence. Modern OpSec involves carefully managing information, concealing defensive capabilities, and controlling communication during an incident to maintain an advantage over adversaries.
5. The Human Element in Cybersecurity: Stoll's passionate rejection of detached, passive language ("access was obtained") underscores that cybersecurity is fundamentally a human problem. Defenders must cultivate a sense of ethical responsibility and emotional investment, recognizing that attacks impact real people and have real consequences. This fosters a more proactive and resilient defense posture, moving beyond merely technical puzzle-solving.
6. Resourcefulness and Creativity: Operating with zero budget and no formal training, Stoll demonstrated that creativity and resourcefulness are paramount. When facing novel threats or resource constraints, defenders must be able to adapt, improvise, and leverage existing tools in unconventional ways to achieve their objectives.
7. Collaboration and Information Sharing: While initial inter-agency cooperation was challenging, the eventual success of the investigation relied heavily on international collaboration with German authorities. Today, information sharing across organizations, industries, and national borders is recognized as crucial for combating sophisticated, globally dispersed threat actors.
8. Zero Trust Architecture: The Arpanet was built on mutual trust, a model that quickly proved unsustainable. Stoll's experience implicitly argues for the "zero trust" principle: never trust, always verify. Every user, device, and application should be authenticated and authorized, regardless of its location or previous access.
9. Importance of Documentation: Stoll’s physical lab notebook was his most powerful tool. Detailed, accurate documentation of incidents, observations, and actions taken is indispensable for forensic analysis, post-incident review, and legal proceedings.

In essence, Stoll's experience teaches that while the tools have evolved, the core principles of vigilance, understanding the adversary, strategic engagement, and human dedication remain the bedrock of effective cybersecurity.

Key Takeaways

• Pioneering Foundations: The "Cuckoo's Egg" incident, driven by a 75-cent accounting error, inadvertently laid the groundwork for modern cybersecurity concepts like intrusion detection, honeypots, and operational security, long before these terms existed.
• Resourcefulness is Paramount: Operating without budget, expertise, or mandate, Cliff Stoll demonstrated that creativity, improvisation, and a "screwdriver as a chisel" mentality are crucial for effective incident response, especially when facing novel threats.
• Understanding the Adversary: Observing the hacker's specific UNIX commands (e.g., PS -GXUA with the -G flag for AT&T System V) provided a critical technical fingerprint, highlighting the importance of deep analysis of attacker behavior for attribution and defense.
• The Power of Low-Tech Monitoring: Stoll's use of physically wired printers and a detailed lab notebook underscored that fundamental, even primitive, methods of data collection and logging can be profoundly effective and remain relevant for understanding system activity.
• Cybersecurity is a Human Problem: Beyond technical puzzles, Stoll emphasized that cybersecurity involves human choices, ethical considerations, and emotional investment. A detached, neutral perspective ("threat actor") fails to capture the full scope and impact of cyber incidents.
• International Collaboration is Essential: The successful apprehension of the hacker ultimately relied on persistent efforts and cross-border cooperation between US agencies and German police, proving that complex, geographically dispersed cyber threats demand a unified global response.

About the Speaker(s)

Cliff Stoll is a celebrated figure in the history of cybersecurity, widely recognized as an astronomer, author, and security pioneer. While initially an astronomer at Lawrence Berkeley Labs, his accidental foray into network security in 1986 led him to track down an international cyber-spy, an experience vividly chronicled in his best-selling book, "The Cuckoo's Egg."

Stoll's work during this period, conducted without formal training or budget, saw him inadvertently invent foundational cybersecurity concepts such as intrusion detection, honeypots, and operational security. He is known for his unique, passionate, and often humorous storytelling style, which brings to life the human element of technology and security. Beyond his pioneering work in cybersecurity, Stoll is also an artisan and proprietor of Acme Klein Bottles, where he continues his fascination with non-orientable surfaces and mathematical curiosities. He remains a beloved "greybeard" of the hacking community, inspiring new generations with his tales of early internet adventures and his unwavering enthusiasm.