Chaospager: How to construct an Open Pager System for c3

Chaospager: How to construct an Open Pager System for c3

Speakers: Max; Julian

Conference: 39C3

YouTube: https://www.youtube.com/watch?v=s9cQ0rsCipg

Overview

This talk, "Chaospager: How to construct an Open Pager System for C3," delves into the ambitious project of developing an entirely custom, open-source pager system designed specifically for large-scale events like the Chaos Communication Congress (C3). Presented by Max and Julian, representing a workgroup of passionate engineers from Hamburg, the project addresses the critical need for a robust, easy-to-use, and reliable communication system for emergency responders, often referred to as "angels," at such complex gatherings. The initiative was primarily driven by the TEAT (Emergency Response Team at C3 events), who sought an integrated and resilient paging solution.

The core of the Chaospager project revolves around building both the pager receivers and the accompanying POCSAG transmitter infrastructure from scratch. This includes custom hardware design, self-manufacturing processes, and the development of a modern, scalable software architecture for transmitter orchestration. The speakers meticulously detail the engineering challenges encountered during prototyping, the innovative solutions implemented, and the ongoing efforts to refine the system for optimal performance and energy efficiency. The talk not only showcases a significant technical achievement but also highlights the power of open-source collaboration in solving real-world emergency communication problems.

The significance of Chaospager lies in its commitment to openness and adaptability. By providing a transparent, community-driven alternative to proprietary paging solutions, it empowers event organizers and emergency services with greater control and customization capabilities. The project's emphasis on user-friendliness for both trained and untrained responders, coupled with its resilience to harsh environmental conditions and heavy abuse, makes it a compelling model for emergency communication systems in dynamic, high-stakes environments. Furthermore, its integration with existing dispatch systems like TOAD ensures seamless operational workflows, reducing additional burdens on dispatchers.

Background

▶ Watch: Introduction and the vision for an open pager system. (0:00)

The genesis of the Chaospager project stemmed from a fundamental need identified by the TEAT at C3 events: a dependable and intuitive pager system for their diverse team of emergency responders. Many of these "angels" are volunteers who use pagers in their professional roles but only interact with the C3 system for a few days a year, necessitating an extremely intuitive interface. The challenge was compounded by the unique environment of a congress, characterized by fluctuating temperatures (e.g., -2°C outside, 25°C inside), which can cause frequency drift in RF components, and the potential for "heavy abuse" during unexpected emergency situations where responders cannot prioritize the delicate handling of their equipment.

The initial prototype, described by Max as "loosely coupled components with some wires," quickly exposed several engineering hurdles. Early iterations of the custom main board, featuring an ESP32 development board and an SX1278 LoRa chip (chosen for its FSK support, crucial for POCSAG), suffered from poor reception performance and a high variance in receive frequency between modules, requiring individual calibration for each pager—a highly inconvenient process for mass deployment. Manufacturing the custom boards presented its own set of difficulties, including footprint misalignments that necessitated bending pins and persistent copper residue from laser structuring, invisible to the naked eye.

The technical foundation of the system is POCSAG (Post Office Code Standardisation Advisory Group), a one-way paging protocol widely used for emergency services. POCSAG operates using Frequency Shift Keying (FSK) modulation with a 4.5 kHz shift, typically running at 1200 baud. A key feature is its message filtering capability using a Radio Identification Code (RIC). Each pager is programmed with a list of RICs, and it only displays messages whose RIC matches one in its list, effectively filtering out irrelevant alerts at the receiver side. When multiple transmitters are required to cover a large area, Time Division Multiple Access (TDMA) is employed. The Net Core implementation uses 16 time slots, allowing multiple transmitters to broadcast the same message sequentially without interference, with receivers intelligently ignoring reception duplicates. This robust protocol formed the backbone of the Chaospager system, around which the custom hardware and software were designed.

Key Findings

▶ Watch: User-centric requirements: ease of use, resilience, long battery life. (4:00)

The Chaospager project yielded several significant findings and contributions, demonstrating the feasibility and efficacy of constructing an open, custom pager system for demanding environments.

First, the team successfully engineered a fully custom main board and a specialized RF board that addresses the shortcomings of initial prototypes. The RF board, built around the ubiquitous and cost-effective SX1262 radio chip, incorporates meticulous filtering and antenna matching to achieve good RF performance. The innovative dual-microcontroller architecture, combining an ESP32 for Wi-Fi/Bluetooth and an ultra-low-power ATtiny dedicated to energy-efficient POCSAG decoding and RIC filtering, stands as a key finding for optimizing battery life and processing.

Second, the talk showcased a remarkably effective self-manufacturing process for custom PCBs. Through careful application of solder paste using stencils, precise component placement (initially with human "pick and place" assisted by tools like Interactive-BOM and a microscope, later leveraging a FinePlace machine for near-perfect alignment), and vapor phase reflow soldering, the team achieved high-quality boards with minimal touch-ups. This hands-on approach allowed for rapid iteration and control over the entire production pipeline, a crucial capability for agile hardware development.

Third, the development and initial alpha testing of Net Core 2.0 alpha represents a pivotal finding in scalable pager infrastructure. This modern orchestration system, built on a microservice architecture using Python, CouchDB for horizontal scaling, and RabbitMQ for inter-service communication, demonstrated impressive performance. Benchmarks showed it could handle 2,000 messages per second with 200 transmitters on a single-core, 3GHz CPU with 2GB RAM. Its MIT license also facilitates broader adoption, including potential commercial applications.

Finally, the 39C3 congress itself served as a live, real-world "alpha test" for the entire Chaospager system. The deployment of three custom POCSAG transmitters across the congress area, orchestrated by Net Core 2.0 alpha, allowed the team to gather crucial data on coverage, reception sensitivity, and the system's overall stability in a challenging environment. This direct validation against real-world conditions, comparing lab measurements with live performance, is a fundamental finding in the project's development lifecycle.

Technical Deep Dive

▶ Watch: Intuitive four-button interface and limited feature set. (6:15)

The technical architecture of the Chaospager system is a testament to thoughtful engineering, balancing performance, energy efficiency, and ease of manufacturing.

Pager Hardware Design:

The pager's evolution began with a prototype featuring an ESP32 development board and an SX1278 radio chip. This initial iteration highlighted issues such as high RX frequency variance and suboptimal user interface. The refined design moved to a custom mainboard housing an ESP32 for higher-level functions like Wi-Fi and Bluetooth, paired with a custom RF board. This RF board is centered around the SX1262 radio chip, chosen for its ubiquitous availability and cost-effectiveness. The board design incorporates specific filtering and antenna matching to optimize RF performance. A crucial addition to the RF board is an ATtiny microcontroller. This dual-MCU approach allows the ATtiny to handle the energy-intensive task of POCSAG packet decoding and RIC filtering, only waking the power-hungry ESP32 when a relevant message is received. This design significantly improves energy efficiency, a critical requirement for a long-lasting pager. The pager features a user-friendly interface with four buttons: two soft keys (Message, Menu) with variable functions, and two hard keys (Up, Down) with fixed functions. The manual is directly on the screen, promoting intuitive operation.

Pager Firmware and Energy Efficiency:

Initially, the ESP32 directly received and decoded POCSAG packets using the RadioLib implementation. While functional, this approach introduced considerable overhead and was not energy-efficient. The ongoing development involves offloading this task to the ATtiny. The ATtiny will receive and decode the POCSAG packet, perform RIC filtering, and only forward necessary packets to the ESP32. This allows the ESP32 to remain in a deep sleep state for extended periods, dramatically reducing power consumption.

Custom Board Production:

A significant aspect of the project was the in-house manufacturing of the custom PCBs. The process involved:

1. Solder Paste Application: Using a stainless steel stencil, solder paste was precisely squeegeed onto all pads, ensuring appropriate coverage for fine-pitched components (0.5 mm spacing).
2. Component Placement: Early stages relied on a "human pick and place machine" leveraging the Interactive-BOM KiCad plugin for efficient manual placement. A soldering microscope was essential for handling tiny components, some roughly the size of a grain of sand. For higher precision and yield, the team utilized a FinePlace machine, which uses an optic mirror system to perfectly align components before placement, resulting in zero touch-ups for those parts.
3. Reflow Soldering: Boards were then processed in a vapor phase reflow oven. This method provides highly uniform heating by condensing vapor (at 230°C for their solder) onto the components. This prevents overheating of sensitive parts and largely excludes oxygen from the process, minimizing solder oxidation. This method is also highly efficient for bulk soldering multiple boards simultaneously. The boards are four-layer designs, with one layer dedicated as a solid ground plane beneath all RF components to ensure optimal RF performance and reduce electromagnetic interference (EMC).

POCSAG Transmitter Architecture:

The transmitter infrastructure is designed for robust and wide-area coverage. The core radio is a Motorola GM1200, a commercial radio chosen for its licensed status, clean RF transmission, and 10-watt output power. This radio is controlled by a Raspberry Pi 3. A custom Raspberry Pi header board facilitates the interface, providing:

• Level conversion for audio signals.
• PTT (Push-To-Talk) control to enable radio transmission.
• Power distribution for both the radio and the Pi, including a voltage converter for the Pi's 5V requirement.
• Reverse polarity protection and fuses for electrical safety.

This custom header minimizes external wiring, simplifying deployment. The transmitter units are housed in 19-inch enclosures, designed for plug-and-play operation (12V power, network connection).

Network Orchestration with Net Core 2.0 Alpha:

For managing multiple transmitters across a large area, the team adopted TDMA (Time Division Multiple Access). The Net Core framework implements 16 time slots, allowing synchronized transmission from different units while enabling receivers to ignore duplicate messages.

The final orchestration solution is Net Core 2.0 alpha, a significant upgrade from its predecessor. It features a modern microservice architecture built with Python. Key components include:

• CouchDB: A NoSQL database used for data storage, chosen for its ease of horizontal scaling and synchronization capabilities (clustering).
• RabbitMQ: An MQTT broker facilitating asynchronous communication between microservices, ensuring reliable message delivery.
• REST API: Provides an interface for external systems, crucially for integration with TEAT's TOAD dispatch system.

Net Core 2.0 alpha is released under an MIT license, making it suitable for both amateur and commercial applications. Benchmarks demonstrate its high performance, handling 2,000 messages per second with 200 transmitters on a 1-core, 3GHz CPU with 2GB RAM.

Demo / Proof of Concept

▶ Watch: Implementing ATtiny for power-efficient packet decoding. (7:50)

The 39C3 congress itself served as the primary demonstration and large-scale proof of concept for the Chaospager system. While a formal, staged demo wasn't presented, the entire setup was deployed and operational for the duration of the event, inviting direct interaction and real-world testing from attendees.

The speakers showcased physical prototypes of the pager, including one fully enclosed and another revealing the internal custom PCBs with the ESP32, SX1262, and ATtiny components. Attendees were encouraged to visit the Chaospager assembly area to inspect the hardware firsthand.

A key aspect of the live demonstration was the active participation of the congress community. The team invited attendees to:

• Try out the new pager prototype: Gaining direct user feedback on the UI/UX.
• Register in the pager call book: Becoming part of the live testbed.
• Send custom POCSAG messages: Licensed amateur radio operators could send messages directly, while unlicensed individuals were offered assistance.
• Receive messages on their own hardware: Many attendees brought their own amateur radio pagers or Flipper Zero devices, receiving messages on the standard amateur radio frequency.
• Submit reception reports: This was crucial for mapping the congress building's reception strength and identifying blind spots, providing invaluable data for system optimization.

To generate meaningful test traffic, the team implemented a "test payload" system utilizing talk announcements from the congress schedule. This data, considered "equal data" for experimental purposes, was broadcast via the transmitter network. A PHP8-based software written by Christian allowed for the selection of different talk groups (RICs), enabling targeted messaging and further testing of the RIC filtering capabilities. This practical, community-driven demonstration underscored the project's open-source ethos and its commitment to iterative development based on real-world feedback.

Defensive Implications

▶ Watch: Julian introduces and demonstrates the physical pager boards. (8:40)

While the Chaospager project focuses on building an open communication system rather than uncovering vulnerabilities, its design choices and operational considerations carry significant defensive implications, particularly concerning reliability, security, and resilience in emergency contexts.

Security Considerations for POCSAG:

A primary defensive implication highlighted by the speakers is the inherent lack of security in the POCSAG protocol itself. POCSAG is an unencrypted and open protocol, meaning anyone with a suitable receiver can listen to transmitted messages, and anyone with a transmitter can potentially spoof messages. The speakers explicitly stated that the system will not transmit personalized data, patient information, or any other critical, sensitive data. It is designed as an addition to existing, more secure communication systems (like DECT, which TEAT angels may already use) rather than a replacement for them. The primary function is to alert responders, who then use secure channels for detailed information. Users of the Chaospager system must be aware of this security limitation. While spoofing is possible, the move to commercial, licensed frequencies for future deployments could mitigate some of the risks associated with unauthorized transmissions, as these bands are typically regulated.

Reliability and Resilience of the Hardware:

The defensive posture of the hardware design is centered on robustness and operational longevity. The requirements for a pager resistant to "heavy abuse" and "ever-changing environments" directly address potential points of failure.

• Environmental Resilience: The design accounts for frequency drift due to temperature variations, a common issue in RF systems, ensuring consistent reception regardless of the environment (e.g., cold outside, warm inside).
• Durability: The physical design aims for resilience against drops and impacts, reflecting the realities of emergency work where equipment can be subjected to harsh treatment.
• Battery Life and Charging: The implementation of a long battery life (ideally lasting the entire congress) and 10-watt fast charging (recharge in under 30 minutes) reduces the risk of pagers being inoperable due to power depletion, a critical defensive measure in sustained operations. The discussion of swappable batteries, while challenging to implement durably, indicates a forward-looking defensive strategy for prolonged events.
• Dual-MCU for Efficiency: The ATtiny and ESP32 dual-microcontroller architecture enhances energy efficiency, extending operational time and reducing the frequency of charging, thereby minimizing downtime.
• Transmitter Redundancy and Orchestration: The use of multiple POCSAG transmitters orchestrated by Net Core 2.0 alpha with TDMA and duplicate reception handling ensures broad coverage and message delivery even if one transmitter experiences issues. The Raspberry Pi header board design with reverse polarity protection and fuses also hardens the transmitter units against electrical faults.

Open-Source for Transparency and Community Defense:

The open-source nature of Chaospager—from hardware schematics to software—is a significant defensive advantage. It allows for:

• Peer Review: The community can scrutinize the design and code for potential vulnerabilities or design flaws, fostering a collective defense mechanism.
• Community Contributions: Other developers can contribute improvements, bug fixes, and new features, enhancing the system's overall resilience and security over time.
• Transparency: No hidden backdoors or proprietary secrets mean greater trust and accountability, crucial for emergency communication infrastructure.

In summary, while the POCSAG protocol itself is not secure, the Chaospager project incorporates defensive strategies through its robust hardware design, energy-efficient operation, redundant transmitter infrastructure, and commitment to open-source transparency, all aimed at ensuring a highly reliable and operationally resilient emergency communication system.

Key Takeaways

• Open-Source POCSAG Pager System: The Chaospager project successfully developed and deployed a custom, open-source POCSAG pager system, including both receivers and transmitter infrastructure, specifically tailored for large events like C3.
• Innovative Hardware Design for Efficiency: The pager utilizes a custom RF board with an SX1262 radio chip and a dual-microcontroller architecture (ATtiny for low-power POCSAG decoding/filtering, ESP32 for Wi-Fi/Bluetooth) to achieve high energy efficiency and extended battery life.
• Effective In-House Manufacturing: The team demonstrated a highly effective self-manufacturing process for PCBs, employing solder paste stenciling, precise component placement with tools like Interactive-BOM and FinePlace machines, and vapor phase reflow soldering for high-quality, rapid prototyping.
• Scalable and Modern Transmitter Orchestration: Net Core 2.0 alpha, built on a microservice architecture with Python, CouchDB, and RabbitMQ, provides a robust, scalable, and MIT-licensed solution for orchestrating multiple POCSAG transmitters via TDMA, capable of handling 2,000 messages/sec with 200 transmitters.
• Designed for Emergency Response Realities: The system prioritizes ease of use for diverse responders, resilience against harsh environments and physical abuse, long battery life with fast charging, and seamless integration with existing dispatch systems like TEAT's TOAD.
• Security by Design (with Protocol Limitations): Acknowledging the inherent insecurity of the POCSAG protocol (unencrypted, open), the system is explicitly designed not to transmit sensitive data, serving as an alert mechanism complementary to more secure communication channels.

About the Speaker(s)

The Chaospager project was presented by Max and Julian, who are part of a core team of nine passionate engineers and "nerds" based in Hamburg, Germany. Many members of their workgroup, including some of the speakers, are also actively involved in emergency service organizations, bringing valuable real-world experience and understanding of the challenges faced by responders to the project.

Max provided the overall concept and vision for the pager system, detailing the requirements, UI/UX design, and the broader POCSAG infrastructure, including the integration with TEAT's TOAD dispatch system and the development of Net Core 2.0 alpha. His role encompassed the high-level architecture and the strategic direction of the project.

Julian focused on the intricate hardware design and manufacturing aspects, particularly the custom RF board with the SX1262 chip and the ATtiny microcontroller. He described the meticulous process of PCB assembly, from solder paste application to component placement using specialized tools and vapor phase reflow soldering, demonstrating his expertise in electronics production and RF engineering.

Together, Max and Julian represent a collaborative effort, combining software and hardware expertise to deliver a functional, open-source solution for critical communication needs. Their deep involvement in both the technical development and the practical application of emergency services underscores their commitment to the project's success.