Excuse me, what precise time is It?

Excuse me, what precise time is It?

Speakers: Oliver Ettlin

Conference: 39C3

YouTube: https://www.youtube.com/watch?v=dOt-zRIG5co

Overview

Oliver Ettlin's talk, "Excuse me, what precise time is It?", delves into the critical role of Precision Time Protocol (PTP) in achieving highly accurate time synchronization across network infrastructures. Primarily drawing from his extensive background in the professional audio and video (AV) broadcast and event technology industries, Ettlin illuminates how PTP addresses the inherent asynchronous nature of IP networks to deliver the ultra-low, predictable, and controlled latency required for synchronous media transport. The presentation goes beyond theoretical concepts, offering live demonstrations that highlight the practical challenges and solutions encountered when deploying PTP in real-world scenarios.

The talk is a vital resource for anyone involved in designing, implementing, or troubleshooting network-based AV systems, industrial control, scientific research, or financial trading platforms where timing accuracy is paramount. Ettlin not only explains the fundamental mechanisms of PTP but also showcases common pitfalls, such as the impact of network traffic, physical layer anomalies, and rogue devices, alongside their respective mitigations. His candid sharing of experiences, including live demo "fails" and vendor-specific quirks, provides invaluable insights into the complexities of achieving and maintaining nanosecond-level synchronization.

This presentation matters because precise time synchronization is no longer a niche requirement; it's fundamental to the reliability and performance of modern networked systems. From ensuring perfect lip-sync in a broadcast studio to coordinating power grids or tracking financial transactions with nanosecond precision, PTP underpins critical infrastructure. Ettlin's discussion serves as a practical guide for network engineers, AV specialists, and security professionals to understand, implement, and secure PTP deployments effectively, emphasizing the need for robust hardware support, careful configuration, and continuous monitoring to prevent costly time-related disruptions.

Background

▶ Watch: Introduction: Synchronizing professional A/V over networks (0:00)

The evolution of professional audio and video (AV) production from traditional, cable-heavy setups to flexible IP-based networks presented a significant challenge: maintaining the stringent timing requirements of high-quality media. Historically, AV systems relied on dedicated cables for each signal, ensuring inherent synchronization but leading to inflexible and complex infrastructures. The shift to sending audio and video over standard networks promised flexibility and efficiency but introduced a fundamental problem: IP networks are asynchronous. Packets can arrive out of order, at variable rates, and with no inherent timing guarantees, directly conflicting with the professional AV demand for ultra-low, predictable, and constant latency.

To overcome this, the industry sought a robust solution for precise clock synchronization, leading to the adoption of Precision Time Protocol (PTP), standardized as IEEE 1588. Unlike its less precise counterpart, Network Time Protocol (NTP) – jokingly referred to as "Not So Precise Time Protocol" – PTP is designed for nanosecond-level accuracy. PTP version 2 (v2) is widely used, with an even more precise version 2.1, known as White Rabbit, employed in highly demanding scientific environments like CERN.

PTP's core concept for synchronous media transport involves a simple yet powerful mechanism: when a transmitting device (e.g., a video camera or microphone) sends a packet, it embeds a timestamp indicating the exact moment of transmission. The receiving device (e.g., a loudspeaker or monitor) reads this timestamp, adds a predefined link offset (also called delay or buffer), and then plays out the media precisely at the calculated time. This entire process hinges on one critical prerequisite: all devices in the system must be synchronized to the exact same time. It doesn't necessarily need to be the actual Coordinated Universal Time (UTC); any consistent, shared time reference will suffice, provided there are no time jumps that could cause audio/video dropouts. However, for systems interacting with external clocks (e.g., multiple sites independently clocked by GPS), the time must be correct.

This synchronization concept is foundational to several open standards in the AV industry:

• AS67: An open standard for sending uncompressed audio over IP networks.
• ST 2110: A suite of standards for professional media over IP, which leverages AS67 for its audio components.
• AVB Milan: Another standard for audio and video over IP, which mandates specific PTP hardware support.

Beyond AV, PTP is a cornerstone in various other industries where precise timing is non-negotiable. This includes telecommunications (synchronizing GSM cell phone stations), scientific measurements (tracking earthquake origins with distributed sensors), financial trading (enabling high-frequency trades with nanosecond precision), and power grids (synchronizing different power plants). While technologies like NDI, Netflix, YouTube, and traditional broadcast TV typically do not employ PTP (leading to phenomena like hearing a neighbor's cheer before seeing the goal on TV), PTP-enabled systems ensure perfect synchronization across all connected devices.

Key Findings

▶ Watch: Synchronous media transport concept: timestamping packets (3:00)

Oliver Ettlin's presentation highlights several critical findings regarding the implementation and operation of Precision Time Protocol (PTP) in real-world network environments:

• PTP is essential for synchronous media: For professional audio and video, as well as critical industrial applications, PTP provides the necessary nanosecond-level synchronization that standard IP networks inherently lack. This enables precise media playout, prevents drift, and ensures a consistent user experience.
• Network traffic is a major source of jitter: Without specialized hardware, PTP's accuracy is highly susceptible to variable network traffic, which introduces jitter and significant time offsets. This necessitates the use of PTP-aware network infrastructure.
• PTP hardware support in switches is crucial: Features like transparent clocks and boundary clocks embedded in network switches are vital for mitigating jitter. Transparent clocks correct for packet processing delays within the switch, while boundary clocks offload the grandmaster and improve scalability by acting as local PTP masters.
• Physical layer anomalies cause asymmetry: Components such as Dispersion Compensation Units (DCUs) in fiber links, or even simple cable length differences, can introduce significant propagation delay asymmetries. PTP must account for these, often requiring specific asymmetry link compensation features in switches, which can be topology-dependent.
• Scalability requires boundary clocks: As the number of PTP time receivers grows, the load on the primary grandmaster (time transmitter) can become unmanageable. Boundary clocks solve this by enabling switches to synchronize their own internal clocks and then act as independent grandmasters for downstream devices, creating a scalable hierarchy.
• Grandmaster election is a potential vulnerability: The Best Master Clock Algorithm (BMCA), which determines the grandmaster based on priority, can be exploited by rogue devices. An unauthorized device with a higher priority can take over as grandmaster, causing network-wide time jumps and system disruptions.
• PTP implementations vary significantly: There is a wide disparity in how different vendors implement PTP, leading to varying levels of feature support (e.g., PTP roll master), performance, and even bugs related to device reboots, hardware timestamping, and compatibility with advanced network features like VLANs or stacking.
• Continuous monitoring and verification are mandatory: Due to the complexities and potential for errors, PTP deployments cannot be "set and forget." Constant monitoring with specialized measurement devices is essential to verify actual time offsets and ensure the system remains within acceptable jitter tolerances (e.g., ±500 nanoseconds for AV).
• Perfect synchronization (zero offset) is not achievable: While PTP aims for extreme precision, achieving an absolute zero-nanosecond offset is practically impossible. PTP v2 typically allows for a best-case accuracy of around ±20 nanoseconds, which is sufficient for almost all applications.

Technical Deep Dive

▶ Watch: PTP versions and terminology evolution (leader/follower) (5:00)

At the heart of PTP lies a hierarchical system of time transmitters (formerly called master, now leader or grandmaster) and time receivers (formerly slave, now follower). The grandmaster, often synchronized to an external, highly accurate source like a GPS/GNSS antenna (supporting Galileo, Glonass, Spu), broadcasts sync messages multiple times per second (e.g., eight times per second). These sync messages contain a timestamp, typically expressed in TAI time (International Atomic Time), which differs from UTC by a fixed offset (currently 37 seconds) to avoid the complexities and potential disruptions caused by UTC's leap seconds.

A crucial aspect of PTP is its ability to compensate for propagation delay over network cables. Signals travel at approximately two-thirds the speed of light through copper cables, equating to about 5 nanoseconds per meter. To account for this, time receivers send delay request messages to the grandmaster, which replies with delay response messages. By measuring the round-trip time and dividing by two, devices can estimate the one-way delay and adjust their clocks accordingly, adding this offset to the sync messages.

However, standard asynchronous networks introduce jitter, causing significant time inaccuracies. Ettlin demonstrated how adding traffic to a network could increase time offsets from negligible levels to 0-500 nanoseconds. To combat this, PTP relies heavily on PTP hardware support in network switches. The first level of support is the transparent clock. A transparent clock switch measures the exact time a PTP sync message spends being processed (queued) within the switch. It then updates a correction field within the PTP packet (or sends a separate follow-up message containing this correction) to inform the downstream time receivers about the accumulated delay. This allows receivers to subtract the switch's processing time, significantly reducing jitter. Ettlin showed this reducing jitter from 0-500ns to 50-80ns. The use of follow-up messages is common because the switch often cannot determine the processing time until after the packet has left its internal queue.

Another significant challenge arises from asymmetry in network links. Ettlin highlighted Dispersion Compensation Units (DCUs), which are essentially coiled lengths of fiber (e.g., 1-10km) used to compensate for chromatic dispersion in long-haul optical networks. While less common in modern SFPs, they are still present in many operational networks. These units, or even simply uneven cable lengths, can introduce asymmetric delays where packets travel faster in one direction than the other, leading to incorrect delay calculations. To address this, some switches offer asymmetry link compensation functions, which allow for manual or automatic adjustments based on Layer 1 topology to correct for these discrepancies.

For large-scale deployments, scalability becomes a concern. A single grandmaster can be overwhelmed by numerous delay requests from many time receivers. This is where the boundary clock comes into play. A boundary clock switch acts as a PTP follower to an upstream grandmaster, synchronizing its own internal clock. Once synchronized, it then functions as a PTP grandmaster for all devices connected to its downstream ports, effectively segmenting the PTP domain and distributing the load. This creates a chain of command, making the PTP system highly scalable.

The selection of the grandmaster is governed by the Best Master Clock Algorithm (BMCA). Each PTP-enabled device advertises its priority, clock quality, and other attributes. The device with the highest priority (lowest numerical value) is elected as the grandmaster. If priorities are equal, other criteria are used to break ties. This election process, while democratic, presents a security vulnerability: a rogue device with a higher priority can hijack the grandmaster role, causing network-wide time jumps. Ettlin demonstrated a solution for this: the PTP roll master feature on boundary clock switches, which allows specific ports to be configured to never accept sync or announce messages, effectively preventing unauthorized devices from becoming grandmasters. This is akin to DHCP snooping for PTP.

PTP's flexibility also leads to various PTP profiles, tailored for different industries. For instance, AV applications typically use multicast for sync messages, while telecommunications often use unicast. AVB Milan, an AV profile, uses MAC addresses instead of IP addresses for PTP packets. Message rates also vary. This fragmentation means that switches are often sold with support only for specific profiles, leading to interoperability challenges between different industry segments.

Finally, for end devices like Linux machines, achieving PTP accuracy requires hardware timestamping capabilities in the network interface card (NIC). This allows timestamps to be captured before the packet enters the operating system's kernel, bypassing OS-induced jitter. The PTP4L daemon, an open-source PTP implementation for Linux, can leverage these hardware timestamps (e.g., from an Intel 210 chipset) to achieve precise synchronization. The follow-up messages are linked to initial sync messages via sequence numbers, which, while enabling functionality, also present a potential attack vector if manipulated.

Demo / Proof of Concept

▶ Watch: Live demo setup and sync messages explanation (7:00)

Oliver Ettlin's talk was heavily punctuated by live demonstrations, showcasing the practical aspects and challenges of PTP implementation. The core setup involved:

• A time transmitter (grandmaster) locked to a GPS/GNSS antenna.
• Two network switches, configured to support PTP.
• Various time receivers, including a custom LED PTP wall clock (designed with ChatGPT's help) for visual confirmation.
• A professional Vaserant measurement device (looking like a small, specialized server) to precisely quantify time offsets and jitter.
• A long cable (100m) running from the time transmitter to the main clock for the initial setup.

The demonstrations unfolded as follows:

1. Initial Synchronization: Ettlin first showed Wireshark capturing the PTP sync messages from the grandmaster, sent eight times per second. The LED wall clock visually confirmed synchronization. The Vaserant measurement device, however, revealed an initial offset of 0 to 650 nanoseconds. This offset, Ettlin explained, was primarily due to the propagation delay of the 100m cable, which PTP attempts to compensate for using delay requests/responses.

1. Impact of Network Traffic: To illustrate a common real-world problem, Ettlin introduced additional network traffic between the two switches. As predicted, the Vaserant device immediately showed a significant increase in jitter, with the time offset becoming much more unstable and varying widely, demonstrating PTP's sensitivity to network congestion without proper hardware support.

1. Enabling Transparent Clocks: To mitigate the traffic-induced jitter, Ettlin logged into the switches and enabled PTP mode E2E transparent. This configures the switches as transparent clocks, allowing them to measure and correct for internal packet processing delays. After this change, the measurement device quickly showed a dramatic improvement, with the jitter reducing significantly to an offset range of 50 to 80 nanoseconds. This demonstrated the effectiveness of PTP hardware support in switches.

1. Dispersion Compensation Unit (DCU) Simulator: Ettlin introduced a "DCU simulator"—a 200-meter coiled cable—into the fiber link. This physical layer asymmetry immediately caused a 500 nanosecond time error on the measurement device, illustrating how such components can disrupt PTP's delay compensation. Removing the cable restored the time accuracy, highlighting the need for specific asymmetry compensation features in real-world deployments.

1. Rogue Grandmaster Attack: This was a compelling security demonstration. Ettlin plugged in a small, PTP-enabled headphone preamp into port 9 of one of the switches. This device, configured with a higher PTP priority (a lower numerical value, e.g., 13 compared to the grandmaster's 14), automatically became the new grandmaster via the Best Master Clock Algorithm (BMCA). The measurement device immediately registered a time jump, indicating the network's synchronization had been disrupted by the unauthorized device. This scenario would cause audio and video dropouts in a live production.

1. PTP Roll Master Defense: To counter the rogue grandmaster, Ettlin enabled the PTP roll master feature on switch port 9. This configuration prevents the port from accepting PTP announce or sync messages, effectively blocking the rogue device from influencing the grandmaster election. After a brief delay (and a humorous "power cycle" suggestion from the speaker as live demo issues arose), the original grandmaster regained control, and the time on the measurement device stabilized back to the corrected range. This demonstrated a critical security measure for PTP deployments.

1. Final State: After resolving the demo challenges and disabling the asymmetry-inducing cable, the system achieved an offset between 0 and 50 nanoseconds in boundary clock mode, which Ettlin humorously declared "good enough for the next ticket sale for 40C3."

Throughout the demos, Ettlin emphasized the importance of using tools like Wireshark and dedicated measurement devices to verify PTP accuracy, as visual indicators like the LED clock, while cool, don't provide the necessary precision for troubleshooting. The live nature of the demos, including the expected "fails," underscored the real-world complexities and the need for thorough testing and understanding of PTP.

Defensive Implications

▶ Watch: PTP time source (GPS/GNSS) and TAI vs. UTC (10:00)

The detailed insights and live demonstrations from Oliver Ettlin's talk provide crucial defensive implications for anyone managing PTP-enabled networks. Protecting PTP synchronization is paramount, as disruptions can lead to significant operational failures, data inaccuracies, and even security vulnerabilities.

1. Prioritize PTP Hardware Support in Network Infrastructure:

• Implement Transparent or Boundary Clocks: Network switches must support PTP hardware acceleration. Transparent clocks are essential to compensate for internal queuing delays, drastically reducing jitter. For larger, more complex networks, boundary clocks are superior, as they create a hierarchical PTP domain, offloading the grandmaster and improving scalability and resilience. When purchasing switches, explicitly verify PTP support and its specific mode (E2E transparent, P2P transparent, boundary clock).
• Activate and Verify: Simply having PTP-capable hardware is not enough; the features must be actively configured and rigorously tested. Ensure the correction fields are correctly applied, or follow-up messages are being sent and processed.

1. Address Physical Layer Anomalies and Asymmetry:

• Account for Cable Delays: Be aware that cable lengths and types (copper vs. fiber) introduce propagation delays. While PTP attempts to compensate, physical layer issues can create asymmetric delays (where travel time differs in each direction), leading to inaccurate clock synchronization.
• Utilize Asymmetry Link Compensation: If using components like Dispersion Compensation Units (DCUs) or if network topology dictates uneven path lengths, leverage switch features like "asymmetry link compensation." This may require careful mapping of Layer 1 topology and manual configuration to correct for known time differences.

1. Secure Grandmaster Election (BMCA):

• Implement PTP Roll Master/Port Security: The Best Master Clock Algorithm (BMCA), which elects the grandmaster, is a significant attack vector. A rogue device with a higher (lower numerical) priority can seize control of the network's time, causing widespread disruption. Implement PTP roll master or similar port security features on switches. This allows administrators to designate specific ports as "master" (allowing PTP messages from an upstream grandmaster) or "slave" (preventing a device from becoming a grandmaster), effectively creating a "DHCP snooping" equivalent for PTP.
• Careful Priority Configuration: Configure PTP priorities on all devices deliberately. Ensure that only designated, secure grandmasters have the highest priority, and non-grandmaster devices are configured with lower priorities.

1. Continuous Monitoring and Verification:

• Invest in Measurement Devices: Do not rely solely on device status indicators. Employ dedicated PTP measurement devices (like the Vaserant device demonstrated) to continuously monitor actual time offsets, jitter, and frequency stability.
• Establish Baselines and Alerts: Set baselines for acceptable jitter tolerance (e.g., ±500 nanoseconds for AV) and configure alerts for deviations. Proactive monitoring can identify drift or disruptions before they impact operations.

1. Understand Vendor-Specific Implementations and Bugs:

• Thorough Testing: PTP implementations vary widely across vendors. Expect differences in feature support, performance, and stability. Conduct thorough testing in a lab environment before large-scale deployment.
• Beware of Reboot Anomalies: Ettlin noted that devices often exhibit bugs related to reboots, sometimes causing temporary time jumps. Plan for graceful restarts and validate PTP re-synchronization behavior.
• Feature Gaps: Be aware that not all PTP features (e.g., PTP roll master) are supported on every switch or interface, nor do they always work across advanced network configurations like VLANs, stacking, virtual port channels, or MLAGs.

1. Mitigate GNSS Jamming and Spoofing Risks:

• Redundant Time Sources: For critical infrastructure, consider multiple, diverse GNSS antennas or alternative time sources to build resilience against GNSS jamming and spoofing attacks.
• Internal Oscillators: Grandmasters should have high-quality internal oscillators (e.g., rubidium, oven-controlled crystal oscillators) that can maintain accurate time for extended periods if external GNSS signals are lost.

1. Ensure PTP Profile Compatibility:

• Match Profiles: Different industries use different PTP profiles (e.g., AVB, AES67, SMPTE 2110, Telco). Ensure all devices and network equipment support the necessary profile. Mixing profiles without proper translation (which many switches do not natively support across different industry-specific firmware) can lead to interoperability issues.

By adhering to these defensive strategies, organizations can build robust, highly accurate, and secure PTP-synchronized networks, safeguarding critical operations from timing-related disruptions.

Key Takeaways

• PTP is Indispensable for Precision: Precision Time Protocol (PTP) is crucial for achieving nanosecond-level time synchronization across IP networks, a requirement far exceeding NTP's capabilities for professional audio/video, industrial control, and scientific applications.
• Hardware Support is Non-Negotiable: Effective PTP deployment absolutely requires hardware support in network switches, either as transparent clocks (correcting for internal delays) or, preferably, boundary clocks (acting as local grandmasters for scalability and resilience).
• Physical Layer Matters: Network asymmetry, caused by varying cable lengths or components like Dispersion Compensation Units (DCUs), can introduce significant time errors that PTP must compensate for, often requiring specialized "asymmetry link compensation" features in switches.
• Secure Grandmaster Election: The Best Master Clock Algorithm (BMCA) is vulnerable to exploitation by rogue devices. Implementing PTP roll master or similar port security features on switches is vital to prevent unauthorized devices from hijacking the network's time synchronization.
• Verify and Monitor Continuously: Due to vendor-specific implementations, potential bugs, and network complexities, PTP deployments demand rigorous monitoring with specialized measurement devices to ensure accuracy and identify any drift or jitter, rather than assuming it's working correctly.
• Expect Implementation Differences: PTP features and performance vary widely across network hardware vendors. Thorough testing and understanding of specific device capabilities are essential for successful and reliable large-scale deployments, especially concerning reboots, VLANs, and different PTP profiles.

About the Speaker(s)

Oliver Ettlin is a seasoned professional with a deep background in the audio and video industry, specifically within broadcast and event technics. His extensive experience has led him to work on projects involving Precision Time Protocol (PTP), where he has gained practical insights into the challenges and solutions of achieving precise time synchronization in complex networked environments. Ettlin is clearly enthusiastic about PTP, describing it as a "cool protocol" with robust feature sets, including redundancies. His talk reflects a hands-on understanding of the technology, sharing real-world experiences and offering practical advice derived from his work in the field.