Invisible Reflections: Leveraging Infrared Laser Reflections to Target Traffic Sign Perception

Invisible Reflections: Leveraging Infrared Laser Reflections to Target Traffic Sign Perception

Speakers: Takami Sato

Conference: NDSS Symposium

YouTube: (no public video)

Overview

This talk introduces a groundbreaking and concerning vulnerability for Connected Autonomous Vehicles (CAVs): the Infrared Laser Reflection (ILR) attack. Presented by Takami Sato, this research highlights how an adversary can manipulate a CAV's traffic sign recognition system using invisible infrared laser light reflected off traffic signs. The core problem addressed is the limitations of prior physical adversarial attacks, which, despite their effectiveness, are visually detectable by humans. The ILR attack, in contrast, is entirely imperceptible to the human eye, leveraging the sensitivity of filterless image sensors common in some CAV cameras and the unique properties of coherent laser light.

The talk delves into the profound safety implications of such an attack, where a CAV might misinterpret a crucial traffic sign—for instance, perceiving a stop sign as a speed limit sign. This research not only formalizes a robust threat model for ILR attacks but also rigorously evaluates their effectiveness against prominent traffic sign recognition architectures using various IR-sensitive cameras. A sophisticated black-box optimization methodology was developed, enabling the attack to achieve a 100% success rate in controlled indoor environments and a remarkable 80.5% success rate in realistic outdoor, moving vehicle scenarios. Furthermore, the findings reveal that state-of-the-art certifiable defenses are largely ineffective, mis-certifying 33.5% of cases. To counter this, the talk proposes a novel detection strategy based on the physical characteristics of IR laser reflections, demonstrating a 96% detection rate in proof-of-concept tests.

Background

[▶ Watch: Introducing the Invisible Infrared Laser Reflection (ILR) Attack (0:00)]()

Vision-Based Traffic Sign Recognition systems are the cornerstone of safe operation for CAVs, transforming camera sensor outputs into real-time object recognition and classification via fast neural networks. These systems are integral to widely adopted platforms like OpenPilot and Tesla. The talk outlines two primary architectural types prevalent in production vehicles: Single-Stage Architectures (e.g., YOLO), which are computationally efficient but limited in the number of sign classes they can handle, often suitable for Level-2 autonomy; and Two-Stage Architectures, which use an initial object detector to crop a Region of Interest (ROI) containing the sign, followed by a second-stage classifier for detailed sign type determination, accommodating a larger array of signs. The ILR attack was designed to target both.

A critical aspect of the ILR attack hinges on the distinction between Human Perception of Visible and IR Light. Infrared (IR) light, with wavelengths between 780 nm and 1 mm, is invisible to humans. However, modern CMOS image sensors are inherently sensitive to a portion of the IR spectrum. To replicate human vision and ensure accurate color reproduction, commercial cameras typically incorporate IR filters to block this light. Crucially, some autonomous vehicles, such as the Tesla Model 3, employ cameras without these IR filters, primarily to enhance low-light detection capabilities. When an IR-sensitive camera images an object illuminated by an IR laser, the IR light can bypass imperfect color filters, appearing as false colors like purple, magenta, or orange in the output image. While a human might discern these false colors if the raw camera feed were displayed, CAVs generally do not show these to drivers, making the attack inherently stealthy.

The research then contextualizes ILR within Previous Adversarial Attacks. Deep Neural Networks are notoriously susceptible to adversarial examples, even in the physical world. Prior attacks on traffic sign recognition have involved physical stickers, visible pattern projections, or shadow shading. The fundamental limitation of these methods is their visibility, making them potentially detectable by humans. The ILR attack fundamentally differentiates itself by using light projection and reflection, avoiding destructive changes or physical alterations to the sign itself, thus leaving no visible artifacts.

Compared to other IR-based attacks like I-Can-See-the-Light (ICSL) and invisible masks, the ILR attack introduces three key differentiators:

1. Invisibility and Attacker Capabilities: Unlike prior IR attacks that often used non-coherent IR LED light, ILR employs coherent laser light. Coherent light maintains its phase, allowing a laser beam to remain tightly focused and persistent over long distances with minimal attenuation, even after reflection. This enables the optimization of projected patterns for long-distance attacks, achieving successful misclassification up to 25 meters away with only 26 mW of laser power. In contrast, LED-based attacks like ICSL often require significantly higher power (e.g., 30 Watts at 12 meters) and demand precise aiming at the victim camera, which is challenging for moving targets.
2. Continuous Tracking: Many light projection attacks necessitate accurate and continuous aiming at the victim sensor, a complex task given dynamic vehicle motion. The ILR attack, by leveraging diffuse reflection off the sign surface rather than direct injection into the camera, eliminates the need for a sophisticated, continuously tracking emitter. The IR laser emitter can remain static, demonstrating stable and prolonged misclassification in moving CAVs without continuous aiming.
3. Ambient Light Variations: Previous light-based attacks frequently struggled under varying lighting conditions, particularly in bright environments where increased total luminance diminishes contrast. The ILR attack is designed to be robust under diverse ambient light conditions.

The ILR attack is classified as a perception attack that alters a small portion of a traffic sign, making it an adversarial patch attack. Its effectiveness was specifically evaluated against state-of-the-art certified defenses such as PatchCleanser.

Key Findings

[▶ Watch: Ineffectiveness of certifiable defenses against ILR attacks (1:30)]()

The research on Invisible Reflections yielded several critical findings that underscore the severity and novelty of the ILR attack:

• Novel and Invisible Attack Vector: The introduction of the Infrared Laser Reflection (ILR) attack as a novel physical-world attack vector is a primary finding. It effectively manipulates CAV traffic sign recognition systems while remaining entirely invisible to human perception, a significant advancement over prior visible adversarial attacks.
• Exceptional Effectiveness: The ILR attack demonstrated remarkably high success rates. It achieved a 100% Attack Success Rate (ASR) in controlled indoor, static scenarios across various single-stage and two-stage traffic sign recognition models. Critically, it maintained a formidable 80.5% ASR in challenging outdoor, dynamic driving scenarios involving moving vehicles, highlighting its real-world applicability.
• Long-Range and Low-Power Capability: The attack is practical over significant distances, achieving successful misclassification up to 25 meters from the target sign with a minimal laser power of just 26 mW. This low power requirement makes the attack stealthier and more feasible.
• Robustness Across Conditions: ILR attacks proved robust under diverse environmental conditions, including varying artificial ambient light levels (e.g., 100-230 Lux for stop signs, 100% ASR across all tested light settings for speed limit signs), different victim camera positions (though lateral distance had a greater impact than longitudinal), and a range of laser projection angles.
• Sophisticated Optimization Methodology: A novel black-box optimization methodology was developed, integrating Image Difference-based IR Trace Modeling and DNN-based frame interpolation (FILM 67). This framework automatically generates optimal attack configurations (laser power, pattern size, position) to maximize misclassification, demonstrating the critical role of optimization for high success rates (compared to <20% ASR for random attacks).
• Inadequacy of Current Defenses: State-of-the-art certifiable defenses like PatchCleanser were found to be largely ineffective. PatchCleanser not only failed to protect against ILR attacks but also significantly degraded clean model accuracy (dropping by an average of 16% for 2%-pixel patch scenarios) and mis-certified 33.5% of cases (and up to 50% for 4%-pixel patch scenarios). This highlights a fundamental mismatch between existing patch defense assumptions and the nature of traffic sign recognition.
• Feasible Novel Detection Strategy: To counter ILR, the research proposes a novel Color-Frequency Detection Methodology that leverages the unique speckle features and false color characteristics of IR laser reflections. A proof-of-concept implementation achieved a high True Positive Rate (TPR) of 96% (daytime) and 92% (nighttime) with low False Positive Rates (FPRs) (2.7% and 6.7% respectively).
• Generality and Transferability: The ILR attack demonstrated generality across four different IR-sensitive cameras (Leopard OnSemi, Raspberry Pi HQ v1.1, Microsoft LifeCam HD-3000, Leopard OmniVision), achieving 100% ASR on all. It also worked with different laser wavelengths (780nm, 830nm, 980nm). Transferability varied; it was high across different training datasets but more nuanced across different model architectures (e.g., 100% ASR for EfficientNet but 0% for DenseNet121/ResNet50 against stop signs) and object detectors (YOLOv3 trained on COCO showed higher robustness with only 20% ASR).

Technical Deep Dive

[▶ Watch: Proposing a novel detection strategy for ILR attacks (1:45)]()

The ILR Optimized Attack Methodology is central to the effectiveness of this novel threat. The threat model formalizes the attacker's goal: to cause a victim CAV's vision-based traffic sign recognition system to misclassify a target sign, focusing on untargeted attacks where any misclassification constitutes success. The adversary employs an IR laser to project an invisible light pattern onto the sign.

The attacker's prior knowledge and assumptions include public camera specifications (e.g., presence of an IR filter), basic optics understanding, and black-box access to the victim CAV's traffic sign recognition model (e.g., by querying results or confidence scores through reverse-engineering vehicle communication). The attack is remote, requiring no firmware access. In typical attack scenarios, the attacker positions a static IR laser emitter along the roadside, in line of sight with a target sign, up to 25 meters away, leveraging adjustable lenses and supports.

The attack is parameterized by:

• Attacker-controlled parameters: das (distance from laser to sign), Pa (laser power in mW), D (diameter of the projected IR pattern, focusing on circular patterns), and (xb, yb) (coordinates of the IR pattern center on the sign surface).
• Scenario parameters (uncontrollable by attacker): dvs (distance from victim camera to sign), dav (distance from emitter to victim camera), and L (intensity of ambient light).

The Physics of IR Laser Reflections is crucial. When a coherent laser light beam strikes the rough surface of a traffic sign (typically aluminum alloy), it scatters in all directions, creating a speckle pattern—a random distribution of bright and dark spots visible to the CAV camera. Despite scattering, the waves retain some coherence, preserving the incident beam's directionality and circular shape.

A detailed ILR Attack Capability Study was conducted using a Leopard camera with an OnSemi AR032ZWDR image sensor (used by Baidu Apollo) and a 780 nm IR laser module (CivilLaser, 1 W max).

• Laser Power vs. Pixel Intensity: Experiments showed that a minimum of 2.4 mW of laser power was sufficient to create a visible speckle pattern and alter pixel values, representing less than 0.25% of the laser's capability. For powers exceeding 20 mW, blue pixels exhibited a significantly larger intensity variation (at least 30 units) compared to red and green pixels. Increasing ambient light L logarithmically decreased the average 8-bit pixel intensity variation, following a rate of 40.1log(I).
• IR Pattern Size vs. Pixel Intensity: The intensity offset in 8-bit pixel values decreased linearly as the IR pattern diameter D increased (controlled between 3.5 cm and 30 cm).
• Attenuation at Increased Lateral Distance: While lateral movement of the victim camera (dav) caused pixel intensity drops (up to 18% for 0 to 1.5 meters), this attenuation was negligible compared to the impact of pattern size D and emitter-to-sign distance das.

The ILR Optimized Attack Methodology (Figure 7 in the paper) comprises three main steps:

1. Image Difference-based IR Trace Modeling: To synthesize accurate attack traces, real-world traces are collected, and image differencing is applied to extract RGB intensity offsets caused by IR patterns at varying power and size. These offsets are then applied to a benign base image (Xbase) of the target sign to create a simulated attack-influenced image (XILR). Temporal image noise is mitigated by averaging ten consecutive frames. The camera's perceived surface color is modeled by measuring the average RGB offset from a baseline traffic sign surface.
2. Trace Image Interpolation: To overcome the impracticality of collecting every possible real-world trace, a novel interpolation method was designed. It leverages a DNN-based frame interpolation algorithm called FILM 67 to generate intermediate attack traces as video frames for increasing laser powers and trace diameters, crucially preserving the unique local spatial information of speckle patterns.
3. Optimization-based ILR Attack Generation: The trace modeling and interpolation are integrated into a black-box optimization framework. The goal is to minimize the confidence value of the target class (for untargeted attacks) by finding optimal values for D, Pa, and (xb, yb). To ensure robustness against environmental changes, the Expectation over Transformation (EoT) technique is used, which samples images from XILR by introducing Gaussian noise, brightness changes, rotation, and shear. Given the non-differentiable nature of the attack formulation, the Tree-structured Parzen Estimator (TPE) algorithm 68 from Optuna 69 is employed, which efficiently converges to the global minimum within the small four-variable search space.

Demo / Proof of Concept

[▶ Watch: ILR's unique advantages: coherent laser, reflection, robustness (4:00)]()

The talk's extensive "Evaluation and Results" sections effectively served as a comprehensive demonstration and proof of concept for the ILR attack's feasibility and effectiveness in various real-world conditions.

Indoor Controlled Environment:

Initial evaluations in a controlled indoor setting, using real aluminum stop and speed limit signs, established a baseline. The ILR attack achieved a 100% ASR across all tested single-stage (Faster R-CNN, YOLOv3) and two-stage (CNNs trained on ARTS, LISA, GTSRB datasets) traffic sign recognition models. This starkly contrasted with a baseline random attack, which yielded less than 20% ASR. The optimized attack configuration was found using laser powers from 2.4 to 80 mW and diameters from 10 to 30 cm. The DNN-based interpolation method proved crucial, achieving a 100% ASR and 92.5% Simulation Consistency Rate (SCR). The attack demonstrated generality across four different IR-sensitive cameras (Leopard OnSemi, Raspberry Pi HQ v1.1, Microsoft LifeCam HD-3000, Leopard OmniVision), consistently achieving 100% ASR. Furthermore, it achieved a 100% ASR up to 25 meters with only 26 mW of laser power, and worked across different laser wavelengths (780nm, 830nm, 980nm).

Outdoor Evaluation (Realistic Scenarios):

To mimic real-world conditions, the ILR attack was tested in both static and dynamic outdoor scenarios using OnSemi and OmniVision automotive cameras.

• Static Scenarios:
• Nighttime Attack (120 Lux): Achieved 100% ASR for GTSRB stop signs (45 mW, 23 cm diameter, 7% sign surface coverage), LISA speed limit signs (46 mW, 17% coverage), ARTS stop signs, and ARTS speed limit signs (115 mW, 28 cm diameter, 10.6% coverage). Higher power was generally required compared to indoor settings.
• Daytime Attack (982 Lux): Achieved 100% ASR for ARTS stop signs (226 mW, 31 cm diameter, 13.1% coverage), ARTS and LISA speed limit signs (52 mW, 17.5 cm diameter, 13% coverage), and GTSRB speed limit signs (115 mW, 31 cm diameter, 7.9% coverage).
• Dynamic Driving Scenarios: Videos were recorded with a victim camera in a car moving towards the sign from 12 meters away at speeds of 5, 8, and 13 km/h.
• Night Scenarios (OnSemi camera): The ILR attack achieved an ASR of 99% for ARTS and LISA speed limit models across all speeds. For stop signs, ASR was 90% for ARTS and 80.5% for GTSRB.
• Day Scenarios (OnSemi camera): ASRs remained high, with 98% for ARTS and LISA speed limit models and 80-100% for stop signs.
• Similar high effectiveness was observed with the OmniVision camera, especially in night driving conditions.

These comprehensive evaluations definitively demonstrate the ILR attack's high effectiveness and practicality in realistic, moving outdoor environments.

Defense Proof-of-Concept:

The research also included a proof-of-concept for a novel defense strategy. Using a random selection of real-world images (75 benign, 75 attack samples for day/night), the proposed Color-Frequency Detection Methodology achieved:

• Daytime: 96% True Positive Rate (TPR) and 2.7% False Positive Rate (FPR).
• Nighttime: 92% TPR and 6.7% FPR.

This effectively demonstrated the feasibility of detecting ILR attacks by analyzing their unique physical characteristics, specifically the high spatial frequency and monotonous false colors of speckle patterns.

Defensive Implications

[▶ Watch: Defining the ILR attack threat model and attacker's goal (4:40)]()

The demonstrated effectiveness of the ILR attack carries significant CAV Safety Implications. Achieving nearly 100% ASR in outdoor driving scenarios, particularly at night, severely undermines the reliability of CAV perception systems. An attacker could easily enhance this threat by deploying multiple laser emitters to affect wider areas or to generate saturated IR speckles, further increasing the attack's impact. The methodology allows for scaling laser power and trace size proportionally and adjusting for different sign surface colors.

Given these findings, the research strongly recommends that CAV companies either implement robust IR filters in their cameras or deploy advanced, robust defenses capable of detecting such invisible threats.

The vulnerability extends to both Single-Stage vs. Two-Stage Architectures. While generic object detectors like YOLOv3 (COCO) showed higher robustness (only 20% ASR for some cases), suggesting they might be suitable for single-stage systems or the first stage of two-stage systems, single-stage architectures are generally not scalable for the high SAE autonomy levels that require recognizing a vast number of traffic sign classes.

Existing generic defenses against adversarial patch attacks, specifically PatchCleanser, proved largely ineffective. Despite ILR traces being visible in camera images, PatchCleanser's core assumptions did not hold for traffic sign recognition. In experiments with simulated ILR attacks, PatchCleanser either offered no benefit or decreased model performance, with clean accuracy for benign cases dropping significantly (e.g., an average of 16% for 2%-pixel patch scenarios and 0% for 4%-pixel patch scenarios). More alarmingly, PatchCleanser mis-certified 33.5% of cases for the 2%-pixel patch scenario and 50% for the 4%-pixel patch scenario, meaning it incorrectly validated a wrong label. This failure stems from the fact that masking even small, critical portions of a traffic sign can lead to misclassification, invalidating PatchCleanser's assumption that predictions on images without adversarial pixels are invariant to masking. Furthermore, the ILR attack trace size is not fixed and can be increased without reducing stealth, and its circular speckle shape is not explicitly handled by PatchCleanser.

To address these limitations, the research proposes an Alternative Defense Strategy based on the unique physics of IR laser reflections, specifically the distinct speckle features and often monotonous false colors (magenta, purple, orange) that appear in the camera image. The Color-Frequency Detection Methodology involves:

1. Extracting salient regions from captured traffic sign images that may contain false colors.
2. Empirically measuring potential IR light color ranges specific to the camera sensor.
3. Applying a Gaussian smoothing low-pass filter to isolate and retain high-frequency regions characteristic of speckle patterns.
4. Identifying a potential ILR attack if more than 1% of an extracted region exhibits these high spatial frequencies.

Regarding Daytime Attack under Strong Sunshine, while the ILR attack demonstrated robustness to various ambient light conditions, it is unlikely to work effectively in direct, strong sunlight without resorting to Class 4 lasers (above 500mW), which introduce significant safety hazards and uncontrollability. However, it should still be at least as robust as attacks using incoherent light.

Finally, the research touches upon Laser Safety. The real-world experiments used Class 3-B lasers (below 500mW). While calculations indicate that a 45mW laser could potentially exceed Maximum Permissible Exposure (MPE) limits, increasing the beam diameter (e.g., to 5 cm) can reduce the energy density below MPE guidelines. The ILR attack configuration, utilizing a diverging beam, resulted in a 17 cm diameter at 45 mW, which falls within safe limits. All experiments were conducted in controlled environments by trained personnel, underscoring the importance of safety protocols.

Key Takeaways

• Novel and Invisible Threat: The Infrared Laser Reflection (ILR) attack is a new, physical-world adversarial attack that is entirely invisible to humans but highly effective against CAV traffic sign recognition systems.
• High Real-World Effectiveness: The ILR attack achieved a 100% success rate in indoor static scenarios and a remarkable 80.5% success rate in realistic outdoor dynamic driving conditions, leveraging low laser power (26 mW) over practical distances (25 meters).
• Advanced Optimization: A novel black-box optimization methodology, combining image difference-based IR trace modeling and DNN-based interpolation, is crucial for generating these highly effective and robust attacks.
• Ineffectiveness of Current Defenses: State-of-the-art certifiable defenses like PatchCleanser are largely ineffective against ILR attacks, failing to protect and even mis-certifying 33.5% of cases due to fundamental assumptions not holding for traffic sign recognition.
• Feasible Countermeasure Proposed: A new defense strategy based on detecting the unique color-frequency characteristics of IR laser speckle patterns demonstrated high detection rates (96% TPR daytime, 92% TPR nighttime) in proof-of-concept tests.
• Urgent Call for Action: CAV manufacturers must address this critical vulnerability by either incorporating IR filters into their cameras or implementing robust detection mechanisms for invisible IR laser attacks to ensure the safety and reliability of autonomous vehicles.

About the Speaker(s)

Takami Sato is a researcher associated with the work on "Invisible Reflections: Leveraging Infrared Laser Reflections to Target Traffic Sign Perception." This presentation at the NDSS Symposium highlights their contributions to understanding and mitigating novel adversarial threats to Connected Autonomous Vehicles.