Experimental Analyses of the Physical Surveillance Risks in Client-Side Content Scanning

Experimental Analyses of the Physical Surveillance Risks in Client-Side Content Scanning

Speakers: Ashish Hooda

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=ybyVVseMTDA

Overview

The proliferation of end-to-end encryption (E2EE) in digital communication platforms, while a boon for user privacy, has presented a significant challenge for detecting and mitigating the spread of illicit content, such as child sexual abuse material (CSAM). To circumvent this, Client-Side Image Scanning (CSIS) has emerged as a proposed solution, where content is scanned directly on a user's device before encryption. These systems employ perceptual hashing algorithms like PhotoDNA or PDQ to identify visually similar images against a database of known illicit content, theoretically preserving privacy by only matching hashes.

However, CSIS proposals have been met with widespread criticism, primarily due to concerns about potential misuse for censorship or surveillance by powerful adversaries like nation-states. This talk, presented by Ashish Hooda, directly addresses a critical gap in understanding these risks: the specific potential for attackers to manipulate CSIS systems to perform physical surveillance on target locations. The research experimentally characterizes how an adversary could exploit the CSIS database to effectively turn users' smartphones into an intermittent, crowd-sourced camera for monitoring physical scenes.

This work is crucial for informing the ongoing debate surrounding CSIS technologies. By rigorously demonstrating the feasibility and stealth of such physical surveillance attacks, it highlights a fundamental and undesirable trade-off inherent in CSIS design: increasing the system's robustness to detect illicit content inevitably amplifies its vulnerability to surveillance. The findings underscore the complex ethical and technical dilemmas that must be confronted before widespread deployment of CSIS.

Background

▶ Watch: Introduction: CSIS and its privacy challenges (0:00)

The impetus for Client-Side Image Scanning (CSIS) stems directly from the widespread adoption of end-to-end encryption (E2EE) across popular communication and cloud storage services. E2EE prevents service providers from accessing plaintext user data, thereby rendering traditional server-side content moderation — where images are scanned after upload but before encryption — impossible. CSIS is designed as a mechanism to selectively relax these encryption guarantees by performing content analysis on the client device, such as a smartphone, before the data is encrypted and transmitted. If a match with known illicit material is detected, the system flags the content, potentially leading to decryption and reporting to law enforcement.

A CSIS system typically involves a triad of entities:

1. Content Curator: Non-profit organizations like the National Center for Missing and Exploited Children (NCMEC) in the US or the Internet Watch Foundation (IWF) in the UK. They collect illicit material, compute perceptual hashes for this content, and maintain a highly confidential database of these non-invertible binary hashes.
2. Service Provider: Companies deploying the CSIS protocol within their applications (e.g., messaging apps, cloud storage). User devices match locally generated perceptual hashes against the curator's database.
3. Law Enforcement: Receives reports of flagged content, which may undergo additional review to handle false positives before legal action.

At the core of CSIS technology are perceptual hashing algorithms. Unlike cryptographic hashes, which are highly sensitive to even minor input changes, perceptual hashes are engineered to produce similar outputs for visually similar images. This allows for the detection of illicit material even if it has undergone minor transformations such as re-coding, re-sizing, or minor color adjustments. Formally, a perceptual hashing function P maps an image X to a bit string. An image X is flagged if the Hamming distance D(P(X), c) is below a threshold t for any hash c in the curated database C. Prominent examples include Microsoft's PhotoDNA and Facebook's PDQ. The research also utilized SSCD-Hash, a learning-based approach based on Self-Supervised Copy Detection.

The security community has previously identified several threat categories concerning CSIS:

• Extraction of Source Images: Attempts by attackers to invert perceptual hashes to retrieve original illicit images.
• Evasion of Detection: Adversaries subtly modifying illicit images using gradient-based or gradient-free techniques to alter their hashes and bypass detection.
• Triggering False Detection: Crafting benign images whose hashes match database entries, leading to wrongful flagging, deanonymization, or framing of innocent users.

This research significantly expands upon the "Triggering False Detection" category by introducing and empirically characterizing a novel dimension: physical surveillance. This attack paradigm allows an adversary, particularly state actors, to exploit hash collision attacks not just to flag benign digital images, but to monitor real-world physical spaces through the CSIS system.

Key Findings

▶ Watch: CSIS system overview: curator, provider, law enforcement (2:00)

The research provides several critical insights into the physical surveillance risks of Client-Side Image Scanning:

• Feasibility of Targeted Physical Surveillance: The study definitively demonstrates that an adversary can conduct highly effective, targeted physical surveillance. By poisoning approximately 0.2% of the illicit hash database (1,000 hashes), surveillance rates averaging 36.2% for SSCD-Hash and 23.7% for PDQ were achieved across diverse target scenes, reaching over 77% in specific indoor settings.
• Stealthy Operation with Minimal False Positives: The attack has a negligible impact on the overall false positive rate (FPR) of the CSIS system. The FPR increased only marginally from 0.412% to 0.474% for SSCD-Hash and from 0.218% to 0.267% for PDQ. This stealthy nature allows the attack to hide within normal system fluctuations, making detection by service providers extremely difficult.
• Fundamental Trade-off in CSIS Design: A core finding is the unavoidable tension between a CSIS system's ability to robustly detect syntactically varied illicit content and its vulnerability to physical surveillance. Increasing the distance threshold to enhance illicit content detection simultaneously and more rapidly increases the potential for surveillance. Conversely, preventing surveillance necessitates a drastic reduction in the system's ability to detect transformed illicit images.
• Effectiveness of GMM-Based Poisoning: The proposed Gaussian Mixture Model (GMM)-based approach for generating optimal poison hashes significantly outperforms random hash selection, providing an average relative improvement of approximately 47% in surveillance rates. This highlights the importance of sophisticated hash selection for covering the diverse visual transformations of a physical scene.
• Impact of Environmental Factors: While environmental variations (lighting, scene layout, presence of unseen persons) can reduce surveillance effectiveness, the attack maintains significant capability. SSCD-Hash generally exhibited greater robustness to these changes compared to PDQ, indicating that more advanced perceptual hash functions might also be more susceptible to robust surveillance.

Technical Deep Dive

▶ Watch: Perceptual hashing: the underlying technology of CSIS (4:00)

The core of this research lies in demonstrating a novel physical surveillance threat model against CSIS systems. The adversary, typically a powerful entity like a nation-state, aims to monitor a physical scene without needing to install covert cameras. Their insight is that users carrying smartphones with CSIS enabled effectively become an intermittent, crowd-sourced surveillance network. This offers advantages in covertness, scalability, and reduced physical presence requirements.

The attack strategy focuses on poisoning the illicit content database, rather than attempting the more difficult and less stealthy task of backdooring the perceptual hash function itself. The illicit content database is confidential and consists of non-invertible hashes, making it an ideal target for subtle manipulation.

The adversary is assumed to possess several capabilities:

• Scene Images: The attacker can obtain approximately 500 images of the target scene, sourced from public platforms (Instagram, 3D tours), real estate websites, or prior physical access.
• Illicit Images: The attacker can obtain or generate new illicit-looking images, potentially through law enforcement data, past CSAM leaks, or AI generation tools.
• CSIS Database Access: The attacker can introduce poisoned images into the CSIS database, requiring the addition of roughly 0.2% additional hashes. This could occur via compromising the database supply chain (tricking curators into inserting perturbed illicit images) or by influencing the content curator through government compulsion to insert specially crafted hashes.
• Flagged Images Read Access: The attacker can access images flagged by the CSIS system, either through legislation mandating reporting or via an insider agent within the service provider.

The attack addresses two primary technical challenges:

Crafting Optimal Poison Hashes

Perceptual hash functions are designed for invariance to small transformations (re-coding, re-sizing). However, physical surveillance requires detecting a wide variety of images from a scene, which involves significant semantic changes due to perspective, viewing angles, and environmental conditions. Even a 5-degree rotation can alter over 10% of hash bits for PDQ and SSCD-Hash, limiting robust detection.

The key insight here is that CSIS detection relies on finding the best matching hash in the illicit database. To overcome the hash function's weak robustness to physical transformations, the adversary must insert an optimal, limited set of k poison hashes, U_k, that maximizes the probability of detecting images from a target scene S. This is formalized as a modified covering code problem, which is NP-complete.

The research proposes an approximate solution using Gaussian Mixture Models (GMMs):

1. Scene Hash Collection: Images of the target scene are hashed, creating a collection of l-dimensional data points (e.g., 256 bits for PDQ/SSCD-Hash).
2. GMM Initialization: A GMM is initialized with K components, where K is the number of poison hashes to be inserted. K-means clustering is used for initial parameter estimation.
3. Expectation-Maximization (EM) Algorithm: The GMM parameters (mixture weights π_k, mean vectors μ_k, and covariance matrices Σ_k) are iteratively updated using the EM algorithm. The E-step calculates posterior probabilities of each data point belonging to each Gaussian component, and the M-step updates parameters to maximize the expected log-likelihood.
4. Binarization: After convergence, the mean vectors (μ_k) of each component are rounded to binary strings, forming the optimal set of poison hashes.

This GMM-based approach effectively approximates the distribution of image hashes from a target scene, allowing for the selection of a compact set of poison hashes that cover a wide range of visual variations within that scene.

Generating Poison Delivery Images

Once the optimal poison hashes are computed, the next step is to insert these hashes into the illicit content database. Directly submitting scene images is not feasible as they are not illicit. Therefore, poison delivery images must be created: images that perceptually look like illicit content to a human curator but whose perceptual hash values are crafted to be close to the desired poison hashes.

This is achieved using techniques from adversarial examples:

• An attacker starts with a known illicit image (x).
• Carefully crafted adversarial perturbations are applied to generate a new image (x') such that:

1. ||x' - x||₂ ≤ ε: x' remains visually similar to the original illicit image x, making it appear illicit to a human curator.
2. P(x') ≈ h: The perceptual hash of x' is very similar (low Hamming distance) to one of the target poison hashes h from the optimal set U_k.

The specific perturbation method depends on the hash function:

• For Learning-Based Hash Functions (e.g., SSCD-Hash): These are often differentiable. White-box optimization techniques like Projected Gradient Descent (PGD) are used. PGD iteratively applies small perturbations in the direction of the gradient of a loss function (minimizing D(P(x'), h)) while staying within an L∞ perturbation budget.
• For Non-Learning Based Hash Functions (e.g., PDQ): These are typically non-differentiable. Black-box attack methods that rely on zero-order gradient estimation, such as Natural Evolutionary Strategies (NES), are employed. NES queries the hash function multiple times with slightly varied inputs to estimate the gradient and guide the perturbation process.

For each GMM-generated poison hash, the attacker attacks every image in their illicit image set (X_inject) and selects the perturbed image that yields the lowest Hamming distance to the target poison hash.

Demo / Proof of Concept

▶ Watch: Physical surveillance: a novel and expanded CSIS threat (5:00)

The experimental setup was designed to rigorously evaluate the feasibility and effectiveness of the proposed physical surveillance attack without using actual illicit content.

1. Datasets and Hash Functions:

• Illicit and Benign Image Approximation: The ImageNet dataset was used. A subset of 500,000 images approximated illicit content for the CSIS database, and 5,000 images served as the attacker's X_inject set. Another 10,000 images approximated benign user content for false positive rate evaluation.
• Scene Image Dataset: Six physical locations were chosen:
• Popular Tourist Spots (4 locations): Pisa Tower, Pyramids Giza, Lennon Wall, and Stone Henge. Images were scraped from Instagram using hashtags and manually validated.
• Indoor Settings (2 locations): Room 1 and Room 2. These were physically captured by the authors, including extensive 6-minute videos for the adversary's reference and 2-minute videos for user simulations. This allowed for varying camera configurations (two phone cameras) and environmental conditions (3 lighting settings, 3 room layouts, presence of unseen moving objects/people).
• Images for each location were split into a Reference Set (for the adversary) and a User Set (simulating user uploads).
• False Positive Dataset: The Places365 dataset (over 10 million images across 400 scene categories) was used for comprehensive false positive rate evaluation.
• Perceptual Hash Functions:
• PDQ: Facebook's official implementation, outputting a 256-bit DCT-based hash.
• SSCD-Hash: A learning-based hash function. A ResNet-50 model was trained on the DISC dataset, outputting a 512-dimension feature vector then quantized to a 256-bit hash.
• Distance Thresholds: Optimal thresholds were determined by analyzing the detection rate of benign images, selecting the inflection point where the false positive rate (FPR) significantly increased. This resulted in a threshold of 0.295 for PDQ and 0.09 for SSCD-Hash, both yielding an FPR of approximately 0.2% (around 2,000 FPs per million images).

2. Optimal Hash Generation:

• Reference Set images were augmented using random affine transforms to increase their effective size to 100,000 images per location.
• The GMM-based approach was applied to these augmented hashes, using 1,000 components (the poison budget), initialized with K-means, and run for 50 iterations. Mean vectors were binarized. This was compared against a random hash selection strategy. The poison budget of 1,000 hashes represented 0.2% of the simulated CSIS database size.

3. Poison Delivery Image Generation:

• An L∞ perturbation budget of 8/255 was used.
• For SSCD-Hash (differentiable), a white-box PGD attack with 1,000 iterations was employed.
• For PDQ (non-differentiable), a query-based black-box NES attack with 10,000 samples for gradient estimation was used.

4. Evaluation Metrics: The primary metrics were surveillance rate (fraction of user images from the target location flagged) and false positive rate (detection of benign images).

Experimental Results

The evaluation addressed five key questions:

Q1. Effectiveness of Physical Surveillance:

The attack achieved high surveillance rates for all 6 target locations with a very low false positive rate.

• Average surveillance rate: 36.2% for SSCD-Hash and 23.7% for PDQ.
• Highest rates: Over 77% for Room 2 with SSCD-Hash and 42.4% for Room 2 with PDQ.
• Indoor settings generally yielded higher rates, suggesting that a larger number of high-quality reference images improve effectiveness. The attack is highly targeted, significantly increasing detection for target locations without impacting other benign images.

Q2. False Positives:

The attack had minimal impact on the overall false positive rate.

• Baseline FPR (without attack): 0.412% for SSCD-Hash and 0.218% for PDQ.
• FPR (with attack): Increased marginally to 0.474% for SSCD-Hash and 0.267% for PDQ.

This small increase ensures the attack remains stealthy and difficult for service providers to detect amidst normal system fluctuations.

Q3. Algorithm Parameters:

• Impact of Poison Budget: Surveillance rates increased significantly with more poisons. From 15% with 500 poisons (0.1% of DB) to 53% with 5,000 poisons (1% of DB) on average. This was particularly pronounced for indoor settings with more reference images.
• GMM vs. Random Hash Selection: The GMM approach consistently outperformed random selection, providing a relative improvement of approximately 47% on average, demonstrating its effectiveness in optimally covering the hash distribution of a scene.

Q4. Environmental Factors:

Ablation studies revealed the attack's robustness to varying conditions:

• Lighting Conditions: Surveillance rate decreased by approximately 34% for SSCD-Hash and 40% for PDQ under unseen lighting. However, it remained above 14%, with SSCD-Hash showing slightly more robustness.
• Scene Layout Changes: SSCD-Hash decreased by approximately 27%, while PDQ experienced a more substantial decrease of 65% for unseen layouts. SSCD-Hash was more robust to layout changes.
• Presence of Unseen Persons: Surveillance rate gradually decreased as the percentage of the image occupied by a person's Field of View (FoV) increased. For SSCD-Hash, it dropped from 65% (0% FoV) to 22% (25% FoV), and to 0% when FoV exceeded 35%. This shows that individual privacy can still be compromised, even if the attack becomes less effective with larger obstructions.

Q5. CSIS Robustness vs. Surveillance:

This analysis revealed a fundamental trade-off:

• Increasing Distance Threshold: Both surveillance rate and CSIS performance (illicit image detection) increased.
• Differential Growth: Crucially, the slope of the surveillance rate's increase was higher than that of CSIS performance, especially under medium and high variation settings. This means that as CSIS becomes more robust to detecting transformed illicit content, its vulnerability to physical surveillance grows at an even faster pace.
• Design Implications: To prevent surveillance, a much lower distance threshold would be needed, drastically reducing CSIS's ability to robustly detect inexact illicit images (e.g., SSCD-Hash detection dropping from 90% to 10% for low variations if threshold is lowered from 0.1 to 0.02). This highlights an undesirable dilemma for CSIS designers.

Defensive Implications

▶ Watch: Attacker's model: poisoning the CSIS database for surveillance (6:00)

The research underscores a fundamental dilemma in designing Client-Side Image Scanning systems: there is an inherent and undesirable trade-off between a system's ability to robustly detect transformed illicit content and its vulnerability to physical surveillance attacks. Maximizing one inherently compromises the other.

Several potential defenses were considered, adopting a systems-view approach:

1. Making Poison Delivery Images Harder: Techniques from adversarial machine learning, such as adversarial training, diffusion-based adversarial purification, or certified robustness, could be employed to increase the distortion required for an attacker to generate effective poison delivery images. However, these methods are primarily effective for deep learning-based hash functions like SSCD-Hash, which are differentiable. They may not be applicable or effective against non-differentiable hash functions like PDQ, leaving a significant vulnerability.

1. Augmenting CSIS with Out-of-Distribution (OOD) Detectors: Integrating an OOD detector could identify content that falls outside the expected distribution of illicit material. This would force attackers to not only achieve hash collisions but also to craft adversarial images that can trick the OOD algorithm into believing they are in-distribution. This adds another layer of complexity for the attacker but does not fundamentally solve the hash collision issue if the OOD detector can be bypassed.

1. CSAM Database Audit Methods: Implementing robust audit mechanisms, potentially involving independent auditors with complete access to the hash database and source content, as proposed by other researchers, could help. However, the challenge of distinguishing genuinely illicit images from subtly perturbed adversarial ones remains an unsolved and difficult problem. The imperceptible nature of the perturbations used in poison delivery images makes auditing for malicious insertions extremely difficult, as human curators would approve them as legitimate illicit content.

From the attacker's perspective, if benign images from non-target locations are inadvertently flagged (which the research showed to be a minimal increase), the attacker can simply filter these out using automated image classification or manual inspection. This further enhances the stealth of the attack, as these false positives are a small fraction of the overall system false positives and easily discarded by the adversary.

Ultimately, the most direct defense against the physical surveillance risk would be to significantly reduce the perceptual hash distance threshold. However, as demonstrated, this would lead to a substantial degradation in the CSIS system's primary function: the robust detection of illicit content under various transformations. This highlights the severe practical challenges in deploying CSIS systems that are both effective at their stated goal and resistant to sophisticated surveillance misuse.

Key Takeaways

• Client-Side Image Scanning (CSIS) systems, designed to detect illicit content on user devices, can be repurposed for targeted physical surveillance by powerful adversaries.
• Attackers can effectively poison the confidential CSIS illicit content database by inserting a small fraction (e.g., 0.2%) of specially crafted hashes, achieving significant surveillance rates (e.g., over 30% average) for target locations.
• The proposed Gaussian Mixture Model (GMM)-based approach for generating optimal poison hashes significantly improves surveillance effectiveness by covering a wide range of visual transformations within a physical scene.
• This physical surveillance attack is stealthy, as it causes only a minimal increase in the overall false positive rate of the CSIS system, making it difficult for service providers to detect.
• There is a fundamental and undesirable trade-off in CSIS design: increasing the system's robustness to detect inexact illicit images (by increasing the hash distance threshold) simultaneously and more rapidly increases its vulnerability to physical surveillance.
• Defending against such surveillance attacks is challenging, as effective measures (like lowering hash thresholds) severely degrade the system's primary function, and other defenses (like adversarial training or OOD detectors) have limitations or do not fully address the core issue.

About the Speaker(s)

Ashish Hooda is a researcher whose work focuses on the security and privacy implications of emerging technologies, particularly in the context of client-side content scanning. As the sole speaker presenting this detailed technical analysis at the NDSS Symposium, Hooda demonstrates expertise in adversarial machine learning, perceptual hashing, and the broader societal impacts of digital surveillance. His research aims to inform critical public and policy discussions by rigorously characterizing the potential for misuse in systems designed to address challenging societal problems like the spread of illicit content.