CrowdGuard: Federated Backdoor Detection in Federated Learning

CrowdGuard: Federated Backdoor Detection in Federated Learning

Speakers: Phillip Rieger

Conference: NDSS Symposium

YouTube: (no public video)

Overview

CrowdGuard introduces a pioneering defense mechanism designed to robustly detect and mitigate backdoor attacks within Federated Learning (FL) environments. Presented by Phillip Rieger, this talk highlights the critical vulnerabilities inherent in distributed machine learning paradigms, particularly the threat of targeted poisoning attacks. Federated Learning, while offering significant privacy benefits by keeping sensitive training data on client devices, simultaneously opens avenues for malicious actors to inject hidden behaviors into global models without directly accessing client data. CrowdGuard addresses this challenge by employing a novel, privacy-preserving architecture that leverages Trusted Execution Environments (TEEs) on both client and server sides, enabling secure client-side model validation and robust server-side aggregation of feedback.

The talk underscores that existing FL safeguards often fall short, struggling with non-IID (non-independently and identically distributed) data, sacrificing model accuracy, or proving vulnerable to adaptive adversaries. Crucially, many prior defenses necessitate server access to local models, which creates new privacy risks. CrowdGuard circumvents these limitations by introducing a secure client-feedback-loop, where clients analyze the behavior of neurons in hidden layers of individual models using a novel Hidden Layer Backdoor Inspection Metric (HLBIM). This analysis, combined with an iterative pruning scheme and a stacked clustering mechanism for feedback aggregation, allows for precise identification and elimination of poisoned models.

CrowdGuard's significance lies in its ability to deliver 100% True-Positive and True-Negative Rates across diverse and challenging scenarios, including highly non-IID data and sophisticated adaptive adversaries, all while preserving the global model's main task accuracy and ensuring client data privacy. By integrating TEEs, CrowdGuard establishes a new benchmark for secure and trustworthy FL, making it a vital contribution to the field as FL adoption expands across sensitive applications like healthcare, mobile computing, and IoT threat detection.

Background

[▶ Watch: Introduction to CrowdGuard and federated backdoor problem (0:00)]()

Federated Learning (FL), as pioneered by McMahan et al., is an iterative process enabling multiple clients to collaboratively train a Deep Neural Network (DNN) without sharing their private local datasets. In each round t, a server S deploys a global model G_t to a subset of selected clients C_i. Each client C_i then initializes its local model L_i with G_t, trains it on its private local dataset D_i, and sends the updated L_i back to the server. The server then aggregates these local models, typically using FederatedAveraging (FedAVG), to form a new global model G_t+1. The formula for FedAVG is G_t+1 = G_t + δ(1/n Σ(L_i - G_t)), where δ* is the global learning rate and n is the number of participating clients.

A critical factor in FL's performance and security is the client data distribution. IID (Independently and Identically Distributed) data assumes all clients' data follows the same distribution, which is rarely the case in real-world applications. Non-IID data scenarios, where data distributions diverge, are common and can manifest as differing label counts, peak non-IID rates where one label dominates, or even 1-class or 2-class non-IID where clients possess data from only one or two labels. Many existing backdoor defenses struggle with these non-IID scenarios, a limitation CrowdGuard explicitly addresses by making no assumptions about data distribution.

Poisoning attacks in FL aim to compromise the learning process. These can be untargeted, seeking to simply degrade the model's overall accuracy, or targeted, commonly known as backdoor attacks. Our focus is on targeted attacks, which are more insidious because they aim to inject specific, hidden functionality into the model while maintaining its main task accuracy on benign inputs. The adversary's goal is to make the poisoned global model *G** predict a specific target label T_A when an input sample contains an adversary-controlled trigger I, effectively maximizing the Backdoor Accuracy (BA) on the trigger set. Trigger types include Pixel backdoors (specific pixel patterns), Label-Swap backdoors (mislabelling samples), and Semantic backdoors (triggers embedded as specific characteristics within the input, e.g., a striped background for a car). Adversaries can inject backdoors via Data Poisoning (manipulating local datasets with trigger-embedded samples, quantified by Poison Data Rate (PDR)) or Model Poisoning (manipulating training algorithms, hyperparameters, or loss functions directly). Adaptive attackers can employ sophisticated techniques like constrain-and-scale to make their poisoned model updates appear benign, bypassing simple defenses.

Trusted Execution Environments (TEEs) are central to CrowdGuard's privacy and security guarantees. TEEs are secure areas within a processor that isolate code and data in cryptographic enclaves, protecting confidentiality and integrity even from privileged software (e.g., OS, hypervisor). Remote attestation allows a party to cryptographically verify the authenticity and integrity of the code running within a TEE. Examples include Intel SGX, AMD SEV, and Nvidia Confidential Computing.

CrowdGuard operates under a cross-silo FL setup involving N clients C_1 to C_N, each with private datasets D_1 to D_N, and an aggregation server S. A core assumption is the availability of TEEs on both clients and the server, isolating code and memory and ensuring data confidentiality. We assume TEEs are trusted against cryptographic attacks and key extraction, while acknowledging that side-channel attacks exist but are outside the scope of this work.

Our threat model considers two main adversaries:

1. Poisoning Attacker A (or AB): Aims to inject a backdoor via data and/or model poisoning.

• Objectives: O1 - Maximize Backdoor Accuracy (BA) on trigger samples. O2 - Stealthiness, ensuring poisoned updates are inconspicuous and do not reduce the aggregated model's Main Task Accuracy (MA) on benign inputs.
• Capabilities: A can fully control a subset of clients (n_a ≤ N/2) in any given round and N_A ≤ N/2 clients overall. A can manipulate local datasets, training processes, or model parameters. A is an adaptive adversary, aware of defense mechanisms and capable of adjusting its strategy.

1. Privacy Attacker AP: Aims to reconstruct information about clients' local data by analyzing local model updates. AP has arbitrary control over the aggregation server. However, benign clients and the server use remote attestation to verify secure enclaves, preventing AP from accessing local models directly.

Based on these adversaries, CrowdGuard derives three key requirements:

• R1: Prevent the backdoor attack. The aggregated model G_t+1 must behave like a benign model for trigger inputs.
• R2: Preserve benign performance. CrowdGuard must not reduce the main task accuracy of the FL model in the absence of an attack.
• R3: Preserve client privacy. The server must not be able to access individual local models for inference attacks, nor should any client access other clients' models.

These requirements lead to several challenges:

• C1: Effectively distinguishing benign and poisoned models, especially in non-IID scenarios.
• C2: Inspecting local models without enabling any party to extract knowledge from them, thus preserving privacy.
• C3: Using clients' local data for backdoor identification without requiring triggered samples, given that benign clients are unlikely to have many.

Key Findings

[▶ Watch: Limitations of current federated learning backdoor defenses (0:40)]()

CrowdGuard presents a significant advancement in federated backdoor detection, effectively addressing critical limitations of existing solutions. Its key findings and contributions are multifaceted:

1. Novel Privacy-Enhancing Architecture: CrowdGuard introduces a pioneering architecture that leverages Trusted Execution Environments (TEEs) on both client and server sides. This creates a secure, privacy-preserving feedback loop, enabling clients to inspect local models without exposing sensitive data or models to the server or other clients. This design guarantees the confidentiality and integrity of local models and client data, preventing inference attacks (R3) and removing the need for trust in the aggregation server.
2. Effective Backdoor Detection Algorithm (HLBIM): The core of CrowdGuard's detection capability lies in its novel Hidden Layer Backdoor Inspection Metric (HLBIM). HLBIM analyzes the outputs of all hidden layers of local models using both Euclidean and Cosine distances. This in-depth, layer-agnostic analysis, combined with an iterative pruning scheme based on multiple statistical significance tests, allows CrowdGuard to effectively distinguish benign from backdoored models. It demonstrates robustness in complex non-IID data scenarios and against sophisticated, adaptive attacks (C1), as adversaries cannot disguise backdoor functionality across all layers without impacting the attack. The use of relative distances in HLBIM further enhances detection by improving the separation capability of the first Principal Component by an average of 10%.
3. Robust Voting Aggregation: To counteract manipulated feedback from malicious clients, CrowdGuard developed a server-located stacked clustering scheme. This two-level approach, which combines Agglomerative Clustering with DBSCAN, robustly aggregates client votes. It effectively filters out manipulated or erroneous feedback, ensuring integrity even when adversaries attempt to mimic benign behavior or benign clients make minor classification errors. This mechanism provides resilience against Poisoned Model Rates (PMRs) of up to 49%.
4. Comprehensive Evaluation and Superior Performance: Extensive evaluation across a wide array of FL scenarios—including IID, diverse non-IID distributions (1-class, 2-class, Dirichlet, Normal), varying poisoning rates (PDR, PMR), different backdoor types (pixel, label swap, semantic, combined), and adaptive adversaries—consistently demonstrated CrowdGuard's exceptional effectiveness. It achieved an unwavering 100% True-Positive Rate (TPR) and 100% True-Negative Rate (TNR), resulting in 0% False Positives (FPR) and 0% False Negatives (FNR).
5. Preservation of Model Performance: A critical finding is that CrowdGuard successfully detects and mitigates backdoors without compromising the Main Task Accuracy (MA) of the protected federated model. This directly fulfills Requirement R2, ensuring that the defense does not degrade the utility or performance of the FL system in practical deployments.
6. Acceptable Runtime Overhead: While introducing a computational overhead, the average client-side validation time of 29.5 seconds when running within an Intel SGX TEE on a CPU is considered acceptable. This is especially true given the substantial security and privacy benefits provided and the typically much longer training durations for DNNs. The evaluation also showed that this overhead can be significantly reduced with the use of GPU accelerators.
7. Real-World Applicability: Recognizing the practical importance of its contributions, the CrowdGuard team is actively integrating its source code into the OpenFL framework, aiming to facilitate its adoption and deployment in real-world applications beyond academic research.

Technical Deep Dive

[▶ Watch: CrowdGuard's secure architecture and core solution overview (1:15)]()

CrowdGuard's core strength lies in its innovative system design and sophisticated detection algorithms, all operating within a privacy-preserving TEE-based architecture. The rationale is to leverage a secure client-feedback-loop where clients perform local validation by analyzing changes in individual neuron behavior. This allows for the identification of poisoned models through iterative pruning based on statistical significance tests, while a multi-layer clustering scheme on the server mitigates manipulated feedback.

System Architecture

CrowdGuard's architecture is structured in seven distinct steps, ensuring end-to-end security and privacy:

1. Setup Phase: Each client C_i and the server S first establish their secure enclaves (TEEs). These enclaves are responsible for client-side validation, feedback provision, and server-side aggregation. Remote attestation is performed by both clients and the server to verify the integrity and authenticity of the executed code within these TEEs, thereby ensuring no unauthorized access to local models or data (fulfilling R3).
2. Local Training: The FL process begins with the server S deploying the global model G_t to selected clients. Each client C_i trains its local model L_i on its private dataset D_i within its secure enclave. After local training, the client encrypts L_i and sends it to the server's secure enclave.
3. Model Distribution for Validation: Once the server's enclave collects all encrypted local model updates, it distributes these encrypted L_is to the client-side validation enclaves of all participating clients. This crucial step addresses C2 by allowing clients to inspect models without direct server access or privacy leakage.
4. Client-Side Validation: Inside their secure enclaves, clients C_j validate the received models L_i (including their own) using their local datasets D_j. This is where the novel Hidden Layer Backdoor Inspection Metric (HLBIM) is applied. HLBIM analyzes changes in the hidden layer outputs of the models. This in-depth analysis, combined with iterative pruning and statistical significance tests, enables CrowdGuard to effectively identify poisoned models, even in non-IID scenarios and without requiring triggered samples (addressing C1 and C3).
5. Feedback Provision: Based on the HLBIM analysis, each client C_j provides binary feedback to the server, voting on whether each model L_i is benign or suspicious.
6. Server-Side Aggregation and Filtering: The server's enclave receives these votes. To mitigate manipulated feedback from malicious clients, the server applies a stacked clustering scheme to robustly aggregate the votes. This scheme identifies and removes manipulated or noisy feedback. Based on the aggregated feedback, the server removes the models identified as poisoned.
7. Global Model Update: Finally, the server aggregates the remaining, benign models using FedAVG to create the new global model G_t+1. This G_t+1 is then distributed back to all clients for the next training round.

HLBIM and Model Pruning

CrowdGuard's core detection algorithm operates client-side within secure enclaves and involves two main steps: HLBIM extraction and probabilistic tests with iterative pruning for voting decisions.

HLBIM Motivation and Matrix Generation (Algorithm 1)

Traditional backdoor defenses often analyze model parameters or final prediction accuracy. However, adaptive adversaries can manipulate parameters to mimic benign models or inject backdoors that do not affect main task accuracy, making them stealthy. CrowdGuard's insight is that even if a backdoor doesn't visibly affect final predictions on benign data, it must alter the behavior of at least a subset of deep-layer neurons to introduce its hidden functionality. HLBIM is designed to detect these subtle, deep-seated changes. It uses both Euclidean and Cosine distances to capture changes in the magnitude and direction of neuron outputs, making it robust against various manipulation strategies.

Algorithm 1, executed by each client C_j in its secure enclave, generates two HLBIM matrices: one for Cosine distance (HLBIMC) and one for Euclidean distance (HLBIME). The process involves:

1. Deep Layer Output (DLO) Generation: For each sample s in D_j, it's fed into all local models L_i and the global model G_t. The outputs of all hidden layers are recorded for each model, creating DLO_locals (for L_i) and DLO_globals (for G_t). Analyzing all layers is crucial to prevent adversaries from hiding backdoors in unconsidered layers.
2. Distance Calculation: For each local model L_i, Cosine and Euclidean distances are computed between its DLOs and G_t's DLOs, yielding DLO_distC and DLO_distE.
3. Ratio Generation: A ratio of these distances is generated, using the validating client's local model L_j as a reference. This normalizes differences relative to a known benign baseline (the client's own model).
4. Scaling: To emphasize deviations, ratio values are scaled by subtracting one and squaring the result, while retaining the original sign. This amplifies even small differences.
5. Averaging and Flattening: The scaled DLO matrices are averaged over the sample dimension for each label, carving out label-specific effects. Then, the label dimension is flattened by concatenating the values. This separation by label is vital as abnormal behavior might be specific to certain classes. The final output is the HLBIM matrix.

Voting Decision via Model Pruning (Algorithm 2)

Algorithm 2 takes the HLBIM matrices and iteratively determines if models are poisoned:

1. Dimension Reduction: CrowdGuard first applies Principal Component Analysis (PCA) to the HLBIM matrix. PCA reduces the two-dimensional matrix (models x layers) to a single dimension, yielding pc_dim1_values. The first PC dimension consistently captures over 65% of the explained variance, effectively segregating data points.
2. Iterative Pruning Loop: The core is a while loop that continues as long as significant differences are detected:

• Significance Tests: The distributions of pc_dim1_values are compared using multiple statistical tests: Student-T-Test (for equal means), F-Test (Levene-Test) (for equal variances), and D-Test (Kolmogorov-Smirnov) (for overall distribution goodness of fit). A strict significance level of 0.01 is used to reduce False Positives.
• Outlier Detection: Even if distributions appear similar, outliers might indicate weakly hidden poisoned models. These are identified using both a boxplot and the 3-sigma rule.
• Clustering and Pruning: If significance tests or outlier detection indicate abnormalities, hierarchical agglomerative clustering with nclusters=2 is employed. This creates two clusters; the smaller cluster is identified as containing malicious_models and pruned from consideration.
• Safety Abort Criterion: The pruning process stops if more than half of the models have been pruned, preventing excessive removal of benign models.

This iterative approach allows CrowdGuard to detect multiple, distinct backdoors within a single FL round. Benign models' PC values should follow a similar distribution, while poisoned models, due to altered neuron behavior, will deviate significantly, making them detectable.

Voting Aggregation (Algorithm 3)

After client-side validation, each client C_j provides a binary vote for each model L_i. However, the server cannot blindly trust these votes, as malicious clients might manipulate them to hide their own models or falsely accuse benign ones. Even benign clients might occasionally misclassify. CrowdGuard addresses this with a robust Stacked Clustering scheme on the server side.

The goal of stacked clustering is to aggregate individual votes and identify the most representative voting vector, filtering out manipulated and noisy feedback.

1. First-Level Clustering: Majority Cluster Detection: The server applies Agglomerative Clustering with nclusters=2 to the voting_matrix (client votes x local models). Based on the majority assumption (that most clients are benign), the larger cluster is identified as the majority_cluster. This step effectively eliminates obvious malicious votes, such as those where adversaries mark all benign models as malicious.
2. Second-Level Clustering: Miss-Classification Compensation: To refine the majority_cluster and extract the most frequent, truly benign voting pattern, DBSCAN clustering is applied. DBSCAN, a density-based algorithm, identifies clusters of varying shapes and sizes and detects outliers. The cluster with the highest vote count is selected, and its voting decision is designated as the aggregated_voting, representing the final robust decision for each model. This step compensates for minor voting errors from benign clients and subtle manipulations from adaptive adversaries.

This stacked clustering approach ensures high robustness, remaining effective against Poisoned Model Rates (PMRs) of up to 49%. It significantly outperforms naïve majority voting, which is highly susceptible to even a few incorrect votes, by systematically identifying and isolating manipulated or erroneous feedback.

Demo / Proof of Concept

[▶ Watch: Background: Federated Learning process and attack types (2:00)]()

While the talk did not feature a live, interactive demonstration, CrowdGuard's effectiveness and technical mechanisms were rigorously validated through an extensive experimental setup, serving as a comprehensive proof of concept. The evaluation leveraged a Python-based implementation using the PyTorch Deep Learning library, executed within a computational environment featuring an Intel Xeon 5318S CPU with Intel SGXv2 for TEE capabilities, Nvidia RTX A6000 GPUs, and substantial memory. The Gramine library was utilized to run Python code within the SGX enclave.

The evaluation used standard benchmark datasets: CIFAR-10 (50,000 training, 10,000 test images, 10 classes) and MNIST (60,000 training, 10,000 test images, 10 classes). These datasets were split into local datasets of 2560 samples per client to simulate FL. Network architectures included a light Resnet-18 for CIFAR-10 and a custom Convolutional Neural Network (CNN) for MNIST.

CrowdGuard's capabilities were tested across a wide array of challenging scenarios, including:

• Diverse non-IID data distributions: 1-class, 2-class, Dirichlet, and Normal distributions, including a completely disjoint data scenario (q=0.0 for 1-class non-IID).
• Adaptive adversaries: Varying adaptation rates (α from 0.1 to 0.9), employing sophisticated techniques like a second loss function or splitting malicious clients.
• Poisoning parameters: Poison Data Rate (PDR from 0.1 to 0.9), Poison Model Rate (PMR from 0.05 to 0.49 for 20-100 clients).
• Backdoor types: Pixel, Label Swap, Semantic, and combined attacks.
• FL round: Detection at t=0 (randomly initialized model) and t=1 (converged model).

The talk highlighted visualizations (e.g., Figure 4 in the paper) that illustrate CrowdGuard's internal workings. For instance, Figure 4a demonstrates the inadequacy of plain Cosine distance for detection, while Figures 4b and 4c, depicting the HLBIM for one label and the full HLBIM plot, clearly show distinct peaks corresponding to malicious models. Figures 4e, 4f, and 4g visually demonstrate the iterative pruning process using PCA, showing malicious models being progressively removed, and Figure 4h illustrates an exemplary boxplot used for outlier detection during pruning. These visualizations, combined with the comprehensive quantitative results, provide strong evidence of CrowdGuard's robust and effective operation.

Defensive Implications

[▶ Watch: System threat model and adversary capabilities defined (4:00)]()

CrowdGuard's findings offer several critical implications for defenders seeking to secure Federated Learning deployments:

1. Embrace Trusted Execution Environments (TEEs): The fundamental reliance on TEEs on both client and server sides is a powerful paradigm shift. Defenders should prioritize FL frameworks and infrastructures that support TEEs (e.g., Intel SGX, AMD SEV, Nvidia Confidential Computing) to enable privacy-preserving model inspection. This is especially crucial for cross-silo FL scenarios involving sensitive data where trust in the aggregation server or other clients cannot be assumed.
2. Move Beyond Superficial Model Inspection: Traditional defenses that only analyze model parameters or the final output layer are insufficient against adaptive and stealthy backdoor attacks. Defenders must adopt advanced techniques like CrowdGuard's HLBIM that delve into the behavioral changes of neurons across all hidden layers of a Deep Neural Network. This deep-layer analysis is more resilient, as adversaries find it far more challenging to hide backdoor functionality across an entire network without detection.
3. Prioritize Robustness to Non-IID Data and Adaptive Adversaries: Real-world FL environments are inherently non-IID, and adversaries will adapt their attacks. Defenses must be specifically designed to perform effectively in highly non-IID settings and withstand sophisticated adaptive attack strategies (e.g., constrain-and-scale, modified loss functions). CrowdGuard's proven 100% TPR/TNR in such scenarios sets a benchmark for this robustness.
4. Implement Robust Feedback Aggregation: Client feedback, even from supposedly benign clients, can be noisy or maliciously manipulated. Defenders should avoid simplistic aggregation methods like naïve majority voting. Instead, robust schemes like CrowdGuard's stacked clustering (combining Agglomerative Clustering and DBSCAN) are essential to filter out erroneous or malicious votes, ensuring the integrity of the detection process even when a significant portion of clients (up to 49% PMR) are compromised.
5. Balance Security, Privacy, and Performance: While TEEs introduce computational overhead (e.g., 29.5 seconds for client-side validation on SGX CPU), this is an acceptable trade-off for the security and privacy guarantees. Defenders should factor this overhead into their FL system design and explore optimizations such as utilizing ML accelerators with TEEs (e.g., GPU TEEs), data subsampling for validation, parallelization, and language optimization to minimize performance impact.
6. Integrate and Adopt Proven Solutions: The ongoing integration of CrowdGuard into the OpenFL framework is a significant step towards practical adoption. Defenders should actively seek out and integrate such proven, robust defense mechanisms into their FL pipelines to enhance the trustworthiness and security of their AI models.
7. Consider Memory Constraints for TEEs: Defenders need to be aware of the memory overhead associated with holding multiple DNN parameters and Deep Layer Outputs (DLOs) simultaneously within TEEs. While newer TEE architectures offer larger memory, optimizing memory usage (e.g., encrypted file systems) or selecting TEEs with adequate resources will be important for large models.
8. Evaluate Communication Overhead for Client-Side Validation: In large-scale FL, distributing all local models to all other clients for validation can introduce communication overhead. Defenders should consider the trade-offs and explore strategies like using a subset of clients for validation, though this might require stronger assumptions about the benign majority.

By adopting these defensive postures, organizations can significantly strengthen their Federated Learning deployments against the growing threat of targeted poisoning attacks, fostering greater confidence in the utility and integrity of collaborative AI.

Key Takeaways

• Privacy-Preserving Architecture: CrowdGuard employs a novel TEE-based architecture on both client and server sides, creating a secure feedback loop that ensures client data and local model confidentiality and integrity during backdoor detection.
• Deep Layer Neuron Behavior Analysis: The Hidden Layer Backdoor Inspection Metric (HLBIM) analyzes outputs from all hidden layers of local models using statistical distances, effectively detecting subtle changes in neuron behavior that signal a backdoor, even in non-IID data.
• Robust Detection through Iterative Pruning: CrowdGuard uses iterative pruning driven by multiple statistical significance tests (Student-T-Test, F-Test, D-Test) and PCA-based dimension reduction to precisely identify and remove poisoned models, even against adaptive adversaries.
• Resilient Feedback Aggregation: A server-located stacked clustering scheme (Agglomerative Clustering + DBSCAN) robustly aggregates client votes, effectively filtering out manipulated or erroneous feedback from malicious clients, maintaining high detection rates up to 49% Poisoned Model Rates.
• Superior Performance with No Accuracy Degradation: CrowdGuard consistently achieves 100% True-Positive Rate and 100% True-Negative Rate (0% FPR/FNR) across diverse FL scenarios, including highly non-IID data and adaptive attacks, without negatively impacting the federated model's Main Task Accuracy.
• Practical Applicability: Despite introducing an acceptable computational overhead (e.g., 29.5s client-side validation on SGX CPU, reducible with GPUs), CrowdGuard offers a robust, practical solution that is being integrated into the OpenFL framework for real-world adoption.

About the Speaker(s)

Phillip Rieger presented CrowdGuard: Federated Backdoor Detection in Federated Learning at the NDSS Symposium. The available metadata and transcript do not provide further details regarding his professional title or institutional affiliation.