A Duty to Forget, a Right to be Assured? Exposing Vulnerabilities in Machine Unlearning Services

A Duty to Forget, a Right to be Assured? Exposing Vulnerabilities in Machine Unlearning Services

Speakers: Hongsheng Hu

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=V76Ld_6Xhz4

Overview

This talk, presented by Hongsheng Hu at the NDSS Symposium, delves into a critical and emerging security challenge within Machine Learning as a Service (MLaaS) environments: the vulnerability of machine unlearning services. In an era governed by data privacy regulations such as GDPR and CCPA, service providers are legally obligated to remove a user's data from their machine learning models upon request. This "Right to be Forgotten" necessitates robust unlearning mechanisms. Traditionally, this meant expensive and time-consuming retraining of models from scratch, a process computationally prohibitive for modern deep neural networks with millions or billions of parameters. The advent of MLaaS, while offering benefits like privacy, accessibility, and cost-effectiveness, introduces a unique constraint: the service provider often lacks direct access to the original training dataset, rendering traditional retraining-based unlearning impractical.

The research presented by Hu identifies a significant gap: existing machine unlearning methodologies have primarily been developed and evaluated in local contexts where the developer has full access to both the model and its training data. This paradigm fails to account for the specific constraints and potential threats inherent in MLaaS, particularly from untrustworthy or malicious users. The core question addressed is whether a user can compromise the normal operations of an MLaaS server by manipulating unlearning requests. The talk introduces a novel threat termed over-unlearning, where a malicious user can manipulate their unlearned data such that the server's model unlearns more information than intended, leading to an unexpected and significant degradation in model utility.

To achieve this, the research proposes and evaluates two black-box strategies: Blending and Pushing. Crucially, these strategies operate even when the malicious user has no knowledge of the model's internal architecture or parameters, reflecting a realistic MLaaS threat model. The findings underscore a critical and underexplored vulnerability, highlighting the urgent need for more secure and robust unlearning solutions as these services become indispensable for privacy compliance in cloud-based AI deployments.

Background

▶ Watch: Introduction to machine unlearning and MLaaS challenges (0:00)

The imperative for machine unlearning is multifaceted, primarily driven by compliance with stringent privacy regulations like GDPR and CCPA, which grant users the Right to be Forgotten. Beyond regulatory mandates, unlearning is also vital for model maintenance, such as updating models or rectifying issues like training on poisoned data. The most straightforward, albeit impractical, approach is full retraining, where the model is retrained from scratch on the remaining data after removing the user's information. While offering perfect unlearning guarantees, its computational cost is prohibitive for the large-scale deep learning models prevalent today.

To circumvent the computational burden, unlearning methods are broadly categorized into exact unlearning and approximate unlearning.

Exact Unlearning methods aim to achieve a model state identical to one trained without the unlearned data. Techniques include SISA (Sharded, Isolated, Sliced, and Aggregated), which partitions the dataset into shards so only relevant shards need retraining, and designing model architectures (e.g., tree structures) that localize data contributions. However, exact unlearning typically requires the service provider to store the original training dataset, which is often not permissible or practical in MLaaS due to privacy regulations and data deletion policies.

Approximate Unlearning methods, more suitable for MLaaS, directly modify the parameters of an already trained model to approximate the state of a retrained model, often without requiring access to the full original training dataset. These are further divided:

1. Influence Function-based methods: These calculate the influence of unlearned data on model parameters and apply a Newton step to remove this influence. While efficient for convex models, they struggle with large-scale deep models due to the computational cost of inverting the Hessian matrix and often still assume access to the training dataset.
2. Gradient-based methods: These calculate the gradients of the unlearned data's contribution and update the model by adding back these gradients. A state-of-the-art method by Warnecke et al. 55 is highlighted, which leverages irrelevant samples (e.g., random noise) to overwrite the unlearned data's contribution. This approach is highly practical for MLaaS as it only requires access to the specific data to be unlearned, not the entire training set.

The research also considers fine-tuning as an empirical unlearning baseline, where the model is fine-tuned on unlearned samples with randomly relabeled incorrect labels, aiming to confuse the model's understanding of that data, though it doesn't guarantee complete information removal. The study primarily focuses on evaluating risks associated with gradient-based approximate unlearning and fine-tuning, as these are the most feasible in an MLaaS environment where full retraining is not an option for the server.

The threat model for this research is specifically tailored for MLaaS scenarios, involving three key entities:

• Model Provider: The developer who trains the initial model (θ*) on the full training dataset (D_train). They have full white-box access.
• MLaaS Server: Hosts the deployed model, provides ML services via APIs, and manages unlearning requests. Crucially, the server does not have access to the original training dataset (D_train) due to privacy concerns and regulations. It does, however, possess a test dataset (D_test) for monitoring performance and uses a pre-selected unlearning method.
• Authorized Unlearning Users: Users who contributed data (D_u) to D_train and are authorized to request its unlearning. They have black-box access to the deployed model via APIs, meaning they can query inputs and receive predictions but cannot inspect internal parameters or architecture.

The focus is on malicious unlearning, where an authorized user, potentially with malicious intent or compromised, exploits the unlearning mechanism. Instead of submitting their original D_u, they submit a perturbed version, D_u', to induce over-unlearning. The properties of such malicious unlearning include:

• Performance Degradation: The server's normal business performance suffers an unexpected and significant drop in model utility.
• Stealthiness of the Unlearned Sample: The manipulated D_u' should be difficult to distinguish from normal samples by the server.
• Stealthy Prediction: If the user submits a label with D_u', it must be consistent with the deployed model's prediction on D_u' to avoid detection.

Over-unlearning is formally defined as occurring if the utility of the model unlearned with malicious data (θ_u') is smaller than the utility of the model unlearned with original data (θ_u). This can manifest in two types:

• Over-unlearning-I: Degradation primarily affects the class of the unlearned data.
• Over-unlearning-II: Degradation extends to other classes, affecting overall model performance.

This work differentiates itself from prior research on unlearning threats like Slow-down Unlearning (36), which aims to increase computational cost and requires white-box access, and Camouflaged Poisoning (18), which requires access to the training dataset and white-box knowledge for targeted misclassification. The current research focuses on a more practical MLaaS threat model with black-box access, aiming to compromise overall model utility rather than just increasing cost or inducing specific misclassifications.

Key Findings

▶ Watch: Overview of exact vs. approximate unlearning methods (2:00)

The research offers a pioneering exploration into the vulnerabilities of machine unlearning services within real-world MLaaS environments, yielding several critical findings and contributions:

1. First Investigation of MLaaS Unlearning Threats: This study marks the first systematic investigation into the specific threats that arise when machine unlearning services are deployed in an MLaaS context. It specifically highlights over-unlearning as a significant and previously underexplored risk, demonstrating how it can severely compromise a model's utility. The unique MLaaS constraints, such as the server's lack of access to original training data and users' black-box access, are central to this threat model.

1. Novel Black-Box Over-unlearning Strategies: The research introduces two novel strategies, Blending and Pushing, enabling a malicious user to achieve over-unlearning with only black-box access to the server's model.

• Blending is a naive, cost-efficient method, demonstrating effectiveness primarily for simple classification tasks by incorporating information from disparate classes into the unlearned data.
• Pushing (comprising Pushing-I and Pushing-II) is an advanced strategy that leverages adversarial perturbations to move unlearned samples closer to or across the model's decision boundary. This method proved to be highly effective, generic across various model types and datasets, and capable of inducing substantial and controlled performance degradation.

1. Extensive Experimental Validation: Comprehensive experiments were conducted on benchmark datasets (CIFAR-10, CIFAR-100, STL-10), across different model architectures (VGG, ResNet), and against representative unlearning approaches (fine-tuning, gradient-based unlearning). These experiments consistently validated the effectiveness of the proposed strategies in inducing over-unlearning, demonstrating significant drops in test accuracy and controlled misclassification.

1. Highlighting Critical Needs: This research uncovers a critical, underexplored gap between current theoretical unlearning research and the practicalities of MLaaS deployments. It underscores the urgent need for careful considerations in balancing data unlearning efficacy, model utility, and the security of MLaaS services. The findings call for the development of more robust countermeasures and secure unlearning systems that can withstand sophisticated black-box attacks.

Technical Deep Dive

▶ Watch: MLaaS-feasible unlearning: gradient-based methods and fine-tuning (3:40)

The core of this research lies in two novel black-box strategies designed to induce over-unlearning: Blending as Naive Over-unlearning and Pushing as Advanced Over-unlearning. Both operate under the MLaaS threat model, where the malicious user has only black-box access to the model.

Blending as Naive Over-unlearning

This strategy is a lightweight, sample-wise modification designed to exploit the inherent difficulty in precisely quantifying the "information" content of a single data sample.

• Motivation: If a malicious user can manipulate their unlearned data (D_u) to contain more information than expected, the server will inadvertently unlearn more, leading to utility degradation.
• Mechanism: The malicious user incorporates additional sample information into their original data x by blending it with a sample x_b from a disparate task or class. For instance, if x belongs to class A, x_b might be from class B. The blending is achieved using an injection function:

Π(x, x_b) = λx + (1 - λ)x_b (Equation 1)

Here, λ is a hyper-parameter (between 0 and 1) controlling the blending ratio. A smaller λ means more of x_b is blended into x, making the injected information more prominent. When the server unlearns this blended data, it removes information related to both class A and the injected class B, causing unexpected performance degradation on class B, which falls under Over-unlearning-II.

• Stealthiness: To avoid detection, the submitted label y for the blended sample x' is modified to be consistent with the prediction of the deployed model on x'. This makes it difficult for the server to identify malicious intent based on label inconsistencies.
• Limitations: While cost-efficient and model-irrelevant, experimental results showed that this blending strategy is "blind" and struggles to generalize to complex datasets with many class categories or intricate patterns. It is more effective for simpler classification tasks.

Pushing as Advanced Over-unlearning

Recognizing the limitations of blending, this strategy introduces a more sophisticated approach.

• Motivation: The key observation is that ML models become more "confused" when predicting samples near their decision boundaries. These samples are highly informative, as even slight perturbations can lead to different predictions. By intentionally moving unlearned data samples closer to the model's decision boundary, the malicious user can significantly amplify the unlearning effect.
• Mechanism: The malicious user perturbs their unlearned data samples x (from D_u) to create a modified version x' that is closer to or crosses the model's decision boundary. This perturbation, denoted as δ, transforms x into x' = x + δ (Equation 2). The perturbed sample x' is required to satisfy Dis(x', θ*) ≤ ε (Equation 3), where Dis(.,.) is the distance to the decision boundary and ε is a small threshold.
• Variants: Two variants of the pushing strategy are explored:

1. Pushing-I: Moves the data sample toward the decision boundary but does not cross it. The model can still correctly predict the sample's original label. This primarily causes Over-unlearning-I, where performance degradation is focused on the original class of the unlearned data.
2. Pushing-II: Moves the data sample across the decision boundary. After manipulation, the model misclassifies the sample. This can lead to both Over-unlearning-I and Over-unlearning-II (secondary degradation on other influenced classes).

• Challenge: Analytically calculating the exact distance to a deep model's decision boundary is computationally challenging due to its non-linear and high-dimensional nature.
• Solution: The research leverages black-box adversarial perturbation techniques, specifically the Carlini and Wagner (CW) attack (12, 13), adapted for a black-box setting using Zeroth Order Optimization (ZOO) (13). The CW attack aims to find the minimum perturbation δ that causes misclassification. The loss function for CW attack is defined as:

L(x, x') = ||x - x'||_2 + c * f(x') (Equation 4)

where f(x') is a function that encourages misclassification. For black-box access, f(x') is formalized using logits:

f(x') = max(max(logZ(x')_i for i ≠ y_t) - logZ(x')_yt, -k) (Equation 6)

Here, Z(x') represents the logits (raw outputs before softmax), Z(x')_i is the logit for class i, y_t is the target class label for misclassification, and k is a margin parameter. The optimization is performed using zeroth-order stochastic coordinate descent (Equations 7 and 8), which approximates gradients using finite differences by querying the black-box model with slightly perturbed inputs.

• Selecting Pushing-I vs. Pushing-II samples: During the iterative perturbation process, if the prediction for x at iteration t-1 (y(t-1)) is still the true label (y_true), but at iteration t (y(t)) it becomes different from y_true, then x(t-1) is chosen for Pushing-I (near the boundary, still correctly classified), and x(t) is chosen for Pushing-II (crossed the boundary, misclassified).

Implementation Details and Experimental Settings

To rigorously evaluate these strategies, extensive experiments were conducted using:

• Datasets: Three widely recognized image classification datasets: CIFAR-10 (60,000 32x32 color images, 10 classes), CIFAR-100 (similar to CIFAR-10, but 100 fine-grained classes), and STL-10 (13,000 96x96 color images, 10 classes, more challenging).
• Models: Two popular deep learning architectures: VGG Model 49 (two variants: shallower with three VGG blocks, deeper with five VGG blocks) and ResNet Model 25 (incorporating one convolutional layer, three ResNet blocks, and two dense layers). All models were trained with the Adam optimizer (31) (learning rate 0.001) and early stopping based on validation loss.
• Evaluation Metric: The primary metric was test accuracy of the model on the test dataset, as MLaaS servers prioritize model utility.
• Unlearning Settings: Malicious users were assumed to modify up to 50% of the training data of a specific class. This translated to 2,000 samples for CIFAR-10 and 200 samples for CIFAR-100 and STL-10.
• Perturbation Magnitude for Pushing: To ensure stealthiness, L2-norm perturbations were bounded to 20. Quantitative evaluation using Structural Similarity Index Measure (SSIM) and Learned Perceptual Image Patch Similarity (LPIPS) confirmed high perceptual similarity (SSIM 0.97, LPIPS 0.03).
• Unlearning Benchmarks: The proposed strategies were evaluated against two common unlearning methods: Fine-tuning based Unlearning Method (57) (randomly relabeling and fine-tuning) and Gradient-based Unlearning Method (55) (overwriting contributions with irrelevant samples), the latter being the primary benchmark due to its certified removal guarantees.

Demo / Proof of Concept

▶ Watch: Defining the MLaaS threat model and involved entities (4:00)

While no live demonstration was explicitly mentioned, the extensive experimental results presented in the talk serve as a robust proof of concept for the proposed over-unlearning strategies. These results quantitatively and qualitatively demonstrate the effectiveness and implications of both the Blending and Pushing methods.

Blending Method Effectiveness

Experiments on CIFAR-10, CIFAR-100, and STL-10 using the VGG model showed the following:

• CIFAR-10: The blending method proved effective. When 400 airplane samples were blended with cat information (λ=0.5), a 1.4% accuracy degradation was observed on the additional class (cat) compared to normal unlearning. Increasing the unlearned samples to 2,000 resulted in a more significant 8.1% degradation on the cat class.
• CIFAR-100 and STL-10: For these more complex datasets, however, no significant performance degradation was observed with the blending method, indicating its limitations for intricate tasks.
• Ablation Studies: Increasing the number of unlearned samples (e.g., from 400 to 2,000 on CIFAR-10 with λ=0.3) generally led to higher accuracy degradation (0.7% to 2.8%). A higher blending ratio (λ) also resulted in greater degradation, but at the cost of reduced stealthiness. Furthermore, testing different target classes (bird, horse, ship) for blending into airplane samples on CIFAR-10 yielded only slight degradation (0.1-0.2%), demonstrating the method's lack of generality.

These results confirm that the naive blending method can induce over-unlearning in simple scenarios but is not reliable for complex, real-world MLaaS applications.

Pushing Methods Effectiveness

The Pushing-I and Pushing-II methods demonstrated significantly more robust and severe impacts across all tested datasets and models:

• Overall Accuracy Degradation (Over-unlearning-II):
• On CIFAR-10, unlearning just 10% of a class with Pushing-I or Pushing-II caused a substantial 5.6-5.8% accuracy degradation compared to normal unlearning. When 50% of a class was unlearned, the degradation became even more severe, with an 11.8-12.3% accuracy drop.
• For the more challenging STL-10 dataset, unlearning 50% of a class using Pushing-II led to a dramatic 17.4% accuracy degradation.
• Class-Specific Utility Degradation (Over-unlearning-I): The pushing methods particularly excelled at degrading performance on the unlearned class. On CIFAR-10, unlearning 10% of airplane data using Pushing-I or Pushing-II reduced the model's utility on that specific class from a pre-unlearning accuracy of 87.8% to around 29.5%. With 50% unlearned data, the utility for the airplane class plummeted to a mere 2%, rendering the model practically useless for identifying airplanes. This vividly demonstrates the severe performance degradation property of malicious unlearning.
• Stealthiness: The perturbed samples generated by the pushing methods maintained high perceptual similarity to the original samples. Visual inspection (e.g., Figure 9 for STL-10 airplane samples) revealed that human eyes could not easily detect the perturbations. Quantitatively, the SSIM values of 0.97 and LPIPS values of 0.03 (Table XIII) confirmed excellent perceptual stealthiness.
• Prediction Distribution Alteration: Malicious unlearning drastically altered the model's prediction behavior. For CIFAR-10 airplane samples, while normal unlearning still correctly predicted most airplane test samples, models subjected to Pushing-I or Pushing-II largely misclassified airplane samples into other classes like bird or ship. This effect was further exacerbated with more unlearned samples, highlighting a significant shift in the model's learned representations.
• Controlled Misclassification: A particularly severe finding was the ability of malicious users to control the target of misclassification. By pushing airplane samples towards the "cat" decision region on CIFAR-10, Pushing-I and Pushing-II significantly increased the number of airplane samples wrongly predicted as cat. For 400 unlearned samples, misclassifications to "cat" rose from 2 (normal unlearning) to 25. With 2,000 samples, this number surged to 378. Similar results were observed for CIFAR-100, demonstrating that attackers can not only degrade utility but also direct the model's confusion to specific, chosen classes.
• Ablation Studies:
• Number of Unlearned Samples: Consistent with blending, more unlearned samples led to greater accuracy degradation (e.g., 6% for 400 samples, 12% for 2,000 samples on CIFAR-10).
• Model Architecture: Both Pushing methods were effective across VGG and ResNet architectures. Interestingly, VGG models, which generally exhibit higher utility, appeared more vulnerable to over-unlearning.
• Model Depth: The methods were effective across different VGG depths, with deeper models tending to be more easily affected.
• Class Option: Pushing-I and Pushing-II proved generic, causing 2-5% degradation even when unlearning samples from different classes (bird, horse, ship) on CIFAR-10, confirming their broad applicability.

These comprehensive experimental results unequivocally demonstrate that Pushing-I and Pushing-II are highly effective, reliable, and generic strategies for achieving severe over-unlearning, including the ability to control misclassification, posing a significant and practical threat to MLaaS services.

Defensive Implications

▶ Watch: Malicious over-unlearning attack concept and properties (4:45)

The research not only exposes a critical vulnerability but also highlights the limitations of existing defense mechanisms against over-unlearning in MLaaS environments. The unique constraints of MLaaS, particularly the server's lack of access to original training data, render many traditional security approaches ineffective.

Several potential defensive strategies were considered, but each presents significant limitations:

1. Hashing Techniques: An MLaaS server could store hash values of original training samples and reject unlearning requests if the submitted data's hash does not match.

• Limitations: This approach could inadvertently expose sensitive dataset information, potentially violating privacy regulations. Furthermore, minor, legitimate data changes (e.g., compression artifacts, network transmission issues, or slight variations in user-submitted data) could lead to false rejections of valid unlearning requests, disrupting legitimate user operations.

1. Membership Inference Techniques: These techniques aim to verify if a submitted sample was part of the original training data. If a malicious, perturbed sample is identified as "not a member," it could be flagged.

• Limitations: Membership inference attacks are computationally expensive to run at scale, often have low accuracy in real-world scenarios, and are less effective on well-generalized models, which are typical in MLaaS deployments. Their unreliability makes them unsuitable as a primary defense.

1. Anomaly Detection Methodologies: The server could scrutinize submitted unlearning samples for anomalies, hoping to detect the subtle perturbations introduced by malicious users.

• Limitations: Generic anomaly detection is often ineffective because malicious submissions, especially those created by the Pushing methods, are specifically tailored to appear perceptually normal and stealthy. The server also lacks a dataset of malicious patterns to train an effective anomaly detector, as such attacks are novel.

1. Heuristic Mitigation: A more practical, albeit reactive, recommendation is for service providers to carefully monitor run-time model performance during deployment for unexpected drops in utility or shifts in prediction distributions.

• Limitations: This is a detection mechanism rather than a preventative one. By the time a significant performance drop is detected, the over-unlearning attack may have already caused substantial damage, requiring costly and time-consuming remediation.

The inadequacy of these existing defense mechanisms strongly suggests that current machine unlearning research and MLaaS security practices are not adequately prepared for sophisticated black-box attacks like over-unlearning. This underscores a critical need for future research to focus on developing more robust, proactive countermeasures. This includes incorporating advanced adversarial attack principles into the design of secure unlearning systems, perhaps by designing unlearning methods that are inherently more resilient to perturbed inputs or by developing novel verification mechanisms that can operate effectively under MLaaS constraints.

Key Takeaways

• Over-unlearning is a significant and novel threat in MLaaS: Malicious users can exploit unlearning services to severely degrade a model's utility, posing a critical risk to service providers.
• Black-box over-unlearning attacks are feasible: Despite lacking internal model knowledge or access to original training data, attackers can successfully induce over-unlearning using sophisticated strategies.
• Pushing methods are highly effective and generic: Leveraging adversarial perturbations, Pushing-I and Pushing-II can cause substantial, class-specific, and overall accuracy degradation across diverse datasets, models, and unlearning methods.
• Attackers can control misclassification: Beyond general utility loss, pushing methods enable malicious users to direct the model's misclassifications to specific target classes, escalating the severity of the threat.
• Naive blending has limited applicability: While cost-efficient, the blending strategy is only effective for simple classification tasks and struggles with complex, real-world datasets, highlighting the need for advanced attack vectors.
• Existing defenses are insufficient: Current mitigation strategies like hashing, membership inference, and anomaly detection are largely ineffective against stealthy over-unlearning attacks due to MLaaS constraints and the nature of the perturbations.

About the Speaker(s)

Hongsheng Hu is the speaker for this presentation, "A Duty to Forget, a Right to be Assured? Exposing Vulnerabilities in Machine Unlearning Services," delivered at the NDSS Symposium. While the provided metadata and transcript do not offer specific biographical details beyond his name, the depth and technical rigor of the research presented demonstrate his expertise in the critical and rapidly evolving fields of machine learning security, privacy regulations, and adversarial machine learning. His work highlights a keen understanding of the practical challenges and vulnerabilities inherent in deploying machine learning systems in cloud-based, service-oriented environments.