Group-based Robustness: A General Framework for Customized Robustness in the Real World

Group-based Robustness: A General Framework for Customized Robustness in the Real World

Speakers: Weiran Lin

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=tDZHDdsVH4I

Overview

In an era where machine learning models are increasingly integrated into critical real-world applications, their vulnerability to sophisticated evasion attacks poses a significant threat. Weiran Lin's presentation at the NDSS Symposium, "Group-based Robustness: A General Framework for Customized Robustness in the Real World," addresses a crucial blind spot in the current understanding and measurement of model robustness. The talk argues that conventional metrics—such as benign accuracy, untargeted robustness, and targeted robustness—are often inadequate for capturing the complexity of real-world adversarial threats, which frequently involve misclassifications across groups of classes rather than single, isolated instances.

This research introduces group-based robustness, a novel and statistically orthogonal metric designed to assess a model's resilience against these more nuanced, group-centric attacks. The framework not only formalizes this new threat model but also provides practical methodologies for both efficiently launching such attacks and effectively defending against them. By moving beyond the traditional paradigm of single-class misclassification, this work offers a critical advancement for evaluating and enhancing the security of machine learning systems deployed in high-stakes environments, from autonomous vehicles to financial security systems.

The core motivation stems from scenarios where an adversary's goal isn't just to cause any misclassification or to target a single specific class, but rather to induce misclassifications from a set of source classes to a set of target classes to achieve a specific adverse outcome. Examples include perturbing speed limit signs into stop or low-speed signs, or impersonating any three distinct bank staff members to open a vault. These complex attack objectives necessitate a more granular and flexible robustness metric that the proposed framework delivers.

Background

▶ Watch: Introduction: Problems with existing robustness metrics and examples (0:00)

The landscape of machine learning security has long been concerned with evasion attacks, where subtle perturbations to model inputs cause misclassifications. Pioneering works like Fast Gradient Sign Method (FGSM), Projected Gradient Descent (PGD), and Carlini and Wagner (CW) attacks have defined the methods for generating such adversarial examples. Correspondingly, model robustness has typically been quantified using a few established metrics:

• Benign Accuracy: The model's performance on clean, unperturbed data.
• Untargeted Robustness: The model's resistance to any incorrect classification, meaning the adversary succeeds if the input is misclassified as any class other than the true one.
• Targeted Robustness: The model's resistance to misclassification as a specific incorrect class, meaning the adversary aims for a particular target label.

However, as highlighted by the speaker, these conventional definitions fall short in scenarios reflecting real-world adversarial objectives. The primary limitation lies in how the "set of desired misclassifications," denoted as I, is traditionally defined. For untargeted attacks, I typically represents any source class s to any other target class t (U_s in Y U_t in Y setminus s (s,t)). For targeted attacks, I is defined as any source class s to a single, fixed target class t_s (U_s in Y (s, t_s)).

These definitions are too narrow for sophisticated threats. Consider the motivating examples:

1. Traffic Congestion/Accident: An adversary doesn't just want a speed limit sign to be misclassified as any other sign. They specifically want it to be misclassified as a stop sign, a no-entry sign, a no-vehicle sign, or a sign indicating a much lower speed limit (e.g., less than half the actual limit). This involves misclassifying a group of source signs (speed limit and delimit) into a group of dangerous target signs.
2. Bank Burglary: Burglars don't need to impersonate a single specific staff member. They need to impersonate any three distinct staff members out of a larger pool. This is a multi-input attack where success depends on a combination of impersonations, not just one.

Existing metrics fail to account for:

• Misclassifications from one set of classes to another, mutually exclusive set of classes.
• Attack goals that involve multiple input instances (e.g., the bank burglary scenario needing three distinct impersonations).

The absence of a metric capable of evaluating model susceptibility to these group-based threats represents a critical gap in current robustness assessment, which this research aims to fill.

Key Findings

▶ Watch: Critique: Why conventional metrics fail for group-based threats (2:00)

The research presents several pivotal findings that collectively redefine how machine learning model robustness should be perceived and measured in complex, real-world contexts:

1. Orthogonality of Group-based Robustness: Empirical evaluations across diverse datasets (GTSRB, PubFig, ImageNet, SST-5) robustly demonstrate that group-based robustness is a statistically orthogonal metric to traditional measures like benign accuracy, untargeted robustness, and targeted robustness. This means it provides new, distinct insights into model susceptibility that conventional metrics cannot capture. Pearson correlation coefficients between group-based robustness and the other three metrics were consistently weak or negligible (between -0.4 and 0.4), affirming its unique analytical value. Furthermore, its distribution (mean and range) often differed significantly from other metrics, emphasizing its ability to reveal vulnerabilities missed by standard evaluations.

1. Efficient Attack Generation via Novel Loss Functions: For single-input, multiple-target class scenarios, two novel loss functions, MDMUL Loss and MDMAX Loss, were introduced. These functions significantly boost attack efficiency by directly optimizing for the group-based attack objective, rather than iterating through individual target classes. Experiments showed that attacks using MDMAX or MDMUL achieved comparable or slightly lower adversarial advantage (0.62-1.04x that of best guess attacks) but were computationally cheaper by a factor as large as the number of targeted classes (e.g., 3 to 7 times faster on GTSRB, 10 to 50 times faster on PubFig/ImageNet). They also found successful perturbations up to 15 times more often than naive average guess attacks.

1. Advanced Attack Strategies for Multi-Input Scenarios: For multi-input, multiple-target scenarios (where X > 1), three new attack strategies were developed to efficiently select which (input, target class) pairs to attack. These strategies drastically reduce the number of attempts required to find a desired number of diverse misclassifications. The most efficient combined strategy (using a prior matrix with MD loss after one attack iteration) achieved a remarkable reduction, needing only 0.79-4.53% of attempts compared to a baseline brute-force approach on ImageNet with L2 norm attacks, and 1.26-9.40% with L-infinity norm attacks. This translates to up to a 99% reduction in attack attempts.

1. Effective Defense Mechanism: A novel defense method, based on modifying existing adversarial training algorithms, was proposed. This defense successfully enhances group-based robustness while simultaneously maintaining high average benign accuracy and high benign accuracy on the impersonated (target) classes. On GTSRB, the defense achieved higher average accuracy and group-based robustness compared to baselines. On PubFig, it boosted group-based robustness by up to 3.52 times while preserving benign accuracy, demonstrating that it is possible to systematically build models robust to group-based threats without compromising general utility.

These findings collectively establish group-based robustness as an essential new dimension for evaluating and securing machine learning models, offering both a theoretical framework and practical tools for dealing with complex, real-world adversarial challenges.

Technical Deep Dive

▶ Watch: Limitations of traditional untargeted and targeted attack definitions (5:30)

The core of this research lies in the formal definition of group-based robustness and the subsequent development of efficient attack and defense mechanisms tailored to this new metric.

Formalizing Group-based Robustness

The framework defines group-based robustness through an experiment, inspired by cryptographic security definitions, parameterized by five key components:

1. Classifier f: The machine learning model under attack (can be randomized or deterministic).
2. Set of Relations I = {I_i}: This is the crucial element defining the adversary's goal. Each I_i is a relation Y x Y (source class, target class). The adversary succeeds if they can implement a misclassification defined by any I_i in I. For example, I could be U_s in S U_t in T (s,t), meaning any source class s from a set S is misclassified as any target class t from a set T. This captures "any student impersonates any TA/professor" or "any speed limit sign becomes any stop sign."
3. Predicate Π: This function indicates whether the perturbed instance x' is sufficiently "close" to the original input x, typically measured by a distance metric (e.g., L_p norm).
4. Algorithm G: Generates a set of X instances (X = {x_1, ..., x_k}) by sampling from some distribution. Crucially, k can be greater than 1, allowing for multi-input attacks (e.g., the bank burglary scenario requiring multiple impersonations).
5. Adversary A: An algorithm that takes the generated instances X and produces a set R of (original, perturbed) input pairs (x, x').

The experiment returns 1 (adversary succeeds) if Π indicates a small perturbation for all pairs in R AND the achieved relation Î = {(f(x), f(x')) : (x,x') in R} matches some I_i in I. Otherwise, it returns 0.

Based on this experiment, two metrics are defined:

• Group-based Advantage: Adv_Π,f,f,G,I(A) = P(Exptimp-rel_Π,f,f,G,I(A) = 1), the probability of adversary success.
• Group-based Robustness: Rob_Π,f,f,G,I(A) = P(Exptimp-rel_Π,f,f,G,I(A) = 0), the probability of adversary failure.

This formalization provides a flexible and powerful way to describe a broad attack space that was previously unaddressed.

Efficient Attack Methods

To efficiently measure group-based robustness, the authors introduce novel loss functions and attack strategies, particularly for scenarios where X=1 (single input, multiple target classes) and X > 1 (multiple inputs, multiple target classes).

Loss Functions for X=1 Scenarios

Traditional targeted attacks require iterating through each possible target class, which becomes prohibitively expensive when the target set T is large. The proposed loss functions aim to achieve the group-based attack goal more directly:

1. MDMUL Loss (Minimal Difference Multiplied Loss): This loss function aims to elevate the logit of any class in the targeted set T above the logit of any class not in T.

• Formally: l_MDMUL = sum_t in T ln(sum_i notin T textReLU(Z_i + delta - Z_t)).
• It evaluates to negative infinity if and only if a successful attack has been found (i.e., there exists a t in T such that Z_t is higher than all Z_i for i not in T). This formulation encourages the logits of T classes to exceed all non-T classes simultaneously.

1. MDMAX Loss (Minimal Difference Maximum Loss): This approach focuses on greedily increasing the current maximum logit from among the targeted classes, relative to all non-targeted classes.

• Formally: l_MDMAX = sum_i notin T textReLU(Z_i + delta - max_t in T Z_t).
• This loss is non-negative and becomes zero if and only if a successful attack has been found (i.e., the maximum logit within T is higher than all logits outside T). It prioritizes making the "best" T class win.

These loss functions, when integrated into gradient-based attack methods like PGD, significantly improve efficiency by avoiding the need for |T| separate targeted attacks.

Attack Strategies for X > 1 Scenarios

For scenarios involving multiple inputs (X > 1), such as finding three distinct impersonations for a bank vault, the challenge is efficiently selecting which (x,t) pairs (original input x, desired target class t) to attack from a potentially vast search space. The authors propose three strategies:

1. Estimate by Computing a Prior from Validation Set: The adversary A pre-computes a prior probability Adv_Π,f,f,G,I(A) for perturbing an input from source class s to target class t using a validation set. During the actual attack, A prioritizes (s,t) pairs with the highest prior success rate, avoiding exhaustive search.
2. Estimate by MD Loss Without Perturbation: For each (x,t) pair, the adversary performs a single forward propagation to obtain logits and then computes the MD loss (MDMAX or MDMUL). Pairs with a smaller MD loss (indicating they are already closer to a successful misclassification) are prioritized.
3. Estimate by MD Loss After One Attack Iteration: This is a more refined version of the previous strategy. The adversary performs one iteration of the chosen attack (e.g., PGD) for each (x,t) pair before computing the MD loss and prioritizing. This provides a slightly more accurate estimate of success likelihood.

These strategies can also be combined, for instance, by using a product of (1-prior) * l_MD to guide prioritization, leading to even greater efficiency.

Defense Approach

The framework also enables the development of defenses that specifically enhance group-based robustness while maintaining other critical performance aspects. The defense aims for three simultaneous objectives:

1. High group-based robustness (Rob_Π,f,f,G,I(A)).
2. High average benign accuracy on all inputs.
3. High benign accuracy on unperturbed inputs associated with classes that might be impersonated (e.g., ensuring bank staff members are correctly identified when not under attack).

The proposed defense modifies existing adversarial training algorithms. Instead of solely training for untargeted robustness (always predicting the correct class), it focuses on maintaining accuracy on benign inputs while allowing misclassifications on adversarial inputs only if they do not further the attacker's objectives. For instance, if an attacker tries to impersonate a bank staff member, it's acceptable if the model misclassifies them as another attacker, but not as the specific staff member.

A new loss function for adversarial training, l_MDTRAIN, is introduced:

• Formally: l_MDTRAIN = kappa * sum_t in T textReLU(Z_t + delta - max_i notin T Z_i).
• This loss is non-negative and becomes zero if an attack has been prevented (i.e., no t in T has higher logits than all i not in T). The weighting factor kappa allows for balancing group-based robustness against benign accuracy.

Additionally, the data-fetching process during adversarial training is modified. Each training batch consists of two partitions:

1. Inputs from all classes, trained with standard cross-entropy loss to maintain general benign accuracy.
2. Inputs specifically from the source classes S (those susceptible to group-based attacks), trained with l_MDTRAIN to prevent the specific group-based misclassifications.

This dual-partition approach ensures the model learns to be robust against group-based threats without compromising its overall performance on benign data.

Demo / Proof of Concept

▶ Watch: Formal definition of the new Group-based Robustness metric (6:00)

While the talk did not feature a live demo, the efficacy of the proposed framework, including the new metric, attack methods, and defense strategy, was rigorously demonstrated through extensive experimental evaluations across diverse datasets and threat models. These experiments serve as the empirical proof of concept.

Experimental Setup

• Datasets:
• GTSRB (German Traffic Sign Recognition Benchmark): 43 traffic sign classes, used for the traffic sign scenario (speed limit/delimit signs to stop/low-speed signs).
• PubFig (Public Figures Face Database): 60 identities, used for scenarios like student/TA impersonation and bank burglary.
• ImageNet: 1000 object classes, used to simulate large-scale face recognition or object identification scenarios.
• SST-5 (Stanford Sentiment Treebank): 5 sentiment classes, used for text classification scenarios (e.g., positive to non-positive sentiment).
• Benchmarks: State-of-the-art defenses were used for comparison, including Free Adversarial Training for GTSRB (with L_infinity=8/255 and L2=0.5 norms) and Wu et al.'s method for PubFig.
• Attack Methods: Auto-PGD was used for L_p norm attacks (image data), and T-PGD for text attacks, with only the loss function modified to incorporate the proposed group-based objectives.

Evaluation Results

The experiments yielded compelling results across four key areas:

1. Metric Orthogonality (C1):

• On GTSRB, group-based robustness exhibited a distinct distribution (mean and range) compared to accuracy, untargeted, and targeted robustness (Fig. 2 in the original paper).
• Crucially, the Pearson correlation coefficients between group-based robustness and the other three metrics were consistently weak or negligible (ranging between -0.4 and 0.4) across various architectures and L_p norms (Fig. 3). This statistically verifies that group-based robustness provides new insights into model susceptibility, orthogonal to existing metrics.
• On PubFig, ImageNet, and SST-5, group-based robustness consistently reported a wide range of values, reflecting the diverse nature of group-based attack scenarios, unlike the often singular or out-of-range values of other metrics (Fig. 4, Fig. 5).

1. Efficient Attacks - Loss Functions (C2):

• Attacks employing MDMAX or MDMUL loss functions achieved comparable or slightly lower adversarial advantage (0.62-1.04x that of best guess attacks) but demonstrated substantial computational efficiency gains.
• These attacks were computationally cheaper by a factor as large as the number of targeted classes, specifically 3 to 7 times faster than best guess attacks on GTSRB, and an impressive 10 to 50 times faster on PubFig/ImageNet (Fig. 6-9, Fig. 11, Fig. 12).
• They were equally fast as average guess attacks but found successful perturbations up to 15 times more often.
• Interestingly, on SST-5, MDMAX loss sometimes yielded an unintuitively larger adversarial advantage than best guess attacks, indicating it found successful attacks earlier and was more effective at increasing the "USE score" (Fig. 10).

1. Efficient Attacks - Strategies for Multi-Input (C3):

• For multi-input scenarios (X > 1), the proposed attack strategies significantly reduced the number of attempts required to find K diverse misclassifications.
• The most efficient strategy, which combined a prior matrix with MD loss computed after one attack iteration, needed only 0.79-4.53% of attempts compared to the baseline on ImageNet with L2 norm, and 1.26-9.40% with L_infinity norm (Fig. 13-15). This represents an astounding up to 99% reduction in attack attempts compared to brute-force methods.

1. Defense Efficacy (C4):

• The proposed defense method successfully enhanced group-based robustness. On GTSRB, the defense achieved higher average accuracy, higher Rob_Π,f,f,G,I(A), and similar accuracy on targeted classes T_s compared to baseline defenses (Fig. 16).
• On PubFig, the defense boosted Rob_Π,f,f,G,I(A) by up to 3.52 times while maintaining similar average accuracy and accuracy on T classes compared to existing defenses (Fig. 17). This clearly demonstrates the defense's ability to systematically achieve the desired objectives: high group-based robustness, high average benign accuracy, and high benign accuracy on impersonated classes.

These comprehensive experimental results provide strong evidence for the practical utility and effectiveness of the group-based robustness framework.

Defensive Implications

▶ Watch: Introducing new loss functions for efficient group-based attacks (8:00)

The introduction of group-based robustness fundamentally alters the defensive landscape for machine learning models, urging security practitioners and model developers to adopt a more sophisticated and nuanced approach to evaluating and enhancing model security.

Firstly, defenders must acknowledge the inherent limitations of conventional robustness metrics. Relying solely on untargeted or single-target robustness leaves models vulnerable to the complex, multi-faceted attacks that group-based threats represent. It is imperative to integrate group-based robustness into routine security assessments, especially for models deployed in critical applications where specific group-to-group misclassifications could lead to severe consequences (e.g., autonomous driving, medical diagnostics, financial fraud detection).

Secondly, the proposed framework offers a concrete path for proactive defense. Security teams should leverage the described adversarial training modifications. Specifically, adopting the l_MDTRAIN loss function and the two-partition batch data-fetching process allows models to be explicitly trained against group-based threats. This ensures that while maintaining high benign accuracy, the model learns to specifically resist misclassifications that align with an adversary's group-based objectives, without becoming overly conservative or unusable in benign scenarios. The kappa weighting factor in l_MDTRAIN provides a tunable mechanism to balance the trade-off between group-based robustness and overall benign performance, allowing defenders to customize the defense based on their specific risk appetite.

Thirdly, defenders need to think critically about the specific group-based threats relevant to their model's context. This involves identifying potential source classes S and target classes T that, if misclassified, would lead to undesirable outcomes. For example, in an identity verification system, S might be all authorized users, and T might be all unauthorized users attempting to impersonate them. Or, in a content moderation system, S could be various forms of legitimate content, while T could be different categories of harmful content. By explicitly defining these S and T sets, defenders can tailor the group-based robustness metric and defense strategy to their unique threat model.

Finally, the research highlights that simply preventing any misclassification to target classes might inadvertently degrade benign performance. The proposed defense's strength lies in its ability to allow "acceptable" misclassifications (e.g., classifying an impersonator as "unknown" or "another attacker" rather than the specific target staff member) while strongly resisting those that would further the adversary's goal. This nuanced approach is vital for building robust systems that remain functional and accurate in benign operations while being resilient to sophisticated attacks.

Key Takeaways

• Conventional robustness metrics (benign accuracy, untargeted, targeted) are insufficient to capture the complexity of real-world, sophisticated group-based evasion attacks.
• Group-based robustness is a novel, statistically orthogonal metric that provides unique insights into a model's susceptibility to attacks involving misclassifications from a set of source classes to a set of target classes.
• Two new loss functions, MDMAX Loss and MDMUL Loss, significantly boost the efficiency of single-input, multi-target attacks by optimizing directly for the group-based objective, achieving comparable success rates with up to 50x speedup.
• For multi-input scenarios, novel attack strategies leverage prior probabilities and MD loss estimates to drastically reduce the number of attack attempts needed (up to 99% reduction), making complex attacks feasible.
• A new adversarial training defense method, utilizing the l_MDTRAIN loss and partitioned data batches, effectively enhances group-based robustness (up to 3.52 times increase) without sacrificing overall benign accuracy or accuracy on impersonated classes.
• The framework encourages a more practical and customized approach to threat modeling, enabling machine learning systems to be secured against specific, high-impact group-based adversarial objectives.

About the Speaker(s)

Weiran Lin, the speaker, presented this detailed work on "Group-based Robustness: A General Framework for Customized Robustness in the Real World." The transcript indicates that the research aims to address critical limitations in current machine learning robustness evaluation, reflecting a deep engagement with the challenges of deploying secure and reliable AI systems in real-world, high-stakes environments.