A Hard-Label Black-Box Evasion Attack against ML-based Malicious Traffic Detection Systems

A Hard-Label Black-Box Evasion Attack against ML-based Malicious Traffic Detection Systems

Speakers: Zixuan Liu

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=yiXkFLCFzXw

Overview

This talk introduces NetMasquerade, a practical hard-label black-box evasion attack against machine learning-based malicious traffic detection systems. The research addresses a critical gap in adversarial ML for network security: existing attacks either require white-box access to the target model or are impractical in constrained real-world settings. NetMasquerade achieves over 90% average attack success rate (ASR) across 72 attack scenarios targeting six advanced detection systems, while operating under strict black-box constraints where the attacker receives only binary pass/fail feedback.

The work demonstrates that ML-based network traffic detection systems, despite their growing adoption, remain fundamentally vulnerable to adversarial manipulation even when attackers have no knowledge of the model architecture, training data, or feature extractors. The system operates 69.6 times faster than existing baselines during inference, making it practical for deployment in real-time network environments.

Background

▶ Watch: Introduction and background on ML-based traffic detection vulnerabilities (0:00)

Machine learning techniques for network traffic analysis have seen rapid adoption in recent years, deployed for tasks including malicious traffic detection, traffic classification, and flow correlation. These models typically operate on either statistical features of traffic (inter-packet delays, packet sizes, throughput) or raw packet content. Statistical feature-based models represent the majority of real-world deployments.

However, ML models are known to be vulnerable to adversarial attacks. The classic example from computer vision -- adding imperceptible perturbations to an image of a panda to make a classifier predict it as a gibbon -- translates to a critical security question for network traffic analysis. The network domain introduces two unique challenges that make adversarial attacks significantly harder than in computer vision. First, real-world attackers operate under a hard-label black-box setting where they cannot access model internals and can only observe binary feedback (e.g., whether a response packet is received). Second, domain constraints prevent arbitrary perturbation: unlike image pixels that can be freely modified in continuous space without destroying visual semantics, arbitrary modifications to network traffic will break protocol constraints, invalidate malicious payloads, or cause the underlying attack to fail.

Key Findings

▶ Watch: Traffic-BERT architecture and bi-cross attention mechanism explained (4:00)

NetMasquerade achieves several significant results across its evaluation:

• Over 90% average ASR against all six advanced detection systems tested, including three traditional ML-based systems and three deep learning-based systems
• Outperforms state-of-the-art baselines by up to 21%, securing the highest ASR in 56 out of 72 evaluated attack scenarios
• 69.6x faster inference speed than existing baseline approaches, enabling real-time adversarial traffic generation
• Training converges within one hour because the pre-trained Traffic-BERT component handles offline learning
• Remains effective with a limited query budget of 1,000-2,000 probes, critical for avoiding detection by security alarm systems
• Maintains high robustness even at 15% feedback noise levels, addressing scenarios where target systems return unreliable feedback to mislead attackers
• Successfully bypasses advanced feature-based defenses like BlindSpot (BS)
• Preserves original malicious intent: bandwidth distribution for DoS traffic remains nearly identical before and after the attack

Technical Deep Dive

▶ Watch: Full NetMasquerade pipeline overview with two-stage synergy (8:00)

NetMasquerade employs a novel two-stage architecture that integrates a tailored pre-training model with a reinforcement learning agent.

Stage 1: Benign Traffic Pattern Mimicking with Traffic-BERT. The first stage involves training a custom model called Traffic-BERT to capture diverse benign traffic patterns from large public datasets. The feature extraction process accounts for the long-tail distribution observed in real-world flows, where short flows dominate traffic volume but long flows contain crucial information entropy. The system applies padding and chunking to standardize sequence inputs without losing semantics.

Feature tokenization is designed around the statistical distributions of benign traffic features. Inter-packet delays vary across multiple magnitudes, so they are hashed into balanced logarithmic intervals. Packet sizes display a distinct bimodal distribution and are directly converted into discrete tokens. Traffic-BERT processes packet sizes and delays as parallel inputs through self-attention layers that generate independent hidden states. A novel bi-cross attention mechanism explicitly fuses these two feature modalities: packet size hidden states act as queries for delay features and vice versa. The model is trained through a masked field task (analogous to BERT's masked language modeling) to reconstruct realistic traffic attributes from surrounding context.

Stage 2: Adversarial Traffic Generation via Reinforcement Learning. The second stage uses an RL framework operating in a loop: (1) a lightweight policy network selects positions to modify, deciding whether to alter an existing packet or insert a new one; (2) the frozen Traffic-BERT fills masked positions with realistic benign features; (3) the feature sequence is restored into adversarial traffic; (4) the agent calculates rewards and updates the policy network.

The reward function comprises three components: an evasion reward based on binary feedback from probe traffic sent to the target system, a dissimilarity penalty that enforces minimal perturbations at each modification step, and an effectiveness penalty that preserves original attack semantics (e.g., maintaining high flow rates for DoS attacks).

During inference, the attacker cannot obtain real-time feedback. To address this, additionally trained Q-networks estimate the value of candidate actions, enabling fully offline decisions. The agent iteratively modifies flows using estimated Q-values and terminates when a predefined threshold is exceeded.

Demo / Proof of Concept

▶ Watch: Results: over 90% ASR across all six detection systems (10:00)

The evaluation testbed covered 12 kinds of malicious traffic across four major categories: reconnaissance, license/hybrid, DoS/botnet, and encrypted web attacks. Six top-performing detection systems served as targets, spanning both traditional ML and deep learning architectures.

The results demonstrated that NetMasquerade consistently hits the highest ASR across virtually all scenarios. Under restricted conditions simulating real-world constraints -- limited query budgets to avoid triggering security alarms and unreliable feedback from target systems -- the attack maintained high effectiveness. The effectiveness penalty was validated by showing that DoS traffic bandwidth distributions remained nearly identical before and after adversarial modification, confirming that the attack preserved its original malicious functionality. The code and related papers were made publicly available.

Defensive Implications

▶ Watch: Bypassing advanced defenses and concluding alarm for the community (12:00)

The researchers explicitly frame their findings as a warning for the security community. The success of NetMasquerade across diverse detection systems and attack categories reveals a systemic weakness in current ML-based traffic analysis approaches. Key defensive recommendations include:

• Traffic-space adversarial training: Detection models must be trained with adversarial examples generated in the traffic domain, not just the feature domain, to build robustness against realistic evasion attacks
• Dynamic model retraining: Static models are insufficient; detection systems need continuous adaptation to evolving adversarial strategies
• Multi-layer detection architectures: Relying on a single ML model for traffic classification creates a single point of failure that NetMasquerade-style attacks can systematically exploit
• Protocol-aware anomaly detection: Defenders should monitor for statistical anomalies in traffic patterns that indicate adversarial manipulation, even when individual packets appear benign
• Query monitoring: Since the attack requires probe traffic for reward computation, monitoring for unusual probing patterns could serve as an early warning indicator

Key Takeaways

• ML-based malicious traffic detection systems are vulnerable to practical black-box evasion attacks that require only binary pass/fail feedback
• NetMasquerade's two-stage architecture combining pre-trained Traffic-BERT with reinforcement learning achieves over 90% ASR across 72 attack scenarios
• The attack is 69.6x faster than existing baselines and converges within one hour of training, making it practical for real-world deployment
• Even under restricted conditions (limited queries, noisy feedback), the attack maintains high effectiveness while preserving original malicious intent
• Current ML-based detection systems must urgently incorporate adversarial training and dynamic retraining to withstand these attacks
• The research underscores that the arms race between ML-based detection and adversarial evasion in network security is far from settled

About the Speaker(s)

The talk was presented by a surrogate speaker (Shanu) on behalf of the first author Zixuan Liu (Seno), who was unable to attend due to visa issues. The research team works on adversarial machine learning applied to network security, with a focus on developing practical attacks that expose weaknesses in deployed ML-based detection systems. The code and papers are publicly available via the research group's repositories.