Low-Quality Training Data Only? A Robust Framework for Detecting Encrypted Malicious Network Traffic

Low-Quality Training Data Only? A Robust Framework for Detecting Encrypted Malicious Network Traffic

Speakers: Yuqi Qing

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=RpCsk5zFPbQ

Overview

The proliferation of encrypted network traffic, while vital for privacy and security, has created a significant blind spot for traditional intrusion detection systems. Malicious actors increasingly leverage encryption to obfuscate their activities, making the detection of malware-generated traffic a critical challenge. Compounding this issue is the inherent difficulty in collecting high-quality training data for machine learning models designed to identify such threats. This talk, presented by Yuqi Qing at the NDSS Symposium, introduces RAPIER, a novel and robust framework specifically engineered to overcome the twin problems of limited and noisy training data in the context of encrypted malicious network traffic detection.

RAPIER tackles a prevalent real-world dilemma: malware constantly evolves, rendering captured samples quickly outdated, and the process of labeling encrypted traffic is expensive, unreliable, and resistant to manual correction due to encryption protocols like SSL/TLS. Existing machine learning approaches, including data augmentation and robust ML models, largely falter when faced with both insufficient samples and significant label noise. These methods either amplify noise, rely on unrealistic assumptions, or incur prohibitive costs and privacy risks.

The core innovation of RAPIER lies in its ability to leverage the distinct distributional characteristics of benign and malicious traffic in the feature space. Benign traffic tends to form dense, consistent clusters, whereas diverse malware generates sparser, more manifold distributions. By exploiting these differences, RAPIER can infer true labels, correct noise, and strategically synthesize new, diverse training data, including samples mimicking evolving malware or zero-day exploits. This comprehensive approach ensures that machine learning models trained with RAPIER achieve superior performance and generalizability, even when starting with low-quality training data.

Background

▶ Watch: Introduction to the challenge: low-quality encrypted traffic data (0:00)

The challenge of detecting malicious activity within encrypted network traffic has long plagued security professionals. Traditional signature-based detection is ineffective against encrypted payloads, pushing the cybersecurity community towards machine learning-based solutions. However, the application of machine learning in this domain faces substantial hurdles, primarily concerning the quality and quantity of available training data.

One major hurdle is data augmentation. Techniques like oversampling or Generative Adversarial Networks (GANs) aim to expand limited datasets. Oversampling replicates existing samples or synthesizes new ones based on nearest neighbors, but often leads to overfitting due as it fails to introduce sufficient diversity. GANs learn the underlying data distribution to generate novel samples. The fundamental flaw, however, is their extreme vulnerability to label noise. If the initial training data contains mislabeled samples, data augmentation methods will learn and subsequently amplify these incorrect distributions, exacerbating the problem rather than solving it. This makes them unsuitable for scenarios where label noise is a significant concern.

Another avenue explored is robust machine learning models. These models are designed to be less sensitive to mislabeled training data. Some approaches incorporate robust loss functions, aiming to achieve similar misclassification probabilities for correctly and incorrectly labeled data. A common limitation here is the assumption of uniform mislabeling probability, which is rarely true in real-world scenarios. Other methods attempt to correct loss values using a label transition matrix, but estimating this matrix accurately without prior knowledge is a formidable task. More recent studies focus on identifying and filtering out incorrectly labeled samples, often by observing their higher loss values during training. Yet, this approach often fails when the number of training samples is severely limited, as the statistical properties required for reliable inference may not hold.

Furthermore, some advanced techniques propose transferring knowledge from large-scale unlabeled data. While this can improve performance under data scarcity, the collection and preprocessing of such vast datasets are expensive and often introduce significant privacy leakage risks, especially when dealing with sensitive network traffic.

In summary, existing data augmentation and robust machine learning techniques, while valuable in other contexts, do not concurrently address the dual challenges of data insufficiency and significant label noise, particularly in the dynamic environment of evolving encrypted malware traffic. This critical gap is precisely what RAPIER seeks to bridge, offering a more holistic and effective solution.

Key Findings

▶ Watch: Limitations of existing data augmentation and robust ML techniques (2:00)

RAPIER represents a significant advancement in the detection of encrypted malicious network traffic, delivering several key findings and contributions that address long-standing challenges in the field:

1. First Comprehensive Solution: RAPIER is identified as the first malware traffic detection system that simultaneously tackles the critical, dual challenges of limited training data and non-negligible label noise in encrypted network traffic. This holistic approach distinguishes it from prior work that typically addresses only one of these issues.

1. Superior Robustness and Performance: The framework consistently and significantly outperforms existing state-of-the-art methods across a wide array of challenging conditions. This includes varying noise ratios (from 20% to 45%), different training sizes (as low as 250 samples), and realistic noise settings (e.g., mislabeling based on unseen domain names or absent threat intelligence). RAPIER achieved average F1 score improvements of 352.6%, 284.3%, and 214.9% on public datasets (DoHBrw, IDS, and combined IDS/DoHBrw, respectively) when facing 45% noise, demonstrating remarkable stability with F1 score fluctuations often less than 0.05.

1. Unsupervised Feature Extraction Immunity: A novel Auto-Encoder (AE) architecture is employed for feature extraction. This unsupervised approach learns fine-grained, representative features from raw encrypted traffic (specifically, sequences of packet lengths) without relying on labels. This design makes the feature learning process inherently immune to the contaminating effects of label noise present in the training data, a crucial factor for robust performance.

1. Distribution-Aware Label Correction: RAPIER introduces a unique label noise correction module that leverages MADE (Masked Autoencoder for Distribution Estimation). By accurately estimating data distributions, particularly the denser distribution of benign traffic and the sparser, manifold nature of malicious traffic, the module can effectively infer and correct true labels. This process consistently reduces original noise ratios to significantly lower levels (e.g., to 8.54% on DoHBrw and 15.81% on IDS, even from an initial 45% noise), correcting up to 76.8% of noise.

1. Adaptive Data Augmentation for Evolving Threats: The framework incorporates an improved GAN-based data augmentation module. Unlike traditional GANs or oversampling techniques, RAPIER strategically synthesizes diverse and relevant training data by predicting and targeting specific distribution regions. These regions include "Malicious Boundary" (M_B) for sophisticated attackers mimicking normal behavior, "Malicious Outlier" (M_O) for emerging zero-day exploits, and "Normal Boundary" (N_B) for maintaining decision boundaries. This adaptive generation significantly improves both recall (identifying more unseen malicious samples) and precision (reducing false alarms), leading to average F1 score improvements of 5.1% to 34.9% over other augmentation methods.

1. Real-World Validation: The effectiveness of RAPIER was robustly demonstrated in real-world cyberspace through a collaboration with a network security enterprise. Tested on a large dataset of over 2.9 million benign and 790,000 malicious encrypted traffic flows collected from 2017 to 2021, RAPIER achieved an F1 score of 0.773 (with 500 training samples and 45% noise), representing an average improvement of 272.5% over existing methods. Its label correction maintained a remaining noise ratio of less than 4.3%, and augmentation significantly boosted recall by 238.8%.

These findings collectively establish RAPIER as a groundbreaking solution for a critical and challenging problem in cybersecurity, providing a robust and practical framework for detecting encrypted malicious traffic in real-world operational environments.

Technical Deep Dive

▶ Watch: Formal problem statement and RAPIER's core observation (4:00)

RAPIER's robust performance stems from its meticulously designed, three-component architecture that addresses the intertwined challenges of data scarcity and label noise. The system aims to detect malware infections within an internal network by monitoring outgoing encrypted traffic at the intranet gateway. The formal problem statement identifies the training set, $D_{train} = (x_i, Y_i)_{i=1}^N$, as low-quality due to its limited size ($N$ is much smaller than the test set $D_{test}$) and non-negligible label noises ($Y_i$ are inconsistent with true labels $y_i$). RAPIER specifically avoids assumptions of access to large-scale unlabeled data or extreme concept drift where malicious traffic perfectly mimics benign traffic.

The framework is built upon a fundamental observation: benign traffic typically exhibits a denser, more consistent distribution in the feature space, while malicious traffic, due to the diversity and evolving nature of malware, tends to be sparser and more manifold. This distributional difference is the cornerstone for inferring true labels and strategically augmenting data.

1. Feature Extraction Module

The first critical component is the Feature Extraction Module, designed to convert raw encrypted network traffic into fine-grained, representative feature vectors in an unsupervised manner. This ensures that the feature learning process is not contaminated by the label noise present in the training data.

Instead of relying on specific TLS handshake metadata, which can be limited or easily manipulated, RAPIER uses the sequence of packet lengths for each network flow. A flow is defined by the standard five-tuple (source/destination IP, source/destination ports, transport protocol). Packet length sequences are more general and effective at capturing subtle behavioral nuances.

The module employs an Auto-Encoder (AE) architecture comprising:

• Embedding Layer: A learnable matrix that converts an input packet length sequence ($l = l_1, l_2, ..., l_n$) into an embedding sequence ($v = v_1, v_2, ..., v_n$), where each $v_i$ is a $V$-dimensional vector representing $l_i$. For RAPIER, the first 50 head packets of each flow are used ($n=50$), with an embedding dimension of $V=32$.
• Encoder: Stacks multiple bi-directional Gated Recurrent Units (bi-GRUs). GRUs are well-suited for sequential data, and the bi-directional nature allows capturing context from both past and future elements in the packet length sequence. The hidden states from the final time steps of all $B$ bi-GRU layers are concatenated to form the final feature vector $f_{encoder}$. RAPIER uses $B=2$ bi-GRU layers, each with a hidden size $H=8$, resulting in a final feature vector dimension of $d=2BH = 32$.
• Decoder: Mirrors the encoder, using stacked bi-GRUs to reconstruct the original embedding sequence from $f_{encoder}$.
• Reconstruction Layer: A multi-layer perceptron (MLP) that converts the decoder's output back into the estimated packet length sequence $\hat{l}$.

During training, the AE minimizes the reconstruction loss between the original packet length sequence $l$ and its reconstructed version $\hat{l}$ using stochastic gradient descent. This unsupervised training is crucial as it allows the AE to learn meaningful representations without being influenced by potentially incorrect labels, unlike supervised methods that use classification layers and cross-entropy loss.

2. Label Noise Correction Module

The second core component is the Label Noise Correction Module, tasked with inferring and correcting true labels within the low-quality training set. This module explicitly leverages the observed density differences: normal data is denser, while malicious data is sparser.

To estimate the probability distribution $p(x)$ of high-dimensional feature vectors, RAPIER employs MADE (Masked Autoencoder for Distribution Estimation). MADE is an efficient auto-regressive generative model that learns $p(x)$ by modeling conditional probability densities $p(x_i | x_1, ..., x_{i-1})$ for each feature $x_i$, effectively capturing latent correlations. Crucially, MADE is trained using only $W_{train}$, the subset of samples initially labeled as normal from the noisy $D_{train}$. This strategy biases MADE to assign high-density values primarily to truly benign traffic, enhancing the distinction.

The label correction process proceeds in five steps:

1. Density Estimation: The trained MADE model calculates $p(x)$ for all samples in $D_{train}$.
2. High-Confidence Normal Selection: A subset $H_{train}$ is selected from $D_{train}$ containing samples with the highest probability density values. The size of $H_{train}$ is controlled by a parameter $\alpha$ (e.g., $\alpha=0.5$, meaning 50% of $D_{train}$).
3. Core Normal Identification ($N_s$): From $H_{train}$, samples that are most similar to each other (i.e., located in the densest regions) are identified by measuring the average Euclidean distance of each sample to others within $H_{train}$. The half with smaller distances form $N_s$, which are then relabeled as normal with high confidence.
4. Core Malicious Identification ($M_s$): From the remaining samples ($D_{train} - N_s$), samples that are least similar to $N_s$ (i.e., located in sparse regions) are identified by measuring their average Euclidean distance to all samples in $N_s$. Samples with larger distances form $M_s$, relabeled as malicious. To prevent data imbalance, the size of $M_s$ is set equal to $N_s$.
5. Ensemble Learning for Remaining Samples: For the remaining samples ($D_{train} - N_s - M_s$), their true labels are inferred using ensemble learning. An ensemble of seven classical machine learning classifiers (Linear Discriminant Analysis, AdaBoost, Random Forest, Logistic Regression, Gaussian Naive Bayes, SVC, and XGBoost) is trained on the high-confidence $N_s$ (normal) and $M_s$ (malicious) sets. Ensemble learning enhances accuracy and robustness to any residual noise.

This process yields a label-corrected training set $D'_{train}$, comprising $D'_{normal}$ and $D'_{malicious}$ samples, ready for data augmentation and final model training.

3. Data Augmentation Module

The third critical component is the Data Augmentation Module, designed to synthesize new, diverse training data to improve detection performance on unseen, evolving malware. This module goes beyond simple replication or mimicking existing distributions by predicting and sampling from possible distribution regions for future malicious data.

RAPIER defines three distinct data augmentation strategies, visualized conceptually in Figure 4 (as mentioned in transcript):

• M_B (Malicious Boundary): Targets regions where new malicious data might appear just outside the normal data but very close to it, mimicking sophisticated attackers.
• M_O (Malicious Outlier): Addresses emerging attack methods, like zero-day exploits, where new malicious data might reside in regions entirely outside all existing normal or malicious data.
• N_B (Normal Boundary): Synthesizes new normal data within the boundary region of the original normal data. This strategy helps prevent generated malicious data from exacerbating data imbalance and aids in maintaining the decision boundary of ML models, reducing false alarms.

These target regions are formally defined using the probability density functions $P_N(x)$ (for normal data) and $P_M(x)$ (for malicious data) estimated by MADE, along with predefined thresholds $\gamma$, $\omega_1$, $\omega_2$, $\omega_3$. Specifically, $\gamma = 0.05$ (5th percentile of $P_M(x)$ values), and $\omega_1=0.1, \omega_2=0.2, \omega_3=0.3$ (10th, 20th, 30th percentiles of $P_N(x)$ values).

To synthesize data conforming to these distributions, an improved GAN model (conceptually Figure 5) is developed. Unlike vanilla GANs, it uses customized loss functions:

• It features three generator networks ($G_{MB}, G_{MO}, G_{NB}$), each dedicated to one augmentation strategy, and a single discriminator network.
• The generator loss function $L_G$ includes a Kullback-Leibler (KL) divergence term, which minimizes the difference between the distribution of generated data $X_G$ and its specific target distribution $P(x)$. This term incorporates the entropy of $X_G$ and log-probabilities of $P_M(x)$ and $P_N(x)$ based on the region definition. For $M_O$, a minimum $P_N(x)$ threshold of 0 allows generation in entirely novel areas.
• A regularization term $L_f$ is also included, minimizing the difference between the generated data's first discriminator layer output and that of original training samples in the corresponding target region. This prevents synthesized data from deviating too far from expected protocol specifications.
• The discriminator $D(x)$ is trained with a standard adversarial loss $L_D$ to distinguish between original and synthesized data and classify them as malicious or normal.

To further enhance diversity and prevent model collapse, multiple independent GAN models ($\eta=5$ models) are trained, and all their synthesized data are combined. The final high-quality training dataset is formed by merging the label-corrected original data $D'_{train}$ with all this synthetic data. This combined dataset is then used to train the final ML-based detector, which leverages Co-teaching and a Multilayer Perceptron (MLP). Co-teaching is specifically chosen for its inherent robustness to any small amount of residual label noise that might persist in the training set.

Demo / Proof of Concept

▶ Watch: Overview of RAPIER's three-component architecture (5:20)

While the talk does not detail a live demonstration, the "Evaluation and Results" sections provide a comprehensive proof of concept for RAPIER's capabilities and effectiveness through extensive empirical validation.

The RAPIER system and all baseline methods were implemented using Python 3.8.5, leveraging scientific computing libraries such as NumPy, PyTorch, TensorFlow, and CUDA 11.7. Experiments were conducted on a Linux server equipped with an Intel Xeon E5-2650 v4 CPU and an NVIDIA GeForce RTX 2080Ti GPU.

Key parameter settings included:

• Feature Extraction: $n=50$ (first 50 head packets), $V=32$ (embedding dimension), $H=8$ (GRU hidden size), $B=2$ (bi-GRU layers), resulting in a $d=32$-dimensional feature vector.
• Label Noise Correction: $\alpha=0.5$ (50% filtering proportion for high-confidence normal samples).
• Data Augmentation: $\gamma=0.05$ (5th percentile of $P_M(x)$), $\omega_1=0.1, \omega_2=0.2, \omega_3=0.3$ (10th, 20th, 30th percentiles of $P_N(x)$), and $\eta=5$ (number of independent GAN models). These values were determined via grid search.

Evaluation metrics included standard Precision, Recall, and F1 Score, with malicious flows considered positive samples. Experiments were performed on three diverse datasets:

1. CIRA-CIC-DoHBrw-2020 (DoHBrw): Focused on normal and malicious DNS-over-HTTPS traffic.
2. CSE-CIC-IDS2018 (IDS): A widely used intrusion detection dataset, augmented with 426 malware samples from CIC-InvesAndMal2019 to enhance encrypted malicious traffic.
3. IDS/DoHBrw (Combined): An extreme scenario simulating malware evolution, where all malicious samples from the newer DoHBrw dataset were used for testing, while training data came from the older IDS dataset (2018-2019).

Datasets were split into T1 (first day for training) and T2 (rest for testing). Training sets were deliberately small (250, 500, or 1000 samples, balanced between malicious and normal). Label noise was introduced via symmetric flipping at various ratios, from 20% to 45%. Testing sets maintained a 1:10 malicious-to-normal ratio to reflect real-world attack rarity, with further tests at 1:20, 1:30, and 1:50 ratios. Realistic noise settings included mislabeling normal traffic with unseen domain names as malicious, and malicious traffic of a chosen malware type as normal.

Key results demonstrating the proof of concept:

• Overall Detection Performance: RAPIER consistently and significantly outperformed all baseline methods (ETA, FS-Net, SMOTE, Co-teaching, DT, ODDS, and combinations). With a 45% noise ratio and 500 training samples, RAPIER achieved F1 scores of 0.770 (DoHBrw), 0.776 (IDS), and 0.855 (IDS/DoHBrw), representing average improvements of 352.6%, 284.3%, and 214.9% respectively. Its performance remained remarkably stable, with F1 score fluctuations less than 0.05.
• Impact of Training Size: Even with a very small training size of 250 samples (and 30% noise), RAPIER achieved F1 scores of 0.705 (DoHBrw), 0.751 (IDS), and 0.817 (IDS/DoHBrw), showing average improvements of 148.5%, 113.0%, and 101.1% over baselines.
• Realistic Label Noise: Under realistic noise scenarios (unseen domain names, absent threat intelligence), RAPIER achieved average F1 scores of 0.797 (DoHBrw), 0.800 (IDS), and 0.867 (IDS/DoHBrw), representing average improvements of 166.5%, 154.6%, and 165.2% over baselines, with F1 score fluctuations less than 0.03.
• Component Effectiveness:
• The AE-based Feature Extraction was shown to be superior to CICFlowMeter (0 recall) and ETA (struggles with high-dimensional output for MADE), extracting more fine-grained 32-dimensional features.
• The Label Noise Correction Module reduced original noise ratios to 8.54% (DoHBrw) and 15.81% (IDS), correcting up to 76.8% of noises at 45% original noise, outperforming SOTA methods like DT and Co-teaching significantly.
• The Data Augmentation Module achieved the highest F1 scores (0.93 on DoHBrw, 0.77 on IDS, 0.70 on IDS/DoHBrw), improving both precision and recall, especially recall on IDS/DoHBrw from 0.75 to 0.94.
• Real-World Experiments: In collaboration with a network security enterprise, RAPIER was tested on a massive dataset (2.9M benign, 790K malicious flows from 2017-2021). Using 2017 data for training and the rest for testing, RAPIER achieved an F1 score of 0.773 (training size 500, 45% noise), an average improvement of 272.5% over existing methods. The label correction module maintained a low remaining noise ratio (less than 4.3%), and augmentation significantly improved recall (by 238.8%).

These extensive results serve as compelling proof of concept, unequivocally demonstrating RAPIER's efficacy, robustness, and practicality in real-world scenarios for detecting encrypted malicious network traffic despite severe data quality issues.

Defensive Implications

▶ Watch: Deep dive into RAPIER's Feature Extraction Module (6:00)

RAPIER offers significant implications for network defenders and security enterprises grappling with the evolving landscape of encrypted cyber threats. Its core value lies in providing a robust and practical framework where traditional methods fail, particularly in environments characterized by limited and noisy threat intelligence.

1. Enhanced Detection of Encrypted Malware: RAPIER directly addresses the critical challenge of detecting encrypted malicious traffic, which often bypasses signature-based systems. By focusing on fine-grained behavioral features derived from packet length sequences rather than easily obfuscated TLS metadata, it provides a deeper, more resilient detection capability. This means security teams can gain visibility into a previously opaque area of their network traffic.

1. Resilience to Low-Quality Data: For network administrators, the reality of collecting high-quality, up-to-date malware samples and accurately labeling encrypted traffic is a persistent headache. RAPIER's ability to perform effectively with limited training data (as few as 250 samples) and high label noise (up to 45% or even 67.4% correction at 45% noise) is a game-changer. It lowers the barrier to entry for deploying ML-based detection, making it feasible even for organizations with constrained resources for threat intelligence collection and expert labeling.

1. Proactive Defense Against Evolving Threats: The adaptive data augmentation module, with its strategies for Malicious Boundary (M_B) and Malicious Outlier (M_O) generation, allows RAPIER to synthesize data representing future and unknown malware behaviors, including potential zero-day exploits or sophisticated evasions. This capability enables a more proactive defense posture, helping organizations prepare for threats that have not yet been observed in their training data.

1. Reduced False Positives and Negatives: By leveraging a distribution-aware label correction module and an improved GAN for balanced data augmentation (including Normal Boundary, N_B), RAPIER significantly improves both precision and recall. This translates directly to fewer false alarms (reducing operational burden) and a higher likelihood of catching actual threats (improving security posture), which is crucial in highly imbalanced real-world network environments.

1. Deployment at Intranet Gateways: The framework is designed for deployment at intranet gateways, enabling real-time monitoring of outgoing traffic. This strategic placement allows for early detection of infected internal hosts attempting to communicate with command-and-control servers or exfiltrate data, thus containing breaches before they escalate.

1. Guidance for Data Management: While RAPIER reduces the burden of data quality, its long-term effectiveness still benefits from strategic data management. Defenders should understand that while the system is robust to noise, extreme levels (>50% mislabeling) will still degrade performance. Future work on periodic re-collection and re-training suggests a need for ongoing, albeit less intensive, data refreshing strategies. Clustering detected malicious samples and pre-processing fresh normal data for future training can further enhance its longevity.

In essence, RAPIER empowers defenders with a powerful, resilient tool to combat encrypted malware, providing actionable intelligence and significantly bolstering the security posture of internal networks against increasingly sophisticated and stealthy cyber threats.

Key Takeaways

• RAPIER is the first comprehensive framework addressing the dual, critical challenges of limited and noisy training data for detecting encrypted malicious network traffic.
• It employs an unsupervised Auto-Encoder (AE) architecture to extract fine-grained, representative features from packet length sequences, making the feature learning process immune to label noise.
• The Label Noise Correction Module uses MADE to accurately estimate data distributions, effectively identifying and correcting mislabeled samples by leveraging the inherent density differences between benign (dense) and malicious (sparse) traffic, significantly reducing noise ratios.
• An improved GAN-based Data Augmentation Module strategically synthesizes diverse training data in target regions (Malicious Boundary, Malicious Outlier, Normal Boundary) to account for evolving malware and maintain model generalizability, improving both precision and recall.
• RAPIER consistently and significantly outperforms existing state-of-the-art methods, demonstrating superior robustness across varying noise ratios (up to 45%), small training sizes (as low as 250 samples), and realistic noise settings, with average F1 score improvements often exceeding 100-300%.
• The framework's effectiveness has been validated in real-world enterprise deployments, successfully detecting encrypted malicious traffic on large datasets despite low-quality training data, proving its practical applicability and impact.

About the Speaker(s)

The talk "Low-Quality Training Data Only? A Robust Framework for Detecting Encrypted Malicious Network Traffic" was presented by Yuqi Qing. Based on the provided metadata and transcript, no specific title or company affiliation for Yuqi Qing is mentioned.