Parrot-Trained Adversarial Examples: Pushing the Practicality of Black-Box Audio Attacks against Speaker Recognition Models

Parrot-Trained Adversarial Examples: Pushing the Practicality of Black-Box Audio Attacks against Speaker Recognition Models

Speakers: Rui Duan

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=nTg2EShMgw8

Overview

Speaker recognition systems, ubiquitous in smart devices and security applications, face a significant and evolving threat from audio adversarial examples (AEs). While advancements in black-box attacks have been made, their real-world practicality has been hampered by the substantial information they demand from target models, often requiring extensive probing or knowledge of similarity scores. This research, presented by Rui Duan at the NDSS Symposium, tackles this critical challenge by proposing a novel mechanism designed to minimize the attacker's required knowledge, thereby significantly enhancing the practicality of black-box audio attacks.

The core innovation lies in parrot training, a method that leverages recent advancements in one-shot voice conversion to synthesize numerous speech samples mimicking a target speaker's voice from just a single, short audio input. These synthetically generated "parrot speech" samples are then used to train a local parrot-trained (PT) surrogate model for the attacker. Operating under an exceptionally stringent attacker assumption—possessing only a few seconds of the target speaker's voice, with no knowledge of the target model's architecture, parameters, training data, or even classification results—the research demonstrates how PT-AEs can achieve high transferability to black-box target models while maintaining human perceptual quality.

The findings are compelling, showcasing impressive attack success rates of 45.8% to 80.8% against open-source models in digital-line scenarios. More critically, the PT-AEs achieved 47.9% to 58.3% success against real-world smart devices like Apple HomePod (Siri), Amazon Echo, and Google Home in over-the-air scenarios. This work represents a significant leap in the feasibility of black-box audio attacks, outperforming existing strategies like Smack and QFA2SR in both attack efficacy and human perception scores, underscoring a pressing need for more robust defenses against such sophisticated threats.

Background

▶ Watch: Addressing practical black-box audio attacks against speaker recognition. (0:00)

Speaker recognition systems are designed to identify individuals based on their unique vocal characteristics. This process typically involves three phases: an initial training phase where the model learns general voice features, an enrollment phase where specific speaker profiles are created, and a recognition phase where an unknown voice is matched against enrolled profiles. Tasks can range from close-set identification (CSI), where the speaker is known to be one of a predefined set, to open-set identification (OSI), where the speaker might not be in the enrolled set, and speaker verification (SV), which is a binary decision to accept or reject a speaker's claimed identity.

Adversarial speech attacks aim to subvert these systems by introducing a small, often imperceptible perturbation ($\delta$) to an original speech signal ($x$). The goal is to produce an adversarial example ($x+\delta$) that fools the speaker recognition function ($f$) into outputting a desired target label ($y_t$), while ensuring the perturbation's distortion $D(x, x+\delta)$ remains below a small perceptual threshold ($\epsilon$).

Historically, adversarial attacks are categorized into two main types:

• White-box attacks assume full knowledge of the target model's internal workings, including its architecture, parameters, and training data. This allows for gradient-based optimization to directly compute the optimal perturbation $\delta$. While effective, this assumption is rarely practical for real-world, proprietary systems.
• Black-box attacks, in contrast, operate without internal model knowledge. Traditional black-box methods often rely on a query or probing process. This involves sending numerous speech signals to the target model and observing its responses, such as soft confidence scores or hard accept/reject labels. This approach can be prohibitively time-consuming and resource-intensive, frequently demanding thousands or even tens of thousands of queries, making them impractical or even impossible against physical devices like smart speakers due to rate limiting or detection mechanisms.

The central motivation for this research stems from the limitations of query-based black-box attacks. The goal was to develop practical black-box attacks without any probing. This work leverages recent advancements in one-shot voice conversion (VC), which can generate speech samples mimicking a target speaker's voice from just a single, short audio input. This capability allows an attacker to construct a local surrogate model, termed a parrot-trained (PT) model, without direct interaction with the black-box target.

This innovative approach led to three core research questions guiding the study:

1. Can a PT model effectively approximate a ground-truth (GT) model (a model trained on actual speaker data)?
2. Are PT-AEs, generated from a PT model, as transferable to a black-box GT model as AEs generated from a GT model itself?
3. How can the generation of PT-AEs be optimized for effective black-box attacks, balancing transferability and human perception?

The defined threat model is highly restrictive: an attacker aims to fool a speaker recognition model, possessing no knowledge of the target model's internals, performing no probing, and having access to only a very short speech sample of the target speaker. The attack is specifically designed to be launched over-the-air against smart devices.

Key Findings

▶ Watch: Speaker recognition basics and limitations of current black-box attacks. (2:00)

The research established several critical findings across the feasibility of parrot training and the optimization of PT-AE generation:

1. Feasibility of Parrot Training:

The initial phase focused on demonstrating that parrot speech can be effectively generated using one-shot voice conversion (VC) techniques to train surrogate models. This process is crucial for the black-box attack strategy, as it eliminates the need for direct interaction with the target model.

• Two-step Parrot Speech Generation:

1. Initial Source Speaker Selection: VC performance is often better for intra-gender conversions. Thus, a source speaker is selected from a public dataset (e.g., VCTK) whose voice has the minimum average pitch distance to the target speaker's voice.
2. Iterative Conversions: An existing one-shot VC method is applied. The output speech, which has now acquired more target speaker characteristics, is then fed back as the new source speaker input for subsequent iterations. This iterative process amplifies the target speaker's audio features in the generated parrot speech.

• Experimental Validation: Five state-of-the-art one-shot VC models (AutoVC, BNE, VQMIVC, FreeVC-s, and AGAIN-VC), trained on the VCTK dataset, were used. Target speakers were six individuals from the LibriSpeech dataset, distinct from VCTK. The attacker's knowledge was limited to a single 4-second speech sample (approximately 10 English words) of the target speaker. The effectiveness of the parrot speech samples was evaluated against a ground-truth TDNN model trained for a close-set identification task, which achieved 99.3% accuracy. The primary metric was the False Positive Rate (FPR), indicating misclassification as the target speaker.
• Key Results on Feasibility:
• Impact of Initial Source Speaker Selection: The pitch distance of the source speaker significantly affected FPR. For instance, FreeVC-s achieved an FPR of 0.7222 when the source speaker had the minimum pitch distance.
• Impact of Iterative Conversions: FPR generally increased and then stabilized with more iterations. FreeVC-s reached its highest FPR of 0.9305 after 5 iterations, which was adopted as the standard for subsequent experiments.
• Impact of Target Knowledge Level (Speech Length): The length of the target speaker's speech sample substantially influenced FPR. For example, AutoVC's FPR increased from 0.0972 with a 2-second sample to 0.5556 with a 12-second sample. FreeVC-s consistently demonstrated the best performance across all knowledge levels.
• Conclusion on Feasibility: These results confirmed that even with very limited target speaker speech, parrot speech samples can be efficiently generated to mimic the speaker's voice features and effectively fool a speaker classifier. Furthermore, experiments showed that a PT model can approximate a GT model with comparable classification performance.

2. PT-AE Generation, Transferability & Perception:

The next phase focused on optimizing PT-AEs for high transferability to black-box GT models and excellent human perceptual quality.

• Audio Perturbation Carriers: Three main types were investigated:

1. Noise carriers: Traditional methods adding Gaussian noise in the unrestricted Lp space, often resulting in perceptibly noisy sounds.
2. Feature-twisted carriers: Manipulating auditory features like pitch or rhythm of the original speech, aiming for stealthy attacks.
3. Environmental sound carriers: Using sounds like traffic or nature to create the perturbation signal.

• Perceptual Quality Quantification: A human study with 30 volunteers rated the similarity between original and perturbed clips (1-to-7 scale). Traditional metrics like SNR or Lp norms were found to be insufficient. The study revealed:
• Perception quality for noise carriers improved with increasing Signal-to-Carrier Ratio (SCR).
• Feature-twisted and environmental sound carriers' perception quality was not strongly correlated with SCR.
• Environmental sound carriers generally yielded better human perception scores than both feature-twisted and noise carriers.
• Speech-Regression Score (SRS): A new metric, SRS, trained using the generated speech samples, was found to provide the best accuracy in evaluating perceptual quality across almost all carrier types, becoming the chosen metric.
• Transferability Evaluation: A diverse set of target models (4 DNN-based: 2 CNN, 2 TDNN) and surrogate models were used. For each target speaker, 24 PT-surrogate models and 24 benchmark GT-surrogate models were created. 20 PT-AEs and 20 GT-AEs were generated per model type using the three carrier types.
• Key Results on Transferability:
• The environmental sound carrier achieved the best AE transferability, with average match rates of 0.23 to 0.27 across target models, significantly outperforming noise (0.10-0.14) and feature-twisted (0.15-0.22) carriers.
• PT-AEs were slightly less transferable than GT-AEs but remained highly effective.
• Transferability-Perception Ratio (TPR): A new joint metric, TPR(C) = m(C) / (8 - SRS(C)), was defined, where m(C) is the match rate and SRS(C) is the perception score. A higher TPR indicates better AE quality considering both transferability and perception.
• TPR Findings: TPRs for all AE carriers increased with more knowledge (speech length). Crucially, the environmental sound carrier consistently exhibited the highest TPR at every knowledge level, confirming its superiority for PT-AE generation.

Technical Deep Dive

▶ Watch: Three core research questions and the no-probing threat model. (3:10)

The core of this research is the two-stage optimized black-box PT-AE attack strategy, meticulously designed to balance attack effectiveness (transferability) with human perceptual quality (stealthiness). Building on the insights that environmental sound carriers generally offer the best Transferability-Perception Ratio (TPR), the researchers further refined the carrier selection and AE generation process.

First, an investigation into combining carriers was conducted. While environmental sounds showed high TPRs, the study explored if combining them with feature-twisting or noise-adding could yield even better results. Two additional carrier types were considered: feature-twisted environmental sounds (manipulating pitch or rhythm of environmental sounds) and noise-based environmental sounds (adding noise to environmental sounds). The findings revealed that the TPR is sensitive to the specific environmental sound chosen; for example, brook sounds achieved a TPR of 0.29, higher than alarm (0.25) or rooster (0.26). More importantly, twisting the pitch of environmental sounds consistently outperformed twisting rhythm or adding noise, with adding noise being the least effective method. Based on these results, the optimal strategy for PT-AE generation was determined to be pitch-twisting the features of environmental sounds.

The two-stage black-box attack formulation proceeds as follows:

Stage 1: Candidate Environmental Sound Selection

The attacker first constructs a parrot-trained (PT) surrogate model. This model is trained using the synthetically generated "parrot speech" samples, which mimic the target speaker's voice based on a minimal initial speech sample. The attacker then evaluates the TPR of various pitch-twisted environmental sounds against this PT-surrogate model. From a diverse pool of environmental sounds, K sounds with the highest TPRs are selected to form a candidate set, denoted as $d_k$ for $k \in \{1, K\}$. Each sound in this candidate set is pre-processed by shifting its pitch to achieve its empirically determined best TPR. This pre-computation step ensures that the most effective environmental sound perturbations are chosen for the subsequent stage.

Stage 2: Ensemble Learning-based AE Generation

In this stage, the attacker builds multiple PT-surrogate models, denoted as $J_n$ for $n \in \{1, N\}$. These models are diversified not only by using parrot speech samples of the target speaker but also by incorporating speech samples from other speakers. This ensemble approach is crucial for improving the transferability of the generated AEs to unknown black-box target models, as an ensemble of diverse surrogate models is more likely to capture generalized adversarial features.

The attack is then formulated as an optimization problem to find the optimal carrier weights $\gamma_k$ for the pitch-twisted candidate set. The objective function is designed to minimize a joint loss function, balancing attack effectiveness and perceptual quality:

$\text{arg min}_{\gamma_k} \sum_{n=1}^{N} W_n J_n(x + \sum_{k=1}^{K} \gamma_k d_k, y_t) + \text{CSRS}(x, x + \sum_{k=1}^{K} \gamma_k d_k)$

Here:

• $x$ is the original speech signal.
• $y_t$ is the target speaker's label (the identity the attacker wants the system to misclassify the speech as).
• $W_n$ are uniform model weights assigned to each of the $N$ surrogate models in the ensemble.
• $J_n( \cdot )$ represents the loss function of the $n$-th surrogate model, aiming to maximize misclassification towards $y_t$.
• $\sum_{k=1}^{K} \gamma_k d_k$ represents the composite adversarial perturbation, formed by a weighted sum of the selected pitch-twisted environmental sound carriers.
• CSRS is a cross-entropy based Speech-Regression Score. This term quantifies the human perception quality of the adversarial example. By including CSRS in the minimization objective, the optimization process is guided to produce AEs that are not only effective in fooling the model but also maintain high perceptual quality, making them stealthy.

Additionally, a constraint is imposed on the total energy of the AE carrier:

$\sum_{k=1}^{K} \gamma_k \le \epsilon$

This constraint limits the magnitude of the overall perturbation, ensuring that the adversarial example remains subtle and difficult for humans to detect.

Since the objective function involves the non-differentiable CSRS metric, a gradient-free optimization algorithm, Simultaneous Perturbation Stochastic Approximation (SPSA), is employed to optimize the carrier weights $\gamma_k$. To mitigate the risk of converging to local minima, the carrier weights $\gamma_k$ are initialized 50 times, and the set of weights yielding the minimal ensemble loss is selected. The maximum computational cost for generating a single PT-AE is capped at 25,000 search steps, making the generation process feasible as an offline pre-computation.

This two-stage formulation, combining sophisticated parrot training, optimized carrier selection, ensemble learning, and perception-aware optimization, forms the backbone of the highly effective and practical black-box audio attack.

Demo / Proof of Concept

▶ Watch: Technical approach: two-step iterative parrot speech generation. (4:00)

The practical efficacy of the proposed PT-AE attack strategy was rigorously demonstrated through extensive evaluations in both digital-line scenarios and, critically, over-the-air (OTA) scenarios against commercial smart devices. These evaluations serve as the proof of concept, illustrating the real-world applicability and potency of the methodology.

For the digital-line evaluations, the PT-AE attack settings involved an ensemble of $N=6$ PT models (3 CNN and 3 TDNN architectures). Each PT model was built using an 8-second speech sample of the target speaker, sourced from either LibriSpeech or VoxCeleb1. To further diversify the PT models and enhance transferability, speech samples from 6 to 16 randomly chosen speakers from the VCTK dataset were also included. A candidate set of $K=50$ environmental sound carriers was selected from a pool of 200, allowing for pitch shifts up to 25 semitones. The total energy threshold $\epsilon$ for the AE carrier was set to 0.08, and the generation of a single PT-AE was capped at 25,000 search steps using the SPSA algorithm.

The digital-line targets were four distinct speaker recognition models: two statistical models (GMM-UBM and i-vector-PLDA) and two DNN-based models (DeepSpeaker and ECAPA-TDNN). These models were trained with 3 male and 3 female speakers from LibriSpeech and VoxCeleb1, using 15-second speech samples for enrollment. Attacks were evaluated across Close-Set Identification (CSI), Open-Set Identification (OSI), and Speaker Verification (SV) tasks, using Attack Success Rate (ASR) for effectiveness and Speech-Regression Score (SRS) for perceptual quality.

The results were striking:

• In the intra-gender scenario, PT-AEs achieved an average ASR of 60.2%, significantly outperforming QFA2SR (40.0%), Smack (29.9%), Occam (19.2%), and FakeBob (11.3%). Crucially, PT-AEs maintained superior perceptual quality with an average SRS of 4.1, compared to QFA2SR (2.3) and Smack (3.1).
• In the inter-gender scenario, while performance generally decreased across all methods, PT-AEs remained effective, achieving an average ASR of 54.6% and an SRS of 3.9, still outperforming QFA2SR (29.7% ASR, 3.2 SRS).

These digital-line results provided a strong foundation, but the ultimate proof of concept came from the over-the-air evaluations against popular smart devices: Amazon Echo Plus, Google Home Mini, and Apple HomePod (Siri). For enrollment, 8-second speech samples from 3 male and 3 female speakers generated by Google's text-to-speech platform were used. OSI and SV tasks were evaluated. For each attack strategy, 24 AEs were generated and played using a JBL Clip3 speaker at a distance of 0.5 meters.

The over-the-air results confirmed the practicality and effectiveness:

• PT-AEs achieved average ASRs of 58.3% (intra-gender) and 47.9% (inter-gender), with good perceptual quality (average SRS of 4.77 intra-gender, 4.45 inter-gender).
• QFA2SR was the second-best in ASR but had significantly lower perceptual quality.
• FakeBob and Occam were largely ineffective over-the-air, highlighting the robustness advantage of PT-AEs in real-world acoustic environments.

Further demonstrations included evaluating the robustness of PT-AEs over distance. ASR degradation was minimal from 0.25m to 0.5m but became evident from 2.0m to 4.0m (e.g., ASR dropped from 27.1% to 14.5% in the inter-gender scenario), primarily due to energy degradation. However, PT-AEs remained quite effective within 2.0 meters.

An ablation study was performed to quantify the contribution of each design component:

1. Parrot Training: Replacing parrot training with non-parrot-training models resulted in a 21.8% drop in average ASR (from 53.1% to 31.3%), confirming its crucial role.
2. Environmental Sound Carrier (Feature-twisted): Excluding these carriers and relying on noise or simple feature-twisted carriers led to a 23.9% and 19.8% ASR drop, respectively, demonstrating their significant enhancement.
3. Ensemble Learning: Using a single PT-CNN or PT-TDNN model yielded ASRs of 31.3% and 32.3%. Combining multiple models of the same architecture improved ASR to 43.8% and 45.8%. The full ensemble, combining different architectures, achieved the highest ASR of 53.1%.

In summary, the demonstrations unequivocally proved that parrot training, feature-twisted environmental sounds, and ensemble learning each contribute approximately equally (around 21-22% ASR improvement) to the overall effectiveness of this highly practical black-box PT-AE attack.

Defensive Implications

▶ Watch: Experimental setup and results on parrot training feasibility. (4:45)

The development of highly practical black-box audio adversarial examples, as demonstrated by the PT-AE attack, poses significant defensive challenges for speaker recognition systems. The research acknowledges these implications and discusses potential mitigation strategies, alongside responsible disclosure efforts.

The researchers engaged in responsible disclosure, reporting their findings to Amazon, Apple, and Google. Google confirmed a "voice mismatch issue" but closed the case, noting that the attack requires the addition of a malicious node. Communications with Amazon and Apple are ongoing. The mention of a "malicious node" by Google is a critical defensive insight, suggesting that while the attack is black-box in terms of model knowledge, its execution in certain real-world contexts might imply a degree of physical access or control over an audio injection point, rather than purely remote over-the-air broadcast.

Two main directions for defense strategies were discussed:

1. Audio Signal Processing: Techniques such as down-sampling, quantization, or low-pass filtering are traditional defenses against noise-carrier AEs. These methods aim to remove or attenuate high-frequency components or subtle perturbations. However, the research explicitly notes that these techniques are less effective against feature-twisted environmental sound carriers. This is because these carriers often share similar frequency ranges and spectral characteristics with legitimate human speech, making them difficult to distinguish and filter out without degrading the quality of the original speech signal itself. This inherent stealthiness of environmental sound carriers requires more sophisticated defenses.
2. Adversarial Training: This involves retraining the target model with worst-case adversarial examples to improve its robustness against such attacks. For PT-AEs, which utilize feature-twisted environmental sounds, effective adversarial training would necessitate the generation of a highly diverse set of AEs. These AEs would need to cover a wide range of environmental sound carriers, pitch manipulation variations, and other auditory features to ensure comprehensive robustness. This remains a significant area for future research, as generating such diverse and effective training examples for this specific attack type is a non-trivial task.

The findings highlight that conventional defenses, particularly simple signal processing methods, are insufficient against this new class of sophisticated, perceptually-aware audio AEs. The reliance on feature-twisted environmental sounds allows PT-AEs to bypass common filtering techniques by blending seamlessly into the acoustic environment. Therefore, developing robust defense mechanisms against PT-AEs, especially those leveraging these advanced carrier types, remains an open and crucial area of research. This necessitates exploring advanced adversarial training techniques, potentially incorporating novel feature-space defenses or anomaly detection mechanisms that can identify subtle, targeted manipulations even when they mimic natural environmental sounds. The challenge is to build defenses that do not inadvertently degrade the performance or usability of legitimate speaker recognition tasks.

Key Takeaways

• Minimizing Attacker Knowledge: The research successfully demonstrates the feasibility of launching highly effective black-box audio adversarial attacks against speaker recognition systems with the most restricted attacker knowledge to date: only a short speech sample of the target speaker, with no probing or internal model information.
• Parrot Training as a Core Enabler: The novel concept of parrot training, leveraging one-shot voice conversion to generate synthetic "parrot speech," is crucial for building effective surrogate models without direct interaction with the black-box target.
• Optimal Adversarial Carriers: Pitch-twisted environmental sound carriers (e.g., brook sounds with pitch shifts) are identified as the most effective perturbation type, offering superior transferability to black-box models while maintaining high human perceptual quality.
• Ensemble Learning Enhances Transferability: An ensemble learning strategy involving multiple, diverse parrot-trained surrogate models significantly boosts the Attack Success Rate (ASR) and transferability of PT-AEs.
• Real-World Effectiveness: PT-AEs achieve impressive attack success rates (47.9% to 58.3%) against commercial smart devices like Apple HomePod, Amazon Echo, and Google Home in over-the-air scenarios, confirming their practical relevance and threat.
• Perceptual Stealthiness: The attacks prioritize perceptual quality (SRS) alongside transferability, making the generated adversarial examples difficult for humans to detect, thus enhancing their stealthiness and real-world applicability.
• Challenges for Defense: Existing audio signal processing defenses are largely ineffective against feature-twisted environmental sound carriers, and robust adversarial training against such sophisticated attacks remains an open and critical area of future research.

About the Speaker(s)

Rui Duan is a researcher who presented this work at the prestigious NDSS Symposium. The details of their specific affiliation or title are not provided in the transcript, but their presentation highlights expertise in the field of adversarial machine learning and speech security.