Acoustic Keystroke Leakage on Smart Televisions

Acoustic Keystroke Leakage on Smart Televisions

Speakers: Tejas Kannan

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=pa_K4vaZwYE

Overview

This talk, presented by Tejas Kannan at the NDSS Symposium, uncovers a novel acoustic side-channel attack targeting the virtual keyboards of popular Smart Televisions (Smart TVs). With projections of over 266 million units sold globally by 2025, Smart TVs have become ubiquitous, offering extensive functionality from web browsing to product purchasing. This expanded utility, however, introduces significant security vulnerabilities, particularly concerning the privacy of sensitive information like passwords and credit card details entered via on-screen virtual keyboards.

The research demonstrates how distinct acoustic feedback generated by Smart TVs during cursor movements, key selections, and character deletions can be exploited. By analyzing these unique sounds, an attacker can infer the number of cursor movements between key selections, and subsequently, reconstruct the user's typed strings. The attack framework, validated through both emulation and realistic user studies on Apple and Samsung Smart TVs, achieved alarming recovery rates, including 33.33% for credit card details and 60.19% for common passwords within 100 guesses. The significance of these findings is underscored by Samsung's acknowledgment of the vulnerability and payment of a bounty, highlighting a critical oversight in Smart TV interface design and the urgent need for enhanced data privacy measures in these widely connected devices.

Background

▶ Watch: Introduction to keystroke leakage on Smart TVs & key findings (0:00)

Smart TVs have evolved significantly beyond their traditional counterparts, now serving as multi-functional hubs for entertainment, communication, and commerce. Users typically interact with these devices using wireless remotes, often equipped with directional pads, to navigate an on-screen virtual keyboard. The talk specifically highlights two popular platforms: AppleTV, which uses an alphabetical keyboard layout, and Samsung Smart TVs, which employ a QWERTY layout.

A crucial aspect of this attack lies in the acoustic properties consistently observed across both AppleTV and Samsung Smart TVs. These properties, originating directly from the TV's operating system (OS), are consistent and user-independent:

• (P1) Consistent Start Position: The cursor always begins on the same key ('a' for AppleTV, 'q' for Samsung) when the keyboard interface opens.
• (P2) Distinct Sounds: The sound produced when moving the cursor (KeyMovement) is audibly different from the sound made when selecting a key (KeySelect).
• (P3) Unique Delete Sounds: The BACK and CLEAR keys generate distinct Delete sounds compared to other key selections.
• (P4) Keyboard-Specific Selection Sound: The KeySelect sound is unique to the keyboard interface, differentiating it from generic SystemSelect sounds produced for actions outside the keyboard.

Samsung keyboards further complicate matters with dynamic inline suggestions (e.g., for web searches), which are notably absent for sensitive fields like passwords. AppleTV keyboards, conversely, lack any dynamic behavior. For the attack, a move count sequence S is defined as an ordered list of M(n) moves, where each M(n) comprises k(n) (the number of cursor movements), s(n) (the sound made, e.g., KeySelect), and t(n) (the timestamps of individual movements). The primary targets for information leakage include sensitive data like credit card details (a 5-tuple of CCN, expiration month, expiration year, CVV, ZIP, validated by Luhn's algorithm for CCNs) and passwords.

This research distinguishes itself from prior keystroke attacks, which have typically focused on mechanical keyboards (exploiting audio differences from physical key presses) or smartwatches (using accelerometers). Smart TV virtual keyboards present a unique challenge due to their requirement for sequential cursor movements rather than instantaneous key presses, and their distinct acoustic profiles. Unlike HomeSpy, which sniffs IR signals, this acoustic attack is agnostic to the remote's communication medium, making it effective even against newer Bluetooth remotes. Furthermore, it avoids vulnerabilities like those exploited by SPOOK, which relies on virtual remote pairing and is mitigated by human attestation, presenting a fundamentally different and more pervasive threat model.

The threat model for this attack assumes an adversary with passive audio access to a single target Smart TV running Samsung's Tizen OS or AppleTV's tvOS. This audio access can be achieved through two primary avenues:

1. Hijacking an adjacent device with a microphone: This is highly plausible given the proliferation of smart speakers (e.g., Amazon Echo) and known vulnerabilities that can expose their microphone feeds, enabling remote exploitation without physical access.
2. Placing a malicious microphone near the target TV: While requiring stronger capabilities, potentially physical access, this is feasible in transient environments like hotel rooms or vacation rentals, or through the use of long-range microphones.

Crucially, the adversary makes no assumptions about: the type of information being entered (passwords, credit cards, etc.); whether the user is typing at all; whether the user follows an optimal path between keys; any personal information about the user; or the ability to validate inferred keystrokes offline. This challenging threat model highlights the stealth and broad applicability of the acoustic side-channel, contrasting with video-based attacks (which require an unobstructed view) or voice input attacks (often blocked for sensitive fields).

Key Findings

▶ Watch: Unique acoustic properties of Smart TV virtual keyboards (2:00)

The research makes several significant contributions to the field of cybersecurity, particularly in the realm of side-channel attacks and Smart TV security:

1. Novel Acoustic Side-Channel Attack: The primary finding is the identification and demonstration of a new acoustic side-channel attack specifically targeting the virtual keyboards of Smart TVs. This attack exploits the distinct and consistent sounds produced by the TV's OS during cursor movements, key selections, and deletions to infer user keystrokes.
2. Comprehensive Attack Framework: The researchers developed a robust framework capable of identifying keyboard activity from raw audio, classifying specific keyboard sounds, and accurately extracting the number of cursor movements between key selections. This framework is adaptable to different Smart TV platforms (Apple and Samsung) by leveraging their unique acoustic profiles.
3. Advanced String Recovery Module: A sophisticated string recovery module was designed to infer the actual typed characters from the extracted movement sequences. This module intelligently incorporates knowledge of keyboard layouts, custom dictionaries tailored to different data types (e.g., credit cards, passwords, English words), and graph search algorithms to reconstruct strings even without knowing the direction of cursor movements.
4. Exploitation of Human Typing Behaviors: A critical finding was the ability to leverage inherent human typing patterns to significantly enhance attack performance. This includes identifying pauses indicative of suboptimal path corrections and inferring horizontal direction during rapid scrolling, thereby refining the search space and improving accuracy in realistic scenarios.
5. Significant Real-World Recovery Rates: Through extensive emulation and a realistic user study, the attack demonstrated alarming efficacy. In the user study, the attack achieved a 33.33% recovery rate for full credit card details (within 100 guesses) and a 60.19% recovery rate for common passwords (within 100 guesses) on real Smart TVs. Even for web searches, which involved dynamic suggestions, a 15% Top-100 accuracy was achieved.
6. Industry Acknowledgment: The severity of this vulnerability was formally acknowledged by Samsung, who awarded a bounty for the findings. This underscores the critical nature of the security flaw and the urgent need for Smart TV manufacturers to address acoustic leakage as a privacy concern.

In essence, the research unequivocally demonstrates that despite their ubiquitous presence and handling of sensitive data, Smart TVs currently leak private user information through their acoustic feedback, a vulnerability previously unaddressed in this context.

Technical Deep Dive

▶ Watch: Acoustic leakage example: distinguishing key movement and selection sounds (2:55)

The core of the attack is an acoustic attack framework (illustrated conceptually in Figure 6 of the original talk), comprising two main modules: Audio Extraction and String Recovery. The attack operates on fixed-length audio recordings, processing them to identify and isolate discrete typing instances.

A. Audio Extraction

The Audio Extraction module is responsible for transforming raw Smart TV audio into meaningful movement information.

1. TV Type Selection (IV-A1):

The first step involves identifying the specific Smart TV platform (e.g., AppleTV or Samsung). This is crucial because different platforms have unique acoustic profiles. The attack achieves this by matching instances of the KeySelect sound (P4) from the recorded audio against pre-recorded reference sounds for known Smart TV platforms. The platform with the highest number of matches is identified. If no matches are found, the attack aborts.

1. Extracting Move Count Sequences (IV-A2):

This is the most intricate part of the audio extraction, converting raw audio into the move count sequence S.

• Candidate Sound Identification: The process begins by identifying potential key sounds by detecting amplitude peaks in the audio waveform, similar to techniques used in prior acoustic attacks.
• Sound Classification: Each candidate sound is then classified into one of five categories: KeySelect, SystemSelect, KeyMovement, Delete, or Unknown. This classification is performed using a nearest-neighbor classifier. The researchers pre-collected extensive reference recordings for each sound type. At runtime, spectrograms (generated using Fourier transform with specific segment lengths and overlaps) are created for both candidate and reference sounds. The L1 distance between the candidate's spectrogram and each reference spectrogram is computed, and the candidate is classified based on the minimum distance, provided it falls below a predefined threshold to filter out Unknown sounds.
• Addressing Challenges:
• (a) Dimension Mismatch: Spectrograms can have varying timesteps. This is resolved by using a sliding window approach to find the minimum L1 distance, accommodating temporal misalignments.
• (b) Background Noise: To mitigate environmental noise, comparisons are limited to specific frequency bands known to contain the reference sounds. Spectrograms are also min-max normalized, and values below an empirically determined threshold (0.7) are masked to reduce noise influence.
• (c) Rapid Scrolling: Users can scroll quickly, causing multiple KeyMovement sounds to conflate into a single audio event. The framework addresses this by deduplicating these conflated sounds. It identifies spectrogram peaks specifically tuned for KeyMovement frequencies and counts the number of distinct peaks to accurately determine individual movements.
• Sequence Construction: After classification, the move count sequence S is constructed. Each element M(n) in S contains k(n) (the number of cursor movements between two selected keys), s(n) (the sound associated with the key selection), and t(n) (the amplitude peak times for each movement).

1. Splitting Keyboard Instances (IV-A3):

A continuous audio recording may contain multiple distinct typing instances. This step segments the raw move count sequence S into disjoint sequences S1, ..., Se, each representing a single instance of keyboard interaction.

• AppleTV: Simpler due to its lack of dynamic behavior. Since the Done key is the only one generating a SystemSelect sound on AppleTV, these sounds are used as delimiters. Only splits ending in a KeySelect (indicating a completed entry) are retained.
• Samsung TV: More complex due to dynamic suggestions and other keys generating SystemSelect sounds. Here, timing information is critical. Applications typically incur noticeable latency upon string submission. These delays are identified as outliers in the time differences between adjacent moves, specifically using a cutoff of avg(move_diffs) + 1.5 * stddev(move_diffs). For entirely numeric inputs like credit card details, SystemSelect sounds are again reliable for splitting, as the Done key is the primary source and dynamic suggestions are absent.

B. String Recovery

The String Recovery module takes the isolated move count sequences (Sr) and infers the actual typed string.

1. Inferring Entry Types (IV-B1):

To optimize recovery, the attack first infers the type of information being entered (e.g., password, credit card, English word).

• Credit Cards (Samsung TV): Detected by observing the characteristic lengths of consecutive keyboard instances corresponding to the 5-tuple of credit card details (CCN, month, year, CVV, ZIP), assuming CCN is entered first.
• Passwords vs. English Words (Samsung TV): Distinguished by the presence or absence of dynamic keyboard suggestions (P2). Suggestions are typically absent for password fields. A Random Forest classifier, trained on movement count histograms, is used to predict dynamic keyboard behavior. The classifier is biased towards passwords due to their higher value.
• AppleTV: All keyboard inputs are assumed to be passwords, given the platform's general usage patterns (no browser, limited credit card entry).

1. Choosing the Prior String Dictionary (IV-B2):

Since acoustic data doesn't reveal movement direction, a single move count sequence can map to many possible strings. This module uses prior dictionaries, customized by the inferred information type, to narrow down possibilities.

• CCN Prior: Enforces prefixes for major card brands (AMEX, Visa, Mastercard) and validates against Luhn's algorithm. Remaining digits are randomized.
• CVV Prior: Consists of random digits.
• Expiration Prior: Includes valid dates up to 2035.
• ZIP Prior: Comprises 33,120 US ZIP codes from 2019, weighted by population frequency.
• Password Prior: Utilizes N-gram models (e.g., 5-grams) derived from leaked datasets like PhpBB and RockYou, which contain millions of real-world passwords.
• English Prior: Based on word prefixes from the Wikipedia corpus, frequency-weighted, including words with at least 100 appearances (yielding 95,892 words).

1. Keyboard Graph Search (IV-B3):

Strings are recovered using a variant of Dijkstra's algorithm on a graph representation of the Smart TV's keyboard layout.

• Search States: Each state z is defined as (key, str, move_num, keyboard_view), representing the current key, the string typed so far, the number of movements processed, and the active keyboard view (e.g., lowercase, uppercase, symbols).
• State Expansion: When expanding state z for a move M(n), the algorithm finds neighboring keys v that are at a distance k(n) from z.key and produce sound s(n). Both wraparound (e.g., moving from the last key to the first in a row) and non-wraparound paths are considered.
• Edge Weight: The weight of an edge is computed using the string prior: weight = -log(Count(candidate_str, Prior) / Count(z.str, Prior)). This biases the search towards more probable strings.
• Addressing Challenges:
• (a) Suboptimal Paths: Users don't always take the shortest path (as shown in Figure 7 of the talk). The search is expanded to find keys within a distance range I = [k(n) - d(n), k(n) + d(n)], where d(n) is a tolerance term. Suboptimal paths are discounted by a factor γ^(d-k(n)).
• (b) Audio Extraction Errors: Errors in k(n) (e.g., conflated movements) are accommodated by setting d(n) to be at least the number of identified rapid scrolls.
• (c) Keyboard Views: Users can change keyboard views (e.g., capitalizing keys on Samsung) without producing distinct audible cues. The algorithm performs an exhaustive search, adding all reachable states through such inaudible view changes.
• (d) Dynamic Suggestions (Samsung TV): For Samsung, dynamic suggestions are predicted by identifying the most common characters in the English dictionary that follow the current string. The top six guessed characters are added to the neighbor set V, and d(n) is always set to 1 to account for the possibility of clearing suggestions.

C. User Timing Patterns

The attack further enhances its performance by leveraging consistent timing patterns observed in user studies.

1. Suboptimal Paths (IV-C1):

Users tend to pause when correcting suboptimal typing paths. The attack uses an iterative solution for d(n). It computes u(n), the largest time difference between adjacent movements for M(n). Moves are sorted by decreasing u(n), and the search is iteratively expanded with suboptimal paths for the highest-delay moves until a sufficient number of guesses (L) are found.

1. Direction Inference (IV-C2):

For rapid scrolls (at least four consecutive key movements), the attack infers horizontal direction, as neither TV platform supports vertical wraparound. This is done by computing time differences a(n) between movements. If all a(n) within a four-movement window are below a cutoff c(n) = median(a(n)), horizontal direction is assigned, significantly reducing the search space.

Demo / Proof of Concept

▶ Watch: Adversary threat model: passive audio access, no assumptions (4:00)

The efficacy of the acoustic keystroke leakage attack was rigorously demonstrated through a comprehensive user study, serving as the primary real-world proof of concept. This study was crucial for validating the attack's performance against actual human typing behaviors and environmental conditions, contrasting with the ideal conditions of emulation.

User Study Setup (VI-A):

The study involved ten subjects aged 22-29, with all procedures approved by the Institutional Review Board (IRB). Each subject was instructed to type three sets of credit card details, ten passwords, and ten web searches into real applications on two specific Smart TV models: a Samsung Smart TV (UN55MU6300) and an AppleTV (A1625). An off-the-shelf Fifine K699B microphone was strategically placed 5.5 feet (approximately 1.68 meters) from the TV, with the TV's volume set to 100% to ensure clear audio capture. The attack framework used specific discount factors (γ = 0.5 for AppleTV, γ = 10^-2 for Samsung) and suboptimal path tolerances (D = 6 for AppleTV, D = 4 for Samsung), empirically determined from observed user behavior during the study.

User Study Results (VI-B, VI-C, VI-D, VI-E):

• Credit Card Recovery (Samsung TV):

The attack successfully identified credit card details in 29 out of 30 interactions. For Credit Card Numbers (CCNs) alone, it achieved a 53.33% Top-100 accuracy. For full credit card details (the 5-tuple), the attack achieved a 43.33% Top-1000 accuracy. These figures represent a significant leakage potential for online payment systems. The study also highlighted the impact of human behavior, with users taking suboptimal paths 10.65% of the time when entering CCNs.

• Password Recovery (AppleTV & Samsung TV):

Using the PhpBB prior dictionary, the attack achieved a 33.00% Top-100 accuracy on AppleTV and a remarkable 60.19% Top-100 accuracy on Samsung TV. These rates significantly outperform random guessing, demonstrating a substantial risk to user accounts. The difference in performance between platforms was attributed to user behavior: AppleTV users took optimal paths only 45.35% of the time, compared to 85.94% on Samsung, likely due to the AppleTV remote's touchpad sensitivity leading to more varied movements.

• Web Search Recovery (Samsung TV):

For web searches, which involve dynamic suggestions and are generally less predictable, the attack achieved a 15% Top-100 accuracy. While lower than password or credit card recovery, this still vastly outperforms random guessing and indicates that even non-sensitive, but potentially private, information can be leaked.

• Impact of Keystroke Timing (VI-E):

The integration of timing-based identification for suboptimal paths proved to be a critical enhancement. For CCNs, incorporating timing increased the Top-100 accuracy from 40.00% to 53.33%. For full credit card details, accuracy improved from 36.67% to 43.33%. Furthermore, the direction inference technique, applied to rapid scrolls, consistently improved or maintained recovery rates, never harming performance.

These results unequivocally prove that the acoustic side-channel attack is not merely a theoretical construct but a practical and effective threat in real-world Smart TV environments, capable of compromising sensitive user data.

Defensive Implications

▶ Watch: System design: Audio Extraction module and sound classification (6:00)

The findings of this research highlight a critical vulnerability in Smart TV design, necessitating robust defensive strategies. Several potential countermeasures were discussed, each with its own set of trade-offs and limitations:

1. Muting Audio During Sensitive Input: The most straightforward defense is to simply mute the Smart TV's audio feedback whenever a user is interacting with a virtual keyboard, especially for sensitive fields like passwords or credit card details. While effective in preventing acoustic leakage, this approach can significantly degrade the user experience, as audible feedback often provides crucial confirmation of input.

1. Randomized Keyboard Layouts: Prior work has suggested randomizing virtual keyboard layouts as a defense against visual and other side-channel attacks. While this could reduce the effectiveness of the acoustic attack, particularly by complicating path inference, the researchers note that their attack framework can model some forms of randomization. Critically, randomization would significantly reduce the efficacy of timing-based suboptimal path detection, which relies on consistent key positions. In emulation, a randomized layout caused a substantial drop in CVV recovery, from 100% to 30.32%, indicating its potential.

1. Randomized Cursor Start Position (P1): Randomizing where the cursor initially appears on the keyboard (violating acoustic property P1) could be another defense. However, for specific input fields, such as Credit Card Numbers (CCNs), where users must ultimately navigate to a fixed "Done" or "Submit" key, an attacker could still effectively search in reverse from the known end point. Emulation showed that even with randomized start positions, 99.88% of CCNs were still recoverable, suggesting this defense is insufficient on its own.

1. Uniform Keyboard Action Sounds (P2-P4): The most fundamental defense against this specific acoustic attack would be for Smart TVs to use the same sound for all keyboard actions – KeyMovement, KeySelect, and Delete. This would effectively break acoustic properties P2, P3, and P4, which are foundational to the attack's ability to differentiate actions and count movements. If all actions sound identical, distinguishing between them becomes significantly harder. However, the researchers caution that even this might not guarantee complete security, as it might still be possible to discern movements and selections using only timing information (e.g., duration between sounds), which would require further investigation.

Samsung's acknowledgment of the vulnerability and payment of a bounty underscores the industry's recognition of this threat. Ultimately, the research highlights a critical need for Smart TV manufacturers to re-evaluate their user interface designs, ensuring that all features, especially those involving sensitive data input, are developed with privacy-by-design principles to mitigate such acoustic side-channel risks.

Key Takeaways

• Smart TV virtual keyboards leak keystrokes: A novel acoustic side-channel attack can extract sensitive user input from the distinct sounds produced by Smart TVs during keyboard interactions.
• Distinct acoustic feedback is exploitable: The attack leverages unique sounds for cursor movements (KeyMovement), key selections (KeySelect), and character deletions (Delete) to infer the number of cursor movements between selected keys.
• Comprehensive framework for recovery: A two-stage framework, comprising Audio Extraction and String Recovery, utilizes platform identification, spectrogram analysis, nearest-neighbor classification, keyboard graph search, and prior dictionaries to reconstruct typed strings.
• Human typing patterns enhance attack efficacy: The attack significantly improves its performance by exploiting real-world human behaviors, such as pauses indicative of suboptimal path corrections and consistent timing patterns for rapid horizontal scrolling.
• Significant real-world data leakage demonstrated: User studies on Apple and Samsung Smart TVs achieved alarming recovery rates: 33.33% for full credit card details and 60.19% for common passwords within 100 guesses.
• Defensive measures are crucial but challenging: While muting audio, randomized layouts, and uniform keyboard sounds offer potential mitigations, each presents user experience trade-offs or has limitations against sophisticated attacks, necessitating a holistic approach to Smart TV privacy.

About the Speaker(s)

The talk "Acoustic Keystroke Leakage on Smart Televisions" was presented by Tejas Kannan. Based on the mention of an "IRB" (Institutional Review Board) in the transcript, it can be inferred that Tejas Kannan is likely affiliated with an academic or research institution. The presentation highlights his expertise in cybersecurity and side-channel analysis, specifically focusing on novel threats to ubiquitous consumer devices like Smart TVs.