Data On Demand The challenges of building a privacy focused AI Device

Data On Demand: The Challenges of Building a Privacy-Focused AI Device

Speakers: Matt Domko

Conference: DEF CON 32

YouTube: https://www.youtube.com/watch?v=jS3iNUfYtLA

Overview

In "Data On Demand: The Challenges of Building a Privacy-Focused AI Device" at DEF CON 32, Matt Domko delivered a critical examination of the complexities inherent in developing consumer-grade AI hardware that prioritizes user privacy. Domko, a cybersecurity professional with a passion for security education, articulated the fundamental tension between delivering a seamless, intuitive AI experience for the general public and rigorously protecting their sensitive personal data. The talk underscored that while AI devices promise convenience and enhanced capabilities, their underlying data architectures often pose significant privacy risks if not meticulously designed with security at their core.

Domko's presentation was particularly timely given the recent proliferation of AI-powered gadgets, often marketed with broad capabilities but vague privacy assurances. He highlighted that the average consumer, unlike a security-conscious enterprise user, may not possess a deep understanding of data handling practices, making robust, built-in privacy safeguards absolutely essential. The core challenge, as presented, lies in creating AI systems that are both "magical" in their utility and uncompromising in their commitment to user data protection, a balance that requires careful consideration of data types, request workflows, and authentication mechanisms.

This article delves into Domko's framework for understanding AI device interactions, the various categories of data involved, and the critical security implications of different authentication strategies. It aims to provide a comprehensive analysis of the talk's key contributions, offering insights for both developers striving to build secure AI products and consumers seeking to understand the privacy landscape of their smart devices. The discussion emphasizes that privacy is not merely a feature but a foundational requirement for ethical and trustworthy AI.

Background

▶ Watch: Introduction and the challenge of privacy-focused AI devices (0:00)

The rapid advancement of artificial intelligence, particularly large language models (LLMs) and speech-to-text technologies, has spurred the creation of a new generation of hardware devices designed to bring AI capabilities into everyday life. These devices, often conceptualized as personal AI assistants or "AI walkie-talkies," aim to simplify interactions with digital services through natural language. However, the very nature of these interactions—processing spoken commands, understanding context, and executing actions on behalf of a user—necessitates the collection, storage, and processing of vast amounts of potentially sensitive personal data.

Historically, consumer technology has often prioritized convenience and functionality over robust privacy by default. Many early internet services and smart devices adopted data collection practices that, while enabling personalized experiences, often led to user data being aggregated, analyzed, and sometimes exposed or exploited. With AI devices, this problem is exacerbated. The data involved is not just browsing history or app usage, but intimate details like voice recordings, personal conversations, location data, familial connections, and access to authenticated third-party accounts (e.g., Spotify, Uber Eats, GrubHub).

Matt Domko's talk addresses this burgeoning privacy gap in the consumer AI hardware space. He frames the problem as a fundamental dichotomy: how to make AI "easy" for the general consumer who may lack technical sophistication, while simultaneously upholding the highest standards of user privacy. This challenge is distinct from enterprise AI deployments, where organizations typically have established data governance policies and a security-aware user base. For consumer devices, the onus is entirely on the manufacturer to implement privacy-by-design principles, anticipating and mitigating risks that users may not even perceive. The talk builds on a landscape where general-purpose AI devices are emerging, often with similar core functionalities but varying approaches to data handling, making a standardized framework for privacy paramount.

Key Findings

▶ Watch: Identifying different data types and their sensitivity (3:55)

Matt Domko's presentation distilled the complex interplay of functionality and privacy in AI devices into several key findings, primarily centered around categorizing AI requests and analyzing the associated data and authentication challenges.

First, Domko identified four distinct types of AI device requests, each with varying privacy implications:

1. General Knowledge Request: These requests require no personal context. An example is "What's the weather out today?" or "Tell me a fact about cats." The data involved is primarily the audio input, its transcript, and the LLM's response. While seemingly innocuous, even these requests can become sensitive (e.g., "I've got a weird rash, should I see a doctor?").
2. Contextual Knowledge Request: These require some non-super-sensitive personal information to provide a tailored response. For instance, asking for "five good restaurants in the area" benefits from knowing a user's location and dietary preferences. This introduces contextual data like location and interests, which must be protected.
3. Generic Action Request: These involve the device performing an action that doesn't require specific user authentication, often through an actuator (a component that performs a task based on input). An example is "Call the Google Weather API for me." The data types are similar to knowledge requests, plus any necessary contextual data.
4. Authenticated Action Request: These are the most privacy-sensitive, requiring the AI device to act on behalf of the user within a third-party service, such as "Call me an Uber" or "Order me some tacos from GrubHub." This type of request introduces authentication information and additional contextual data related to identity and familial connections (e.g., "Did Jimmy do his homework?").

Second, Domko highlighted the critical types of data handled by AI devices:

• Audio Data: The raw recording of the user's voice.
• Transcript Data: The text conversion of the audio.
• Response Data: The LLM's generated answer.
• Contextual Data: Information like location, interests, and personal relationships.
• Authentication Information: Credentials or tokens required to access third-party services.

Third, a central finding revolved around the trade-offs in authentication strategies for consumer AI devices. Domko analyzed three common approaches:

• Storing Username and Password: While offering maximum convenience for the service provider (can log in anytime, anywhere, perform any user action), this method carries catastrophic security risks. If stolen, an attacker gains complete control over the user's linked accounts. Domko explicitly states, "this one doesn't feel like the right thing to do."
• Storing Session Cookies: This improves security slightly by limiting access to an existing session, meaning an attacker cannot create new sessions. However, sessions can expire, posing an engineering challenge. Crucially, stolen cookies can still grant broad access to user actions without requiring password re-entry, similar to how GitHub requires a "step up" authentication (password and MFA) for sensitive actions like adding collaborators to a repository, even if a session is active.
• Using OAuth (Open Authorization): This is presented as the most secure and privacy-respecting option. OAuth grants limited, revocable permissions (tokens) to specific actions, rather than full account access. Users can revoke access at any time without changing their password. The major limitation, however, is Oauth adoption: many consumer-facing websites and local businesses (e.g., a local mechanic shop) do not support OAuth, making it impractical for a general-purpose AI device that aims to interact with a wide array of services. This highlights a significant industry-wide challenge for universal secure integration.

Finally, Domko implicitly found that the drive for market adoption and ease-of-use often clashes directly with security best practices, particularly regarding authentication. Businesses need to make money, and implementing robust, but potentially less convenient, security measures like OAuth can be a hurdle if the broader ecosystem isn't ready. This necessitates a careful balancing act and a clear understanding of the tradeoffs involved in each design decision.

Technical Deep Dive

▶ Watch: Role of actuators and contextual data in AI actions (4:20)

The technical core of Domko's talk lies in dissecting the data flows and security mechanisms within a conceptual AI hardware device, which he often refers to as an "AI walkie-talkie." This device serves as a model for understanding how consumer-grade AI interacts with users, cloud services, and third-party applications.

At its most basic, an AI device workflow involves the following sequence:

1. User Input: The user speaks a command into the device.
2. Cloud Transmission: The audio data is sent over the internet to a cloud provider.
3. Routing, Storage, and Replay Engine: A backend system manages the flow, potentially storing data for processing or audit.
4. Speech-to-Text (STT): The audio is converted into a text transcript.
5. Large Language Model (LLM) Processing: The transcript is fed to an LLM to interpret the request and formulate a response.
6. Text-to-Speech (TTS): The LLM's text response is converted back into audio.
7. Device Output: The audio response is played back to the user.

This core flow is augmented depending on the type of request. For Generic Action Requests, an actuator is introduced. An actuator is defined as "a thing that can perform a task when you provide an input." This could be an API call to a service like the Google Weather API. The critical aspect here is that the actuator performs a task without needing to know the user's specific identity or credentials for that service, acting as a general-purpose interface.

The most technically complex and privacy-sensitive requests are Authenticated Action Requests. These require the AI device's backend to store and utilize authentication information to interact with third-party services (e.g., Uber Eats, Spotify) on the user's behalf. Domko meticulously detailed the technical implications of three primary authentication strategies:

1. Storing Usernames and Passwords:

• Mechanism: The service provider stores the user's plaintext or weakly hashed username and password in their database.
• Pros (from a service provider's perspective, though highly problematic): Allows the service to log in "whenever we want, wherever we want" without managing session state. If logged out, the service can simply log back in. The service can perform "literally anything the user can do."
• Cons (severe security implications):
• Data Breach Catastrophe: A breach of the service provider's database immediately exposes user credentials, allowing attackers to compromise all linked third-party accounts.
• Irrevocable Access: To revoke the service's access, the customer must change their password for the third-party service, which is a cumbersome and often overlooked step.
• Excessive Permissions: The service gains full, unfettered access to the user's account, far beyond what might be necessary for specific AI commands. This violates the principle of least privilege.
• Conclusion: Domko unequivocally states this is "probably not" cool and "doesn't feel like the right thing to do," highlighting its unacceptable security posture.

1. Storing Session Cookies:

• Mechanism: The service provider stores active session cookies obtained when the user logs into a third-party service.
• Pros: Prevents the service from creating new sessions, limiting it to using an existing, authenticated session. It offers slightly better security than storing raw credentials.
• Cons:
• Session Expiration: Cookies expire regularly, requiring the engineering team to manage re-authentication or session renewal, which can be a "downside for the engineering team."
• Broad Access: A stolen session cookie can still grant access to "almost everything that a user can do" within the duration of the session. Domko uses the example of GitHub: while a logged-in user can create a repository, adding a new collaborator often requires a "step up" authentication (password + MFA push), a safeguard not easily replicated with just a stored cookie.
• Revocation Challenges: Revoking access typically involves logging out of the third-party service, which might not be immediately apparent or easy for a consumer to do for an AI device.

1. Using OAuth (Open Authorization):

• Mechanism: OAuth provides a secure delegation framework where users grant specific, limited permissions (scopes) to an application (the AI device) to access their data or perform actions on a third-party service, without sharing their actual credentials. It uses access tokens that are typically short-lived and refreshable.
• Pros:
• Least Privilege: Grants only necessary permissions, reducing the impact of a breach.
• Revocable Access: Users can easily revoke an application's access at any time directly from the third-party service's settings, without changing their password.
• No Credential Storage: The AI device never stores the user's actual username or password for the third-party service.
• Session Management: OAuth tokens have defined lifespans and refresh mechanisms, providing inherent session management.
• Cons:
• Limited Adoption: This is the "biggest limitation." Domko points out that while B2B enterprises (e.g., DataDog, GitLab) widely adopt OAuth due to security demands, many consumer-facing websites and smaller businesses "are not incentivized to invest in Oauth." This means a general-purpose AI device aiming to interact with a broad spectrum of online services will frequently encounter services that simply do not support OAuth, forcing developers to compromise on security or limit functionality.

Domko's deep dive reveals that while OAuth is the technically superior and privacy-preserving solution, its lack of universal adoption in the consumer web ecosystem presents a significant barrier to building truly privacy-focused, yet broadly functional, AI devices. This forces developers to confront uncomfortable tradeoffs between user convenience, broad functionality, and robust security.

Demo / Proof of Concept

▶ Watch: Authenticated actions: the need for secure credential storage (5:30)

While Matt Domko's talk did not feature a live software or hardware demonstration of a specific AI device or exploit, he extensively utilized the concept of an "AI walkie-talkie" as a conceptual proof of concept to illustrate the various data flows and privacy challenges. This conceptual device served as a tangible example for discussing the four types of requests (general knowledge, contextual knowledge, generic action, authenticated action) and the associated data types (audio, transcript, response, contextual, authentication information).

Domko walked the audience through hypothetical interactions with this "AI walkie-talkie," such as asking "What's the weather out today?" or "Order me some tacos from GrubHub." These scenarios, while not live demonstrations, effectively simulated the user experience and the underlying technical processes that would occur. He detailed how audio input would travel to the cloud, undergo speech-to-text conversion, be processed by an LLM, and potentially trigger actions via an actuator or authenticated API calls.

The strength of this approach was its focus on the architectural and data-handling aspects, rather than a specific implementation. By using a clear, relatable device concept, Domko was able to highlight the abstract security and privacy concerns in a concrete manner, making the technical challenges of data protection and authentication strategies more understandable for the audience. The "demo" was, therefore, a detailed walkthrough of potential system architectures and their inherent vulnerabilities and privacy considerations, serving to illuminate the problem space for developers building such devices.

Defensive Implications

▶ Watch: Analyzing secure authentication methods for AI devices (6:15)

Matt Domko's talk provides crucial insights for security professionals and developers tasked with building and securing AI-powered consumer devices. The defensive implications revolve around meticulous data handling, robust authentication, and a proactive security-first mindset.

1. Data Minimization and Granular Control:

• Identify and Classify Data: Developers must meticulously categorize all data types processed by the AI device: audio data, transcript data, response data, contextual data, and authentication information. Each category has different sensitivity levels.
• Minimize Data Collection: Only collect data absolutely necessary for the requested function. For a "general knowledge request," contextual data like location or interests should not be collected or retained.
• Ephemeral Data Handling: Whenever possible, audio and transcript data should be processed and then immediately discarded, especially for non-contextual requests. Avoid unnecessary storage of raw voice recordings.
• User Consent and Transparency: Clearly communicate to users what data is collected, why, and how it's used. Provide granular controls for users to opt-in or out of specific data collection (e.g., location tracking, interest profiling).

1. Prioritize Secure Authentication (OAuth First):

• Default to OAuth: For any "authenticated action request," OAuth should be the primary and preferred authentication mechanism. It provides least privilege, revocability, and prevents the AI service from ever storing the user's direct credentials.
• Mitigate Non-OAuth Scenarios: When integrating with services that do not support OAuth, developers must carefully evaluate the risks of alternative methods (session cookies, direct credentials). If session cookies are used, implement strong session management (short lifespans, forced re-authentication for sensitive actions, robust encryption) and provide clear user mechanisms for session revocation. Storing raw usernames and passwords should be considered an absolute anti-pattern due to its catastrophic security implications.
• Step-Up Authentication: For highly sensitive actions (e.g., financial transactions, making significant account changes), even with OAuth or session cookies, implement "step-up" authentication requiring explicit user re-verification (e.g., password re-entry, MFA push) on the device itself.

1. Implement Robust Data Pipelines and Security Controls:

• "Four Questions" for Data Handling: Domko presented a crucial set of questions that security teams should ask for any new feature or data flow [29:00]:

1. Are you logging a new type of data? If yes, a security review is warranted to assess sensitivity, storage requirements, and retention policies.
2. Are you moving data outside of prod? Moving production data to development, testing, or analytics environments requires stringent controls, anonymization, or pseudonymization to prevent exposure.
3. Are you connecting to prod databases? Direct connections from new services or features to production databases are high-risk and necessitate careful access control and auditing.
4. Are you bringing in a new tool or vendor? Third-party integrations introduce supply chain risks. Thorough vendor security assessments are essential.

• Encryption at Rest and In Transit: All data, especially sensitive audio, transcript, contextual, and authentication data, must be encrypted both when stored and when transmitted between the device, cloud, and third-party services.
• Access Control: Implement strict role-based access control (RBAC) for internal teams accessing user data. Log and audit all access attempts.
• Regular Security Audits and Penetration Testing: Continuously assess the security posture of the AI device, its cloud backend, and integrated services.

1. Educate and Empower Users:

• Clear Privacy Policies: Draft privacy policies that are easy for the general consumer to understand, avoiding legal jargon.
• In-Device Privacy Settings: Provide intuitive settings on the AI device or its companion app that allow users to manage their data, revoke permissions, and understand what information is being shared.
• Transparency on Trade-offs: Be transparent about the limitations of privacy controls, especially when integrating with services that lack robust security features like OAuth.

By adopting these defensive strategies, developers can move closer to building AI devices that not only offer convenience but also earn and maintain user trust through principled privacy by design.

Key Takeaways

• Privacy is Paramount for Consumer AI: Unlike enterprise solutions, consumer AI devices require privacy to be a foundational design principle, as general users may not understand data handling complexities.
• Four Request Types Dictate Data Needs: AI interactions can be categorized into General Knowledge, Contextual Knowledge, Generic Actions, and Authenticated Actions, each demanding different levels and types of data.
• Authentication is a Critical Security Battleground: The choice of authentication mechanism (storing credentials, session cookies, or OAuth) has profound security and privacy implications, with OAuth being the most secure.
• OAuth's Limited Adoption is a Major Hurdle: While OAuth is the ideal for secure delegation, its absence in many consumer-facing services forces developers into less secure alternatives or limits AI device functionality.
• Data Minimization and Ephemeral Processing: For privacy, AI devices should collect only necessary data, process sensitive information ephemerally, and offer granular user controls over data retention.
• "Four Questions" for Data Governance: Any new data type, data movement outside production, connection to production databases, or new vendor/tool integration requires a rigorous security review.

About the Speaker(s)

Matt Domko is a dedicated cybersecurity professional with a strong passion for security education. He emphasizes that the insights shared in his talk are his own, reflecting his personal commitment to making technology safer, and not necessarily the views of his employer. Beyond his professional work, Matt is an enthusiastic hobbyist, deeply involved in 3D printing, laser cutting, and cosplay, particularly steampunk themes. He also proudly holds the unofficial, unpaid title of "Director of Fun at Tines," a role that underscores his engaging and approachable demeanor in the security community. Matt is always keen to "nerd out" about his hobbies or discuss security education, demonstrating his continuous drive to share knowledge and foster a more secure technological landscape.