Hacker vs AI perspectives from an ex spy

Hacker vs AI perspectives from an ex spy

Speakers: Harriet Farlow

Conference: DEF CON 32

YouTube: https://www.youtube.com/watch?v=WC-tY-gEIPc

Overview

This DEF CON 32 talk, "Hacker vs AI perspectives from an ex spy," delivered by Harriet Farlow, offers a unique perspective on the intersection of artificial intelligence and security, viewed through the critical lens of national security and policy. Farlow, an alumna of the Australian Signals Directorate (ASD) – Australia's equivalent of the NSA – and now the founder of Mileva Security Labs, brings a decade of experience in data science, defense projects, and adversarial machine learning research to the forefront. The presentation, explicitly designated for the policy village, aimed to address the strategic implications and vulnerabilities of AI systems rather than delve into specific hacking techniques or APT evasion methodologies.

Farlow's core message emphasizes the growing and often underestimated threat landscape surrounding AI, particularly the significant gap in AI security awareness and robust defensive measures within the private sector. Drawing from her background in intelligence and her ongoing PhD research, she highlights that while governments and defense agencies are increasingly recognizing and investigating methods to "hack models," the commercial sphere lags, leaving critical AI deployments vulnerable. The talk serves as a call to action, urging a deeper understanding of AI-specific risks and a more proactive, policy-driven approach to securing these burgeoning technologies, which are rapidly becoming integral to national infrastructure and defense capabilities.

The significance of this talk lies in its ability to bridge the gap between technical cybersecurity and national policy, advocating for a holistic security posture that encompasses AI. Farlow's insights from both government intelligence and a cutting-edge AI security startup provide a crucial dual perspective, underscoring that the vulnerabilities inherent in AI are not merely academic curiosities but represent tangible risks with potentially severe national security implications. Her work, including her PhD in adversarial machine learning, directly investigates how these models can be exploited, making her a leading voice in a field that demands urgent attention from both technical practitioners and policymakers.

Background

▶ Watch: Introduction: AI, security, national security policy (0:00)

The rapid proliferation of artificial intelligence across various sectors, from critical infrastructure and defense systems to everyday commercial applications, has introduced a new dimension to the cybersecurity threat landscape. Traditional cybersecurity paradigms, while robust for conventional IT systems, often fall short when confronted with the unique vulnerabilities inherent in complex AI and machine learning models. This talk arises from the urgent need to understand and address these novel attack surfaces, particularly from a national security standpoint, where the stakes are inherently higher.

Harriet Farlow's extensive background provides the foundational context for this critical discussion. With a decade of experience at the confluence of AI and security, her journey began in data science, specializing in defense projects within Australia. Her early career involved augmenting human roles with technologies like robotic process automation (RPA) in Air Force projects and contributing to defense operations, such as those on patrol boats in Darwin. These experiences provided firsthand insight into the practical application of data science in high-stakes environments. A pivotal phase of her career was spent at the Australian Signals Directorate (ASD), Australia's equivalent of the NSA. There, she served as a data scientist within cybersecurity teams and later as a technical director for AI streams, leading international technical collaborations on AI security with Canada and the USA. This intelligence background instilled a deep appreciation for the strategic implications of technology and the imperative of robust security.

During her tenure at ASD, Farlow embarked on a PhD in adversarial machine learning, a field dedicated to investigating methods to "hack models." This academic pursuit, combined with her practical intelligence experience, revealed a significant disparity: while the theoretical and governmental understanding of AI vulnerabilities was growing, the private sector's engagement in AI security was notably sparse. She observed that despite the clear and present threat posed by compromised AI systems, there was "not that much activity happening in that space" commercially. This perceived gap prompted her to leave government service and establish Mileva Security Labs, an AI security company focused on building technological solutions to address these very challenges. Her entrepreneurial venture aims to translate advanced research into practical defenses, driven by the conviction that the private sector needs to urgently elevate its AI security posture to mitigate emerging risks that have profound national security ramifications. The problem, as framed by Farlow, is not just the existence of AI vulnerabilities, but the collective underestimation and underinvestment in securing these systems, particularly outside of specialized government agencies.

Key Findings

▶ Watch: Joining Australian Signals Directorate (ASD) during COVID (2:20)

While this particular DEF CON talk was positioned within the policy village and thus did not delve into specific technical "findings" in the traditional sense (e.g., new exploits or CVEs), Harriet Farlow's presentation and background illuminate several critical observations and overarching "findings" regarding the state of AI security, especially from a national security perspective. These are derived from her extensive professional experience and ongoing research:

Firstly, a primary finding is the demonstrable vulnerability of AI models to malicious manipulation. Through her PhD research in adversarial machine learning, Farlow has been actively "investigating ways to hack models," including specific work on "hacking facial recognition AI models." This work underscores that AI systems, far from being infallible, possess inherent weaknesses that can be exploited by adversaries. These vulnerabilities are not merely theoretical but represent tangible attack vectors that can lead to misclassification, data exfiltration, or complete model subversion, with potentially catastrophic outcomes in sensitive applications.

Secondly, Farlow highlights a significant chasm between the awareness and proactive measures taken by national security agencies versus the private sector regarding AI security. Her experience at the Australian Signals Directorate (ASD) revealed a dedicated focus on understanding and mitigating AI threats within intelligence communities. In stark contrast, she found that "in the private sphere, there was just not that much activity happening in that space, even though there was... a real threat." This disparity is a critical finding, indicating a collective underestimation of AI risks by commercial entities, which are increasingly deploying AI in critical business functions and infrastructure. This gap implies that many commercially developed AI systems may be deployed without adequate security considerations, creating widespread vulnerabilities that could be exploited by state-sponsored actors, cybercriminals, or other malicious entities.

Thirdly, the talk implicitly finds that AI security is not merely a technical problem but a policy imperative with national security implications. By presenting in the policy village and framing the discussion from a "national security lens," Farlow emphasizes that the integrity and resilience of AI systems are crucial for national defense, economic stability, and societal trust. The ability to "hack an AI" can undermine critical government functions, intelligence gathering, and military capabilities, making AI security a matter of strategic importance that requires not only technical solutions but also robust policy frameworks, regulations, and public-private partnerships.

Finally, her decision to found Mileva Security Labs is a direct consequence of these findings, indicating that existing solutions and market engagement in AI security are insufficient. The "grind" of building a tech product in this space, even after receiving funding, points to the nascent stage of the AI security industry and the considerable work still needed to develop and deploy effective defensive technologies and practices. This highlights a market failure where the demand for AI security solutions far outstrips their current availability and adoption, necessitating dedicated entrepreneurial efforts to bridge this critical gap.

Technical Deep Dive

▶ Watch: Role at ASD: AI/cyber security technical director (2:45)

While Harriet Farlow explicitly stated that this particular DEF CON talk was not intended for "cool hacking techniques or how to evade an APT" due to its policy village context, her background and consistent references to her PhD research in adversarial machine learning provide a clear technical framework for the types of vulnerabilities and attack vectors she investigates. Her work centers on understanding how to "hack models" and, specifically, how to compromise "facial recognition AI models," which falls squarely within the domain of AI security.

Adversarial machine learning is a field dedicated to studying the vulnerabilities of machine learning models to malicious inputs and developing robust defenses against such attacks. Unlike traditional cybersecurity, which often focuses on software bugs or network intrusions, adversarial machine learning targets the data inputs, the model's training process, or the model's internal representations to manipulate its behavior. The goal is often to cause a model to make incorrect predictions, reveal sensitive training data, or even inject backdoors.

Key attack types within adversarial machine learning include:

1. Evasion Attacks: These are perhaps the most well-known. An attacker crafts a subtly perturbed input (an "adversarial example") that is designed to be misclassified by the model while appearing normal to a human observer. For instance, in an image classification task, a few imperceptible pixel changes can cause a model to classify a stop sign as a yield sign. In the context of facial recognition, an attacker might wear specific glasses or makeup that, to the human eye, doesn't alter their identity but causes a facial recognition system to incorrectly identify them as someone else or fail to recognize them altogether. Farlow's mention of "new methods to hack facial recognition AI models" directly points to research in this area, where adversaries aim to bypass or spoof biometric systems.

• Techniques often involve gradient-based methods like the Fast Gradient Sign Method (FGSM), Projected Gradient Descent (PGD), or Carlini & Wagner (C&W) attacks, which compute the gradient of the model's loss function with respect to its input to find the optimal perturbation direction.

1. Poisoning Attacks: These attacks occur during the model's training phase. An attacker injects malicious data into the training dataset, influencing the model's learning process. This can lead to a "backdoored" model that behaves normally on most inputs but exhibits malicious behavior when specific trigger inputs are provided. For example, a poisoning attack on a spam filter could train it to classify certain malicious emails as legitimate, provided they contain a specific, hidden phrase. For facial recognition, a poisoning attack could embed a backdoor that causes the system to grant access to an unauthorized individual if they present a specific, seemingly innocuous visual cue.

1. Model Inversion Attacks: In these attacks, an adversary attempts to reconstruct sensitive information about the training data from a deployed model. Given a model's output (e.g., a person's name from a facial recognition system), an attacker might try to reconstruct the original input image (the face) used to train the model for that individual. This poses significant privacy risks, especially for models trained on sensitive personal data.

1. Membership Inference Attacks: Similar to model inversion, these attacks aim to determine if a specific data point was part of the model's training dataset. By querying the model and observing its confidence scores, an attacker can infer whether a particular individual's data was included in the training set, again raising privacy concerns, particularly for sensitive applications like healthcare or intelligence.

1. Model Extraction Attacks (Stealing Attacks): An attacker interacts with a black-box model (where internal parameters are unknown) to replicate its functionality. By making numerous queries and observing the outputs, they can train their own "copycat" model that mimics the target model's behavior. This can lead to intellectual property theft and can also be a precursor to other attacks, as the extracted model might then be used for more effective adversarial example generation.

Farlow's work, particularly from her time at ASD, would have involved not just understanding these attack vectors but also developing methods to detect and defend against them. Defenses often include:

• Adversarial Training: Augmenting the training data with adversarial examples to make the model more robust.
• Defensive Distillation: Training a second model on the softened outputs (probability distributions) of a first model, which can make the second model less sensitive to small input perturbations.
• Input Preprocessing: Applying transformations to inputs to remove adversarial perturbations.
• Certified Robustness: Developing models with mathematical guarantees against certain types of adversarial attacks.

The shift from government intelligence to founding Mileva Security Labs implies a focus on operationalizing these technical understandings into practical security products and services for the private sector. While the talk itself abstained from live demonstrations of these techniques, the very foundation of Farlow's career and entrepreneurial endeavor is deeply rooted in these sophisticated technical challenges of securing AI.

Demo / Proof of Concept

▶ Watch: Founding Mileva Security Labs from adversarial ML research (3:40)

This particular DEF CON 32 presentation, delivered in the policy village, was explicitly not structured to include live demonstrations or technical proofs of concept. Harriet Farlow stated at the outset that attendees should not expect "cool like hacking techniques or how to evade an APT" as this was a policy-oriented discussion. Her focus was on the strategic implications of AI security from a national security perspective, rather than showcasing specific exploits or defensive tools in action. Therefore, no demo or proof of concept was presented as part of this talk.

Defensive Implications

▶ Watch: Entrepreneurial journey: Funding, social media, podcast efforts (4:00)

Harriet Farlow's talk, underpinned by her extensive experience in national security and adversarial machine learning, carries profound defensive implications for organizations across both public and private sectors. The core message is a resounding call for a paradigm shift in how AI systems are secured, moving beyond traditional cybersecurity practices to embrace a proactive, AI-specific defense strategy.

Firstly, a critical implication is the urgent need for organizations to recognize AI as a distinct and vulnerable attack surface. It is insufficient to apply generic cybersecurity measures to AI systems; specific vulnerabilities related to data integrity, model robustness, and interpretability must be addressed. Defenders need to understand that AI systems can be manipulated not just through software bugs but through subtle alterations to input data (adversarial examples), poisoning of training data, or inference of sensitive information from model outputs. This requires specialized expertise in adversarial machine learning and AI security engineering.

Secondly, organizations, particularly those deploying AI in critical or sensitive applications, must invest in robust AI security frameworks and practices throughout the entire AI lifecycle. This includes:

• Secure Data Governance: Ensuring the integrity and provenance of training data to prevent poisoning attacks. Strict access controls and auditing for data pipelines are essential.
• Adversarial Robustness Testing: Proactively testing AI models against known adversarial attacks (e.g., evasion, poisoning, model inversion) before deployment. This involves generating adversarial examples and evaluating the model's resilience.
• Continuous Monitoring and Anomaly Detection: Implementing systems to detect unusual inputs or outputs that might indicate an ongoing adversarial attack in deployed AI models.
• Explainability and Interpretability: Developing AI models that can explain their decisions, making it easier to identify when a model is behaving maliciously or erroneously due to an attack.
• Secure Model Deployment: Protecting models from unauthorized access, reverse engineering, and extraction attacks.

Thirdly, Farlow's observation about the private sector's lagging engagement in AI security highlights the need for increased collaboration and knowledge transfer between government intelligence agencies and commercial enterprises. Governments possess invaluable insights into state-sponsored threats and advanced adversarial techniques, which can inform private sector defense strategies. Conversely, the private sector's rapid innovation in AI development can provide new challenges and solutions. Establishing forums, sharing threat intelligence, and developing common standards for AI security are crucial steps.

Fourthly, the policy-focused nature of the talk implies that regulatory and policy frameworks for AI security are essential. Governments should consider developing guidelines, certifications, or even mandatory security requirements for AI systems, particularly those deemed critical infrastructure or high-risk. This would incentivize organizations to prioritize AI security and ensure a baseline level of protection across industries.

Finally, the talk implicitly advocates for upskilling and specialized training for cybersecurity professionals in AI security. Traditional cybersecurity roles may not have the necessary machine learning expertise to identify and mitigate AI-specific threats. Investing in education and training programs focused on adversarial machine learning, AI ethics, and secure AI development is vital to build a competent workforce capable of defending against these emerging challenges. Farlow's own entrepreneurial venture, Mileva Security Labs, is a direct response to this defensive need, aiming to provide the necessary tools and expertise to bridge the current security gap.

Key Takeaways

• AI Security is a National Security Imperative: The talk underscores that AI vulnerabilities are not just technical issues but have profound implications for national defense, intelligence, and critical infrastructure, demanding a strategic, policy-driven approach.
• Adversarial Machine Learning Poses Significant Threats: AI models are demonstrably vulnerable to various attacks, including evasion, poisoning, model inversion, and membership inference, which can lead to misclassification, data theft, and model subversion.
• Private Sector Lags in AI Security: There is a critical gap in awareness and proactive measures regarding AI security within the commercial sphere, despite the increasing deployment of AI in sensitive applications and the existence of "real threats."
• Proactive Defense Across the AI Lifecycle is Essential: Organizations must adopt comprehensive AI security frameworks, including secure data governance, adversarial robustness testing, continuous monitoring, and secure deployment practices, rather than relying solely on traditional cybersecurity.
• Collaboration Between Government and Industry is Crucial: Effective AI defense requires enhanced knowledge transfer, threat intelligence sharing, and partnerships between national security agencies and private sector AI developers.
• Policy and Regulation are Needed: Establishing clear policy frameworks, guidelines, and potentially mandatory security standards for AI systems will be vital to incentivize robust defenses and ensure a baseline level of protection across industries.

About the Speaker(s)

Harriet Farlow is a distinguished expert at the intersection of artificial intelligence and security, boasting over a decade of experience in the field. Her academic journey began with an undergraduate degree in physics, which ultimately led her into the domain of data science. She commenced her career specializing in defense projects in Australia, contributing to initiatives such as augmenting roles with robotic process automation in the Air Force and working on patrol boats in Darwin.

A significant portion of her career was spent at the Australian Signals Directorate (ASD), Australia's equivalent of the NSA. At ASD, Farlow served as a data scientist within their cybersecurity teams and rose to the position of acting technical director on one of their critical AI streams. In this capacity, she led technical collaborations on AI with Canada and the USA, gaining invaluable insights into national security applications and vulnerabilities of AI.

While at ASD, Farlow embarked on a PhD focusing on adversarial machine learning, specifically investigating methods to "hack models" and, as she mentioned, "hack facial recognition AI models." This research revealed a concerning lack of activity in AI security within the private sector despite the clear and present threats. Driven by this observation and her entrepreneurial spirit, she left her government role to found Mileva Security Labs, an AI security company dedicated to building technological products to address these critical vulnerabilities.

Beyond her professional and academic pursuits, Harriet Farlow maintains a public presence under the handle "@HarrietHacks" on platforms like YouTube and X (formerly Twitter), and also hosts a podcast, aiming to further engage with and educate the broader community on AI security topics. Her unique background, spanning defense, intelligence, advanced research, and entrepreneurship, positions her as a leading voice in the evolving landscape of AI security.