AI Safety and You: Perspectives on Evolving Risks and Impacts

AI Safety and You: Perspectives on Evolving Risks and Impacts

Speakers: Nathan Hammel (Senior Director of Research, Kadelski Security, and Black Hat AI/ML/Data Science Track Lead), Nikki Pope (Head of AI and Legal Ethics, Nvidia), Michael (Research Scientist, Google Deep Mind), Amanda Minnick (AI Red Team, Microsoft)

Conference: Black Hat USA

YouTube: https://www.youtube.com/watch?v=s9cLeHR8CBA

Overview

This Black Hat USA session, "AI Safety and You: Perspectives on Evolving Risks and Impacts," delves into the critical and often misunderstood domain of AI safety from the perspective of cybersecurity professionals. Nathan Hammel, the Black Hat AI/ML/Data Science track lead, initiates the discussion by challenging the common misconception that AI safety is solely concerned with speculative "existential risk" scenarios involving AI destroying humanity. Instead, the panel reframes AI safety as a pragmatic concern for "safe to use" AI, directly aligning it with core cybersecurity principles like security, privacy, alignment, and reliability (SPAR). The talk emphasizes that the skills inherent to the security community are uniquely positioned to address the burgeoning challenges of AI safety, which are rapidly creeping into the security domain.

The panel features a diverse group of experts: Nikki Pope, a lawyer leading AI and Legal Ethics at Nvidia; Michael, a research scientist leading Google DeepMind's AI red team; and Amanda Minnick, an operator and researcher on Microsoft's AI Red team. Together, they dissect the real-world implications of unsafe AI, ranging from biased algorithms impacting individuals' access to financial services or healthcare, to the limitations of current mitigation strategies like guardrails and post-training alignment. The discussion highlights the urgent need for proactive, integrated safety measures throughout the AI development lifecycle, stressing that the cybersecurity community's expertise in threat modeling, red teaming, and risk management is indispensable for building trustworthy AI and navigating the complex regulatory landscape, exemplified by initiatives like the EU AI Act.

Background

▶ Watch: Introduction to AI safety and SPaR framework (0:00)

The conversation around AI safety has long been plagued by a public relations issue, as highlighted by Nathan Hammel. The term often conjures images of apocalyptic "Skynet" or "Terminator" scenarios, commonly referred to as existential risk or "P Doom" (probability of doom). While these speculative fears capture significant media attention, they inadvertently lead many to dismiss AI safety as a niche concern irrelevant to immediate, tangible problems. This narrow focus on extreme, hypothetical risks overshadows the very real and present dangers that AI systems pose in everyday life, thereby hindering broader engagement from the security community.

The panel redefines AI safety in a more actionable and immediate context: ensuring AI is safe to use. For personal AI applications, this involves a multi-faceted approach encapsulated by the acronym SPAR: Security, Privacy, Alignment, and Reliability. Each component is crucial for building AI systems that are not only robust against malicious attacks but also respectful of user data, consistent with human values and intentions, and perform as expected without errors or biases.

Prior to the generative AI boom, exemplified by the public release of ChatGPT in late 2022, the landscape of AI safety was already evolving. However, the rapid proliferation of large language models (LLMs) and generative AI applications like Microsoft's Co-pilot dramatically accelerated the urgency and complexity of these issues. Regulators worldwide, including the European Union with its impending EU AI Act, quickly moved to establish frameworks for AI governance and accountability. This regulatory push underscores the growing recognition that AI safety is not merely a technical challenge but also a legal and ethical imperative, requiring organizations to integrate safety requirements into their product development processes and establish robust systems for compliance. The challenge lies in translating abstract ethical principles and legal mandates into concrete, technical safeguards that can be effectively implemented and verified.

Key Findings

▶ Watch: Panelist introductions and expertise (2:25)

The panel's discussion reveals several critical findings regarding the current state and future direction of AI safety:

Firstly, the prevalent public-facing AI safety controls such as guardrails and post-training alignment are deemed largely insufficient on their own. Amanda Minnick points out that while these measures are widely used and do offer some level of protection, they are "easily circumvented." Classifiers designed to enforce guardrails can quickly become "stale," and the benefits of post-training alignment — intended to make models safer — can be "undone by fine-tuning" if developers or users have quick and inexpensive access to modify the model's parameters. This highlights a fundamental fragility in current reactive safety mechanisms.

Secondly, the panel reframes existential risk from a speculative, humanity-ending event to immediate, real-world harms affecting individuals. Nikki Pope vividly illustrates this by describing scenarios where biased algorithms in financial services could deny a single mother a mortgage loan, or in healthcare, could lead to incorrect recommendations for surgical procedures or medications, potentially resulting in death. For the affected individual, these constitute an "existential risk" to their livelihood, well-being, or even life. This perspective grounds AI safety in tangible, human-centric impacts, making it a more relatable and urgent concern for security professionals.

Thirdly, the importance of proactive testing and the integration of AI safety into the entire product development lifecycle is emphasized. Amanda Minnick notes a significant shift at Microsoft following President Biden's executive order, which mandated independent red teaming for high-risk or high-impact generative AI applications. This order transformed red teaming from a post-deployment advisory role into an integral part of the "ship process," providing data that directly informs "ship no ship" decisions. This integration gives "teeth" to safety concerns, allowing red teams to effectively block the release of products with unacceptable risks.

Finally, the panel identifies a crucial tension between utility and safety in AI development. Michael from Google DeepMind highlights that this is a problem familiar to the cybersecurity community, where balancing functionality with security is a constant challenge. This suggests that existing security paradigms, which involve trade-offs and risk management, are highly relevant to AI safety. The call is for the security community to leverage its long-standing experience in navigating this tension to contribute innovative solutions to AI safety.

Technical Deep Dive

▶ Watch: Addressing AI regulatory compliance, including EU AI Act (3:00)

The technical discussion centers on the practical implementation of AI safety, particularly through advanced red teaming methodologies and robust risk management frameworks. Amanda Minnick, from Microsoft's AI Red Team, provides insight into their operational model. Her team's day-to-day focus is on "attacking our Gen AI models and our AI applications like Co-pilot" to ensure they are "safer and more secure for the end user." This involves actively probing models for vulnerabilities that could lead to undesirable or harmful outputs.

The limitations of common AI safety controls are a key technical point. Guardrails, typically implemented as classifiers or filters, are designed to prevent models from generating prohibited content or performing disallowed actions. However, as Minnick explains, these classifiers can quickly grow stale, meaning they fail to adapt to new adversarial techniques or evolving linguistic patterns. Furthermore, post-training alignment, a process often involving techniques like Reinforcement Learning from Human Feedback (RLHF) to align model behavior with human values, can be easily undermined. If fine-tuning access is permitted to the model, these safety alignments can be "undone... very quickly and inexpensively," potentially reintroducing biases or vulnerabilities that the alignment process sought to remove. This highlights a fundamental challenge: ensuring the persistence of safety properties across the model lifecycle, especially when models are continuously updated or adapted.

Michael from Google DeepMind advocates for foundational cybersecurity practices within the AI domain, specifically threat modeling and risk modeling. He suggests that organizations should, at a minimum, establish mechanisms for these activities to "ground that understanding through some form of like threat and risk modeling." This allows organizations to identify potential attack vectors, assess the likelihood and impact of various risks, and allocate resources effectively, preventing a reactive approach to "hypothetical problems." Michael specifically endorses the MITRE ATLAS framework, which stands for "Adversarial Threat Landscape for Artificial-Intelligence Systems." ATLAS is akin to the well-known MITRE ATT&CK framework but is tailored to AI/ML systems, providing a comprehensive knowledge base of adversary tactics and techniques used against AI. This framework helps security professionals systematically analyze and understand the threats to their AI deployments, from data poisoning and model evasion to prompt injection and exfiltration of sensitive training data.

The panel also touches on the concept of trustworthy AI, which Nvidia's Nikki Pope is responsible for integrating into product development. This goes beyond mere security to encompass ethical considerations and regulatory compliance, such as those mandated by the EU AI Act. Implementing trustworthy AI involves establishing AI governance systems that ensure transparency, fairness, accountability, and robustness throughout the AI lifecycle. Technically, this translates to rigorous testing for bias, explainability mechanisms, data provenance tracking, and robust model validation processes. Pope's role as a lawyer in this technical space underscores the interdisciplinary nature of AI safety, where legal and ethical requirements must be translated into engineering specifications and technical controls.

Finally, Amanda Minnick's prior work at Twitter, where she used unsupervised methods to identify and action information operations, election interference, and misinformation (e.g., during COVID-19 related to QAnon), demonstrates a practical application of AI/ML techniques to address large-scale trust and safety challenges. While not directly a defensive measure for AI systems, it illustrates the broader context of how AI can be both a tool and a target in complex societal issues, and how data science expertise is crucial for identifying and mitigating harms.

Demo / Proof of Concept

▶ Watch: Microsoft AI Red Team's practical safety approach (4:10)

While a live, interactive demonstration was not performed during the panel, Amanda Minnick described a practical and scalable approach to AI red teaming that functions as an internal proof-of-concept for assessing AI safety. When Microsoft's AI Red Team faced the challenge of insufficient personnel to thoroughly test new generative AI applications like Bing Co-pilot during its early development, they implemented a "very tactical" solution: leveraging volunteers from across the company.

This approach involved:

1. Volunteer Recruitment: Gathering individuals from various departments within Microsoft.
2. Training and Guidance: Providing a "little bit of training" and "examples" to equip volunteers with the basic knowledge of how to probe AI models for vulnerabilities. This likely included instruction on prompt injection, jailbreaking, and other adversarial techniques relevant to LLMs.
3. Manual Probing: Volunteers engaged in manual probing of the AI models. This hands-on testing allowed for a diverse range of inputs and interaction styles, mimicking real-world user behavior and potential adversarial attempts.
4. Time-Boxing: The testing efforts were "time boxed," ensuring focused and efficient use of volunteer resources.
5. Results Collection and Sharing: The outcomes, including identified risks and their potential impact, were "collected... and shared widely." This not only provided crucial data for product development decisions but also ensured that participants "felt that it was a valuable use of their time," fostering a culture of shared responsibility for AI safety.

This method, though not a traditional live "demo," serves as a compelling proof-of-concept for how organizations, even those with limited dedicated AI safety resources, can initiate and scale effective AI red teaming efforts. It demonstrates a practical, crowdsourced approach to identifying weaknesses in AI models before deployment, providing actionable data to inform "ship no ship" decisions and enhance the overall safety and security of AI applications.

Defensive Implications

▶ Watch: Reframing existential AI risk to real-world impacts (5:00)

The panel offers several crucial defensive implications for organizations grappling with AI safety and security:

1. Integrate Safety into the Product Development Lifecycle: Nikki Pope emphasizes the need for companies like Nvidia to integrate AI safety requirements directly into their product development processes from the outset. This means moving beyond reactive measures to a proactive approach where safety, privacy, alignment, and reliability (SPAR) are considered fundamental design principles, not afterthoughts. This includes establishing robust AI governance systems to ensure compliance with emerging regulations like the EU AI Act.

1. Adopt Proactive AI Red Teaming: Amanda Minnick's experience at Microsoft highlights the transformative power of proactive red teaming. Organizations should not wait until models are deployed to test for vulnerabilities. Instead, AI red teams should be integrated into the "ship process," providing data that influences "ship no ship" decisions. For organizations starting from scratch, Minnick suggests a tactical approach: recruit internal volunteers, provide basic training on manual probing techniques (e.g., prompt injection, adversarial attacks), time-box their efforts, and widely share the findings to build collective ownership and improve models.

1. Leverage Established Threat and Risk Modeling Frameworks: Michael advocates for grounding AI safety efforts in familiar cybersecurity practices, specifically threat modeling and risk modeling. He recommends that organizations implement mechanisms to systematically identify, assess, and prioritize AI-specific risks. The MITRE ATLAS framework is highlighted as an excellent starting point, providing a structured approach to understanding adversarial tactics and techniques against AI systems. This allows organizations to allocate resources efficiently and address the most impactful threats rather than chasing hypothetical problems.

1. Understand the Limitations of Current Controls: Defenders must recognize that common guardrails and post-training alignment techniques are not foolproof. Guardrails can become stale, and alignment efforts can be undone by fine-tuning. This necessitates a multi-layered defense strategy that includes continuous monitoring, adaptive controls, and a deeper understanding of the model's underlying vulnerabilities.

1. Address Bias and Fairness as Security Concerns: Nikki Pope's examples of biased algorithms in finance and healthcare underscore that fairness and ethical considerations are not merely "soft" issues but direct security risks with "existential" consequences for individuals. Defenders must integrate testing for algorithmic bias into their security assessments to prevent discriminatory outcomes and ensure equitable access and treatment.

1. Balance Utility and Safety with Privacy: Michael points out the inherent tension between utility and safety, a common challenge in cybersecurity. Nikki Pope adds the critical dimension of privacy, noting the conflict between monitoring systems for anomalies (a security imperative) and protecting user privacy. Defenders need to develop sophisticated strategies that balance these competing demands, perhaps by implementing privacy-preserving analytics or differential privacy techniques, and educating stakeholders about appropriate testing and monitoring practices.

1. Invest in Data Science and Data Engineering Principles: Amanda Minnick stresses the importance of learning from the lessons of data science, data engineering, and statistics. This implies a need for robust data governance, secure data pipelines, and meticulous data validation processes to ensure the integrity and quality of training data, which is foundational to building safe and secure AI models.

In essence, the defensive posture against AI risks requires a holistic, interdisciplinary approach that combines traditional cybersecurity rigor with specialized AI/ML understanding, ethical considerations, and a proactive, continuous testing mindset.

Key Takeaways

• AI Safety is Practical, Not Just Existential: Move beyond "Skynet" fears to focus on real-world harms caused by biased algorithms in areas like finance and healthcare, which can have "existential" impacts on individuals.
• Current Safety Controls are Insufficient: Public-facing guardrails and post-training alignment are easily circumvented and can be undone by fine-tuning, necessitating more robust and integrated safety measures.
• Proactive Red Teaming is Essential: Integrate AI red teaming into the product development lifecycle to identify and mitigate risks before deployment, influencing "ship no ship" decisions. Even small organizations can start with volunteer-based manual probing.
• Leverage Established Security Practices: Apply familiar cybersecurity concepts like threat modeling and risk modeling to AI systems. Frameworks like MITRE ATLAS provide a structured approach to understanding AI-specific threats.
• Balance Utility, Safety, and Privacy: Recognize the inherent tension between these factors and develop strategies that optimize for all three, ensuring secure, effective, and privacy-respecting AI systems.
• Interdisciplinary Approach is Crucial: AI safety requires collaboration between engineers, security professionals, lawyers, and ethicists to build trustworthy AI that complies with regulations like the EU AI Act and addresses societal impacts.

About the Speaker(s)

Nathan Hammel is the Senior Director of Research at Kadelski Security, where he leads the fundamental and applied research team, focusing on the security of emerging technologies, including AI. He is also the AI, Machine Learning, and Data Science track lead for Black Hat, having served on the review board for many years.

Nikki Pope is the Head of AI and Legal Ethics at Nvidia. As a lawyer, not an engineer (though she humorously notes taking a Fortran course in high school), she is responsible for integrating requirements into Nvidia's product development process to meet objectives for trustworthy AI and comply with pending regulations, such as the EU AI Act.

Michael is a Research Scientist at Google DeepMind. He helps lead their AI red team, with a primary focus on security and privacy as they pertain to AI systems.

Amanda Minnick is on the AI Red Team at Microsoft, where she has worked as an operator and researcher for three years. Her day-to-day role involves attacking generative AI models and applications like Co-pilot to enhance their safety and security for end-users. Prior to Microsoft, she worked at Twitter, focusing on trust and safety, information operations, election interference, and misinformation, often applying unsupervised methods to address these issues at scale. Her background includes a PhD in Computer Science and Machine Learning.