Securing AI: What We’ve Learned and What Comes Next

Securing AI: What We’ve Learned and What Comes Next

Speakers: Vasu Jakkal, Corporate Vice President, Microsoft Security

Conference: RSAC 2024

YouTube: https://www.youtube.com/watch?v=pQDmskN0qtg

Overview

In this compelling RSA Conference talk, Vasu Jakkal, Corporate Vice President of Microsoft Security, delves into the unprecedented rise of Artificial Intelligence (AI) and its profound implications for cybersecurity. The presentation, titled "Securing AI: What We’ve Learned and What Comes Next," highlights the dizzying pace of AI innovation and adoption, underscoring both its immense promise for societal good and the urgent, complex security challenges it introduces. Jakkal articulates a clear vision for how defenders must adapt to this new era, emphasizing that AI transformation cannot succeed without a corresponding security transformation.

The talk serves as a critical call to action for the cybersecurity community, positioning defenders as the "yes" for AI innovation – the foundational element that enables safe and trusted exploration of AI’s limitless possibilities. Jakkal details Microsoft's learnings and a practical three-pillar framework—Discover, Protect, and Govern—designed to help organizations manage AI risks comprehensively. This framework addresses the full spectrum of AI security, from identifying usage patterns and mapping risks to implementing robust technical controls and establishing ethical governance, providing a roadmap for securing one of the most consequential technologies of our time.

The urgency of the message is underscored by staggering statistics on AI adoption, such as ChatGPT reaching 100 million users in just two months, a feat that took mobile phones 16 years. Today, over a billion people globally use Large Language Models (LLMs) in some capacity, reflecting an acceleration of innovation that is "absolutely incredible." This rapid adoption, while promising immense benefits across healthcare, education, climate, and cybersecurity, simultaneously creates new attack surfaces and empowers adversaries with sophisticated tools, making the comprehensive security of AI not just a technical challenge but a societal imperative.

Background

▶ Watch: Introduction and the unprecedented pace of AI innovation (0:00)

The landscape of AI has transformed dramatically in recent years, evolving at a pace that has defied even optimistic predictions. Jakkal recounts that a hypothesis presented two years prior regarding the timeline for generative AI innovation in security had shrunk significantly, with what was expected to take three years materializing in just one. This acceleration is evident in the rapid adoption of generative AI technologies; for instance, ChatGPT garnered 100 million users in approximately two months, a stark contrast to the 16 years it took mobile phones to achieve the same milestone. Fast forward 18 months from ChatGPT's launch, and over a billion people worldwide are now using large language models in some form, signaling a technological shift of unparalleled magnitude.

This rapid proliferation of AI, while offering immense potential to solve some of the world's most pressing challenges—from faster cancer diagnoses and personalized education to climate monitoring and efficient transportation—also introduces a complex and unprecedented threat landscape. The cybersecurity domain, already grappling with persistent challenges, finds itself at a critical juncture. Jakkal highlights that identity-related attacks have increased tenfold year-over-year, and cybercrime has evolved into a global "gig economy" with a GDP equivalent to the world's third-largest country, growing at 15%. Compounding these issues are a severe talent shortage in cybersecurity and an increasingly complex regulatory environment, with Microsoft tracking over 250 regulatory updates daily.

The advent of generative AI exacerbates these existing threats while creating entirely new vectors for attack. Attackers can now leverage generative AI creatively to proliferate malware rapidly, generate new variants, conduct more intelligent password cracking with contextual understanding, and craft highly convincing phishing campaigns that exploit human curiosity. The rise of deepfakes, particularly voice synthesis, poses a significant concern; a mere three-second voice sample can train a GenAI model to mimic anyone, turning something as innocuous as a voicemail greeting into a potent tool for fraud and social engineering. Crucially, AI introduces entirely new attack surfaces alongside traditional ones (devices, identities, data, cloud, infrastructure). These include prompts, LLM models, AI data, and orchestration layers, demanding an expanded and holistic approach to cybersecurity.

Key Findings

▶ Watch: How AI empowers defenders in cybersecurity (4:25)

The talk reveals several critical findings regarding the dual nature of AI—its immense promise and its inherent risks. On the one hand, AI is depicted as a transformative force capable of elevating human potential across diverse sectors. In healthcare, it promises faster, more accurate diagnoses. In education, it offers personalized tutoring at scale, fostering a more equitable future. For climate change, AI can monitor endangered species, detect deforestation, and even predict natural disasters. Within cybersecurity, AI is already demonstrating "incredible things," such as reverse-engineering malware in seconds, shifting the balance in favor of defenders, and making natural language the most powerful coding tool for global collaboration. Data from AI users indicates significant improvements: organizations are becoming "faster, more productive, more accurate," and "better protected."

However, this transformative power comes with a significant security paradox. While 93% of organizations are already utilizing AI in some capacity, a staggering 99% of leaders feel ill-equipped to manage the associated risks. This disconnect highlights a critical gap between AI adoption and security preparedness. The speaker details how adversaries are exploiting generative AI to enhance their capabilities:

• Malware Proliferation: Rapid creation of new malware variants.
• Intelligent Password Cracking: Leveraging context for more effective attacks.
• Advanced Phishing: Preying on human curiosity with sophisticated, personalized lures.
• Deepfakes: Generating realistic voice samples from as little as three seconds, enabling highly convincing impersonation attacks.

Furthermore, AI introduces entirely new categories of attack surfaces that demand attention beyond traditional security perimeters. These include vulnerabilities within prompts (e.g., prompt injection), the LLM models themselves (e.g., model poisoning, jailbreak), the underlying AI data, and the orchestration layers that manage AI workflows. Recognizing these new attack vectors is paramount for developing effective defensive strategies, as traditional cybersecurity approaches alone are insufficient to secure the rapidly expanding AI ecosystem. The core takeaway is that while AI offers limitless possibilities, its secure and ethical deployment hinges on a comprehensive understanding and proactive mitigation of these emergent risks.

Technical Deep Dive

▶ Watch: Crucial questions about AI safety and current threats (6:00)

Microsoft's approach to securing AI is structured around a comprehensive three-pillar framework: Discover, Protect, and Govern. This framework is designed to provide organizations with a systematic method for managing the complex risks introduced by generative AI.

Discover

The first pillar, Discover, emphasizes the critical need for organizations to gain a clear understanding of their AI landscape. Jakkal notes that fear stemming from the unknown often paralyzes organizations, preventing them from harnessing AI's benefits. With 93% of organizations already using AI but only 1% of leaders feeling equipped to manage its risks, discovery is paramount. This phase involves:

1. Understanding AI Usage: Identifying which Generative AI (GenAI) applications are being used, how they are being leveraged, the extent of their use, who the users are, and the flow of data into and out of these applications. The goal is to create a detailed "blueprint" of AI adoption within the enterprise.
2. Mapping Risk to the Blueprint: Once the AI usage blueprint is established, organizations must map potential risks across three key categories:

• Application and Model Risk: This includes vulnerabilities inherent to the AI models and applications themselves, such as model poisoning (malicious data altering model behavior), prompt injections (manipulating LLMs through crafted inputs), jailbreak techniques (bypassing safety filters), and supply chain vulnerabilities within the AI development lifecycle.
• Data Risk: This category focuses on the sensitive data that fuels AI. Risks include data loss, insider risk (malicious or accidental data exposure by internal actors), and unintended use of data (data being used for purposes beyond its original intent or consent).
• Governance Risk: With a rapidly evolving regulatory landscape (e.g., EU AI Act, India's Digital Act), this risk category addresses compliance with new AI-specific regulations, internal codes of conduct, and organizational policies. Microsoft alone tracks over 250 regulatory updates daily, highlighting the complexity.

Protect

The second pillar, Protect, focuses on mitigating identified risks, preventing anticipated threats, and establishing robust guardrails for unforeseen challenges. This pillar mandates an expansion of existing cybersecurity measures to encompass AI-specific threats:

1. Application and Model Protection:

• Zero Trust Architecture: Jakkal advocates for extending the foundational principles of Zero Trust—verify explicitly, use least privilege access, and assume breach—to AI environments. This means meticulously verifying every request, ensuring users and AI systems only have the minimum necessary access, and designing defenses assuming a breach is inevitable.
• Expanded Threat Protection: Traditional threat protection and Extended Detection and Response (XDR) technologies must evolve to incorporate AI-specific Tactics, Techniques, and Procedures (TTPs) and Indicators of Compromise (IOCs). This involves conducting threat modeling tailored to AI risks.
• Posture Management: Integrating AI threats into existing Cloud Security Posture Management (CSPM) and Extended Security Posture Management (XSPM) solutions is crucial to continuously assess and improve the security posture of AI systems.
• Content Safety Controls: Implementing filters and controls to scrutinize LLM processing and outputs is vital. This ensures that harmful content or malicious content generated or processed by AI models is not served up or propagated.

1. Data Protection:

• Data Understanding and Classification: Given that data is the "fuel of AI," understanding where data resides, how it is used, and its sensitivity is paramount. Labeling and classification are identified as critical, especially for sensitive data.
• Auto-Classification: Due to the rapid generation and consumption of data by AI, auto-classification is highlighted as a key learning from Microsoft, enabling data to be categorized at the speed of its creation and use.
• Risk-Based Access Controls: Access to AI systems and the data they consume or generate must be dynamic, combining both user risk and data risk assessments to determine appropriate levels of access. This integrates into the Zero Trust architecture.
• Holistic Data Loss Prevention (DLP): Organizations need to revisit and expand their DLP policies to encompass AI data flows, ensuring sensitive information is not exfiltrated or misused by AI systems or through their outputs.

Govern

The third pillar, Govern, centers on establishing the ethical and responsible deployment and use of AI, emphasizing human agency and control. Governance is seen as integral, drawing lessons from past technological and industrial revolutions to ensure AI is built, deployed, and used safely.

1. Regulatory Compliance: Organizations must proactively address the influx of new AI regulations globally. The EU AI Act, with its risk-based approach, is lauded as a model, allowing organizations to prevent access to high-risk apps while implementing tailored controls for lower-risk applications. Tools must integrate compliance checks in an automated way to manage complexity.
2. Policy and Code of Conduct Enforcement: This involves defining clear internal policies for AI use and mechanisms to detect and address AI misuse that violates organizational codes of conduct.
3. Content Safety Filters: Reinforcing content safety, these filters are crucial for ensuring that AI outputs align with ethical guidelines and do not produce harmful or inappropriate content.
4. User Education: This is identified as a critical, overarching component across all pillars. It involves educating users on how AI models work, the data they use, how to evolve threat protection for AI, how to create and label content responsibly, and which tools to use. Educated users are the first line of defense and critical enablers of safe AI adoption.

Together, these three pillars form a robust framework for organizations to navigate the opportunities and challenges of AI securely and responsibly, transforming security to meet the demands of the AI era.

Demo / Proof of Concept

▶ Watch: Identifying new generative AI attack surfaces (8:40)

The talk provided a high-level strategic overview of AI security, focusing on Microsoft's framework and learnings. While the speaker referenced a slide shown two years prior about their hypothesis on AI innovation, the presentation did not include a live demonstration or a detailed description of a specific technical proof of concept. The content was primarily analytical and prescriptive, outlining strategies and principles for securing AI rather than showcasing a particular tool or exploit.

Defensive Implications

▶ Watch: The need for comprehensive AI protection (9:00)

The rapid ascent of AI and its dual-use nature fundamentally reshapes the defensive landscape, demanding an immediate and comprehensive recalibration of cybersecurity strategies. Defenders must recognize that AI is not merely a new tool but a paradigm shift that requires a security transformation to match its potential.

Firstly, defenders should actively embrace AI for defense. Generative AI offers "superpowers" to security teams, enabling them to defend at machine speed and scale, significantly reducing the barriers to entry for new defenders. This means leveraging AI for tasks like rapid malware reverse engineering, intelligent threat detection, and automating response workflows. Organizations should invest in AI-powered security tools and integrate them into their existing Security Operations Center (SOC) infrastructure.

Secondly, a proactive and structured approach to risk management is non-negotiable. Organizations must initiate a detailed Discover phase to understand all AI usage within their environment—from GenAI applications to data flows and user access. This blueprint must then be meticulously mapped to the three categories of AI risk: application and model risk (e.g., prompt injection, model poisoning), data risk (e.g., data loss, insider threat), and governance risk (e.g., regulatory non-compliance). This comprehensive discovery prevents "fear and anxiety" from hindering AI adoption.

Thirdly, the Protect pillar dictates a significant evolution of existing security controls.

• Zero Trust Architecture must be expanded to explicitly cover AI systems, ensuring explicit verification, least privilege access for AI components and users, and an assumption of breach within the AI ecosystem.
• Threat protection mechanisms, including XDR technologies, need to be updated to detect and respond to AI-specific TTPs and IOCs. This requires continuous threat modeling focused on AI attack vectors.
• Security posture management solutions (CSPM, XSPM) must integrate AI risks, providing visibility and control over the security configurations of AI models, data pipelines, and related infrastructure.
• Crucially, data security must be elevated. Defenders need to implement robust data labeling and classification strategies, with an emphasis on auto-classification to keep pace with AI's data generation speed. Risk-based access controls that combine user and data context are essential, and Data Loss Prevention (DLP) policies must be holistically re-evaluated and extended to cover AI-driven data flows.
• Finally, content safety controls are vital for LLM outputs, acting as filters against harmful or malicious content generated by AI.

Fourthly, the Govern pillar underscores the importance of a robust ethical and compliance framework. Defenders must stay abreast of the rapidly evolving regulatory landscape (e.g., EU AI Act) and ensure that AI deployments comply with both external regulations and internal codes of conduct. Automated tools for compliance management will be critical.

Lastly, user education is highlighted as a foundational defensive measure. All stakeholders, from developers to end-users, must be educated on AI risks, safe interaction practices, responsible content creation, and the importance of data labeling. This empowers the human element, which remains crucial in navigating the complexities of AI security. By integrating these defensive implications, organizations can build a secure foundation for AI, transforming security to enable, rather than impede, innovation.

Key Takeaways

• Unprecedented AI Adoption Demands Urgent Security Focus: Generative AI's adoption rate is unparalleled, with over a billion LLM users in 18 months. This rapid pace necessitates immediate and comprehensive security strategies to manage the associated risks effectively.
• New Attack Surfaces and Enhanced Adversary Capabilities: AI introduces novel attack surfaces like prompts, LLM models, and AI data while simultaneously empowering attackers with advanced tools for malware proliferation, intelligent phishing (including deepfakes from 3-second voice samples), and more sophisticated cybercrime.
• Microsoft's Three-Pillar Framework for AI Security: Organizations should adopt a structured approach based on Discover (understand AI usage and map risks), Protect (mitigate risks with adapted controls), and Govern (establish ethical and compliant use) to manage AI risks holistically.
• Evolution of Core Security Principles is Essential: Foundational security concepts such as Zero Trust Architecture, XDR technologies, and security posture management must be expanded and adapted to incorporate AI-specific threats, TTPs, and IOCs.
• Data Security is Paramount for AI: Given that data fuels AI, critical focus areas include robust data labeling and classification (especially auto-classification), risk-based access controls combining user and data risk, and comprehensive Data Loss Prevention (DLP) policies tailored for AI data flows.
• Governance and User Education are Foundational: Navigating the complex regulatory environment (e.g., EU AI Act) and enforcing internal policies are crucial. Simultaneously, continuous user education on AI models, risks, safe interaction, and responsible content handling is vital for embedding security into the AI lifecycle.

About the Speaker(s)

Vasu Jakkal is the Corporate Vice President of Microsoft Security, a role that places her at the forefront of securing one of the world's largest technology ecosystems. Throughout her talk, Jakkal demonstrates a deep passion for technology and its potential to improve lives, positioning herself as an "AI optimist" who believes the age of AI will unlock massive human potential and help build a "more equitable world." A self-proclaimed "hardcore Trekkie" who grew up with Star Trek, Jakkal frequently weaves in themes of limitless possibilities and shared responsibility, echoing the sentiment of "with great power comes great responsibility." She emphasizes the critical role of cybersecurity defenders, calling them the "heart of an organization's trust in AI" and the "yes" that enables safe innovation. Her presentation reflects a commitment to fostering community, belonging, and continuous learning within the cybersecurity domain.