Securing New Limits: Protecting the Pathway for AI Innovation

Securing New Limits: Protecting the Pathway for AI Innovation

Speakers: Kevin Skapinetz, Vice President of Security Strategy, IBM; Rosa Bolger, Vice President and Distinguished Engineer, IBM

Conference: RSAC 2024

YouTube: https://www.youtube.com/watch?v=om8AAVFwe_8

Overview

In this compelling keynote at RSAC 2024, Kevin Skapinetz, Vice President of Security Strategy at IBM, joined by Rosa Bolger, Vice President and Distinguished Engineer at IBM, addressed the profound and dual impact of artificial intelligence (AI), particularly Generative AI (GenAI), on the cybersecurity landscape. The talk underscored the critical juncture the industry faces: AI represents an unprecedented force multiplier capable of extending human limits and enhancing productivity, yet it simultaneously introduces a vast, complex, and largely unsecured new attack surface. The central theme revolved around the urgent imperative for the security community to proactively build security into AI and its burgeoning ecosystem, rather than attempting to retrofit existing defenses.

Skapinetz framed the discussion by highlighting the pervasive issue of burnout among security professionals, with 60% reporting burnout and 83% linking it to increased data breaches. He posited AI as a potential antidote to this chronic problem, offering the ability to automate mundane tasks and free up human resources for higher-value work. However, this promise is shadowed by the rapid expansion of AI adoption in businesses, which are increasingly training models on highly sensitive intellectual property and creating new forms of valuable data, all while remaining woefully unprepared to secure these advancements. The speakers presented IBM's new framework for securing GenAI, emphasizing the need to protect data, models, and usage across the entire AI pipeline, and advocating for a proactive, governance-centric approach to navigate this evolving frontier.

The talk served as a clarion call for the cybersecurity industry to recognize AI not merely as a tool, but as the "next new platform to secure." It outlined the unique threats posed by AI, such as prompt injection and the exfiltration of model weights, and proposed actionable strategies for defenders. By leveraging AI to enhance defensive capabilities and establishing robust security measures around its development and deployment, the industry can harness AI's transformative power responsibly, moving from a state of constant overload to one where security teams are empowered and more effective.

Background

▶ Watch: Introduction and the pervasive security team burnout challenge (0:00)

The talk began by drawing a stark picture of the challenges plaguing the cybersecurity industry, particularly the pervasive issue of burnout among security professionals. Skapinetz cited alarming statistics: 60% of security professionals report experiencing burnout, and a staggering 83% believe this burnout directly contributes to more data breaches and security challenges. This chronic stress, lack of control, long hours, and limited resources create an asymmetric battle where a finite number of defenders face an "infinite number of threats." This context sets the stage for AI as a potential solution to augment human capabilities and alleviate this burden.

Skapinetz then drew parallels between human physical limitations—such as attention span, sleep deprivation, and memory capacity—and how humanity has historically overcome these through science, technology, and medicine. AI, he argued, represents the next great advancement, a force multiplier poised to reimagine these human limits. This has two major implications for security: it will transform how security teams work daily, and it will transform what needs protection.

The historical evolution of cybersecurity defenses further underscored the need for a paradigm shift. Skapinetz provided examples illustrating that "last era's security technology" is often ineffective against "next era's challenges." Firewalls, designed for networks, proved insufficient for the web era, necessitating Layer 7 defenses. Virtual appliances struggled with cloud environments, leading to the development of cloud-native defenses. Similarly, traditional antivirus solutions were outmatched by Advanced Persistent Threats (APTs), paving the way for behavioral-based defenses. The clear message: in the world of AI, we need new AI-powered and AI-intelligent defenses to keep pace.

While the emergence of generative AI has brought AI into the spotlight, Skapinetz noted that AI in cyber is not entirely new. IBM, for instance, has been leveraging AI for years, albeit often "behind the scenes." Examples include automating approximately 85% of security alerts for customers and reducing triage times by 55% in the first year of implementation, thereby freeing up human resources for higher-value tasks. This "invisible" progress, however, contributed to skepticism within the security community regarding AI's tangible benefits. The recent emergence of highly visible generative AI has fundamentally changed this perception, making AI a direct augmenter of human decision-making and productivity, enabling analysts to quickly understand complex security events and automatically generate reports, content, queries, and policies. This shift underscores the growing conviction that future security professionals who do not leverage AI daily may struggle to remain effective.

However, the talk also pivoted to the second, more concerning implication: AI fundamentally changes what needs protecting. Companies are now training AI models on critical intellectual property, product designs, internal codebases, and engineering logs—highly proprietary datasets previously "tucked away behind our highest security systems." These crown jewels are now moving to the surface, being hooked up to models, or even encoded within models that "could fit on a thumb drive." Furthermore, AI is uniquely capable of creating new crown jewels, as prompts and responses become a "printing press of high-value data," massively expanding the scope of what needs protection. This creates a new mission for the security community: to secure AI as the "next new platform," much like networks, servers, cloud, and workloads before it. The challenge is stark, as a joint IBM and AWS study revealed that while 82% of business leaders consider secure and trustworthy AI essential, only 24% have secured or are actively working on securing their existing GenAI projects. This widespread unpreparedness forms the urgent backdrop for the strategies and frameworks presented in the talk.

Key Findings

▶ Watch: Why new AI-powered defenses are essential for the AI era (4:40)

The talk highlighted several critical findings regarding the intersection of AI and cybersecurity, shaping the current and future landscape for defenders:

• Dual Transformation: AI is fundamentally transforming two core aspects of cybersecurity: how security teams operate, by augmenting human capabilities and boosting productivity, and what security teams need to protect, by creating new forms of valuable data and expanding the attack surface.
• Widespread Unpreparedness: Despite a high awareness of the importance of AI security, there is a significant gap in implementation. An IBM and AWS study found that 82% of business leaders believe secure and trustworthy AI is essential for business success, yet only 24% have actually secured or are actively working on securing their existing GenAI projects. This indicates a critical lack of preparedness across industries.
• New Crown Jewels and Expanded Attack Surface: AI models are being trained on vast amounts of highly sensitive, proprietary data, including intellectual property, product designs, and internal codebases. This data, previously well-protected, is now more exposed within AI systems. Furthermore, AI itself generates new high-value data through prompts and responses, effectively creating "new crown jewels" that must be secured, massively expanding the scope of protection.
• Unique AI-Specific Threats: The talk identified novel attack vectors inherent to AI systems. These include model evasion (manipulating model inputs to produce incorrect outputs) and prompt injection attacks (crafting malicious inputs to make the model reveal sensitive information or perform unintended actions). Beyond these, the exfiltration or theft of model weights was highlighted as a critical, emerging threat, as these weights represent the "brains" and core intellectual property of a trained model.
• Supply Chain Vulnerabilities: The increasing reliance on external AI platforms, open-source models, and codebases introduces new avenues for supply chain attacks. Malicious behavior or vulnerabilities can be introduced at various stages of the AI development pipeline, impacting the integrity and security of the deployed AI.
• The "Shadow AI" Problem: Organizations face challenges in discovering and managing unknown and unmanaged AI usage within their environments. This Shadow AI can introduce significant new risks and vulnerabilities that bypass established security controls.
• A New Security Framework: IBM introduced a new framework for securing GenAI, designed to prioritize defensive approaches against the likeliest attacks and risks. This framework simplifies the complex problem by focusing on three core areas: securing the data, the models, and their usage across the entire AI pipeline, while also emphasizing a secure underlying infrastructure and robust governance.
• AI as a Defensive Force Multiplier: Beyond the threats, the talk underscored AI's potential to significantly enhance defensive capabilities. Examples include automating threat analysis and vulnerability detection, summarizing complex security events for faster response, augmenting data for incident investigation, automating threat hunts, and even developing capabilities for finding and fixing software vulnerabilities, moving towards "self-healing products."

Technical Deep Dive

▶ Watch: Security professionals must use AI or risk obsolescence (5:50)

The technical deep dive of the talk centers on understanding the AI pipeline and the specific security challenges and recommendations associated with each stage. IBM's framework breaks down the AI lifecycle into three main phases, emphasizing that security must be integrated end-to-end.

1. Data Collection and Handling Phase:

This initial stage involves gathering the vast quantities of data required to train and tune AI models. As Rosa Bolger highlighted, petabytes of data are often needed for tasks like converting COBOL to Java, much of which constitutes confidential business information. Historically, such data resided in "cocoons," well-protected within internal networks. With AI, this sensitive data is now moving to the surface, being ingested, processed, and eventually embedded within the models themselves.

• Threats: The primary concern here is data exfiltration. Attackers may target data sources during collection or during the preprocessing stages to steal sensitive information before it's even incorporated into the model. The sheer volume and proprietary nature of this data make it an extremely attractive target.
• Defensive Focus: Robust data security practices are paramount, including stringent access controls, encryption at rest and in transit, data loss prevention (DLP) solutions, and careful data anonymization or de-identification where possible.

2. Model Development and Training Phase:

In this central phase, the AI models are built, trained, and fine-tuned using the collected data. This is where the core logic and intelligence of the AI are formed.

• Threats:
• Supply Chain Attacks: The use of numerous new AI platforms, open-source libraries, and codebases introduces significant supply chain risks. Malicious behavior or vulnerabilities can be injected into the model training process, or even directly into the foundational models themselves, leading to compromised AI systems.
• Model Poisoning/Manipulation: Attackers can attempt to inject malicious data into the training set, subtly altering the model's behavior or introducing backdoors that can be exploited later.
• Intellectual Property Theft (Model Weights): Rosa Bolger specifically emphasized the critical importance of protecting model weights. These weights are the numerical parameters within a neural network that determine the strength and direction of connections between neurons. They are, in essence, the "brains of the model" and encapsulate the learned intelligence and, often, the embedded confidential training data. Their exfiltration or theft represents a significant loss of intellectual property.
• Defensive Focus: Secure development lifecycle (SDL) practices for AI models, rigorous vetting of third-party components and open-source models, and robust version control. Protecting model weights requires specialized techniques, potentially including encryption of model files, secure storage, and strict access controls over the model artifacts themselves.

3. Inferencing and Live Use Phase:

This final phase involves the deployment of the trained AI model for real-world applications, where users interact with it and it generates responses or performs tasks.

• Threats:
• Prompt Injection Attacks: This is a novel and significant threat unique to generative AI. Attackers craft malicious inputs (prompts) to manipulate the model into generating misleading responses, revealing sensitive data it shouldn't, or performing actions outside its intended scope. For example, a model trained on internal documents could be prompted to reveal confidential information.
• Model Evasion Attacks: Adversaries create carefully crafted inputs that cause the model to misclassify or fail to detect malicious activity, allowing attacks to bypass AI-powered security systems.
• Data Exfiltration from Outputs: While the model is generating responses, sensitive data could inadvertently be exposed through its outputs, particularly if the model was over-trained on confidential information or if prompt injection is successful.
• Access Control and API Security: Many AI models are exposed via APIs, making them vulnerable to traditional API security flaws and unauthorized access if not properly secured. Rosa Bolger highlighted concerns about access controls and application security for protecting the execution of models.
• Defensive Focus: Implementing input validation and sanitization for prompts, deploying AI firewalls or guardrails to filter malicious inputs and outputs, and continuous monitoring of model behavior for anomalies. Robust API security, including authentication, authorization, and rate limiting, is crucial. The talk also mentioned the combination of Watson X (IBM's data and AI platform focusing on governance) with Guardium (IBM's data security platform) as a potential solution for securing this space, enabling comprehensive data governance and protection across AI environments.

Underlying Infrastructure and Governance:

Beyond the pipeline stages, the talk emphasized two overarching principles:

• Secure Foundation: Protecting the underlying infrastructure where AI models are developed and deployed is a non-negotiable starting point. This includes securing cloud environments, containers, virtual machines, and network components.
• Governance Throughout: Uniquely in the AI space, governance must be considered throughout the entire lifecycle. This involves establishing policies, controls, and accountability for AI development, deployment, and usage to ensure ethical, responsible, and secure AI adoption. This includes controlling how employees use AI, preventing the inadvertent entry of confidential data into public models, and addressing Shadow AI.

The detailed technical concerns raised by Rosa Bolger, particularly the protection of model weights and the challenges of securing petabytes of confidential training data, underscore the complexity and novelty of securing AI. These elements move beyond traditional cybersecurity concerns, demanding a new class of specialized defenses.

Demo / Proof of Concept

▶ Watch: AI creates new 'crown jewels' requiring protection (6:15)

The keynote presentation did not include a live technical demonstration or a detailed proof of concept. Instead, it focused on outlining the strategic challenges, introducing a conceptual framework for securing generative AI, and discussing IBM's ongoing initiatives and tools in this evolving space. The speakers primarily relied on statistics, conceptual diagrams of the AI pipeline, and expert insights to convey their message.

Defensive Implications

▶ Watch: We are unprepared: IBM/AWS study on generative AI security (7:40)

The insights shared in "Securing New Limits" carry profound defensive implications for organizations grappling with the rapid adoption of AI. Defenders must pivot their strategies to account for AI's unique attack surface and leverage its capabilities to enhance their own defenses.

1. Adopt an AI-Centric Security Framework: Organizations should immediately adopt a structured framework for securing AI, such as the one proposed by IBM. This involves understanding the entire AI pipeline—from data collection and model training to inferencing and live use—and implementing targeted security controls at each stage. A holistic approach that secures the data, the models, and their usage is paramount.

1. Elevate Data Security for AI Training and Inference: The sheer volume and sensitivity of data used to train AI models (e.g., critical intellectual property, proprietary codebases) demand enhanced data protection. Defenders must implement robust data loss prevention (DLP), encryption, and strict access controls for AI training datasets. Furthermore, they must ensure that sensitive data is not inadvertently exposed through model outputs or by successful prompt injection attacks during inference. Tools like IBM Watson X for data governance and Guardium for data security can be integrated to manage and protect these vast datasets.

1. Prioritize Model Integrity and Protection: Models themselves are now intellectual property and critical assets. Defenders must focus on:

• Protecting Model Weights: Given that model weights are the "brains of the model," organizations must develop specific strategies to prevent their exfiltration or theft. This could involve stringent access controls on model repositories, encryption of model files, and monitoring for unusual access patterns.
• Secure Model Execution: Implement robust access controls, application security, and API security for deployed AI models, treating them as critical applications. This includes ensuring proper authentication, authorization, and vulnerability management for model APIs.
• Mitigating Supply Chain Risks: Vet all AI platforms, open-source models, and third-party components used in the AI development lifecycle to prevent the introduction of vulnerabilities or malicious code during model development and training.

1. Defend Against AI-Specific Attacks: New attack vectors like prompt injection and model evasion require novel defensive mechanisms. Defenders should implement:

• Input/Output Validation and Sanitization: Rigorous validation and sanitization of user prompts and model outputs to detect and neutralize malicious inputs or prevent the leakage of sensitive information.
• AI Firewalls/Guardrails: Deploying specialized AI security tools that act as "firewalls" to monitor and filter interactions with AI models, identifying and blocking suspicious prompts or anomalous model behavior.

1. Combat Shadow AI and Implement Strong Governance: The proliferation of AI tools and services means employees may be using AI without IT or security oversight, creating Shadow AI. Defenders must:

• Discover and Inventory AI Usage: Implement tools and processes to discover and inventory all AI applications and services being used across the organization.
• Establish AI Governance Policies: Develop clear policies for the responsible and secure use of AI, including guidelines for handling confidential data, acceptable use of public AI services, and mandatory security reviews for internal AI projects. This governance should span the entire AI lifecycle.

1. Leverage AI to Augment Defensive Capabilities: Crucially, defenders should actively embrace AI as a force multiplier for their own operations to combat burnout and enhance effectiveness:

• Automate Threat Intelligence and Vulnerability Analysis: Use AI to analyze vast amounts of threat data, identify emerging vulnerabilities, and summarize complex security incidents for faster triage and response.
• Automate Threat Hunting: Develop AI-powered solutions to automatically detect and hunt for adversaries within networks, reducing manual effort and improving detection rates.
• Enable Self-Healing Systems: Invest in AI capabilities that can automatically find and fix software vulnerabilities, moving towards a vision of "self-healing products" and significantly reducing the burden on development and security teams. This directly addresses the burnout problem by offloading repetitive and time-consuming tasks.
• Educate and Empower Security Teams: Provide training and resources to security professionals to understand the new AI threat landscape and effectively utilize AI-powered defensive tools, ensuring they remain relevant and productive.

By proactively integrating these defensive strategies, organizations can secure the pathway for AI innovation, mitigate emerging risks, and transform their security posture in this new era.

Key Takeaways

• AI is a Double-Edged Sword: AI, particularly GenAI, is a powerful force multiplier that can significantly boost productivity and augment human capabilities in security, but it simultaneously creates a vast new attack surface and introduces novel threats.
• Urgent Need for AI Security: Despite widespread recognition of AI's importance, most organizations are critically unprepared to secure their GenAI projects, creating a significant risk gap that needs immediate attention.
• The AI Pipeline is the New Attack Surface: Security must be integrated across the entire AI lifecycle—from data collection and model training to inferencing and live use—to protect against threats like data exfiltration, supply chain attacks, model poisoning, and prompt injection.
• Protecting Model Intellectual Property is Paramount: Beyond traditional data, securing model weights and the execution environment of AI models as critical intellectual property is a novel and essential defensive priority.
• AI Can Combat Security Burnout: AI offers a vital opportunity to automate mundane security tasks, analyze threats faster, and even enable self-healing software, thereby empowering human security teams and mitigating the pervasive issue of professional burnout.
• Governance and "Shadow AI" are Critical Challenges: Establishing robust governance frameworks for AI usage and actively discovering and securing Shadow AI within organizations are crucial steps to manage new risks and ensure responsible AI adoption.

About the Speaker(s)

Kevin Skapinetz is the Vice President of Security Strategy at IBM. He is a seasoned professional in the cybersecurity industry, having attended RSA Conference for 17 years, with 16 of those years as an attendee before taking the stage. His extensive experience provides him with a deep understanding of the industry's challenges, including the pervasive issue of professional burnout among security teams. Skapinetz is focused on how emerging technologies like AI will reshape the security landscape and IBM's strategic approach to these transformations.

Rosa Bolger is a Vice President and Distinguished Engineer at IBM, where she leads the company's cyber defense strategy. With a strong background in cybersecurity, Bolger brings practical, hands-on insights into protecting complex systems. Her role involves understanding the evolving threat landscape, particularly as it relates to AI, and developing frameworks and solutions to secure IBM's own operations and its clients. She emphasized the detailed technical challenges of securing AI, such as protecting model weights and managing vast datasets, reflecting her deep engineering expertise.