Innovate Now, Secure Later? Decisions, Decisions…

Innovate Now, Secure Later? Decisions, Decisions…

Speakers: Mohammed, COO, IBM Consulting; Sri, Security Expert, IBM Consulting; Ryan, Security Expert, IBM Consulting

Conference: RSAC 2024

YouTube: https://www.youtube.com/watch?v=zw4Li3Ih7IM

Overview

In an era defined by rapid technological advancement, the security landscape faces unprecedented challenges, particularly with the widespread adoption of Generative AI (GenAI). This talk, "Innovate Now, Secure Later? Decisions, Decisions…", delivered by Mohammed, Sri, and Ryan from IBM Consulting at RSAC 2024, delves into the critical tension between accelerating innovation and embedding robust security from the outset. The speakers compellingly argue that the traditional approach of retrofitting security is no longer viable in the context of AI, advocating instead for an "innovate and secure by design" philosophy.

The core message revolves around addressing the alarming disconnect observed among business leaders: while an overwhelming majority (82%) acknowledge the essential nature of secure and trustworthy AI for business success, a significant portion (69%) still prioritize innovation speed over security. Furthermore, a mere 24% of organizations are actively securing their AI projects, with nearly half (47%) expressing uncertainty about what to secure or how much to invest. This session aims to bridge this gap by outlining the novel threats posed by AI, demonstrating how existing security controls can be adapted, and introducing a comprehensive framework for securing the entire AI pipeline.

This presentation is highly relevant for security professionals, C-suite executives, and anyone involved in AI development and deployment. It highlights that AI not only creates new attack surfaces and vulnerabilities but also empowers bad actors with sophisticated tools for more effective attacks, such as deepfakes and tailored phishing. By providing a practical framework and actionable recommendations, IBM Consulting underscores the urgency for organizations to proactively integrate security throughout the AI lifecycle, emphasizing that this is a collective challenge requiring strong partnerships across the industry.

Background

▶ Watch: Introduction to 'Innovate Now, Secure Later?' dilemma (0:00)

The rapid proliferation of Artificial Intelligence, particularly Generative AI, has introduced a paradigm shift in how businesses operate and innovate. While the potential for efficiency gains and novel applications is immense—such as the hospital chain example shared by Mohammed, which used AI to serve 700 more patients weekly—this acceleration often comes at the expense of security considerations. The prevailing mindset among many executives, as revealed by a study conducted by IBM Consulting and AWS involving over 1,000 CXOs, is that "innovation takes precedence over security." This creates a perilous environment where AI systems are deployed without adequate defenses, exposing organizations to significant risks.

The problem is compounded by a lack of clarity and investment in AI security. The study found that only 24% of organizations are actively securing their AI projects, and a staggering 47% are uncertain about what precisely needs to be secured and how much investment is appropriate. This uncertainty stems from the unique nature of AI, which introduces entirely new classes of threats that conventional security frameworks are ill-equipped to handle. As Sri emphasized, simply "applying existing guidelines better" is insufficient because "there are new threats that are upon us, threats that are just different."

These new threats manifest in two primary ways: attackers leveraging AI to enhance their offensive capabilities, and attackers directly targeting AI models and applications. Examples of real-world incidents underscore the urgency: terabytes of training data exposed due to cloud misconfigurations (leading to potential data exfiltration or poisoning), Denial-of-Service (DoS) attacks on large AI service providers causing outages, pre-release code of an electronics company submitted to an AI for quality assessment and subsequently exposed, and a finance worker tricked into transferring $25 million via deepfake technology. These incidents highlight not only the financial and operational impact but also the potential for broader societal consequences, as seen with deepfake robocalls influencing elections. The "entry barrier has lowered," meaning more bad actors can now leverage AI tools to launch sophisticated attacks, making the "cat and mouse" game between defenders and attackers faster and more complicated.

Key Findings

▶ Watch: The solution: Innovate AND secure by design (4:35)

The central tenet of the talk is a resounding call to action: it is not about "innovate or secure," but rather "innovate and secure by design." This paradigm shift acknowledges that security must be an intrinsic part of the AI development and deployment lifecycle, not an afterthought. The speakers outlined several critical findings that underpin this philosophy:

Firstly, AI fundamentally opens up new attack surfaces and introduces novel attack types. These include:

• Model evasion attacks, where subtle changes to inputs can trick an AI model into making incorrect decisions (e.g., a car failing to stop at a stop sign).
• Prompt injection attacks, where malicious input (often a "jailbreak" prompt like "Do anything now") can bypass guardrails, leading to data extraction or unintended model behavior.
• Vulnerabilities in models and applications, where bad actors embed malicious code or backdoors into models hosted on public repositories, which are then downloaded and used by unsuspecting organizations.

Secondly, AI significantly enhances attacker capabilities. Bad actors are leveraging GenAI to:

• Create more convincing spam content and tailored phishing attacks, making traditional detection methods (like looking for grammatical errors) obsolete.
• Produce highly effective deepfakes (e.g., voice clones created for $1 in 10 minutes), capable of sophisticated fraud and misinformation campaigns with massive implications.
• Rapidly adapt to evolving defenses, accelerating the "cat and mouse" game between attackers and defenders.

Thirdly, the talk proposes a three-fold defense strategy to counter these threats:

1. Apply existing controls to conventional attacks: Many AI-related incidents, such as DoS attacks or cloud misconfigurations, are essentially conventional attacks with a new target. Existing cloud-based services or data center-based services for DDoS protection, and robust Cloud Security Posture Management (CSPM), can be configured to protect AI applications.
2. Enhance conventional controls for new interpretations: Traditional security measures need to be expanded to cover AI-specific assets. For instance, data discovery and classification must now identify sensitive training data, understand its lineage, and apply appropriate access controls.
3. Implement new technologies for novel attacks: For truly AI-specific threats like prompt injection and deepfakes, organizations will need to adopt new security technologies designed for these challenges.

Finally, the speakers introduced IBM's comprehensive framework for securing the entire AI pipeline, emphasizing the need to secure the data, secure the model, secure the usage, secure the underlying infrastructure, and establish robust governance. They also stressed the crucial role of partnerships across the industry, highlighting that no single company or individual can effectively combat the evolving threat landscape alone.

Technical Deep Dive

▶ Watch: AI opens up new attack surfaces and strategies (5:00)

Securing Generative AI requires a multi-layered approach that spans the entire AI lifecycle, from data ingestion to model deployment and ongoing use. IBM's framework organizes these efforts into three primary phases of the AI pipeline: Data Collection and Management, Model Development and Training, and Model Inference and Live Use. Underpinning these are the foundational layers of Infrastructure Security and Governance.

Data Collection and Management

This initial phase focuses on the preparation and handling of training data sets. AI models require vast amounts of data, often including critical intellectual property (IP) like product designs, internal codebases, and engineering logs, which have historically been locked away in highly secure systems. Now, this sensitive and proprietary "crown jewel data" is being centralized and connected to AI systems for training, making it a highly attractive target for attackers seeking data exfiltration or data poisoning.

The recommended defensive approaches in this phase are largely based on conventional security solutions, emphasizing strong data security and Identity and Access Management (IAM) fundamentals:

• Data Discovery and Classification: Organizations must accurately identify what types of sensitive data are being used to train models and where they are housed.
• Encryption: Protecting data at rest and in transit through robust encryption mechanisms is essential.
• Data Lineage: Tracking the origin, transformations, and usage of data helps maintain integrity and identify potential compromises.
• User Behavior Analytics (UBA): Monitoring user activity can detect insider threats or unauthorized access to sensitive data repositories by third parties.
• Least Privilege: Implementing strict IAM controls ensures that only authorized individuals and systems have the minimum necessary access to sensitive training data.
• Security Awareness and Training: Educating non-security practitioners, such as data scientists and researchers, on secure data handling practices is crucial to prevent accidental exposures, such as data being placed in unsecured shared repositories.

A notable example mentioned was a cloud misconfiguration that exposed terabytes of training data, underscoring how conventional security failures can have magnified impacts in the AI context.

Model Development and Training

This phase addresses the security of the AI models themselves, particularly given the reliance on open-source pre-trained models downloaded from model sharing repositories. Organizations often lack the resources to build their own Large Language Models (LLMs) from scratch, leading them to leverage these readily available models to accelerate application development. However, these repositories often lack robust security controls, making them prime targets for supply chain attacks.

Attackers can download legitimate open-source models, trojanize them with backdoors or malware, and then re-upload them. Unsuspecting organizations subsequently download these compromised models, introducing vulnerabilities deep within their AI pipeline. The "black box" nature of many models makes it challenging to detect such compromises.

Key defensive strategies here include a mix of conventional and AI-specific solutions:

• API Security: As many organizations consume AI capabilities through APIs, securing these interfaces is paramount. This involves API discovery to map all endpoints and hardening API integrations to third-party models.
• IAM for LLM Agents: As LLM agents are increasingly being trained to autonomously perform business functions with significant privileges, implementing least privilege access control for these agents is critical to prevent privilege escalation and exploitation of excessive permissions to gain access to downstream systems.
• Model Scanners: These are new AI-specific solutions that continuously scan models for vulnerabilities, including embedded malicious code or backdoors.
• Red Teaming: Regular pen testing of AI models, either through automated tools or specialized services, helps uncover vulnerabilities and assess their resilience against various attacks.

The talk cited research by Hidden Layer, which demonstrated the feasibility of downloading, trojanizing, and re-uploading open-source models for subsequent malicious download.

Model Inference and Live Use

This final phase focuses on securing the operational use of AI models, particularly against model evasion and prompt injection attacks. These attacks represent a new "AI spin" on age-old application security problems like malicious injection, but with natural language inputs, detection becomes significantly more challenging. Attackers aim to trick models into providing misleading results, exposing sensitive data, or performing unintended actions (e.g., "jailbreaking").

The primary defense in this phase relies on new AI-specific detection and response solutions:

• AI Detection Response (AIDR) / Machine Learning Detection Response (MLDR): These emerging technologies are designed to continuously monitor for AI-specific attacks on models during live usage. The talk referenced last year's winner of the Sandbox competition as an example.
• Integration with SOC/SIEM/EDR: Alerts from AIDR/MLDR tools should be integrated into existing Security Operations Center (SOC), Security Information and Event Management (SIEM), and Endpoint Detection and Response (EDR) systems to enable responsive actions such as blocking model access, quarantining models, or disabling them entirely.

The speakers highlighted research from Carnegie Mellon and the AI Institute for Advanced Safety, which demonstrated prompt injection attacks using simple string appendages or semantic masking techniques, proving that such attacks do not require high sophistication.

Underlying Infrastructure and Governance

Beyond the AI pipeline stages, two overarching layers are critical:

• Infrastructure Security: This is a foundational element, requiring organizations to apply existing expertise and tools to harden the underlying infrastructure. This includes robust network security, network access control, and intrusion detection and response systems. Investment in both conventional and new AI-specific infrastructure security solutions is necessary.
• Governance: Establishing robust governance across the entire AI pipeline is paramount. This involves continuously monitoring and managing for accuracy, privacy, bias, drift, and trustworthiness of AI models. Critically, security teams and governance teams, historically separate functions, must align and integrate more closely to ensure responsible AI adoption.

The talk also briefly touched upon IBM's internal platform, IBM Consulting Advantage. This internal layer sits between GenAI models (like GPT-3, IBM Watson, Amazon Bedrock) and client applications, using an LLM adapter. It provides real-time security by checking prompts for bias, Personally Identifiable Information (PII), and GPL code usage, while also managing cost tradeoffs. This platform exemplifies a comprehensive, integrated approach to securing AI at scale within a large consulting organization.

Demo / Proof of Concept

▶ Watch: Exploring the risks of prompt injection attacks (6:20)

While the presentation did not feature a live, interactive demonstration of specific tools or exploits, the speakers provided compelling examples and referenced real-world research to illustrate the concepts and validate the urgency of their recommendations.

For instance, in the context of model development and training, Ryan specifically mentioned research by Hidden Layer, where researchers successfully demonstrated the ability to download a model from an open-source repository, trojanize it with malicious code, and then re-upload it. This served as a concrete proof of concept for the feasibility and danger of AI supply chain attacks.

Similarly, when discussing model inference and live usage, Ryan referenced research conducted by Carnegie Mellon University and the AI Institute for Advanced Safety. These researchers were able to execute prompt injection attacks by simply appending a string of characters, akin to a buffer overflow, or by using "synapses to kind of mask the intent." These examples underscored the relative ease with which unsophisticated prompt injection attacks can be carried out and their potential to bypass security controls, providing practical evidence for the need for AI Detection Response (AIDR) solutions.

The speakers also cited various real-world incidents that have "hit the press," such as cloud misconfigurations leading to exposed training data, DoS attacks on AI service providers, pre-release code leakage, and the $25 million deepfake fraud. These incidents, though not live demonstrations, served as powerful "proof of concept" examples of the severe implications of neglecting AI security.

Defensive Implications

▶ Watch: How attackers leverage AI: The $1 deepfake demo (7:20)

The defensive implications are clear and require immediate, structured action from organizations. The speakers provided a timeline-based roadmap, emphasizing that security must be integrated into AI initiatives now, not later.

Immediate Actions (Within the Next Few Weeks):

• Inventory AI Consumption: Organizations should identify and inventory all instances of AI consumption, whether it's using AI as a service (like ChatGPT) or internally developed AI applications.
• Assess Current Security Posture: For identified AI consumption, a thorough assessment of the current security posture is critical.
• Enhance Existing CSPM: Leverage existing Cloud Security Posture Management (CSPM) tools and configurations to prevent common issues like data leaks and misconfigurations in AI environments. This involves understanding where training data sets, models, and sensitive data reside, who is accessing them, and enhancing configuration levels. This can be done with existing conventional technologies.

Short-Term Actions (Within the Next Three Months):

• Data Discovery and Classification: Conduct comprehensive data discovery to identify and classify all training data sets, especially sensitive data within them.
• Establish Data and Model Lineage: Understand the full lifecycle of data and models, tracking their origins, transformations, and dependencies.
• Implement Conventional Security Controls: Apply and enhance existing security controls for data (encryption, access controls) and models.
• Pilot New Technologies: Begin piloting new AI-specific technologies for advanced threats, such as deepfake detection and prompt injection detection. The speakers explicitly recommended seeking help from partners for these specialized solutions.

Mid-Term Actions (Within the Next Six Months):

• Integrate Security into AI Governance: Security practitioners must secure a "seat at the table" of overall corporate AI governance councils. This enables them to proactively help teams understand and moderate AI-related risks and exposures.
• Develop a Comprehensive Operations Model: Establish a robust operational model for how AI is consumed and built, ensuring security is embedded throughout.

Beyond these timelines, the speakers underscored the vital role of partnerships. Mohammed shared a poignant anecdote from an FBI briefing, where an agent lamented the difficulty and cost of obtaining crucial threat intelligence from private companies. This highlights a historical lack of collaboration in the security industry. Going forward, in this "new age of GenAI-powered attacks and GenAI vulnerabilities," collaboration among technology providers (like Microsoft, IBM, AWS, Palo Alto) and consulting partners will be essential to effectively fight threats. The collective strength of a network of partners is needed to combat the sophisticated and rapidly evolving tactics of bad actors who will otherwise "divide and conquer."

Finally, the talk implicitly advocated for platform-based approaches, like IBM's internal IBM Consulting Advantage. This platform provides a consistent layer for building AI applications, offering real-time security checks for bias, PII, GPL code, and cost optimization, ensuring that all client projects are built securely. Organizations should consider similar internal frameworks to standardize and scale secure AI development.

Key Takeaways

• Innovate AND Secure by Design: The fundamental principle is to embed security from the very beginning of AI development and deployment, rather than treating it as an afterthought.
• New Attack Surfaces and Enhanced Threat Capabilities: Generative AI introduces novel attack vectors like model evasion and prompt injection, while simultaneously empowering bad actors with tools for sophisticated deepfakes and targeted phishing.
• Hybrid Defense Strategy: Effective AI security requires a combination of enhancing existing conventional security controls (e.g., CSPM, IAM, data encryption) and adopting new, AI-specific technologies (e.g., model scanners, AI Detection Response).
• End-to-End AI Pipeline Security: A comprehensive approach must secure every stage of the AI pipeline: data collection and management, model development and training, and model inference and live usage, all built upon a secure infrastructure.
• Crucial Role of Partnerships and Governance: No single entity can tackle AI security alone. Industry-wide partnerships for threat intelligence and technology sharing are vital. Additionally, security teams must be integrated into corporate AI governance to proactively manage risks and ensure responsible AI adoption.
• Act Now with a Phased Approach: Organizations should immediately inventory AI use, enhance existing security postures, and within months, implement data discovery/lineage for AI training data, pilot new AI-specific detection tools, and integrate security into AI governance.

About the Speaker(s)

The talk was delivered by a team from IBM Consulting, bringing a wealth of experience in enterprise technology and security.

Mohammed, as the Chief Operating Officer (COO) of IBM Consulting, provided the strategic overview and highlighted the critical business imperative of securing AI. With 30 years in the security space, including a previous role as CEO of a security company, Mohammed emphasized the ever-increasing dangers in the security domain and the necessity for industry-wide collaboration.

Sri, a Security Expert at IBM Consulting, delved into the specifics of emerging AI threats and the urgency of addressing them. Sri detailed how attackers leverage AI, how AI models themselves are attacked, and outlined the three-fold strategy for moderating these exposures using both conventional and novel technologies.

Ryan, also a Security Expert at IBM Consulting, presented the detailed framework for securing Generative AI. Ryan elaborated on the defensive approaches applicable across the AI pipeline—data handling, model development, and live usage—and discussed the importance of underlying infrastructure security and robust governance.