GenAI Opportunities and Challenges: Where 370 Enterprises Are Focusing Now

GenAI Opportunities and Challenges: Where 370 Enterprises Are Focusing Now

Speakers: David Gruber, Industry Analyst, Enterprise Strategy Group

Conference: RSAC 2024

YouTube: https://www.youtube.com/watch?v=R57fwtXpWDI

Overview

In this insightful session from RSAC 2024, David Gruber, an Industry Analyst at the Enterprise Strategy Group (ESG), presented a comprehensive overview of how 370 enterprises are navigating the nascent landscape of Generative AI (GenAI). Drawing from a recent survey involving nearly 400 companies, Gruber illuminated the prevailing opportunities, challenges, and strategic priorities shaping GenAI adoption within organizations. This talk serves as a critical benchmark, offering attendees a peer-driven perspective on GenAI utilization, governance strategies, and the evolving cybersecurity posture required to harness this transformative technology responsibly.

Gruber's presentation delved into four key areas: the perceived potential and opportunities of GenAI, specific actions being taken around governance, the subjective and objective risks associated with GenAI use by both application development teams and general user populations, and the critical capabilities enterprises are seeking from security vendors. The session aimed not to be a "how-to" guide, but rather a strategic compass, helping security leaders and practitioners model their own GenAI strategies against current industry trends and the collective experiences of their peers.

The talk underscored that while GenAI is still in its early days, its impact on the cybersecurity industry is profound and rapidly accelerating. Gruber highlighted the dual nature of GenAI—both a potent tool for adversaries and a powerful enabler for defenders. By dissecting the survey data, he painted a clear picture of the pervasive security concerns, the optimistic outlook of defenders, and the urgent need for formalized governance and robust security controls to manage the inherent risks while maximizing the benefits of GenAI.

Background

▶ Watch: Introduction and session objectives (0:00)

The rapid emergence and widespread adoption of Generative AI have presented enterprises with a familiar challenge: integrating a powerful, innovative technology into existing operational frameworks while simultaneously grappling with its inherent security implications. As David Gruber noted, the current situation with GenAI mirrors historical patterns observed with other significant technological shifts, such as the early days of cloud computing. A pervasive set of security concerns often accompanies such innovations, primarily stemming from a lack of knowledge, visibility, and established best practices regarding the associated risks.

This problem is exacerbated by the speed at which GenAI capabilities have been adopted by end-users. Security teams often find themselves playing catch-up, with a "huge amount of generative AI consumption of end users that security teams just have no visibility in." This lack of visibility is a critical gap, as existing security tools and frameworks were not initially designed to monitor or control interactions with sophisticated AI models. Consequently, organizations have responded with initial resistance, with a significant 86% of surveyed organizations reporting they have blocked access to one or more GenAI sites, highlighting an immediate, reactive approach to perceived threats like data leakage.

Beyond internal consumption, the adversary landscape is also rapidly evolving. Concerns around GenAI's potential for offensive use include the expansion of the attack surface, supply chain vulnerabilities, the proliferation of deepfakes for social engineering and fraud, and the generation of highly believable phishing campaigns at an unprecedented pace. The prospect of net new malware strains developed with AI assistance further compounds the uncertainty, presenting an unknown frontier for defenders. This dual pressure—from internal adoption outpacing controls and external threats leveraging the same technology—creates an urgent need for strategic guidance and effective defensive measures.

Key Findings

▶ Watch: Pervasive security concerns and GenAI site blocking (4:20)

The Enterprise Strategy Group's survey of nearly 400 organizations unveiled several critical insights into the enterprise adoption and security posture surrounding Generative AI.

A key finding is the pervasive security concerns surrounding GenAI. The data revealed that 86% of organizations have proactively blocked access to one or more GenAI sites, primarily driven by worries about data leakage and a general lack of visibility into end-user GenAI consumption. This reactive stance underscores the initial apprehension and the gap in security tooling.

From an offensive perspective, the primary concerns include GenAI's potential to expand the attack surface, introduce vulnerabilities through the supply chain, facilitate highly realistic deepfake attacks, enable more convincing and rapid phishing campaigns, and potentially generate entirely new strains of malware. These fears highlight the urgent need for enhanced threat intelligence and adaptive defensive strategies.

Despite these significant concerns, defenders exhibit a remarkable degree of optimism. A substantial 75% of organizations reported already actively using some form of GenAI capability within their security environments. While many of these applications are currently "simple, basic interactive functions," this widespread adoption signifies a strong belief in GenAI's potential to augment security operations.

The survey identified the top five areas where organizations are currently applying GenAI capabilities:

1. Improving security hygiene and posture: Leveraging GenAI to analyze vulnerabilities, identify rogue devices, and prioritize risks across the attack surface, enabling faster data analysis and decision-making.
2. Up-leveling junior security analysts: Utilizing GenAI as a learning tool, providing interactive workflows, script analysis, and signal correlation to accelerate the development of less experienced personnel. It also offers a unique way to assess acumen during onboarding.
3. Accelerating threat detection and response: Employing GenAI for automation, data enrichment (e.g., pre-sandbox activities, script analysis), and streamlining investigative processes.
4. Security technology consolidation: While not directly consolidating tools, GenAI assists by tapping multiple data sources, bringing disparate information together, and reducing the "swivel chair" effect for analysts.
5. General process automation: Covering a broad spectrum from proactive security and attack surface exploration to detection, investigation, and automated responses.

Further breaking down automation priorities, organizations are focusing GenAI applications on: security investigations (for speed and faster decision-making), orchestration across heterogeneous security controls, general process automation, case management (for report writing, summarization, and diverse communication), alert enrichment (automating data pull-in and threat intelligence mapping), and ticketing systems.

A critical finding regarding trust and automation is that while GenAI offers recommendations, a significant 71% of organizations still require a staff member to review recommendations before taking action. This "human in the loop" approach reflects ongoing concerns about hallucination issues and overall quality, but Gruber anticipates a cultural shift over time as trust in GenAI capabilities grows.

Governance emerged as a significant work in progress, with less than 20% (specifically 10%) of organizations having formalized governance structures in place. Larger companies tend to be more advanced in this area. Key governance actions being explored or implemented include required end-user training, approval processes for GenAI use, vetting processes for GenAI-leveraging products, risk assessments for each GenAI application, and the creation of acceptable use policies. The weakest links in governance were identified as third-party risk management, data leakage protection, regulatory compliance, general enforcement capabilities, and user awareness training.

Regarding application development teams, 82% reported believing they understand GenAI risks, though Gruber expressed skepticism about this optimistic figure. Two-thirds of dev teams have received at least some formal training. The biggest perceived gaps and risks for app dev were trust boundaries and data management/exposure risks, echoing concerns from early cloud adoption.

Finally, when selecting vendors for GenAI-assisted security technology, over half of organizations (50%+) lean towards trusting a single large vendor for their security platform, while roughly a third prefer a best-of-breed strategy. Top vendor considerations include a formal, documented process for security and data privacy, support for multiple LLMs (including open-source versions), an open architecture that can accommodate data from other security vendors, a strong history with AI and Machine Learning (ML), and capabilities built directly on top of LLMs.

Technical Deep Dive

▶ Watch: Lack of visibility into end-user GenAI consumption (5:20)

The technical core of GenAI's application in cybersecurity, as illuminated by Gruber's presentation, revolves around its ability to process, analyze, and synthesize vast quantities of data at speeds and scales unattainable by human analysts alone. The foundational technology enabling these capabilities is the Large Language Model (LLM), which provides the interactive and analytical horsepower.

In the realm of improving security hygiene and posture, GenAI acts as an advanced analytical engine. It can ingest and contextualize data from various sources—vulnerability scanners, configuration management databases, network logs, and asset inventories. The LLM's capacity for pattern recognition and natural language understanding allows it to "pick apart all the different vulnerabilities, flaws in my attack surface, rogue devices, unknown aspects," and then, crucially, prioritize these risks. This involves correlating disparate data points to identify critical exposure pathways, effectively serving as an intelligent assistant for Attack Surface Management (ASM) and Vulnerability Management (VM), enabling faster analysis and more informed decision-making.

For up-leveling junior security analysts, GenAI manifests as co-pilot or co-pilot-like capabilities. These interactive tools guide analysts through complex investigations. For instance, when analyzing a malicious script, the GenAI assistant doesn't just provide a conclusion; it can break down its analytical process, explaining how it identified key functions, suspicious calls, or obfuscation techniques. This "shows the workflow that the generative engine went through," transforming the tool into a powerful learning mechanism. It helps junior staff "learn faster about what they need to do" and understand the investigative process, signal analysis, and conclusion derivation. Furthermore, Gruber highlighted a fascinating use case where CISOs leverage these interactive capabilities during onboarding to assess a new hire's acumen and drive, effectively using GenAI as a "shadowing a learning process" tool.

Accelerating threat detection and response heavily leverages GenAI for automation and data enrichment. Instead of manually gathering context for an alert, GenAI can automate tasks like pre-sandbox activities, script analysis, and correlating alerts with internal and external threat intelligence. This significantly reduces the manual "crap work" for analysts, allowing them to focus on higher-level strategic analysis and decision-making. The LLM can retrieve and summarize relevant information from disparate security tools, enriching alerts with context such as affected assets, user behavior profiles, and known threat indicators before they even reach an analyst's queue.

The concept of security technology consolidation is addressed by GenAI's ability to act as a unifying layer. Instead of analysts having to write queries across "five separate data sources," the LLM model performs this task transparently. It aggregates and synthesizes data from various security controls—such as Security Information and Event Management (SIEM), Endpoint Detection and Response (EDR), and Cloud Security Posture Management (CSPM)—presenting a consolidated view. While not physically consolidating the underlying tools, it significantly reduces operational friction and the "swivel chair" effect, improving efficiency.

In general process automation, GenAI extends its reach across the entire security lifecycle. This includes automating aspects of proactive security, such as attack surface exploration, streamlining detection processes, enhancing investigations through rapid data correlation, and even facilitating automated responses. Specific areas where this is applied include security investigations (focused on speed and faster decision-making), orchestration across heterogeneous security controls (connecting disparate systems), case management (automating report generation, summarization, and tailored communication for technical and non-technical audiences), alert enrichment, and ticketing systems.

For defensive controls, Gruber suggested leveraging existing security infrastructure. Firewalls can be used for basic access control to GenAI services. Cloud Access Security Brokers (CASB) can help manage which user groups have access to specific data when interacting with GenAI, and monitor data flows. User and Entity Behavior Analytics (UEBA) capabilities can detect suspicious activity related to GenAI use, identifying anomalies that might indicate data leakage or misuse. The upcoming NIST CSF 2.0 update was also highlighted as a crucial framework to guide organizations in formalizing their GenAI governance and risk management processes.

Demo / Proof of Concept

▶ Watch: Emerging threats: deep fakes and advanced phishing (7:00)

This particular conference talk was a presentation of survey findings and strategic insights, rather than a demonstration of a specific GenAI security product or proof of concept. David Gruber, as an industry analyst, focuses on observing and reporting on market trends and vendor offerings.

However, Gruber did describe observing various demonstrations from security vendors at the conference. He noted that "most all the security vendors brought something to the table here today" in terms of leveraging GenAI capabilities. These demonstrations showcased interactive functions within security solutions, such as tools that use GenAI to "pick apart all the different vulnerabilities, flaws in my attack surface," and others that provide "co-pilot and co-pilot like capabilities" to assist security analysts. He also mentioned seeing vendors demonstrate how GenAI helps with "automation, like really basic automation, data enrichment use cases," such as pre-sandbox activities and script analysis, and how it can "go out and tap many, many data sources" to bring data together for analysts. While no specific product was demoed by Gruber, his observations confirm that the industry is rapidly moving from theoretical discussions to tangible GenAI-powered security solutions.

Defensive Implications

▶ Watch: Defenders' optimism and GenAI application opportunities (8:40)

The insights from David Gruber's presentation provide clear and actionable implications for security defenders grappling with the rapid evolution of Generative AI. The overarching message is one of proactive engagement, strategic governance, and intelligent leveraging of both existing and new technologies.

Firstly, prioritizing AI governance is paramount. Organizations must establish or formalize AI governance within their cybersecurity steering committees, or create such committees if they don't exist. Leveraging frameworks like the NIST CSF 2.0 update, which has formalized its govern capability, can provide a robust structure for this. This isn't just an IT or security responsibility; it requires cross-functional stakeholder involvement, including line of business leaders, technical leads, and legal, to ensure policies are comprehensive and widely adopted. Key governance actions should include mandatory end-user training for all staff (basic users, technical users, and developers), establishing approval processes for GenAI tool usage, implementing vetting processes for third-party GenAI-enabled products, conducting risk assessments for every GenAI application, and formalizing acceptable use policies.

Secondly, defenders must leverage existing security infrastructure for immediate controls. Before investing in net new, purpose-built GenAI security tools, organizations should assess and deploy controls using their current tech stack. This includes using firewalls for basic access control to GenAI services, implementing CASB (Cloud Access Security Broker) solutions to manage user access to data and monitor sensitive data leakage in GenAI interactions, and deploying UEBA (User and Entity Behavior Analytics) to detect suspicious activities and anomalous behaviors related to GenAI use. While specialized gateways for filtering traffic to LLMs are emerging, existing tools can provide significant foundational control.

Thirdly, a strong focus on trust boundaries and data management is critical. Learning from the early days of cloud adoption, organizations must anticipate and mitigate risks associated with data exposure and the flow of sensitive information into GenAI models. This means scrutinizing vendor processes for security and data privacy, ensuring that any GenAI applications developed internally adhere to stringent data protection standards, and addressing the weakest links in governance, particularly third-party risk management and data leakage protection.

Finally, strategic vendor engagement is essential. Defenders should actively engage with their security vendors to understand their GenAI roadmaps. Key criteria for vendor selection should include: a formal, documented process for security and data privacy in GenAI development and operations; an architecture that supports multiple LLMs, including open-source options, to avoid vendor lock-in and enable future innovation; an open ecosystem approach that can accommodate data from other security vendors; and a proven track record with AI and ML technologies. Defenders should challenge vendors to move beyond rhetoric and provide concrete, specific capabilities that align with their organization's GenAI strategy.

In summary, the defensive posture against GenAI risks and the strategy for leveraging its opportunities require a blend of immediate tactical actions (using existing tools), strategic governance implementation (policies, training, risk assessments), and forward-looking vendor partnerships to build a resilient and adaptive security ecosystem. The goal is to move from a reactive, blocking approach to a proactive, enabling framework that securely integrates GenAI into the enterprise.

Key Takeaways

• Establish Robust AI Governance: Prioritize the development and formalization of AI governance policies, leveraging frameworks like NIST CSF 2.0, and ensure cross-functional stakeholder involvement from all business units and technical teams.
• Leverage Existing Security Investments: Deploy immediate GenAI controls using current security infrastructure, including firewalls for access, CASB for data governance, and UEBA for behavioral anomaly detection, before rushing into new, specialized solutions.
• Proactively Manage Emerging Risks: Address critical concerns such as data leakage, third-party GenAI risks, deepfake threats, and advanced phishing by enhancing visibility, implementing rigorous risk assessments, and integrating GenAI-specific training into user awareness programs.
• Harness GenAI for Defensive Advantage: Actively integrate GenAI to improve security hygiene and posture, up-level junior analysts through interactive learning tools, accelerate threat detection and response through automation and data enrichment, and streamline security operations across heterogeneous controls.
• Critically Evaluate Vendor Capabilities: Select security vendors based on their commitment to documented security and data privacy processes, support for multiple LLMs (including open-source), an open architecture for data integration, and a proven history in AI/ML, rather than just marketing claims.
• Maintain Human Oversight While Fostering Trust: Recognize the current necessity for human review of GenAI recommendations, but anticipate and prepare for a cultural shift towards greater trust and automation as the technology matures and its reliability improves.

About the Speaker(s)

David Gruber is an Industry Analyst at the Enterprise Strategy Group (ESG), a prominent IT analyst, research, and strategy firm. In his role, David specializes in the cybersecurity industry, meticulously studying both the practices of cybersecurity practitioners and the offerings and innovations of vendors. He describes his function as a "matchmaker," connecting the evolving needs and challenges of technology users with the solutions and capabilities being developed by vendors. His work provides critical insights into industry trends, helping organizations understand what their peers are doing and guiding vendors in delivering relevant features and innovations.