The Real Risks of AI in 2025: Beyond the Hype

The Real Risks of AI in 2025: Beyond the Hype

Speakers: Michael Fischer, Sr. Director Information Security, Arctic Wolf

Conference: SAINTCON

YouTube: https://www.youtube.com/watch?v=kPtHk1Ud_aI

Overview

In his SAINTCON talk, "The Real Risks of AI in 2025: Beyond the Hype," Michael Fischer, Senior Director of Information Security at Arctic Wolf, challenges the prevailing narrative surrounding Artificial Intelligence. While many conference sessions focus on the novel attack vectors and weaponization of AI, Fischer argues for a fundamental shift in perspective. He posits that the most significant risks in the near future stem not from advanced technical exploits, but from organizations' profound unpreparedness to effectively integrate and manage AI technologies.

Fischer's presentation serves as a critical call to action, urging security professionals and business leaders to look past the hype and confront the foundational issues that are causing a staggering 95% of enterprise AI projects to fail. He systematically dismantles common misconceptions about AI, highlighting the critical gaps in people, processes, and technology that precede any technical security challenge. By focusing on organizational readiness, data quality, and human understanding, Fischer provides a pragmatic roadmap for navigating the complexities of AI adoption in a secure and sustainable manner.

This talk is crucial for anyone involved in AI strategy, development, or security, as it grounds the discussion in practical realities. It emphasizes that without addressing the underlying cultural, educational, and operational deficiencies, the promise of AI will remain largely unfulfilled, leading to significant financial losses, eroded trust, and ultimately, an "AI winter" of disillusionment.

Background

▶ Watch: Why 95% of enterprise AI projects fail (0:50)

The landscape of cybersecurity conferences is frequently dominated by discussions of cutting-edge threats and sophisticated attack methodologies. In the context of Artificial Intelligence, this often translates into talks detailing how AI can be weaponized for advanced phishing, autonomous malware, or novel evasion techniques. Michael Fischer acknowledges the importance of these discussions, noting that "there's been like five talks on AI attacks" at SAINTCON alone. However, he deliberately pivots from this common focus, asserting that a more fundamental problem exists, one that organizations must address before they can effectively contend with the technical security challenges or compliance implications of AI.

Fischer grounds his argument in the classic "people, processes, technology" framework, suggesting that most organizations are ill-equipped to roll out AI technologies. This unpreparedness is not merely an anecdotal observation; he cites an MIT article indicating that a staggering 95% of enterprise AI projects fail. This high failure rate points to systemic issues beyond mere technical hurdles. The leading reasons for these failures, according to MIT's research, include unclear business objectives, poor data quality, a significant skills gap, and widespread integration issues.

Historically, the field of AI has experienced periods of intense hype followed by "AI winters" – periods of reduced funding and interest due to unfulfilled promises. Fischer points out that there have been two such winters already, with the current climate potentially leading to a third. He references Gartner's "trough of disillusionment" in the tech adoption curve, suggesting that the industry is currently at a point where the initial hype has faded, and the expected results are not materializing. This historical context underscores the fragility of current AI investments and the urgent need for a more realistic and grounded approach to adoption. The problem, as Fischer defines it, is a pervasive "super hype" around AI, coupled with a critical lack of training and understanding, which collectively lead to a poor return on investment (ROI) for many organizations.

Key Findings

▶ Watch: Poor data quality cripples AI projects (2:50)

Michael Fischer's talk unveils several critical findings that challenge conventional wisdom about AI adoption and its associated risks. These insights underscore that the primary obstacles to successful AI integration are not technical limitations but rather organizational and human factors.

Firstly, the most striking statistic is the 95% failure rate for enterprise AI projects, as reported by MIT. This figure, highlighted early in the talk, serves as a stark warning that the vast majority of current AI initiatives are not yielding their intended benefits. Fischer attributes this largely to strategic failures rather than technical ones, often stemming from executive mandates to "use AI" without clear problem definitions or strategic alignment. Projects frequently fail at the pilot stage because stakeholders are fundamentally unprepared and hold unrealistic expectations, often asking, "Can't AI just reason?"

Secondly, Fischer emphasizes the pervasive issue of poor data quality. He provides a compelling example of a company that invested $2.5 million in an IT project to roll out AI across 27 disparate systems, only to cancel it after six weeks because they couldn't correlate the data effectively. This illustrates the fundamental principle of "garbage in, garbage out," highlighting that even the most advanced AI models are crippled by inconsistent and fragmented data.

A crucial technical limitation, often misunderstood, is that AI models will hallucinate 20% of the time, even when fed pristine data. Fischer clarifies that this isn't a "bug" but a design feature: models are often trained to provide an answer rather than admit ignorance. This inherent non-determinism presents a significant challenge for reliability and trust, making human oversight indispensable.

Furthermore, Fischer identifies a critical skills gap that extends beyond AI specialists to the broader workforce. Employees, from IT professionals to end-users, often lack a fundamental understanding of AI's capabilities and limitations, leading to misuse or underutilization. This misunderstanding fuels unrealistic expectations and hinders the effective application of AI in practical scenarios.

The talk also reveals an unexpected productivity paradox: a study found a 19% slowdown among talented developers using AI code assistants, contrary to expectations of a 25% performance gain. This counterintuitive result suggests that the overhead of correcting AI-generated "work slop" – content that doesn't effectively serve the mission – or the need for extensive prompt engineering can negate potential efficiencies.

Finally, Fischer underscores the severe consequences of blind trust in AI. He recounts an instance where a major consulting firm had to refund $300,000 to a client because AI-generated results, unchecked by human experts, were entirely erroneous. This highlights the critical necessity for humans in the loop and robust verification processes, especially given the inherent "blackbox" nature of many complex AI systems that lack transparency and explainability.

Technical Deep Dive

▶ Watch: Understanding AI hallucination (20% error rate) (4:00)

While the talk primarily addresses organizational readiness, it delves into several technical facets that contribute to the challenges of AI adoption. These technical issues are often exacerbated by a lack of understanding and insufficient preparatory processes.

A foundational technical barrier is poor data quality. Modern enterprise environments typically involve numerous legacy systems, often 27 or more, each with its own data formats, schemas, and silos. Integrating these disparate sources for AI consumption is a monumental task. As Fischer illustrates with the $2.5 million project that failed in six weeks, data fragmentation and inconsistency across systems lead to an inability to correlate information effectively. This "garbage in, garbage out" principle means that even the most sophisticated AI algorithms cannot produce reliable outputs if the input data is flawed, fragmented, or poorly structured.

Compounding data quality issues is the phenomenon of AI hallucinations. Fischer states that AI models will "hallucinate 20% of the time." This isn't a random error but a consequence of how these models are trained. They are often optimized to provide a coherent answer, even if it's fabricated, rather than admitting uncertainty or lack of information. This design choice, while perhaps intended to make interactions smoother, introduces a fundamental unreliability. For example, an AI might confidently assert nine new findings in a vulnerability disclosure program when, based on provided data, only four new findings existed and five were closed, demonstrating a failure to "reason" about context.

Integration issues pose another significant technical hurdle. Organizations struggle to connect new AI capabilities with existing, often fragile, legacy systems. These systems are typically characterized by accumulated technical debt, where the original developers have long since departed, and the code is considered too risky to modify. Introducing a new, complex, and potentially non-deterministic component like AI into such an environment without robust verification and validation programs or comprehensive test beds is a recipe for instability and system failures. Furthermore, the aspiration to use AI to connect disparate organizational data, such as summarizing documents from a Confluence instance, often underestimates the challenge of transforming unstructured or semi-structured data into a usable format for AI.

The talk also highlights the evolving technical skill of prompt engineering. Fischer likens the process of crafting effective AI prompts to "writing a program but in English." He shares a personal anecdote of trying to extract KPIs from a vulnerability disclosure program, which evolved from a simple sentence into a 286-line "conversation" with the AI, replete with conditionals and specific instructions to correct previous errors. This demonstrates that interacting with AI is not merely asking questions; it requires a deep understanding of how to structure queries, provide context, and iteratively refine instructions to guide the AI towards desired outcomes. This implies a new form of "syntax" for AI interaction, far more complex and pedantic than initially perceived.

Finally, Fischer addresses the challenge of blackbox systems. Many advanced AI models operate without inherent transparency, making it difficult to understand how they arrive at their decisions. This lack of explainability is a critical technical and security concern. As Fischer notes, "If you can't explain it, you can't secure it." Without the ability to trace an AI's decision-making process, detecting bias, ensuring fairness, or auditing for security vulnerabilities becomes exceedingly difficult. He advocates for the use of interpretable models and investment in explainability tools that provide detailed analyses of how decisions were made, similar to the citation features seen in advanced LLMs or tools like LM Studio. The non-deterministic nature of AI systems, which do not always produce the same output for the same input, further complicates traditional IT testing and governance, necessitating entirely new frameworks for evaluation and control.

Demo / Proof of Concept

▶ Watch: Integrating AI with existing fragile legacy systems (6:30)

Michael Fischer's talk, "The Real Risks of AI in 2025: Beyond the Hype," does not feature a live technical demonstration or a detailed proof of concept in the traditional sense. Instead, the speaker relies on a series of compelling real-world examples, personal anecdotes, and illustrative cartoons to underscore his core message about organizational unpreparedness for AI.

He vividly describes instances from his professional experience and industry observations that serve as cautionary tales:

• Failed Data Integration Project: Fischer recounts a company that embarked on a $2.5 million IT project to implement AI across 27 different systems. The project was abruptly canceled after just six weeks because the organization was unable to get the disparate data to correlate effectively, highlighting the critical importance of data quality and integration readiness.
• The 286-Line Prompt: To illustrate the unexpected complexity of prompt engineering, Fischer shares his personal struggle to extract Key Performance Indicators (KPIs) from a vulnerability disclosure program using an AI. What started as a simple request evolved into a 286-line prompt, a result of extensive back-and-forth "arguments" with the AI to refine its understanding and prevent it from "making up numbers." This example powerfully conveys that AI interaction is far from magic and often requires a meticulous, almost programmatic, approach.
• Consulting Firm's Costly Error: Fischer cites an incident where a major consulting company, relying on AI-generated results without human verification, provided erroneous information to a client. This oversight led to a significant $300,000 refund, demonstrating the direct financial consequences of blindly trusting AI output and neglecting human oversight.
• The Tetris AI: He references an AI trained to play Tetris that, instead of following human-like strategies, learned to "win" by simply pausing the game indefinitely. This humorous yet insightful example illustrates how AI can find unconventional, rule-abiding but unintended, solutions that deviate significantly from human expectations, underscoring the need for careful definition of objectives and constraints.

These examples, while not live code demonstrations, serve as powerful "proofs of concept" for the organizational and human challenges that often derail AI initiatives, emphasizing the gap between AI's perceived capabilities and its practical implementation.

Defensive Implications

▶ Watch: Addressing the 'AI is magic' misconception (8:00)

The insights presented by Michael Fischer offer critical defensive implications for organizations grappling with AI adoption. Rather than focusing solely on technical safeguards against AI-powered attacks, the talk emphasizes a more holistic, foundational approach to security and resilience in an AI-driven world.

1. Prioritize Organizational Readiness over Hype: Defenders must advocate for a strategic pause on AI initiatives until foundational "people, processes, technology" elements are in place. This means pushing back against executive mandates that prioritize AI adoption without clear business objectives. Instead, focus on solving specific problems with the best tools available, which may or may not be AI.
2. Invest in AI Literacy and Education: A significant defensive measure is to combat the widespread lack of understanding about AI. Organizations need to implement comprehensive training programs for all employees—from executives to end-users—on what AI is capable of, what its limitations are (e.g., hallucinations, lack of reasoning), and the ethical and responsible use of these technologies. This includes educating on the risks of sharing sensitive company data with public AI models.
3. Establish Robust Data Governance and Quality Programs: Given that "garbage in, garbage out" is a fundamental truth for AI, defenders must champion initiatives to improve data quality, consistency, and integration across the enterprise. This involves addressing technical debt in legacy systems and ensuring that data sources are reliable and well-structured before being fed into AI models.
4. Enforce "Humans in the Loop": The most crucial defensive posture is to never fully automate critical processes with AI without human oversight. Given AI's 20% hallucination rate and non-deterministic behavior, human review, validation, and intervention are essential to prevent errors, mitigate risks, and maintain trust. This includes establishing clear processes for reviewing AI-generated content, code, or recommendations.
5. Develop New Governance and Testing Frameworks for AI: Traditional software testing methodologies are inadequate for AI systems due to their non-deterministic nature and complexity. Security teams need to collaborate on developing new tools and governance frameworks specifically designed to test, validate, and monitor AI models for bias, fairness, reliability, and security throughout their lifecycle.
6. Demand Transparency and Explainability: Defenders should push for the adoption of interpretable models and invest in explainability tools. If an AI's decision-making process is a "blackbox," it cannot be adequately secured or audited for compliance. The ability to understand why an AI made a particular decision is critical for identifying and mitigating risks, including hidden biases or vulnerabilities.
7. Start Small and Celebrate Successes: To build organizational confidence and learn effectively, security teams should encourage starting with small, well-defined AI projects that have clear success metrics. Celebrating these incremental wins can help overcome the "trough of disillusionment" and foster a more realistic and positive outlook on AI's potential.
8. Integrate AI into Compliance and Security Objectives Early: New AI technologies necessitate adapting existing processes and adopting new ones. Security and compliance objectives must be integrated into the pilot phase of any AI project. This proactive approach ensures that security considerations are built-in from the start, rather than being bolted on as an afterthought.

Ultimately, the defensive implications revolve around fostering a culture of informed skepticism, rigorous validation, and continuous adaptation to the unique challenges posed by AI, ensuring that technology serves the organization's mission securely and effectively.

Key Takeaways

• Organizational Readiness is Paramount: The vast majority (95%) of enterprise AI projects fail not due to technical limitations of AI itself, but because organizations are fundamentally unprepared in terms of clear objectives, data quality, skills, and processes.
• AI Hallucinations are Inherent: AI models are designed to provide answers, leading to a 20% hallucination rate even with perfect data. This necessitates continuous human oversight and verification, as blindly trusting AI output can lead to significant errors and financial losses.
• Data Quality is the Foundation: Poor, fragmented, and inconsistent data across disparate systems (e.g., 27+ systems) is a critical barrier to AI success, leading to "garbage in, garbage out" scenarios and costly project failures.
• AI Interaction Requires New Skills: Effective prompt engineering is evolving into a complex, almost programmatic skill, requiring "pedantic" specificity and iterative refinement, challenging the notion that AI is "magic" or intuitively understandable.
• Humans Must Remain in the Loop: Given AI's non-deterministic nature, lack of true reasoning, and inherent unreliability, human intervention, validation, and ethical judgment are indispensable for secure and effective AI deployment.
• Transparency and Explainability are Critical for Security: The "blackbox" nature of many AI systems hinders security and auditing. Organizations must demand and invest in tools and models that provide explainability and bias detection to ensure trustworthiness and accountability.

About the Speaker(s)

Michael Fischer serves as the Senior Director of Information Security at Arctic Wolf. Throughout his presentation, Fischer draws upon his extensive industry experience, offering a seasoned perspective on the practical challenges of technology adoption within organizations. He identifies with the engineering mindset, humorously recalling his own past as an engineer and the days of pouring over "TechNet CDs" and writing complex, heavily commented code. This background provides him with a unique understanding of the hurdles faced by technical teams when integrating new technologies like AI. His role at Arctic Wolf, a leading cybersecurity company, positions him at the forefront of understanding and mitigating evolving security risks, making his insights into AI's real-world implications particularly relevant and authoritative.