Findings From Real-World AI Application Assessments

Findings From Real-World AI Application Assessments

Speakers: Jake Williams (MalwareJake)

Conference: SAINTCON

YouTube: https://www.youtube.com/watch?v=MyRIZZYFgiE

Overview

In this insightful talk from SAINTCON, security expert Jake Williams, widely known as MalwareJake, dissects the complex and often overlooked security challenges inherent in the burgeoning field of Artificial Intelligence (AI) application development. Drawing upon his extensive background in theoretical computer science and real-world experience conducting AI risk assessments for heavily regulated organizations, Williams offers a sobering yet practical look at the current state of AI security. The presentation serves as a critical wake-up call for an industry rapidly embracing large language models (LLMs) without fully grappling with their fundamental differences from traditional software.

Williams highlights that the commoditization and "democratization" of LLMs have led to their widespread deployment by individuals and organizations lacking deep machine learning expertise. He vividly illustrates this with an anecdote about a CEO eager to dump sensitive claims data into ChatGPT for pattern analysis, bypassing an in-house team of data scientists for the sake of speed. This scenario, Williams explains, underscores a pervasive misunderstanding of AI's capabilities and risks, setting the stage for a new wave of vulnerabilities and breaches. The talk meticulously unpacks why established security paradigms and tools are often ill-suited for AI applications, proposing a shift in mindset and methodology for effective risk management.

The core message is that securing AI applications demands a fresh perspective, acknowledging their unique characteristics like non-determinism and the inherent challenges in distinguishing between data and executable instructions within natural language prompts. Williams is not a "doomer" but a pragmatist, emphasizing that while AI offers immense potential, its secure integration requires a fundamental re-evaluation of how we approach threat modeling, testing, logging, and access control. This article delves into the critical findings and recommendations presented, offering a comprehensive guide for security professionals navigating the complexities of AI-enabled systems.

Background

▶ Watch: Speaker intro and current AI boom context (0:00)

The current era marks what Jake Williams identifies as potentially the fourth or fifth "AI boom," characterized by an unprecedented level of accessibility and commoditization of AI technologies, particularly large language models (LLMs). Unlike previous iterations where AI development was largely confined to specialized machine learning engineers and theoretical computer scientists, LLMs have fundamentally democratized the ability to build and deploy AI applications. This widespread availability, while accelerating innovation, has also introduced significant security challenges.

Williams, with his strong foundation in theoretical computer science, recognized early on that this democratization would lead to non-experts deploying powerful AI systems without a deep understanding of their underlying mechanisms or inherent risks. His personal "aha!" moment came from a client, the CEO of a major insurance company, who expressed a desire to feed all claims data into ChatGPT to identify underwriting patterns. The CEO's rationale—that ChatGPT offered instantaneous results compared to his team of PhD data scientists—revealed a critical gap in understanding: the trade-off between speed and accuracy, and the propensity of LLMs to "make up a bunch of stuff."

This scenario crystallized the need for a focused approach to AI risk assessments. The core problem, as Williams articulates, is that many deploying these systems fail to grasp fundamental characteristics like non-determinism or the subtle ways LLMs process information. Traditional software development principles, often honed over decades, are proving inadequate when applied directly to AI. The industry is, in many ways, "relearning all the same lessons" from past decades of application security, but with new and more complex variables introduced by AI's unique operational model. This context underscores why current security practices need significant adaptation rather than direct application to AI-driven systems.

Key Findings

▶ Watch: The CEO's misperception of ChatGPT for data analysis (1:50)

Jake Williams's assessments of real-world AI applications consistently reveal a set of recurring challenges, which he likens to the repetitive nature of Groundhog Day. These findings highlight fundamental differences between securing traditional software and AI-enabled systems:

1. Non-Determinism is a Core Challenge: Unlike deterministic software where identical inputs yield identical outputs, LLMs are probabilistic and non-deterministic. This means the same input can produce varied outputs, making traditional security testing incredibly difficult. A test might run hundreds or thousands of times with acceptable output, only to produce dangerous results on the n+1th attempt. While parameters can be tuned for "almost deterministic" behavior, this often sacrifices the "creativity" or verbosity users seek, and still doesn't guarantee security. This characteristic renders standard regression testing insufficient and demands a new approach to validation.

1. LLMs Are Often a Small Component: In most AI-enabled applications, the LLM itself constitutes a surprisingly small portion of the overall functionality, particularly outside of pure chatbot scenarios. Authentication, data storage, access control, and other core application logic are typically handled by traditional software components. This means security efforts must be judiciously focused on the specific areas where the LLM interacts with the application, rather than treating the entire application as LLM-centric.

1. Blackbox Testing is Inefficient: Performing blackbox tests on LLM-enabled applications is, as Williams bluntly states, "synonymous with lighting money on fire." Due to non-determinism, a single input might need to be repeated multiple times to observe all possible outputs. Without whitebox access to understand the application's data flow and identify precisely where LLM calls are made, security testers cannot efficiently target their non-deterministic testing efforts, leading to wasted resources and incomplete coverage.

1. Reporting Non-Repeatable Findings is Difficult: Security findings in AI applications often manifest as transient, non-repeatable errors. This presents a significant challenge for reporting to development teams accustomed to clear, reproducible bugs. Williams emphasizes that findings must be organized by impact, not the likelihood of exploitation success. Since the probability of an LLM generating a specific malicious output cannot be mathematically measured through observation alone, security professionals must report all observed impactful conditions, even if they are rare or difficult to reproduce.

1. Common Vulnerabilities Persist (The "Groundhog Day" List): Williams consistently encounters the same critical issues across diverse AI applications:

• Prompt Injection: This is perhaps the most foundational and "unsolvable" problem. LLMs interpret natural language prompts as both data and instructions without a defined syntax. Unlike Cross-Site Scripting (XSS), which has known HTML tags as anchors for defense, LLMs lack such structural cues. Attackers can manipulate prompts to bypass instructions, extract sensitive data, or generate malicious outputs. Williams uses verbal prompt injection in interviews as a practical example.
• Insecure Use of LLM Output: Application developers frequently use LLM output directly as input for other parts of the application without adequate sanitization, treating it as a trusted source. This is a critical trust boundary violation. Sanitizing the unstructured data produced by LLMs is far more complex than sanitizing structured data (e.g., phone numbers, SSNs), forcing a "whack-a-mole" approach against bad patterns rather than validation against known good patterns.
• Improper Authentication: A common and dangerous practice is proxying authentication material or placing API keys directly in system prompts. Williams's "golden rule" is clear: "If the LLM can see it, the LLM can leak it." Furthermore, many AI agents are designed to spoof identity (a STRIDE 'S' threat) by acting on behalf of a user, complicating traditional threat modeling where spoofing is inherently negative.
• Identity Governance and Entitlements Management: Agents often require non-human identities to operate. Organizations face a dilemma: provision a single identity with broad permissions (violating least privilege) or create numerous specialized identities. The latter, while adhering to least privilege, introduces a massive identity governance burden, an area where cybersecurity has historically struggled. Low-code/no-code platforms often exacerbate this by only supporting the broad-permission model.
• Insufficient Logging: Many AI applications lack comprehensive logging, particularly of LLM outputs. While inputs are often logged, the non-deterministic nature of LLMs means logging only inputs is insufficient to reconstruct how a conclusion was reached. Furthermore, prompts and responses are data, not metadata, and should not be simply dumped into a Security Information and Event Management (SIM) system without considering data locality, access controls, and regulatory implications (e.g., GDPR).
• OAUTH is Foundationally Broken for AI Agents: Designed for user delegation, OAUTH struggles with the nuances of AI agents. It cannot differentiate between an interactive agent performing a direct task, an autonomous agent executing a sub-task, or a fully digital employee acting independently. All these actions appear identical in logs, making it impossible to apply granular scope or condition-based access controls, posing significant challenges for auditing and accountability.

Technical Deep Dive

▶ Watch: Starting the discussion on AI risk assessment challenges (3:00)

The technical underpinnings of AI application security, as outlined by Jake Williams, reveal a landscape fundamentally different from traditional software. The core differentiator is the non-deterministic nature of large language models. Unlike a standard function f(x) = y where x always produces y, an LLM's output y' for a given input x is probabilistic. This means f(x) might produce y1 at one instance and y2 at another, even with identical inputs. This characteristic is not a bug but a feature, enabling the "creativity" and varied responses users desire. However, for security, it means that traditional testing methodologies, which rely on repeatable steps and predictable outcomes, are severely hampered. Even efforts to make LLMs "almost deterministic" by tuning parameters often reduce their utility and still don't guarantee security, as subtle variations can still lead to unexpected outputs.

Prompt Injection, a pervasive vulnerability, stems from the LLM's reliance on natural language processing (NLP). LLMs interpret human language, which inherently blurs the lines between data and instructions. Unlike programming languages that have strict syntax rules (e.g., if, for, while statements) or even web technologies like HTML that use tags (<script>, <a>) to delineate code from data, natural language offers no such anchors. Williams draws a compelling analogy to Cross-Site Scripting (XSS), a decades-old problem where a browser interprets user-supplied data as executable code. While XSS remains largely unsolved, its defenses often involve stripping out specific HTML tags. LLMs lack this structural defense mechanism; a simple phrase like "ignore all previous instructions" can fundamentally alter the model's behavior, making robust mitigation extremely challenging, if not mathematically impossible, in a generalized sense.

The issue of Insecure Use of LLM Output arises directly from this lack of syntax and the unstructured nature of LLM responses. When an LLM generates text, that text often flows into other parts of an application. If this output is treated as inherently trusted, it becomes a potent vector for further attacks. Sanitizing unstructured data is a monumental task. For structured data, like a phone number or social security number, developers can define clear patterns and acceptable characters. For arbitrary text, however, sanitization becomes a "whack-a-mole" game, attempting to identify and block known bad patterns, which is never exhaustive. Williams's practical mitigation involves forcing LLMs to output data in a structured schema, such as JSON, even for seemingly unstructured responses. This allows the application to validate the output against a known schema, enabling easier detection when the LLM "goes off the rails" and fails to produce valid JSON, thereby serving as an early warning system.

Improper Authentication and Authorization are significant concerns, particularly with the rise of agentic applications. The principle "If the LLM can see it, the LLM can leak it" highlights the danger of exposing sensitive credentials, such as API keys, within system prompts. These prompts are susceptible to injection attacks that can cause the LLM to reveal its internal context. Furthermore, AI agents, by their very nature, often spoof identity (acting on behalf of a user or another system) and introduce non-repudiation challenges. This clashes with traditional STRIDE threat modeling, where spoofing is typically an undesirable outcome. The complexity extends to Identity Governance and Entitlements Management, where the choice between a single, overly permissive non-human identity (violating least privilege) and numerous narrowly scoped identities (creating an unmanageable governance burden) presents a difficult trade-off, especially in environments utilizing low-code/no-code platforms like Copilot Studio that often lack granular permission controls.

Finally, Insufficient Logging poses a critical challenge for incident response and regulatory compliance. Due to non-determinism, logging only the input to an LLM is inadequate; the output must also be captured to understand how a specific conclusion or action was reached. Williams notes that many organizations fail to log outputs or, if they do, store them inappropriately. Prompts and responses are classified as data, not metadata, and thus require different considerations for storage, access control, and data locality (especially in the context of regulations like GDPR), rather than being indiscriminately jammed into a SIM. While tools like Langsmith offer an "easy button" for LLM-level logging with minimal code changes, Williams emphasizes that application-level logging is crucial to provide the necessary context to correlate LLM interactions with user actions and system events. The existing OAUTH protocol further complicates matters, as it was not designed for the complex delegation scenarios of AI agents. It provides scope for resources but lacks mechanisms to define conditions under which those resources can be accessed, making it impossible to distinguish between authorized and unauthorized autonomous actions in audit logs.

Demo / Proof of Concept

▶ Watch: Understanding non-determinism in large language models (4:15)

While Jake Williams's talk did not feature a live demonstration or specific proof-of-concept during the presentation, he effectively illustrated the discussed vulnerabilities through compelling real-world incident examples and references to practical exercises conducted in his classes.

One particularly striking example Williams recounted involved a customer service chatbot that had been overhauled to use a large language model on the backend, replacing traditional decision trees. The organization had existing filters to block malicious input, such as Cross-Site Scripting (XSS) payloads, from users. However, these filters were designed for direct script injection. An attacker successfully bypassed these defenses by describing to the LLM how to generate an XSS payload in plain language.

The non-deterministic nature of the LLM played a crucial role here: the attacker ran the prompt 13 times before the LLM successfully generated a functional XSS payload. The first 12 attempts produced garbage or non-functional code. Once the successful payload was generated by the LLM, it was then transferred to a customer service representative as part of the chatbot's escalation process. This LLM-generated, and therefore "trusted" by the internal system, XSS payload was then processed, allowing the attacker to steal a session token from the Customer Relationship Management (CRM) system.

This incident vividly demonstrates several key findings:

• Prompt Injection: The attacker's ability to instruct the LLM to generate malicious code, bypassing input filters.
• Non-Determinism: The need for multiple attempts to achieve the desired malicious output, highlighting why single-run tests are insufficient.
• Insecure Use of LLM Output: The application's failure to sanitize the LLM's output before processing it, treating the LLM as a trusted source across a trust boundary.
• Threat Modeling Gaps: The failure to anticipate that an LLM could produce a payload, even if direct user input was filtered.

This real-world example serves as a powerful proof-of-concept for the vulnerabilities discussed, emphasizing the critical need for whitebox testing, comprehensive logging, and a re-evaluation of trust boundaries in AI-enabled applications.

Defensive Implications

▶ Watch: Why 'seeming creativity' leads to suboptimal AI outputs (5:45)

Securing AI applications necessitates a fundamental re-evaluation of traditional cybersecurity controls and strategies. Jake Williams emphasizes that the definition of a "control" itself broadens beyond mere blocking mechanisms to include process controls, detection, and review.

1. Prioritize Logging as a Primary Control:

• Quality over Quantity: It's not about logging everything, but logging the right data for specific investigative use cases. Organizations must define these use cases before implementing logging.
• Log Inputs AND Outputs: Due to non-determinism, logging only LLM inputs is insufficient. Both the prompt and the LLM's response must be logged at the application level to reconstruct events and understand how conclusions were reached, especially for regulatory compliance.
• Data vs. Metadata: LLM prompts and responses are data, often containing sensitive information, not mere metadata. They require different storage considerations, access controls, and data locality policies (e.g., in relation to GDPR) than traditional SIM logs. Avoid simply jamming them into a SIM without proper protection.
• Langsmith: For emergency or rapid debugging, Langsmith (from the creators of LangChain) offers a "Konami cheat code" for adding LLM-level logging with minimal code changes. However, it's a SaaS platform, which may not meet regulatory requirements for data locality and control.

1. Implement AI Content Safety and Prompt Firewalls:

• Meta Llama Guard: An open-source, ready-built platform for AI content safety. It uses a separate LLM to filter harmful outputs or prompts based on predefined or custom categories. While it increases compute requirements and latency (in milliseconds), it's a non-commercial solution to prevent undesirable content generation (e.g., "Nazi fanfiction").
• Prompt Firewalls: While prompt injection cannot be fully mitigated, semi-reliable defenses exist to detect and block adversarial prompts before they reach the LLM. Tools like Rebuff (open-source from Protect AI) and commercial solutions (e.g., Witness AI, Azure Prompt Shield) offer this capability.
• Azure Prompt Shield: A notable exception, this Microsoft tool doesn't use a separate LLM on the backend, making it significantly cheaper (an order of magnitude) but also less reliable.
• The Cost of AI Security: Williams highlights that security for LLM-enabled applications can be a double-digit percentage (e.g., 30%) of the overall operational cost, a drastic shift from traditional web application firewalls (WAFs) which are often a rounding error in operational budgets. This cost must be factored into development and deployment.

1. Strategic Prompt Engineering and Input Validation:

• Limit Input Size: The single most effective tip for reducing prompt injection risk is to limit the size of user input prompts. Many use cases do not require arbitrarily large inputs. This must be enforced on the server side, not just the client side, to prevent circumvention.

1. Rethink Testing and Development Methodologies:

• Breadth vs. Depth in Testing: Due to non-determinism and fixed budgets, security teams must decide whether to test a broader range of inputs less frequently or hammer high-value inputs repeatedly. This decision should be made by business stakeholders who can accept the associated risks, not by security practitioners alone.
• Create Test Harnesses: For whitebox testing, develop test harnesses that can reliably inject specific malicious outputs downstream in the application, bypassing the LLM's non-deterministic behavior. This allows for consistent testing of the consequences of a vulnerability, even if the LLM's output is hard to reproduce.
• Enumerate Failure Modes for Agents: For autonomous agents, assume worst-case scenarios given the agent's tools and data access. If a potential failure mode (e.g., sending regulated data to an unauthorized recipient) is unacceptable, implement guardrails or descoping the application's capabilities.
• Purpose-Built Code: Instead of generic tools, write very purpose-built code for agents. For example, an email tool for an agent should be restricted to specific internal domains if external emailing is a risk.
• Algorithmic Guardrails for MCP: For applications using Model Context Protocol (MCP), implement algorithmic guardrails to restrict tool use. While this limits agent autonomy and complicates programming (and is often not supported by low-code/no-code frameworks), it is essential to control agent behavior.

1. Upskill Staff and Evolve Threat Modeling:

• AI Specializations: The industry needs dedicated AI security specializations within roles like pentesting. Throwing traditional network/infrastructure pentesters at AI applications without foundational knowledge (likened to the early days of Ajax and Web 2.0) will lead to widespread breaches. Managerial staff must invest in upskilling their teams.
• New Threat Modeling Frameworks: Traditional frameworks like STRIDE are inadequate for AI applications because they don't account for autonomy. New tools, such as Maestro from the Cloud Security Alliance, are needed to effectively model threats in AI systems.

1. Refined Reporting:

• Report Weaknesses, Not Just Findings: Security reports for AI applications must articulate identified weaknesses, even if direct, repeatable exploitation cannot be demonstrated. The probabilistic nature of LLMs means a weakness that manifests rarely is still a risk.

Key Takeaways

• Non-Determinism is a Game Changer: The inherent probabilistic nature of LLMs fundamentally alters security testing, making traditional deterministic approaches insufficient. Testing must account for varied outputs from identical inputs, requiring whitebox access and focused efforts.
• Prompt Injection is an Unsolvable Problem (Practically): Unlike XSS, which has syntax anchors for defense, LLMs' natural language processing blurs data and code, making prompt injection a foundational vulnerability with no easy or complete mitigation. Defenses must focus on detection, input size limits, and output sanitization.
• AI Security is Expensive: Implementing robust security for LLM-enabled applications significantly increases operational costs, often representing a double-digit percentage of the total budget, a paradigm shift from traditional application security.
• Comprehensive Logging is Critical but Complex: Logging both LLM inputs and outputs at the application level is essential for transparency, incident response, and regulatory compliance. However, treating prompts/responses as sensitive data (not metadata) requires careful consideration of storage, access controls, and data locality, making traditional SIM integration problematic.
• Traditional Security Frameworks Are Insufficient: Existing threat modeling tools like STRIDE and authentication protocols like OAUTH were not designed for the autonomy and unique trust models of AI agents, necessitating the development or adaptation of new frameworks and methodologies.
• Upskilling is Non-Negotiable: To effectively secure AI applications, security professionals must develop foundational knowledge of AI/ML concepts. Organizations must invest in specialized training rather than simply assigning existing staff to AI security challenges.

About the Speaker(s)

Jake Williams, widely recognized by his alias MalwareJake, is a distinguished figure in the cybersecurity landscape with a unique blend of theoretical and practical expertise. He is noted for his background in theoretical computer science, which provides him with a deep understanding of the mathematical and computational underpinnings of complex systems, including AI. Williams gained prominence for his work in malware reverse engineering and for attributing the infamous Sandworm group's activities.

Currently, Jake Williams focuses his extensive experience on AI risk assessments, working with numerous large organizations, particularly those in heavily regulated fields, to help them securely deploy AI technologies. His talks are characterized by a confident, analytical prose, interspersed with candid war stories and humorous anecdotes that underscore the real-world challenges and absurdities encountered in the field. He is known for his pragmatic approach, shying away from "doomer" predictions while emphasizing the critical need for a new security paradigm tailored for AI. Williams regularly shares his insights through conference presentations and educational initiatives, urging the industry to adapt and upskill to meet the demands of this evolving technological frontier.