A Year(ish) of Countering Malicious Actors' Use of AI: What Have We Learned?

A Year(ish) of Countering Malicious Actors' Use of AI: What Have We Learned?

Speakers: Sherrod DeGrippo (Microsoft), Cynthia Kaiser (FBI), Adam Maruyama (Everfox), Lyn Brown (Wiley Rein LLP)

Event: RSA Conference 2025 — April 28–May 1, 2025, Moscone Center, San Francisco

Watch: YouTube — A Year(ish) of Countering Malicious Actors' Use of AI

Reading time: ~9 minutes

TL;DR

A panel of cybersecurity practitioners, law enforcement, and legal experts convened at RSA Conference 2025 to take stock of a year's worth of real-world lessons from countering AI-enabled cyber threats. While existing laws have proven more adaptable than feared, critical gaps remain in how investigators detect AI-generated content, how thresholds for government disruption operations are determined, and how defenders can separate synthetic noise from real harm. The clearest conclusion: the legal framework is ahead of the narrative, but investigative and operational frameworks are still catching up.

Introduction

The scenario Cynthia Kaiser posed to open the session was simple and unsettling. Imagine receiving a video call from your CEO on a regular messaging app. The background is a residence you recognize. The CEO says there is an urgent meeting and sends a link. Would you click it?

Kaiser, Deputy Assistant Director at the FBI's Cyber Division, was describing a real case — a criminal who impersonated a software company's CEO via deepfake video to trick an employee. It resulted in significant financial losses. It was also, she noted, just one of many such cases from the past year.

That scenario set the stage for a candid, practitioner-focused assessment at RSA Conference 2025. The panel — which also included Sherrod DeGrippo of Microsoft, Adam Maruyama of Everfox, and Lyn Brown of Wiley Rein LLP — worked through what a year of actual enforcement, threat tracking, and legal action against AI-enabled adversaries has actually taught defenders, prosecutors, and policymakers. The session, held on the Policy Stage, was less about theoretical risk and more about what the field now knows from hard experience.

Nation-States Are Using AI to Work Smarter, Not to Do the Impossible

Sherrod DeGrippo opened the threat landscape portion of the discussion with a data-grounded summary of what Microsoft's threat intelligence research has established about nation-state AI use. China, Russia, North Korea, and Iran — the so-called "big four" — are all actively using large language models and generative AI to support their operations.

▶ Watch: Nation-state AI use overview (02:00)

But the nature of that use is more mundane than alarming headlines suggest. "We don't see threat actors at this time using AI for something that they couldn't do a little slower on their own," DeGrippo said. The observed applications include research acceleration, communication refinement, translation, content generation for social engineering, and code improvement — the same productivity gains that legitimate users extract from the same tools.

The implication is important for how defenders prioritize resources. The threat is not a step-change in adversary capability so much as an acceleration of existing workflows. Nation-state actors are becoming faster and more scalable, not fundamentally more capable. That said, DeGrippo's framing also contains a warning: scale and speed are not trivial advantages when applied to already-sophisticated operations targeting critical infrastructure.

The Criminal Ecosystem Is Lowering the Technical Bar — and the Numbers Prove It

Adam Maruyama, Field CTO for Digital Transformation and AI at Everfox, focused on the criminal side of the AI threat, and here the picture is more acutely worrying for typical organizations. While APT actors use AI primarily to improve targeting precision and communication credibility, the criminal ecosystem is using it to dramatically expand the volume and believability of attacks.

▶ Watch: AI-powered phishing and criminal LLMs (04:00)

The numbers are stark. Since the release of ChatGPT, the quantity of phishing emails has increased by more than a thousand percent. Phishing-related domains have increased approximately 120 percent over the same period. The emails themselves are no longer the easily spotted, grammatically fractured messages of a few years ago — they are contextually accurate, personally plausible, and often reference real institutional details that would be difficult for a recipient to verify quickly.

Maruyama flagged a particularly concerning development: the emergence of criminal-purpose LLMs with local compute resources, which are harder for commercial AI providers to take down. He referenced research from the University of Illinois Urbana-Champaign showing that based solely on CVE descriptions — not detailed code — researchers were able to generate proof-of-concept attack code against eight of thirteen tested vulnerabilities. Criminal tools like what Maruyama described as "Xanthrox AI" are being positioned as essentially AI-powered Metasploit frameworks — lowering the technical entry bar for exploitation at scale.

▶ Watch: Criminal LLMs and technical exploitation (04:30)

Existing Law Is More Adaptable Than the Discourse Suggests — With Key Exceptions

Lyn Brown, a partner at Wiley Rein LLP specializing in cybersecurity law, addressed what is arguably the most underappreciated aspect of AI-enabled crime: the legal framework for prosecution is more capable of addressing these cases than commonly assumed.

▶ Watch: Legal tools for prosecuting AI-enabled crime (08:00)

The clearest example is North Korea's remote IT worker fraud scheme, in which individuals use AI-enhanced identity documents and fabricated LinkedIn profiles to trick U.S. companies into hiring them — funneling income back to Pyongyang's weapons programs. These cases are being successfully prosecuted under pre-existing statutes: identity theft, wire fraud, money laundering, and sanctions violations. The AI component makes the fraud more convincing and harder to detect, but it does not create a legal vacuum.

Brown also walked through the more complex territory of synthetic content. Early media reporting had suggested that AI-generated child sexual abuse material might not violate U.S. law — an assertion Brown flatly rejected. The Child Pornography Prevention Act of 1996 covers visual depictions that have been "created, adapted, modified." The PROTECT Act of 2003 goes further, specifying that the government does not need to prove a real minor exists and covers "any kind of" depiction, including drawings and digital imagery. These laws predate generative AI by decades, but they apply directly.

▶ Watch: AI-generated CSAM and existing law (12:00)

The harder challenge Brown identified is investigative capacity: when adversaries flood investigators with synthetic content, the ability to quickly distinguish fake from real becomes critical for rescuing actual victims. No legal sophistication can substitute for the technical tooling required to parse through high volumes of AI-generated material at speed.

Microsoft Takes Adversaries to Court — and Makes the Case for Civil Remedies

Sherrod DeGrippo described Microsoft's lawsuit against a threat actor designated Storm-2139, which had been using Microsoft's services and products to generate deepfake images for fraud. The case was filed in Virginia, and DeGrippo framed it as a deliberate signal: misuse of AI products for criminal activity will result in legal action.

▶ Watch: Microsoft's Storm-2139 civil lawsuit (14:00)

The broader point was about the role of civil legal tools alongside criminal prosecution. The Computer Fraud and Abuse Act has civil provisions allowing victimized companies to sue for injunctive relief and damages. Copyright and trademark law provides additional civil remedies. Microsoft's Digital Crimes Unit has built a track record of using these tools not merely to seek compensation, but to affirmatively disrupt threat actor operations — a model that other large technology companies are increasingly studying.

DeGrippo paired this with a sharp observation about threat intelligence itself: "If your threat intelligence is not actionable, it's threat entertainment." The critique applied equally to how organizations consume AI threat reporting — consuming alarming narratives without translating them into operational changes produces no security benefit.

Government Disruption Operations: Where AI-Specific Thresholds Remain Unsettled

Kaiser turned to the hardest question in the panel: what happens when a government agency decides that an adversary's AI-enabled campaign justifies active disruption? The FBI has existing authorities, but the procedural and technical questions around AI-specific operations remain largely unresolved.

▶ Watch: Thresholds for government disruption operations (16:00)

Currently, the criticality and scope of targets — not the novelty of adversary methods — typically drives decisions about sophisticated counter-operations. But as AI enables adversaries to conduct attacks at greater scale with more convincing tradecraft, Kaiser suggested the field will need to grapple with whether novel AI-enabled methods themselves should trigger higher-priority response, independent of target criticality.

The technical challenge is equally formidable: if an adversary is running operations from a custom AI model, countering it requires either obtaining a copy of that model and developing adversarial techniques against it, or generating enough variant samples to identify patterns. Neither path is straightforward, and neither maps cleanly onto existing operational playbooks.

Kaiser noted one unintended intelligence dividend: the FBI had identified a previously unknown adversary persona precisely because that actor was using similar techniques to circumvent AI model guardrails — effectively revealing themselves through their evasion methods. As adversaries increasingly rely on AI, their characteristic patterns of AI use may become a new category of attribution signal.

Notable Quotes

"We don't see threat actors at this time using AI for something that they couldn't do a little slower on their own." — Sherrod DeGrippo, Microsoft

"There has been an exponential increase since the release of ChatGPT in the quantity of these emails — greater than a thousand percent in phishing emails." — Adam Maruyama, Everfox

"Fake content can still violate real laws." — Lyn Brown, Wiley Rein LLP

"If your threat intelligence is not actionable, it's threat entertainment." — Sherrod DeGrippo, Microsoft

"We identified a new adversary, especially their persona, because they were using similar methods to try to get around AI models' guardrails." — Cynthia Kaiser, FBI

Key Takeaways

1. Nation-state AI use is about acceleration, not novel capability — the big four adversary nations are using LLMs for research, translation, social engineering content, and code improvement, not for attacks that would otherwise be impossible.

1. The criminal ecosystem is the more acute near-term concern — phishing volume has exceeded a thousand percent increase since ChatGPT's release, and criminal-purpose LLMs are lowering the technical entry bar for exploitation.

1. Existing legal frameworks are more capable than assumed — wire fraud, identity theft, CFAA civil provisions, and decades-old CSAM statutes are all being applied successfully to AI-enabled crime without requiring new legislation.

1. The investigative challenge is parsing volume, not finding legal authority — when synthetic content floods case evidence, the ability to quickly identify real victims depends on technical tooling that is still being developed.

1. Civil enforcement is a meaningful complement to criminal prosecution — Microsoft's Storm-2139 lawsuit demonstrates that technology companies can use civil legal tools to affirmatively disrupt adversary operations.

1. Adversary AI use creates new attribution signals — the characteristic patterns of how threat actors interact with AI systems, including attempts to bypass guardrails, are becoming a new category of fingerprint for identification.