Detecting the Undetectable: Threat Hunting in Appliance Environments

Detecting the Undetectable: Threat Hunting in Appliance Environments

Speakers: Sagi Tzadik; Shahar Dorfman

Conference: fwd:cloudsec North America 2025

YouTube: https://www.youtube.com/watch?v=1rfB0Pb0t2o

Overview

Sagi Tzadik and Shahar Dorfman, security researchers at Wiz, present a methodology for hunting sophisticated malware in virtual appliance environments in the cloud. Virtual appliances -- products like Ivanti, Palo Alto PAN-OS, Fortinet, Aviatrix, and Zscaler -- are notoriously difficult to monitor because they restrict root access, prevent agent installation, and operate as black boxes. The researchers leverage cloud snapshot APIs to create forensic images of appliance volumes, then build file prevalence histograms across multiple deployments of the same appliance to identify anomalies. By focusing on the "long tail" of files that appear on fewer than 5% of monitored instances, they discovered previously undetected malware including web shells and Sliver implants, and traced the activity to what appears to be a state-sponsored threat actor operating across multiple appliance exploitation campaigns.

Background

▶ Watch: Virtual appliance landscape: 23% of exploited CVEs found in appliances (1:30)

Virtual appliances are preconfigured software packages where the vendor controls the operating system, application stack, and dependencies. They are deployed as-is and traded flexibility for simplicity. In cloud environments, these appliances are ubiquitous -- providing security, networking, and user management functions -- and they are frequently exposed to the internet with elevated permissions. The researchers found that 65% of Aviatrix instances they monitor are granted high IAM permissions in AWS.

The security challenge is severe: 23% of all known exploited CVEs in enterprise environments were found in appliances. Recent years have seen waves of critical remote code execution vulnerabilities in products from Ivanti, Palo Alto Networks, Fortinet, and others, many exploited as zero-days by both opportunistic attackers and nation-state groups. Traditional security tools -- EDR agents, antivirus, host-based monitoring -- typically cannot be installed on these appliances. Ivanti's own documentation explicitly states that antivirus installation on their appliance is restricted.

This visibility gap creates a perfect hunting ground for attackers: high-value targets with elevated cloud permissions, internet exposure, and no monitoring capability.

Key Findings

▶ Watch: Snapshot API solution: treating cloud volumes as forensic images (5:30)

The research produced several significant findings across two case studies:

PAN-OS Case Study: Exploiting CVE vulnerabilities in Palo Alto Networks' PAN-OS (two vulnerabilities from late 2024 that chain to unauthenticated RCE), the researchers built file prevalence graphs across all monitored PAN-OS instances. In the long tail (files present on fewer than 5% of instances), they discovered numerous malware instances, four of which were previously undetected with zero hits on VirusTotal. The majority were web shells, and the team was able to extract attacker IOCs including IP addresses, user agents, and browsing sequences from the compromised machines' log files.

Ivanti EPMM Case Study: Analyzing the Ivanti Endpoint Mobile Manager appliance following two chained vulnerabilities enabling unauthenticated RCE (with a public POC released just 3 days after disclosure), the team searched the long tail for files with anomalously large file sizes. They identified two Sliver implants (an open-source C2 framework increasingly used by threat actors as an alternative to Cobalt Strike), along with web shells, reverse shells, and MySQL database dumps.

Threat Actor Attribution: By pivoting on mutual IOCs, analyzing malware, and examining DNS history, the researchers linked the PAN-OS and Ivanti EPMM compromises to what appears to be a state-sponsored threat actor responsible for multiple appliance exploitation campaigns over the preceding two years.

Technical Deep Dive

▶ Watch: PAN-OS case study: file prevalence graph reveals long tail with malware (10:00)

The core methodology leverages a fundamental property of virtual appliances: their file systems are largely immutable across deployments. Since administrators rarely have root access and don't upload custom files, most files remain identical across all instances of a given appliance technology. Any deviation stands out.

The threat hunting lifecycle proceeds as follows:

1. Select a target appliance technology based on prevalence or known vulnerability.
2. Collect file inventories from multiple instances across different environments using the cloud snapshot API. This involves creating a snapshot of the appliance's volume, attaching it to an analysis system, and treating it as a forensic image.
3. Build file prevalence histograms mapping file paths and hashes against the population of monitored instances.
4. Establish baseline categories: Stock files present on 100% of instances, version-variant files at 90-100%, plugin/add-on files at 10-70%, and the long tail below 5%.
5. Investigate the long tail using enrichment from logs -- who deployed the file, when, and how.
6. Generate IOCs and YARA rules from discovered malware to hunt across other environments.

The file prevalence graph is the key analytical artifact. On the x-axis are file paths; on the y-axis is the percentage of monitored instances where each file appears. The researchers found that stock appliance files cluster at 100%, version variations explain the 90-100% band, and plugins populate the 10-70% range. Below 5%, files are overwhelmingly suspicious and frequently malicious.

For the PAN-OS case, the team asked two specific questions: Are there files with interesting extensions (PHP, EXE, etc.) present on only some machines? And are there common files (like authorized_keys) where a single instance has a unique hash, indicating tampering?

The approach has known limitations: it does not work on appliances with encrypted disks where the vendor holds the keys, and it cannot detect fileless or in-memory malware. The researchers suggest that vendors could significantly improve the ecosystem by publishing baseline file path lists or file hashes for their appliances.

Demo / Proof of Concept

▶ Watch: Four previously undetected malware samples found with zero VirusTotal hits (11:30)

No live demo was presented. The researchers showed graphs and findings from their production hunting operations, including the file prevalence histogram for PAN-OS, the long-tail analysis that surfaced zero-day malware, and the Ivanti EPMM analysis that identified Sliver implants. The methodology was presented as an operational workflow rather than a controlled demonstration.

Defensive Implications

▶ Watch: State-sponsored threat actor attribution across PAN-OS and Ivanti campaigns (16:00)

Defenders managing virtual appliance environments should take several immediate actions:

• Use cloud snapshot APIs for forensic analysis. Even when SSH access and agent installation are impossible, the snapshot API provides full disk-level visibility.
• Build file prevalence baselines across all instances of each appliance technology in your environment. Even with a small number of instances, comparing against a stock installation can surface anomalies.
• Focus hunting on the long tail. Files present on fewer than 5% of instances of a given appliance warrant immediate investigation.
• Monitor IAM permissions on appliances. With 65% of Aviatrix instances running with high IAM permissions, a compromised appliance provides a direct path to cloud account takeover.
• Advocate for vendor telemetry. Work with appliance vendors to expose more logging, file integrity monitoring, and diagnostic capabilities. The researchers note that vendor cooperation on publishing baseline file inventories would dramatically improve the community's ability to detect compromise.
• Treat appliances as high-priority patch targets. With 23% of known exploited CVEs targeting appliances and POCs often available within days of disclosure, patching speed is critical.

Key Takeaways

• Virtual appliances represent 23% of known exploited CVEs and are high-value targets due to their internet exposure, elevated permissions, and resistance to traditional monitoring.
• Cloud snapshot APIs provide a scalable forensic analysis capability that bypasses the visibility limitations of appliance environments.
• Building file prevalence histograms across multiple deployments of the same appliance technology creates a powerful anomaly detection baseline, with the long tail (below 5%) being the primary hunting zone.
• The methodology surfaced four previously undetected malware instances (zero VirusTotal hits) in PAN-OS and two Sliver implants in Ivanti EPMM environments.
• Threat actor attribution linked both campaigns to what appears to be a state-sponsored actor operating across multiple appliance exploitation campaigns.
• The approach does not work on encrypted disks or for fileless/in-memory malware, and vendors could dramatically improve the ecosystem by publishing baseline file inventories.

About the Speaker(s)

Sagi Tzadik is a security researcher on the Wiz Research team, focused on threat hunting and vulnerability research in cloud environments. Shahar Dorfman is a threat researcher at Wiz. Both presented remotely rather than in person at the conference. Their work focuses on developing scalable methodologies for detecting sophisticated threats in cloud environments where traditional security tools cannot operate.