Windows Keylogger Detection: Targeting Past & Present Keylogging Techniques- Asuka

Windows Keylogger Detection: Targeting Past & Present Keylogging Techniques

Speakers: Asuka Nakajima, Senior Security Research Engineer, Elastic

Conference: Nullcon

YouTube: https://www.youtube.com/watch?v=1zVSS7NfyR0

Overview

In this insightful Nullcon talk, Asuka Nakajima, a Senior Security Research Engineer at Elastic, delves into the persistent threat of keyloggers on Windows systems. The presentation meticulously dissects both traditional and emerging keylogging techniques, offering a dual-pronged approach to detection. Nakajima shares her team's experience in developing a behavior-based detection feature for an Endpoint Detection and Response (EDR) solution, focusing on API monitoring for common keylogger types, and then introduces a novel method for detecting a more stealthy "hotkey-based" keylogger.

The talk is highly relevant for security professionals, incident responders, and EDR developers seeking to bolster their defenses against keystroke exfiltration. By exploring the underlying mechanisms of keyloggers and the intricacies of Windows' input handling, Nakajima provides actionable intelligence for building robust detection capabilities. Her work highlights the continuous cat-and-mouse game between attackers and defenders, emphasizing the need for deep system understanding and innovative detection strategies to combat evolving threats.

The persistent abuse of keyloggers by malware families like Agent Tesla underscores the critical importance of effective detection. Early identification of these tools is paramount, as stolen credentials and sensitive data can rapidly escalate into financial fraud, intellectual property theft, or further sophisticated cyberattacks. This talk serves as a crucial guide for understanding and mitigating one of the most fundamental data exfiltration techniques in the attacker's arsenal.

Background

▶ Watch: Introduction: Windows keylogger detection and EDR focus (0:00)

Keyloggers, software designed to record keystrokes, have been a staple in an attacker's toolkit for decades. While they can have legitimate uses, their primary abuse lies in stealing sensitive information such as passwords, credit card details, and personal communications. The talk focuses exclusively on software keyloggers operating in user mode on Windows, specifically those leveraging Windows API calls.

To understand how keyloggers operate, it's essential to grasp the fundamental Windows input model, particularly on Windows 10. When a key is pressed on a USB keyboard, a HID keyboard report is sent to the kernel-side USB keyboard driver stack. This report is converted into a device-specific scan code, which the Win32k driver then translates into a virtual key code—a standardized representation of the keypress. Before reaching an application, this virtual key code is stored in the Async Key State array. Finally, it's packed into a Windows message (e.g., WM_KEYDOWN) and added to the target application's thread message queue, where the window procedure processes it.

Nakajima categorizes traditional software keyloggers into four main types:

1. Polling-based Keyloggers: These continuously check the state of each key at very short intervals. They commonly use the GetAsyncKeyState API to retrieve key states from the Async Key State array in kernel space. By doing so, they can detect which key was pressed before the next one is typed.

1. Hooking-based Keyloggers: Windows provides a hooking mechanism that allows programs to intercept certain window messages before they reach their intended application. Many keyloggers abuse this by using the SetWindowsHookEx API to intercept messages like WM_KEYDOWN as they are posted to an application's thread message queue.

1. Raw Input Model Keyloggers: Introduced in Windows XP to support a wider range of input devices, the raw input model delivers input directly to an application without prior processing by the OS. To receive raw input, an application must register the device using the RegisterRawInputDevices API. Keyloggers exploit this by secretly registering the keyboard and then collecting raw input data via the GetRawInputData API.

1. DirectInput Keyloggers: DirectInput, part of Microsoft DirectX, is typically used for multimedia tasks like gaming. It provides APIs for retrieving keyboard states. While keyloggers can leverage these APIs, Nakajima's research revealed that DirectInput internally calls the RegisterRawInputDevices API. This means DirectInput-based keyloggers fundamentally operate similarly to raw input model keyloggers.

The existence and continuous evolution of these techniques necessitate sophisticated and multi-layered detection mechanisms within EDR solutions.

Key Findings

▶ Watch: Pulling-based keyloggers using GetAsyncKeyState API (5:10)

The talk presents two primary key findings, each addressing a different facet of keylogger detection:

1. Behavior-based EDR Detection for Traditional Keyloggers via ETW: Asuka Nakajima and her team successfully developed a behavior-based detection feature for Elastic's EDR by leveraging Event Tracing for Windows (ETW). They identified that the Microsoft-Windows-Win32k kernel-level provider can trace critical API calls abused by polling, hooking, and raw input-based keyloggers. By monitoring specific ETW events associated with GetAsyncKeyState, SetWindowsHookEx, and RegisterRawInputDevices APIs, and analyzing their field data, they could reliably detect these common keylogger techniques. This approach provides a robust, low-overhead method for EDRs to identify suspicious keystroke capture activities.

1. Novel Detection Method for Hotkey-based Keyloggers: Following the release of their initial EDR feature, Nakajima encountered a new, more stealthy technique introduced by Microsoft researcher Jonathan Bar-Or: hotkey-based keyloggers. These keyloggers abuse the RegisterHotKey API to intercept keystrokes, a method not directly monitored by existing ETW providers. Nakajima's key finding here was the discovery and reverse engineering of the gpHkHashTable within Win32kfull.sys – a kernel-mode hash table that stores information about registered hotkeys. By developing a kernel-mode device driver to access and scan this table, she devised a novel detection method that identifies when all main virtual key codes are registered as hotkeys, indicating the presence of this advanced keylogger type. This finding highlights the need for deep kernel-level analysis to counter sophisticated, undocumented keylogger techniques.

Technical Deep Dive

▶ Watch: Raw Input Model keyloggers: RegisterRawInputDevices API (7:00)

The technical deep dive of this talk is divided into two distinct parts: the ETW-based detection for common keyloggers and the novel kernel-level detection for hotkey-based keyloggers.

Part 1: ETW-based Detection for Common Keyloggers

The core of the EDR's behavior-based detection relies on Event Tracing for Windows (ETW). ETW is a powerful framework for tracing and logging application and system component execution within Windows. Events are generated by providers (applications, drivers), buffered, and then consumed by tools like EDRs.

Nakajima's team found that the Microsoft-Windows-Win32k provider, a kernel-level provider, is crucial for monitoring keylogger-related API calls. When enabled, this provider emits ETW events whenever APIs like GetAsyncKeyState are called. This happens because calls to GetAsyncKeyState invoke the NtUserGetAsyncKeyState system call, which in turn triggers an ETW event within the Win32k driver.

Challenges with ETW:

Despite its power, working with ETW presents challenges:

• Event names and field definitions are not always explicitly listed in manifest files.
• Event generation conditions are often undocumented.
• Events and fields can vary significantly across different Windows versions.

To overcome these, Nakajima and her team engaged in reverse engineering and extensive testing, calling relevant APIs to observe generated events and cross-referencing with published research.

Identified ETW Events and Detection Rules:

After rigorous investigation, three key ETW events from the Microsoft-Windows-Win32k provider were identified as effective for keylogger detection:

1. GetAsyncKeyState Event (for Polling-based Keyloggers):

• Relevant Fields: MsSinceLastKeyEvent (milliseconds since the last key event) and BackgroundCallCount (total GetAsyncKeyState calls, including unsuccessful ones, since the last successful keypress capture).
• Detection Rule: A keylogger is suspected if BackgroundCallCount is 400 or higher. This threshold indicates an abnormal frequency of polling attempts.

1. SetWindowsHookEx Event (for Hooking-based Keyloggers):

• Relevant Field: FilterType (represents the type of hook procedure specified).
• Detection Rule: An alert is raised if FilterType is 13. This value specifically corresponds to a low-level keyboard hook (WH_KEYBOARD_LL), a common technique used by hooking-based keyloggers to intercept all keyboard input system-wide.

1. RegisterRawInputDevices Event (for Raw Input/DirectInput Keyloggers):

• Relevant Fields: Usage (indicates the device type being registered for raw input) and Flag (defines settings for raw input data collection).
• Detection Rule: The rule first checks if the Usage field identifies the registered device as a keyboard. If so, it then scrutinizes the Flag field. An alert is triggered if the Flag includes the RI_INPUT_SYNC flag. This flag setting allows an application to capture key inputs even when it is not in the foreground, a characteristic behavior of keyloggers aiming for stealth and persistence.

Part 2: Novel Detection for Hotkey-based Keyloggers

The second part of the talk addresses a more advanced keylogging technique introduced by Jonathan Bar-Or, leveraging the Windows hotkey mechanism. A hotkey is a keyboard shortcut that invokes a specific function, like Alt+Tab for task switching. Windows allows custom hotkeys to be registered using the RegisterHotKey API.

Hotkey Keylogger Mechanism:

1. Registration: The keylogger registers each virtual key (e.g., 'A', 'B', 'C') as a system-wide hotkey using RegisterHotKey.
2. Interception: When a user presses a registered key (e.g., 'J'), a WM_HOTKEY message containing the virtual key code is sent directly to the keylogger's thread message queue.
3. Simulation: To prevent detection and ensure normal system operation, the keylogger temporarily unregisters the hotkey using UnregisterHotKey, simulates the keypress using the KeyboardEvent API, and then re-registers the hotkey. This makes the keystroke appear normal to the user.

The Challenge: ETW Limitations:

A significant hurdle for detecting hotkey-based keyloggers is that ETW, in its current state, does not monitor the RegisterHotKey or UnregisterHotKey APIs. Nakajima demonstrated this by comparing the compiled code for NtUserGetAsyncKeyState (which includes an ETW logging call) with NtUserRegisterHotKey (which does not).

The Solution: Kernel Memory Analysis of gpHkHashTable:

Undeterred by ETW's limitations, Nakajima hypothesized that information about registered hotkeys must be stored somewhere in the kernel. Her investigation led to the discovery of gpHkHashTable within the Win32kfull.sys driver. This is a kernel-mode hash table that stores HOTKEY objects, each containing details like the virtual key code and modifiers specified during registration.

• HOTKEY Object Structure: Each HOTKEY object holds crucial information, including the virtual key code and any modifiers (e.g., Shift, Ctrl, Alt).
• gpHkHashTable Structure: The hash table uses virtual key code MOD HEX 80 as an index. HOTKEY objects that share the same index are linked together in a list, allowing the system to manage multiple hotkeys even with identical virtual key codes but different modifiers.

Building the Detection Tool:

Accessing gpHkHashTable presented several challenges:

1. Kernel Space Access: gpHkHashTable resides in kernel space, making it inaccessible from user-mode applications. This necessitated the development of a device driver for detection.

1. Obtaining gpHkHashTable Address: The address of gpHkHashTable is not directly exported. Nakajima reverse-engineered Win32kfull.sys and found that the IsHotkey function (an internal, unexported function) accesses gpHkHashTable via a LEA instruction at its beginning. She used the opcode byte sequence hex 48 8D and a 32-bit/4-byte offset from this instruction as a signature to locate the table's address.

• To find IsHotkey, she first located the exported xxxIsHotkey function, which calls IsHotkey, and used the call instruction as a signature.
• The overall process involves:

1. Determining the base address of Win32kfull.sys using PsLoadedModuleList.
2. Resolving xxxIsHotkey's address using RtlFindExportedRoutineByName.
3. Finding IsHotkey's address within xxxIsHotkey via a call instruction signature.
4. Finally, locating gpHkHashTable's address via the LEA instruction signature within IsHotkey.

1. Session Drivers: Win32kfull.sys is a session driver, meaning its data (like hotkey information) is isolated per user session. To access hotkey information for a specific user, the detection tool needed to attach to a GUI process within that target session. Nakajima used KeStackAttachProcess to temporarily attach the current thread to the address space of winlogon.exe (responsible for user login operations), assuming only one user is logged in.

Detection Logic:

The developed hotkey keylogger detector scans all HOTKEY objects in gpHkHashTable. If all alphanumeric keys are registered as hotkeys, it raises an alert. While the demo focused on alphanumeric keys for simplicity, the approach can easily be extended to check all virtual key codes with modifiers.

Demo / Proof of Concept

▶ Watch: Direct Input keyloggers and their raw input connection (8:10)

Asuka Nakajima showcased two distinct demonstrations, each illustrating the effectiveness of her detection methodologies against different classes of keyloggers.

Demo 1: ETW-based API Monitoring for Polling Keyloggers

In the first demonstration, Nakajima illustrated the behavior-based detection feature for traditional keyloggers. She initiated a polling-based keylogger in the background. Subsequently, she launched a custom monitoring tool developed to capture Win32k ETW events. As she typed "notepad" into a text editor, the monitoring tool immediately displayed a large number of GetAsyncKeyState events. This visual surge of events, particularly related to the BackgroundCallCount exceeding the established threshold, clearly indicated that a keylogger was actively monitoring and capturing keystrokes in the background. The demo effectively validated the EDR's capability to detect common API-abusing keyloggers through ETW event analysis.

Demo 2: Kernel-level Detection for Hotkey-based Keyloggers

The second demonstration showcased the novel hotkey-based keylogger detector. Nakajima first ran a "hotkey" keylogger, the proof-of-concept tool developed by Jonathan Bar-Or. She then installed and executed her specialized hotkey-based keylogger detector. Immediately upon execution, the detector began scanning the kernel's gpHkHashTable. Within moments, it successfully identified the presence of the hotkey keylogger and raised an alert. This demo powerfully demonstrated the ability of her kernel-level analysis tool to detect sophisticated keyloggers that evade traditional user-mode API monitoring, highlighting the necessity of deep system introspection for advanced threats.

Defensive Implications

▶ Watch: Detailed explanation of ETW logging mechanism (11:00)

The insights presented by Asuka Nakajima offer critical guidance for defenders looking to strengthen their posture against keyloggers. The talk underscores that a multi-layered approach is essential, as different keylogger techniques require distinct detection strategies.

1. Enhance EDR with ETW-based API Monitoring: EDR solutions should integrate robust monitoring of the Microsoft-Windows-Win32k ETW provider. Specifically, defenders should configure their EDRs to alert on:

• Excessive calls to GetAsyncKeyState (e.g., BackgroundCallCount over 400), indicating polling.
• Registration of low-level keyboard hooks (SetWindowsHookEx with FilterType 13).
• Registration of raw input devices for keyboards with the RI_INPUT_SYNC flag, which allows background keystroke capture.

This provides effective, low-overhead detection for most common user-mode keyloggers.

1. Consider Kernel-Level Monitoring for Advanced Threats: The hotkey-based keylogger highlights a significant blind spot for many conventional EDRs. Defenders need to recognize that advanced adversaries may employ techniques that bypass user-mode API hooks or standard ETW providers. While implementing a full kernel-mode device driver for detection is complex and typically falls within the purview of specialized security vendors, organizations should:

• Prioritize EDRs with Kernel Visibility: Evaluate EDR products for their ability to perform deep kernel-level introspection, even if it's for specific, high-risk behaviors.
• Monitor for Driver Installation: Be vigilant for the installation of unsigned or suspicious kernel drivers, as these are necessary for techniques like hotkey keylogger detection and other kernel-mode attacks.

1. Educate Users and Implement Strong Authentication: While technical controls are paramount, user education remains vital. Phishing and social engineering often precede keylogger deployment. Additionally, implementing multi-factor authentication (MFA) significantly mitigates the impact of stolen credentials, even if a keylogger successfully exfiltrates a password.

1. Virtual Keyboards (Limited Protection): During the Q&A, a question arose about the efficacy of virtual keyboards in protecting against keyloggers. Nakajima implicitly confirms that virtual keyboards offer limited protection against the types of keyloggers discussed. Since virtual keyboard input still generates Windows messages and interacts with the input subsystem (albeit without physical key presses), advanced keyloggers (especially hooking or kernel-level ones) can still intercept this input. The system still needs to process the "key press" from the virtual keyboard, meaning the same APIs and kernel structures can be targeted. Therefore, relying solely on virtual keyboards for sensitive input is not a sufficient defense against sophisticated keyloggers.

Key Takeaways

• Keyloggers remain a critical threat: Even traditional keylogger techniques are frequently abused by modern malware.
• ETW is powerful for behavior-based detection: The Microsoft-Windows-Win32k provider offers crucial visibility into GetAsyncKeyState, SetWindowsHookEx, and RegisterRawInputDevices for EDRs.
• Deep system understanding is essential: Overcoming ETW limitations required reverse engineering and extensive testing to understand undocumented events and their behavior.
• New keylogger techniques emerge: Hotkey-based keyloggers demonstrate the continuous evolution of evasion tactics, bypassing standard API monitoring.
• Kernel-level analysis is necessary for advanced threats: Detecting hotkey keyloggers required a novel approach involving kernel memory scanning of gpHkHashTable within Win32kfull.sys.
• Multi-layered defenses are paramount: A combination of robust EDR capabilities and kernel-level introspection is needed to counter the full spectrum of keylogger threats.

About the Speaker(s)

Asuka Nakajima is a Senior Security Research Engineer at Elastic, specializing in endpoint security with a focus on EDR research and development. With over a decade of experience in cybersecurity, Asuka is a prominent figure in the industry. She is the founder of C for Girls, the first information security community for women in Japan. Additionally, her expertise is recognized through her roles as a review board member for prestigious conferences such as Black Hat USA, Black Hat Asia, and Nullcon. Her work consistently contributes to advancing endpoint protection and understanding complex threat landscapes.