Building canaries with ELK and ElastAlert2

Building canaries with ELK and ElastAlert2

Speakers: Andrew Januszak (Lehigh University); Keith Erekson (Lehigh University)

Conference: BSides NYC

YouTube: https://www.youtube.com/watch?v=IAQIOaYHKRM

Overview

In "Building Canaries with ELK and ElastAlert2," Andrew Januszak and Keith Erekson from Lehigh University present a practical, cost-effective approach to enhancing organizational security through the strategic deployment of canaries and honey tokens. As members of a systems engineering team, their focus is on leveraging existing infrastructure to create "low effort, high gain" defensive tools that provide immediate, event-driven alerts when something unexpected or malicious occurs. This talk delves into how their team implemented these proactive measures using the ELK Stack (Elasticsearch, Logstash, Kibana) and ElastAlert2, specifically targeting gaps left by traditional security controls and vendor tooling.

The core premise of their work is to shift from reactive incident response to proactive detection. While conventional security monitoring often relies on analysts reviewing dashboards to identify past events, canaries enable organizations to catch malicious activity as it happens, or even before it fully escalates. By strategically placing fake resources or embedding hidden triggers, Januszak and Erekson demonstrate how to create an early warning system that provides critical time to respond to threats like phishing campaigns and unauthorized access attempts, ultimately making their computing infrastructure more resilient.

This approach is particularly relevant in today's threat landscape, where sophisticated attackers constantly seek to bypass standard defenses. The speakers emphasize that their solution fills the void where EDR, AV, and other vendor tools are "ineffective or can't touch" certain attack vectors. Their journey, inspired by industry leaders and other security researchers, showcases how an in-house, open-source-driven solution can deliver significant security value and empower blue teams with actionable intelligence.

Background

▶ Watch: Introduction: What are canaries and honey tokens? (0:00)

The concept of using canaries and honey tokens in cybersecurity is not new, drawing its terminology from historical intelligence and espionage practices. The speakers reference "canary traps," where slightly altered documents are given to different individuals to identify the source of a leak, and "honey tokens," which are fake credentials or data designed to lure attackers. This foundational idea—using a decoy to detect unauthorized access—forms the basis of their defensive strategy.

A significant impetus for their project came from the cybersecurity community. At the 2023 RSA Conference, Kevin Mandia, CEO of Mandiant, highlighted the importance of building honeypots as one of seven pieces of advice for organizations to avoid attacks requiring incident response. Mandia's observation that these tools fill critical gaps where traditional security vendor solutions are ineffective resonated deeply with the Lehigh team. Further inspiration was drawn from Jacob Torrey's talk at ShmooCon 2023, which focused on defending by understanding attacker incentives and the utility of canary tokens. These external validations spurred Januszak and Erekson to explore building their own canary system rather than relying on "canary as a service" offerings, leveraging their existing infrastructure.

Lehigh University already possessed a robust ELK Stack (Elasticsearch, Logstash, Kibana) that had been in use for over a decade for centralized log aggregation and analysis. While security analysts monitored dashboards, this provided a retrospective view, often identifying issues days or weeks after they occurred. The need for immediate, event-driven alerting for specific, high-priority scenarios became clear. This necessity led them to integrate ElastAlert2, an open-source alerting engine, into their existing ELK environment. The goal was to transform their log data from a historical record into a real-time threat detection system, moving beyond traditional security controls to proactively identify and respond to malicious activity.

Key Findings

▶ Watch: Inspiration for using canaries: Mandiant CEO & ShmooCon talk (2:00)

The Lehigh University team’s implementation of canaries with ELK and ElastAlert2 yielded several significant findings, demonstrating the high value of this low-effort defense strategy:

• Filling Security Gaps: Canaries proved highly effective in detecting malicious activities that their existing EDR, AV, and other vendor security tools either missed or couldn't effectively monitor. They specifically targeted areas prone to phishing and unauthorized access, creating an early warning system for sophisticated attacks.
• Proactive Threat Intelligence: The system provided immediate, event-driven alerts, enabling the security team to identify and respond to threats in real-time. For instance, JavaScript canaries on their Single Sign-On (SSO) page allowed them to detect attackers testing cloned phishing pages an "hour or two" before mass email campaigns were launched, providing a crucial window for defensive action.
• Attacker Behavior Insights: The canaries offered invaluable insights into attacker Tactics, Techniques, and Procedures (TTPs). The team observed attackers cloning their login pages, testing them from various global locations (e.g., South Africa, Belgrade), and sometimes even mixing and matching login components from different universities, highlighting their laziness and reliance on automated scraping.
• Actionable Alerts: Despite initial challenges with false positives, the system was successfully tuned to provide highly actionable alerts. Once refined, the alerts were specific enough to prompt immediate investigation and response, such as blocking malicious DNS entries or implementing email filter rules.
• Overcoming Internal Skepticism: Initial pushback from internal security teams, concerned about alert fatigue and the legitimacy of "fake" accounts, quickly dissipated once the canaries started producing tangible results. The immediate detection of active phishing campaigns demonstrated the undeniable value of the system.
• Versatility of Canaries: The project demonstrated that almost any resource or loggable event could be transformed into a canary. From fake Active Directory accounts to specific URL paths and admin login pages, the potential for creating custom detection mechanisms is vast, limited only by creativity.

Technical Deep Dive

▶ Watch: Choosing ELK Stack and ElastAlert2 for canaries (4:00)

The technical backbone of Lehigh University's canary system is the ELK Stack combined with ElastAlert2, providing a powerful, flexible, and event-driven alerting platform.

The ELK Stack forms the data ingestion, storage, and visualization layer:

• Elasticsearch serves as the distributed, RESTful search engine and data store. It's where all the log data from various sources, including canary hits, is indexed and made searchable.
• Logstash acts as the dynamic data pipeline, responsible for ingesting, transforming, and enriching logs before they are sent to Elasticsearch. It can parse raw log entries, extract relevant fields (like IP addresses, user agents, referrer domains), and apply filters, making the data more structured and useful for alerting.
• Kibana is the visualization layer, offering dashboards and graphs for exploring and analyzing the data stored in Elasticsearch. While the goal was to move beyond retrospective dashboard monitoring for critical alerts, Kibana remains essential for deeper analysis of canary hits and tuning.

It's important to note that while Lehigh leveraged their full ELK Stack, the speakers clarified that a complete ELK deployment isn't strictly necessary for just canaries. Any centralized log aggregation system that supports real-time search and alerting, such as OpenSearch, Splunk, or potentially Greylog, could serve a similar purpose, provided it is event-driven—meaning it can trigger actions immediately upon log ingestion, rather than relying on delayed human review.

The critical component for real-time alerting is ElastAlert2. This is a community-maintained, drop-in replacement for the original ElastAlert, which had been abandoned. ElastAlert2 operates by running configurable queries against Elasticsearch at regular intervals, typically every minute. Its core features include:

• YAML Rule Files: Alerts are defined using simple YAML configuration files. These files specify the Elasticsearch query to run, the conditions for triggering an alert (e.g., specific event types, thresholds), and the action to take.
• Thresholding: Rules can incorporate thresholding logic, such as "only alert me if there have been more than 50 of these events in some time frame," preventing excessive alerts from minor anomalies.
• Extensible Alerting Mechanisms: ElastAlert2 boasts over 37 different plugins for sending alerts. Lehigh primarily utilizes email and Slack for immediate notifications, though other options like opening Jira tickets exist (which they actively avoid for this purpose).

A simple YAML rule example provided in the talk illustrates the structure for detecting cron entry errors:

This example, while not a canary, demonstrates how ElastAlert2 can be configured to immediately flag specific log messages.

Lehigh's primary canary implementations included:

1. Windows Event ID Canaries: They set up fake Active Directory accounts (e.g., a fake domain admin account) that should never be logged into. Any login attempt to these accounts triggers an immediate alert via ElastAlert2, leveraging specific Windows Event IDs logged to Elasticsearch. These accounts are configured with strict restrictions (e.g., "restricted hours to never") to ensure they cannot be genuinely used, mitigating internal security audit concerns.
2. SSO Login Page JavaScript Canary: This was a highly effective measure against phishing. A few lines of JavaScript were injected into their university's Single Sign-On (SSO) login page. This script was designed to trigger a "canary hit" only if the page was loaded from a domain that was not their legitimate university domain. The script would capture the loading domain and the referrer (similar to how Google Tag Manager collects analytics) and send this information back to their ELK stack.

• This canary successfully caught instances where attackers cloned their login page and hosted it on malicious domains like micro-desktop-state.top/leehigh.edu.
• The referrer information was crucial, showing, for example, that a phishing page was being tested from South Africa or Belgrade before mass email campaigns were sent to students and faculty.
• Observations included attackers being "pretty lazy," often just cloning pages, sometimes even de-obfuscating and still including the canary code, and occasionally mixing elements from different universities' login flows.

The team also mentioned exploring more advanced honeypot scenarios, such as using OpenCanary, which is a deployable honeypot layer that can run on a VM or Raspberry Pi, mimicking various services to trap and alert on attacker interactions.

Demo / Proof of Concept

▶ Watch: ElastAlert2 explained: Event-driven alerting with YAML (6:00)

While the talk did not feature a live, interactive demo, Andrew Januszak and Keith Erekson effectively demonstrated the efficacy of their canary system through real-world examples and screenshots of actual Slack notifications. These visual proofs of concept highlighted the immediate and actionable intelligence provided by their setup.

A prominent example involved a phishing campaign detected on Thanksgiving weekend. The Slack notification screenshot showed a canary hit originating from micro-desktop-state.top/leehigh.edu, indicating that their SSO login page had been cloned and was being served from a malicious domain. The referrer information further specified the origin as South Africa. This alert, received on a holiday when "no one's working," was critical because it showed attackers testing their infrastructure. The speakers explained that this test-run typically occurs "an hour or two before they actually send out the emails to all of our users," providing a crucial window for the Lehigh team to implement defensive measures like blocking the malicious DNS on campus or establishing email filter rules.

Another screenshot illustrated a similar incident from early December, where hits were observed from Belgrade and other global locations. This indicated either a distributed attacker team or the use of compromised servers/VPNs across different countries. Again, the system provided early warning, allowing the team to block the malicious source.

The speakers also touched upon how their fake Active Directory account canaries would trigger an alert upon any login attempt, providing the IP address from which the phished credentials were being tested. This direct evidence of attempted unauthorized access allowed for immediate blocking and further investigation.

These examples clearly served as proof of concept, demonstrating that their custom-built canaries with ELK and ElastAlert2 were not only functional but also highly effective in detecting and pre-empting real-world threats, providing concrete evidence of malicious activity and enabling proactive defense.

Defensive Implications

▶ Watch: Initial canaries: Windows Event IDs, fake AD, SSO & tuning (8:00)

The insights and practical implementation shared by Januszak and Erekson offer several crucial defensive implications for organizations looking to bolster their security posture:

1. Prioritize Low-Hanging Fruit: Defenders should start by implementing simple, high-value canaries that are easy to deploy and quickly demonstrate value. Phishing-prone areas, like cloned login pages, are excellent initial targets. The immediate results from these canaries can help overcome internal skepticism and build momentum for more complex deployments.
2. Embrace Event-Driven Alerting: Relying solely on retrospective log analysis or dashboard monitoring is insufficient for modern threats. Implementing an event-driven alerting system, like ElastAlert2, ensures that security teams are notified of suspicious activity immediately, enabling rapid response and potentially preventing incidents from escalating.
3. Tailor Canaries to Your Environment: The talk emphasizes that "anything that logs" can be a canary. Defenders should be creative, identifying unique assets, specific URL paths, or privileged accounts that, if accessed, would indicate malicious activity. Focus on areas where existing security controls are known to be weak or ineffective.
4. Strategic Communication Channels: To avoid alert fatigue and internal pushback, establish dedicated communication channels (e.g., specific Slack channels, email lists) for canary alerts. This ensures that only relevant personnel receive notifications, reducing noise for others and fostering a focused response environment.
5. Invest in Tuning and Iteration: The most significant amount of work involved in deploying canaries is tuning out false positives. This is an ongoing process that requires careful analysis of each alert ("Why did this happen? Is it legitimate? Do I have enough data?") and continuous refinement of rules. As attacker Tactics, Techniques, and Procedures (TTPs) evolve, canary rules must also be adapted.
6. Maintain Authenticity for Honeypots: For more advanced honeypot scenarios, ensure that the decoy resources appear as legitimate as possible. This includes using realistic hostnames, server naming schemes, and blending them seamlessly into the existing infrastructure to deceive attackers.
7. Be Prepared for Internal Pushback: Expect initial skepticism from security teams concerned about "more noise" or the perceived risk of "fake" accounts. However, be confident that demonstrating tangible results from early, simple canaries will quickly win over doubters.
8. Practice Operational Security (OpSec) During Investigation: When a canary triggers, and investigation leads to attacker infrastructure (e.g., an unsecured admin panel for a phishing toolkit), responders must exercise caution. Avoid probing from the organization's network; instead, use isolated environments like a guest network or a tethered mobile device to prevent inadvertently painting a larger target on your organization.
9. Anticipate and Filter Common False Positives: The team encountered various unexpected false positives, including web crawlers (e.g., from ByteDance), cache services executing JavaScript, translation services, users saving login pages locally (resulting in Windows file paths as referrers), and inexplicable traffic from sources like Bing. Building a curated list of known benign events and filtering them is essential for reducing noise and maintaining alert fidelity.
10. Consider Management Tools for Rule Creation: If multiple teams or individuals are creating ElastAlert2 rules, consider building a simple web interface to generate YAML configurations. This helps enforce consistency, prevent errors, and streamline the rule creation process, especially for users less familiar with YAML syntax.

Key Takeaways

• Canaries bridge security gaps: They provide a "low effort, high gain" defense mechanism, detecting threats that traditional security controls and vendor tooling often miss, especially proactive phishing detection.
• Event-driven alerting is critical: Leveraging tools like ElastAlert2 with log aggregation platforms (e.g., ELK Stack) enables immediate notification of suspicious activity, transforming retrospective analysis into real-time threat intelligence.
• JavaScript canaries on login pages are highly effective: Injecting simple JavaScript to detect page loads from unauthorized domains provides an early warning system for phishing campaigns, often before attackers launch mass emails.
• Tuning false positives is essential for success: Initial setup requires significant effort to filter out benign events (e.g., web crawlers, translation services), but this investment is crucial for ensuring alerts are actionable and preventing alert fatigue.
• Creativity and authenticity are key: Almost any loggable resource can become a canary. For honeypots, making them blend seamlessly into the existing infrastructure enhances their deceptive power and effectiveness.
• Tangible results overcome skepticism: Demonstrating the immediate value of canaries in catching active threats quickly builds internal support and validates the effort invested in their deployment.

About the Speaker(s)

Andrew Januszak and Keith Erekson are integral members of a systems engineering team at Lehigh University, located in Bethlehem, Pennsylvania. Their primary responsibilities involve the comprehensive management, monitoring, securing, and automation of the university's computing infrastructure. Beyond their core duties, they are self-described "purple teamy" individuals and security enthusiasts, dedicating their downtime to exploring and implementing advanced security measures. While not formally part of the university's dedicated security team, their work on projects like the ELK and ElastAlert2 canary system stems from a practical necessity to proactively identify and mitigate threats, ultimately making their operational lives easier by preventing incidents before they escalate.