Protecting Snowflake and Critical Data Systems from Unauthorized Access

Protecting Snowflake and Critical Data Systems from Unauthorized Access

Speakers: Shelley (Product Manager, Opal Security), Stephen Spano (Engineer, Opal Security)

Conference: BSides NYC

YouTube: https://www.youtube.com/watch?v=gzXMwnCc0S8

Overview

In an era where data is often a company's most valuable asset, securing critical data systems like Snowflake against unauthorized access has become paramount. This talk, delivered by Shelley, a Product Manager, and Stephen Spano, an Engineer, both from Opal Security, delves into the inherent vulnerabilities of high-value data targets and presents robust strategies for their protection. The speakers highlight the increasing sophistication of attackers targeting cloud data platforms and underscore the necessity of proactive, automated security measures.

The presentation provides a comprehensive framework for organizations to identify, centralize, and optimize access controls for their "crown jewels." It dissects the intricacies of Snowflake's flexible access model, explaining how custom roles and hierarchies can be leveraged to enforce granular permissions. A significant portion of the talk is dedicated to the practical implementation of just-in-time (JIT) access, a critical security control designed to minimize the window of opportunity for attackers by granting temporary, need-based privileges. This article will explore the architectural considerations and lessons learned from building such a system, offering actionable insights for defenders.

The relevance of this topic is underscored by recent high-profile breaches, such as the Ticketmaster incident, which saw attackers exfiltrate significant volumes of customer data from Snowflake environments. This talk serves as a timely reminder that while platforms like Snowflake offer powerful data capabilities, the ultimate responsibility for securing access lies with the customer. By implementing the best practices and technical solutions discussed, organizations can significantly bolster their defenses against data exfiltration and unauthorized access, safeguarding their most sensitive information.

Background

▶ Watch: Introduction to speakers and talk on Snowflake security. (0:00)

The concept of "crown jewels" or "high-value targets" (HVT), as defined by NIST, is central to understanding the motivation behind securing critical data systems. These systems typically house sensitive information such as customer data (PII), intellectual property (proprietary code, algorithms), financial records, HR employee data, and cryptographic key material. They also include data protected under regulations like PCI and GDPR, or business-critical services that are essential for a company's operations. The financial and reputational damage from a breach of these systems can be catastrophic, as exemplified by the Ticketmaster incident, where an estimated $500,000 worth of data was compromised.

Securing access to these HVTs typically involves a three-phase process:

1. Identification: Inventorying all critical data systems, which can range from SaaS applications, cloud providers (AWS S3, GCP BigQuery), internal tools, to specific Snowflake databases or tables.
2. Centralization and Best Practices: Implementing fundamental security controls such as multi-factor authentication (MFA) and single sign-on (SSO) for authentication. For authorization, practices like just-in-time (JIT) access and tracking human and non-human identities are crucial. Culturally, organizations must foster an environment where access is requested only when absolutely necessary.
3. Optimization: Establishing robust detection and response capabilities, supporting continuous and conditional access, and potentially using behavioral or cohort analysis to right-size privileges and roles.

Snowflake, as a data-rich platform, has become a prime target for attackers due to the sheer volume and sensitivity of data it often hosts. Recent incidents, such as the Mandiant-attributed attacks involving InfoStealer campaigns to acquire Snowflake credentials, highlight the urgency. These campaigns enabled attackers to gain unauthorized access to customer environments, read, and exfiltrate sensitive data. While such incidents are inevitable in the cybersecurity landscape, the speakers emphasize that controls like MFA, short-lived access, and finely scoped roles could significantly limit the blast radius of a compromise.

To counter these threats, a set of best practices for Snowflake access security has been compiled, drawing from Snowflake's own documentation, CIS benchmarks, and other industry standards. These include:

• Enabling MFA and SSO organization-wide.
• Implementing role-based access control (RBAC) via custom roles.
• Configuring account-level or user-level network policies for defense in depth.
• Strictly controlling the assignment of highly privileged roles like ACCOUNTADMIN and SECURITYADMIN.
• Implementing "use it or lose it" policies, revoking access for inactive users (e.g., after 90 days).
• Setting up just-in-time or time-bound access for privileged roles. The talk particularly focuses on the practical implementation of this last point, recognizing that while best practices are readily available, their actual deployment requires specific technical effort.

Key Findings

▶ Watch: Overview of securing access: identify, centralize, optimize. (2:20)

The talk's key findings revolve around the critical importance of robust access control for Snowflake and other data systems, and the practical implementation of just-in-time (JIT) access as a foundational defensive strategy.

First, the speakers underscore that Snowflake's flexible access model, built around users, secureable objects, privileges, and especially roles, provides a powerful foundation for granular control. The ability to create custom roles and organize them into role hierarchies allows organizations to align permissions precisely with job functions and access needs. This composable nature, where roles inherit privileges from those below them, is a double-edged sword: it allows for powerful, fine-grained control but also necessitates careful management to prevent over-privileging.

Second, the talk identifies access sprawl as a significant and common vulnerability in traditional access management workflows. The manual, ticket-based process for granting access often leads to delays, repetitive work for administrators, and, crucially, long-standing, unrevoked permissions. This sprawl creates a larger attack surface, as dormant or forgotten privileges can be exploited by attackers. The just-in-time (JIT) access model is presented as the primary solution to this problem, ensuring that permissions are temporary and automatically revoked, thereby drastically reducing the window of exposure.

Third, the speakers share practical insights and lessons learned from developing a JIT system for Snowflake. They highlight the trade-offs between using SHOW commands and account usage views for retrieving information about users, roles, and grants. While SHOW commands are cost-effective as they don't require a running warehouse and are suitable for smaller organizations (up to 10,000 rows), they offer limited filtering. Account usage views, conversely, provide comprehensive data but come with a two-hour latency for the latest system state. This finding is crucial for organizations designing their JIT systems, as the choice impacts real-time visibility and operational costs.

Finally, a critical security finding emphasized is the necessity of using bind variables when interacting with Snowflake's REST API, especially when incorporating user input into queries. This practice is vital for preventing SQL injection attacks, a fundamental security vulnerability that can arise when building automated systems that process dynamic query components.

In essence, the talk finds that while Snowflake offers a strong native access control framework, its full security potential is only realized through the diligent application of best practices, particularly the automation of short-lived, JIT access, and careful architectural choices to prevent common vulnerabilities.

Technical Deep Dive

▶ Watch: Analyzing the Ticketmaster Snowflake data breach incident. (4:10)

Snowflake's access model is built on a robust framework designed for granular control, revolving around four core entities: users, secureable objects, privileges, and roles.

• A user represents any human or non-human identity in Snowflake.
• A secureable object is any Snowflake entity on which permissions can be granted or revoked, such as a database, schema, or table.
• A privilege defines an operation that can be performed on a secureable object (e.g., SELECT on a table).
• Roles are collections of privileges. Crucially, privileges cannot be granted directly to users; they must be granted to roles, which are then assigned to users.

The power and flexibility of Snowflake's access model stem from custom roles and their ability to form role hierarchies. System-defined roles, like ACCOUNTADMIN or SYSADMIN, exist at the top. Below these, custom roles can be created, such as read-only or engineering. Roles higher in the hierarchy automatically inherit all privileges of the roles below them. For example, ACCOUNTADMIN might inherit from SYSADMIN, which in turn inherits from engineering.

A common and recommended approach to managing roles involves differentiating between access roles and functional roles:

• Access roles (e.g., read-only, read-write) are typically designed to contain only specific privileges and are often at the lower levels of the hierarchy.
• Functional roles (e.g., engineering, sales) align with job functions. These roles usually inherit their privileges from access roles and are then assigned to users or other functional roles. This allows for a clean separation of concerns: access roles define what can be done, while functional roles define who can do it based on their job. An engineering role might inherit both read and write privileges, while a sales role might only inherit read privileges.

Implementing just-in-time (JIT) access at scale for Snowflake involves several key technical components and steps:

1. Service Account Configuration:

• Authentication Mechanism: Snowflake recommends key pair authentication for service accounts. This involves generating a public/private key pair. The private key is used to sign a JSON Web Token (JWT), which is then passed as a bearer token to authenticate requests to Snowflake's REST API. A significant advantage is that a user can be associated with two public keys, enabling seamless key rotation without downtime.
• Service Account Role: A dedicated role should be created for the service account. This role should be granted the minimum necessary privileges, primarily the MANAGE GRANTS privilege, which allows it to grant and revoke roles. This dedicated role ensures better auditing and separation from human user roles.
• Service Account User: A specific user is created to act as the service account, assigned the public key, and granted the dedicated service account role.

1. Granting and Revoking Roles:

• Snowflake provides simple SQL commands for managing grants:
• GRANT ROLE <role_name> TO USER <user_name>; for granting access.
• REVOKE ROLE <role_name> FROM USER <user_name>; for revoking access.
• SHOW GRANTS OF ROLE <role_name>; for viewing existing grants. These commands form the core logic of the JIT system.

1. JIT System Architecture:

A sample architecture for a JIT system involves several interacting components:

• Request Server: Receives and processes access requests from employees (e.g., via a web portal).
• Database: Stores details of approved requests, including the role, requested duration, and status.
• Job Queue: Enqueues tasks for granting or revoking access.
• Worker: Picks up jobs from the queue and executes the appropriate GRANT ROLE or REVOKE ROLE command against Snowflake using the configured service account.
• Cron Job/Scheduler: Periodically checks the database for expired grants and queues REVOKE jobs for them, ensuring automatic access revocation.

1. Lessons Learned:

• Retrieving User/Role/Grant Information: Organizations must choose between SHOW commands and account usage views.
• SHOW commands (e.g., SHOW GRANTS OF USER <user_name>) offer limited filtering and a maximum of 10,000 rows. Their advantage is that they don't require a running warehouse, thus saving credits.
• Account usage views (e.g., SNOWFLAKE.ACCOUNT_USAGE.GRANTS_TO_USERS) provide comprehensive data but have a significant two-hour latency for the latest system state. The choice depends on the organization's scale and need for real-time data versus cost-effectiveness.
• SQL Injection Prevention: When building systems that interact with Snowflake's REST API and incorporate user input into queries, it is absolutely essential to use bind variables. This prevents malicious input from altering the query structure and executing unintended commands.

This technical deep dive illustrates that while Snowflake provides the primitives, building a scalable and secure JIT access system requires careful architectural design, adherence to least privilege principles for service accounts, and awareness of platform-specific nuances like data retrieval mechanisms and SQL injection vectors.

Demo / Proof of Concept

▶ Watch: Preparing for a practical implementation of just-in-time access. (6:40)

While a live demonstration was not explicitly performed in the talk, Stephen Spano walked through a detailed sample architecture that serves as a robust proof of concept for implementing just-in-time (JIT) access in a Snowflake environment. This architecture outlines the components and workflows necessary to automate the granting and revocation of temporary, role-based access.

The core of the "demo" architecture illustrates two primary workflows:

1. Granting Access Workflow:

• An employee initiates an access request, typically through a web portal, specifying the desired role, duration, and justification.
• This request is received by a server component, which might involve an approval process (manual or automated, depending on the sensitivity of the resource).
• Once approved, the request details (user, role, duration) are stored in a database.
• A job is then enqueued into a message queue.
• A worker process picks up this job.
• The worker, authenticated as the specially configured service account, issues a GRANT ROLE command to Snowflake's REST API.
• Upon successful execution, the user is provisioned with the requested role for the specified duration.

1. Expiring and Forfeiting Access Workflow:

• Forfeiting Access: If a user finishes their task early, they can proactively forfeit their access (e.g., via a button in the web portal). This triggers a similar process: a REVOKE job is enqueued, picked up by a worker, and the REVOKE ROLE command is executed in Snowflake.
• Expiring Access: To handle automatic revocation, a cron job or scheduler is configured. This job periodically queries the database for all grants that have passed their expiration time. For each expired grant, a REVOKE job is queued.
• The worker then processes these REVOKE jobs, issuing the REVOKE ROLE command to Snowflake, thereby automatically removing the temporary role from the user.

The architecture emphasizes the role of the service account, which is configured with key pair authentication and granted only the MANAGE GRANTS privilege. This account acts as the automated agent for all grant and revoke operations, ensuring that human administrators are removed from the "hot path" of access provisioning.

This conceptual demonstration highlights how organizations can leverage Snowflake's native access control capabilities, combined with custom automation, to create a scalable, secure, and efficient JIT access system. It addresses the common pain points of slow, manual access requests and the perennial challenge of access sprawl by automating the entire lifecycle of temporary privileges.

Defensive Implications

▶ Watch: High-level overview of Snowflake's flexible access model. (7:25)

The insights from this talk offer several critical defensive implications for organizations aiming to secure their Snowflake environments and other critical data systems.

1. Prioritize "Crown Jewel" Identification and Inventory: Defenders must start by meticulously identifying and inventorying their high-value targets (HVTs). This includes not just Snowflake instances but also S3 buckets, BigQuery datasets, internal tools, and any system containing sensitive customer data, intellectual property, or business-critical information. Understanding what needs protecting is the first step to effective defense.

1. Enforce Foundational Security Controls: The talk reiterates the non-negotiable importance of fundamental security controls. Multi-factor authentication (MFA) and single sign-on (SSO) should be universally enforced for all users accessing Snowflake. Additionally, implementing account-level and user-level network policies provides an essential layer of defense-in-depth, restricting access to trusted IP ranges or VPNs, even if credentials are compromised.

1. Adopt a Least Privilege, Just-in-Time Access Model: This is perhaps the most significant defensive implication. Organizations must move away from long-standing, broad access grants. Instead, they should:

• Implement custom roles in Snowflake that align precisely with job functions and grant only the absolute minimum privileges required.
• Automate just-in-time (JIT) access for all privileged roles. This means access is granted only when needed, for a specific duration, and automatically revoked. This drastically shrinks the window of opportunity for attackers, even if they manage to acquire valid credentials.
• Control highly privileged roles: Strictly limit the assignment of roles like ACCOUNTADMIN and SECURITYADMIN. When these roles are needed, they should be subject to JIT access.

1. Implement "Use It or Lose It" Policies: Proactively revoke access for inactive users or those who haven't utilized their granted permissions within a defined period (e.g., 90 days). This helps to combat access sprawl and reduces the attack surface presented by dormant accounts.

1. Build Robust Automation and Auditing:

• Automate grant and revocation workflows: Leverage service accounts with key pair authentication and minimal privileges to programmatically manage Snowflake roles. This reduces human error, improves efficiency, and ensures consistent application of policies.
• Leverage Snowflake's auditing capabilities: Regularly review SHOW commands and account usage views to monitor who has access to what, when, and for how long. Be aware of the two-hour latency for account usage views when designing real-time detection systems.
• Implement SQL Injection prevention: When building custom tools or integrations with Snowflake's REST API, always use bind variables to prevent SQL injection vulnerabilities, especially when handling user-supplied input.

1. Focus on Detection and Response: Beyond preventative measures, organizations need robust detection and response capabilities. This includes monitoring for anomalous access patterns, unusual data exfiltration attempts, or unauthorized changes to roles and permissions. The JIT system's logs can be invaluable for forensic analysis during an incident.

By integrating these defensive strategies, particularly the shift towards automated, temporary, and least-privilege access, organizations can significantly harden their Snowflake environments against the sophisticated threats targeting cloud data platforms today.

Key Takeaways

• Critical data systems are prime targets: Organizations must identify their "crown jewels" (e.g., PII, IP, financial data) and prioritize their protection, as exemplified by the Ticketmaster breach costing an estimated $500,000 in data compromise.
• Snowflake's access model is powerful but requires careful management: Leverage custom roles and role hierarchies to implement fine-grained, role-based access control (RBAC) that aligns with least privilege principles.
• Just-in-time (JIT) access is essential to combat access sprawl: Implement automated systems to grant temporary, time-bound access to privileged roles and revoke it automatically, significantly reducing the attack surface.
• Foundational security controls are non-negotiable: Enforce MFA, SSO, and network policies for all Snowflake users to build a strong defense-in-depth strategy.
• Automate access management with dedicated service accounts: Utilize key pair authenticated service accounts with minimal privileges (e.g., MANAGE GRANTS) to programmatically handle role assignments and revocations, taking human administrators out of the "hot path."
• Be aware of Snowflake's data retrieval nuances: Understand the trade-offs between SHOW commands (cost-effective, limited data) and account usage views (comprehensive, two-hour latency) when building auditing and monitoring systems. Always use bind variables to prevent SQL injection in custom integrations.

About the Speaker(s)

Shelley is a Product Manager at Opal Security. Prior to her current role, she gained extensive experience in various security domains, including infrastructure security, platform security, SecOps, and detection and response at multiple companies. She is an active participant and enthusiast of the B-sides security conference community, with involvement in events in San Francisco, Las Vegas, and New York.

Stephen Spano is an Engineer at Opal Security. His background includes building and securing servers and developer platforms. In his current role, he contributes to the technical development of Opal Security's offerings, focusing on securing critical data systems.