Cloud's Dirty Little Secret: It Was Misconfigs All Along

Cloud's Dirty Little Secret: It Was Misconfigs All Along

Speakers: Karl Ots, Consultant, Cloud Security Lead, EPAM

Conference: BSides NYC

YouTube: https://www.youtube.com/watch?v=KU2g6OheMxM

Overview

In "Cloud's Dirty Little Secret: It Was Misconfigs All Along," Karl Ots, a Cloud Security Lead and consultant at EPAM, delivers a pragmatic and critical assessment of the persistent threat posed by cloud misconfigurations. Drawing from extensive experience as both a cloud builder and breaker, Ots argues that despite the industry's focus on advanced exploits and zero-days, simple misconfigurations remain the overwhelmingly dominant cause of data breaches in cloud environments. The talk emphasizes that this isn't a new problem, but one that continues to plague organizations due to ingrained human error, reliance on insecure defaults, and the complexity of assembling secure cloud components.

Ots's presentation serves as a wake-up call, urging security professionals and developers to shift their mindset from traditional on-premises security paradigms to a cloud-native, identity-centric approach. He dissects the problem into four "harbingers of doom," illustrating common pitfalls with practical examples across various cloud providers, particularly Azure, while stressing the universal applicability of his insights. The core message is clear: while cloud technology evolves rapidly, the fundamental security challenge often lies in how humans configure and manage these powerful, yet intricate, systems.

The significance of this talk lies in its actionable guidance for both red and blue teams. Ots provides concrete strategies, tool recommendations, and policy-as-code principles designed to audit, prevent, and respond to misconfigurations effectively. By focusing on practical, implementable solutions rather than theoretical threats, the session empowers organizations to build more resilient cloud infrastructures and move beyond the "dirty little secret" that continues to undermine cloud security efforts globally.

Background

▶ Watch: Introduction: The persistent problem of cloud misconfigurations (0:00)

The pervasive issue of cloud misconfigurations is not a novel problem; rather, it has been the leading cause of data breaches for over a decade. As Karl Ots highlights, this trend was evident ten years ago, five years ago, and remains equally pertinent today. Despite significant advancements in cloud security tools and practices, the industry continues to grapple with the same fundamental challenge. The speaker expresses a desire to move beyond discussing misconfigurations, yet the reality dictates its continued prominence as the primary attack vector.

The persistence of misconfigurations stems largely from the "human in the loop." Developers and architects, often under pressure, make mistakes when building cloud solutions. A critical contributing factor is the reliance on out-of-the-box or default configurations, which, for many vendors, are inherently not secure. Even when individual components are configured correctly, the intricate process of assembling these building blocks into a larger system can inadvertently create complex attack paths. As Ots notes, an environment with individually properly configured components can still become vulnerable when these components interact in unexpected ways.

Ots defines misconfigurations broadly, encompassing classic scenarios such as accidentally exposing a storage account to the public internet, securing resources with static keys then publishing them to GitHub, or publicly exposing a Kubernetes cluster that can be discovered by tools like Shodan. These examples underscore the range of errors, from simple oversight to complex architectural missteps, that allow attackers to gain unauthorized access or exploit vulnerabilities without relying on zero-day exploits or sophisticated techniques. The talk aims to categorize these common misconfiguration patterns, provide examples, and offer concrete solutions for remediation and prevention.

Key Findings

▶ Watch: Common examples of cloud misconfigurations (2:30)

Karl Ots structures his analysis of cloud misconfigurations around four "harbingers of doom," which he estimates cover 90% or more of typical misconfiguration scenarios. These categories highlight persistent challenges in cloud security:

1. Legacy Mindset: This refers to the practice of lifting and shifting not only applications but also security controls, mindsets, and architectures directly from on-premises environments to the cloud. Ots points out that traditional controls, such as static firewall rules or heavy, process-driven security operations, are ill-suited for the dynamic, ephemeral nature of cloud workloads. Network engineers, for instance, might lack the context of how cloud access control configurations can override network settings, leading to vulnerabilities.

• Solution: Cloud environments demand adaptation to their ephemeral nature. This means moving away from agent-based security towards event-driven models that tap into the control plane of the cloud or orchestration layers like Kubernetes. Crucially, identity-based access control should be prioritized, as identity is typically less ephemeral than compute or network resources. For complex network configurations in Kubernetes, service meshes (leveraging technologies like eBPF or sidecar proxies) can automate much of the heavy lifting.

1. Over-Permissive Access ("Just Give Them More"): This common anti-pattern arises when developers, facing blockers in new cloud environments, request more and more permissions to get their job done. Without careful management, these excessive permissions can become permanent, leading to privilege creep. This applies to both user access and machine/workload identities. Cloud environments offer a vast number of roles (e.g., Azure has ~1200 built-in roles, plus custom roles), making prescriptive, upfront design of permissions extremely challenging.

• Solution: Organizations must move towards centralized Identity Providers (IDPs) such as Azure Entra ID, Google Identity, or AWS IAM, eliminating disconnected, key-based access controls. Instead of focusing solely on the exact set of API calls allowed, prioritize defining the scope where permissions are provisioned. Tools (both commercial and open-source like air-iam from Checkmarx/Czecho) can analyze actual permission usage against provisioned roles, identifying unused or overly permissive identities (e.g., a root user booth admin unused for 192 days). This analysis should inform the implementation of Identity and Access as Code (IaC), making permission management prescriptive and auditable.

1. Disconnected Access Controls (Legacy Authentication): This category specifically addresses the use of non-centrally managed authentication methods, such as connection strings or static keys, for data services like storage accounts or databases. These legacy methods bypass the robust, centralized identity and access management systems of the cloud provider, creating isolated security perimeters that are harder to track and secure. Kubernetes environments, by default, often have cluster-local credentials that can fall into this trap.

• Solution: Actively disable or reconfigure defaults to enforce the use of centralized identity providers for all services. Where key-based authentication is genuinely required, implement strong compensating controls and lock down future key issuance. Crucially, enforce data plane level Role-Based Access Control (RBAC) or Attribute-Based Access Control (ABAC) (in GCP or AWS) to manage access to data, integrating network and other controls into role assignments. Policies are essential for enforcing these configurations.

1. Accidental Public Exposure: The classic misconfiguration: unintentionally exposing cloud resources (e.g., storage buckets, container instances, Kubernetes dashboards) to the public internet. Ots illustrates the immediate danger, noting that a container instance running in the public cloud will likely be scanned, pinged, and subjected to application-level scanning within five minutes, often leading to login attempts.

• Solution: Adopt a "disconnected by default" philosophy for all cloud deployments. This requires a multi-faceted approach involving shift-left security (pre-deployment checks), runtime monitoring, and robust policy enforcement.

Technical Deep Dive

▶ Watch: Introducing the 'Four Harbingers of Doom' for misconfigurations (3:28)

Addressing the four harbingers of doom requires a comprehensive approach leveraging cloud-native tools and methodologies. Ots emphasizes integrating security throughout the development lifecycle, from pre-deployment to runtime.

Shift-Left Security (Pre-Deployment)

The most cost-effective and easiest way to secure cloud resources is before deployment. This involves embedding security checks into the development pipeline:

• Infrastructure as Code (IaC) Scanning: If all deployments are defined as IaC, tools like Terrascan or other IaC security scanners can be used to enforce policies and identify misconfigurations in templates (e.g., Terraform, CloudFormation, ARM templates) before they are provisioned. This is analogous to Static Application Security Testing (SAST) but applied to infrastructure definitions.
• Developer Education: Tools like Czecho (now part of Checkmarx) can integrate directly into Integrated Development Environments (IDEs) like VS Code. This provides developers with real-time feedback and guidance on security best practices, preventing misconfigurations at the source rather than just breaking builds later.
• Kubernetes-Specific Scanners: For Kubernetes environments, tools such as Kube-scan or Kubescape can analyze YAML manifests for security vulnerabilities and misconfigurations.
• Version Control Integration: Integrating security scans as GitHub commit hooks or within Continuous Integration/Continuous Deployment (CI/CD) pipelines ensures that every code change is automatically checked for policy compliance.

Runtime Security and Posture Management (Post-Deployment)

Once resources are deployed, the dynamic and ephemeral nature of the cloud necessitates continuous monitoring and adaptation:

• Cloud Native Application Protection Platforms (CNAPP): This is the current "hype" term for comprehensive cloud security platforms that go beyond traditional virtual machine protection. CNAPPs offer event-based monitoring, covering a broader range of cloud assets. Cloud providers often offer their own built-in CNAPP capabilities (e.g., Azure Security Center, AWS Security Hub), which are typically the most cost-effective starting point.
• Container and Kubernetes Runtime Protection: For containerized environments, Falco is a powerful open-source tool. Falco uses eBPF to tap into the kernel of the host system, monitoring all calls from container applications to the node itself. This allows for detailed security behavioral analysis at runtime, detecting anomalous activities or policy violations.
• Posture Management and Red Teaming Tools: Tools like ScoutSuite can be used for both red teaming exercises and ongoing posture management. ScoutSuite audits cloud environments (including multi-cloud setups) to identify misconfigurations and provides a snapshot of the security posture, serving as a valuable double-check for CNAPP tooling.
• Identity and Access Management (IAM) Analysis: As mentioned, tools like air-iam are crucial for analyzing actual permission usage versus provisioned permissions, helping to identify and revoke overly permissive or unused roles and machine identities. Commercial Cloud Identity Entitlement Management (CIEM) solutions also operate in this space.

Policy as Code: The Foundation of Enforcement

Ots strongly advocates for Policy as Code as the ultimate mechanism for enforcing security controls across the cloud lifecycle. It allows organizations to define security rules in a machine-readable, version-controlled format, enabling automated auditing and prevention:

• Native Cloud Policies: Each major cloud provider offers its own policy enforcement engine (e.g., Azure Policy, AWS Service Control Policies (SCPs), GCP Organization Policies). These can audit existing configurations, enforce new deployments, and prevent non-compliant changes.
• Terraform and Kubernetes Policy Engines: For organizations using Terraform, HashiCorp Sentinel provides policy as code capabilities. In Kubernetes, Kyverno offers a similar approach, allowing administrators to define policies that govern deployments, enforce best practices, and audit cluster configurations. Kyverno supports audit-only policies, which are excellent for gaining visibility without immediately breaking existing workflows.

By combining these technical strategies, organizations can establish a robust security framework that systematically addresses misconfigurations across the entire cloud environment.

Demo / Proof of Concept

▶ Watch: Harbinger 1: Legacy mindset in cloud security (4:00)

While Karl Ots's talk did not feature a live, interactive demonstration or a full proof-of-concept exploit, he did illustrate the effectiveness of certain tools through static examples. Specifically, he presented a screenshot of air-iam, an open-source tool used for analyzing AWS Identity and Access Management (IAM) roles. This visual example, though presented on a dark background, highlighted how air-iam can parse existing AWS roles, analyze permissions, and correlate them with logs to pinpoint security issues. The image specifically showed an entry for a root user named booth admin that had not been used for 192 days, serving as a clear illustration of an overly permissive and inactive identity that could pose a significant attack path.

Ots also mentioned ScoutSuite as a tool capable of performing red teaming exercises and good posture management for self-assessment. While no direct demonstration was given, the implication was that such tools provide the necessary visibility to understand how an environment is configured and to double-check the efficacy of other CNAPP solutions. The talk's focus was more on architectural principles and a broad array of tools and strategies rather than a single, in-depth technical demonstration.

Defensive Implications

▶ Watch: Fixing legacy mindset: Adapting to ephemeral cloud workloads (6:00)

Karl Ots provides clear, actionable call-to-actions for both red teamers looking to identify misconfigurations and blue teamers aiming to prevent and remediate them.

For Red Teamers:

Red teamers should focus on the "low-hanging fruit" of misconfigurations, as these are often the most effective entry points:

1. Simple Scans: Don't underestimate the basics. Regularly scan for publicly exposed resources:

• Public storage accounts/buckets (e.g., S3 buckets, Azure Storage accounts).
• Public container registries.
• Public dashboards from Kubernetes clusters. The speaker notes that "no one cares if it's only for dev" when a breach hits the press.

1. Legacy Connection Methods: Actively search for connection strings and other legacy authentication methods that are not tied to centrally managed Identity Providers (IDPs). These often represent isolated security perimeters.
2. Machine Identities: Dive deeper into service accounts (for Kubernetes), service principles (for Azure), and other highly privileged or overpowered machine identities. Analyze how often they are used, their provisioned permissions, and whether their scope is appropriately limited. Inactive but highly privileged identities are prime targets.
3. Exposed Kubernetes Components: Specifically look for Kubernetes components exposed to the public internet. Manually built Kubernetes clusters (as opposed to cloud provider-managed services like AKS or EKS) are often more prone to such misconfigurations due to greater human involvement.

For Blue Teamers:

Blue teamers have a critical role in proactive prevention and robust monitoring:

1. Deploy at Least One Policy: Ots urges blue teamers to deploy at least one policy, even if it's an audit-only policy that doesn't prevent actions. The goal is to gain visibility and challenge assumptions. "You'll be surprised" by what non-compliant actions are happening.
2. Kubernetes Runtime Analytics: Implement Falco for Kubernetes environments. It provides excellent runtime analytics by monitoring container calls to the node's kernel via eBPF, offering deep insight into security behaviors. Pair this with Kyverno for audit-only policies to get a comprehensive list of potential issues without breaking production.
3. Azure-Specific Logging: On Azure, turn on the Graph Activity Log. This is distinct from standard activity logs and Graph logs; it pre-packages activities from various endpoints, providing a valuable tenant-level monitoring source for all activities within the environment.
4. Audit Service Principles: Regularly audit service principles and other machine identities. Many are often unused. While cloud providers now enforce a maximum lifetime (e.g., two years for Azure service principles), consider shortening these lifetimes even further where possible.
5. Comprehensive Logging: Ensure diagnostic settings and logging are turned on across all cloud environments. Without logs, detecting and investigating incidents is impossible.
6. Container Image Hygiene: Check container images for embedded credentials in code and avoid running containers as the root user. These are fundamental security best practices.
7. Credential Rotation Practice: As a personal exercise, Ots suggests logging into a mobile dashboard and attempting to rotate at least one set of credentials. This helps understand the process, identify potential breakage points, and ensure readiness for an actual incident.

By adopting these defensive strategies, blue teams can proactively harden their cloud environments against the most common and impactful attack vector: misconfigurations.

Key Takeaways

• Misconfigurations Remain the Top Threat: Despite advances in cloud security, misconfigurations—driven by human error, insecure defaults, and complex interdependencies—are still the leading cause of data breaches, a trend that has persisted for over a decade.
• Shift from Static to Ephemeral Security: Traditional on-premises security mindsets and controls (e.g., static firewalls, agent-based systems) are ineffective in the dynamic, ephemeral nature of cloud environments. Embrace event-driven, identity-centric security models.
• Prioritize Least Privilege and Centralized Identity: Over-permissive access is a critical vulnerability. Implement least privilege by focusing on the scope of permissions, leveraging centralized Identity Providers (IDPs), and adopting Identity and Access as Code (IaC) to manage permissions programmatically.
• Eliminate Legacy Authentication: Disconnected authentication methods like static keys and connection strings bypass centralized IAM. Enforce policies to use data plane level Role-Based Access Control (RBAC) or Attribute-Based Access Control (ABAC) exclusively.
• Embrace "Disconnected by Default" and Shift-Left: Prevent accidental public exposure by defaulting all resources to private. Integrate Infrastructure as Code (IaC) scanning (e.g., Terrascan, Czecho) into CI/CD pipelines to catch misconfigurations before deployment.
• Policy as Code is Your Friend: Implement Policy as Code (e.g., Azure Policy, Kyverno, HashiCorp Sentinel) across all cloud environments. This is crucial for both auditing existing configurations and proactively preventing new misconfigurations, allowing security teams to enforce controls at scale.

About the Speaker(s)

Karl Ots is a seasoned Consultant and Cloud Security Lead at EPAM, a global software engineering and IT consulting company. With extensive experience in both building and breaking cloud environments, Ots primarily works with enterprise customers, focusing heavily on the Azure ecosystem, though his insights and expertise span across multiple cloud providers. He identifies himself as the part of EPAM that "loves to break things apart," highlighting his passion for offensive security to inform and strengthen defensive strategies. His background provides a unique perspective on the challenges and solutions in cloud security, making him a credible voice on topics like misconfigurations and cloud-native security architectures.