How to CTF Infra - Beyond the Challenges and Flags

How to CTF Infra - Beyond the Challenges and Flags

Speakers: Jayesh Singh Chauhan, Founder, Cloud Village; Max, Infrastructure Lead, Cloud Village

Conference: RSAC 2024

YouTube: https://www.youtube.com/watch?v=me4taJZAp0

Overview

This talk, "How to CTF Infra - Beyond the Challenges and Flags," delves into the often-overlooked complexities of building and operating a Capture The Flag (CTF) event, particularly in the cloud security domain. Presented by Jayesh Singh Chauhan and Max from Cloud Village, the session moves beyond merely solving challenges to provide a comprehensive blueprint for designing scalable infrastructure, managing volunteer teams, and fostering a vibrant community around CTFs. The speakers draw upon their extensive experience running the Cloud Village CTF at RSA for six years, sharing a battle-tested framework that empowers others to host their own events.

The core motivation behind this presentation is to democratize the knowledge required to host CTFs. While numerous resources exist for participants, there's a significant void in practical guidance for organizers on architectural considerations, automation, and team management. Jayesh and Max aim to fill this gap by outlining their technical infrastructure, built largely on AWS and open-source tools, and detailing their unique human-centric framework for volunteer engagement and skill development. Their goal is to inspire and enable individuals and communities worldwide to create more CTFs, thereby propagating hands-on learning in cybersecurity.

This talk is crucial for anyone interested in organizing security events, fostering community-driven learning, or simply gaining a deeper understanding of resilient cloud infrastructure. It offers invaluable insights into the technical challenges of maintaining a live, high-stakes platform under pressure, as well as the organizational strategies for building and sustaining a passionate, skilled volunteer team. By sharing their successes and failures, the Cloud Village team provides a practical roadmap for transforming the abstract concept of a CTF into a tangible, educational, and engaging experience.

Background

▶ Watch: Talk introduction and speaker welcome (0:00)

The Cloud Village has been a consistent presence at RSA and other major security conferences since its inception in 2019, primarily known for its popular cloud security-focused Capture The Flag events. Over these six years, Jayesh Singh Chauhan, the founder, and Max, the infrastructure lead, observed a significant knowledge gap within the cybersecurity community. While there's an abundance of resources and discussions centered around solving CTF challenges and specific cloud security vulnerabilities, very little public knowledge exists on the intricate process of running a CTF event. This includes everything from architecting a robust platform to managing the diverse skills and efforts of a volunteer team.

The problem stems from the inherent complexity of hosting a live, interactive security competition. It requires not only a deep understanding of the security challenges themselves but also expertise in scalable infrastructure, automation, incident response, and community building. Many CTFs are run by passionate volunteers, often without a formal framework or extensive prior experience in event operations at scale. This often leads to ad-hoc solutions, potential stability issues, and a steep learning curve for new organizers. The Cloud Village team sought to address this by codifying their accumulated knowledge and sharing a repeatable framework that has served them well.

A foundational component of their approach, and a common choice in the CTF community, is CTFd. As an open-source Capture The Flag framework, CTFd provides the essential platform for user registration, challenge display, flag submission, and scoreboard management. Its focus on ease of use and customizability, coupled with solid documentation and an active community, made it the logical choice for Cloud Village. However, merely adopting CTFd is only the first step; the real challenge lies in deploying, scaling, and maintaining it, along with all the custom challenges, in a production-like environment for a demanding conference audience. The talk aims to demystify this entire lifecycle, offering a holistic view of CTF operations.

Key Findings

▶ Watch: Sharing Cloud Village's 6-year CTF framework (2:30)

The talk presents several key findings across both technical infrastructure and organizational management, derived from Cloud Village's six years of running cloud CTFs.

Technical Findings:

1. CTFd as a Robust Core: CTFd is identified as the de-facto open-source framework for running CTFs, lauded for its stability, customizability via plugins and themes, and strong community support. Its availability for both self-hosting and as a managed SaaS offering (CTFd.io) provides flexibility for organizers.
2. Infrastructure as Code (IaC) is Non-Negotiable: The consistent use of Terraform for infrastructure provisioning and Packer for declarative virtual machine image building is paramount. IaC ensures repeatability, documentation, testability, and reduces the "bus factor" by codifying infrastructure knowledge.
3. Scalability through Auto-Scaling and Remote Services: Employing auto-scaling groups for compute instances (VMs or containers) is critical for absorbing unpredictable traffic spikes during registration and gameplay, while simultaneously managing costs by scaling down. Decoupling core services like databases (AWS RDS for MySQL), asset storage (S3), and caching (Elastic Cache) from the compute layer prevents service interruptions during scaling events.
4. Containerization Offers Significant Advantages: Re-platforming from virtual machines to containers on Amazon ECS (Elastic Container Service) provided improved isolation, faster spin-up times, and enforced a more disciplined approach to documentation and accountability, further reducing the bus factor. AWS's underlying Firecracker technology for micro-VMs in ECS provides robust isolation.
5. Prudent Secrets Management: Utilizing services like AWS Secrets Manager and Parameter Store for sensitive data (database credentials, API keys) is essential. Parameter Store is favored for its cost-effectiveness and ease of use in development.
6. CDNs Enhance Performance and Security: Implementing a Content Distribution Network (CDN) like AWS CloudFront or CloudFlare offloads traffic, caches static assets, improves application performance, and adds a layer of security by absorbing potential malicious traffic before it reaches the core infrastructure.
7. Common Operational Pitfalls and Remedies:

• Gunicorn Misconfiguration: In 2020, the CTFd application was CPU-bound due to incorrect Gunicorn (the web server) worker and thread configurations, preventing auto-scaling from being effective. Tuning these parameters unlocked the underlying compute capacity.
• Memory Leaks: Subsequent years (2022, 2023) revealed memory leaks within the CTFd application, necessitating fleet rolls and capacity adjustments during live events.
• Database Connection Leaks: The application occasionally failed to close database connections correctly, leading to a self-inflicted denial of service against the database. Remediation involved actively monitoring and terminating rogue connections.
• The crucial insight here is that while these problems are persistent, familiarity allows for quick, on-the-fly remediation without impacting user experience during a live contest.

Organizational and Human Findings:

1. Structured Volunteer Framework: A clear division of roles—Challenge Builders, Challenge Owners, Challenge Testers, and Platform Infrastructure Team—enables efficient task distribution and skill development.
2. Mentorship and Deliberate Pairing: New volunteers are paired with more experienced ones (e.g., Challenge Builders with Challenge Owners) to facilitate learning and ensure quality.
3. Open Ideation Fosters Innovation: Involving new volunteers in the initial challenge ideation phase brings fresh perspectives and prevents "myopic views" from experienced organizers, leading to more diverse and engaging challenges.
4. Learning Through Contribution: The framework is designed to upskill volunteers, regardless of their starting point in their cloud security journey. Challenge Testers, for instance, gain exposure to vulnerabilities and often return as Challenge Builders in subsequent years.
5. Proactive Incident Response and Communication: Acknowledging that "things would go wrong," the team prioritizes a robust incident response plan with active synchronization between the platform infrastructure team and a social media team. This ensures timely communication with players during outages, maintaining trust and engagement.
6. Community and Iterative Improvement: The Cloud Village CTF is entirely volunteer-run, thriving on a strong community that consistently returns due to the rewarding learning experience. The team embraces an iterative approach, learning from past incidents to continuously mature their framework and reduce operational burden over time.

Technical Deep Dive

▶ Watch: Talk structure: Technical infrastructure and human framework (4:20)

The Cloud Village CTF infrastructure has evolved significantly since its inception, demonstrating a commitment to robust, scalable, and automated deployment practices. At its heart lies CTFd, an open-source Capture The Flag framework that serves as the central platform for managing the competition. CTFd is highly customizable, supporting plugins and themes, which the Cloud Village team leverages for specific event needs. While CTFd.io offers a managed SaaS solution, Cloud Village opts for a self-hosted model, providing full control over their environment and enabling deep customization.

Initial Architecture (2020) - Virtual Machines

The initial deployment in 2020 utilized a traditional virtual machine-based architecture on AWS. Traffic from the public internet first hit a Content Distribution Network (CDN), such as AWS CloudFront, to absorb load, cache static assets, and filter potentially malicious requests. This significantly reduced the burden on the backend infrastructure. Requests then proceeded to AWS Load Balancers which distributed traffic across multiple EC2 virtual machines running the CTFd application.

Supporting services were decoupled and managed remotely:

• A remote mail server handled user registrations and communications.
• AWS S3 provided highly available blob storage for static assets.
• AWS RDS for MySQL served as the managed relational database.
• AWS Elastic Cache was used for caching session cookies and other frequently accessed data, improving application responsiveness.

Crucially, the entire infrastructure was defined and managed using Infrastructure as Code (IaC). Terraform, from HashiCorp, was the chosen tool for its declarative approach, allowing the team to define what they wanted their infrastructure to look like, rather than how to build it step-by-step. For the EC2 virtual machines themselves, Packer, another HashiCorp tool, was used to create standardized, repeatable machine images. This ensured consistency across instances and simplified updates. The EC2 instances were deployed within auto-scaling groups, enabling the system to automatically scale out in response to increased traffic (e.g., during registration spikes) and scale in when demand subsided, optimizing both performance and cost. AWS Secrets Manager was employed for securely storing sensitive credentials like database passwords and API keys.

Evolution to Containers (2021) - Amazon ECS

Recognizing the benefits of containerization, the Cloud Village team re-platformed their CTFd deployment to Amazon Elastic Container Service (ECS) in 2021. While the overall logical architecture (CDN, load balancer, remote database, cache, mail server) remained largely similar, the core CTFd application now ran within containers.

The move to containers offered several key advantages:

• Improved Isolation: Containers provide a higher degree of process isolation compared to applications running directly on VMs, enhancing security and stability.
• Faster Spin-up Times: Container images are significantly lighter and faster to provision than entire virtual machines, allowing for quicker scaling responses. Max highlighted AWS's Firecracker technology, which uses micro-VMs to provide machine-level isolation for containers, as a fascinating underlying mechanism.
• Enhanced Accountability and Reduced Bus Factor: Containerization inherently pushes towards more explicit documentation and configuration, as changes are typically baked into images rather than made ad-hoc on live VMs. This reduces reliance on individual knowledge and improves team collaboration.
• Simplified Secrets Management: With ECS, the team leveraged AWS Parameter Store in addition to Secrets Manager. Parameter Store was preferred for its cost-effectiveness and ease of use in development and knowledge transfer.
• Streamlined Auto-Scaling: Instead of writing custom auto-scaling logic from scratch in Terraform, the team utilized existing Terraform module registry components. This allowed them to offload complex scaling logic to a community-contributed, well-tested library, further accelerating development and improving reliability.

Operational Challenges and Learnings

Despite a robust architecture, running a live CTF event invariably presents operational challenges:

• 2020 - Gunicorn Misconfiguration: The primary issue during the 2020 event was the CTFd application becoming CPU-bound and failing to scale effectively. The root cause was identified as a misconfiguration of Gunicorn, the Python WSGI HTTP server used by CTFd. Default Gunicorn settings were insufficient to fully utilize the provisioned EC2 instances, leading to slow page loads and 500 errors. The fix involved tuning Gunicorn's worker and thread counts to match the available CPU capacity, immediately resolving the bottleneck. This highlighted the critical importance of understanding and correctly configuring application-level components, not just the underlying infrastructure.
• 2022/2023 - Persistent Application Issues: In subsequent years, new challenges emerged. The CTFd application exhibited what appeared to be memory leaks, requiring manual intervention to roll the container fleet and adjust capacity during the live competition. Additionally, the application occasionally failed to correctly close database connections, effectively launching a self-inflicted denial-of-service attack against its own MySQL database. While these issues were persistent, the team's accumulated experience meant they could diagnose and remediate them quickly behind the scenes, preventing any noticeable impact on players. This underscored the reality of managing live applications: problems will occur, and the ability to rapidly identify and mitigate them is paramount, even if a permanent upstream fix isn't immediately feasible during a high-stakes event.

Future Infrastructure Directions

The Cloud Village team is continuously looking to refine their infrastructure:

• Enhanced Logging and Observability: Current reliance on CloudWatch logs is functional but not optimal. Max expressed interest in exploring Grafana Loki for more user-friendly log aggregation and OpenTelemetry for comprehensive application performance monitoring, potentially contributing upstream patches to CTFd.
• CI/CD Pipelines for Infrastructure: To further reduce the bus factor and automate deployments, the goal is to implement robust Continuous Integration/Continuous Delivery (CI/CD) pipelines for their Terraform code. This would enable automated testing, deployment, and rollback capabilities, improving reliability and freeing up engineers.
• Terragrunt for Advanced IaC Management: Exploring Terragrunt, an opinionated wrapper around Terraform, is on the roadmap. Terragrunt offers advanced abstractions for managing multiple Terraform modules and environments, potentially enabling more granular role-based access control (RBAC) for infrastructure components, ensuring engineers only have access to what they need.
• Potential Language Shift: Max also considered a potential rewrite of the infrastructure code using the AWS Cloud Development Kit (CDK), possibly in Python. While Terraform's declarative nature is preferred, a Python-based CDK approach might broaden accessibility given Python's wider user base, making it easier for more volunteers to contribute.

The Cloud Village / CTFd Infra GitHub repository serves as the public embodiment of this infrastructure, containing all the Terraform code, container definitions, and configurations that power their CTF events. This open-source contribution allows other organizers to leverage their work directly, including support for custom container images to enable bespoke CTFd modifications.

Demo / Proof of Concept

▶ Watch: Introducing CTFd: The chosen CTF framework (5:50)

While the talk itself did not feature a live, interactive technical demonstration of a specific exploit or tool, the entire presentation serves as a comprehensive "proof of concept" for Cloud Village's methodology in building and running a successful CTF. The most tangible demonstration of their work is the Cloud Village CTF event itself, which was actively running at RSAC 2024 during the time of the talk, showcasing their infrastructure and organizational framework in a live, production environment.

The speakers highlighted that their GitHub repository, Cloud Village / CTFd Infra, is publicly available and contains all the Terraform code and configurations that underpin their CTF platform. This repository is the practical embodiment of their technical deep dive, allowing anyone to inspect, fork, and deploy their own CTFd infrastructure. It includes provisions for containerized deployments, database setup, caching, and custom container images, effectively serving as a blueprint for others to replicate their success. This open-source sharing is a direct demonstration of their commitment to empowering the community to host more CTFs, making their entire operational model a living, breathing proof of concept.

Defensive Implications

▶ Watch: CTFd's open-source, customizable, and community-driven nature (6:30)

The insights shared by the Cloud Village team, while focused on running a CTF, carry significant defensive implications for organizations managing their own cloud infrastructure and applications.

1. Embrace Infrastructure as Code (IaC): The consistent use of Terraform and Packer for declarative infrastructure management is a critical defensive practice. IaC ensures that infrastructure is built consistently, is auditable, and can be rapidly rebuilt in case of compromise or failure. It eliminates configuration drift, reduces human error, and facilitates security reviews by treating infrastructure like application code. Defenders should strive to codify all their cloud resources.
2. Architect for Scalability and Resilience: The challenges faced with traffic spikes and application performance underscore the need for resilient architectures. Implementing auto-scaling groups, leveraging CDNs for traffic absorption and caching, and decoupling critical services (databases, storage, cache) are fundamental for defending against DDoS attacks, handling legitimate load fluctuations, and ensuring high availability.
3. Prioritize Secrets Management: The emphasis on AWS Secrets Manager and Parameter Store highlights the importance of never hardcoding credentials. Organizations must adopt robust secrets management solutions, integrate them into their deployment pipelines, and rotate keys regularly to minimize the impact of a potential breach. Furthermore, advocating for IAM role support over direct credentials wherever possible is a strong security posture.
4. Understand Application Server Nuances: The Gunicorn misconfiguration issue demonstrates that even with robust infrastructure, application-level components can introduce critical bottlenecks and vulnerabilities. Defenders need to deeply understand the configuration and performance characteristics of their application servers, web servers, and frameworks to ensure they are optimally tuned and not inadvertently creating attack surfaces or performance issues.
5. Implement Comprehensive Monitoring and Alerting: The ability to quickly identify and remediate issues like memory leaks and database connection issues (even self-inflicted ones) during a live event is paramount. Organizations must invest in comprehensive application and infrastructure monitoring solutions (e.g., exploring Grafana Loki, OpenTelemetry) to gain deep visibility into performance, resource utilization, and potential anomalies. Proactive alerting is key to rapid incident response.
6. Develop Robust Incident Response and Communication Plans: The Cloud Village team's approach of "normalizing that it's all right if things go wrong" and having a clear plan for incident response, including synchronized social media communication, is a vital lesson. Defenders must have well-rehearsed incident response playbooks and established communication channels to inform stakeholders transparently and minimize panic during security incidents.
7. Foster a Culture of Continuous Learning and Improvement: The iterative nature of the Cloud Village's framework, learning from past failures to continually refine processes and infrastructure, is a mindset that every security team should adopt. Post-incident reviews should not only focus on technical fixes but also on process improvements and knowledge sharing to prevent recurrence and enhance overall resilience.
8. Leverage Containerization for Security and Efficiency: The move to Amazon ECS showcases the defensive benefits of containerization, including enhanced isolation between application components and faster patching/rollbacks. Defenders should explore containerization for their applications to improve security boundaries and operational efficiency.

By applying these lessons, security teams can build more resilient, secure, and manageable cloud environments, capable of withstanding both internal operational challenges and external threats.

Key Takeaways

• CTFd is the Go-To Open-Source CTF Framework: For anyone looking to host a Capture The Flag event, CTFd provides a robust, customizable, and community-supported platform that significantly streamlines the process of user management, challenge display, and scoring.
• Infrastructure as Code (IaC) is Essential for Scalability and Reliability: Tools like Terraform and Packer are critical for defining, deploying, and managing CTF infrastructure. IaC ensures repeatability, auditability, and the ability to scale resources (e.g., using auto-scaling groups and remote services like AWS RDS and Elastic Cache) efficiently and cost-effectively.
• Containerization Offers Significant Operational Advantages: Migrating from virtual machines to containers on platforms like Amazon ECS provides better application isolation, faster deployment and scaling capabilities, and promotes a more disciplined, documented approach to infrastructure management, reducing reliance on individual expertise.
• Anticipate and Plan for Application-Level Pitfalls: Even with robust infrastructure, application-specific issues like Gunicorn misconfigurations, memory leaks, and database connection problems can cripple a live event. Proactive monitoring, understanding application internals, and having rapid remediation strategies are crucial for maintaining uptime and user experience.
• A Structured Volunteer Framework Fosters Community and Learning: Cloud Village's model of Challenge Builders, Owners, Testers, and a Platform Infrastructure Team, coupled with deliberate pairing and open ideation, creates a self-sustaining ecosystem that continuously upskills volunteers and generates innovative content.
• Embrace Iterative Improvement and Incident Response: Acknowledging that "things will go wrong" is a healthy mindset. Prioritize swift incident response, clear communication channels (like syncing with social media), and continuous learning from operational challenges to iteratively refine processes and infrastructure over time.

About the Speaker(s)

Jayesh Singh Chauhan is the founder of Cloud Village, which he started in 2019, establishing a prominent platform for cloud security CTFs. With extensive experience in the cybersecurity domain, Jayesh has delivered numerous trainings at prestigious conferences such as Black Hat, AppSec NZ, and Nullcon. He is also the author of several open-source tools. Currently, Jayesh serves as the CISO for CoinSwitch, a leading company in the financial technology sector, bringing a wealth of practical security leadership to his work.

Max has been an integral part of the Cloud Village team, specifically leading the infrastructure work for their CTF events since 2020. Based in New York, Max brings a unique background to his technical role, having previously run a comic book store and possessing an art school education. He describes himself as motivated by a blend of coffee, whiskey, fruits, veggies, peace, and love, reflecting a creative and grounded approach to his demanding infrastructure responsibilities.