Is there a before and an after, or a happily ever after?

Is there a before and an after, or a happily ever after?

Speakers: Patricia Lindén

Conference: Security Fest

YouTube: https://www.youtube.com/watch?v=8nUNZVAtl94

Overview

In this insightful Security Fest talk, Patricia Lindén, IT Security Manager at the Swedish sports retail giant Stadium, presents a candid and detailed post-mortem of a significant ransomware incident that crippled Stadium's operations through a third-party supplier. Titled "Is there a before and an after, or a happily ever after?", the presentation offers a rare, boots-on-the-ground perspective of navigating chaos when a critical business partner falls victim to a cyberattack. Lindén recounts the harrowing experience, from the initial moments of confusion and uncertainty to the arduous recovery process and the profound organizational lessons learned.

The talk is a powerful testament to the importance of preparedness, resilience, and collaborative teamwork in the face of a major security crisis. Lindén not only walks the audience through the timeline of events but also delves into the human element, highlighting the emotional toll on the team and the critical role of leadership and communication. Her willingness to share Stadium's vulnerabilities and successes provides invaluable insights for any organization grappling with the ever-present threat of ransomware and the complexities of supply chain security. This incident serves as a stark reminder that in today's interconnected digital landscape, a security incident at a single supplier can have far-reaching and devastating consequences for its customers.

Background

▶ Watch: Speaker Patricia Lindén introduces Stadium, her company. (0:50)

Stadium is a prominent sports retail company based in Sweden, operating 200 physical stores, an extensive e-commerce platform, and three warehouses. Their operations are complex, encompassing everything from product design and purchasing to customer interaction in physical stores and online. Patricia Lindén, as IT Security Manager, is responsible for both IT and information security across this multifaceted business.

The incident unfolded in January 2024 (approximately 500 days prior to the talk, placing the event in late 2022/early 2023). Lindén describes the week leading up to the crisis as initially normal, painting a picture of routine security work:

• Monday: Lindén started the week by attending a demo for a Security Operation Center (SOC) vendor. This was a crucial initiative, as Stadium previously relied heavily on the "loyalty" of Lindén and her colleagues for monitoring outside office hours, leading to inconsistent incident response times. The need for 24/7 monitoring was already a recognized priority.
• Tuesday: She conducted five interviews to quantitatively measure security awareness, updated the company's risk policy (shifting ownership higher up the organizational ladder to the board), and evaluated a risky user whose login activity appeared anomalous (logging in from Hungary while physically in Sweden).
• Wednesday: As an "office day," Lindén led a workout session for employees (a stark contrast to the challenges of enforcing IT security requirements) and held meetings to plan for fulfilling new PCI DSS 4.0 requirements, which were set to take effect in March.
• Thursday: A minor, but significant, personal data breach occurred. A flaw allowed some members to potentially view information about other members. This incident was quickly identified, fixed, and a limited number of affected individuals were informed. Lindén attended a crisis meeting with group management to discuss the communication strategy, including whether to involve the media. This experience provided an immediate, albeit smaller, test of their crisis management capabilities.
• Friday: The day was spent conducting a post-mortem of the personal data breach, analyzing what went well and what could be improved. Lindén concluded her work day expecting a quiet weekend.

However, that evening, around 9:00 PM, a chat message reported issues with Stadium's ERP system. Initially dismissed as minor, the situation escalated dramatically by Saturday morning. Lindén woke to urgent messages that their supplier was experiencing a major incident, rendering Stadium's ERP system and website inaccessible. News reports confirmed widespread outages, with a cinema chain, Gordon, and Rusta also affected. The root cause was soon identified: a ransomware attack on their critical data center supplier in Sweden. To protect its customers and itself, the supplier had shut down all integrations, severing Stadium's access to its essential systems and data. This immediately plunged Stadium into a full-blown crisis, far exceeding the scope of the previous day's data breach.

Key Findings

▶ Watch: Describing a typical work week before the major incident. (2:20)

The core of the incident was a ransomware attack on a critical third-party data center supplier in Sweden. This attack led the supplier to proactively shut down all integrations to protect its customers, including Stadium. Stadium's ERP system and website were immediately rendered inaccessible, triggering a company-wide crisis.

Stadium's response unfolded in distinct phases:

1. Emergency Crisis (1-3 days):

• The first crisis management meeting convened at 9:00 AM on Saturday, involving approximately 35 individuals.
• Initial focus: Determining the scope of the incident (which systems were affected), assessing consequences (can stores operate? can employees be paid?), and developing a rough plan.
• Crucially, the nature of the incident (ransomware) was not immediately known or disclosed by the supplier, adding to the uncertainty.
• Lindén described this period as "pretty dark," with uncertainty about data recovery and the need to plan for worst-case scenarios.

1. Recovery and Prioritization:

• After a few days, the supplier confirmed that data recovery was possible, though it would take time.
• Stadium prioritized which data and systems to restore first, focusing on critical business flows like store operations, the ERP system, and the website.
• Solutions groups were formed, aligned with Stadium's business flows (e.g., warehouse, product delivery).
• A critical step involved verification of recovered data by Stadium, even after the supplier had tested it.
• During the recovery, Stadium also used the opportunity to fix known issues and update systems, aiming for a more robust setup.
• One week after the incident began (the following Friday), the ERP system and website were successfully brought back online, marking a significant milestone.
• Three weeks post-incident (February 16th), the formal crisis management was closed, though work continued on restoring lower-priority systems.

Internal Experience and Team Dynamics:

• Co-location and Collaboration: Departments like Finance, Customer Service, and HR moved to the IT department's floor, fostering intense collaboration in "solutions groups."
• Energy and Support: An "energy buffet" (candy, drinks, chips) was established, and other teams showed support (e.g., bringing cinnamon buns), highlighting strong internal solidarity.
• Intensive Meetings: Numerous Teams meetings were held daily, both internally and with the supplier. Patricia Lindén served as the crucial link between the technical crisis management group and the executive group management.
• Burnout Prevention: Initial long working hours, including weekends, led to a realization that a sustainable approach was needed. A schedule was implemented for shifts and mandatory free time, with regular check-ins on team well-being. Lindén herself was "ordered home" on the Friday the ERP/website came back up.
• Whiteboards: Conference rooms were designated as "crisis management" hubs, with whiteboards used extensively for structuring meetings, information, and tasks, despite Lindén's personal preference for PowerPoint.
• Post-Incident Recovery: The company encouraged employees to take advantage of the Swedish "Botto" activity break for family time after the intense period.

Post-Mortem Strengths Identified:

• Crisis Management: The group management had an existing crisis plan, practiced during the pandemic, which provided a framework. Crucially, a technical crisis management plan, only months old, acted as a "safety blanket" for the IT department, providing clear next steps.
• Security Work: Prior investments in security (financially and in terms of personnel like Lindén) and a proactive approach had built a foundation of resilience.
• Engagement and Teamwork: A strong sense of common goal, high energy, and collaboration across all departments (IT, finance, customer service, stores, warehouses) was observed.
• Communication:
• Internal: A philosophy of informing frequently, even if there were no new updates, helped manage anxiety. Daily meetings for the tech team and at least daily updates for the wider office were standard.
• External: Suppliers with integrations were informed promptly. Media communication was kept low-key, which suited Stadium well as other, more affected companies drew greater attention.
• Luck: Lindén acknowledged that while preparedness was key, some elements of luck (e.g., their backup solution, the specific supplier team, their existing security posture) also played a role.

Post-Mortem Areas for Improvement:

• Continuity Plan: A major focus was developing robust continuity plans specifically for supplier-side incidents, including payroll contingency and clear requirements for supplier recovery and data validation.
• Documentation: Recognized as vital for consistency, traceability, and knowledge transfer, Stadium committed to improving documentation for systems, processes, and continuity plans.
• Individuals/Bottlenecks: Identifying key roles and competencies that were single points of failure (bottlenecks) and implementing succession plans and cross-training to distribute knowledge and reduce dependencies.
• Supply Chain Security: Strengthening requirements for suppliers regarding information handling, security agreements, monitoring, follow-up, and secure data off-boarding upon contract termination. This involved updating agreements with existing suppliers and setting higher standards for new ones.
• System Landscape: A comprehensive review of the entire system landscape to ensure proper segmentation, robust authentication controls, effective access management, and overall fulfillment of security requirements. This aimed to identify and address any architectural weaknesses.

The overarching lesson from this experience, as articulated by Lindén, is that a major security incident is not a matter of if, but when. The critical factors are preparedness, the magnitude of the impact, and the organization's ability to respond effectively.

Technical Deep Dive

▶ Watch: Highlighting the critical need for 24/7 security monitoring. (3:40)

The technical core of this incident revolved around a ransomware attack that targeted a third-party data center supplier in Sweden. This wasn't an attack directly on Stadium's infrastructure, but rather on a critical external provider housing Stadium's essential systems and data. As a direct consequence, the supplier implemented a protective measure by shutting down all integrations, effectively isolating Stadium from its primary operational systems.

Stadium's key affected systems included:

• The ERP system, which is fundamental for managing business processes such as inventory, sales, purchasing, and finance.
• The e-commerce website, directly impacting online sales and customer interaction.
• Other potentially integrated systems reliant on the supplier's infrastructure.

Stadium's technical response, while not delving into the specifics of the ransomware itself (as the incident occurred at the supplier's end), focused heavily on recovery and resilience:

1. Scope Assessment and Prioritization: In the immediate aftermath, a critical technical task was to assess the full scope of the outage. This involved identifying all affected systems and understanding the downstream impact on business operations. A rapid prioritization strategy was developed, focusing on bringing back the most critical systems first. Top priorities included systems supporting store operations, the ERP system, and the website, as these directly impacted revenue generation and core business functions. The ability to process payroll was also a high-priority technical consideration.

1. Data Recovery and Verification: The recovery process relied on the supplier's ability to restore data from backups. However, Stadium did not passively accept restored data. A crucial technical step involved Stadium's teams actively testing and verifying the integrity and completeness of the recovered data and systems after the supplier had performed their own checks. This due diligence ensured that the restored environment was functional and trustworthy before being brought back online.

1. System Rebuilding and Hardening: As systems were brought back online, Stadium leveraged the opportunity to perform maintenance and improvements. This included updating systems and potentially reconfiguring infrastructure to address known issues and enhance their overall security posture. This "do it a little bit rightly" approach suggests a focus on hardening the environment during recovery.

1. Pre-existing Security Initiatives: The talk highlighted pre-incident technical efforts, such as planning for PCI DSS 4.0 compliance. This indicates Stadium already had a framework for adhering to strict security standards, which likely contributed to their ability to respond effectively, even if the direct attack vector was external. The ongoing evaluation of a risky user and the demo for a Security Operation Center (SOC) also point to a proactive stance on monitoring and threat detection, which would be crucial for future incidents.

1. Post-Incident System Landscape Review: A significant technical outcome of the post-mortem was a commitment to a thorough review of Stadium's entire system landscape. This review focused on several key areas:

• Segmentation: Ensuring proper network and system segmentation to limit the blast radius of any future incidents.
• Authentication Controls: Strengthening mechanisms for verifying user identities.
• Access Management: Refining who has access to what resources and ensuring the principle of least privilege.
• Security Requirements Fulfillment: Verifying that all systems meet established security policies and standards. This comprehensive technical audit aimed to build a more resilient and secure architecture moving forward.

While the speaker refrained from detailing the supplier's technical vulnerabilities or the specifics of the ransomware variant, Stadium's internal technical response and subsequent strategic shifts demonstrate a profound understanding of how to manage and learn from a major IT disruption, even when the initial compromise occurs outside their direct control.

Demo / Proof of Concept

▶ Watch: The first incident: discovering a personal data breach. (6:29)

This talk was a detailed post-mortem analysis of a real-world ransomware incident and its impact on Stadium's operations and its critical supplier. It did not feature a live demonstration of a specific tool, exploit, or proof of concept. Instead, Patricia Lindén presented a narrative account supported by internal photographs of their crisis management efforts, whiteboards, and a structured discussion of lessons learned.

Defensive Implications

▶ Watch: Data breach goes public, leading to a crisis meeting. (7:50)

The Stadium ransomware incident, though originating with a third-party supplier, offers crucial defensive implications for organizations across all sectors:

• Robust Third-Party Risk Management (TPRM): This incident underscores the paramount importance of comprehensive TPRM. Organizations must not only assess but also continuously monitor the security posture of their critical suppliers. This includes:
• Contractual Requirements: Enforce stringent security clauses in contracts, detailing requirements for data protection, incident response, recovery time objectives (RTOs), and recovery point objectives (RPOs).
• Audit and Oversight: Regularly audit suppliers' security controls and practices.
• Communication Protocols: Establish clear, pre-defined communication channels and protocols for incident notification from suppliers, including expected timelines for disclosure and updates.
• Exit Strategy: Plan for how data and services would be transitioned or recovered if a supplier becomes compromised beyond repair or goes out of business.

• Comprehensive Business Continuity Planning (BCP) and Disaster Recovery (DR): The incident highlighted the need for BCPs that explicitly address supplier-side outages. This means:
• Contingency for Critical Functions: Develop alternative methods for critical operations like payroll, order processing, and customer service if primary systems are unavailable. Stadium, for instance, had already transferred payroll information to the bank, preventing a major issue.
• Manual Workarounds: Document and train staff on manual workarounds for essential tasks during system downtime.
• Static Fallback Information: Consider static, resilient website pages or communication channels that can be activated to inform customers and stakeholders during prolonged outages, rather than just showing an error message.

• Proactive Crisis Management Framework: Stadium's existing, albeit nascent, technical crisis management plan proved invaluable. Organizations should:
• Establish Dedicated Teams: Define clear roles and responsibilities for incident response and crisis management teams (both technical and executive).
• Regular Exercises: Conduct tabletop exercises and simulations annually (as Stadium now does) to test plans, identify gaps, and ensure muscle memory for critical personnel.
• Cross-Departmental Involvement: Ensure BCPs and DR plans involve all relevant departments (IT, legal, HR, communications, finance, operations).

• Invest in Internal Security Posture: While the attack was external, Stadium's prior investments in security (e.g., a security manager role, moving risk policy ownership higher, PCI 4.0 planning) provided a "safety blanket." Key areas include:
• Security Awareness Training: Continuous training helps identify risky user behavior and fosters a security-conscious culture.
• System Hardening: Regularly review and strengthen internal system configurations, including network segmentation, robust authentication controls, and access management policies based on least privilege.
• Proactive Monitoring: Implement 24/7 monitoring solutions, such as a SOC, to detect and respond to threats efficiently.

• Strengthen Documentation and Knowledge Transfer: The post-mortem identified documentation as a key area for improvement.
• Clear, Concise Documentation: Maintain up-to-date documentation for all systems, processes, and recovery procedures.
• Reduce Bottlenecks: Identify single points of failure in expertise and implement succession planning and cross-training to ensure multiple individuals can perform critical tasks during a crisis.

• Strategic Communication Plan: Effective communication is vital to manage internal and external perceptions during a crisis.
• Internal Transparency: Prioritize frequent, honest internal communication, even if it's just to say there's no new update. This reduces anxiety and misinformation.
• External Messaging: Develop pre-approved external communication templates for customers, partners, and media. Control the narrative by being proactive and factual.

• Prioritize Employee Well-being: The incident highlighted the risk of burnout. Implementing schedules for rest, mandatory time off, and regular check-ins on team well-being is crucial for sustained incident response.

Ultimately, Stadium's experience reinforces that cybersecurity is not just an IT problem but a business-wide risk. A holistic approach that integrates security into business continuity, supplier management, and organizational culture is essential for navigating the inevitable "when" of a major security incident.

Key Takeaways

• Ransomware is a "When, Not If" Scenario: Organizations must accept that a major security incident, potentially involving ransomware, is highly probable. Preparedness, not prevention alone, is the key to resilience.
• Supply Chain Security is Critical: An organization's security posture is only as strong as its weakest link, often found in its third-party suppliers. Robust contracts, monitoring, and clear incident response requirements for suppliers are non-negotiable.
• Practice Your Crisis Management: Pre-existing and regularly practiced crisis management plans, both at the executive and technical levels, provide a vital "safety blanket" and structure during chaotic events.
• Teamwork and Communication are Paramount: Strong internal collaboration, transparent communication (even if there are no new updates), and leadership focused on employee well-being are crucial for navigating a prolonged crisis.
• Proactive Security Investments Pay Off: Prior investments in security tools (e.g., SOC), personnel (e.g., IT Security Manager), policy updates, and compliance efforts (e.g., PCI 4.0) provide a foundation of resilience that aids recovery.
• Continuous Learning and Sharing: Learning from incidents, both internal and external, and openly sharing experiences within the cybersecurity community, strengthens collective defense and improves future preparedness.

About the Speaker(s)

Patricia Lindén is the IT Security Manager at Stadium, a prominent sports retail company operating across Sweden with 200 physical stores and an e-commerce presence. With a background rooted in the family-owned business (which she humorously notes started "five decades ago"), Lindén is responsible for overseeing both IT security and information security. Her role involves navigating complex retail flows, from customer interactions to supply chain and product design. She is actively involved in security awareness initiatives, risk policy management (having successfully elevated policy ownership to the board level), and ensuring compliance with standards like PCI DSS 4.0. Lindén also serves as an instructor for office workouts, a unique position that gives her a different perspective on influencing her colleagues compared to her security requirements. She is a strong advocate for sharing knowledge and experiences within the cybersecurity community to foster collective strength and preparedness.