Changing from Probability to Prioritization Method in Risk Measurement

Changing from Probability to Prioritization Method in Risk Measurement

Speakers: Glauco Sampaio, Chief Information Security Officer, Cielo

Conference: RSAC 2024

YouTube: https://www.youtube.com/watch?v=1ygqIg7T-eA

Overview

In an era where organizations face an ever-growing deluge of security vulnerabilities, the traditional methods of risk assessment and prioritization often fall short, leading to overwhelmed teams and persistent exposure. Glauco Sampaio, Chief Information Security Officer at Cielo, a major Brazilian payment systems company, addressed this critical challenge at RSAC 2024. His talk, "Changing from Probability to Prioritization Method in Risk Measurement," detailed Cielo's transformative journey from a reactive, probability-centric approach to a proactive, prioritization-driven model that is deeply aligned with business objectives.

Sampaio's presentation highlighted the inherent limitations of relying solely on vulnerability severity scores like CVSS or statistical probabilities, which frequently fail to account for an asset's business criticality, environmental context, or existing mitigating controls. He articulated a pragmatic framework that not only helps security teams manage "thousands of issues" more effectively but also fosters stronger partnerships with product teams. This new methodology has enabled Cielo to elevate its overall security posture while optimizing resource allocation and streamlining remediation efforts.

The insights shared by Sampaio are particularly relevant for CISOs and security leaders grappling with unmanageable backlogs, strained relationships with business units, and the continuous pressure to demonstrate tangible security improvements. By shifting focus from abstract probabilities to concrete prioritization based on asset importance and exposure, Cielo has forged a path towards more intelligent risk management, demonstrating a model that can significantly enhance organizational resilience against evolving cyber threats.

Background

▶ Watch: Introduction and the problem with current prioritization (0:00)

The genesis of Cielo's innovative approach stemmed from a pervasive industry problem: the overwhelming volume of security issues and the struggle to effectively prioritize them. As Glauco Sampaio explained, upon his arrival at Cielo five years ago, the company, responsible for approximately one-third of Brazil's payment systems market, was contending with "huge, huge numbers" of vulnerabilities and security issues. These issues originated from a diverse array of sources, including vulnerability scans (utilizing tools like Tenable), penetration tests, war games in production environments, threat intelligence feeds, and static/dynamic code scanning integrated into development pipelines. All these inputs collectively generated "more than thousands of issues to be solved."

The core challenge was that traditional prioritization methods, often based on standardized vulnerability classifications like CVSS scores, were proving inadequate. Sampaio emphasized that simply labeling a vulnerability as "high" and mandating a 30-day fix window was "not fair with the technical teams" who lacked the capacity to address everything simultaneously with equal urgency. This led to a reactive posture where security teams were "running all the time against the attackers," with remediation efforts often lagging behind the pace of exploitation.

Sampaio referenced Gartner research, which corroborates this industry-wide struggle, indicating that organizations are frequently unable to resolve vulnerabilities before they are actively exploited. Compounding this issue, the time window between a vulnerability's discovery and its exploitation is "reducing fast" year after year, demanding a more proactive and efficient approach. Gartner's recommendations, aligning with Sampaio's vision, advocate for a focus on asset risk rating rather than solely on vulnerability classification. This means considering the asset importance to the business, recognizing that a "high" vulnerability on a non-critical asset or one protected by multiple layers of defense (e.g., Web Application Firewall (WAF), virtual patching, microsegmentation) does not carry the same immediate risk as an identical vulnerability on a critical, exposed system.

Furthermore, Sampaio noted that the NIST SP 800 series also underscores the significance of asset importance in determining remediation timelines. The fundamental question from product and IT teams – "What do we have to fix first?" – highlighted the desperate need for a refined prioritization mechanism that could guide efforts effectively, moving beyond a generic "fix everything" mandate to a strategic, risk-informed approach. This recognition formed the bedrock for Cielo's shift towards a system that truly reflects the company's unique risk landscape and business priorities.

Key Findings

▶ Watch: Gartner's focus on asset risk, not just CVSS (2:30)

Cielo's journey led to several pivotal findings and a fundamental shift in its risk management philosophy. The most significant change was the move from a probability-based to a prioritization-based method for risk measurement. Sampaio explicitly stated that Cielo is "not using probability anymore" in its core risk calculations, having observed that even extremely low-probability events (e.g., "0.0001%") can have devastating real-world impacts when they do occur. Instead, the focus is squarely on the importance of the assets and their exposure.

Central to this new methodology is the concept of a Protection Level Agreement (PLA). Similar in structure to a Service Level Agreement (SLA), a PLA is a formal agreement established collaboratively between security teams and business unit leaders (specifically, product managers). These agreements define the acceptable level of security for a given product or system, directly aligning with the company's overall risk appetite. PLAs serve as a crucial mechanism to bring "the guys from the business in the game," ensuring that security requirements are not just dictated but mutually understood and agreed upon, providing the necessary visibility and accountability.

The risk classification at Cielo now employs "five levels of classification" which are derived by integrating various factors that act as modifiers. These modifiers adjust the urgency and remediation timelines for identified issues. A key finding was the profound impact of differentiating remediation efforts based on a product team's adherence to security best practices and architecture. Teams that act as "security champions," actively implementing cybersecurity requirements and utilizing approved architectures, receive the benefit of "much more time to solve the issues." Conversely, teams not adhering to these standards face significantly shorter, more immediate remediation demands, sometimes needing to "fix this today." This differential treatment serves as a powerful incentive, encouraging proactive security integration upstream in the development lifecycle.

Another transformative finding was the cultural shift within the organization. The implementation of PLAs and the new prioritization method fostered a dramatically improved relationship between security and product teams. Sampaio noted that "the product teams are friends of the cybersecurity guys," a significant departure from the typical adversarial dynamic. This collaboration facilitated easier conversations with the board, who now have a clearer, more business-relevant understanding of the risk landscape. Furthermore, the process ensures that all vulnerabilities are ultimately addressed, as Cielo operates under a strict corporate policy of not accepting risk indefinitely. Instead, unaddressed issues incur financial penalties (e.g., "5%, 10% depend of the level of the penalty") on annual bonuses for non-compliant teams, driving consistent remediation.

Finally, the new process is designed to be "alive," continuously integrating information from threat intelligence and incident response teams. This dynamic nature ensures that prioritization remains relevant and responsive to the evolving threat landscape, making it a "movie, not a picture." The long-term vision also includes managing PLAs in relation to costs, optimizing budget allocation for security efforts.

Technical Deep Dive

▶ Watch: Addressing the common 'what to fix first?' question (4:00)

Cielo's transition from a probability-based to a prioritization-based risk management method is underpinned by a sophisticated framework that integrates multiple data points and business contexts. The core of this system revolves around an asset risk rating model that moves significantly beyond simplistic CVSS scores.

At the heart of the technical approach is the comprehensive evaluation of an asset's importance. Sampaio highlighted that "not all the products have the same importance for the company." For instance, a public-facing corporate website, while important, does not carry the same systemic risk as a core payment processing system. This differentiation is a "game changer," allowing resources to be concentrated on the most critical business functions. This importance is determined in collaboration with business stakeholders, feeding into the definition of the Protection Level Agreements (PLAs).

The PLAs themselves are not static documents but dynamic frameworks. They are established through a collaborative process involving security teams and product managers, aligning directly with the company's risk appetite. This means that for a highly critical product, the PLA will stipulate a very aggressive remediation timeline and a stringent security posture, whereas for a less critical asset, the PLA might allow for more extended remediation periods. The PLA effectively codifies the acceptable risk threshold for each specific asset or product.

To refine the prioritization, Cielo employs a system of modifiers that dynamically adjust the urgency of remediation. These modifiers take into account several factors:

1. Exploitability: While Sampaio noted that public exploits are primarily relevant for patching vulnerabilities (often detected by tools like Tenable), their existence significantly escalates the criticality of an issue. For other types of vulnerabilities, such as those found during penetration tests, the concept of a "public exploit" is less applicable and is therefore not used in their classification.
2. Environmental Exposure: This factor considers the actual context of the vulnerability. Is the affected asset exposed to the internet? Is it protected by existing security controls like a Web Application Firewall (WAF), virtual patching, or microsegmentation? An issue on an internally segmented system with limited access carries a different risk profile than one on an externally facing, unprotected asset, even if the underlying vulnerability is identical.
3. Security Architecture Adherence: This is a powerful, behavioral modifier. Product teams that consistently implement robust cybersecurity architectures and adhere to established security standards are "benefited" with extended remediation timelines. This incentivizes proactive security-by-design, as teams understand that good security practices upfront translate to less pressure and more flexibility down the line. Conversely, teams that disregard security architecture or requirements face accelerated remediation demands, often needing to fix issues "today" if a critical vulnerability arises.

All security findings, whether from vulnerability scans, pentests, war games, threat intelligence, or code scanning, are fed into this unified risk management process. The system then applies the PLA for the specific asset, factoring in the various modifiers, to generate a final, prioritized remediation requirement. This output is presented to the product teams, who are then expected to integrate these fixes into their regular development backlogs. Cielo's strategy involves allocating a dedicated "percentage of the team's backlog" to security issues, treating them as normal bugs or features, rather than disruptive, ad-hoc tasks.

A cornerstone of Cielo's technical philosophy, and a point Sampaio reiterated, is the explicit rejection of probability in risk calculation. The company has moved entirely away from quantifying the likelihood of an event, choosing instead to focus on the impact (derived from asset importance) and exposure (derived from environmental context and controls). This allows for a more direct and actionable assessment of risk, unburdened by the complexities and potential misinterpretations of probabilistic models. The entire process is designed to be "alive" and continuously updated, integrating new threat intelligence and incident data to ensure that prioritization remains current and effective, reflecting the dynamic nature of the threat landscape.

Demo / Proof of Concept

▶ Watch: Implementing five levels of risk classification and board reaction (6:00)

The talk delivered by Glauco Sampaio at RSAC 2024 focused on the strategic and operational transformation of Cielo's risk management processes rather than a technical demonstration of a specific tool or proof of concept. Sampaio detailed the conceptual framework, implementation challenges, and organizational benefits of their new methodology, including the adoption of Protection Level Agreements (PLAs) and asset-centric prioritization. No explicit software demonstration, code walkthrough, or live exploit scenario was presented during the session. The emphasis was on the systemic change within a large enterprise environment and the practical application of these principles.

Defensive Implications

▶ Watch: Benefiting security champions with extended resolution time (7:00)

Cielo's innovative approach to risk management offers profound defensive implications for organizations grappling with similar challenges. The shift from probability to prioritization, grounded in asset importance and business context, provides a blueprint for building a more resilient and business-aligned security program.

For Security Leadership (CISOs, Security Managers):

1. Adopt an Asset-Centric Risk Model: Move beyond generic vulnerability scores (like CVSS) to prioritize based on the actual business criticality of assets and their exposure. This allows for intelligent resource allocation, focusing efforts where they deliver the most impact.
2. Implement Protection Level Agreements (PLAs): Establish formal, collaborative agreements with business units to define acceptable security postures and remediation timelines for specific products or services. This fosters shared ownership of risk, aligns security objectives with business goals, and provides clear accountability.
3. Incentivize Security-by-Design: Differentiate remediation expectations based on a team's adherence to security architecture and best practices. Reward "security champions" with more flexible timelines, creating a positive feedback loop that encourages proactive security integration rather than reactive firefighting.
4. Embrace Continuous Evolution: Recognize that risk management is a "movie, not a picture." Integrate real-time threat intelligence and incident response data into the prioritization process to ensure it remains dynamic and responsive to the evolving threat landscape.
5. Leverage Financial Incentives/Penalties: Consider implementing mechanisms, such as financial penalties on annual bonuses for consistent non-compliance with PLAs, to enforce security policies and drive accountability across the organization. This provides a tangible consequence for inaction, as seen at Cielo (5-10% penalty).
6. Foster Business Partnership: Actively work to change the perception of security from a "police" function to a "partner" function. Collaborative PLA definition and differentiated remediation timelines can significantly improve relationships with product and IT teams.
7. Engage Governance and Auditors Early: Involve internal governance structures and external auditors in the development and modification of new risk policies. This ensures buy-in, compliance, and avoids future policy bypasses or audit issues.

For Technical Teams (Developers, Operations, Security Engineers):

1. Understand Asset Importance: Be aware of the business criticality of the systems you manage. This context is crucial for understanding the true impact of vulnerabilities and prioritizing your work.
2. Adhere to Security Architecture: Actively follow established security architectures and best practices. This not only improves the security posture of your products but can also provide more favorable remediation timelines for your team.
3. Integrate Security into Backlogs: Incorporate security fixes as a regular part of your development backlog. Dedicate a percentage of sprint capacity to security issues, treating them as features or bugs, to ensure continuous remediation without disruptive, ad-hoc efforts.
4. Collaborate on PLAs: Participate actively in defining PLAs for your products. This ensures that the agreements are realistic, achievable, and reflect the operational realities of your team.

Broader Organizational Implications:

• Cultural Transformation: Be prepared for significant cultural resistance. Changing mindsets from a purely technical "fix everything" approach to a business-aligned risk management model takes time (Cielo's process took nearly three years to fully establish).
• Iterative Policy Development: Risk management policies and PLAs will likely require multiple iterations and adjustments. Start with willing "security champions" or product teams to build momentum and demonstrate success before rolling out company-wide.
• Focus on Exposure and Impact: By de-emphasizing abstract probability, organizations can focus on tangible factors: what assets are most important, and how exposed are they? This leads to more actionable and defensible risk decisions.
• No Indefinite Risk Acceptance (Cielo's model): While many organizations accept certain risks, Cielo's corporate decision to not accept risk indefinitely means a continuous drive towards remediation. This approach, while strict, ensures that all identified vulnerabilities are eventually addressed, even if the timeline is adjusted.

By embracing these defensive implications, organizations can move towards a more mature, efficient, and integrated security posture that genuinely supports business objectives.

Key Takeaways

• Shift from Probability to Prioritization: Organizations should move away from risk assessments solely based on statistical probability, which can be misleading, and instead prioritize remediation based on asset importance and exposure.
• Establish Protection Level Agreements (PLAs): Collaborate with business and product teams to define Protection Level Agreements (PLAs) that align security requirements and remediation timelines with the company's risk appetite for specific assets.
• Incentivize Proactive Security: Differentiate remediation timelines based on a team's adherence to security architecture and best practices. Reward "security champions" with more flexibility, while holding non-compliant teams to stricter, shorter deadlines.
• Integrate and Evolve Risk Management: Consolidate all security findings (from scans, pentests, threat intel, etc.) into a single, comprehensive risk process. This process must be "alive," continuously updated with new information to remain relevant and effective.
• Drive Accountability and Collaboration: Foster stronger relationships between security and product teams, transforming security from a policing function to a business partner. Implement mechanisms, potentially including financial penalties, to ensure accountability for security policy adherence.
• Cultural Change is Paramount: Implementing such a system requires significant cultural transformation and leadership buy-in. Start with willing teams and iteratively refine policies, involving governance and auditors throughout the process.

About the Speaker(s)

Glauco Sampaio is the Chief Information Security Officer (CISO) at Cielo, a prominent Brazilian company based in São Paulo. Cielo is a significant player in the payment systems market in Brazil, responsible for approximately one-third of the country's payment transactions. Sampaio has been with Cielo for five years and brings a strong technical background to his leadership role, having previously worked within technical teams. His experience informs his pragmatic approach to cybersecurity, focusing on fair and effective risk management strategies that balance security imperatives with business realities.