A Framework for Evaluating National Cybersecurity Strategies

A Framework for Evaluating National Cybersecurity Strategies

Speakers: Fred Heiding (Research Fellow, Harvard), Alex O'Neil (Cybersecurity Researcher, Harvard), Lachlan (Former Australian National Cyber Security Strategy Team), Eric Rosenbach (Senior Lecturer, Harvard Kennedy School)

Conference: Black Hat USA

YouTube: https://www.youtube.com/watch?v=D3KKYD0JPcc

Overview

In an era where the cyber threat landscape is constantly evolving, driven by advancements like Artificial Intelligence, the efficacy of national cybersecurity strategies has become paramount. This Black Hat USA talk, presented by Fred Heiding and his esteemed colleagues, introduces a novel framework designed to evaluate these crucial policy documents. The core challenge addressed is whether existing national strategies, typically 20-100 pages and less than a decade old, truly achieve their stated goal of protecting people and critical institutions. The speakers argue that current evaluation methods often fall short, relying on ambiguous absolute scores that fail to provide actionable insights.

The research team, comprising experts from computer science, policy, and national security, developed a relative scoring framework to identify leading practices and areas for improvement across different national strategies. By analyzing documents from nine diverse governments, the framework aims to help policymakers craft more effective and practical cyber strategies for future generations. This work is significant because national cybersecurity strategies profoundly impact technical practitioners, business leaders, and policymakers alike, influencing everything from regulatory environments to the overall digital resilience of a nation.

Background

▶ Watch: Speaker introduction and project team (0:00)

The proliferation of national cybersecurity strategies is a relatively recent phenomenon, with most documents published within the last decade. These strategies typically outline a nation's approach to securing its digital landscape, covering broad themes such as protecting citizens, safeguarding critical infrastructure, and fostering cybersecurity capacity. While the creation of such strategies is a positive step, a fundamental question remains: are they truly effective in practice? The talk highlights the inherent difficulty in crafting a "good" cybersecurity strategy that simultaneously satisfies diverse stakeholders, including policymakers, deep technical experts, and business leaders, while ensuring tangible protection.

Prior attempts at evaluating these strategies often employed absolute scoring systems, assigning numerical grades (e.g., a "7 out of 10") to individual countries. The presenters criticized this approach, noting the inherent complexity of justifying such scores given the myriad variables in cybersecurity. More importantly, they argued that evaluating countries in isolation misses a crucial dimension: how different national approaches interact and what lessons can be learned from comparative performance. The analogy was drawn to securing a company or country – one often only needs to be more secure than their "neighbor" or competitor. This perspective underpinned the team's decision to move away from absolute scoring towards a more nuanced, relative evaluation framework that emphasizes identifying leading practices for others to emulate. The selection of nine initial governments for analysis was based on criteria ensuring a strong cyber capability reputation, geographical and operational diversity, recent strategy publication (after 2020), and English accessibility.

Key Findings

▶ Watch: Fundamental questions about national cyber strategies (3:00)

The research yielded several significant findings regarding the structure and content of national cybersecurity strategies:

1. Relative Scoring Framework: The primary contribution is the development of a relative scoring framework. Unlike previous absolute scoring methods, this framework focuses on identifying "good points" and leading examples from which other countries can learn. This approach avoids the pitfalls of arbitrary numerical scores and provides more actionable insights into best practices.
2. Comprehensive Evaluation Criteria: The framework utilizes an extensive set of 268 quantifiable questions, primarily in a yes/no format, to meticulously evaluate various aspects of a national strategy. These questions are grouped into key evaluation areas:

• Codification of Responsibilities: How clearly are roles and responsibilities defined within the strategy (e.g., which agency is responsible for securing the energy sector)?
• People Protection: The measures outlined to protect citizens in the digital realm.
• Infrastructure Protection: Strategies for safeguarding critical national infrastructure.
• Capacity Generation: Initiatives aimed at addressing the shortage of skilled cybersecurity labor.
• Political Context: How the unique political environment of a country influences its strategic approach and implementation.

1. Strengths in Codified Responsibilities: Across the board, the strategies analyzed generally performed "rather well" in codifying responsibilities. Most documents explicitly mention which persons or agencies are accountable for specific tasks, which is crucial for effective follow-up and implementation. However, there's still room for improvement in the level of detail and specificity.
2. Universal Focus on Capacity Generation: A recurring and strong focus in nearly all national strategies is capacity generation, particularly addressing the global shortage of skilled cyber labor. This indicates a widespread recognition of the human element as a critical factor in national cybersecurity.
3. Varying Effectiveness in Protection: While all strategies emphasize protecting people and critical institutions, the effectiveness and detail vary significantly across different sectors. For instance, a country might excel in protecting its communication industry but show deficiencies in safeguarding its healthcare sector. This highlights the need for more granular and sector-specific protective measures.
4. Singapore as a Leading Example: Singapore's national cybersecurity strategy was highlighted as a particularly strong document. Its distinct political context allows for a unique approach to national protection. Key strengths include:

• A robust zero-trust protection plan for government systems, which appears to be highly effective.
• A proactive regional leadership role, where Singapore actively assists its neighbors in improving their cybersecurity posture, recognizing that enhanced regional security benefits everyone.

These findings underscore the importance of moving beyond generic policy statements to concrete, actionable strategies with clearly defined responsibilities and measurable outcomes.

Technical Deep Dive

▶ Watch: Overview of talk structure: method and highlights (4:30)

The "technical deep dive" in this context refers not to code or system architecture, but to the rigorous methodology and structured framework developed for evaluating national cybersecurity strategies. The innovation lies in its systematic approach to policy analysis, moving beyond subjective interpretations to quantifiable metrics.

The core of the framework is built upon the aforementioned 268 quantifiable questions. These questions are designed to be specific and answerable with a simple "yes" or "no," reducing ambiguity and enabling consistent evaluation across different documents. For example, instead of asking "Does the strategy protect critical infrastructure well?", a quantifiable question might be "Does the strategy explicitly identify the energy sector as critical infrastructure and assign a responsible agency for its cyber protection?" This specificity ensures that the evaluation is grounded in the explicit content of the strategy document.

The framework's structure is hierarchical, breaking down broad strategic objectives into granular, evaluable components:

1. Codified Responsibilities: This is considered a foundational element. The framework scrutinizes how clearly and comprehensively responsibilities are assigned. It examines whether specific government agencies, ministries, or even private sector entities are explicitly tasked with particular cybersecurity duties. This includes not just broad statements but also mechanisms for follow-up and accountability. The rationale is that without clear ownership, even well-intentioned strategies risk becoming ineffective due to diffusion of responsibility.
2. Protection of People and Institutions/Critical Infrastructure: This category delves into the actual defensive measures and policies outlined. It examines how the strategy addresses cyber hygiene for citizens, protection of personal data, resilience of essential services (e.g., water, electricity, transportation, healthcare, finance), and the response mechanisms for cyber incidents affecting these areas. The framework is designed to identify gaps, such as a strong focus on one sector (e.g., communications) while neglecting others (e.g., healthcare), allowing for targeted recommendations.
3. Capacity Generation: Recognizing the chronic global shortage of cybersecurity talent, this section evaluates the strategic initiatives aimed at building a skilled workforce. This includes education programs, professional development, talent recruitment, and retention strategies. The framework assesses the specificity and ambition of these plans, looking for concrete actions rather than vague aspirations.
4. Political Context and Implementation: This less tangible but crucial aspect acknowledges that a strategy's effectiveness is influenced by a nation's political system, legal framework, and geopolitical position. For instance, a country with a more centralized governance structure might be able to implement certain policies (like mandatory zero-trust architectures for government systems, as seen in Singapore) more readily than a decentralized democracy. The framework implicitly accounts for these contextual differences when comparing "leading practices" rather than imposing a single ideal.

The choice of relative scoring is a key methodological innovation. Instead of assigning an arbitrary score of 1-10, the framework aims to answer questions like "Who is leading in critical infrastructure protection?" or "Which country has the most robust plan for capacity generation?" This comparative analysis allows for the identification of benchmarks and exemplars that other nations can study and adapt. By focusing on what works well in practice, the framework fosters a learning environment among nations, promoting the adoption of proven strategies rather than striving for an abstract, universally "perfect" score. The initial application of this framework to nine diverse governments, all with strategies published after 2020 and accessible in English, demonstrated its practicality in extracting actionable insights from complex policy documents.

Demo / Proof of Concept

▶ Watch: Introducing relative scoring and the scorecard framework (6:00)

The talk did not feature a live technical demonstration or a proof-of-concept of a software tool. Instead, the presenters focused on explaining the methodology of their evaluation framework and sharing the key findings derived from its application to nine national cybersecurity strategies. The "proof of concept" was the successful application of their relative scoring framework to real-world policy documents, illustrating its utility in identifying leading practices and areas for improvement across diverse geopolitical contexts.

Defensive Implications

▶ Watch: Criteria for selecting governments for the study (7:00)

The findings from this evaluation framework carry significant implications for various stakeholders involved in national cybersecurity defense:

1. For Policymakers and Strategy Creators:

• Prioritize Codified Responsibilities: Ensure that national strategies clearly and explicitly assign responsibility for every critical cybersecurity task to specific agencies or entities. Ambiguity in ownership is a direct path to inaction and vulnerability. This means moving beyond high-level statements to detailed operational mandates.
• Adopt Relative Benchmarking: Instead of aiming for an abstract "perfect" strategy, policymakers should actively study and learn from the leading practices identified through frameworks like this. Countries like Singapore, with its zero-trust protection plan for government, offer concrete examples of effective implementation that can be adapted to different national contexts.
• Holistic Sector Protection: While general protection is good, strategies must include detailed and specific plans for all critical infrastructure sectors (e.g., energy, finance, communications, healthcare, water). Gaps in one sector can create systemic vulnerabilities.
• Foster Regional Cooperation: Emulate Singapore's approach of taking a regional leadership role by actively assisting neighboring countries in strengthening their cybersecurity. A stronger regional posture enhances collective security and resilience against transnational threats.
• Integrate Technical Expertise: Ensure that deep technical experts are meaningfully involved in the strategy creation process, translating high-level policy goals into actionable, implementable technical measures.

1. For Business Leaders:

• Understand National Strategy: Businesses must be intimately familiar with their country's national cybersecurity strategy, as it directly influences regulatory frameworks (e.g., GDPR in Europe vs. US data privacy laws), compliance requirements, and government expectations for private sector involvement in national defense.
• Align with National Goals: Where possible, align internal cybersecurity investments and initiatives with national strategic priorities, especially concerning critical infrastructure protection and capacity generation. This can lead to synergistic benefits and potential government support.
• Advocate for Clarity: Engage with policymakers to advocate for clearer responsibilities, more specific guidance, and better support mechanisms within national strategies, particularly where they impact private sector operations.

1. For Technical Practitioners and Researchers:

• Influence Policy: Recognize that policy decisions at the national level profoundly shape the operational environment for cybersecurity. Technical practitioners should seek avenues to contribute their expertise to policy development, ensuring strategies are technically sound and implementable.
• Focus on Capacity Building: Actively participate in initiatives aimed at capacity generation, such as mentoring, training, and educational programs, to help address the cybersecurity skills gap identified in most national strategies.
• Develop Actionable Solutions: Researchers should focus on developing solutions that can directly inform and enhance national strategies, such as robust zero-trust architectures, advanced threat intelligence sharing mechanisms, and effective incident response frameworks.
• Understand the "Why": Comprehending the strategic objectives behind national policies can help practitioners better prioritize their technical efforts and understand the broader impact of their work.

By adopting these defensive implications, nations can move towards more resilient, well-governed, and effectively implemented cybersecurity postures, better prepared to face the evolving digital threat landscape.

Key Takeaways

• National cybersecurity strategies are essential but require a robust, comparative evaluation framework to assess their effectiveness and utility.
• The developed framework employs relative scoring based on 268 quantifiable questions, moving beyond ambiguous absolute scores to identify leading practices and areas for improvement.
• Clear codification of responsibilities is paramount for effective strategy implementation, ensuring accountability and follow-through.
• Capacity generation, addressing the cybersecurity labor shortage, is a universal focus across national strategies, highlighting its critical importance.
• Countries like Singapore demonstrate exemplary practices, including robust zero-trust protection plans for government and proactive regional leadership in cybersecurity.
• Understanding and influencing national cybersecurity strategies is crucial for all stakeholders—policymakers, business leaders, and technical practitioners—due to their profound impact on regulations and operational security.

About the Speaker(s)

The presentation was primarily delivered by Fred Heiding, a Research Fellow in Computer Science at Harvard. Heiding brings a strong technical background to the discussion, with extensive experience in red teaming for various devices, including IoT and embedded systems. Notably, he has experience "hacking the King of Sweden" and previously spoke at Black Hat on how AI empowers cyberattacks, and at B-Sides on AI-empowered phishing. In recent years, Heiding has shifted his focus towards the holistic view of computer security, bridging the gap between technical solutions and policy/business considerations, emphasizing the translation of security into growth and profitability for board members and policymakers.

Heiding introduced his collaborators who were instrumental in this research:

• Alex O'Neil, a cybersecurity researcher focusing on the policy side, is an expert on US policies and has conducted significant work on emerging threats such as cryptocurrencies and threat actors like Russia and North Korea.
• Lachlan, a key member of the team that developed Australia's national cyber security strategy, a document widely recognized and emulated by other nations. His practical experience informed the framework's design, aiming to provide tools he would have valued during his strategy creation process.
• Eric Rosenbach, a Senior Lecturer at Harvard Kennedy School, contributes extensive experience from senior security and national security positions within the White House, providing invaluable insights into high-level policy and strategic implementation.

Together, this diverse team combined technical expertise, policy acumen, and practical strategy development experience to create a comprehensive and insightful framework for evaluating national cybersecurity strategies.