AI Governance & Ethics: A Discussion with the Big Players

AI Governance & Ethics: A Discussion with the Big Players

Speakers: Barbara Cosgrove, Chief Privacy Officer, Workday; Ruby Zefo, Chief Privacy Officer, Uber; Emma Redmond, Associate General Counsel and Head of Privacy and Data Protection, OpenAI; Anu Talus, Commissioner, Finnish Data Protection Authority & Chair, European Data Protection Board; Trevor Hughes (Moderator), President and CEO, IAPP

Conference: RSAC 2024

YouTube: https://www.youtube.com/watch?v=pkl0ngaBRiA

Overview

This panel discussion at RSAC 2024 brought together leading voices from industry and regulation to delve into the critical and rapidly evolving landscape of Artificial Intelligence (AI) governance and ethics. Moderated by Trevor Hughes of the IAPP, the session featured Chief Privacy Officers from Workday and Uber, the Head of Privacy and Data Protection from OpenAI, and the Chair of the European Data Protection Board. The central theme revolved around the recently passed EU AI Act, exploring its unprecedented global significance, the practical implications for organizations operating worldwide, and the strategic approaches companies are adopting to navigate this new regulatory frontier.

The talk is crucial because it addresses a fundamental challenge of our digital age: how to responsibly develop and deploy powerful AI technologies while safeguarding fundamental rights, fostering trust, and ensuring a level playing field. With the EU AI Act representing the world's first comprehensive legal framework for AI, its influence is expected to ripple globally, much like the GDPR before it. The panel offered invaluable insights into how this landmark legislation will shape corporate governance, risk management, and ethical considerations for AI systems, making it a must-understand for security professionals, legal experts, and business leaders alike.

Background

▶ Watch: Introduction and expert panelist overview (0:00)

For several years, the rapid advancement of AI technologies outpaced the development of corresponding regulatory frameworks, creating what moderator Trevor Hughes aptly described as a "policy vacuum." While numerous organizations and academics proposed various AI governance frameworks—some "dozens, hundreds even"—there was a distinct lack of a single, comprehensive, and legally binding standard that could provide a clear roadmap for responsible AI development and deployment. This absence led to fragmentation, uncertainty, and a complex environment for global enterprises striving to innovate with AI while managing inherent risks.

The European Union, drawing on its experience with the General Data Protection Regulation (GDPR), recognized the urgent need to establish a harmonized set of rules for AI. The GDPR, enacted in 2018, demonstrated the EU's capacity to set a global benchmark for data protection, influencing legislation and corporate practices far beyond its borders due to the sheer size of its consumer market. This precedent set the stage for the EU AI Act, which aimed to achieve a similar impact in the realm of artificial intelligence. The Act’s development was driven by a shared understanding of common values across international organizations and countries, making it a natural fit for global influence. Anu Talus, Chair of the European Data Protection Board, highlighted the pragmatic consequence of the EU being a massive consumer market with over 450 million consumers, necessitating harmonized rules to protect fundamental rights and create a level playing field for businesses operating within its jurisdiction. The Act thus emerged as a direct response to the policy void, intending to provide the first major, globally influential framework for managing AI risks and fostering responsible innovation.

Key Findings

▶ Watch: Panel dives into the EU AI Act (2:56)

The panel discussion underscored several critical findings regarding the trajectory of AI governance, with the EU AI Act at its core.

Firstly, the EU AI Act has unequivocally passed the EU Parliament, receiving overwhelming political support with 526 votes in favor, 523 against, and 46 abstentions. This significant endorsement positions it as the world's first comprehensive legal framework for AI, a monumental achievement in global regulation. While some procedural "housekeeping" remains, its final publication is imminent, with phased implementation periods stretching over six to eighteen months to allow for adaptation. This phased rollout will also see the addition of further guidance, codes of conduct, and interpretations, providing necessary clarity over time, particularly concerning general purpose AI and systemic risks.

Secondly, the Act is poised to exert enormous global influence, echoing the precedent set by the GDPR. Anu Talus emphasized that given the EU's vast consumer market, any company wishing to do business in the EU will need to respect these harmonized rules, which protect consumers and fundamental rights. This pragmatic reality ensures that the EU AI Act will be closely watched for its implementation and impact, likely serving as a de facto global standard for AI governance.

Thirdly, organizations are proactively developing sophisticated AI governance frameworks to prepare for and comply with emerging regulations. Barbara Cosgrove of Workday outlined a model based on principles, practices, and people. This includes defining executive-level commitments (e.g., amplifying human potential, championing transparency and fairness), translating these into practical responsible AI guidelines and standards for developers, and establishing cross-functional Responsible AI Steering Committees with diverse representation (legal, privacy, diversity, product, technology). Workday has leveraged the NIST AI Risk Management Framework as a foundational guide for building these guidelines.

Fourthly, there is a strong consensus on the efficiency of leveraging existing governance structures, particularly those established for privacy and data protection. Ruby Zefo of Uber highlighted their programmatic, risk-based system for traditional machine learning, which embeds model analysis into existing Privacy Impact Assessment (PIA) processes. This approach avoids creating entirely new evaluation systems, making compliance more efficient and repurposing existing expertise. OpenAI's Emma Redmond also advocated for "recycling" muscle memory from GDPR, such as Privacy by Design and DPIAs, as contextually relevant tools for AI governance.

Finally, the discussion revealed the emerging importance of external, third-party assessments for AI systems. Ruby Zefo noted Uber's proactive step in commissioning a full civil rights, third-party assessment that included both privacy and AI recommendations. This signals a growing recognition that complex issues like algorithmic transparency, fairness, and bias may require independent validation to ensure accountability and build trust, a direction many organizations are expected to follow.

Technical Deep Dive

▶ Watch: OpenAI's Emma Redmond details AI Act's phased rollout (3:58)

While the panel’s discussion focused primarily on policy, ethics, and organizational frameworks, the "technical deep dive" can be understood through the lens of how these regulatory and corporate governance architectures are constructed to manage the inherent complexities of AI systems. The EU AI Act itself represents a technical architecture for risk management, and the companies discussed their internal technical approaches to operationalizing AI governance.

The EU AI Act introduces a risk-based approach to AI regulation, a foundational "technical" element of its design. It categorizes AI systems into four levels:

1. Unacceptable Risk: AI systems that pose a clear threat to fundamental rights, such as social scoring by governments or real-time remote biometric identification in public spaces, are outright banned.
2. High-Risk: This category is the core of the Act, covering AI systems used in critical areas like employment and worker management, critical infrastructure, law enforcement, biometric identification, education, and healthcare. For these systems, the Act imposes stringent technical and organizational requirements. These include:

• Robust Risk Management Systems: Continuous processes to identify, analyze, and mitigate risks throughout the AI system's lifecycle.
• Data Governance: Requirements for high-quality training, validation, and testing datasets to minimize risks of bias and discrimination.
• Technical Documentation: Detailed records of the system's design, purpose, capabilities, and performance, ensuring traceability and transparency.
• Human Oversight: Mechanisms to ensure that human beings can effectively oversee and intervene in the AI system's operation, preventing over-reliance or autonomous harmful decisions.
• Robustness, Accuracy, and Cybersecurity: Technical specifications to ensure AI systems are resilient to errors, manipulation, and security vulnerabilities.
• Conformity Assessment: Before deployment, high-risk AI systems must undergo a conformity assessment to verify compliance with the Act's requirements.

1. Limited Risk: AI systems with specific transparency obligations, such as chatbots or deepfakes, where users must be informed that they are interacting with an AI or seeing AI-generated content.
2. Minimal Risk: The vast majority of AI systems, which are subject to minimal or no specific obligations, but developers are encouraged to adhere to voluntary codes of conduct.

This tiered approach is a technical design choice to focus regulatory burden where the risks are highest, while fostering innovation in lower-risk areas.

From an organizational perspective, companies like Workday and Uber have developed internal "technical" frameworks for AI governance. Workday’s approach, built on principles, practices, and people, provides a structured methodology:

• Principles: These serve as the high-level architectural blueprint, guiding the fundamental "should we build this?" questions. Workday’s principles include amplifying human potential, positively impacting society, championing transparency and fairness, and delivering on privacy and security commitments.
• Practices (Responsible AI Guidelines and Standards): These are the detailed technical specifications and operational procedures for developers. Workday has baked the EU AI Act's emerging requirements into these guidelines, leveraging frameworks like the NIST AI Risk Management Framework as a "checklist" to ensure comprehensive coverage. These standards dictate how AI models are designed, developed, tested, and deployed, ensuring adherence to ethical and legal requirements.
• People (Cross-functional Responsible AI Steering Committee): This committee, comprising diverse leadership (General Counsel, Chief Responsible AI Officer, Chief Diversity Officer, Chief Product Officer, Chief Technology Officer), acts as the governance "architecture review board," ensuring buy-in and consistent application of guidelines across product decisions.

Uber’s approach to machine learning (ML) governance similarly integrates technical and operational components. Ruby Zefo distinguished between traditional machine learning (where algorithms are specially trained for specific tasks, like marketplace efficiency or ID verification) and Generative AI. Uber's focus on traditional ML for its core operations (e.g., matching drivers/riders, personalized recommendations in Uber Eats) necessitates a programmatic, risk-based system. This system technically embeds model analysis directly into existing product and engineering development processes, specifically by revising and extending their Privacy Impact Assessment (PIA) questionnaires. This integration streamlines the evaluation of AI models for potential risks, including fairness and the prevention of bias or disparate impact. Furthermore, Uber incorporates human review as a critical technical control, particularly for decisions that could lead to someone losing access to the platform. Their executive cross-functional council and the unique third-party civil rights assessment represent technical processes for external validation and continuous improvement of their AI systems.

OpenAI, as a leading developer of generative AI, emphasizes a "research lab" mindset that facilitates a holistic and global perspective. Their internal "ways of working" involve specialized teams like red teamers (who technically probe systems for vulnerabilities and misuse) and preparedness teams (who assess and plan for future risks). Emma Redmond stressed the importance of "context is king" – understanding the specific technical use cases of AI tools within an organization – and "recycling" existing muscle memory from privacy by design and Data Protection Impact Assessments (DPIAs) to build robust AI governance. This involves adapting established technical risk assessment methodologies to the unique challenges of AI.

In essence, the "technical deep dive" for this talk involves understanding the structural design of the EU AI Act's risk management framework and the architectural approaches companies are taking to build integrated, principle-driven, and risk-based governance systems for AI development and deployment.

Demo / Proof of Concept

▶ Watch: AI Act's potential global influence, akin to GDPR (7:10)

This panel discussion was focused on policy, governance frameworks, and strategic organizational responses to AI regulation, rather than demonstrating specific AI tools or security vulnerabilities. As such, no live demo or proof of concept was presented during the talk.

Defensive Implications

▶ Watch: EDPB Chair Anu Talus on AI Act's broad global impact (7:52)

The insights from this panel offer crucial defensive implications for organizations grappling with the complexities of AI governance and cybersecurity. Adopting these strategies can help mitigate legal, ethical, and reputational risks.

1. Establish a Principle-Driven AI Strategy: Organizations must start by defining their own core principles and values for AI development and deployment. As Barbara Cosgrove articulated, this "North Star" dictates not just compliance, but "the right thing to do." These principles (e.g., human-centered, fair, transparent, privacy-preserving) should guide all AI initiatives from ideation to deployment, providing a foundational defense against misaligned or unethical AI use.
2. Cultivate a "Village" Approach to Governance: AI governance is not a siloed function. It requires broad, cross-functional collaboration across legal, privacy, product, engineering, diversity, and executive leadership. Establishing a Responsible AI Steering Committee or an executive cross-functional council (as seen at Workday and Uber) ensures diverse perspectives, shared ownership, and comprehensive risk assessment. This collective responsibility is a robust defense against blind spots and fragmented decision-making.
3. Integrate AI Governance into Existing Structures: Avoid building entirely new, parallel governance systems. Leverage existing "muscle memory" and operational frameworks, particularly those developed for privacy and data protection. Extending Privacy Impact Assessments (PIAs) to include model analysis, human rights impact assessments, or fairness/bias impact assessments is an efficient defensive strategy. This streamlines processes, reduces overhead, and capitalizes on established expertise, as highlighted by Ruby Zefo and Emma Redmond.
4. Implement Continuous Upskilling and Training: The AI landscape is rapidly evolving. Regular, enterprise-wide training is paramount, not just for specialists but for all employees who interact with or are impacted by AI tools. This ensures that junior staffers using AI in marketing or HR platforms, as well as senior leadership, understand their responsibilities, organizational guidelines, and the inherent risks. This continuous education acts as a critical human firewall against misuse and non-compliance.
5. Prioritize Risk-Based Assessments and Controls: Adopt a programmatic, risk-based approach to evaluating AI models, distinguishing between traditional machine learning and generative AI. For high-risk systems, implement robust data governance, technical documentation, human oversight, and measures for robustness, accuracy, and cybersecurity, as outlined by the EU AI Act. This proactive risk identification and mitigation is essential for preventing harmful outcomes.
6. Consider External Third-Party Validation: For complex and sensitive AI applications, especially those touching on fairness, bias, and algorithmic transparency, consider engaging independent third-party assessments. Uber's civil rights assessment is a pioneering example. External review adds a layer of objective scrutiny, builds trust, and provides a strong defensive posture against accusations of bias or lack of transparency, which are notoriously difficult to address internally.
7. Emphasize "Context is King": Organizations must deeply understand how AI tools are actually being used within their specific operational context. Emma Redmond's emphasis on this point is crucial. Generic policies are insufficient; defensive strategies must be tailored to the specific applications, data involved, and potential impact on individuals and society. This granular understanding allows for the deployment of targeted controls and mitigations.

By adopting these defensive implications, organizations can move beyond mere compliance, building resilient, ethical, and trustworthy AI systems that foster innovation while protecting users and upholding fundamental rights in an increasingly AI-driven world.

Key Takeaways

• The EU AI Act is a landmark piece of legislation, representing the world's first comprehensive legal framework for AI, poised to exert significant global influence akin to the GDPR.
• Organizations should proactively develop robust AI governance frameworks grounded in clear principles, practical guidelines, and cross-functional collaboration, involving diverse stakeholders from legal to engineering.
• Efficiency in AI governance can be achieved by leveraging and extending existing structures, particularly those established for privacy and data protection, such as Privacy Impact Assessments (PIAs) and Privacy by Design methodologies.
• Continuous upskilling and training across the entire enterprise are critical to ensure all staff understand their roles and responsibilities in the rapidly evolving AI landscape.
• External, third-party assessments are emerging as a vital tool for validating fairness, mitigating bias, and enhancing algorithmic transparency, particularly for high-risk AI systems.
• A contextual understanding of how AI tools are used within an organization is paramount for effective governance, tailoring strategies to specific applications and their potential impacts.

About the Speaker(s)

Trevor Hughes is the President and CEO of the IAPP (International Association of Privacy Professionals), a global professional home for individuals working in privacy, data protection, and AI governance. He served as the moderator for this panel.

Barbara Cosgrove is the Chief Privacy Officer at Workday, a global provider of enterprise HR and financial services. She is responsible for guiding Workday's approach to AI governance, emphasizing a framework built on principles, practices, and people.

Ruby Zefo is the Chief Privacy Officer at Uber, where she focuses on programmatic, risk-based systems for machine learning, integrating model analysis into existing privacy processes. She also initiated an executive cross-functional council for AI governance at Uber.

Emma Redmond serves as Associate General Counsel and Head of Privacy and Data Protection for OpenAI, a leading developer of generative AI technologies. She also contributes to the AI Advisory Council for the Irish Government, offering a unique perspective on both industry innovation and regulatory development.

Anu Talus is the Commissioner of the Finnish Data Protection Authority and, crucially, serves as the Chair of the European Data Protection Board (EDPB). In this role, she convenes all data protection authorities in Europe, making her a preeminent data protection regulator globally, particularly regarding the influence of the GDPR and now the EU AI Act.