Creating your AI Risk Management Framework (Part 1)

A practical outline of how to build the core risk management framework for AI Governance, starting with approaches to risk categorisation and an overview of useful guidance and resources.

Mar 24, 2025

Most Australians know Bunnings for their weekend sausage sizzles - that unmistakable smell of onions and sausages wafting across the parking lot on a Saturday morning. It's about as Australian as it gets. But between 2018 and 2021, while we were lining up for our snags, Bunnings was grappling with a serious problem: retail theft and violence against their staff were on the rise.

Their solution was technically sophisticated and yet seemed straightforward to implement at the time: use facial recognition technology across dozens of stores to identify known troublemakers before they could cause harm. The system would scan faces as customers enter, compare them against a database of individuals previously linked to theft or violence, and alert security if a match was found. For everyone else, their images would vanish within seconds.

But in 2024, this well-intentioned security measure ran headlong into a privacy crisis. The Office of the Australian Information Commissioner (OAIC) ruled that Bunnings had breached privacy laws - the same stores where Australians felt at home spending their weekends had been secretly scanning customers' faces without proper consent.1

The fallout was significant. Bunnings had collected sensitive biometric data without explicit consent, failed to adequately inform customers, and omitted crucial details from their privacy policy. They were ordered to cease their use of facial recognition and destroy all collected data. Their reputation took a hit as customers questioned whether their local hardware store had covertly collected their biometric data while they browsed for potting mix or picked up supplies for weekend projects.

Bunnings appealed the decision, arguing for staff safety, and the case has become a watershed in how Australian businesses approach AI adoption - particularly the delicate balance between security and privacy rights. It exposed a critical loophole in privacy laws – the concept of "implied consent" through signage. The controversy is ongoing, areminder of AI's double-edged nature: even well-intentioned applications can spiral into unforeseen ethical dilemmas if not carefully governed.

Identifying and mitigating AI risks is exceptionally challenging, especially in a landscape where technology and regulations are constantly evolving. What seems like a reasonable solution today might breach tomorrow's privacy standards or raise unexpected ethical concerns. That's why the topic of my next few articles will be about building a robust and adaptable risk management framework. Having a strong risk managemetn approach isn't just good practice—it's essential. Such a framework helps organisations systematically identify potential issues, assess their implications, and implement appropriate controls before they become headlines. The Bunnings case reminds us that when it comes to AI governance, good intentions need to be backed by systematic risk management that works.

Traditional risk management frameworks are great at managing what we understand well. We know how to approximate the likelihood of a security breach, estimate the impact of system downtime, or calculate the costs of data loss. But AI systems introduce deep uncertainties, and agentic AI systems have make consequential decisions despite those uncertainties. Models can develop subtle biases that emerge only gradually, their behaviour drifting as the world around them changes. New generations of AI models can outperform their predecessors in all bench evaluations, yet introduce new ways to fail unpredictably. The chain of causation between action and harm isn't always clear or immediate. Failures cascade and feed back in unpredictable ways.

This uncertainty doesn't mean we should abandon structured approaches to risk management. Quite the opposite, AI's complexity makes it even more essential to put in place such frameworks - but we need ways to identify, meaningfully assess, and mitigate dynamic, fast-moving risks that we may not fully understand. The key is building frameworks that embrace this uncertainty while remaining practical enough for teams to actually use. Over the few articles, I’ll describe some of the unique aspects of AI risk management and go through techniques I apply to risk identification and risk assessment. In a final article, I’ll provide a complete AI Risk Management Policy that reflects these approaches, a template that you might find to be a useful starting point for your own organisation.

The good news is we're not starting from scratch - there are many great resources available. I’m a big fan of the MIT AI Risk Repository2, a resource that has recently become even more useful by incorporating the AI Incident Tracker3. The combination gives us a framework of over 1000 risks extracted from over 50 frameworks, along with a record of almost a thousand real-world AI incidents. It’s a goldmine of information, providing a rich understanding of how AI systems fail in practice, and an incredibly useful resource to think through and communicate the breadth and complexity of risks that can emerge. Then we have NIST's AI Risk Management Framework4 which provides a thoughtful methodology for approaching AI risks systematically, and we can borrow from the well-established ISO31000 standard5 and to a lesser extent, it’s AI-specific variant ISO238946. Control guidance like that from the Cloud Security Alliance7 help us understand emerging security threats specific to AI and relevant controls. There’s an abundance of useful inputs; the challenge lies in synthesising the insights they contain into something that works in the messy reality of day-to-day operations.

The governance structures we explored in the previous article provide the foundation - clear decision rights, escalation paths, and oversight mechanisms. But that isn't enough. Now we need systematic ways to identify what could go wrong, assess the implications, and implement controls that work. So let’s get started.

Learning from established frameworks

Risk management frameworks for AI can feel like Lego building blocks scattered across a table. Each piece - NIST's methodology, MIT's risk repository and incident tracker, ISO's risk principles and standards, OWASP's security insights, CSA’s controls - they all offer valuable perspectives. The art lies in assembling these pieces into something cohesive and practical, fit for your own organisation.

Let's start with NIST's AI Risk Management Framework (AI RMF). Released in January 2023, it provides a structured way to think about AI risk across the system lifecycle. What makes it particularly valuable is how it breaks down AI risk into manageable workflow steps: map, measure, manage, govern. For instance, when evaluating a new AI system, the 'map' function guides you through identifying key characteristics and potential impacts before diving into specific risks. This systematic approach helps ensure you don't miss critical considerations.

But NIST's framework alone isn't enough, even with the detailed Profile for Generative AI8. While it tells you what to think about, it doesn't provide a comprehensive picture of what can actually go wrong. That's where MIT's Risk Repository and AI Incident Tracker proves invaluable. With over 1,000 risks within structured domain and causal taxonomies, it provides a vast panorama of what can go wrong, along with real-world examples. Examining these incidents reveals patterns that might not be obvious from theoretical frameworks. For instance, many incidents involve systems working exactly as designed but producing unintended consequences - like recommendation engines amplifying polarising content while optimising for engagement, or as per the Bunnings case, an application of AI for workplace safety that breached privacy consent laws.

Sidebar: A Closer Look at the MIT AI Risk Repository
In navigating the complex landscape of AI risks, the MIT AI Risk Repository is invaluable. It is a comprehensive catalog of AI-related risks. The repository really stands out for its structured yet adaptable approach, organising AI risks into by two principal taxonomies: a high-level Causal Taxonomy and a detailed Domain Taxonomy. The Causal Taxonomy helps specify the underlying factors of AI risks, categorising them based on three aspects:
Entity differentiates whether a risk originates from human decisions, AI system actions, or other ambiguous sources.
Intent addresses whether the risks arise intentionally (expected outcomes) or unintentionally (unexpected outcomes).
Timing identifies if the risks occur during pre-deployment (development and testing phases) or post-deployment (operational usage).
Complementing this is the Domain Taxonomy, which delves into specific AI risks, categorising them across seven domains: (1) Discrimination & toxicity, (2) Privacy & security, (3) Misinformation, (4) Malicious actors & misuse, (5) Human-computer interaction, (6) Socioeconomic & environmental impacts, and (7) AI system safety, failures & limitations. Each domain further subdivides into detailed subdomains, providing practical granularity.
One significant strength of the MIT AI Risk Repository is its integration with the AI Incident Database, which now documents over 4,000 reports relating to over 800 actual incidents. This real-world evidence base helps you see patterns and implications of AI failures beyond theoretical frameworks. It’s also hugely valuable to help illustrate and communicate those risks to leaders and others who can be more receptive and understanding of real-world precedents.

AI Incident Tracker - https://airisk.mit.edu/ai-incident-tracker

ISO 31000 brings a risk methodology perspective. While not AI-specific, its principles for risk management - from establishing context through treatment and monitoring - provide a proven foundation that translates well to AI systems at a basic level. Its emphasis on integrating risk management into organisational processes rather than treating it as a separate activity is particularly relevant for AI, where risks often emerge from the interaction between technical systems and organisational processes. There is a standard called ISO 23894 that attempts to adapt ISO 31000 to AI but in practice, I’m not sure that this adds meaningfully. In my opinion, a major challenge with the ISO approach to risk (embodied in all of 31000, 23894, 42001 and others) is in how they formulate risk as quite a static concept, without a real reflection that AI risks are incredibly dynamic and interrelated. We’ll return to this point in depth within the second and third articles of this series.

Resources like The Cloud Security Alliance’s AI Controls Matrix is a useful recent addition, focusing specifically on security vulnerabilities in AI systems. It highlights unique attack vectors like training data poisoning or model inversion attacks that traditional security frameworks might miss. More importantly, it provides practical guidance on controls and mitigations specific to AI systems.

These frameworks complement each other. Consider evaluating a new machine learning model for deployment. NIST's framework helps you structure the overall assessment process. MIT's risk repository and incident database helps identify potential failure modes based on similar systems. ISO 31000 provides the methodological rigor for analysing and treating risks. The CSA AI Controls Matrix guidance ensures you haven't missed critical security considerations.

But here's a key thing you need to keep in mind: these frameworks serve as guides, not gospel. They're most valuable when adapted thoughtfully as a reference point or grounding to your specific context. For instance, if your industry is healthcare, maybe take NIST's structure but modify the assessment criteria to align with your existing clinical risk frameworks. Incorporate relevant risks and examples of health sector incidents from MIT's database into your scenario planning while adding more healthcare-specific considerations. Map the CSA security controls to your existing security architecture rather than implementing them as separate measures.

NIST's methodology provides an excellent skeleton, MIT's incidents add flesh to theoretical bones, ISO 31000 contributes proven risk management discipline, and control frameworks like the CSA AI Matrix ensures critical security aspects aren't overlooked. Together, they enable a comprehensive foundation for AI risk management, albeit one that needs to be augmented with some real-world practices.

The Foundation: Understanding your AI risk landscape

To capture the unique aspects of AI risk, we need to think across four key dimensions of risk: technical, ethical, operational and strategic.

🔧Technical risks encompass the familiar territory of system failures, security vulnerabilities, and performance degradation. But they also include AI-specific concerns like model drift, where system performance degrades subtly over time as the world changes around it. A fraud detection system might maintain high accuracy scores while slowly becoming less effective because patterns of fraudulent behaviour have evolved.
⚖️Ethical risks represent perhaps the biggest departure from traditional IT risk models. These aren't just about compliance or privacy - they're about fundamental questions of fairness, transparency, and societal impact. A hiring algorithm might be technically perfect but systematically disadvantage certain groups in ways that aren't immediately obvious. A content recommendation system might amplify harmful narratives even while optimising for engagement metrics that looked reasonable on paper.
⚠️Operational risks emerge from how AI systems integrate into business processes and human workflows. These often manifest in the gap between what designers expected and how systems actually get used. A customer service chatbot might work flawlessly in testing but create frustration and inefficiency when deployed because it doesn't align with how service representatives actually handle complex cases.
🌐Strategic risks often prove the most challenging to manage because they emerge from the broader implications of AI deployment. A company might successfully deploy an AI system that works exactly as intended, only to face reputational damage or regulatory scrutiny because they didn't fully consider how stakeholders would perceive it. The rapid evolution of AI capabilities and regulations makes these risks particularly dynamic.

MIT's taxonomy is great at helping identify blind spots in how we think about these risks. (By the way - I recommend reading the preprint of their paper9, then scanning a selection of the 50+ frameworks they map from in the database itself, referring back to the database as you read).

As you consider AI risks, it is worth paying attention to how patterns and connections emerge that might not be obvious from first principles. For instance, how often AI failures stem not from technical bugs but from misalignment between system behaviour and human expectations, and how they cascade between categories. Consider, just as an example, how fraudsters' adoption of AI voice cloning technology demonstrates a cascade of risks10. What begins as a technical capability - the ability to synthesise highly convincing voice replicas - quickly evolves into an operational challenge for banks and businesses who must revamp their authentication systems. Their existing security frameworks, built around traditional voice verification, become vulnerable to increasingly sophisticated impersonation attacks. This technical vulnerability then spills into ethical territory as fraudsters disproportionately target elderly individuals, exploiting their trust in familiar voices and potentially eroding faith in legitimate digital banking services.

The ripple effects continue into strategic risks as financial institutions face mounting pressure to address this threat. They must balance the need for stronger authentication against customer experience, all while managing potential reputational damage from successful scams. When fraud losses from AI-enabled attacks are projected to reach $40 billion in the US by 202711, the strategic implications become clear. Organisations must not only defend against current threats but anticipate how rapidly evolving AI capabilities might create new attack vectors, turning what seems like a purely technical or operational challenge into a complex web of interconnected risks that span operations, ethics, and long-term strategy.

A traditional risk mindset might have caught each individual aspect but missed how they can interact, and how fast they can cascade. AI risks often materialise through the interplay of these dimensions rather than in isolation. Technical issues cascade into ethical concerns, operational problems amplify strategic risks. Understanding these interactions proves crucial for effective risk management. This multidimensional nature of AI risk demands a sophisticated approach to risk identification and assessment.

And so, armed with understanding of the four major categories of AI risks, an outline of available frameworks and resources, my next article will go on to explore practical techniques for identifying the most important risks to focus on. I’ll illustrate five approaches I’ve used in the past: incident pre-mortem, pattern mining, time-horizon scanning, red-teaming and dependency chain analysis.

Thank you for reading, as always I welcome any feedback or suggestions.

Doing AI Governance

Discussion about this post