What is your AI assurance mindset?
Does your company apply high-integrity assurance, checkbox compliance, or even malicious compliance to the governance of AI?
If you were to discover your company's AI system made a consequential wrong decision using data it wasn't trained to interpret, how would your company respond? Could you trigger an immediate systematic review, with transparent disclosure, and confidently gain leadership support for a temporary pause in deployment? Or would it be quietly logged and dismissed as an edge case, with encouragement from leadership to treat it as a minor event so that operations continue unchanged?
Your answers may reflect the true mindset of your organisation's approach to AI governance – whether it adopts a mindset of high-integrity assurance, checkbox compliance or even malicious compliance. In this article I go through these three different mindsets, how we can strengthen the positive and guard against the worst. I also provide a simple tool you can use to diagnose the true nature of your company’s mindset towards AI Governance – fair warning, it might tell an uncomfortable truth.
Good governance is cultural
The stakes in AI governance have never been higher as intelligent systems increasingly drive decisions that directly impact human lives. While accidents like the GM Cruise incident (that I described in my last article)1 represent the most visible failures, the reality is far more nuanced, insidious and pervasive. The assurance problems of most organisations don't announce themselves with dramatic headlines – they build quietly, seeping into corporate culture through a thousand small decisions that prioritise speed over safety, efficiency over transparency, or compliance over genuine risk management. By the time these accumulated compromises surface in a crisis, the cultural rot has often progressed too far for quick fixes.
Three patterns of AI governance, ranging from proactive rigour to lazy theatre to actively destructive behaviour, reflect fundamentally different organisational philosophies about safety, trustworthiness and responsibility. While some companies build governance as an essential capability that enables innovation, others treat it as a bureaucratic overhead to be minimised. Some even actively work to undermine safety frameworks while maintaining an elaborate theatre of compliance. Understanding these stark differences is crucial because they determine not only how organisations handle crises, but whether they're actively creating the conditions for future failures.
Intelligent systems present governance challenges that fundamentally transcend traditional software engineering. Unlike conventional programs that faithfully repeat the same errors when triggered by specific bugs, AI systems can exhibit unexpected behaviours when faced with scenarios outside their training data. In September 2024, ChatGPT exhibited an unexpected behaviour known as the "Speak First" incident. During this incident, ChatGPT began initiating conversations without being prompted by a user, contrary to its designed behaviour and instruction only to respond when given input. This surprisingly unexpected autonomy raised significant concerns about AI safety and control2. This unpredictability, coupled with the opaque nature of AI decision-making processes, creates a complex landscape for trust and safety governance that demands more than just technical expertise.
The role of an AI assurance professional might seem, on the surface, to be purely technical – a matter of validating models, scrutinising training data, and verifying algorithmic behaviour. But this view misses a crucial truth: effective AI governance requires a sophisticated understanding of human systems, organisational dynamics, and the subtle ways that institutional pressures can erode safety cultures. An assurance professional who can spot every technical flaw in a model but fails to recognise the warning signs of organisational drift toward checkbox compliance is like a doctor who can interpret test results perfectly but misses their patient's obvious distress.
Large-scale engineered systems are more than just a collection of technological artifacts: they are a reflection of the structure, management, procedures and culture of the engineering organisation that created them.' - Nancy Leveson3
This insight from Nancy Leveson, author of the inspiring book “Engineering a Safer World” (well, inspiring to engineers who care about trust and safety) becomes even more critical with AI systems, where the complexity of the technology can often serve as a convenient smokescreen for inadequate governance. Technical sophistication without organisational wisdom creates the dangerous illusion of safety while masking deeper vulnerabilities.
As AI's influence expands across critical sectors – from infrastructure management to medical diagnosis, from financial systems to public safety – the stakes of this challenge grow exponentially. Each new deployment weaves AI more deeply into the fabric of society, amplifying both its potential benefits and its risks. The present hype of agentic AI may only accelerate this enmeshing. Such integration across high-stakes domains demands a new breed of assurance professional: one who can bridge the gap between technical excellence and organisational integrity, who can read both algorithms and human systems with equal clarity, and who understands that the most dangerous failures often originate not in code, but in culture4.
I've witnessed how even well-intentioned companies can drift into dangerous territory. The path to inadequate governance is rarely marked by dramatic decisions or obvious red flags. Instead, it's paved with seemingly reasonable compromises, each small enough to justify in the moment, each adding to an invisible accumulation of risk.
The most dangerous situation is not when organisations have the courage to acknowledge their governance gaps. It's when they've built such a convincing facade of compliance that they've started believing it themselves: they have mastered the art of crafting narratives that make their actions appear reasonable after the fact, they have convinced themselves that zero findings from a complex audit is a great result, rather than a bright red flag. Their leadership have become so skilled at rationalisation that they no longer recognise the difference between managing risks and managing appearances.
The stakes here aren't theoretical. Every time an organisation chooses checkbox compliance over genuine trust and safety practices, they're not just cutting corners, they're actively contributing to a culture that prioritises the appearance of safety over actual safety. Every time a leader pressures teams to minimise "inconvenient" findings, creates elegant but artificial narratives, streamlines oversight processes to the extent they become meaningless, they're not just optimising efficiency – they're signalling that speed matters more than safety.
The good news is that recognition is the first step. I have witnessed how organisations can shift from superficial compliance to a genuine safety culture, but only if they're willing to first acknowledge where they truly stand.
The most dangerous failures often originate not in code, but in culture, especially if that culture prioritises the appearance of safety over actual safety.
The Three Mindsets of AI Assurance
Understanding these three distinct mindsets – high-integrity assurance, checkbox compliance, and malicious compliance – requires looking beyond superficial indicators. Each approach reveals itself not through what organisations say about their commitment to safety and trustworthiness, but through how they behave when faced with difficult choices.
Organisations who practice high-integrity assurance treat safety as a living practice, not a static achievement. When unexpected behaviour in a rare but consequential scenario emerges, they don't just document the issue – they launch a comprehensive review of their entire validation framework. Their leadership don't ask "How quickly can we fix this?" but rather "What else might we be missing?" Engineering teams view the resulting delay not as a frustrating setback, but as an essential part of the development process. They recognise that trust and safety is always about the interplay of technical, human and social factors so they value meaningful human oversight.
In these organisations, documentation isn't just thorough – it's honest about uncertainties and limitations. I’ve had the experience of watching engineering teams systematically dismantle their own safety claims, probing for weaknesses with the same vigour an external critic might apply. Their incident investigations read like scientific papers, focused relentlessly on understanding root causes rather than assigning blame. When they engage with regulators, they share not just their successes but their struggles, treating oversight as a partnership in building safer systems.
Checkbox compliance tells a different story. These organisations master the art of looking good without being good. I've seen instances of immaculate audit trails while actual development practices bear little resemblance to the documented procedures. They'll spend more energy crafting the perfect incident report template than investigating actual incidents. Their audits of vast scope covering multiple compliance frameworks produce zero findings, immaculate and pristine. Their safety metrics always trend positively – not because safety is improving, but because they've learned exactly which numbers to track and how to frame them.
The tragedy of checkbox compliance lies in its waste of potential. These organisations often have talented people and robust technical capabilities, but their energy gets diverted into maintaining appearances rather than building genuine safety. Their safety meetings become exercises in choreography, where everyone knows their lines but nobody is having a real conversation about risk.
But it's the malicious compliance mindset that poses the greatest danger. These organisations don't just fail to build safety – they actively work to undermine it while maintaining an elaborate illusion of responsibility. They approach governance like a game to be won rather than a capability to be built. The cautionary tales of Theranos and FTX provide stark illustrations of this mindset in action. Theranos constructed a sophisticated facade of scientific rigour while actively subverting safety protocols and manipulating test results. Their quality assurance documentation was deliberately crafted to obscure fundamental technical failures. Meanwhile, FTX demonstrated how malicious compliance operates in the financial sector, creating complex risk management frameworks that gave the appearance of responsible governance while systematically concealing massive risks and misappropriations.
The most insidious aspect of these mindsets is how they shape organisational culture over time. High-integrity practices build trust and capability, checkbox compliance breeds cynicism and disengagement, and malicious compliance corrodes the very foundation of a trust and safety culture. What begins as a choice about how to approach assurance eventually becomes a defining characteristic of the organisation itself.
Sidebar: Diagnosing your organisations assurance mindset
Are you prepared to diagnose where your own organisation stands? Here's a simple but revealing diagnostic that cuts through the complexity. You don’t have to share it with anyone, you don’t even have to write it down. Just consider how your organisation would respond to these ten questions, not based on your official policies, but on your actual lived experience. The pattern in your honest answers will reveal your organisation's true mindset.
High-integrity organisations will recognise themselves in answers that consistently favour transparency and learning, even at the cost of short-term progress. If your answers paint a picture of carefully managed appearances and problem-avoidance, you're likely operating in a checkbox compliance environment. But if you find yourself thinking about how to frame your answers rather than simply giving them, how to evade or minimise responsibilities, how to avoid and evade with elegant facade, that's a warning sign of malicious compliance taking root.
Understanding these mindsets is crucial, but the harder question is how to systematically cultivate high-integrity practices while guarding against the subtle drift toward checkbox or malicious compliance. To me, this is where a robust AI Management System becomes essential – not as another layer of bureaucracy, but as organisational architecture that makes doing the right thing the path of least resistance.
A well-designed management system doesn't just prevent failures – it builds capability. Effective AI Management Systems operate at three levels simultaneously: they provide practical mechanisms for safety governance and validation of trustworthiness, they shape organisational culture toward high-integrity practices, and they build institutional muscle memory for responsible innovation. When a team faces a difficult trade-off between rapid deployment and thorough testing, the management system doesn't just provide decision criteria – it creates a context where thoughtful, open deliberation is valued over hasty action.
But here's a crucial point: an AI Management System isn't a shield against malicious compliance – in fact, a poorly implemented system can enable it. The difference lies in how the system is woven into the organisation's daily operations and decision-making. Does it make safety considerations more transparent or more opaque? Does it empower front-line workers to raise concerns or create new layers of bureaucracy between problems and solutions? Does it support an elegant facade while masking real practices, in the end providing only false confidence and a hollowed safety culture?
These aren't just theoretical questions – they have direct business implications. The investment required to build genuine safety capabilities through an AI Management System might seem substantial, but it pales in comparison to the costs of getting it wrong. I believe the investment case for high-integrity assurance of innovative technology is always strong. Because in my experience, organisations that treat high-integrity assurance as an investment rather than an expense aren't just managing risk – they're building a sustainable competitive advantage in an increasingly AI-driven world. That said, it’s not always an easy case to make.
That will be the focus of my next article: how to build a compelling business case for leadership commitment and investment in an AI Management System.
https://www.forbes.com/sites/lanceeliot/2024/09/17/chatgpt-speak-first-incident-stirs-worries-that-generative-ai-is-getting-too-big-for-its-britches/
https://direct.mit.edu/books/oa-monograph/2908/Engineering-a-Safer-WorldSystems-Thinking-Applied