Really appreciated this article. It nails the importance of building multi-layered governance structures with dynamic safety controls and real-time oversight. And yes, absolutely: engineers need to be at the heart of AI governance. One thing I’ve been thinking about a lot lately: so much of today’s AI, especially open-source models and API-first tools, comes out of decentralized, fast-moving environments. There's often no single “owner” or controller. So how do we build safety and governance mechanisms that work in those settings? How do you design for distributed control and escalation when there's no central gatekeeper? Would love to hear your take or if you know others tackling this challenge.
Hello James, I really like the idea of using STPA to inform AI system design!
I can think of a couple of challenges: first, STPA is difficult to learn, and I haven't seen it adopted as a practice in software companies, with two exceptions - it was partially implemented at Akamai, and more recently was more fully adopted at Google. The engineers at Google have found STPA useful, and recently presented it at SREcon: https://www.usenix.org/conference/srecon25americas/presentation/klein, but there are few orgs outside Google with the resources and incentives to implement STPA.
Second, as the article "A manifesto for Reality-based Safety Science" points out, STAMP as an accident model is largely unchanged since its introduction in 2004 (https://doi.org/10.1016/j.ssci.2020.104654). I should point out that the goal of the paper is to call attention to a more widespread problem in safety science - a lack of empiricism.
Hi John, I think you're right that “full-fat” STPA can feel too heavy for most software teams, but I think there are some core elements in the mindset that are still worthwhile. I think it's valuable to start with just three ideas: safety is an emergent property of the whole sociotechnical system; losses flow from unsafe control actions, not only component failures; and every layer needs constraints from above plus feedback from the one below. Framed that way, engineers can understand that safety needs to be designed for, and incidentally non-engineers can see why (for example) ISO 42001 certification or EU-style product conformance cannot guarantee real-world safety. Even a sketch of the control structure often reveals blind spots like an organisational loop that never updates its process model.
Lightweight adaptations of STPA like these have worked for pockets of other software companies besides Google (including Amazon). Definitely, much more work (and probably some simplification necessary) if we are to see STAMP/STPA/CAST style approaches being more widely used.
Just about to publish the next article that gets into some more detail of that - thanks for your feedback, it helped when I was writing the next one. Let me know what your thoughts are on that.
Thanks, James, I agree with that perspective; I see benefit in adopting that perspective, and other companies have used lightweight versions of STPA, also including Akamai. Reading the next article now!
Hi James, great article and good points. Minor, so please delete / dont show - but to help your "perfect" document - "That error set an cascade in motion that ultimately broke through every defence." Reload as "That error set a cascade ...". Kind regards Gavin
Really appreciated this article. It nails the importance of building multi-layered governance structures with dynamic safety controls and real-time oversight. And yes, absolutely: engineers need to be at the heart of AI governance. One thing I’ve been thinking about a lot lately: so much of today’s AI, especially open-source models and API-first tools, comes out of decentralized, fast-moving environments. There's often no single “owner” or controller. So how do we build safety and governance mechanisms that work in those settings? How do you design for distributed control and escalation when there's no central gatekeeper? Would love to hear your take or if you know others tackling this challenge.
Hello James, I really like the idea of using STPA to inform AI system design!
I can think of a couple of challenges: first, STPA is difficult to learn, and I haven't seen it adopted as a practice in software companies, with two exceptions - it was partially implemented at Akamai, and more recently was more fully adopted at Google. The engineers at Google have found STPA useful, and recently presented it at SREcon: https://www.usenix.org/conference/srecon25americas/presentation/klein, but there are few orgs outside Google with the resources and incentives to implement STPA.
Second, as the article "A manifesto for Reality-based Safety Science" points out, STAMP as an accident model is largely unchanged since its introduction in 2004 (https://doi.org/10.1016/j.ssci.2020.104654). I should point out that the goal of the paper is to call attention to a more widespread problem in safety science - a lack of empiricism.
Hi John, I think you're right that “full-fat” STPA can feel too heavy for most software teams, but I think there are some core elements in the mindset that are still worthwhile. I think it's valuable to start with just three ideas: safety is an emergent property of the whole sociotechnical system; losses flow from unsafe control actions, not only component failures; and every layer needs constraints from above plus feedback from the one below. Framed that way, engineers can understand that safety needs to be designed for, and incidentally non-engineers can see why (for example) ISO 42001 certification or EU-style product conformance cannot guarantee real-world safety. Even a sketch of the control structure often reveals blind spots like an organisational loop that never updates its process model.
Lightweight adaptations of STPA like these have worked for pockets of other software companies besides Google (including Amazon). Definitely, much more work (and probably some simplification necessary) if we are to see STAMP/STPA/CAST style approaches being more widely used.
Just about to publish the next article that gets into some more detail of that - thanks for your feedback, it helped when I was writing the next one. Let me know what your thoughts are on that.
Thanks, James, I agree with that perspective; I see benefit in adopting that perspective, and other companies have used lightweight versions of STPA, also including Akamai. Reading the next article now!
Hi James, great article and good points. Minor, so please delete / dont show - but to help your "perfect" document - "That error set an cascade in motion that ultimately broke through every defence." Reload as "That error set a cascade ...". Kind regards Gavin
Thanks for that Gavin - appreciate you reading and letting me know. Humans make errors even when they're writing about how to control for errors!