Hello James, I really like the idea of using STPA to inform AI system design!
I can think of a couple of challenges: first, STPA is difficult to learn, and I haven't seen it adopted as a practice in software companies, with two exceptions - it was partially implemented at Akamai, and more recently was more fully adopted at Google. The engineers at Google have found STPA useful, and recently presented it at SREcon: https://www.usenix.org/conference/srecon25americas/presentation/klein, but there are few orgs outside Google with the resources and incentives to implement STPA.
Second, as the article "A manifesto for Reality-based Safety Science" points out, STAMP as an accident model is largely unchanged since its introduction in 2004 (https://doi.org/10.1016/j.ssci.2020.104654). I should point out that the goal of the paper is to call attention to a more widespread problem in safety science - a lack of empiricism.
Hi James, great article and good points. Minor, so please delete / dont show - but to help your "perfect" document - "That error set an cascade in motion that ultimately broke through every defence." Reload as "That error set a cascade ...". Kind regards Gavin
Hello James, I really like the idea of using STPA to inform AI system design!
I can think of a couple of challenges: first, STPA is difficult to learn, and I haven't seen it adopted as a practice in software companies, with two exceptions - it was partially implemented at Akamai, and more recently was more fully adopted at Google. The engineers at Google have found STPA useful, and recently presented it at SREcon: https://www.usenix.org/conference/srecon25americas/presentation/klein, but there are few orgs outside Google with the resources and incentives to implement STPA.
Second, as the article "A manifesto for Reality-based Safety Science" points out, STAMP as an accident model is largely unchanged since its introduction in 2004 (https://doi.org/10.1016/j.ssci.2020.104654). I should point out that the goal of the paper is to call attention to a more widespread problem in safety science - a lack of empiricism.
Hi James, great article and good points. Minor, so please delete / dont show - but to help your "perfect" document - "That error set an cascade in motion that ultimately broke through every defence." Reload as "That error set a cascade ...". Kind regards Gavin
Thanks for that Gavin - appreciate you reading and letting me know. Humans make errors even when they're writing about how to control for errors!