Superintelligence Risk Is Managed by Organizations, Not Only by Models
The most practical part of the catastrophic risk literature is about safety culture, separation of duties and stolen model weights. That is organizational and security ground, and IS already studies it.
Most writing on superintelligence risk is about the model, its goals and its abilities. When I read Hendrycks, Mazeika and Woodside (2023), the section that held my attention was the one on organizations. They describe "organizational risks, highlighting how human factors and complex systems can increase the chances of catastrophic accidents". That is IS ground. We study how organizations build, run and govern complex systems, and how they fail.
Their advice reads like something from a safety management text. "Thus, the focus for organizations dealing with complex systems, especially deep learning systems, should not be solely on eliminating accidents, but rather on ensuring that accidents do not cascade into catastrophes." They borrow the Swiss cheese model, in which "safety is not achieved with a monolithic airtight solution, but rather with a variety of safety measures." And they describe a culture. "A strong safety culture means that members of an organization view safety as a key objective rather than a constraint on their work." The opposite case will be familiar to anyone who has watched compliance work go wrong. "For AI developers with weak organizational safety, safety is merely a matter of box-ticking."
Two of their warnings connect directly to things IS researchers care about. The first is about evidence. "Currently, most organizations proceed by gut feeling, appeals to authority, and intuition to determine whether a safety intervention would reduce overall risk." The second is about claims. They warn against safetywashing, which they define as "overstating or misrepresenting one's commitment to safety". Both are questions about how organizations produce and use information about their own risks.
Parts of the same section read like a security text, which is why I think this topic belongs next to cybersecurity. "A security mindset, widely valued among computer security professionals, is also applicable to organizations developing AIs. It goes beyond a questioning attitude by adopting the perspective of an attacker and by considering worst-case, not just average-case, scenarios." They recommend separation of duties, to "distribute control among different agents, preventing any single individual from wielding undue influence over the entire system". And they name a threat that any security team would recognize. "State, industry, and criminal actors are motivated to steal model weights and research IP." The most capable systems are also information assets, and many of the controls that protect them are familiar ones.
Tests on the model alone are not enough. Bengio et al. (2024) write that evaluations "can sometimes demonstrate dangerous capabilities but cannot reliably rule them out". They also warn that "Purely behavioral evaluations may fail for advanced AI systems", since such systems "might behave differently under evaluation, faking alignment". The International AI Safety Report (2025) names the problem this creates for decision makers. "The pace of advancement in general-purpose AI creates an 'evidence dilemma' for decision-makers." Acting early "might prove unnecessary, but waiting for conclusive evidence could leave society vulnerable to risks that emerge rapidly".
IS research already has a concept for deploying systems that cannot be fully inspected. Asatiani et al. (2021) present "an approach for implementing inscrutable (i.e., nonexplainable) artificial intelligence (AI) such as neural networks in an accountable and safe manner in organizational settings." They call it envelopment, which means "establishing clear boundaries within which the AI is to interact with its surroundings, choosing and curating the training data well, and appropriately managing input and output sources". They argue that "envelopment is fundamental to enabling an organization to use inscrutable systems safely even in settings that necessitate explainability".
I am not saying that envelopment solves superintelligence risk. Their work was about systems in ordinary organizational use, not about the most advanced models. I am saying that the unit that manages the risk is the organization, and IS already studies how organizations wrap controls around systems they cannot fully see into. Bostrom (1998) was careful to say that "Entities such as companies or the scientific community are not superintelligences according to this definition." That is fair. But Drexler (2019) reminds us that "It is a truism that human organizations can achieve tasks beyond individual human competence by employing, not just many individuals, but individuals who perform differentiated tasks using differentiated knowledge and skills." The organizations that build these systems are themselves complex systems, and their failures are the ones Hendrycks and colleagues worry about.
For an IS research agenda, this suggests questions that do not need a superintelligent system to study. How do AI developers decide that a safety step is worth its cost? Who can stop a release, and has anyone ever done it? How is separation of duties set up, and does it survive deadline pressure? How are model weights protected, and who has access? These are questions about governance and control in organizations, and we have the methods to answer them. They also connect to how control is handed over, because the organization that delegates decisions to its systems is the same organization that decides how carefully to check them. And they connect to an older worry of mine, that many AI policies are old IT policies with AI pasted in. A safety culture that is only on paper is exactly the box-ticking that Hendrycks and colleagues describe.