Superintelligence Is a Construct Without an Agreed Measure

Every major definition of superintelligence uses a different yardstick, and expert forecasts move by decades when the question is reworded. For an IS reader that looks like a construct problem before it looks like a risk problem.

5 min read AI & Agentic SystemsIS Theory

I started reading on superintelligence expecting the hard part to be the risk arguments. The hard part turned out to be the definition. Sabherwal and Grover (2024) describe the wider debate on generative AI as swinging between "utopian visions of unprecedented productivity and dystopian fears of humanity's demise", and they add that "This polarized perspective neglects the nuanced, pragmatic manifestation of GAI." Ågerfalk (2020) asks a question I keep coming back to. "Is it not our responsibility as IS scholars to bring clarity to the discourse rather than contributing to its decline?" For superintelligence, I think clarity has to start with the construct.

Here are the definitions I found in the sources I opened. Good (1966) wrote, "Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever." Bostrom (1998) meant "an intellect that is much smarter than the best human brains in practically every field, including scientific creativity, general wisdom and social skills". Bostrom (2009) wrote of "a general intelligence that vastly outperforms the best human brains in every significant cognitive domain". Chalmers (2010) described "AI of far greater than human level (say, at least as far beyond the most intelligent human as the most intelligent human is beyond a mouse)". Morris et al. (2024) set their highest performance level at "outperforming 100% of humans".

These are related, but they are not the same thing. All the intellectual activities of any person, every field, every significant cognitive domain, a gap as wide as the one between a person and a mouse, and every human on a given task. The yardsticks differ, and none of the verbal definitions comes with a way to measure it. Chalmers is candid about the assumption underneath. "The argument depends on the assumptions that there is such a thing as intelligence and that it can be compared between systems". Drexler (2019) doubts the standard definition itself, arguing that "to define superintelligence in terms of adult intellectual competence fails to capture what we mean by human intelligence".

When a construct has no agreed measure, the surveys show it. Müller and Bostrom (2016) defined high-level machine intelligence as "one that can carry out most human professions at least as well as a typical human" and then asked experts, "How likely do you then think it is that within (2 years / 30 years) thereafter there will be machine intelligence that greatly surpasses the performance of every human in most professions?" They are clear that "the aim was to 'gauge the perception', not to get well-founded predictions". In the survey by Grace et al. (2018), high-level machine intelligence "is achieved when unaided machines can accomplish every task better and more cheaply than human workers". In the same survey, "a subset were asked a logically similar question that emphasized consequences for employment." The answers were far apart. "Researchers believe there is a 50% chance of AI outperforming humans in all tasks in 45 years and of automating all human jobs in 120 years". The authors saw the problem. "Yet our survey respondents do not treat them as logically equivalent. We observed effects of question framing in all our prediction questions, as well as in pilot studies."

The gap did not close with a bigger sample. Grace et al. (2025) "surveyed 2,778 researchers who had published in top-tier artificial intelligence (AI) venues", and they told respondents to "Think feasibility, not adoption." Still, "There is about a 70-year difference between the mean 50% prediction for HLMI and the mean 50% prediction for FAOL." HLMI is high-level machine intelligence and FAOL is full automation of labor. Their comment is honest. "We do not know what accounts for this gap in forecasts." One possibility they raise is that people read the labor question as being about adoption, "not mere feasibility", even though the wording asked about feasibility.

For an IS reader this is a familiar problem in an unfamiliar place. When two wordings of what should be the same question move the answer by decades, the trouble is not only the forecasters. The construct is not yet stable enough to be measured. I would not accept that in a survey instrument for a new IS construct. I would ask for a definition, a domain, and some evidence that different items measure the same thing. I do not see why superintelligence should get a pass because the stakes are high. If anything, high stakes are a reason to be stricter.

Good's own paper is a useful reminder here. He predicted that "It is more probable than not that, within the twentieth century, an ultraintelligent machine will be built and that it will be the last invention that man need make". He also wrote that his conclusions "are primarily the opinions of the writer, as they must be in a paper on ultraintelligent machines written at the present time". I find that sentence more useful than the prediction. Grace and colleagues say something similar about their own data. "While AI experts' predictions should not be seen as a reliable guide to objective truth, they can provide one important piece of the puzzle."

What would a measure look like? Morris and colleagues attempt one for general AI, with "levels of AGI performance, generality, and autonomy, providing a common language to compare models, assess risks, and measure progress along the path to AGI." I do not think it settles the question. But it does what Ågerfalk asks of us. It replaces one dramatic word with dimensions that can be argued about one at a time. The next notes in this series take two of those dimensions seriously, autonomy and control and the organizations that build these systems.