Human-AI Collaboration Is Judged Against the Wrong Bar
Most human-AI teams beat the person working alone. Far fewer beat the better of the two. The bar we choose decides what we end up calling success.
When someone tells me their team now collaborates with AI, my first question is a dull one. Compared with what? Information Systems research spent a long time asking whether people accept a system and use it. Schemmer et al. (2023) describe the shift well. "Appropriate reliance in AI advice is the next milestone after a decade of research focused on AI adoption and acceptance." They add, "In the use phase of AI, researchers need to find ways to ensure appropriate reliance and, thus, effective use of AI." I agree with the direction. But after reading the evidence on human-AI teams, I think the problem starts one step earlier, with the comparison we choose.
Vaccaro, Almaatouq and Malone (2024) wrote the meta-analysis I wish I had read first. They "analysed 370 unique effect sizes from 106 different experiments published between January 2020 and June 2023". They kept two outcomes apart. The first is "human-AI synergy, where the human-AI group performs better than both the human alone and the AI alone". The second is human augmentation, where the team only has to beat the person working without the model.
The two bars give opposite answers. Against the person alone, the result was strong. "The human-AI systems performed significantly better than humans alone, and this pooled effect size was positive (g = 0.64; t98 = 11.87; two-tailed P = 0.000; 95% CI, 0.53 to 0.74)". Against the better of the two, the result went the other way, because "on average, human-AI combinations performed significantly worse than the best of humans or AI alone (Hedges' g = -0.23; 95% confidence interval, -0.39 to -0.07)". The studies are the same and the teams are the same. Only the bar changed, and the verdict changed with it.
Hemmer et al. (2025) give the stricter bar a name, complementary team performance, which they define as "a level of performance that neither of them can attain individually". Their reading of the field is blunt. "So far, however, CTP has rarely been observed, suggesting an insufficient understanding of the principle and the application of complementarity." They find that people do get better when they work with a strong model, and then they add, "In the vast majority of cases, however, the team performance remains inferior to that of the AI model when performing the task alone". Bansal et al. (2021) read the earlier explanation studies the same way. Explanations moved the team closer to the model, yet "one would achieve an even better result by stripping humans from the loop and letting the AI operate autonomously".
This is where I think IS has something to add, because the bar is a design decision before it is a statistic. If a project reports only the gain over the unaided worker, almost any capable model passes. The team looks like a success even when the model alone would have done better. That is not a small reporting issue. It decides whether the organization keeps a person in the loop for a good reason or only for comfort. It decides what that person is trained to do. And it decides what we later call effective use. A team can look productive against the worker's own baseline and still sit below the model, and the first number will hide the second.
I wrote earlier about augmentation and automation as a framing choice. The two bars are the measurement side of the same choice. Augmentation asks whether the person got better. Synergy asks whether the pair is better than its best member. Most of what gets called collaboration has only been shown to pass the first test.
I want to be careful about how far this goes. Most of the evidence comes from the lab. Vaccaro and colleagues note that "the vast majority (about 85%) of the effect sizes were for decision-making tasks", and that the study setups "may not be representative of the ways human-AI systems are configured in practical uses of AI outside the laboratory". Hemmer and colleagues say something similar about their own experiments, which "do not yet consider the wider range of complex human factors that shape the effectiveness of human-AI teams". So I read these results as a warning about the comparison. They are not proof that teams always fail.
My rule for my own reading is now simple. When a paper or a vendor says a human-AI team works, I look for both baselines. If only one is reported, I assume it is the easy one. The harder question is the one I want to carry into IS studies of AI at work. Under what conditions does the team beat the better of its members, and who in the organization is responsible for knowing which condition holds? The next note takes up the first half of that question, on why collaboration needs two parties who know different things.