Human-AI Collaboration Needs Two Parties Who Know Different Things

A model and a person who know the same things make a weak team. In the evidence, the gains come from what one side knows and the other does not, and that is an information system design question.

The question I now ask about any human-AI team is what the person brings that the model does not have. It sounds obvious. When the person and the model see exactly the same case and the same inputs, though, it is fair to ask what the person is there to add. The evidence I read for this note says the answer decides whether the team works at all.

Dellermann et al. (2019) define hybrid intelligence as "the ability to achieve complex goals by combining human and artificial intelligence, thereby reaching superior results to those each of them could have accomplished separately, and continuously improve by learning from each other." What I find most useful is the contrast they draw. Collective intelligence "typically refers to large groups of homogenous individuals (i.e., humans or animals), whereas Hybrid Intelligence combines the complementary intelligence of heterogeneous agents (i.e., humans and machines)." The word doing the work in that definition is heterogeneous. If the two agents know the same things, there is nothing to combine.

Hemmer et al. (2025) make the idea testable. They separate two reasons why a person and a model might reach different answers. "Discrepancies in the decision outcome may either stem from different levels of available information (information asymmetry) or from different capabilities to process this information (capability asymmetry)." In their first experiment the model predicted real estate prices from tabular data. Participants were randomly assigned to one of two conditions. In one they saw only the same tabular data, and in the other they also saw images of the house. The model's mean absolute error was $163,080. Teams with the images reached $148,009, and teams without them reached $160,095. The authors write that "the human-AI team outperforms the AI" in both conditions, but for the teams without images "the difference in the treatment without UHCI does not constitute a significant improvement". UHCI is their short name for unique human contextual information. The gain over the model came from the photo that only the person could see.

Vaccaro, Almaatouq and Malone (2024) found the same pattern across more than a hundred experiments. In their words, "when the human alone outperformed the AI alone, the combined human-AI system outperformed both alone with an average pooled effect size for human-AI synergy of g = 0.46". When the model was the stronger party, the combined system lost ground against the model alone, with a synergy effect of g = -0.54. Bansal et al. (2021) put it in one line. "Comparable accuracy does not guarantee complementary partners." Two parties can be equally good and still make a poor team if they get the same cases right and the same cases wrong.

I think this is an IS problem more than a modelling problem, and Dellermann and colleagues say as much. "The rise of AI is now changing the capabilities of IS and the potential distribution of tasks between human and IS dramatically and, hence, affects the core of our discipline." The design question is not only how accurate the model is. It is which information goes to whom. Who sees the photo of the house, the customer's tone, the history that never made it into the database? That is information system design in the oldest sense, routing information to the people and machines that can use it. In the Hemmer experiment, the gain came from a choice about what the person could see. It did not come from a better model or a better explanation.

For practice, this changes the question I would ask before deploying an assistant. What does the person know here that never reaches the model? If the honest answer is nothing, then the collaboration may be a person checking a model on cases where the model is already better, which is the condition under which Vaccaro and colleagues found losses. If the answer is something, the system should be built so that this knowledge actually reaches the decision.

The same reasoning tells me which projects to be wary of. Some deployments are sold as collaboration because a person stays in the loop, but the person gets the same inputs as the model and no time to gather more. By the evidence above, that arrangement has little room to beat the model. It may still be the right arrangement for accountability reasons, which I come back to in the note on who hands work to whom. But it should not be sold as a performance gain.

The limits are the same as in the previous note on the wrong bar. These are mostly lab tasks and mostly decisions. Hemmer and colleagues write that their controlled settings "do not yet consider the wider range of complex human factors that shape the effectiveness of human-AI teams". I take the pattern as strong enough to change the questions I ask. It is not strong enough to predict any one workplace.