Human-AI Collaboration Needs a Stopwatch, Not Only a Survey
In three studies of AI help at work, what people believed about their speed and what was measured did not match, and the gap ran in both directions.
A lot of what we know about technology at work comes from asking people. Did the system help you? Do you work faster with it? I understand why. Many outcomes are hard to observe, and what people believe about a tool shapes what they do with it. But three studies of AI assistance that I read side by side made me less comfortable treating the answer to that question as a measure of the outcome itself.
Becker et al. (2025) ran a randomized controlled trial in which "16 developers with moderate AI experience complete 246 tasks in mature projects on which they have an average of 5 years of prior experience." The result is the reason I wrote this note. "Before starting tasks, developers forecast that allowing AI will reduce completion time by 24%. After completing the study, developers estimate that allowing AI reduced completion time by 20%. Surprisingly, we find that allowing AI actually increases completion time by 19%". The authors call this "a large disconnect between perceived and actual AI impact on developer productivity".
Peng et al. (2023) found a gap in the opposite direction. In their experiment, "The treatment group, with access to the AI pair programmer, completed the task 55.8% faster than the control group." Yet "participants in both treated and control groups estimated a 35% increase in productivity, which is an underestimation compared with the 55.8% increase in their revealed productivity." So in one study people believed in a gain that was not there, and in the other they believed in a smaller gain than the one they actually had.
Brynjolfsson, Li and Raymond (2025) did not rely on self-report at all. They worked from "data from 5,172 customer-support agents" in one firm. Their main finding is that "Access to AI assistance increases worker productivity, as measured by issues resolved per hour, by 15% on average, with substantial heterogeneity across workers." The spread interests me more than the average. "Less experienced and lower-skilled workers improve both the speed and quality of their output, while the most experienced and highest-skilled workers see small gains in speed and small declines in quality." For the less skilled and less experienced the gain was large, "including a 30% increase in the number of issues resolved per hour".
Put the three together and the sign of the effect changes with the person and the task. Experienced developers working in code they knew well were slowed down. Programmers given a standardized task were sped up. New support agents gained a lot, and top agents gained a little speed and lost a little quality. And the people inside these studies could not reliably tell which case they were in. If a survey had been the only instrument in the first study, it would have reported a gain of about a fifth, roughly the size of the loss that the clock recorded.
For IS research this does not mean throwing perceptions away. I still think perceptions matter, because people decide whether to keep a tool partly on what they believe it does for them. It means not using perceptions as a stand-in for performance when the question is performance. Becker and colleagues make a related point about earlier field studies, which measured things like lines of code, commits and pull requests. They describe these as outcome measures "that are not fixed in advance", since AI assistance can move such counts without making anyone more productive. When I design a study of AI at work, I want an outcome that is fixed in advance and observed, with self-report next to it as a separate variable. The gap between the two may be one of the more interesting things to explain.
Brynjolfsson and colleagues add a finding that I did not expect. "In our data, top workers increase their adherence to AI recommendations, even though those recommendations marginally decrease the quality of their conversations." Their worry is about the next version of the tool. "Yet with fewer original contributions from the most skilled workers, future iterations of the AI model may be less effective in solving new problems." That is a measurement problem as much as a productivity problem, because no measure of this quarter's output shows what the next version of the tool loses.
Each author group warns against reading too much into one setting. Brynjolfsson and colleagues say that "our findings apply for a particular AI tool, used in a single firm, in a single occupation, and should not be generalized across all occupations and AI systems". Becker and colleagues write, "We caution readers against overgeneralizing on the basis of our results." Peng and colleagues chose "a standardized programming task in an experiment to obtain a precise measure of productivity, instead of a task where developers collaborate on large projects". I agree with all three warnings. My point is narrower. Whatever the setting, the clock and the survey can disagree, and only one of them was built to measure time.
That is also why I read claims about AI and productivity in organizations with one question in mind. Was the gain measured, or was it reported? Both are worth knowing, and they are not the same thing. The previous note on the wrong bar asked what the team is compared with. This one asks who is doing the measuring.