Common Method Bias and Why Your Survey Results Might Be Wrong

Podsakoff et al. (2003) is one of the most cited papers in IS and management research. That alone says something. It is cited not because it introduced a new theory or identified a new phenomenon, but because it named and documented a threat that almost every survey researcher faces, and it offered remedies. The paper is about common method variance (CMV): the artificial inflation of observed correlations when both the independent variable and the dependent variable are measured using the same method, from the same source, at the same time.

The study-hub I use for my comprehensive exam preparation confirmed that Podsakoff et al. (2003) is local StudyHub evidence in IS research methods, specifically in the context of validity and measurement. The core finding is straightforward and uncomfortable: if you measure perceived system usefulness and user satisfaction in the same Likert survey given to the same person on the same day, your correlation between those two constructs may be partly or largely driven by the measurement context rather than any real relationship between the underlying constructs.

Let me make this concrete. Say I ask you to rate how useful a system is. Then I ask you, in the same survey, to rate your job satisfaction. By the time you get to the job satisfaction questions, your mood has been shaped by thinking about the system. If the system is frustrating, you are in a slightly worse mental state. If it is pleasant, your job satisfaction rating edges up. Neither of these effects has anything to do with whether the system actually causes changes in your job satisfaction. It is just response context bleeding from one set of questions into the next. Multiply this across hundreds of studies and you get a literature where correlations are systematically inflated, sometimes substantially.

This is not a minor bookkeeping error. It is a threat to construct validity. When I observe a correlation of, say, 0.45 between perceived ease of use and user satisfaction in a survey study, I cannot easily tell how much of that 0.45 reflects a real relationship and how much is the shared-method artifact. The problem is that IS research has been built heavily on exactly this kind of data. A lot of what we believe about technology acceptance, system quality, and user behavior comes from Likert surveys where someone rated their perceptions of one thing and then their perceptions of another thing, all in the same session.

The remedies Podsakoff et al. (2003) recommend are worth knowing because the field uses them, sometimes well and sometimes as ritual. The cleanest fix is procedural: separate the measurement of your predictor and criterion variables in time, or use different methods or different sources for each. If you are studying whether system quality predicts user satisfaction, you could measure system quality through objective log data or a panel of evaluators, and measure user satisfaction through a survey. Now you have different methods for different variables. The shared-method problem is largely gone. Temporal separation is the next best thing: collect the predictor at time one and the criterion at time two, weeks or months later. This does not eliminate CMV entirely, but it reduces it substantially and also gives you better evidence for a causal ordering of the relationship.

The statistical remedy that gets used most often is Harman's single-factor test. The idea is that if common method variance is a serious problem, a single factor should emerge when you run exploratory factor analysis on all of your items, or a single factor should explain most of the variance. The problem is that this test is weak. Podsakoff et al. (2003) are explicit about this: Harman's single-factor test is inadequate as a diagnostic because it is sensitive to the number of items and factors in your model. A study can fail the single-factor test (which looks like "CMV is not a problem") while still having substantial method variance biasing its correlations. The study-hub document confirms this directly: "Harman's single-factor test is widely used but inadequate. Stronger statistical remedies include the unmeasured latent method factor approach and the marker variable approach."

The stronger statistical approach is to specify a common method factor in a structural equation model that captures variance shared by all measured items beyond the theorized constructs. This approach is more demanding but gives you a better estimate of how much shared method variance is actually present. The marker variable approach involves including a theoretically unrelated variable in your survey and using its correlations with other items to estimate method variance. These are rarely used in practice, which says something honest about the gap between what methods papers recommend and what published research actually does.

Here is the part that makes me a little uncomfortable about how the field handles this. A lot of papers run Harman's test, report that the single factor explained less than 50 percent of variance, and move on with a sentence like "common method bias does not appear to be a serious concern." That claim is almost never justified by Harman's test alone. The test is so weak that passing it tells you very little. But the convention is established, so reviewers often accept it, and the paper moves through. This is a case where a ritual has replaced actual diagnosis.

I also want to be honest about the other side of this. CMV can be over-corrected. There are contexts where collecting predictor and criterion from the same source at the same time is the only feasible option and also the theoretically appropriate option. If you are studying user perceptions of a system, both variables exist in the user's head. Separating them temporally might introduce other threats, like memory decay or changes in the context. And sometimes the relationship you are studying is genuinely perceptual, meaning you want to know how one perception relates to another perception, and in that case, same-source measurement is not obviously wrong.

The real issue is that too many IS papers treat survey design as a box to check rather than a methodological choice that shapes what you can and cannot conclude. Common method bias is not always disqualifying. But it requires you to be honest about what you can claim from your data.