Qualitative Coding and the Rigor Problem

Qualitative research in IS often involves taking interview transcripts, documents, or field notes and organizing them into categories. You read through the material, you label sections with codes, you group codes into themes, and eventually you produce a set of categories that you claim represent something meaningful about the phenomenon you studied. This process looks informal from the outside. It is not. But "rigorous" is a word that gets applied to it very loosely, and the field has not agreed on what it means.

The dominant way IS researchers demonstrate rigor in qualitative coding is intercoder reliability. You have two people independently code the same material and then you compare how often they agree. Cohen's kappa is the most common statistic, though percentage agreement is also used. A high kappa looks like objectivity. It suggests that the categories are real enough that different people see them independently. A low kappa suggests the categories are in the eye of the beholder.

This is a useful check. I am not dismissing it. But it is a narrow answer to a broader question. Two coders can agree consistently and still be coding the wrong thing. If both coders share the same theoretical assumptions, the same disciplinary background, and the same interpretation of the research question, they will reliably produce the same codes regardless of whether those codes are the most defensible reading of the data. The agreement is real, but it is agreement among similar minds, not evidence that the coding captures something independent of the researchers' prior views.

Klein and Myers (1999), which has local evidence in the study-hub I use for my comprehensive exam preparation, laid out seven principles for rigorous interpretive IS field research. The one I find most underused in practice is the principle of suspicion, which requires the researcher to question the assumptions embedded in their own interpretation and in the accounts given by participants. Participants have interests. They present themselves in ways that are favorable to them. They emphasize some things and omit others. A rigorous qualitative researcher does not just record what people say and call it data. She reads against the grain of the accounts, looks for silences and contradictions, and asks what the data might be hiding as well as what it reveals.

The Gioia methodology, developed by Dennis Gioia and colleagues and widely cited in management and IS research, offers a structured approach to qualitative theory building. The basic move is to start with "first-order concepts" that are grounded in participants' own language, aggregate these into "second-order themes" that reflect theoretical patterns, and then move to "aggregate dimensions" that map to the theoretical contribution. This structure is appealing because it creates a visible audit trail from raw data to theory. A skeptical reader can trace how the researcher got from what an interview participant said to the theoretical claims in the paper. That transparency is a form of rigor.

But the Gioia methodology has also been criticized, fairly, for how it can become a template rather than a genuine engagement with the data. Researchers sometimes produce the first-order, second-order, aggregate dimensions table as a formatting convention without doing the hard analytical work that the structure is supposed to represent. The table looks tidy. It has quotes. It has arrows pointing from codes to themes to dimensions. And it can still be wrong if the researcher found what they were looking for rather than what was actually in the data.

This is the deeper rigor question in qualitative IS research: are your codes theory-driven or data-driven? This is not a binary distinction. Most good qualitative research involves both. You enter the field with theoretical sensitization, meaning you know the relevant literature and you cannot pretend you do not. That shapes what you notice. But if your categories are so predetermined that you are essentially sorting data into boxes you brought with you, then you are not doing inductive qualitative research, you are doing deductive categorization dressed up as qualitative work.

Suddaby (2006), which is confirmed as local StudyHub evidence in IS research methods, is brutal about misuses of grounded theory. He identifies several ways researchers misuse the label: treating it as an excuse to skip the literature, presenting rich description as if it were theory, counting codes without doing theoretical abstraction, and claiming the software did the analysis. The analytic work is the hard part. The codes are just a way of managing the data. What matters is the theoretical logic that links the codes to each other and to existing theory.

Reflexivity is one honest response to all of this. Being explicit about your own theoretical commitments, your positionality, and your relationship to the research site does not eliminate the interpretive character of qualitative work. Nothing does. But it makes visible what would otherwise be hidden. A researcher who acknowledges that she is studying an organization she previously worked for, or that she entered the study expecting to find a particular dynamic, is more credible than one who presents the analysis as if she arrived from nowhere with no prior commitments.

The other piece that matters and that IS research underemphasizes is negative case analysis. If your proposed categories hold in most of your data but not all of it, what do you do with the exceptions? Reporting and explaining the deviant cases is one of the strongest signals of rigor in qualitative work, because it shows you were not just selecting confirming evidence. It also often forces theoretical refinement. The case that does not fit your emerging theory is frequently the most informative one.

Intercoder reliability is a real check and worth doing. But it is not the ceiling of rigor in qualitative IS research. It is closer to the floor.