Ninety percent login rates tell you almost nothing about whether a system is doing its job. Burton-Jones and Grange showed why measuring adoption instead of effective use is the wrong question.
Ninety percent of employees logged into the new CRM last quarter. The dashboard shows a healthy adoption curve. Leadership declares victory. The IS research question nobody asks: did any of those logins produce faithful representations of customer relationships, or did salespeople open the app, enter the minimum data their manager required, and close it?
I kept running into this problem from different angles while studying for my comprehensive exams. The IS field has spent decades measuring "system use" as a behavioral count: frequency, duration, login rate. DeLone and McLean (1992) built their IS Success model with six interrelated dimensions, and "use" sat right in the middle, connecting system quality and information quality to user satisfaction and net benefits. It was a breakthrough. For the first time, IS had a multidimensional framework that said use is both an outcome of good system design and an antecedent of organizational benefit. The 2003 update kept the structure at six dimensions by adding service quality and folding individual and organizational impact into a broader construct they called net benefits. Feedback loops ran between use and satisfaction: use shapes satisfaction, and satisfaction shapes future use.
But the model still treated use as a relatively thin behavioral measure. How much. How often. How long. And that thinness created a practical problem that I think most organizations still have not noticed.
Burton-Jones and Grange (2013) made the correction that I believe is the most important reformulation in the use literature. They defined effective use as using a system in a way that faithfully appropriates the domain representations the system was designed to support. Not how often you log in. Not how much you click. Whether you engage the deep structure of the system to accomplish real domain tasks.
They grounded this in representation theory. Every information system has three structures. Deep structure is the real-world domain the system represents: patients in a hospital, customers in a CRM, financial transactions in an ERP. Surface structure is the representation scheme the system uses to present that domain to users: forms, dashboards, data fields. Physical structure is the technological implementation that delivers the representation: the database, the interface, the code. Effective use happens when a user engages the deep structure. A salesperson who opens the CRM and fills out contact names is interacting with the surface structure. A salesperson who uses the CRM to understand and maintain the actual customer relationship is engaging the deep structure. Same login. Different use. One is counting. The other is representing faithfully.
The distinction Burton-Jones and Grange drew is sharp. Use is not adoption. Adoption is the decision to begin engaging. Use is the engagement itself. Use is also not evaluation. Satisfaction is a judgment about your engagement. Use is the behavioral pattern of engagement. These three constructs, adoption, use, and evaluation, get collapsed in practice all the time. An organization measures logins and calls it adoption, then measures satisfaction and calls it use, then wonders why the correlation between "use" and performance is weak.
Torres and Sidorova (2019) pushed this further in the business intelligence context. They showed that information quality is not a property the system outputs. It is a property constructed through effective use. Their construct decomposes BI and analytics use into transparent interaction, representational fidelity, and actionability. Transparent interaction means the user can see and understand how the system processes data. Representational fidelity means the system's output faithfully maps the domain. Actionability means the user can act on what the system reveals. These three mediate the path from system quality, data quality, and producer expertise to actual performance benefits. Use does not deliver value directly. It delivers value through the quality of engagement that effective use names. This is the deeper point from Burton-Jones and Grange: the system represents a domain, and use is meaningful only against that representation.
I think this distinction explains something I keep seeing in enterprise software deployments. A company rolls out Salesforce or SAP or Workday. The adoption metrics look great: login rates above 85%, session duration numbers climbing, training completion rates at 100%. But the actual domain representation is broken. Salespeople enter data to satisfy managers, not to represent customer relationships. Managers read dashboards to justify decisions they already made, not to understand what the data reveals. Nurses click through EHR templates to document compliance, not to support clinical reasoning. The system is "used" in the behavioral sense. It is not used effectively in the representational sense.
This is where the DeLone and McLean model and the effective use construct connect in a way that I think most practitioners miss. DeLone and McLean gave us the dimensions of success: system quality, information quality, service quality, use, satisfaction, net benefits. Burton-Jones and Grange gave us the construct that makes the "use" dimension meaningful. Without effective use, the IS Success model has a black box where use sits. You can measure system quality. You can measure satisfaction. But if use is just frequency, the chain from quality to benefit breaks because frequency does not guarantee that the system is doing what it was designed to do.
The practical implication is straightforward but uncomfortable. If effective use is faithful domain representation, then organizations need to measure whether their systems are helping people represent their domain better, not whether people are logging in. For a CRM, that means measuring whether customer records actually reflect real customer relationships over time, not whether 90% of the sales team opened the app this week. For a BI platform, it means measuring whether analysts make better decisions with the data, not how many reports they downloaded. For an EHR, it means measuring whether clinical documentation supports patient care, not how many template fields clinicians filled out.
Burton-Jones and Straub (2006) laid the groundwork for this by arguing that use is not a simple behavior but a complex adaptive process involving the user's information processing. When an organization treats use as a count, it is measuring the physical structure, not the deep structure. It knows people are clicking. It does not know what those clicks represent.
I wrote about how delegation theory replaces use for agentic systems, and that reformulation matters precisely because the use construct was already broken before AI entered the picture. If we could not measure use properly for databases and ERPs, we are certainly not going to measure it properly for systems that act on our behalf. Burton-Jones and Grange diagnosed the problem. Baird and Maruping (2021) built the next construct for a world where the user is not the only actor. And as I argued in my post on why the productivity paradox is still alive, the gap between spending and value persists partly because we keep measuring the wrong dependent variable.
The uncomfortable truth for anyone running an enterprise system is that your adoption metrics are probably lying to you. A 90% login rate with 10% faithful domain representation is a failing system that looks like a success. A 40% login rate with 90% faithful domain representation might be a system that is doing exactly what it was designed for, just not by everyone. The question is not whether people use your system. The question is whether, when they use it, they engage the deep structure and represent the domain well enough to produce the benefit the system was built to deliver.
About the author
Share
More notes
Related notes