Critical Realism Explains Why AI Models Are Not the Mechanisms They Simulate

I kept reading about distribution shift and model collapse last month and something would not let go. A neural network predicts loan defaults with 95% accuracy on its test set. The bank puts it into production. Six months later the economy changes and the same model drops to 60% and nobody in the bank understands why, because the model did not learn why defaults happen. It learned a correlation in historical data. When the mechanism that generated that data changed, the correlation broke. I had just finished prepping critical realism for my comps and the connection was too clean to ignore.

Bhaskar (1975) splits reality into three domains. The real domain contains mechanisms, structures, and powers that exist whether or not they are activated. The actual domain contains events that happen when those mechanisms are triggered. The empirical domain contains the subset of those events that we observe and measure. Wynn and Williams (2012) spell out what this means for research: observed regularities may not reveal underlying causal mechanisms because mechanisms can be activated, suppressed, or counteracted by contextual conditions in open systems. I had underlined that sentence in my study notes and treated it as a general philosophical caution. Then I realized it is a direct description of what happens when an AI model fails in production. The model learned an empirical regularity. The generative mechanism that produced that regularity changed. The model had no access to the mechanism, only to the surface pattern the mechanism left behind. When the pattern shifted, the model kept predicting from the old pattern.

Mingers et al. (2013) argue that critical realism provides ontological depth that positivism cannot deliver, and I think that same ontological depth is the missing language for the ML deployment problem. An AI model is by construction an empirical domain device. It processes patterns in observed data. It does not and cannot access the real domain, the actual structures and mechanisms and powers that generate the events it was trained on. A large language model that predicts the next token with impressive accuracy has not identified the social, economic, or physical mechanisms that produced the text in its training corpus. It has learned to reproduce the empirical trace of those mechanisms under the conditions that existed at training time. When those conditions change, the model does not know they changed, because it never had a representation of the mechanism itself.

This is the 2008 financial crisis in miniature. Banks had built credit risk models on data from 2002 to 2007. During those years, loose credit, securitization incentives, and rising home prices formed a mechanism that produced a stable pattern of low default rates. The models captured that empirical regularity with high precision. When the mechanism collapsed, when subprime mortgages started failing and the securitization chain froze, the empirical regularity the models had learned disappeared. The models were not wrong when they were built. They were correct descriptions of the empirical domain under the old mechanism. They became wrong when the real domain changed, because they had never modeled the mechanism, only the pattern it generated. The same pattern repeated during COVID. Supply chain forecasting models trained on years of stable global logistics broke overnight because the mechanism that produced predictable lead times and consistent shipping routes had been disrupted by a pandemic that did not appear in any training set. A model trained on the empirical domain under Mechanism A cannot generalize to Mechanism B, because it has no representation of mechanisms at all.

Domain shift and concept drift are the ML field's terms for this gap, and they describe the symptom without naming the cause. The cause is ontological. Models degrade when the generative mechanism shifts because they were never designed to model mechanisms. They were designed to model empirical regularities. When a fraud detection model trained before a new fraud technique emerges misses every attack using the new method, the problem is not that the model has insufficient data. The problem is that the fraud mechanism changed and the model only learned the old mechanism's empirical signature. Retraining on new data from the shifted distribution does not solve the underlying issue either. It just re-fits the empirical model to the new surface pattern produced by the new mechanism. The model is still operating in the empirical domain. It has simply updated its description of the empirical trace. The ontological limitation persists.

I think this is the most useful thing critical realism gives the IS field in the context of AI, and it is not a method or a framework. It is a way to talk about why purely empirical AI has a hard ceiling that no amount of data or compute will break through. The ceiling is ontological, not technical. An empirical model describes the empirical domain. The real domain contains mechanisms that are not directly observable. You cannot train a neural network on a mechanism because mechanisms are not events. They are the generative powers and structures that produce events. You cannot put a mechanism in a feature vector. You can only reason about it retroductively, as Wynn and Williams (2012) specify, by observing its effects and constructing explanations of how it might work. That kind of reasoning is what humans do when they build theory. It is not what gradient descent does.

The practical consequence for organizations deploying AI is that periodic retraining is not a sufficient adaptation strategy for changing environments. If the generative mechanism has shifted, retraining on the new empirical pattern is just updating the description of the surface. What the organization actually needs is infrastructure for detecting when the mechanism has shifted, mechanisms for flagging uncertainty rather than producing confident predictions based on obsolete patterns, and human judgment loops that can recognize when the empirical pattern no longer matches the real mechanism. These are not technical features. They are architectural requirements that the ontology of the domain imposes on any system that operates in open, changing conditions. I covered the general critical realism argument for IS research in an earlier post about how the stratified ontology explains why the same technology produces different outcomes in different contexts. The AI version of that argument is more urgent. The model does not know why the world works the way it does. It knows what the world looked like during training. When the world changes, the model cannot understand the change because it never had access to the mechanisms that drive the change in the first place.

Every ML practitioner with enough production experience knows something that Bhaskar formalized in 1975. Correlation trained on past data does not predict future events when the causal structure changes. The practitioner calls it domain shift or concept drift and treats it as a data engineering problem. Critical realism gives the same observation a philosophical vocabulary and an explanation for why it is not a solvable technical problem. The empirical and the real are different layers of reality. No model trained on one layer gives you access to the other. The limit is not a research question that a clever transformer architecture will solve. It is a fact about the relationship between observation and causation.

I keep thinking about how many AI pilot projects fail and get attributed to bad data, wrong model choice, or insufficient compute. All of those are real failure modes. But underneath them is a deeper pattern. The organization deployed an empirical domain device into a world where the generative mechanisms are constantly shifting, and expected it to work permanently. The model will work until the mechanism changes. The mechanism always changes eventually. Critical realism did not predict the transformer or the attention mechanism. It predicted the failure mode. That is worth sitting with.