Why Your IS System Behaves in Ways Nobody Designed

In 2010, a single trading algorithm on the New York Stock Exchange executed a series of orders that, in combination with similar algorithms at other firms, produced a self-reinforcing feedback loop. In about thirty-six minutes, the Dow Jones dropped nearly a thousand points. It recovered within hours. No individual actor intended this. No single bug caused it. The behavior emerged from the interaction of multiple systems running at millisecond speed in ways that their designers had not anticipated and, in some respects, could not have anticipated. The systems worked exactly as designed. The outcome was not designed at all.

That is what emergence means in the context of complex adaptive systems, and it is what Benbya and colleagues (2020) argue applies to digital systems generally. My study materials confirm this paper in the local library: Benbya et al. (2020) characterize digital systems as exhibiting emergence, coevolution, and chaos. Emergence means that system-level behavior cannot be predicted from knowledge of the individual components. Coevolution means that technology, users, organizations, and the environment adapt to each other continuously. Chaos means sensitivity to initial conditions: small differences in how a system is deployed or how users first interact with it can produce large divergences in outcomes over time.

The traditional engineering model of information systems treats a system as a designed artifact whose behavior can be fully specified in advance. Build the requirements document, write the code, test against the requirements, deploy. If the system behaves unexpectedly, that is a bug, and bugs can be fixed. This model works well for simple systems with limited interactions and stable environments. It works increasingly poorly as systems grow in scale, as the number of interacting components increases, and as user populations become large and heterogeneous. At sufficient scale, the gap between designed behavior and actual behavior is not just a bug. It is a structural feature of the system's complexity.

IS researchers have had the conceptual tools to describe this for a while. Markus and Robey (1988) established that the emergent view of IS impacts, the view that outcomes arise from the interaction of technology, people, and context rather than being determined by any single force, is almost always the correct framing. This is in my local study-hub materials. The emergent causal stance means that you cannot predict IS outcomes from technology features alone, or from user intentions alone, or from organizational context alone. You need all three, interacting, and the interaction produces something that none of the inputs can explain independently.

Benbya et al. (2020) push this further by connecting it explicitly to complexity science, the tradition that includes work on complex adaptive systems from researchers like Stacey and Kauffman in management and organizational theory. I want to be careful here: my study-hub materials reference Benbya et al. but do not include local evidence for Stacey or Kauffman directly, so I am hedging on those specific names. What I can say with confidence from my local sources is that the study-hub frames digital systems as exhibiting emergence, coevolution, and chaos, and that these properties make linear IS models inadequate for explaining the behavior of large-scale sociotechnical systems.

The practical implication that follows from this, and it is one I keep coming back to, is about the design goal. If emergence is a structural property of large-scale IS, then designing for full predictability is the wrong target. The system will behave in ways its designers did not anticipate. The question is not how to prevent unexpected behavior but how to detect it quickly and respond. This shifts design priorities toward monitoring, rapid detection, and recovery capacity rather than exhaustive prior specification. It is the difference between designing a system that never fails and designing a system that fails gracefully and recovers well. The first goal is unreachable at sufficient scale. The second is achievable.

Resilience engineering makes exactly this argument in the context of critical infrastructure, and I think it transfers directly to enterprise IS. A system with strong monitoring and rapid incident response is safer than a system with exhaustive prior specification and no monitoring, because the exhaustive specification will be wrong in ways nobody predicted, while the monitoring will catch the unexpected behavior when it manifests. The CrowdStrike incident of 2024, where a single software update caused millions of Windows systems to enter a crash loop simultaneously, is a recent example. The update was tested against the requirements. The emergent behavior at scale was not something the tests captured.

The design implication connects to what I wrote about coevolution between technology and organizations. If technology and organizations coevolve, then a system's behavior at time two is shaped by how users adapted to it at time one, which shaped how they use it at time two, which shapes what the system is actually doing at time three. A recommendation algorithm deployed to a million users is not the same system six months later, because the interactions have changed the user population's behavior, and that changed behavior feeds back into the algorithm's learning. The designed artifact and the social system are not separate objects. They are one complex system that evolves together.

Gartner's work on technology risk management has increasingly moved away from purely linear risk models. The newsroom at https://www.gartner.com/en/newsroom covers enterprise technology risk and resilience topics that reflect this shift, particularly in the context of AI systems and critical IT infrastructure. The framing has changed from "prevent all incidents" toward "build the capacity to detect and respond," which is the practical translation of the complexity argument even when it is not named that way.

What I think IS researchers can contribute here is making the mechanism explicit. Emergence is not just a metaphor for "systems are complicated." It is a specific claim about the relationship between component behavior and system behavior: system behavior cannot be derived from component behavior. That claim has testable implications. It means variance models that treat system-level outcomes as linear functions of component-level predictors will systematically underfit. It means process models that trace how behavior changes over time through feedback will outperform cross-sectional snapshots. It means research that studies systems at one point in time will often fail to capture the dynamics that produce the outcomes it is trying to explain. That is not a small limitation. It is a structural challenge for most of the IS research canon.