Why IT Projects Are Always Late and Over Budget

The Standish Group started publishing its CHAOS research in the early 1990s. The core finding, repeated year after year, is that a large share of IT projects finish late, over budget, or with fewer features than planned. A meaningful portion get cancelled entirely. For decades, every time a new project management framework arrived, every time a certification program expanded, every time agile replaced waterfall as the fashionable answer, people assumed the numbers would finally get better. By most accounts, they have not moved dramatically. I find that either depressing or clarifying, depending on the day.

The clarifying interpretation is that this is a structural problem. It is not that the people running IT projects are incompetent, or that the tools are bad, or that the right methodology has not been discovered yet. The problem is in the nature of software itself, and the problem is compounded by how organizations make decisions about it.

Software projects are different from physical construction in one important way. When you build a bridge, the material constraints are visible. Steel has known properties. Ground conditions can be tested. You are working with physics that does not change halfway through the project. Software has no such constraint. Requirements are invisible. They live in people's heads, and what people say they want often differs from what they mean, which often differs from what they actually need. By the time you build what they described, they can see it for the first time and realize the description was wrong. This is not a failure of communication. It is a feature of complex problem-solving. You often do not fully understand a problem until you have partially solved it.

Daniel Kahneman's work on the planning fallacy, built on ideas he developed with Amos Tversky, describes a pattern that applies universally: people systematically underestimate how long tasks will take and how much they will cost, even when they have direct experience with similar tasks going wrong before. Software projects are a particularly severe case because the unknowns are genuinely unknown at the start. You cannot estimate accurately what you do not understand yet, and in software development, the hard parts are almost always the parts you did not anticipate.

The standard response to this is to spend more time on requirements. Write better specifications. Build more detailed project plans. This helps at the margins but does not solve the core problem, because the requirements change. Not because stakeholders are fickle, though some are. Because the business changes. Because the competitive environment shifts. Because the technology options that were not available when the project started become available during it. A two-year project designed in 2022 is being built against a world that already looks different in 2024. The plan is a document. The world is not.

The IS literature has wrestled with this for a long time. Researchers have documented the pattern with names like escalation of commitment, where teams and executives continue investing in troubled projects long after the evidence suggests they should stop, because stopping feels like admitting failure. The Standish CHAOS research has consistently noted that large projects fail at much higher rates than small ones, which is not surprising once you think about it: larger projects carry more complexity, more stakeholders, more hidden dependencies, and longer timelines during which the world can shift.

What the literature talks about less is the political economy of project estimates. IT projects require commitments to be made before the problem is fully understood. Leadership needs a number to put in a budget. The board needs a timeline. Procurement needs a contract. All of this has to happen before anyone has written a line of code or tested any assumptions with real users. And the person who provides the estimate faces an asymmetric incentive structure. If you say "we need two years and fifteen million dollars," the project might get cancelled or scaled back. If you say "twelve months and eight million," the project gets approved. The difference between those two scenarios, from the perspective of the person making the estimate, is one of social risk, not technical accuracy.

So estimates compress. Not through dishonesty, usually, but through optimism and the very human desire to tell people what they want to hear. The compressed estimate goes into the plan. The plan becomes the baseline. Every deviation from the plan is a failure, even if the deviation reflects new information that was not available when the plan was written. This is where I think the industry conflates the symptom with the disease. We call a project "late" when it takes longer than the estimate. But if the estimate was unrealistic from the start, the project was never on time in any meaningful sense. It was just temporarily consistent with a number someone wrote down before the work began.

What the CHAOS research reveals, if you read it carefully, is that the definition of success in IT projects has always been the wrong one. A project that delivers on time and on budget but builds the wrong thing is not a success. A project that takes longer than planned but delivers something the organization actually uses and values is not straightforwardly a failure. The well-documented large-scale national government IT programs that ran billions over budget and years over schedule are treated as cautionary tales, and they should be. But they also reveal something about the impossibility of applying rigid planning to problems that are poorly understood at the start. The failure mode is not laziness or incompetence. It is the mismatch between the certainty that planning and contracting require and the uncertainty that complex software development involves.

The reason we have not solved this after thirty years of trying is that the incentives have not changed. Organizations still require upfront commitments before they release funds. Vendors still win contracts with optimistic proposals. Executives still set deadlines based on business need rather than technical reality. No methodology changes these facts. Agile helps teams respond to change faster, which is real and valuable, but agile does not give a CIO a credible answer to a CFO who is asking what the total cost will be and when it will be done. The structural tension between the uncertainty of the work and the certainty that governance requires is what produces late, over-budget IT projects. Until the governance changes, the pattern repeats.