Why Projects Go Over Budget Before They Start: The Estimation Problem That AI Is Finally Solving

That sinking feeling is familiar. The project is six weeks old and already burning through contingency. The original estimate, signed off with confidence, now looks like a work of fiction. The truth is, that budget was wrong the day it was approved. It was based on a price for steel that was three months out of date, labour rates that ignored the latest union agreement, and an optimistic timeline that assumed every single dependency would land perfectly. It was, in other words, doomed from the start.

The Anatomy of a Flawed Estimate

This isn’t a case of bad luck or poor management. It’s a cognitive bias, first identified by psychologists Daniel Kahneman and Amos Tversky in 1979, called the “planning fallacy.” It is the demonstrable, repeatable tendency for people and organisations to underestimate the time, cost, and risk of future actions, even when they have direct experience of similar tasks running over. The average large IT project, for instance, runs 45% over budget while delivering 56% less value than expected, according to one major McKinsey study. For megaprojects, the numbers are even worse.

This bias isn't born from incompetence. It's a product of how we think. When planning, we naturally adopt an “inside view,” focusing on the specifics of the project at hand and visualising a best-case sequence of events. We build a plan based on what we can control and what we want to happen. What this view excludes is the “outside view”—the statistical reality of how similar projects have actually performed. We ignore the vast dataset of past overruns because, of course, *this time is different*.

In a business setting, this psychological quirk is amplified by organisational pressure. A department head needs a project to fit a pre-approved capital budget. A salesperson promises a delivery date to close a deal. The estimate is massaged, not out of malice, but to make the numbers work. Each assumption is individually defensible, but collectively they create a fragile plan with no resilience.

The Compounding Cost of Disconnected Data

The planning fallacy thrives on stale, fragmented information. A project estimate to build a new production line isn’t a single number; it's the sum of hundreds of assumptions drawn from dozens of different systems, most of them spreadsheets. The cost of raw materials comes from purchasing. Labour costs come from HR. Machine capacity comes from operations. Overhead allocations come from finance.

Each of these data points is a potential point of failure. They exist in separate silos, updated on different cadences, with no shared context. By the time someone manually collates them into a master project budget, the information is already obsolete. A price change for a key component won’t be reflected. A shift in logistics costs won't be included. Small errors, born from data latency, multiply. A 2% error in a bill of materials cost here, a 5% miscalculation of labour hours there. It all compounds, hidden within the false precision of a final budget number.

This is the core operational weakness that traditional project management tools cannot solve. They are brilliant at visualising a plan. They are terrible at validating the real-world assumptions that underpin it. They manage the project, but not the data ecosystem the project depends on.

One in six IT projects becomes a “black swan” with a cost overrun of 200% on average and a schedule overrun of almost 70%. The statistical distribution of project risk is fat-tailed, meaning the mean or “average” overrun significantly understates the true financial exposure.

Flyvbjerg & Budzier, Harvard Business Review

The ‘What If’ We Can’t Answer

Because the data is static and the assembly process is manual, the resulting project plan is brittle. It represents a single, idealised path to completion. It has no capacity to answer the critical “what if” questions that define real-world operations.

What if the price of a key raw material increases by 15%? What if a key supplier pushes their lead time out by four weeks? What if a new tariff adds 10% to a critical sub-assembly? Answering these questions in a spreadsheet-driven environment is a monumental task. It requires re-running the entire manual data-gathering exercise. As a result, it rarely happens. Projects are launched with a single-point estimate and a vague contingency percentage, but with no genuine understanding of the risks attached. The team is flying blind.

How AI Changes the Estimation Game

This is the estimation problem that AI is finally solving. Not by creating a magical black box, but by doing what disconnected systems never could: creating a live, unified model of the business. When an AI-powered platform operates on a single database, the concept of stale data vanishes. The estimate is no longer a static snapshot; it’s a dynamic calculation based on real-time information.

Imagine a project plan where the Bill of Materials is directly linked to the live supplier price book in your purchasing software. When a steel supplier updates their price, every open quote and every planned project that uses that steel is instantly re-costed. The Response365 Project Management module’s profitability watches can then flag any quote whose margin has fallen below a set threshold *before* it gets sent to the customer.

This is the first step: connecting the data. The next step is learning from it. AI models can analyse historical performance across hundreds of past projects, identifying the hidden patterns that precede an overrun. It can learn that projects involving a certain type of custom fabrication typically require 15% more engineering hours than planned. It can see that projects for a specific customer consistently experience scope creep at the midway point. These insights are then used to build more realistic base estimates, moving from a flawed “inside view” to a data-driven “outside view.”

Finally, AI can run thousands of simulations in seconds. It can model the impact of commodity price swings, labour shortages, and logistics delays to generate a probabilistic forecast, not a single number. The output is no longer “This project will cost £5.2 million.” It is “There is an 80% probability this project will cost between £5.1 million and £5.8 million, with a 5% chance of exceeding £6.5 million if steel prices and shipping delays both hit.” This allows for a completely different quality of conversation about risk and contingency.

From Reactive Reporting to Proactive Control

A better estimate is the start, not the end. The real operational advantage comes from carrying this live, data-driven approach through the entire project lifecycle. When your project management system is part of the same platform as your finance and operations, control becomes proactive, not reactive.

Instead of waiting for a month-end report to discover a budget variance, you see it the moment it happens. Earned Value metrics like Cost Performance Index (CPI) and Schedule Performance Index (SPI) aren’t historical artefacts; they are live health indicators for every project. A CFO can see a live, drillable P&L for every single project, updated with every transaction. A procurement manager is alerted when a purchase order commitment will breach a project-specific budget *before* it can be approved. This isn’t just better reporting. It is a system of automated guardrails that prevents the small deviations that snowball into catastrophic overruns.

The End of the Guess

For decades, project estimation has been more art than science, a blend of experience, intuition, and political negotiation. The result, as the data shows, has been the consistent destruction of value. The shift to AI-driven estimation isn't about replacing human judgment. It's about augmenting it with a real-time, statistically sound model of reality. It's about making decisions based on probability, not personality. The goal is no longer to create a perfect plan, but to build a resilient one that can absorb the shocks of a world that refuses to follow a Gantt chart.

---