From Pilot to Production: What Changes When AI Has to Operate Inside a Real Organisation

Two of the most cited statistics about enterprise AI seem to contradict each other, until you look closely at what each one actually measures. Gartner predicts at least 30% of generative AI projects will be abandoned after proof of concept by the end of 2025, citing poor data quality, inadequate risk controls, escalating costs and unclear business value. MIT's Project NANDA, in a July 2025 study, found that 95% of the roughly 300 generative AI deployments it analysed had produced no measurable profit-and-loss impact by the time of the report. One number describes projects that never got deployed. The other describes projects that did get deployed and still didn't move a number anyone was tracking. Those are different failures, and treating them as the same statistic leads to the wrong fix.

Why the numbers disagree

Gartner's figure counts projects abandoned before they ever reach production. MIT NANDA's figure counts something different: deployments the researchers reviewed that had reached production or something close to it, and still hadn't produced a measurable financial return within the study's window. A project can clear the first bar and still fail the second. It can reach production, run for months, and still not move any number a finance team cares about.

This distinction matters more than the headline percentages. An organisation that reads "95% show no return" as "95% never launch" will draw the wrong lesson and focus effort on getting more pilots across a finish line that was never the real problem. The evidence points somewhere else: getting to production is necessary but not sufficient. What happens to the system once it's there is where the value actually gets lost or found.

What the people building these systems actually say

RAND Corporation's 2024 study on AI project failure took a different approach from a market survey. Researchers interviewed 65 experienced data scientists and engineers about their own projects, rather than asking executives to estimate failure rates. The most frequently cited root cause, mentioned by 84% of industry interviewees, was not a technical one. It was a breakdown between what business leadership actually needed and what the technical team built.

RAND's own example makes this concrete. A business leader might ask for a model that predicts the right price to set for a product. What they actually need is the price that maximises profit margin, rather than the price that moves the most units. Those are different optimisation targets, and a data science team without that context will build a model that answers the wrong question competently. The pilot can work exactly as specified. It just wasn't specified correctly, because the specification depended on business context that never made it from the leadership team to the people building the system.

Data quality was RAND's second most commonly cited cause. The remaining three, in descending order of frequency, were data scientists chasing the latest technology instead of the problem at hand, underinvestment in the infrastructure needed to deploy and maintain models, and cases where the underlying problem turned out to be beyond what AI could reliably solve. That last cause is a genuine technical limit rather than an organisational one, and deserves to be named accurately rather than folded into the others. But it was the least frequently cited of the five, and the other four all describe breakdowns in how the organisation defined the problem, prepared its data, chose its tools and built its infrastructure, rather than limits in what the underlying models could actually do.

Why a pilot can succeed and production can still expose the difference

A pilot typically runs on curated data, in a sandbox, evaluated against a narrow success metric that was easy to define up front. Production runs on the organisation's actual data, inside its actual workflows, judged against outcomes that matter to people who weren't in the room when the pilot was scoped.

MIT NANDA's report describes a related pattern from a different angle. Its authors attribute the 95% figure primarily to a shortfall in the tools' ability to learn: most of the generative AI tools reviewed lacked persistent memory and the ability to adapt to a specific workflow over time, so a system that performed well in a demo kept behaving the same generic way once it was live, because nothing about the tool or its deployment forced it to absorb the organisation's own terminology, decision logic or feedback. That's partly a real limitation in current generative AI tools and partly a deployment and integration problem, and the report treats the two as connected rather than separate. The study is described by its own authors as preliminary, has not been peer reviewed, and has drawn criticism over its interview sample size and its definition of success as measurable profit-and-loss impact within a matter of months, so its precise percentage should be read as one data point rather than a settled figure. What it shares with RAND's much smaller, differently structured study is a similar shape: organisational and integration factors, alongside real technical limitations, account for more of the difference between deployments that produced value and ones that didn't than infrastructure or budget alone would suggest.

This is not the only explanation for why production is harder than piloting. Cost overruns, data pipeline fragility and unclear ownership after launch are all real, well-documented contributors in their own right. Both studies point to a mix of organisational, integration and, in RAND's case, genuine technical factors rather than a single cause. Reading the two together, our own view is that organisational knowledge which never made it into the system is a common thread running through several of those factors, even though neither study frames its findings that way itself.

Agentic AI raises the stakes on the same problem

Gartner's more recent prediction sharpens this further. In June 2025, the firm forecast that more than 40% of agentic AI projects will be cancelled by the end of 2027, driven by escalating costs, unclear business value and organisations underestimating what it takes to run an autonomous agent at scale rather than in a proof of concept. Gartner's own framing is direct: many current agentic AI projects are early-stage experiments driven by hype, and "agent washing", rebranding existing automation as agentic AI, is inflating the apparent pipeline of genuine deployments.

An agent that acts on its own, rather than producing a recommendation for a person to review, has less room for the kind of context mismatch RAND and MIT NANDA both describe. A chatbot that misunderstands business context produces an unhelpful answer. An agent that misunderstands business context and is empowered to act on it produces a wrong outcome, at whatever speed the agent operates.

What this means for the next AI initiative

None of this argues against piloting AI, and none of it suggests infrastructure and budget discipline don't matter. They clearly do, and the research is consistent on that point. What it argues for is treating the transition from pilot to production as a distinct project in its own right, with its own success criteria, rather than as a formality that follows automatically once a proof of concept looks promising.

The specific, practical question worth asking before scaling a pilot is not just whether the infrastructure and budget are ready. It's whether the people who understand why the organisation makes the decisions it makes were involved early enough to shape what the system was actually built to do, and whether that understanding is something the system carries with it once it's out of the sandbox and into daily use.

AinTools works with organisations moving AI from early adoption into trusted, everyday use. Our Platform page sets out how we approach that journey, from readiness through to production.

Sources

• Gartner, official press release, "Gartner Predicts 30% of Generative AI Projects Will Be Abandoned After Proof of Concept By End of 2025" (30 July 2024): gartner.com
• Gartner, official press release, "Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027" (25 June 2025): gartner.com
• RAND Corporation, Ryseff, De Bruhl and Newberry, "The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed" (RR-A2680-1, August 2024): rand.org
• MIT Project NANDA, "The GenAI Divide: State of AI in Business 2025" (July 2025)