The numbers, and what they actually say
If you run operations at a mid-sized company, you have probably lived this: an agency arrives, builds a slick proof-of-concept on an off-the-shelf model, and presents a deck showing it working on clean, curated dummy data. You pay the invoice, everyone claps, and the system is never used again.
You are not an outlier. You are the base case.
| Source | Finding |
|---|---|
| MIT, State of AI in Business 2025 | 95% of GenAI pilots deliver no measurable P&L impact |
| Gartner (2025) | 50% of GenAI pilots abandoned after PoC, revised up from a 30% prediction |
| IDC | 88% of pilots fail to reach production |
| RAND | 80% project failure rate |
These are different samples measuring different things, which is exactly why you should distrust anyone quoting a single round number. We used to run a "90%" figure in the headline of this very post. It came from nowhere. We removed it, because a company selling measurement accuracy has no business inventing statistics.
What survives the disagreement is the direction: most AI pilots do not become systems. And across MIT, IDC and Gartner the recurring cause is the same: data readiness, weak integration into real workflows, and unclear definitions of success. Not model quality.
Why does a pilot that demoed perfectly die in production?
Because demos run on curated data and production does not. In production, documents are scanned sideways, end-users mistype their email addresses, APIs rate-limit you, and the light over the inspection station changes between the day and night shift. The edge cases are not a small remainder of the work. Handling them is most of the work.
A demo proves a model can produce the right answer. It proves nothing about whether a system will, every time, at 2am, when the input is malformed and nobody is watching.
What does building for production actually require?
We refuse to sell pilot projects or strategy consulting, because they reward the wrong behaviour: making something that looks like it works rather than something that actually works. A production system has to account for three things a proof-of-concept never does.
Error handling and fallbacks. If the primary OCR engine fails on a smudged receipt, does the pipeline crash, or does it route to a fallback vision model and flag the record for review? Every external dependency needs a defined failure path.
Human-in-the-loop. Autonomous agents are useful, but high-stakes decisions need a deterministic pause where a person verifies output before it proceeds. Our real-estate agent deployment handles 70% of inquiries autonomously: the design question that mattered was what happens to the other 30%, not how to push the number higher.
Data residency and security. You cannot pass PII to a public API endpoint and call it an architecture. Compliant infrastructure is a build requirement, not a phase two.
What we do instead
We start by identifying the exact, measurable operational drain. If a team spends 20 hours a week extracting invoice data, that is the target, not "AI transformation."
Then we build a deterministic pipeline wrapped around an inference engine, not a generic assistant. For invoice extraction that means a system that reads your invoices and structures the data exactly as your ERP expects, with sub-second latency. For manufacturing QC it meant a YOLOv8 model trained on 10,000+ images of the actual product, running on the actual line: 99.2% classification accuracy at 120 items per minute, and a 94% reduction in inspection errors in the first month. Shipped in three weeks.
The deliverable is not a deck. It is 20 hours back in the week, or a defect that stops reaching your customer.
When is a pilot actually the right call?
There is a version of a pilot worth running, and it is narrow: one falsifiable question, answered on real data. "Can a model hit 98% on our actual defect images, in our actual lighting?" is a good pilot. It has a pass/fail answer and it de-risks the real build.
"Let's do a pilot to see if AI is worth it" is not. That question is answered by arithmetic on what your current process costs, not by a proof-of-concept. If the manual process does not cost enough to justify the build, no demo will change that, and a vendor who runs the pilot anyway is selling you the pilot, not the outcome.
Not a fit if…
We are the wrong choice in several situations, and it is cheaper for both of us to say so now.
If you genuinely do not know which process to automate, you need an operations review first, and we do not sell those. The honest move is to hire someone who does, or to spend a week measuring where hours actually go. If your data does not exist yet, no vendor can help until it does; you cannot train on images nobody has taken. If your inspection or process is standard enough that an off-the-shelf product covers it, buy the product. We say the same thing in our comparison of computer vision vendors, where the recommendation for standard inspections is a Cognex or Keyence sensor, not us.
And if nobody on your side will own the system after go-live, it will drift regardless of how well it was built. That is not a vendor problem and no contract fixes it.
The one question to ask any AI vendor
Ask them what happens when it fails.
Not whether it works. Everyone's demo works. Ask what the system does when the input is malformed, when the API is down, when the confidence score is 0.6 instead of 0.95. A vendor who has built production systems will answer immediately and in detail, because they have been paged about it. A vendor who is selling you a pilot will tell you about the model.
