The gap nobody budgets for
A prototype that answers correctly on a laptop is a good afternoon's work with modern tooling. A system that answers correctly on real data, knows when it should not answer, costs a predictable amount per call, and tells you when it starts drifting — that is a quarter of engineering.
The gap between them is not glamorous and it is not optional, and it is where the large majority of stalled AI programmes are sitting right now.
What the missing 60% actually is
In our experience rescuing these projects, the same list comes up almost every time:
- No measured baseline
- Nobody recorded what the process achieved before the model. Without that number, no accuracy figure means anything — 94% is a triumph against a 70% manual process and a regression against a 96% rule.
- No real evaluation set
- A few hundred examples annotated by people who know the domain, covering the awkward cases. It is the highest-value artefact in the project and the first thing cut when a deadline tightens.
- Training on information that would not exist
- Features computed with data unavailable at prediction time. The model scores brilliantly in testing and collapses in production, usually a month after anyone stopped watching.
- Random splits on time-series data
- Train on the past, test on the future. A random split flatters almost any model built on business data, which nearly always has time structure.
- No defined failure behaviour
- What happens when the provider is slow, the model is confidently wrong, or the input is unlike anything in training? If this is not written down and tested, the answer is decided at 2am by whoever is on call.
- No cost ceiling
- Per-request cost scales with success. Systems without an enforced budget become expensive at exactly the moment they become popular.
- No monitoring on inputs
- Model decay is normal. Discovering it from a customer complaint is not.
Why it gets cut
Not through incompetence. It gets cut because the prototype is visible and the hardening is not. A demo can be shown to a board; a regression gate in CI cannot. When a deadline compresses, teams cut what nobody in the room can see the absence of — and the absence only becomes visible months later, as an incident.
The commercial version of the same mistake is a proposal that prices the prototype and treats production as a phase two that never gets funded.
What to do instead
Establish the baseline before anything else, and treat the evaluation set as a deliverable rather than a task. Build one thin slice end to end on production-shaped data before building anything broad — it is deliberately unimpressive and it surfaces the architectural mistakes while they are still cheap to fix.
Then budget the hardening explicitly, as its own phase with its own acceptance criteria. If a supplier's proposal does not contain that phase, the right question is not 'can you do it faster' — it is 'what happens when this is wrong?'
More
Also worth reading.
Other arguments we make often enough to have written down.
Twelve questions to ask an AI vendor
Questions that separate a supplier who has shipped production systems from one who has shipped demos. Use them on us too.
The DPDP Act: what AI projects in India need to get right
India's Digital Personal Data Protection Act is a design constraint on AI systems, not a document you write afterwards. What it changes in practice.
Next step
Have a version of this problem?
Tell us what is slow, expensive or error-prone. A diagnostic will tell you honestly whether it is worth solving with machine learning — including when it is not.