Twelve questions to ask an AI vendor

Why these questions

Evaluating an AI supplier is hard because the demo is not the product and everyone's demo works. These questions are chosen because a firm that has actually run systems in production answers them immediately, and a firm that has not tends to answer a different question.

They apply to us as much as to anyone else. If our answers are unsatisfying, that is useful information.

On the problem

What would make you tell us not to build this?

A supplier with no answer has never turned work down, which tells you what their recommendation is worth.

What is the current process achieving?

If they have not asked, they cannot claim an improvement later. Watch whether they measure it or ask you to estimate it.

Which decision does this model inform, and who acts on it?

Vague answers here produce systems that are technically working and operationally ignored.

On evidence

What will you evaluate against, and who annotates it?

'We'll use a holdout set' is not an answer. Ask who creates the ground truth and how much of your team's time it needs — because it will need some.

Will you split the data by time?

For anything with time structure, a random split inflates the result. If they do not know why this matters, that is decisive.

Show me performance broken out by segment.

Aggregate accuracy routinely hides a region, product or customer type where the model is worse than useless.

What is the cost of each error type, and who decided the threshold?

The operating point is a commercial decision. If a data scientist set it alone, it was set wrongly.

On production

What happens when the model is wrong or the provider is down?

Ask for the specific behaviour: fallback, escalation, timeout, retry. 'It's very reliable' is not a failure plan.

What does one request cost, and what is the ceiling?

If nobody has modelled this, the bill scales with adoption and nobody notices until it is large.

How will we know it is degrading?

Monitoring on input drift and, where ground truth arrives later, on realised accuracy. Without it, decay is discovered by a customer.

On what you are left with

Who owns the code, the models and the evaluation sets?

The answer should be 'you do, in your repository, from the first commit'. Anything else is a dependency being sold to you as a service.

Can our team operate this after you leave?

Ask what documentation and handover looks like concretely. If the honest answer is no, that is a legitimate choice — but you should be making it knowingly.