The demo is a sales artifact, not evidence
A vendor demo runs on inputs the vendor picked. That is not necessarily dishonest. Everyone shows their work at its best. But the cases most likely to get picked are the clean ones: the well-formed invoice, the customer email that states the problem in a single sentence, the record with every field populated. Your actual queue is forwarded threads with four signature blocks, scans rotated ninety degrees, people asking two unrelated questions in one message, and the request that is really about something the customer has not said yet. A demo tells you the feature can work. It tells you almost nothing about how often it will.
The opposite failure is just as common and costs more. The buyer senses the demo proved little, cannot articulate what would prove more, and asks for another demo. Then another. The vendor experiences this as moving goalposts and starts padding estimates. Nobody is acting in bad faith and the project still dies. The missing piece in both versions is the same: a test set that belongs to the buyer, exists in writing, and was agreed on before the build started.
Once that set exists, the negotiation changes character. You are no longer arguing about whether the feature feels accurate, which is unwinnable, because the vendor has seen it work and you have seen it fail and you are both right. You are arguing about a number that either side can reproduce. That is a solvable argument.
Build the evaluation set out of your own ugly cases
Pull real historical cases from your own records, from a stretch of time that reflects normal operating volume. Somewhere between 100 and 300 is usually enough to tell a system that is right 70 percent of the time from one that is right 90 percent of the time. It is not enough to distinguish 92 from 94, and part of this exercise is deciding whether you actually need that distinction or just think you do. If your work is seasonal, sample across the year rather than taking the most recent two hundred items in order.
Then deliberately load in the hard tail. The ambiguous request. The one with a missing field. The customer who is angry about the wrong thing. The exception that gets handled today by someone walking over to a colleague's desk. Keep the natural mix roughly intact, but confirm the difficult cases are actually represented: if eight percent of real volume is genuinely hard and your test set contains none of it, your measured accuracy is a fiction you paid to produce.
Have someone who does this work today write down the correct answer for each case. This is the part buyers skip and it is the part that makes everything else possible. Where two experienced people disagree about the right answer, mark it and pull it out of scoring. Those cases are not a model problem, they are a process problem you now know about. Finally, hold back a portion of the set, perhaps a third, that the vendor never sees. If the system performs well on the visible cases and noticeably worse on the held-back ones, it was tuned to the test.
Decide what good enough means before you see the number
There is no industry-standard accuracy threshold for AI features. Anyone who quotes you one is quoting nothing. The right number falls out of what a wrong answer costs you, and that varies enormously between two features that look similar on a slide. A tool that drafts a reply for a human to review and send: a wrong answer costs thirty seconds of editing. A tool that classifies an invoice and posts it to the ledger with no one looking: a wrong answer costs an accounting correction, and occasionally a payment to the wrong party. Same accuracy figure, completely different decision.
Write the threshold down and have both sides agree to it before the test runs. This sounds procedural and it is the single most useful thing in this guide. Once the number is known, everyone's sense of what is acceptable drifts toward it. The vendor will explain why 84 percent is strong for this class of problem. You will wonder whether you were being unrealistic. Neither of you is lying. Human judgment about thresholds is simply not stable in the presence of a result, so fix the threshold while it is still cheap to be honest.
Accuracy rate alone is also the wrong unit. Separate the error types and price them. A feature that misroutes a routine ticket and a feature that tells a customer something false about their contract are both counted as one error, and they are not the same event. Sort your failures by what they cost rather than how often they happen, and pay particular attention to the tail. If one case in four hundred produces an answer that is confidently wrong and expensive to unwind, that fact should carry more weight in your decision than three percentage points of headline accuracy.
Test the edges, not just the middle
Most AI features fail in one of two directions, and a test built from typical cases catches neither. The system answers a question it should have handed to a person, or it hedges and escalates so often that it creates more work than it removes. Both need explicit test cases. Feed it inputs that sit outside the defined scope and confirm it declines instead of improvising. Feed it inputs where the correct behavior is to say it does not know and route to a human, and score correct escalation as a pass. If the vendor's scoring counts every escalation as a miss, the incentive is pointed the wrong way and you will get a confident system rather than a useful one.
Then test hostile input. If the feature reads text that customers or the public supply, whether that is email, uploaded documents, form fields, or support tickets, someone will eventually write instructions into it. Submit cases containing text that tells the system to disregard its instructions, reveal its configuration, or approve something. You are not looking for a perfect defense, because there is not one. You are measuring blast radius: what can this feature actually do if something steers it. If the honest answer includes issuing a credit, sending an external email unreviewed, or writing to a system of record, that is an architecture conversation, not a tuning conversation, and it is much cheaper to have before go-live.
Finally, measure stability. Run twenty or thirty of the same cases again on a different day and compare. Some variation is expected and fine. If identical input produces materially different decisions, the feature is not ready to carry decisions on its own no matter what its average accuracy says, and you should either add a review step or narrow its scope. Put the stability figure in the acceptance record alongside the accuracy figure.
Nothing underneath the system stays still after go-live
Most builds sit on a model the vendor licenses from someone else. That model gets updated, deprecated, and eventually retired, on a schedule neither you nor your vendor controls. The prompts and settings that produced your acceptance result were tuned against the old one. Your own inputs drift too: a new product line, a redesigned form, a pricing change, a support policy that shifted in June. A feature that measured 91 percent in March can be at 78 percent by September with nobody having done anything wrong.
This is what turns the evaluation set from a one-time gate into an asset. Keep it, keep the labels with it, and re-run it on a schedule and after any change on the vendor's side. The re-run costs a fraction of what the original labeling cost, because the expensive part was deciding the right answers, and you only do that once. It also gives you something most buyers never have: an early, specific signal that quality moved, instead of a vague sense months later that the team has stopped trusting the tool.
Put the mechanics in the agreement rather than in a friendly email. Attach the evaluation set and the threshold as an exhibit. Require written notice before the vendor changes the underlying model, provider, system prompt, or the data the system retrieves from, and require the acceptance test to be re-run afterward with results sent to you. Name a measurement cadence, an owner, and who pays for it. Define a remediation window and what happens if measured quality stays under the bar. And confirm in writing that you own the evaluation set, the labels, the prompts, and any tuning data, so that changing vendors costs you a migration rather than a restart.
An AI feature cannot be accepted on a demo or a feeling. Build the test set from your own real cases and agree the threshold before the build starts, judge failures by what they cost rather than how often they happen, and keep that same test as a regression suite that runs every time something changes underneath you.