Testing an AI System Before the Demo: A 5-Step Method

What an AI evaluation actually is, and why a demo cannot replace it

A good demo proves exactly one thing: the system worked on the example you picked, under the conditions you picked, on the day you picked. It proves nothing about how the system behaves across the dozens of variations real users will throw at it. That gap is precisely what an evaluation is built to close.

An AI evaluation, or eval, is an automated or semi-automated test that measures a system’s performance across a set of representative cases before a real user ever sees the output. Unlike a demo, which shows a single favorable scenario, an eval measures a success rate across dozens or hundreds of cases, including the edge cases nobody would pick to impress a client.

Among teams shipping AI features in weeks rather than quarters, the absence of structured evaluation is the single most common cause of the gap between "it worked in the demo" and "it broke for the first real user." That gap is usually discovered at the worst possible time, once the client is already committed.

The trap of succeeding on one example

A generative system can produce an excellent answer to one question and a false, incoherent, or off-topic answer to an almost identical one. That variance is not a bug you fix once, it is a property of language models that you measure continuously. Picking a single example for a demonstration is effectively drawing one card from a deck whose distribution you have not checked.

Take a common case: an assistant answering customer questions about invoices. In a demo, the question is always clear, well phrased, with a valid invoice number. In production, a share of questions arrive misspelled, without an invoice number, in a different language, or mixing two separate requests in the same sentence. An eval that only covers the clean case will never catch that the system produces nonsense as soon as the input strays slightly from the script.

Eval-driven development flips the usual order: you write a measurable, testable definition of what a correct answer looks like for a given category of input, including explicit edge-case handling, before you even touch the model's instructions. The demo then becomes a consequence of a system that passes its tests, not a goal in itself.

Three families of methods, for three different jobs

There is no single correct way to evaluate an AI system. Three families of methods coexist, and most serious projects combine at least two of them.

In practice, the most effective setup lets the automated judge handle most of the volume, and reserves human review for disagreements, edge cases, and periodic drift checks on the judge itself. A judge that is never recalibrated ends up scoring lower-quality answers generously, a phenomenon usually called judge drift.

Building an evaluation set before the first demo

You do not need a full evaluation platform to get started. Five steps are enough to turn a gut feeling ("it seems to work") into an actual measurement:

1. Collect 50 real cases, ideally past failures or genuine requests from the target use, rather than examples invented for the occasion.
2. Grade each case as pass or fail using someone who understands the domain, with a short written critique explaining why.
3. Calibrate an automated judge against those human verdicts, adjusting the judge's prompt until it agrees with the expert on the large majority of cases.
4. Set a blocking threshold on the metric tied to your most expensive failure mode, not on a general average that hides serious cases.
5. Rerun the evaluation on every change to the prompt, the model, or the data source, before showing anything to a client.

OpenAI documents a transcript-summarization example evaluated on a held-out set of 1,000 reference transcripts, which gives a sense of scale for a system already in production. Fifty cases is a reasonable starting point, not a finish line: the set grows with every new failure mode discovered once the system meets real usage.

What to measure depending on the type of system

An agent that triggers actions

For an agent, the question is not only "is the final answer correct" but also "was the right tool called, with the right arguments, in the right order." An agent that reaches the correct conclusion through a risky path, wrong tool, wrong arguments, corrected by chance, is an agent that will fail as soon as conditions shift slightly. Measuring only the final outcome gives a false sense of reliability.

A retrieval-augmented (RAG) system

Two separate measurements matter: retrieval quality (were the right documents found) and generation quality (does the answer actually rely on those documents, without fabrication). Conflating the two hides the real cause of a failure: a bad answer can come from retrieval missing the right document, or from generation ignoring it entirely and inventing a plausible-sounding answer instead. Teams that only track a single end-to-end accuracy number for a RAG system routinely spend weeks tuning the wrong component, because the score never tells them which half of the pipeline actually broke.

A simple generative system

Here, an automated judge calibrated against human review is usually enough, provided edge cases are explicitly covered: empty inputs, off-topic requests, ambiguous phrasing, or an unexpected language. That is exactly the category of case a demo, by design, never shows.

Reading the results without drawing the wrong conclusion

A high overall score can hide a systematic failure on one specific category of requests, often the costliest one for the business. Imagine a hypothetical 92% correct-answer average: on its own, that number is decision-useless if the remaining 8% of failures concentrate on the biggest clients' requests or on legally sensitive cases. Good practice means breaking the score down by request category rather than looking at a single average, and setting the blocking threshold on the most sensitive category rather than on the whole set.

Watch out for an automated judge that is too lenient: a judge calibrated once and never rechecked drifts over time as the evaluated system changes. A periodic check, even a lightweight one, against a fresh human-graded sample remains the only way to confirm the judge is still measuring what you think it measures.

When a formal evaluation is not the right choice

Building a full evaluation pipeline is not always justified. For a throwaway prototype meant to validate a product direction before any real investment, or for an ordinary deterministic script that standard unit tests already cover well, the effort of a formal eval outweighs its value. In a short delivery cycle measured in weeks, spending several days building an evaluation set for a feature that might be dropped after the first round of user feedback is a poor trade-off.

The signal that should trigger investment in structured evaluation is different: it is the moment a real user is about to see the system's output, a decision starts relying on it, or the system moves from occasional use to repeated use. Before that point, a quick manual review of a few dozen cases by someone who knows the domain remains sufficient. Building the full five-step pipeline earlier than that mostly produces a false sense of rigor around a feature whose scope has not stabilized yet.

What it costs, in time and discipline

The most expensive part is almost never the tooling, it is the domain expert's time to grade the first cases and write usable critiques. That step cannot be delegated to a model without ending up calibrating an automated judge against itself, which amounts to measuring nothing at all. Once that human baseline exists, rerunning the evaluation on every change costs a few minutes and avoids hours of production fixes later, often under time pressure and in front of the client.

One additional caution: evaluation platforms change quickly, and some features get retired with little notice, which argues for documenting the method and the test cases independently of any single tool rather than depending on one proprietary platform to keep the history of past verdicts. A plain spreadsheet with the fifty graded cases and the written critiques is often more durable than a dashboard tied to a single vendor’s roadmap.

Read more about the forward deployed engineer role, where evaluating before a demo is part of the daily routine, and our piece on the real cost of an AI feature in production, which breaks down where the budget goes once the demo is over. For an overview of our packages, see our packages and pricing, and for common questions about our method, our FAQ.