How to evaluate an AI agent demo before you rely on it
Manoa editorial
https://manoa.health/resources/explainer-how-to-evaluate-an-ai-agent-demo-before-you-rely-on-it

An AI demo can go perfectly while the real workflow fails because the two test different things. A demonstration runs prepared records once, in a single sitting. Daily purchasing work brings missing details, slow replies and approvals spread over days. Here, an AI agent means software that uses a model and connected tools to choose and carry out steps toward a goal.

Imagine a vendor shows your team its purchasing software. In 15 minutes, its AI agent compares two glove quotes, recommends the cheaper one and writes to a supplier about a late order. Every figure is right and every step finishes. The scene is invented, but many operators will recognize it.

Now imagine the same product a month into a trial. Staff are checking quote comparisons by hand because some totals were wrong. A supplier’s reply has sat unanswered for three days. Nobody is sure whether the late order was ever cancelled. Both scenes can be true of the same software.

A demonstration is usually prepared in advance. The records are complete and chosen for the occasion. Every item has a pack size, every quote states its freight, and the supplier answers the question it was asked. The software can read and change whatever it needs, because nobody has set limits on its access. The task runs once, from start to finish, in a single sitting. Nothing has to wait for a reply that arrives next Tuesday.

Real purchasing work rarely offers all of those conditions. A quote leaves out the pack contents or the account the price applies to. A supplier confirms a new delivery date and ignores the question about a substitute. The order says two cases and the packing slip says one. A change needs someone with authority to approve it, and that person is with patients until four. The software can read the supplier’s email but can’t see the contract terms. The work pauses for days, then has to pick up again with the right context.

Each step an agent takes depends on the one before it. Anthropic’s [guide to evaluating agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) notes that agents act across many turns and change the systems they work in, so “mistakes can propagate and compound.” A wrong pack size in the first step becomes a wrong total, then a wrong recommendation, then a wrong order.

Federal guidance on AI risk describes the same gap in general terms. The National Institute of Standards and Technology’s [AI Risk Management Framework](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf) says measurements taken in a laboratory or controlled environment “may differ from risks that emerge in operational, real-world settings.” It asks for performance to be demonstrated under conditions similar to where a system will be used, and for its behavior to be monitored once it’s in production.

Take the quote comparison from the demonstration. Suppose one supplier sells gloves in cases of ten boxes and the other in cases of 20. A correct comparison converts both quotes to the number of gloves your site needs, then adds freight. Software can do that arithmetic reliably when both packs are stated. When a quote leaves the pack out, the software has to ask someone or guess. Our [guide to comparing quotes](/resources/guide-medical-supply-quote-comparison-checklist-and-calculator) works through that conversion with fictional prices. A good answer depends on the pack and the account terms being written down.

So one useful question for a vendor is what the product does when that information is missing. It might stop and ask the supplier. It might show the gap to your team. It might also fill in a likely value and present a confident total. You want to know which of these it does before you rely on its comparisons.

The late order shows a second gap. Reporting a problem and carrying out the follow-up are separate pieces of work. A product can read a supplier’s email and tell you that Thursday’s delivery has moved to Monday. Following up means asking the supplier for options, bringing them to the person who can approve a change, requesting a cancellation if one is approved and checking that the supplier confirmed it. That work spans days and several people, and it has to resume each time a reply arrives. Our [guide to backorders across locations](/resources/guide-medical-supply-backorders-across-multiple-locations) lists those steps and who owns each one.

A 15-minute demonstration can show the first message to the supplier. It can’t show the third follow-up four days later, or what happens when the supplier answers only half the question. Those are the moments when work tends to stall.

It helps to separate three things a vendor might show you. Advertised capability is what the product is said to do, on a website, in a proposal or in a sales conversation. Demonstrated behavior is what you watched it do on the examples chosen for the demonstration. Sustained performance is what it keeps doing on your own work, over weeks, with missing information, slow suppliers and the people who approve changes.

Each tells you something different. A claim tells you what the vendor intends the product to do. A demonstration shows that the product can do it under prepared conditions. Only a trial on your work, measured over time, shows whether it keeps doing it. An impressive demonstration moves a product from the first category to the second. It tells you little yet about the third.

You can ask for evidence closer to the third before you commit. Start by asking the vendor to run the demonstration on your own fictional records. Take a few recent purchases, change the names and prices, and keep the untidy parts: a quote without a pack size, an item listed under two numbers, a supplier reply that answers half the question. Include at least one request with information missing, and watch what the product does with it.

Then check what changed in the records. An agent’s summary of its work describes what it says happened. The order, the quote comparison and the message to the supplier show what actually happened. Anthropic’s guide makes the same point with a travel example: an agent can report that a flight is booked, and the outcome is whether a reservation exists. After the demonstration, open the records and confirm the totals, the pack sizes and who approved each change.

Finally, count the work that stays with your team. In a trial, record the time staff spend setting the product up, reviewing its output, correcting mistakes and following up on anything it left unfinished. Record whether each task finished correctly and whether the right person made each decision. Keep counting for several weeks, so the tally includes slow replies, missing details and changed orders.

Those numbers answer the question a demonstration can’t. If the product saves staff an hour a day and needs 50 minutes of checking, little has changed. If it handles missing information by asking, shows its work in the records and leaves your team only the decisions that belong to them, it may be ready to rely on.

Manoa does purchasing work of this kind, including the follow-up, and the same three questions apply to us. Our [purchasing examples](/example) let you check those records and decisions yourself. You can also ask us to run the demonstration on your own records.

Sources
Anthropic: Demystifying evals for AI agents: https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents (checked 2026-09-23)
NIST: Artificial Intelligence Risk Management Framework (AI RMF 1.0): https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf (checked 2026-09-23)