How should buyers evaluate AI agents?

Judge an AI agent on finished jobs from your own cases: the full cost of each one, how it hands off what it can't finish, and whether that cost keeps falling.

Skywake Labs5 min read

Evaluate an AI agent on finished work from your own cases, not on a demo or a headline accuracy score. Measure the full cost of each finished job, including the time your people spend reviewing, fixing and completing it. Test what the agent does with cases it can't handle. Then watch whether that cost falls month after month: an agent that stays flat is a weaker choice than one that gets cheaper to run with every job.

Why do demos and accuracy scores mislead buyers?

A demo runs on cases the seller picked. An accuracy score usually counts steps, not jobs. Neither tells you what it costs to get one of your invoices or claims all the way to done.

The gap sits in the outputs that are mostly right. An invoice with every field right except the amount isn't almost done. Someone still has to spot the wrong field, fix it and check that nothing downstream used it. A score that counts that output as a near-success hides the person who had to finish it.

So the useful unit of evaluation is the finished job: the case that reaches the result you would have accepted from your own team, with no one redoing the work.

What should you measure when you evaluate an AI agent?

Five measures tell you most of what you need to know.

  1. Finished-job rate. The share of cases that reach done with no rework. Count partly right outputs as unfinished.
  2. Rework time. How often a specialist changes the output, and how long each change takes. A short fix and a full redo are not the same cost.
  3. Handoff quality. When the agent can't finish a case, does it say so? A good handoff stops, passes the case to a person and explains what it found and why it stopped. A bad one reports success on work that isn't done.
  4. Cost per finished job. The agent's fee plus every minute of people time spent reviewing, fixing and completing cases, divided by the number of finished jobs. This is the number to compare with what the work costs you today.
  5. Predictability. Whether you can forecast next month's bill from next month's volume. A price that swings with retries, usage spikes or reruns is hard to budget, however low it looks per unit.
The number to compare is the agent's fee plus every minute of people time spent reviewing, fixing and completing cases, divided by the number of finished jobs.
The number to compare is the agent's fee plus every minute of people time spent reviewing, fixing and completing cases, divided by the number of finished jobs.

How do you build a fair test from your own work?

Start with a batch of recent cases in their real mix, not a clean sample. For invoice processing, that means last month's invoices as they arrived: the routine ones, the duplicates, the partial deliveries and the new supplier with unusual terms. For claims, it means the straightforward intake alongside the files with missing documents or conflicting evidence.

Then work through four steps.

  1. Write down what done means for each kind of case before the test starts, in terms your team already uses.
  2. Score whole cases against what your team actually did, not individual steps.
  3. Keep some cases back and run them later, so the result doesn't depend on cases the setup was tuned for.
  4. Run the same batch for every option you're comparing, including the way you do the work today.

Whether a workload suits an agent at all is a separate question, which we cover in When should you not use AI agents?.

Why does the trend matter more than the first score?

Because the first score tells you what the agent costs today. The trend tells you what it will cost for as long as you use it.

We judge a business by how fast the work gets cheaper, and the same test applies to an agent. When a specialist steps in on a case, what happens to that judgment? If it is captured with its reason and built back in, the same kind of case needs less of a person's time next month. If it stays in one person's head or a one-off fix, you pay for the same judgment every time the case comes back.

The first score tells you what the agent costs today. If each judgment is built back in, people time per finished job falls; if it is not kept, that people cost stays permanent.
The first score tells you what the agent costs today. If each judgment is built back in, people time per finished job falls; if it is not kept, that people cost stays permanent.

You can see the difference in a pilot. Measure people time per finished job in the first weeks, then again after several cycles of normal volume, including a month-end or seasonal peak if your work has one. A line that stays flat means the people cost behind the agent is permanent. A line that keeps falling means the agent is learning from the work. We explain how that loop works in How does a data flywheel make each job cheaper in AI-native services?.

Are you evaluating an agent or buying finished work?

Buying an agent means you run it. Your team sets it up, defines the cases, finishes what it hands back and keeps it current as suppliers, policies and rules change. The five measures above tell you whether that trade is worth it.

There is another option: buy the finished work itself. The same measures apply, but the seller carries the cost of rework and exceptions, and the price attaches to each finished unit. We describe how that pricing works in What is outcome-based pricing? and the model behind it in What is an AI-native services company?.

Either way, the question is the same: what does one finished job cost, and how fast is that cost falling?

Questions people ask

What metrics should you use to evaluate an AI agent?

Start with the finished-job rate, rework time, handoff quality, cost per finished job and how predictable the bill is. Track people time per finished job over several weeks, because the trend matters as much as the starting point.

How do you test AI agent accuracy?

Run the agent on a batch of your own recent cases in their real mix, and score whole cases against a written definition of done. Count partly right outputs as unfinished, because someone still has to find and fix the error.

How long should an AI agent pilot run?

Long enough to see a trend, not just a first result. That usually means several cycles of normal volume, including a month-end or peak if your work has one, so you can tell whether people time per finished job is falling.

What are red flags when evaluating an AI agent?

An agent that reports success on work it didn't finish, a seller who can show only hand-picked cases, and a bill you can't forecast from your own volume. A people cost that stays flat across the pilot is another warning sign.

Questions or ideas? Email hello@skywakelabs.com.

Buyers, investors and partners: hello@skywakelabs.com

More notes

  1. Services5 min read

    What is agentic BPO?

    Outsourcing where AI agents work most cases and specialists take the exceptions. Real agentic BPO prices the finished case, not seats or agents.

    Read
  2. Judgment6 min read

    Will AI replace procurement teams?

    Not the team. AI takes the paperwork of buying; people keep the decisions with no written answer: which supplier, on what terms and whether to buy at all.

    Read