A pilot only earns its keep if it ends with a number you can defend. Yet most pilots end with a vibe: “it felt faster,” “people liked it.” Here is how to measure one so the result makes the next decision for you, instead of starting another debate.
Measure the baseline before you build anything
The single most common measurement mistake is starting to measure after the AI is already in the loop. By then you have nothing to compare against. Before a line of the pilot is built, capture how the work happens today: how long it takes, how often it is redone, what it costs, and how often it goes wrong. A rough baseline you wrote down beats a precise one you reconstruct from memory later.
If you are still choosing what to pilot, the use-case finder helps you pick a candidate that is actually measurable in the first place.
Pick one primary metric, not five
A pilot with five success metrics has none. Choose the one number that would genuinely change your decision, and treat everything else as supporting context. In practice the primary metric is almost always one of four things:
- Time. Hours saved per week on a specific task. The easiest to measure and usually the easiest to believe.
- Quality. Errors, rework, or complaints, counted before and after.
- Cost. Money spent to get the same output, including the cost to run the AI.
- Revenue. Harder to attribute cleanly, so only lead with this when the link is direct and obvious.
Count the full cost, not just the licence
A saving is only real if you have subtracted what it took to get it. Net benefit means the value the pilot created minus the cost to build it, run it, and keep a human reviewing it. A tool that saves ten hours a week but needs eight hours of oversight is a very different story than it first appears. The ROI calculator is built to force that subtraction, so the figure you present is the honest one.
Watch for the measurements that lie
A few traps quietly inflate a pilot result. Guard against them up front:
- The novelty bump. People try harder with a new toy. Measure over a few weeks, not the first exciting afternoon.
- The cherry-picked sample. Test on the messy, real cases, not the three clean examples that always work.
- The hidden human. If a person is quietly fixing the output, that work is part of the cost. Count it.
Write the result as a decision, not a report
The output of a good pilot is one page: the baseline, the primary metric before and after, the full cost, and a plain recommendation to scale, adjust, or stop. That is exactly what a Pilot Project is scoped to deliver. If the numbers point the wrong way, that is a good outcome too, and a pilot that says no is worth reading about before you run one.