How to measure AI project success

If measurement is not designed before the work begins, it does not get designed. What happens instead is a conversation six months later in which the supplier presents whichever numbers look best and you have nothing to check them against.

Not usually dishonesty. Just the absence of a baseline, and the absence favors the person presenting.

Take the baseline first, and take it narrow

Two or three numbers, captured before anything changes.

Resist measuring everything. A wide dashboard nobody reads is worse than three figures somebody actually looks at, because breadth gives you a way to find a good story regardless of what happened.

Pick numbers that would move if the project worked and would not move much for other reasons. That second condition is what most people skip, and it is the one that makes the result mean anything.

Four types worth tracking

A volume number. How many of the thing happens, and how many get handled. Inquiries received against inquiries answered, quotes sent against quotes followed up. The most straightforward and the least arguable.

A time number. How long between something arriving and something happening about it. Response time, turnaround time, how long a task takes end to end. Usually the metric that moves first and most visibly.

A conversion number. What share of the things reach the outcome you wanted. This is the one that decides whether the project paid, and it is the one most often left out because it is the least flattering.

An error number. How often the system got it wrong, how often it escalated, how often somebody had to redo the work. Track this deliberately: a project that improves speed while quietly degrading quality will look excellent for a quarter.

The last one is the number a supplier is least likely to volunteer. Ask for it.

Vanity metrics to refuse

Response time in seconds. It will be seconds, always, because that is what software does. It proves the thing is switched on.

Messages sent, tasks processed, conversations handled. Volume of activity is not evidence of value, and "handled" is doing a lot of work in that sentence. A conversation that ended with somebody promising a callback was handled, and lost.

Hours saved, calculated theoretically. Multiplying a task count by an assumed duration produces an impressive figure with no connection to anything. If hours were genuinely saved, something else should be visibly different: more output, or fewer people on that task, or somebody doing higher-value work. Measure that instead.

Model accuracy in isolation. Interesting technically, disconnected from business outcomes. A system can be accurate on the cases it handles and useless because of what it fails to handle.

Give it long enough, and pick the right window

A month minimum, and pay attention to seasonality.

Anything measured during your busiest period will flatter a coverage or capacity improvement, and the same system will look disappointing in a quiet month while working exactly as well. If your business has a season, either measure across both or compare to the same period last year.

Two months spanning different conditions is the honest window for most things.

Attribution, honestly handled

The awkward part. Business results have many causes, and a supplier presenting a revenue increase alongside their invoice is inviting a conclusion the data may not support.

Two protections. Keep the metric close to the work: if you automated inbound response, measure inbound response, not quarterly revenue. And note what else changed in the period, because you probably also hired someone, ran a promotion, or had unusual weather.

Being rigorous here occasionally means concluding that something worked less well than it appeared. That is the price of being able to trust the times it worked.

What cannot be measured, and should not be faked

Some real value does not produce a number, and naming it honestly beats inventing a proxy.

Work that is now done consistently rather than depending on somebody remembering. Not carrying an unfinished list around in your head. A customer relationship retained for years because you responded once, at the right moment, which will never appear in a monthly report.

Put those in words in the review, not in a spreadsheet with a fabricated figure attached.

One thing to calibrate before the first review: most projects have a stretch in month two that feels worse than before it started, and it is normal rather than a failure. What to expect, by phase.

The review that matters

Set a date three months after go-live and hold it. One page: the baseline, the current numbers, the error rate, what changed that was not the project, and one decision about whether to continue, adjust, or stop.

That last option needs to be available or the review is theater. Projects that cannot be cancelled do not get evaluated, they get defended. Which is how good work drifts into irrelevance.

The measurement chain for the specific case of inbound handling is worked through in measuring ROI on call handling, and the roadmap we produce carries an estimate per recommendation so there is something to check the result against: how that works.