Impressions are not evidence
After a few weeks with a new tool, people usually have a feeling about it. Enthusiasts say it is a game changer; skeptics say it is more trouble than it is worth. Both may be right about their own experience, yet neither feeling tells you whether the organization should keep paying for it.
A few well-chosen measures turn opinion into evidence. They do not need to be sophisticated. They need to be agreed in advance and applied the same way before and after.
Start with a baseline
You cannot show improvement without a starting point. Before the trial begins, record how the task performs now. Depending on the work, that could mean:
- how long a typical item takes,
- how many items are completed in a week,
- how many mistakes or rework cycles occur,
- how long customers or colleagues wait for a response,
- how people rate the task’s effort or frustration.
If you did not capture a baseline, you can sometimes rebuild one from timestamps, tickets or a short sample of manual work. Rough is far better than nothing, but note how rough it is.
Choose a small set of measures
Three to five measures are usually enough. A balanced set looks at several angles:
Speed. Time per task, turnaround time or backlog size. Count the whole job, including the time spent checking and correcting the tool’s output. A tool that drafts quickly but requires long fixes may not save time overall.
Quality. Error rate, number of corrections, consistency between items, or a reviewer’s rating against a simple scale. Decide how you will judge quality before you see results, and have the reviewer score blind if you can, without knowing which items used the tool.
Effort and experience. A short survey asking whether the task felt easier, whether people trusted the output and whether they would choose to keep using the tool.
Cost. The full total cost, including review time, compared with the cost of the current approach.
Risk. Incidents such as sensitive data entered by mistake, outputs that needed escalation, or complaints. Zero incidents is an important result in its own right.
Avoid counting things just because they are easy. Number of logins or number of prompts says little about value.
Define success in advance
Write down what result would make you keep the tool, what would make you stop, and what would mean extending the trial. For example, you might decide that the tool must reduce time per task without increasing the error rate, and that any data-handling incident pauses the trial. Setting thresholds beforehand protects against moving the goalposts once everyone is attached to the tool.
Keep the thresholds modest and honest. Precise targets pulled from nowhere create false confidence. The agreed bar should reflect what matters for your task, not what a vendor’s brochure suggests.
Collect data fairly
A few habits keep the comparison honest.
- Use comparable work. Compare similar tasks, not easy ones with the tool against hard ones without.
- Allow a learning period. The first days are often slower. Measure after people have had time to get used to it, and note the early dip separately.
- Include enough items. A handful of examples can mislead. Collect as many as is practical, and be cautious about drawing firm conclusions from small samples.
- Check the outputs. Because tools can produce fabricated answers, sample results for accuracy and avoid assuming fluent text is correct. Structured human review makes this routine.
- Record context. Note holidays, staff changes and unusual workloads, which can distort results.
Watch for common traps
Novelty effect. People may use a new tool more diligently at first, which flatters early results.
Selection bias. If only enthusiastic volunteers take part, results may not carry over to the wider team. A pilot project should say who took part and why.
Ignoring hidden work. Time spent rewriting prompts, formatting, or fixing outputs belongs in the tally.
Cherry-picking. Remembering the impressive successes and forgetting the failures. Keep a simple log of both.
Overgeneralizing. A good result on one task does not prove the tool suits every task.
Make the decision and write it down
At the end, compare results with your thresholds and summarize in a page: what was measured, what happened, what it cost, what risks appeared and what you recommend. Options include adopting the tool, adopting it with a narrower scope, extending the trial or stopping. Because outcomes vary, a decision to stop is a legitimate and useful result, and the notes on tool retirement cover how to do it cleanly.
If you adopt the tool, keep measuring at intervals. Performance can drift as the work, the product or the team changes, and a lightweight yearly check keeps the decision current.