Freeze the task list and acceptance rules before testing. Compare only matched correctly completed tasks for timing, report every outcome against the original denominator, and include setup, checking, correction, refusal and fallback effort.
Choose tasks and acceptance rules first
Pick recurring tasks and freeze their company, period, records and required output before seeing results. Include ordinary and awkward cases. Record who may access the data and which approved sample or snapshot each task uses.
| Decision | Owner-set rule |
|---|---|
| Task coverage | ___ of ___ named tasks must complete |
| Correctness | Required reference and allowed tolerance: ___ |
| Maximum checking effort | ___ minutes per task or ___ total |
| Unsupported/refusal handling | Which cases should refuse, and what fallback completes the job: ___ |
| Cost ceiling | Setup + recurring + usage + support: ___ |
| Data permission | Approved records, people, retention and withdrawal process: ___ |
There is no universal pass threshold. The task owner sets it before results are known and applies it consistently.
Count the whole matched task
In this fictional example, both methods correctly complete the same ten tasks. Both start with the same prepared records. The previous method takes 12 minutes per task. The trial takes three minutes to ask and receive, four minutes to check, and two answers need five additional correction minutes each.
| Work | Minutes |
|---|---|
| Previous method: 10 × 12 | 120 |
| Trial: 10 × (3 + 4) | 70 |
| Extra correction: 2 × 5 | 10 |
| Total trial time | 80 |
| Matched-task difference | 40 saved |
The difference is 40 minutes, or one-third of 120 minutes. It is not 90 minutes because checking and correction are part of the task. These are example inputs, not Dhandha customer results.
Keep the original denominator
If only eight of ten tasks finish correctly, report 8 ÷ 10 = 80% correct completion. Compare timing only for the same eight completed tasks and label that narrower comparison. Keep effort on the two unfinished tasks, manual fallback and recovery visible.
A refusal can be correct when required information is missing. It still does not complete a task the business needs. Record correct, wrong, appropriately refused and unfinished separately.
Keep an executable evidence log
| Task ID | Outcome | Reference and evidence | Effort to completion | Verdict |
|---|---|---|---|---|
| T01 | Correct | Snapshot S1; reference R1 | 7 min | Pass |
| T02 | Wrong, corrected | Difference D2; corrected against R2 | 12 min | Pass after correction |
| T03 | Appropriate refusal | Required period absent; evidence E3 | 5 min + 9 min fallback | Refusal correct; task completed manually |
| T04 | Unfinished | Error E4; no accepted answer | 11 min trial effort | Fail for completion |
Link each answer, source snapshot, reference, reviewer and discrepancy. Record one-time onboarding and preparation separately from recurring effort so later runs are not credited with work done elsewhere.
Count cost and decide per task
Include setup labour, subscription or usage fees, approved record preparation, checking, corrections, fallback and support. Time released is not automatically cash saved.
For each task, decide: retain as-is, retain with a named fix, require manual fallback, or stop. Then make a portfolio decision against the pre-trial contract. A fast result that fails the reference does not pass.
The NIST AI Risk Management Framework is a broader risk reference. This log is an operating worksheet, not a NIST certification.
Questions you might ask
How should I compare time when some AI trial tasks remain unfinished?
Compare time only for the same correctly completed tasks and label that scope. Report completion against the original task list and retain unfinished-task and fallback effort.
What is an acceptable AI trial success rate?
There is no universal threshold. The task owner should set coverage, correctness, checking-effort and cost rules before seeing results.
What should an AI trial evidence log contain?
Keep stable task IDs, inputs or snapshot, answer, independent reference, outcome, review and recovery effort, evidence links and verdict. Separate one-time setup from recurring work.