Define what “finished” means
Pick one repeatable workflow, such as service intake or an invoice exception. Record its starting event and the evidence required to accept its result. Keep the definition identical across the current process and each proposed AI path.
Separate first response, model completion and accepted task completion. Streaming an early token can improve the experience without making the finished business task faster. If an employee must correct the output, count that time too.
Break the delay into stages
| Stage | What to record |
|---|---|
| Queue and preparation | Waiting, retrieval, input validation and context assembly. |
| Model execution | Each provider request, time to first response and complete response. |
| Handoffs and retries | Switching decisions, context transfer, backoff and repeated calls. |
| Validation and review | Automated checks, reviewer time, corrections and final acceptance. |
Use one case identifier to connect the events. Distinguish wall-clock time from the sum of stage durations when work runs in parallel. Report waiting time separately from staff effort.
Compare three paths on representative cases
- Current baseline: the process people actually use today, including existing automation.
- Simple AI path: one suitable model with the same checks and business acceptance rules.
- Proposed routing path: the additional decisions and handoffs being evaluated.
Use a held-out set of representative cases with a documented mix of straightforward, difficult and interrupted work. Record model versions, settings, input size, traffic conditions, concurrency and cache state. Avoid comparing an empty test environment with a busy production baseline.
Report slow cases and failed cases together
The median, or p50, describes the middle case. The 95th percentile, or p95, exposes the slower end of the workload. Include the number of observations and the evaluation period so reviewers can judge how much evidence supports those figures.
Show timeouts, rejected results, retries and review rates beside latency. A fast path that fails difficult cases is not a complete improvement. Count unresolved cases explicitly rather than quietly dropping them from the report.
Agree on acceptable quality and delay before reviewing results. Continue only if the proposed path meets those thresholds and improves the cost per accepted case. Steprill has not published measured latency benchmarks; the pilot is where these assumptions must be tested.