Run a Small-Business AI Pilot from Baseline to Decision

Choose a problem with a visible outcome

This guide uses enquiry triage for a training business as an illustrative pilot. Messages arrive through several channels and need to be routed to the right person. The pilot suggests a category and extracts missing information; a person reviews the result. It does not autonomously send offers, set prices or confirm bookings.

Choose a task that is frequent enough to observe, bounded enough to understand and low enough in consequence for a controlled trial. If the team cannot agree on the current routing rule, resolve that process issue first. Automation is not a substitute for deciding who owns the work.

Record the manual baseline

Observe a representative sample of enquiries. Record handling time, the final category, missing information, rerouting and errors. Include incomplete and ambiguous messages. Keep personal data out of the evaluation copy when synthetic or anonymized examples can serve the same purpose.

Do not measure only the time spent typing. The complete task includes finding context, checking the suggested result, correcting it and recording the handoff. A model may generate a category quickly while increasing review work. The baseline makes that trade-off visible.

Define the pilot contract

Build a reviewable first version

Use a small controlled category set with an explicit unclear outcome. Preserve the original message and distinguish extracted facts from inferred labels. If a college-training enquiry omits participant count and preferred format, the output should flag those gaps rather than manufacture complete-looking data.

Show the suggestion beside the source message. Let the reviewer accept, correct or escalate it, and record the correction reason. This interface generates evidence about which distinctions are difficult. It also keeps the team in control while the pilot is still learning about real inputs.

Measure the entire workflow

Compare reviewed outcomes with the manual baseline. Track correct routing, important omissions, review time and cases that require escalation. Keep errors with different consequences separate: mislabelling a general question is not equivalent to making an unauthorized commitment. Record the cost of setup and ongoing maintenance as well as per-request usage.

Use the same definitions throughout the comparison. If the category scheme changes halfway through the pilot, annotate the change and avoid presenting the before-and-after numbers as directly comparable without explanation. Include examples of failures so the decision is not based only on an average score.

Set a clear continuation and stop rule

Write the decision criteria before reviewing final results. A continuation rule might require useful routing accuracy, manageable review effort and no unresolved critical boundary failure in the tested scope. The exact thresholds should reflect your business and sample; there is no universal number that makes every AI pilot ready.

Stop or revise if the task rules remain unclear, if review erases the benefit or if the system cannot protect the required data and action boundaries. A stopped pilot can still produce useful process improvements, such as better enquiry categories and clearer ownership. Treat that learning as a result rather than hiding it.

Prepare operations before expanding

Assign ownership for category changes, source material, incident review and cost monitoring. Keep a manual routing path available and record manually completed work so recovering automation does not duplicate it. Expand to a larger sample or an additional channel only after the current workflow is understood.

• One task and one responsible owner are named.
• The manual baseline includes review and correction effort.
• The pilot preserves source messages and uncertainty.
• Quality and operational criteria were defined in advance.
• Failures and exceptions remain visible.
• Manual fallback and reconciliation have been practiced.

The finished deliverable is a pilot decision memo with evidence, not a promise of guaranteed savings. Record what was tested, what changed and what remains unknown so the next decision starts from an honest foundation.