A catalogue team watches a model produce a product description in seconds. Support watches it suggest a ticket category. The description attracts more attention because it looks like a finished asset. The category looks mundane. That is a poor basis for choosing the first investment.
For a small commerce team, the better first task often has a clear input, a checkable output and a cheap way to recover when it is wrong. The case for it should survive the cost of that checking. Speed at generating an answer is only one part of the calculation.
Compare complete jobs, not model response time
Illustrative planning example: Aster Home is a fictional retailer considering two first-month pilots. Every workload, timing, labour value and tool cost below is a design assumption to test, not a measured result, supplier quote or Aright project outcome.
The same $30 per hour internal labour value is used for both options. It values staff capacity; it does not mean payroll will fall when a task takes less time.
| First-month planning input | Catalogue-description assistance | Internal support-ticket triage |
|---|---|---|
| Workload | 120 product records | 300 tickets |
| Current human work per item | 15 minutes | 3 minutes |
| Current total human time | 30 hours | 15 hours |
| Proposed human time per assisted item | 3 minutes preparing input + 8 minutes checking/correcting | 1 minute checking/correcting a suggested category |
| Assisted recurring human time | 22 hours | 5 hours |
| Additional setup and evaluation time | 3 hours | 2 hours |
| Total proposed first-month human time | 25 hours | 7 hours |
| Capacity released if assumptions hold | 5 hours | 8 hours |
| Incremental first-month tool cost | $90 | $45 |
| Released-capacity value less tool cost | $60 | $195 |
Catalogue arithmetic is 120 × 15 ÷ 60 = 30 hours before, and 120 × 11 ÷ 60 + 3 = 25 hours after. Its modelled value is 5 × $30 − $90 = $60. Triage is 300 × 3 ÷ 60 = 15 hours before and 300 × 1 ÷ 60 + 2 = 7 hours after: 8 × $30 − $45 = $195.
These figures are an incomplete investment case until quality is checked. The proposed minutes must include correction and escalation work, not just acceptance of the easy items. If the catalogue reviewer spends longer verifying unsupported claims, or triage mistakes create extra handoffs, the assumed benefit shrinks.
Why the less impressive output gets the first trial
Aster would first test internal triage. A suggestion such as “delivery question—human order lookup required” can help a staff member open the right work queue without deciding the customer's remedy. The pilot has no permission to send a reply, close a case, change priority invisibly or issue a refund.
That boundary is part of the economic choice. A catalogue mistake can become a customer-facing product promise if checking fails. A ticket-category mistake can also cause harm, especially if it hides an injury, repeated complaint or urgent delivery issue. Aster therefore keeps the original ticket visible, requires a person to confirm the category, and sends safety or unclear cases directly to the relevant human queue. The model's confidence score does not remove that responsibility.
The NIST Generative AI Profile recommends evaluating capabilities empirically and warns against generalising from narrow demonstrations. Its guidance supports testing in the intended work setting; it does not substantiate Aster's timings or certify a tool.
The alternative is not necessarily another AI purchase. A better form, clearer routing labels or a saved response might remove the same effort more cheaply. Compare the proposed pilot with that simpler intervention before treating a language model as a requirement.
Make the trial answer a question management can use
Aster's pilot decision could read:
Trial: Suggested internal categories for a bounded batch from the normal support queue, with human confirmation and the original message retained. No automatic customer action.
Question: Does this reduce total staff handling time while preserving correct routing and visibility of important exceptions?
Measure: Input preparation, checking, correction, extra handoffs and evaluation time; missed or delayed important cases; time to the appropriate human response.
Reject: Any apparent saving obtained by leaving cases unreviewed, suppressing complaints or delaying customer help. Pause if a material safety or remedy issue is missed and repair the failure before continuing.
Use comparable task types and an explicit timing method. Include difficult and ambiguous cases rather than demonstrating only clean requests. Preserve a way to work manually during the trial. A small evaluation can expose a failure and revise a time estimate; it cannot establish universal reliability.
The table gives management a sensitivity question. With two hours of setup and a $45 tool cost, triage releases value above that cost only if recurring human work stays below 11.5 hours: 15 − 2 − 45/30 = 11.5. At 300 tickets, that is below 2.3 minutes per ticket. Quality remains a separate condition; a faster but harmful workflow does not pass because it clears the financial threshold.
Released hours need a destination
Eight hours released in a spreadsheet are not $240 arriving in the bank. They become useful when the team can redeploy them to a real constraint, avoid an otherwise necessary paid cost, or reduce a backlog without lowering service quality. If staff must remain available and there is no valuable replacement work, the labour-value estimate overstates the commercial benefit.
Aster might assign the time to unresolved customer questions or product-page corrections. It should then check whether the hours actually became available. Do not count the same hours once as reduced payroll and again as extra productive capacity.
The recommendation would change if catalogue facts became easier to verify, triage required more sensitive data than the business could appropriately use, or human routing was already almost frictionless. Likewise, a tool or workflow change requires a fresh estimate. The first AI task should earn a modest place in the operation through a better complete job—not through the most polished demonstration.