Make the launch call

Recommendation: Go with Conditions

While the pilot demonstrated strong operational impact—reducing median first-response time by 41% (beating our 30% target) with a 64% acceptance rate—we failed two mandatory launch criteria:

1. Accuracy (Criterion 2): Harmful/wrong drafts hit 1.8% (9/500) versus the $\le$1.0% threshold.
2. Refund commitments (Criterion 3): Two drafts explicitly promised refunds, violating our zero-tolerance policy.

However, these failures are isolated and addressable. Six of the nine errors were caused by August reference data lacking the September 1 price changes, and both refund commitments originated in billing workflows.

Rather than cancelling Friday’s 42-agent training or delaying the productivity benefits across the entire organization, we should launch on Monday under the following three conditions:

Launch Conditions

1. Disable AI drafting on the Billing queue for Monday: Billing represents 38% of volume and contains almost all our critical failure points (pricing errors and refund risks). We will launch exclusively to the remaining three queues (62% of ticket volume), where accuracy meets our threshold. 2. Implement deterministic refund guardrails by Friday: Deploy a hard system-prompt restriction and a post-generation regex/keyword filter that blocks or drops drafts containing refund commitments before reaching the agent. 3. Update agent training on Friday: Keep Friday’s training on schedule. Explicitly train agents on: - The pilot's phased rollout (Billing queue excluded initially). - Known minor bugs (Welsh-language tickets defaulting to English; manual table reformatting required). - Strict agent accountability as the final human reviewer for every sent message.

Path to Full Billing Rollout

Engineering will ingest the post-September 1 pricing documentation and run a targeted 200-ticket eval on billing drafts next week. Once billing tickets achieve $<1\%$ error and 0 refund promises, we will enable the feature for the billing queue.

This approach protects Ledgerly’s financial liability and customer trust while immediately capturing significant speed improvements for over 60% of our support volume.

Launch recommendation: AI-drafted support replies

Recommendation: no-go for all 42 agents this Monday. Go the following Monday (5 Oct) if three conditions are met.

Scorecard against PRD criteria

Two of four criteria fail. The value is proven, but the safety bar isn't met yet.

Why not Monday

• Refunds are a hard line, and we crossed it. Two drafts promised refunds. Agents caught both, but the criterion says no draft may do this. Human review is the backstop, not the control. The pilot used 8 motivated agents. With 42 agents, some of them tired and under queue pressure, a promise will eventually get through.²
• Billing errors land on our biggest queue. Six of the nine failures quote pre-1 September prices. Billing is 38% of volume, so this is a systematic error, not an edge case.
• Training comes after launch. Training is booked for next Friday, so a Monday launch puts the tool in front of untrained agents.¹ The pilot agents were hands-on and engaged, and we can't assume the rest will review as carefully.

Conditions for a 5 October go

1. Refund guardrail. Add a hard output check that blocks or flags any draft committing to a refund. Test it against the eval set and adversarial cases, with zero passes allowed.
2. Refresh pricing data and re-run the full 500-ticket eval. Launch only if wrong or harmful drafts are at most 1%. Removing the six pricing errors would leave 0.6% on paper. We should prove that with a re-run, not assume it.
3. Train before rollout. Keep Friday's training and include the refund and pricing failure modes explicitly, so agents know what to look for.

Not blocking

• Welsh replies coming back in English (0.3%). Turn drafting off for Welsh-language tickets until this is fixed. It costs little and avoids a visible failure.
• Table formatting. This is cosmetic, and agents edit every draft anyway. Fix it in the normal backlog.

Meanwhile

Keep the 8 pilot agents on the tool so we don't lose momentum.³ In the first two weeks after launch, spot-check a sample of sent billing replies.

The ask

Agree to the three conditions. Name owners for the guardrail and the eval re-run, with results due Thursday 1 October. If both pass, we confirm go on 2 October after training.

Recommendation: No-go for the all-agent launch on Monday

The pilot demonstrates value, but we have not met the agreed safety gates. Agent review is a safeguard—not a substitute for those gates.

Seven of eight pilot agents want to keep the feature, which supports its usefulness. However, catching the refund promises before sending does not satisfy the requirement that no draft make that commitment.

What must change before launch

• Refresh billing reference data with current prices and verify retrieval uses it. Six of nine harmful drafts used old prices, suggesting a concentrated, fixable problem—but we must also investigate the remaining three.
• Enforce the refund restriction, with testing that shows prohibited commitments are blocked before drafts reach agents. Agent training alone is insufficient.
• Rerun a representative 500-ticket evaluation after fixes. Require no more than five wrong or harmful drafts and zero refund commitments; explicitly cover current pricing and refund requests.
• Handle known quality defects: route Welsh-language tickets to manual drafting until language support is verified; suppress tables or fix their formatting.

Monday plan

Keep Friday’s training, but frame it as readiness training, not a launch commitment. Until safety fixes are verified, pause AI drafting for billing and use manual replies there.

Billing represents 38% of tickets, so excluding it is a material scope change. Any continued non-billing pilot should be explicitly approved, safety-validated, monitored, and reversible—not presented as meeting the original launch criteria. Reassess full rollout once the gates pass.