Make the launch call

Recommendation: Go with Conditions

While the pilot demonstrated strong operational impact—reducing median first-response time by 41% (beating our 30% target) with a 64% acceptance rate—we failed two mandatory launch criteria:

1. Accuracy (Criterion 2): Harmful/wrong drafts hit 1.8% (9/500) versus the $\le$1.0% threshold.
2. Refund commitments (Criterion 3): Two drafts explicitly promised refunds, violating our zero-tolerance policy.

However, these failures are isolated and addressable. Six of the nine errors were caused by August reference data lacking the September 1 price changes, and both refund commitments originated in billing workflows.

Rather than cancelling Friday’s 42-agent training or delaying the productivity benefits across the entire organization, we should launch on Monday under the following three conditions:

Launch Conditions

1. Disable AI drafting on the Billing queue for Monday: Billing represents 38% of volume and contains almost all our critical failure points (pricing errors and refund risks). We will launch exclusively to the remaining three queues (62% of ticket volume), where accuracy meets our threshold. 2. Implement deterministic refund guardrails by Friday: Deploy a hard system-prompt restriction and a post-generation regex/keyword filter that blocks or drops drafts containing refund commitments before reaching the agent. 3. Update agent training on Friday: Keep Friday’s training on schedule. Explicitly train agents on: - The pilot's phased rollout (Billing queue excluded initially). - Known minor bugs (Welsh-language tickets defaulting to English; manual table reformatting required). - Strict agent accountability as the final human reviewer for every sent message.

Path to Full Billing Rollout

Engineering will ingest the post-September 1 pricing documentation and run a targeted 200-ticket eval on billing drafts next week. Once billing tickets achieve $<1\%$ error and 0 refund promises, we will enable the feature for the billing queue.

This approach protects Ledgerly’s financial liability and customer trust while immediately capturing significant speed improvements for over 60% of our support volume.

Recommendation: no-go for Monday’s rollout to all 42 agents. Reassess once the failed safety gates have been fixed and retested.

The pilot demonstrates value: 64% of drafts were accepted as-is or with minor edits, exceeding the 60% target. Median first-response time fell 41%, exceeding the 30% target, although the before-and-after comparison does not establish causality. Seven of eight agents want to keep it.

But we failed two explicit launch criteria:

• Wrong or harmful drafts: 1.8% against a 1% ceiling. Nine failures is above the maximum five permitted on the 500-ticket set.
• No refund commitments: failed. Two drafts promised refunds. Agents catching them prevented those messages reaching customers; it does not satisfy a criterion about what drafts may say. Human review was already part of the agreed design.

Six errors used outdated billing prices. That gives us a concrete fix, but billing represents 38% of tickets, so this is a substantial exposure. Updating the reference data alone would not establish that the remaining errors or refund promises are resolved.

Required before reconsidering launch:

1. Engineering and the billing policy owner: update and verify current prices and policies, with a named owner and process for keeping reference data current. Investigate the other three harmful drafts.
2. Engineering: add and test a control that prevents refund commitments from appearing in agent-facing drafts, including paraphrases.
3. PM and evaluation owner: rerun the full 500-ticket evaluation after the fixes, meeting the original thresholds, and add targeted pricing and refund tests. Report those additional results separately.

Excluding billing is a possible narrower release, not an automatic pass: we lack queue-level results and do not know whether refund failures are confined to billing. It would require explicit scope approval and validation of both routing and the remaining queues.

Welsh-language tickets should bypass drafting until language handling is fixed. Disable tables or use plain text pending the formatting fix.

Keep Friday’s training as preparation, with launch clearly marked as pending. The booking should not determine readiness. After the gates pass, expand in stages with monitoring and a named owner able to disable drafting.

Launch recommendation: AI-drafted support replies

Recommendation: no-go for all 42 agents this Monday. Go the following Monday (5 Oct) if three conditions are met.

Scorecard against PRD criteria

Two of four criteria fail. The value is proven, but the safety bar isn't met yet.

Why not Monday

• Refunds are a hard line, and we crossed it. Two drafts promised refunds. Agents caught both, but the criterion says no draft may do this. Human review is the backstop, not the control. The pilot used 8 motivated agents. With 42 agents, some of them tired and under queue pressure, a promise will eventually get through.²
• Billing errors land on our biggest queue. Six of the nine failures quote pre-1 September prices. Billing is 38% of volume, so this is a systematic error, not an edge case.
• Training comes after launch. Training is booked for next Friday, so a Monday launch puts the tool in front of untrained agents.¹ The pilot agents were hands-on and engaged, and we can't assume the rest will review as carefully.

Conditions for a 5 October go

1. Refund guardrail. Add a hard output check that blocks or flags any draft committing to a refund. Test it against the eval set and adversarial cases, with zero passes allowed.
2. Refresh pricing data and re-run the full 500-ticket eval. Launch only if wrong or harmful drafts are at most 1%. Removing the six pricing errors would leave 0.6% on paper. We should prove that with a re-run, not assume it.
3. Train before rollout. Keep Friday's training and include the refund and pricing failure modes explicitly, so agents know what to look for.

Not blocking

• Welsh replies coming back in English (0.3%). Turn drafting off for Welsh-language tickets until this is fixed. It costs little and avoids a visible failure.
• Table formatting. This is cosmetic, and agents edit every draft anyway. Fix it in the normal backlog.

Meanwhile

Keep the 8 pilot agents on the tool so we don't lose momentum.³ In the first two weeks after launch, spot-check a sample of sent billing replies.

The ask

Agree to the three conditions. Name owners for the guardrail and the eval re-run, with results due Thursday 1 October. If both pass, we confirm go on 2 October after training.