Make the launch call

Recommendation: Go with conditions

Not a clean go. Two of the four agreed launch criteria are currently failed, and I don't think we should waive criteria we set ourselves.

The value is real: faster responses, good acceptance, and 7 of 8 pilot agents want to keep it. Both failures also have identifiable causes.

• Wrong drafts: Six of the nine errors come from stale pricing, because the reference data stops in August. Fixing that should bring us to about 3/500 (0.6%), but that is a projection until we re-run the eval.
• Refunds: Agents caught both drafts, but the criterion says no draft, and with 42 agents we shouldn't rely on catching them by habit.

Conditions for Monday

1. Refresh the reference data with post-1 September pricing and re-run the full 500-ticket eval. It must come in at 1% or below (5 or fewer wrong).
2. Add a refund guardrail that blocks or flags commitment language such as "we'll refund." It must show zero refund commitments on the re-run plus a targeted set of refund-request tickets.
3. Kill switch and flag button. Ops can disable the feature instantly, and agents can flag bad drafts in one click.

Fallback

• If condition 2 fails, no-go for Monday.
• If only condition 1 fails, launch on the three non-billing queues (62% of tickets) and keep billing off until it passes.
• Friday training goes ahead for all 42 agents either way. It should cover the refund rule and checking prices before sending.

Known issues, not blockers

• Welsh tickets (~0.3%) get English replies. Agents will write these manually, and we'll suppress drafts for them if detection is feasible.
• Table formatting is cosmetic, since agents edit every draft. We'll fix it after launch.

After launch

• Sample wrong-draft rates weekly for the first month.
• Track how often agents edit or discard drafts, as a check on whether review discipline slips.
• Ask the one pilot agent who didn't want to keep it why.

I'd like the eval re-run results by Thursday so we can make the final call before Friday's training.

Recommendation: Go with conditions; otherwise no-go for Monday.

The pilot met two launch criteria: 64% of drafts were accepted as-is or with minor edits, above the 60% target, and median time to first response fell 41%, exceeding the 30% target. Seven of eight agents also want to keep the feature.

But the launch does not currently meet the safety criteria. The eval found harmful drafts in 1.8% of cases, above the 1% ceiling. Six errors came from outdated billing prices—a significant exposure because billing handles 38% of tickets. And two drafts promised refunds, directly violating the “no draft may commit to a refund” criterion, even though agents caught them.

Before enabling drafts broadly, require: - Updated billing reference data and a rerun of the 500-ticket eval with no more than five wrong or harmful drafts. - A reliable safeguard against refund commitments, verified to produce zero such drafts. - Welsh-language tickets routed away from AI drafting until that issue is fixed.

If those gates pass before Monday, proceed with the launch and monitor closely. If they do not, delay the launch rather than rely on agents to catch known failure modes. The formatting issue should also be addressed or clearly covered in agent training, but it is not the main launch blocker.

Recommendation: no-go for Monday’s rollout to all 42 agents. Reassess once the failed safety gates have been fixed and retested.

The pilot demonstrates value: 64% of drafts were accepted as-is or with minor edits, exceeding the 60% target. Median first-response time fell 41%, exceeding the 30% target, although the before-and-after comparison does not establish causality. Seven of eight agents want to keep it.

But we failed two explicit launch criteria:

• Wrong or harmful drafts: 1.8% against a 1% ceiling. Nine failures is above the maximum five permitted on the 500-ticket set.
• No refund commitments: failed. Two drafts promised refunds. Agents catching them prevented those messages reaching customers; it does not satisfy a criterion about what drafts may say. Human review was already part of the agreed design.

Six errors used outdated billing prices. That gives us a concrete fix, but billing represents 38% of tickets, so this is a substantial exposure. Updating the reference data alone would not establish that the remaining errors or refund promises are resolved.

Required before reconsidering launch:

1. Engineering and the billing policy owner: update and verify current prices and policies, with a named owner and process for keeping reference data current. Investigate the other three harmful drafts.
2. Engineering: add and test a control that prevents refund commitments from appearing in agent-facing drafts, including paraphrases.
3. PM and evaluation owner: rerun the full 500-ticket evaluation after the fixes, meeting the original thresholds, and add targeted pricing and refund tests. Report those additional results separately.

Excluding billing is a possible narrower release, not an automatic pass: we lack queue-level results and do not know whether refund failures are confined to billing. It would require explicit scope approval and validation of both routing and the remaining queues.

Welsh-language tickets should bypass drafting until language handling is fixed. Disable tables or use plain text pending the formatting fix.

Keep Friday’s training as preparation, with launch clearly marked as pending. The booking should not determine readiness. After the gates pass, expand in stages with monitoring and a named owner able to disable drafting.