Make the launch call

Recommendation: No-go for the all-agent launch on Monday

The pilot demonstrates value, but we have not met the agreed safety gates. Agent review is a safeguard—not a substitute for those gates.

Seven of eight pilot agents want to keep the feature, which supports its usefulness. However, catching the refund promises before sending does not satisfy the requirement that no draft make that commitment.

What must change before launch

• Refresh billing reference data with current prices and verify retrieval uses it. Six of nine harmful drafts used old prices, suggesting a concentrated, fixable problem—but we must also investigate the remaining three.
• Enforce the refund restriction, with testing that shows prohibited commitments are blocked before drafts reach agents. Agent training alone is insufficient.
• Rerun a representative 500-ticket evaluation after fixes. Require no more than five wrong or harmful drafts and zero refund commitments; explicitly cover current pricing and refund requests.
• Handle known quality defects: route Welsh-language tickets to manual drafting until language support is verified; suppress tables or fix their formatting.

Monday plan

Keep Friday’s training, but frame it as readiness training, not a launch commitment. Until safety fixes are verified, pause AI drafting for billing and use manual replies there.

Billing represents 38% of tickets, so excluding it is a material scope change. Any continued non-billing pilot should be explicitly approved, safety-validated, monitored, and reversible—not presented as meeting the original launch criteria. Reassess full rollout once the gates pass.

Launch Recommendation: Go with Conditions

As the Product Manager, my recommendation for Monday’s rollout to all 42 agents is Go with Conditions.

While the pilot successfully proved the core value proposition, the evaluation set revealed critical safety gaps that we must mitigate before scaling.³

---

Why We Are Ready (The "Go" Factors)

• Efficiency Gains Exceeded Target: Median time to first response dropped by 41% (from 7.1 to 4.2 hours), surpassing our 30% goal.
• High Team Adoption: 7 out of 8 pilot agents want to keep using the feature.
• Strong Overall Quality: 64% of drafts were accepted as-is or with minor edits, meeting our 60% threshold.

---

Critical Conditions for Monday's Launch

To protect our customers and brand, the following four conditions must be met prior to Monday:

1. Fix the Billing Knowledge Gap (Mandatory): * The Issue: 6 out of the 9 harmful drafts on the eval set quoted old prices because the model's training data cutoff is August⁴, and prices changed September 1st. * Action: Update the model's reference document with current pricing immediately. 2. Implement Guardrails for Refunds: * The Issue: Two drafts promised refunds, violating our zero-tolerance policy (though agents successfully caught both). * Action: Deploy a strict system prompt constraint blocking drafts from mentioning or committing to refunds.² 3. Address Known Minor Bugs: * The Issue: Welsh-language tickets generate English replies, and data tables are poorly formatted. * Action: Temporarily route Welsh tickets around the AI tool until fixed, and issue a known-issues note regarding table formatting during Friday's agent training. 4. Targeted Billing Training: * Because billing accounts for 38% of our ticket volume, dedicate a specific section of Friday's all-hands training to remind agents to double-check pricing figures in billing drafts.

---

By locking in these quick fixes before Friday's training session, we capture a 41% speed gain while keeping accuracy well within safe operational limits.¹

Recommendation: no-go for Monday’s rollout to all 42 agents. Reassess once the failed safety gates have been fixed and retested.

The pilot demonstrates value: 64% of drafts were accepted as-is or with minor edits, exceeding the 60% target. Median first-response time fell 41%, exceeding the 30% target, although the before-and-after comparison does not establish causality. Seven of eight agents want to keep it.

But we failed two explicit launch criteria:

• Wrong or harmful drafts: 1.8% against a 1% ceiling. Nine failures is above the maximum five permitted on the 500-ticket set.
• No refund commitments: failed. Two drafts promised refunds. Agents catching them prevented those messages reaching customers; it does not satisfy a criterion about what drafts may say. Human review was already part of the agreed design.

Six errors used outdated billing prices. That gives us a concrete fix, but billing represents 38% of tickets, so this is a substantial exposure. Updating the reference data alone would not establish that the remaining errors or refund promises are resolved.

Required before reconsidering launch:

1. Engineering and the billing policy owner: update and verify current prices and policies, with a named owner and process for keeping reference data current. Investigate the other three harmful drafts.
2. Engineering: add and test a control that prevents refund commitments from appearing in agent-facing drafts, including paraphrases.
3. PM and evaluation owner: rerun the full 500-ticket evaluation after the fixes, meeting the original thresholds, and add targeted pricing and refund tests. Report those additional results separately.

Excluding billing is a possible narrower release, not an automatic pass: we lack queue-level results and do not know whether refund failures are confined to billing. It would require explicit scope approval and validation of both routing and the remaining queues.

Welsh-language tickets should bypass drafting until language handling is fixed. Disable tables or use plain text pending the formatting fix.

Keep Friday’s training as preparation, with launch clearly marked as pending. The booking should not determine readiness. After the gates pass, expand in stages with monitoring and a named owner able to disable drafting.