Make the launch call

Recommendation: Go with Conditions

While the pilot demonstrated strong operational impact—reducing median first-response time by 41% (beating our 30% target) with a 64% acceptance rate—we failed two mandatory launch criteria:

1. Accuracy (Criterion 2): Harmful/wrong drafts hit 1.8% (9/500) versus the $\le$1.0% threshold.
2. Refund commitments (Criterion 3): Two drafts explicitly promised refunds, violating our zero-tolerance policy.

However, these failures are isolated and addressable. Six of the nine errors were caused by August reference data lacking the September 1 price changes, and both refund commitments originated in billing workflows.

Rather than cancelling Friday’s 42-agent training or delaying the productivity benefits across the entire organization, we should launch on Monday under the following three conditions:

Launch Conditions

1. Disable AI drafting on the Billing queue for Monday: Billing represents 38% of volume and contains almost all our critical failure points (pricing errors and refund risks). We will launch exclusively to the remaining three queues (62% of ticket volume), where accuracy meets our threshold. 2. Implement deterministic refund guardrails by Friday: Deploy a hard system-prompt restriction and a post-generation regex/keyword filter that blocks or drops drafts containing refund commitments before reaching the agent. 3. Update agent training on Friday: Keep Friday’s training on schedule. Explicitly train agents on: - The pilot's phased rollout (Billing queue excluded initially). - Known minor bugs (Welsh-language tickets defaulting to English; manual table reformatting required). - Strict agent accountability as the final human reviewer for every sent message.

Path to Full Billing Rollout

Engineering will ingest the post-September 1 pricing documentation and run a targeted 200-ticket eval on billing drafts next week. Once billing tickets achieve $<1\%$ error and 0 refund promises, we will enable the feature for the billing queue.

This approach protects Ledgerly’s financial liability and customer trust while immediately capturing significant speed improvements for over 60% of our support volume.

Launch Recommendation: Go with Conditions

As the Product Manager, my recommendation for Monday’s rollout to all 42 agents is Go with Conditions.

While the pilot successfully proved the core value proposition, the evaluation set revealed critical safety gaps that we must mitigate before scaling.³

---

Why We Are Ready (The "Go" Factors)

• Efficiency Gains Exceeded Target: Median time to first response dropped by 41% (from 7.1 to 4.2 hours), surpassing our 30% goal.
• High Team Adoption: 7 out of 8 pilot agents want to keep using the feature.
• Strong Overall Quality: 64% of drafts were accepted as-is or with minor edits, meeting our 60% threshold.

---

Critical Conditions for Monday's Launch

To protect our customers and brand, the following four conditions must be met prior to Monday:

1. Fix the Billing Knowledge Gap (Mandatory): * The Issue: 6 out of the 9 harmful drafts on the eval set quoted old prices because the model's training data cutoff is August⁴, and prices changed September 1st. * Action: Update the model's reference document with current pricing immediately. 2. Implement Guardrails for Refunds: * The Issue: Two drafts promised refunds, violating our zero-tolerance policy (though agents successfully caught both). * Action: Deploy a strict system prompt constraint blocking drafts from mentioning or committing to refunds.² 3. Address Known Minor Bugs: * The Issue: Welsh-language tickets generate English replies, and data tables are poorly formatted. * Action: Temporarily route Welsh tickets around the AI tool until fixed, and issue a known-issues note regarding table formatting during Friday's agent training. 4. Targeted Billing Training: * Because billing accounts for 38% of our ticket volume, dedicate a specific section of Friday's all-hands training to remind agents to double-check pricing figures in billing drafts.

---

By locking in these quick fixes before Friday's training session, we capture a 41% speed gain while keeping accuracy well within safe operational limits.¹