Write a PRD

Product Requirements Document (PRD)

Feature: AssistRoute (Automated Queue Routing & First-Response Drafting) Target Delivery: Q3 Status: Ready for Engineering Review Owners: Engineering Lead, ML Lead, Support Operations Lead

---

1. Problem Statement & Business Opportunity

Ledgerly processes 9,000 support tickets per week with a team of 42 agents across four queues: Billing (38%), Technical, Account Access, and General.

• Current FRT: 7.1 hours average first response time.
• Target FRT: < 2.0 hours.
• Manual Bottlenecks: Every incoming ticket requires manual reading, classification, and initial triage. Furthermore, approximately 60% of all initial inquiries map cleanly to one of ~40 canonical saved replies (macros), yet agents spend minutes manually searching, adapting, and populating them.

Solution Overview

AssistRoute is a two-part machine learning pipeline that acts on incoming tickets: 1. Predictive Routing: Automatically assigns incoming tickets to one of the four queues. 2. First-Response Drafting: Generates an editable, context-aware draft response pre-populated in the agent’s console before ticket open.

Core Invariant: Human-in-the-Loop (HITL). No draft is ever sent directly to a customer. Support agents retain 100% send authority.

---

2. Key Objectives & Metrics

---

3. Scope & Non-Goals

In Scope

• Asynchronous classification and routing pipeline for all newly created tickets via email and web form.
• Generation of a single personalized initial draft based on historical resolutions and the 40 standard macros.
• Guardrail pipeline strictly forbidding autonomous refund/credit commitments.
• Feedback loop telemetry (tracking agent accepts, edits, discards, and re-routes).

Out of Scope (Phase 1)

• Autonomous sending (auto-resolution without human click).
• Chat/live messaging support (limited strictly to asynchronous ticketing channels).
• Processing multi-turn responses (AssistRoute generates first responses only).
• Multi-language support (English only).

---

4. System Architecture & Workflow

``` [Customer Submits Ticket] │ ▼ [Event Ingestion: Webhook] ──▶ [PII Masking & Sanitization] │ ┌───────────────────────┴───────────────────────┐ ▼ ▼ [Routing Classifier] [Draft Generation Pipeline] │ │ Confidence $\ge$ Threshold? │ ├── Yes ──▶ Assign Queue 1. Macro Match / Retrieval └── No ──▶ Assign "General" (Flagged) 2. Prompt Compilation (LLM) │ 3. Deterministic Policy Guardrails │ │ └───────────────────────┬───────────────────────┘ ▼ [Agent Console: Pre-populated View] ├── Action A: Accept & Send ├── Action B: Edit & Send ├── Action C: Discard Draft └── Action D: Re-assign Queue ```

End-to-End Latency SLA

From `ticket.created` webhook ingestion to draft persistence in the ticketing database: $\le 5.0$ seconds (P95).

---

5. Functional Requirements

5.1 Routing Engine (ML Service)

• FR-1.1: The model must classify each ticket into one of four queues: `Billing`, `Technical`, `Account Access`, or `General`.
• FR-1.2: Confidence Scoring:
• If `confidence >= 0.85`: Automatically set the ticket's `queue_id` attribute.
• If `confidence < 0.85`: Set `queueid` to `General` and append the tag `needstriage`.
• FR-1.3: The Routing Engine must evaluate and route the ticket before generating the draft, as draft prompts require queue-specific system contexts.

5.2 Retrieval & Generation Pipeline

• FR-2.1 (Context Hydration): The generation service must pull:
• The ticket subject and body.
• The authenticated user's metadata: Plan tier (`Solo`, `Growth`, `Enterprise`), account age, and active add-on modules.
• The closest semantic match from Ledgerly's 40 standard macros.
• FR-2.2 (Draft Generation): The system must generate a friendly, concise first response that adheres to Ledgerly brand voice, incorporates the customer's name, and directly addresses the primary issue using the retrieved macro logic.
• FR-2.3 (Confidence Gating): If the model's semantic similarity score against approved knowledge bases or historical solutions falls below `0.70`, no draft shall be rendered. The UI must show: "Draft unavailable: Low context confidence."

5.3 Legal & Compliance Guardrail (Refund Protection)

* FR-3.1 (Deterministic Regex & Semantic Filter): Every draft must pass through a two-stage financial commitment filter prior to saving: 1. Lexical Check: Negative keyword blocking: `refund`, `credit`, `reimburse`, `waive fee`, `reverse charge`, `money back`, `comp`. 2. Semantic Entailment Check: Verification that the text does not commit Ledgerly to monetary compensation or liability. * FR-3.2 (Violation Action): If a draft contains a prohibited financial commitment: * The drafting engine must automatically strip the phrase and substitute safe boilerplate: "I have escalated your billing inquiry to our finance specialists for immediate review." * The draft must display a visible amber badge in the Agent UI: "Financial Review Required — No Commitments Permitted."

5.4 Support Agent Console UX

• FR-4.1 (Draft Presentation): The draft must render inside the native response editor as an uncommitted state (distinct background styling or watermarked).
• FR-4.2 (Interaction Actions):
• Tab / Single-Click Acceptance: Agent hits `Tab` or clicks `Use Draft` to copy the generated text into the active editor.
• Direct Editing: Editing the draft directly must be seamless without extra confirmation modals.
• Clear / Reject: Single-click action to clear the generated text and open a blank response field.
• FR-4.3 (Metadata Panel): A sidebar panel must show:
• Queue classification and confidence score.
• Which of the 40 macros informed the draft.
• A one-click queue reassignment dropdown.

``` +-------------------------------------------------------------------------+ | Ticket #10492: "Charged twice for Payroll add-on" | | Sender: Sarah Jenkins (Growth Tier) | Queue: [Billing (96%)] [Re-route] | +-------------------------------------------------------------------------+ | [ AI Drafted Response ] [Macro Source: Billing_Dup]| | | | Hi Sarah, | | | | Thanks for reaching out. I see that your account was debited twice | | for the Payroll add-on on October 1st. | | | | I have escalated this to our payment operations team to review the | | duplicate transaction details. We will update you here as soon as this | | review is complete. | | | | [ Accept Draft (Tab) ] [ Edit Draft ] [ Reject ] | +-------------------------------------------------------------------------+ | [ Agent Response Box ] | +-------------------------------------------------------------------------+ ```

---

6. ML & Data Engineering Specifications

6.1 Data Preparation (2-Year Historical Corpus)

1. Sanitization: Strip all historical PII (tax identifiers, SSNs, credit card numbers, passwords) using Microsoft Presidio or an equivalent NER pipeline before training/indexing. 2. Filtering: * Drop all historical tickets that required more than 4 re-routes (noisy labels). * Filter out tickets closed with negative customer satisfaction (CSAT $\le 2$). * Exclude responses superseded by outdated accounting rules or old pricing tiers (Ops team to define date cutoffs). 3. Macro Ground-Truth Alignment: Map the 40 canonical macros against historical agent responses to serve as gold-standard reference pairs.

6.2 Model Specifications

• Routing Classifier:
• Architecture: Fine-tuned lightweight encoder (e.g., `modern-bert-base` or `RoBERTa-base`) or an optimized classification endpoint.
• Input: Ticket Subject + Body.
• Output: Softmax distribution over `[Billing, Technical, Account Access, General]`.
• Latency Target: $< 200\text{ ms}$.
• Generative Drafting Model:
• Architecture: Hosted LLM (e.g., Claude 3.5 Sonnet or GPT-4o-mini) via secure enterprise API with zero-data-retention agreements.
• Prompt Design: System prompt containing Ledgerly tone guidelines, user context JSON, the selected macro instructions, and strict instructions forbidding financial promises.
• Temperature: `0.1` (low variability, high determinism).

6.3 Telemetry & Event Logging

The frontend and backend must emit the following events to the analytics lakehouse: * `ticketrouted`: `{ ticketid, predictedqueue, confidence, autoassigned: bool }` * `ticketrerouted`: `{ ticketid, oldqueue, newqueue, agentid }` * `draftgenerated`: `{ ticketid, macroid, promptversion, modelid, generationtimems }` * `draftactioned`: * `action`: `ACCEPTEDASIS` | `EDITED` | `REJECTED` * `originaldraft`: string * `finalsenttext`: string * `levenshteindistance`: int * `agentid`: string

---

7. Safety, Compliance, and Security

1. Refund Protection Hard Filter: As specified in Section 5.3, any draft containing automated refund promises is blocked from the UI by an air-gapped deterministic validation check that runs after LLM generation and cannot be bypassed by prompt engineering.
2. Zero Training on Customer Data: LLM vendors must have explicit Data Processing Agreements (DPAs) confirming customer ticket content is not used to train foundation models.
3. Data Storage & SOC 2: Masked inputs and outputs stored for model evaluation must adhere to standard data retention policies (auto-purged after 90 days).

---

8. Rollout Plan & Milestones

``` Phase 0: Offline Validation & Guardrail Auditing (Weeks 1-3) ├── Train routing model on historical data; evaluate against held-out test set. ├── Run 10,000 synthetic adversarial prompts targeting the refund guardrail. └── Success Gate: Routing accuracy > 93%, 0 adversarial refund breaches.

Phase 1: Shadow Mode (Weeks 4-5) ├── Pipeline processes live tickets; outputs are logged to Databricks/Snowflake. ├── No UI visible to agents. └── Evaluate: Model inference latency, real-time routing accuracy vs. manual triage.

Phase 2: Single-Queue Canary Pilot (Weeks 6-7) ├── Deploy exclusively to the Billing Queue (10 designated agents). ├── Daily sync with Support Operations leads to review draft quality. └── Success Gate: Draft utilization > 50%, no severe agent blockers.

Phase 3: General Availability (GA) & Iteration (Weeks 8+) ├── Enable routing and drafting across all 42 agents and 4 queues. ├── Implement weekly retraining cadence for classifier based on override logs. └── Track progress toward < 2.0 hour FRT target. ```

---

9. Risks & Mitigations

PRD: AI-Assisted First Response and Ticket Routing

Product: Ledgerly Support Assist Audience: Engineering, ML and Support Operations leads Status: Proposed MVP; thresholds below are launch targets, not measured results Accountable owner: Support Operations lead, with Engineering and ML leads owning delivery and model quality

1. Problem and objective

Ledgerly’s 42 support agents handle 9,000 tickets per week across Billing, Technical, Account access and General. Billing represents 38% of volume—approximately 3,420 tickets weekly. Average first-response time is seven hours, against a target of under two hours. Approximately 60% of tickets receive one of roughly 40 saved replies.

The feature will classify new tickets into the appropriate queue and prepare a grounded first-response draft for an agent to review. It should reduce queue-selection work and repetitive writing without delegating customer communication or financial commitments to the model.

Every outbound response requires explicit agent approval. Legal prohibits automated sending of refund commitments. The MVP will have no automated-send path and will not generate refund promises; agents must author any commitment using the existing authorized refund process.

2. Goals and success measures

Primary goals:

• Reduce average time to the first substantive human-approved response to under two hours during a staffed pilot.
• Reduce median active agent time spent preparing first responses by at least 30%.
• Route tickets accurately while avoiding hidden misroutes.
• Preserve response correctness, account security and customer satisfaction.

Measure first-response time from ticket creation to the first substantive response sent by an agent. Automated receipts do not count. Report calendar-hour and staffed-hour results separately, plus median and p90, to prevent averages hiding long waits.

Additional proposed launch and pilot thresholds:

Zero observed errors is a release gate, not proof of zero production risk. ML must report sample sizes and confidence intervals. Support Operations must confirm staffing coverage: drafting improvements alone cannot guarantee the response-time target.

3. Scope

MVP includes:

• Classification into Billing, Technical, Account access or General.
• Automatic routing only above validated, queue-specific confidence thresholds.
• Retrieval of approved saved replies and current support documentation.
• A first-response draft with internal source references, risk flags and suggested clarifying questions.
• Agent controls to edit, discard, regenerate, change queue and approve/send.
• Audit logs, monitoring and immediate disable controls.

Excluded: automatic sending, subsequent-turn assistance, ticket resolution, refunds, account changes, security verification decisions and customer-facing AI chat.

Ticket channels, languages and attachment formats are not specified. Support Operations will inventory them before implementation. MVP supports only validated text channels and languages; unsupported inputs receive normal human triage without a generated draft.

4. User workflow and functional requirements

1. Ingest: On creation of a new eligible ticket, capture its text, subject and approved metadata. Use only authorized customer/account context available to the assigned support role.
2. Assess: Predict a queue, calibrated confidence and risk flags such as refund request, account compromise or insufficient information.
3. Route: Assign high-confidence tickets to the predicted queue. Send uncertain cases to the existing manual-triage destination, provisionally General. Surface uncertainty prominently; do not treat General as a confident classification.
4. Draft: Retrieve relevant approved material and produce a concise response addressing the request. Prefer adapting an applicable saved reply over open-ended generation.
5. Review: Show the draft, suggested queue, risk flags and source links in the agent workspace. Clearly label the text “AI draft—not sent.”
6. Approve/send: The existing send action requires an explicit authenticated agent action. No background job, timeout or model output may trigger sending.
7. Learn: Record queue corrections, edits, rejection reasons and quality reviews for controlled evaluation and future retraining.

Drafts must not invent transactions, troubleshooting outcomes, refund eligibility or account status. Where evidence is missing, ask for necessary information or acknowledge that an agent must investigate. Avoid requesting passwords, full payment-card details or other unnecessary sensitive data.

Refund-related drafts may acknowledge the request and explain approved next steps, but must not promise an amount, eligibility or processing date. Flag these tickets for Billing review. Agents may add commitments only after authorized verification.

A draft becomes stale if relevant ticket content or account context changes. Disable approval until it is refreshed or explicitly reviewed against the updated context. Manual queue changes take precedence over later model results.

5. Data and ML approach

Two years of resolved tickets and agent replies are available. Historical replies are examples, not authoritative policy: outdated guidance and unauthorized commitments must not be reproduced.

ML and Support Operations will:

• Define queue labels and rules for mixed-intent tickets. Account-security concerns take precedence over routine billing questions; Support Operations must approve the complete precedence matrix.
• Audit final queue labels and sample ambiguous cases for expert adjudication.
• Remove duplicates, signatures and irrelevant quoted history; redact unnecessary personal and payment information.
• Preserve ticket/thread/customer grouping across splits to reduce leakage.
• Use a chronological training, validation and held-out test split, with the newest period reserved for testing.
• Evaluate only information available when the ticket arrived. Later replies and resolutions may provide labels but must never become runtime inputs.

Benchmark saved-reply retrieval and a conventional classifier before introducing more complex models. Select the simplest approach meeting quality, latency and operational requirements.

Generation will use current, versioned approved content. Historical replies may support offline training subject to privacy and quality approval, but must not serve as an unrestricted runtime knowledge source. Conflicting or missing sources trigger a clarification or human-investigation draft rather than a guessed answer.

Customer text and retrieved content are untrusted inputs. Instructions embedded in tickets must not change system rules, authorize actions or bypass review.

6. Engineering design and controls

Implement an asynchronous pipeline behind feature flags:

Ticket event → eligibility check → classifier → routing decision → retrieval → generation → policy validation → draft storage → agent UI.

Persist ticket ID/version, model and prompt versions, retrieved document versions, queue scores, draft state, risk flags and agent actions. Use idempotency keys to prevent duplicate processing and concurrency controls to avoid overwriting agent work.

The AI service must have no credential or permission to send messages, issue refunds or modify accounts. Routing permissions must be limited to approved queues. Outbound messages remain controlled by the existing authenticated support application.

Post-generation checks will block prohibited promises and unsupported sensitive claims. Failed checks suppress the draft and display a reason; they must not merely append a disclaimer.

Apply role-based access, encryption and existing retention rules. Any external model provider requires Security and Legal approval, including contractual restrictions on retention and training use. Log identifiers and operational metadata where possible, not unrestricted ticket bodies.

On timeout, provider outage or validation failure, preserve normal ticket handling. Keep the ticket visible, apply manual triage where necessary and show “Draft unavailable.” Never delay ticket intake while waiting for AI.

7. Evaluation and release gates

Build an adjudicated test set covering all queues, common saved-reply cases, mixed intent, sparse descriptions, refund disputes, account compromise, outdated-policy examples and prompt-injection attempts. Report overall and per-queue performance; oversampled risk cases must also be reported separately from production-weighted results.

Two support reviewers will score drafts for correctness, relevance, completeness, tone and policy compliance, with disagreements adjudicated. Distinguish cosmetic edits from substantive corrections. Track source support and appropriate abstention, not acceptance rate alone.

Required acceptance tests include:

• A refund-request draft contains no commitment.
• No model or pipeline component can invoke sending.
• Unsupported facts cause omission or escalation.
• Ticket updates invalidate stale drafts.
• Human rerouting is not overwritten.
• Provider failure leaves tickets accessible.
• Duplicate events create neither duplicate drafts nor duplicate sends.
• Agent permissions also constrain retrieved context.

Production expansion requires meeting quality thresholds and passing all safety-critical tests. Any unauthorized financial promise in evaluation blocks launch pending remediation.

8. Rollout, ownership and operations

Indicative eight-week plan:

• Weeks 1–2: Support Operations approves taxonomy, saved replies, triage rules and baseline measurements; ML audits data; Engineering completes integration and security design.
• Weeks 3–4: Build the pipeline, UI, evaluation harness and operational controls.
• Weeks 5–6: Run offline evaluation and shadow mode without changing queues or showing drafts.
• Weeks 7–8: Pilot with selected agents, then expand by queue only after gates pass.

Use a randomized holdout where practical, balanced by queue and shift, to compare handling time, response time, corrections and customer outcomes.

Engineering owns availability, access controls and rollback. ML owns evaluation, calibration and drift monitoring. Support Operations owns policy content, QA, agent training and staffing. Legal approves refund wording and provider terms.

Provide separate kill switches for routing and drafting. Suspected unauthorized commitments, data exposure or account-security mishandling trigger immediate suspension of the affected capability and incident review. Review corrections and rejected drafts weekly; retraining requires fresh holdout evaluation, not automatic ingestion of agent edits.

PRD: AI Draft & Route for Support Tickets

Sep 24, 2026 · @Martin Slaney

1. Summary and problem

Ledgerly's support team (42 agents, four queues: Billing, Technical, Account access, General) handles \~9,000 tickets a week. Average first response time (FRT) is 7 hours against a target of under 2.

Most of that delay is not writing time. It is triage and lookup<sup>1</sup>: \~60% of tickets are answered with one of \~40 saved replies, and 38% are billing. Tickets wait unassigned or in the wrong queue, then an agent spends minutes finding the right macro and filling in account details.

This feature does two things when a ticket arrives:

1. Routes it to the correct queue, with a confidence score.
2. Drafts a first response, grounded in the saved-reply library, the customer's account data and similar past resolutions, for an agent to review.

Nothing is sent without an agent approving it. On top of that, any draft containing a refund commitment is hard-gated per Legal (§4.3).

2. Goals, non-goals and success metrics

Goals

• Bring average FRT under 2 hours without adding headcount.
• Cut misrouted tickets and the reassignment delay they cause.
• Reduce agent handling time on saved-reply tickets, so agents spend more time on the \~40% that need real investigation.

Non-goals (v1)

• Auto-sending any response, of any type.
• Drafting follow-ups beyond the first response.
• Taking actions in Ledgerly (issuing refunds, resetting passwords, changing plans). The model drafts text; agents act.
• Customer-facing chatbot or deflection.

Success metrics (measured per queue, against a 4-week pre-launch baseline)

"Light edit" is defined as a normalised edit distance below 0.2 between draft and sent text. ML owns the metric definition; Support Ops signs it off before pilot.

3. Users and core workflow

Users: support agents (reviewers of every draft); queue leads (monitor routing, handle overrides); Support Ops (owns saved replies, policies and the refund-approval rota).

Flow for a new ticket

1. Ticket created (email or in-app form).
2. Router predicts queue + confidence within 30s. High confidence → assigned to that queue. Low confidence → Triage view for a lead to assign in one click.
3. Drafter produces a first response and attaches it to the ticket as an internal draft, with: the saved reply(s) it drew on, account facts it inserted, and any flags (refund, low confidence, missing data).
4. Agent opens the ticket, reviews the draft, and chooses Send, Edit & send, or Discard (with a reason code).
5. If the draft is refund-flagged, Send is replaced by Request refund approval (§4.3).
6. Final queue, sent text and action are logged as training and evaluation signal.

4. Functional requirements

4.1 Routing

• R1. Classify every new ticket into Billing, Technical, Account access or General, with a calibrated confidence score.
• R2. Auto-assign when confidence ≥ threshold (set per queue from eval, starting target: ≥ 95% precision at that threshold). Below threshold → Triage view.
• R3. Account access is the costliest miss (locked-out customers).<sup>3</sup> Tune for recall on this class; a ticket with any access signal and ambiguous classification goes to Account access, not General.
• R4. Agents can reassign in one click; every reassignment is logged with the original prediction.
• R5. Support Ops can switch routing to suggest-only per queue without a deploy.

4.2 Drafting

• D1. Generate a draft for every routed ticket in English within 60s of creation. Other languages: no draft in v1, flag only.
• D2. Retrieval first: identify the best-matching saved reply (or "none"). When one matches, the draft is that reply personalised with ticket and account context, not free text. When none matches, draft from similar resolved tickets and help-centre articles, and label it Free-form draft.
• D3. Account facts (plan, billing dates, invoice amounts, last payment status) come only from read-only lookups against Ledgerly's billing/account APIs, never from model memory. Any fact the model could not verify is left as a visible `[placeholder]` that blocks sending until filled.
• D4. Show sources inline: saved-reply ID, linked tickets/articles, API fields used.
• D5. Never promise timelines, credits, discounts, policy exceptions or refunds unless the saved reply itself contains them<sup>2</sup> (refunds additionally gated, §4.3).
• D6. Skip drafting (flag only) for: legal threats, suspected fraud, data deletion/GDPR requests, security incidents, and abusive or distressed customers. Keyword + classifier; list owned by Support Ops.

4.3 Refund guardrail (Legal requirement)

Legal requires no automated sending of refund commitments. Because every draft already needs agent approval, we go further so a refund promise cannot slip through in a routine approve click:

• G1. A dedicated refund-commitment detector runs on every draft and on the final edited text at send time (agents may add refund language themselves). Tuned for recall ≥ 99% on a Legal-reviewed test set; false positives are acceptable.
• G2. If triggered, the ticket cannot be sent via one-click approval. The agent must confirm the refund is authorised under current policy (Support Ops defines who can authorise which amounts) via a separate confirmation step that is logged.
• G3. The drafter must never generate a refund commitment from free-form reasoning; refund language may only come from approved refund saved replies.
• G4. No bulk-approve action exists anywhere in the product.
• G5. Legal reviews the detector test set and the confirmation UX before pilot, and receives a monthly log of refund-flagged sends.

Open for Legal: does "automated sending" cover a one-click approve by an agent? This PRD assumes one-click approval is not automated but adds G2 as defence in depth. Confirm before build.

4.4 Agent experience

• A1. Draft appears pre-filled in the reply box, visibly marked as AI-drafted until edited.
• A2. Discard requires a reason: wrong answer, wrong tone, missing info, wrong queue, should not be drafted.
• A3. Agents never lose their normal tools; saved replies remain available manually.
• A4. Customers are not told a draft was AI-assisted (agent authors the sent message). Support Ops to confirm this against Ledgerly's AI disclosure policy.

5. ML approach, data and evaluation

Data. Two years of resolved tickets (\~900k) with final queue, agent replies and saved-reply usage.

• Label routing from the final queue, not the initial one; tickets that were reassigned are the most valuable examples.
• Map historic replies to saved-reply IDs where possible (exact/near-match), giving a supervised "which reply fits" dataset.
• Down-weight or exclude replies older than any policy or pricing change; Support Ops supplies the change dates. Stale answers are the main quality risk.
• Strip PII before any use outside the production data boundary; confirm data-processing terms with Legal/DPO for customer ticket content.

Approach. Start with the simplest thing that hits the targets: an LLM classifier (or fine-tuned small model if cost/latency requires) for routing; retrieval over saved replies + resolved tickets + help centre, then LLM personalisation for drafting; a separate refund detector. Model choice is ML's call, constrained by the latency targets in §3 and D1 and per-ticket cost.

Offline evaluation (gate to pilot)

• Held-out set: most recent 3 months, time-split (no leakage).
• Routing: per-queue precision/recall and calibration; auto-assign coverage at the chosen threshold.
• Drafting: saved-reply selection top-1 accuracy; 500-ticket human-graded sample scored by senior agents (correct, complete, on-policy, tone) with a ≥ 80% "send with light or no edit" bar.
• Refund detector: recall ≥ 99% on Legal-reviewed set.
• Factual accuracy: 0 invented account facts in the graded sample.

Online monitoring. Dashboards per queue for acceptance, edit distance, discard reasons, reassignments, FRT, CSAT and refund flags. Weekly review of a random sample of sent drafted replies by Support Ops QA.

6. Rollout, risks and open questions

Rollout

Billing goes first: highest volume, most saved-reply coverage, and it exercises the refund guardrail early under close watch.

Risks

• Rubber-stamping. Agents approve drafts without reading them. Mitigate: placeholders block send, refund gate, QA sampling, track time-on-draft before send.
• Stale policy in drafts. Mitigate: saved replies as primary source, policy-change exclusion in training data, owner for saved-reply freshness.
• Misrouting Account access. Mitigate: recall-weighted threshold (R3), Triage fallback.
• Agent trust. Early bad drafts kill adoption. Mitigate: shadow phase, drafts only above a quality-confidence score; otherwise show the suggested saved reply only.

Open questions

1. Legal: does agent one-click approval count as "automated sending" (§4.3)?
2. Support Ops: who can authorise refunds, and at what amounts?
3. Eng: which helpdesk platform APIs support pre-filled internal drafts and blocking send?
4. DPO: approval to use historic ticket content with a model provider.
5. Support Ops: AI-assistance disclosure to customers (A4).

Owners: Eng lead (integration, guardrail enforcement, UI), ML lead (models, eval, monitoring), Support Ops lead (saved replies, policy, pilot, QA).