Write a PRD

Product Requirements Document (PRD)

Feature: AssistRoute (Automated Queue Routing & First-Response Drafting) Target Delivery: Q3 Status: Ready for Engineering Review Owners: Engineering Lead, ML Lead, Support Operations Lead

---

1. Problem Statement & Business Opportunity

Ledgerly processes 9,000 support tickets per week with a team of 42 agents across four queues: Billing (38%), Technical, Account Access, and General.

• Current FRT: 7.1 hours average first response time.
• Target FRT: < 2.0 hours.
• Manual Bottlenecks: Every incoming ticket requires manual reading, classification, and initial triage. Furthermore, approximately 60% of all initial inquiries map cleanly to one of ~40 canonical saved replies (macros), yet agents spend minutes manually searching, adapting, and populating them.

Solution Overview

AssistRoute is a two-part machine learning pipeline that acts on incoming tickets: 1. Predictive Routing: Automatically assigns incoming tickets to one of the four queues. 2. First-Response Drafting: Generates an editable, context-aware draft response pre-populated in the agent’s console before ticket open.

Core Invariant: Human-in-the-Loop (HITL). No draft is ever sent directly to a customer. Support agents retain 100% send authority.

---

2. Key Objectives & Metrics

---

3. Scope & Non-Goals

In Scope

• Asynchronous classification and routing pipeline for all newly created tickets via email and web form.
• Generation of a single personalized initial draft based on historical resolutions and the 40 standard macros.
• Guardrail pipeline strictly forbidding autonomous refund/credit commitments.
• Feedback loop telemetry (tracking agent accepts, edits, discards, and re-routes).

Out of Scope (Phase 1)

• Autonomous sending (auto-resolution without human click).
• Chat/live messaging support (limited strictly to asynchronous ticketing channels).
• Processing multi-turn responses (AssistRoute generates first responses only).
• Multi-language support (English only).

---

4. System Architecture & Workflow

``` [Customer Submits Ticket] │ ▼ [Event Ingestion: Webhook] ──▶ [PII Masking & Sanitization] │ ┌───────────────────────┴───────────────────────┐ ▼ ▼ [Routing Classifier] [Draft Generation Pipeline] │ │ Confidence $\ge$ Threshold? │ ├── Yes ──▶ Assign Queue 1. Macro Match / Retrieval └── No ──▶ Assign "General" (Flagged) 2. Prompt Compilation (LLM) │ 3. Deterministic Policy Guardrails │ │ └───────────────────────┬───────────────────────┘ ▼ [Agent Console: Pre-populated View] ├── Action A: Accept & Send ├── Action B: Edit & Send ├── Action C: Discard Draft └── Action D: Re-assign Queue ```

End-to-End Latency SLA

From `ticket.created` webhook ingestion to draft persistence in the ticketing database: $\le 5.0$ seconds (P95).

---

5. Functional Requirements

5.1 Routing Engine (ML Service)

• FR-1.1: The model must classify each ticket into one of four queues: `Billing`, `Technical`, `Account Access`, or `General`.
• FR-1.2: Confidence Scoring:
• If `confidence >= 0.85`: Automatically set the ticket's `queue_id` attribute.
• If `confidence < 0.85`: Set `queueid` to `General` and append the tag `needstriage`.
• FR-1.3: The Routing Engine must evaluate and route the ticket before generating the draft, as draft prompts require queue-specific system contexts.

5.2 Retrieval & Generation Pipeline

• FR-2.1 (Context Hydration): The generation service must pull:
• The ticket subject and body.
• The authenticated user's metadata: Plan tier (`Solo`, `Growth`, `Enterprise`), account age, and active add-on modules.
• The closest semantic match from Ledgerly's 40 standard macros.
• FR-2.2 (Draft Generation): The system must generate a friendly, concise first response that adheres to Ledgerly brand voice, incorporates the customer's name, and directly addresses the primary issue using the retrieved macro logic.
• FR-2.3 (Confidence Gating): If the model's semantic similarity score against approved knowledge bases or historical solutions falls below `0.70`, no draft shall be rendered. The UI must show: "Draft unavailable: Low context confidence."

5.3 Legal & Compliance Guardrail (Refund Protection)

* FR-3.1 (Deterministic Regex & Semantic Filter): Every draft must pass through a two-stage financial commitment filter prior to saving: 1. Lexical Check: Negative keyword blocking: `refund`, `credit`, `reimburse`, `waive fee`, `reverse charge`, `money back`, `comp`. 2. Semantic Entailment Check: Verification that the text does not commit Ledgerly to monetary compensation or liability. * FR-3.2 (Violation Action): If a draft contains a prohibited financial commitment: * The drafting engine must automatically strip the phrase and substitute safe boilerplate: "I have escalated your billing inquiry to our finance specialists for immediate review." * The draft must display a visible amber badge in the Agent UI: "Financial Review Required — No Commitments Permitted."

5.4 Support Agent Console UX

• FR-4.1 (Draft Presentation): The draft must render inside the native response editor as an uncommitted state (distinct background styling or watermarked).
• FR-4.2 (Interaction Actions):
• Tab / Single-Click Acceptance: Agent hits `Tab` or clicks `Use Draft` to copy the generated text into the active editor.
• Direct Editing: Editing the draft directly must be seamless without extra confirmation modals.
• Clear / Reject: Single-click action to clear the generated text and open a blank response field.
• FR-4.3 (Metadata Panel): A sidebar panel must show:
• Queue classification and confidence score.
• Which of the 40 macros informed the draft.
• A one-click queue reassignment dropdown.

``` +-------------------------------------------------------------------------+ | Ticket #10492: "Charged twice for Payroll add-on" | | Sender: Sarah Jenkins (Growth Tier) | Queue: [Billing (96%)] [Re-route] | +-------------------------------------------------------------------------+ | [ AI Drafted Response ] [Macro Source: Billing_Dup]| | | | Hi Sarah, | | | | Thanks for reaching out. I see that your account was debited twice | | for the Payroll add-on on October 1st. | | | | I have escalated this to our payment operations team to review the | | duplicate transaction details. We will update you here as soon as this | | review is complete. | | | | [ Accept Draft (Tab) ] [ Edit Draft ] [ Reject ] | +-------------------------------------------------------------------------+ | [ Agent Response Box ] | +-------------------------------------------------------------------------+ ```

---

6. ML & Data Engineering Specifications

6.1 Data Preparation (2-Year Historical Corpus)

1. Sanitization: Strip all historical PII (tax identifiers, SSNs, credit card numbers, passwords) using Microsoft Presidio or an equivalent NER pipeline before training/indexing. 2. Filtering: * Drop all historical tickets that required more than 4 re-routes (noisy labels). * Filter out tickets closed with negative customer satisfaction (CSAT $\le 2$). * Exclude responses superseded by outdated accounting rules or old pricing tiers (Ops team to define date cutoffs). 3. Macro Ground-Truth Alignment: Map the 40 canonical macros against historical agent responses to serve as gold-standard reference pairs.

6.2 Model Specifications

• Routing Classifier:
• Architecture: Fine-tuned lightweight encoder (e.g., `modern-bert-base` or `RoBERTa-base`) or an optimized classification endpoint.
• Input: Ticket Subject + Body.
• Output: Softmax distribution over `[Billing, Technical, Account Access, General]`.
• Latency Target: $< 200\text{ ms}$.
• Generative Drafting Model:
• Architecture: Hosted LLM (e.g., Claude 3.5 Sonnet or GPT-4o-mini) via secure enterprise API with zero-data-retention agreements.
• Prompt Design: System prompt containing Ledgerly tone guidelines, user context JSON, the selected macro instructions, and strict instructions forbidding financial promises.
• Temperature: `0.1` (low variability, high determinism).

6.3 Telemetry & Event Logging

The frontend and backend must emit the following events to the analytics lakehouse: * `ticketrouted`: `{ ticketid, predictedqueue, confidence, autoassigned: bool }` * `ticketrerouted`: `{ ticketid, oldqueue, newqueue, agentid }` * `draftgenerated`: `{ ticketid, macroid, promptversion, modelid, generationtimems }` * `draftactioned`: * `action`: `ACCEPTEDASIS` | `EDITED` | `REJECTED` * `originaldraft`: string * `finalsenttext`: string * `levenshteindistance`: int * `agentid`: string

---

7. Safety, Compliance, and Security

1. Refund Protection Hard Filter: As specified in Section 5.3, any draft containing automated refund promises is blocked from the UI by an air-gapped deterministic validation check that runs after LLM generation and cannot be bypassed by prompt engineering.
2. Zero Training on Customer Data: LLM vendors must have explicit Data Processing Agreements (DPAs) confirming customer ticket content is not used to train foundation models.
3. Data Storage & SOC 2: Masked inputs and outputs stored for model evaluation must adhere to standard data retention policies (auto-purged after 90 days).

---

8. Rollout Plan & Milestones

``` Phase 0: Offline Validation & Guardrail Auditing (Weeks 1-3) ├── Train routing model on historical data; evaluate against held-out test set. ├── Run 10,000 synthetic adversarial prompts targeting the refund guardrail. └── Success Gate: Routing accuracy > 93%, 0 adversarial refund breaches.

Phase 1: Shadow Mode (Weeks 4-5) ├── Pipeline processes live tickets; outputs are logged to Databricks/Snowflake. ├── No UI visible to agents. └── Evaluate: Model inference latency, real-time routing accuracy vs. manual triage.

Phase 2: Single-Queue Canary Pilot (Weeks 6-7) ├── Deploy exclusively to the Billing Queue (10 designated agents). ├── Daily sync with Support Operations leads to review draft quality. └── Success Gate: Draft utilization > 50%, no severe agent blockers.

Phase 3: General Availability (GA) & Iteration (Weeks 8+) ├── Enable routing and drafting across all 42 agents and 4 queues. ├── Implement weekly retraining cadence for classifier based on override logs. └── Track progress toward < 2.0 hour FRT target. ```

---

9. Risks & Mitigations

PRD: AI-Assisted First Response and Ticket Routing

Product: Ledgerly Support Assist Audience: Engineering, ML and Support Operations leads Status: Proposed MVP; thresholds below are launch targets, not measured results Accountable owner: Support Operations lead, with Engineering and ML leads owning delivery and model quality

1. Problem and objective

Ledgerly’s 42 support agents handle 9,000 tickets per week across Billing, Technical, Account access and General. Billing represents 38% of volume—approximately 3,420 tickets weekly. Average first-response time is seven hours, against a target of under two hours. Approximately 60% of tickets receive one of roughly 40 saved replies.

The feature will classify new tickets into the appropriate queue and prepare a grounded first-response draft for an agent to review. It should reduce queue-selection work and repetitive writing without delegating customer communication or financial commitments to the model.

Every outbound response requires explicit agent approval. Legal prohibits automated sending of refund commitments. The MVP will have no automated-send path and will not generate refund promises; agents must author any commitment using the existing authorized refund process.

2. Goals and success measures

Primary goals:

• Reduce average time to the first substantive human-approved response to under two hours during a staffed pilot.
• Reduce median active agent time spent preparing first responses by at least 30%.
• Route tickets accurately while avoiding hidden misroutes.
• Preserve response correctness, account security and customer satisfaction.

Measure first-response time from ticket creation to the first substantive response sent by an agent. Automated receipts do not count. Report calendar-hour and staffed-hour results separately, plus median and p90, to prevent averages hiding long waits.

Additional proposed launch and pilot thresholds:

Zero observed errors is a release gate, not proof of zero production risk. ML must report sample sizes and confidence intervals. Support Operations must confirm staffing coverage: drafting improvements alone cannot guarantee the response-time target.

3. Scope

MVP includes:

• Classification into Billing, Technical, Account access or General.
• Automatic routing only above validated, queue-specific confidence thresholds.
• Retrieval of approved saved replies and current support documentation.
• A first-response draft with internal source references, risk flags and suggested clarifying questions.
• Agent controls to edit, discard, regenerate, change queue and approve/send.
• Audit logs, monitoring and immediate disable controls.

Excluded: automatic sending, subsequent-turn assistance, ticket resolution, refunds, account changes, security verification decisions and customer-facing AI chat.

Ticket channels, languages and attachment formats are not specified. Support Operations will inventory them before implementation. MVP supports only validated text channels and languages; unsupported inputs receive normal human triage without a generated draft.

4. User workflow and functional requirements

1. Ingest: On creation of a new eligible ticket, capture its text, subject and approved metadata. Use only authorized customer/account context available to the assigned support role.
2. Assess: Predict a queue, calibrated confidence and risk flags such as refund request, account compromise or insufficient information.
3. Route: Assign high-confidence tickets to the predicted queue. Send uncertain cases to the existing manual-triage destination, provisionally General. Surface uncertainty prominently; do not treat General as a confident classification.
4. Draft: Retrieve relevant approved material and produce a concise response addressing the request. Prefer adapting an applicable saved reply over open-ended generation.
5. Review: Show the draft, suggested queue, risk flags and source links in the agent workspace. Clearly label the text “AI draft—not sent.”
6. Approve/send: The existing send action requires an explicit authenticated agent action. No background job, timeout or model output may trigger sending.
7. Learn: Record queue corrections, edits, rejection reasons and quality reviews for controlled evaluation and future retraining.

Drafts must not invent transactions, troubleshooting outcomes, refund eligibility or account status. Where evidence is missing, ask for necessary information or acknowledge that an agent must investigate. Avoid requesting passwords, full payment-card details or other unnecessary sensitive data.

Refund-related drafts may acknowledge the request and explain approved next steps, but must not promise an amount, eligibility or processing date. Flag these tickets for Billing review. Agents may add commitments only after authorized verification.

A draft becomes stale if relevant ticket content or account context changes. Disable approval until it is refreshed or explicitly reviewed against the updated context. Manual queue changes take precedence over later model results.

5. Data and ML approach

Two years of resolved tickets and agent replies are available. Historical replies are examples, not authoritative policy: outdated guidance and unauthorized commitments must not be reproduced.

ML and Support Operations will:

• Define queue labels and rules for mixed-intent tickets. Account-security concerns take precedence over routine billing questions; Support Operations must approve the complete precedence matrix.
• Audit final queue labels and sample ambiguous cases for expert adjudication.
• Remove duplicates, signatures and irrelevant quoted history; redact unnecessary personal and payment information.
• Preserve ticket/thread/customer grouping across splits to reduce leakage.
• Use a chronological training, validation and held-out test split, with the newest period reserved for testing.
• Evaluate only information available when the ticket arrived. Later replies and resolutions may provide labels but must never become runtime inputs.

Benchmark saved-reply retrieval and a conventional classifier before introducing more complex models. Select the simplest approach meeting quality, latency and operational requirements.

Generation will use current, versioned approved content. Historical replies may support offline training subject to privacy and quality approval, but must not serve as an unrestricted runtime knowledge source. Conflicting or missing sources trigger a clarification or human-investigation draft rather than a guessed answer.

Customer text and retrieved content are untrusted inputs. Instructions embedded in tickets must not change system rules, authorize actions or bypass review.

6. Engineering design and controls

Implement an asynchronous pipeline behind feature flags:

Ticket event → eligibility check → classifier → routing decision → retrieval → generation → policy validation → draft storage → agent UI.

Persist ticket ID/version, model and prompt versions, retrieved document versions, queue scores, draft state, risk flags and agent actions. Use idempotency keys to prevent duplicate processing and concurrency controls to avoid overwriting agent work.

The AI service must have no credential or permission to send messages, issue refunds or modify accounts. Routing permissions must be limited to approved queues. Outbound messages remain controlled by the existing authenticated support application.

Post-generation checks will block prohibited promises and unsupported sensitive claims. Failed checks suppress the draft and display a reason; they must not merely append a disclaimer.

Apply role-based access, encryption and existing retention rules. Any external model provider requires Security and Legal approval, including contractual restrictions on retention and training use. Log identifiers and operational metadata where possible, not unrestricted ticket bodies.

On timeout, provider outage or validation failure, preserve normal ticket handling. Keep the ticket visible, apply manual triage where necessary and show “Draft unavailable.” Never delay ticket intake while waiting for AI.

7. Evaluation and release gates

Build an adjudicated test set covering all queues, common saved-reply cases, mixed intent, sparse descriptions, refund disputes, account compromise, outdated-policy examples and prompt-injection attempts. Report overall and per-queue performance; oversampled risk cases must also be reported separately from production-weighted results.

Two support reviewers will score drafts for correctness, relevance, completeness, tone and policy compliance, with disagreements adjudicated. Distinguish cosmetic edits from substantive corrections. Track source support and appropriate abstention, not acceptance rate alone.

Required acceptance tests include:

• A refund-request draft contains no commitment.
• No model or pipeline component can invoke sending.
• Unsupported facts cause omission or escalation.
• Ticket updates invalidate stale drafts.
• Human rerouting is not overwritten.
• Provider failure leaves tickets accessible.
• Duplicate events create neither duplicate drafts nor duplicate sends.
• Agent permissions also constrain retrieved context.

Production expansion requires meeting quality thresholds and passing all safety-critical tests. Any unauthorized financial promise in evaluation blocks launch pending remediation.

8. Rollout, ownership and operations

Indicative eight-week plan:

• Weeks 1–2: Support Operations approves taxonomy, saved replies, triage rules and baseline measurements; ML audits data; Engineering completes integration and security design.
• Weeks 3–4: Build the pipeline, UI, evaluation harness and operational controls.
• Weeks 5–6: Run offline evaluation and shadow mode without changing queues or showing drafts.
• Weeks 7–8: Pilot with selected agents, then expand by queue only after gates pass.

Use a randomized holdout where practical, balanced by queue and shift, to compare handling time, response time, corrections and customer outcomes.

Engineering owns availability, access controls and rollback. ML owns evaluation, calibration and drift monitoring. Support Operations owns policy content, QA, agent training and staffing. Legal approves refund wording and provider terms.

Provide separate kill switches for routing and drafting. Suspected unauthorized commitments, data exposure or account-security mishandling trigger immediate suspension of the affected capability and incident review. Review corrections and rejected drafts weekly; retraining requires fresh holdout evaluation, not automatic ingestion of agent edits.

Product Requirements Document (PRD): AI-Powered Ticket Triage and Response Draft Generation ("AutoDraft & Route")

---

1. Document Overview

---

2. Background & Problem Statement

2.1 Current State & Metrics

Ledgerly is a premier accounting software platform designed specifically for small businesses. As our customer base scales, our support infrastructure is facing severe volume pressures: * Support Team Size: 42 agents distributed across four specialized queues: Billing, Technical, Account Access, and General. * Ticket Volume: ~9,000 incoming tickets per week, with 38% (~3,420 tickets/week) falling into the Billing queue. * First Response Time (FRT): Current average is 7 hours, severely lagging behind our target of under 2 hours. * Repetitive Workflows: Approximately 60% of all incoming tickets are resolved using one of roughly 40 standard saved replies (e.g., password resets, invoice retrieval, subscription tier changes).

2.2 The Opportunity

We possess two years of historical resolved tickets, including full conversation histories and agent replies. This corpus serves as a robust training and evaluation dataset for an ML-driven system that can: 1. Accurately classify and route incoming tickets to the correct queue. 2. Automatically draft context-aware first responses using our proven saved replies and historical resolutions.

2.3 Core Constraint (Legal Compliance)

• Zero Automated Sending: Legal has mandated a strict human-in-the-loop requirement. The AI system must never autonomously send messages to customers.
• Financial Safety: The model is strictly prohibited from making automated commitments regarding refunds, credits, or billing adjustments without explicit human review and approval.

---

3. Goals and Success Metrics (KPIs)

---

4. User Personas & Workflows

4.1 Support Agent

• Role: Resolves customer queries, reviews AI drafts, edits when necessary, approves responses, and handles complex escalations.
• Workflow: Opens the ticketing dashboard, sees incoming tickets pre-sorted into their designated queue with an AI-generated draft response attached. The agent reviews the draft, clicks "Approve & Send" or modifies the text, ensuring zero financial compliance risk.

4.2 Support Operations Lead

• Role: Manages queue health, tracks SLAs, audits agent performance, and maintains the library of saved replies and routing rules.
• Workflow: Monitors routing precision dashboards, updates macro/saved reply templates, and analyzes draft acceptance/rejection feedback loops to refine prompts.

4.3 Engineering & ML Leads

• Role: Build, deploy, scale, and monitor the inference pipeline, classification models, LLM orchestration, and UI integration.

---

5. Functional Requirements

5.1 Automated Ticket Routing (Triage Engine)

• Queue Classification: Upon ticket ingestion, the ML classification service must categorize incoming text into one of four queues: `Billing`, `Technical`, `Account Access`, or `General`.
• Confidence Thresholds:
• Confidence $\ge$ 85%: Automatically assign the ticket to the predicted queue.
• Confidence < 85%: Route the ticket to a "Triage Review" sub-queue or default General queue with a visual tag flagging low classification confidence.
• Metadata Tagging: Tag tickets with sub-intent labels (e.g., `Billing: Refund Request`, `Technical: Bank Sync Failure`) to assist the response generation engine.

5.2 Response Draft Generation

• Context Window & Retrieval: The generation engine must leverage the incoming user query, customer account metadata (plan tier, active status), and relevant historical examples from the 2-year resolved ticket dataset (via RAG / vector search).
• Saved Reply Integration: The model must prioritize mapping inquiries to Ledgerly's 40 core saved replies where applicable, maintaining brand voice, accuracy, and tone consistency.
• Draft Status: Every generated response must be marked with a distinct internal status: `Draft - Pending Agent Review`. It must remain locked from customer view until explicit human sign-off.

5.3 Agent Workspace UI/UX Integration

• Side-by-Side Review: The agent UI must display the incoming customer message alongside the AI-generated draft in a clear, editable text box.
• Action Controls:
• [Approve & Send]: Immediately transmits the draft to the customer and logs the interaction.
• [Edit & Send]: Allows agents to modify text inline before sending. Edits must be logged for ML fine-tuning feedback loops.
• [Discard & Rewrite]: Clears the draft if the AI misunderstood the prompt.
• Visual Safety Badges: Clear UI warnings must appear on any ticket flagged as containing billing or refund keywords, reminding agents of compliance protocols.

---

6. Non-Functional & Safety Requirements

6.1 Legal & Financial Guardrails (Critical)

• No Auto-Dispatch: Zero API endpoints or automation rules are permitted to transition a draft state to `Sent` without a cryptographic token or database action originating from an authenticated human agent session.
• Refund Keyword Interceptor: A deterministic rule-based filter must scan all generated drafts for high-risk financial terminology (`refund`, `credit`, `waiver`, `chargeback`). If detected, the UI must highlight the section in amber and require a secondary manual checkbox confirmation from the agent before sending.

6.2 Performance & Latency

• Inference Latency: End-to-end classification and draft generation must complete within < 3.0 seconds of ticket ingestion via webhook, ensuring drafts are ready before an agent opens the ticket.
• Availability: 99.9% uptime SLA for the ML inference and routing service.

6.3 Data Privacy & Security

• PII Redaction: Customer tax IDs, banking credentials, and sensitive financial figures must be handled in compliance with GDPR, CCPA, and Ledgerly security policies. Training data pipelines must scrub extraneous PII prior to embedding generation.

---

7. Data, ML, & Training Strategy

7.1 Dataset Utilization

• Corpus: 2 years of historical resolved tickets (approx. 900,000+ interaction pairs across 42 agents).
• Data Splitting: 80% training/fine-tuning, 10% validation, 10% held-out test set reflecting recent seasonal billing spikes.

7.2 Model Architecture & Approach

• Classification: Fine-tuned encoder model (e.g., BERT/DeBERTa variant) for multi-class queue routing and intent tagging.
• Generation: Instruction-tuned LLM accessed via secure API or private VPC deployment, augmented with a vector database (RAG) containing the 40 approved saved replies and top-performing historical resolutions.
• Feedback Loop: Log all agent edits (diffs) and rejections to construct a weekly RLHF (Reinforcement Learning from Human Feedback) or supervised fine-tuning dataset to reduce hallucination rates.

---

8. High-Level Architecture & Data Flow

``` [Customer Email / Web Form] │ ▼ [Ingestion Service / Webhook] │ ├──► [ML Classification Engine] ──► Assigns Queue & Sub-intent │ └──► [RAG Vector DB (Saved Replies & History)] │ ▼ [LLM Draft Generation Service] ──► Generates Initial Response Text │ ▼ (Strictly Locked: Status = Pending Review) [Support Ticketing Database] │ ▼ [Agent UI Dashboard] ◄─── Human Agent Reviews, Edits, & Approves │ ▼ (Explicit Human Action) [Customer Dispatch API] ```

---

9. Phased Rollout Plan

• Phase 1: Shadow Mode (Weeks 1–4)
• Deploy classification and draft generation in the background.
• Log routing accuracy and draft quality without surfacing drafts to agents. Establish baseline ML accuracy metrics.
• Phase 2: Internal Pilot / Beta (Weeks 5–8)
• Roll out the UI feature to a pilot group of 5 senior Billing agents.
• Measure acceptance rates, time saved, and friction points. Refine prompt engineering and guardrails.
• Phase 3: General Availability Across All Queues (Weeks 9–12)<sup>2</sup>
• Roll out to all 42 agents across Billing, Technical, Account Access, and General queues.
• Establish continuous monitoring dashboards for Support Ops and Engineering leads.

---

10. Open Questions & Risks

1. Edge Cases in Billing: How should the model handle complex multi-invoice dispute threads where historical context spans multiple months? (Mitigation: Surface the last 3 ticket summaries alongside the draft).
2. Agent Adoption: Will agents trust the AI drafts, or will they rewrite them from scratch? (Mitigation: Emphasize time savings during training and incorporate agent feedback buttons).