Write a PRD

PRD: AI-Drafted First Responses and Ticket Routing

Status: Draft for build planning Audience: Engineering, ML, and Support Operations leads Product: Ledgerly Support

1. Problem

Ledgerly receives about 9,000 support tickets a week, handled by 42 agents across four queues: Billing, Technical, Account access, and General. Billing is 38% of volume (about 3,400 tickets a week). Average first response is 7 hours against a target of under 2.

Two things likely drive the delay, and we don't yet know their split:

• Wait time. Tickets sit in a queue, or in the wrong queue, before anyone opens them.
• Handle time. Agents spend time writing or assembling a first reply. About 60% of tickets are already answered with one of roughly 40 saved replies, so much of this work is repetitive.

This feature has two parts. It routes each new ticket to the right queue, and it drafts a first response for the agent to review, edit, and send. Nothing is ever sent without an agent's approval.

2. Goals and non-goals

Goals 1. Bring mean first response time under 2 hours within 90 days of full rollout. 2. Route at least 92% of tickets to the correct queue on first assignment. 3. Have agents send at least 50% of drafts with no or light edits. 4. Ensure no draft or sent message contains an unreviewed refund commitment (see §5).

Non-goals - Auto-sending any message, in any queue, at any confidence level. - Resolving tickets, or taking actions such as issuing refunds, changing plans, or resetting access. - Replacing follow-up replies after the first response (later phase). - Changing queue structure or staffing.

3. Users

• Support agents (42): review, edit, approve, or discard drafts, and correct routing.
• Support-ops leads: own saved replies, policy content, thresholds, and quality review.
• Legal/compliance: consulted on the refund guardrail and audit logs.

4. Functional requirements

4.1 Routing

• R1. On ticket creation, classify into one of the four queues and assign it automatically.
• R2. Store a confidence score with each assignment. Below a threshold set by support ops (initially tuned so about 10% of tickets fall below it), assign to General with a "needs triage" tag and show the top two suggested queues.
• R3. Agents can re-route in one click and give an optional reason. Every re-route is logged as a labeled correction.
• R4. Also emit tags used by drafting: a topic label (for example, invoice question, failed payment, login lockout) and a refund-related flag.

4.2 Draft generation

• R5. A draft must be ready in the agent's ticket view within 60 seconds of ticket creation, and never block the ticket from appearing. If drafting fails or times out, the ticket appears as it does today.
• R6. Use two drafting paths:
• Saved-reply path. If a ticket matches one of the ~40 saved replies above a similarity threshold, use that reply with variables filled in (name, plan, invoice number). The wording stays as approved by support ops.
• Generated path. Otherwise, generate a draft grounded in retrieved help-center articles and similar past resolved tickets. Every factual claim about policy, pricing, or product behavior must come from retrieved content. If the content is insufficient, the draft asks the customer a clarifying question or says an agent is investigating, rather than guessing.
• R7. Each draft shows which path produced it and its sources (saved reply ID or article links) so agents can verify quickly.
• R8. Match the tone and format of the best-rated historical agent replies. Support ops maintains the style guide.

4.3 Agent workflow

• R9. Drafts appear pre-filled in the reply composer with three actions: Send (after optional edits), Discard, and Regenerate. Discarding takes an optional reason from a short list.
• R10. The system has no auto-send code path. Sending requires an authenticated agent action, and this is enforced in the sending service, not just the UI.
• R11. Log for each ticket: draft text, final sent text, edit distance, agent ID, time from ticket open to send, and discard or re-route reasons.

4.4 Account-access safeguards

• R12. Drafts for Account access tickets must never confirm whether an account exists, reveal account details, or state that access was restored or changed. They may give standard verification instructions from approved content only.

5. Refund guardrail (legal requirement)

Legal requires no automated sending of refund commitments. Because agents approve every draft, the primary control is R10. We add layered controls so the model never puts a commitment in front of an agent as if it were approved policy.

• G1. Prompt and template rules. Drafts may acknowledge a refund request and say it is being reviewed. They must not promise, imply, or estimate a refund, credit, waiver, or timeline, and no saved reply used by this feature may contain one.
• G2. Output classifier. Every draft passes a commitment detector (rules plus model) before display. If it fires, replace the offending sentence with a neutral placeholder, such as "[Agent: refund decision needed]", and tag the ticket.
• G3. Send-time check. The detector also runs on the final edited text. If an agent's own text contains a refund commitment, Send requires an explicit confirmation ("This message commits to a refund"), which is logged. Agents may make commitments in their own words. The system may not.
• G4. Audit. Retain drafts and final messages for the period Legal specifies. Support ops reviews a weekly sample of 100 refund-related tickets.
• G5. Release bar. The detector must reach at least 98% recall on a labeled set of refund-commitment phrasings, including implicit ones like "we'll take care of that charge", before Billing goes live. Legal signs off on the test set.

6. Data and evaluation

Available: two years of resolved tickets with agent replies.

Data preparation (ML lead owns) - Use the queue where a ticket was resolved as the routing label, not where it first landed, since misroutes are the problem we're fixing. Also keep the initial queue to measure the historical misroute rate. - Remove PII before training or indexing. Get security review of the vendor and hosting setup if any external model is used. - Filter out replies that are outdated (old pricing, retired features, superseded policies). Support ops flags policy change dates so the pipeline can exclude earlier replies. - Mark replies that used a saved reply. This gives the saved-reply matching set and a baseline for the other 40%. - Have support ops label about 1,500 recent tickets for refund-commitment presence, routing, and draft quality. This set doubles as the gold evaluation set.

Evaluation - Split by time, not randomly: train on the earliest ~21 months, test on the most recent 3. Random splits will overstate performance because of seasonality and policy drift. - Routing: report accuracy and per-queue precision and recall, plus confusion between Billing and General, and Account access and Technical, which are likely weak spots. - Drafts: blind human review by senior agents on a 5-point rubric (accuracy, policy correctness, tone, completeness), with automatic policy-violation checks. Compare against the original agent reply. - Slice results by queue, topic, and saved-reply versus generated path.

7. Success metrics

First task: instrument the current split of first-response time into queue wait and agent handle time. Drafting mainly cuts handle time. If wait time dominates, we also need queue-ordering changes, such as surfacing the oldest tickets first, or the 2-hour target will be missed even with high draft quality. Throughput is about 43 tickets per agent per day, so handle time matters, but it is not necessarily the bottleneck.

8. Rollout

1. Shadow mode (2–3 weeks). Run routing and drafting on live tickets without showing anything to agents. Compare with actual outcomes, set thresholds, and validate the refund detector.
2. Pilot (3 weeks). Enable routing for all tickets. Show drafts to about 12 agents across General, Technical, and Account access.
3. Expand. Enable drafts for all agents in those three queues once the pilot meets the quality targets.
4. Billing. Enable only after G5 is met and Legal has signed off. Start with 6–8 agents, then expand.

Any queue can be switched off by a feature flag within minutes. Rollback triggers: any refund-commitment incident, a policy-incorrect rate above 5% in weekly review, or a CSAT drop of more than 3 points.

9. Risks and open questions

• Automation bias. Agents may approve drafts without reading them. Mitigations: show sources, sample-audit sent messages, and track the edit rate by agent. A near-zero edit rate on generated-path drafts is a warning sign.
• Stale content. Saved replies and help articles drift from policy. Support ops needs a named owner and a review cadence.
• Language. We don't know the share of non-English tickets. Confirm before launch. If it is significant, scope it out or handle it separately.
• Legal definition. Does "refund commitment" include credits, prorations, or fee waivers? Legal to define by the end of shadow mode, since it determines G1–G3.
• Hosting. Confirm data-residency and vendor constraints for ticket content.
• Threshold ownership. Decide who can change confidence thresholds in production (proposed: support-ops lead, with ML review).

10. Ownership

• Engineering: integration with the ticketing system, sending-service enforcement (R10), logging, feature flags, latency.
• ML: data preparation, routing model, drafting pipeline, refund detector, evaluation and monitoring.
• Support ops: labeling, saved-reply and style-guide upkeep, pilot agents, weekly quality review, and thresholds.
• Legal: the refund definition, the detector test set sign-off, and audit retention.

Product Requirements Document (PRD): AI-Powered Ticket Triage and Response Draft Generation ("AutoDraft & Route")

---

1. Document Overview

---

2. Background & Problem Statement

2.1 Current State & Metrics

Ledgerly is a premier accounting software platform designed specifically for small businesses. As our customer base scales, our support infrastructure is facing severe volume pressures: * Support Team Size: 42 agents distributed across four specialized queues: Billing, Technical, Account Access, and General. * Ticket Volume: ~9,000 incoming tickets per week, with 38% (~3,420 tickets/week) falling into the Billing queue. * First Response Time (FRT): Current average is 7 hours, severely lagging behind our target of under 2 hours. * Repetitive Workflows: Approximately 60% of all incoming tickets are resolved using one of roughly 40 standard saved replies (e.g., password resets, invoice retrieval, subscription tier changes).

2.2 The Opportunity

We possess two years of historical resolved tickets, including full conversation histories and agent replies. This corpus serves as a robust training and evaluation dataset for an ML-driven system that can: 1. Accurately classify and route incoming tickets to the correct queue. 2. Automatically draft context-aware first responses using our proven saved replies and historical resolutions.

2.3 Core Constraint (Legal Compliance)

• Zero Automated Sending: Legal has mandated a strict human-in-the-loop requirement. The AI system must never autonomously send messages to customers.
• Financial Safety: The model is strictly prohibited from making automated commitments regarding refunds, credits, or billing adjustments without explicit human review and approval.

---

3. Goals and Success Metrics (KPIs)

---

4. User Personas & Workflows

4.1 Support Agent

• Role: Resolves customer queries, reviews AI drafts, edits when necessary, approves responses, and handles complex escalations.
• Workflow: Opens the ticketing dashboard, sees incoming tickets pre-sorted into their designated queue with an AI-generated draft response attached. The agent reviews the draft, clicks "Approve & Send" or modifies the text, ensuring zero financial compliance risk.

4.2 Support Operations Lead

• Role: Manages queue health, tracks SLAs, audits agent performance, and maintains the library of saved replies and routing rules.
• Workflow: Monitors routing precision dashboards, updates macro/saved reply templates, and analyzes draft acceptance/rejection feedback loops to refine prompts.

4.3 Engineering & ML Leads

• Role: Build, deploy, scale, and monitor the inference pipeline, classification models, LLM orchestration, and UI integration.

---

5. Functional Requirements

5.1 Automated Ticket Routing (Triage Engine)

• Queue Classification: Upon ticket ingestion, the ML classification service must categorize incoming text into one of four queues: `Billing`, `Technical`, `Account Access`, or `General`.
• Confidence Thresholds:
• Confidence $\ge$ 85%: Automatically assign the ticket to the predicted queue.
• Confidence < 85%: Route the ticket to a "Triage Review" sub-queue or default General queue with a visual tag flagging low classification confidence.
• Metadata Tagging: Tag tickets with sub-intent labels (e.g., `Billing: Refund Request`, `Technical: Bank Sync Failure`) to assist the response generation engine.

5.2 Response Draft Generation

• Context Window & Retrieval: The generation engine must leverage the incoming user query, customer account metadata (plan tier, active status), and relevant historical examples from the 2-year resolved ticket dataset (via RAG / vector search).
• Saved Reply Integration: The model must prioritize mapping inquiries to Ledgerly's 40 core saved replies where applicable, maintaining brand voice, accuracy, and tone consistency.
• Draft Status: Every generated response must be marked with a distinct internal status: `Draft - Pending Agent Review`. It must remain locked from customer view until explicit human sign-off.

5.3 Agent Workspace UI/UX Integration

• Side-by-Side Review: The agent UI must display the incoming customer message alongside the AI-generated draft in a clear, editable text box.
• Action Controls:
• [Approve & Send]: Immediately transmits the draft to the customer and logs the interaction.
• [Edit & Send]: Allows agents to modify text inline before sending. Edits must be logged for ML fine-tuning feedback loops.
• [Discard & Rewrite]: Clears the draft if the AI misunderstood the prompt.
• Visual Safety Badges: Clear UI warnings must appear on any ticket flagged as containing billing or refund keywords, reminding agents of compliance protocols.

---

6. Non-Functional & Safety Requirements

6.1 Legal & Financial Guardrails (Critical)

• No Auto-Dispatch: Zero API endpoints or automation rules are permitted to transition a draft state to `Sent` without a cryptographic token or database action originating from an authenticated human agent session.
• Refund Keyword Interceptor: A deterministic rule-based filter must scan all generated drafts for high-risk financial terminology (`refund`, `credit`, `waiver`, `chargeback`). If detected, the UI must highlight the section in amber and require a secondary manual checkbox confirmation from the agent before sending.

6.2 Performance & Latency

• Inference Latency: End-to-end classification and draft generation must complete within < 3.0 seconds of ticket ingestion via webhook, ensuring drafts are ready before an agent opens the ticket.
• Availability: 99.9% uptime SLA for the ML inference and routing service.

6.3 Data Privacy & Security

• PII Redaction: Customer tax IDs, banking credentials, and sensitive financial figures must be handled in compliance with GDPR, CCPA, and Ledgerly security policies. Training data pipelines must scrub extraneous PII prior to embedding generation.

---

7. Data, ML, & Training Strategy

7.1 Dataset Utilization

• Corpus: 2 years of historical resolved tickets (approx. 900,000+ interaction pairs across 42 agents).
• Data Splitting: 80% training/fine-tuning, 10% validation, 10% held-out test set reflecting recent seasonal billing spikes.

7.2 Model Architecture & Approach

• Classification: Fine-tuned encoder model (e.g., BERT/DeBERTa variant) for multi-class queue routing and intent tagging.
• Generation: Instruction-tuned LLM accessed via secure API or private VPC deployment, augmented with a vector database (RAG) containing the 40 approved saved replies and top-performing historical resolutions.
• Feedback Loop: Log all agent edits (diffs) and rejections to construct a weekly RLHF (Reinforcement Learning from Human Feedback) or supervised fine-tuning dataset to reduce hallucination rates.

---

8. High-Level Architecture & Data Flow

``` [Customer Email / Web Form] │ ▼ [Ingestion Service / Webhook] │ ├──► [ML Classification Engine] ──► Assigns Queue & Sub-intent │ └──► [RAG Vector DB (Saved Replies & History)] │ ▼ [LLM Draft Generation Service] ──► Generates Initial Response Text │ ▼ (Strictly Locked: Status = Pending Review) [Support Ticketing Database] │ ▼ [Agent UI Dashboard] ◄─── Human Agent Reviews, Edits, & Approves │ ▼ (Explicit Human Action) [Customer Dispatch API] ```

---

9. Phased Rollout Plan

• Phase 1: Shadow Mode (Weeks 1–4)
• Deploy classification and draft generation in the background.
• Log routing accuracy and draft quality without surfacing drafts to agents. Establish baseline ML accuracy metrics.
• Phase 2: Internal Pilot / Beta (Weeks 5–8)
• Roll out the UI feature to a pilot group of 5 senior Billing agents.
• Measure acceptance rates, time saved, and friction points. Refine prompt engineering and guardrails.
• Phase 3: General Availability Across All Queues (Weeks 9–12)²
• Roll out to all 42 agents across Billing, Technical, Account Access, and General queues.
• Establish continuous monitoring dashboards for Support Ops and Engineering leads.

---

10. Open Questions & Risks

1. Edge Cases in Billing: How should the model handle complex multi-invoice dispute threads where historical context spans multiple months? (Mitigation: Surface the last 3 ticket summaries alongside the draft).
2. Agent Adoption: Will agents trust the AI drafts, or will they rewrite them from scratch? (Mitigation: Emphasize time savings during training and incorporate agent feedback buttons).