Write a PRD

Product Requirements Document (PRD)

Feature: AssistRoute (Automated Queue Routing & First-Response Drafting) Target Delivery: Q3 Status: Ready for Engineering Review Owners: Engineering Lead, ML Lead, Support Operations Lead

---

1. Problem Statement & Business Opportunity

Ledgerly processes 9,000 support tickets per week with a team of 42 agents across four queues: Billing (38%), Technical, Account Access, and General.

• Current FRT: 7.1 hours average first response time.
• Target FRT: < 2.0 hours.
• Manual Bottlenecks: Every incoming ticket requires manual reading, classification, and initial triage. Furthermore, approximately 60% of all initial inquiries map cleanly to one of ~40 canonical saved replies (macros), yet agents spend minutes manually searching, adapting, and populating them.

Solution Overview

AssistRoute is a two-part machine learning pipeline that acts on incoming tickets: 1. Predictive Routing: Automatically assigns incoming tickets to one of the four queues. 2. First-Response Drafting: Generates an editable, context-aware draft response pre-populated in the agent’s console before ticket open.

Core Invariant: Human-in-the-Loop (HITL). No draft is ever sent directly to a customer. Support agents retain 100% send authority.

---

2. Key Objectives & Metrics

---

3. Scope & Non-Goals

In Scope

• Asynchronous classification and routing pipeline for all newly created tickets via email and web form.
• Generation of a single personalized initial draft based on historical resolutions and the 40 standard macros.
• Guardrail pipeline strictly forbidding autonomous refund/credit commitments.
• Feedback loop telemetry (tracking agent accepts, edits, discards, and re-routes).

Out of Scope (Phase 1)

• Autonomous sending (auto-resolution without human click).
• Chat/live messaging support (limited strictly to asynchronous ticketing channels).
• Processing multi-turn responses (AssistRoute generates first responses only).
• Multi-language support (English only).

---

4. System Architecture & Workflow

``` [Customer Submits Ticket] │ ▼ [Event Ingestion: Webhook] ──▶ [PII Masking & Sanitization] │ ┌───────────────────────┴───────────────────────┐ ▼ ▼ [Routing Classifier] [Draft Generation Pipeline] │ │ Confidence $\ge$ Threshold? │ ├── Yes ──▶ Assign Queue 1. Macro Match / Retrieval └── No ──▶ Assign "General" (Flagged) 2. Prompt Compilation (LLM) │ 3. Deterministic Policy Guardrails │ │ └───────────────────────┬───────────────────────┘ ▼ [Agent Console: Pre-populated View] ├── Action A: Accept & Send ├── Action B: Edit & Send ├── Action C: Discard Draft └── Action D: Re-assign Queue ```

End-to-End Latency SLA

From `ticket.created` webhook ingestion to draft persistence in the ticketing database: $\le 5.0$ seconds (P95).

---

5. Functional Requirements

5.1 Routing Engine (ML Service)

• FR-1.1: The model must classify each ticket into one of four queues: `Billing`, `Technical`, `Account Access`, or `General`.
• FR-1.2: Confidence Scoring:
• If `confidence >= 0.85`: Automatically set the ticket's `queue_id` attribute.
• If `confidence < 0.85`: Set `queueid` to `General` and append the tag `needstriage`.
• FR-1.3: The Routing Engine must evaluate and route the ticket before generating the draft, as draft prompts require queue-specific system contexts.

5.2 Retrieval & Generation Pipeline

• FR-2.1 (Context Hydration): The generation service must pull:
• The ticket subject and body.
• The authenticated user's metadata: Plan tier (`Solo`, `Growth`, `Enterprise`), account age, and active add-on modules.
• The closest semantic match from Ledgerly's 40 standard macros.
• FR-2.2 (Draft Generation): The system must generate a friendly, concise first response that adheres to Ledgerly brand voice, incorporates the customer's name, and directly addresses the primary issue using the retrieved macro logic.
• FR-2.3 (Confidence Gating): If the model's semantic similarity score against approved knowledge bases or historical solutions falls below `0.70`, no draft shall be rendered. The UI must show: "Draft unavailable: Low context confidence."

5.3 Legal & Compliance Guardrail (Refund Protection)

* FR-3.1 (Deterministic Regex & Semantic Filter): Every draft must pass through a two-stage financial commitment filter prior to saving: 1. Lexical Check: Negative keyword blocking: `refund`, `credit`, `reimburse`, `waive fee`, `reverse charge`, `money back`, `comp`. 2. Semantic Entailment Check: Verification that the text does not commit Ledgerly to monetary compensation or liability. * FR-3.2 (Violation Action): If a draft contains a prohibited financial commitment: * The drafting engine must automatically strip the phrase and substitute safe boilerplate: "I have escalated your billing inquiry to our finance specialists for immediate review." * The draft must display a visible amber badge in the Agent UI: "Financial Review Required — No Commitments Permitted."

5.4 Support Agent Console UX

• FR-4.1 (Draft Presentation): The draft must render inside the native response editor as an uncommitted state (distinct background styling or watermarked).
• FR-4.2 (Interaction Actions):
• Tab / Single-Click Acceptance: Agent hits `Tab` or clicks `Use Draft` to copy the generated text into the active editor.
• Direct Editing: Editing the draft directly must be seamless without extra confirmation modals.
• Clear / Reject: Single-click action to clear the generated text and open a blank response field.
• FR-4.3 (Metadata Panel): A sidebar panel must show:
• Queue classification and confidence score.
• Which of the 40 macros informed the draft.
• A one-click queue reassignment dropdown.

``` +-------------------------------------------------------------------------+ | Ticket #10492: "Charged twice for Payroll add-on" | | Sender: Sarah Jenkins (Growth Tier) | Queue: [Billing (96%)] [Re-route] | +-------------------------------------------------------------------------+ | [ AI Drafted Response ] [Macro Source: Billing_Dup]| | | | Hi Sarah, | | | | Thanks for reaching out. I see that your account was debited twice | | for the Payroll add-on on October 1st. | | | | I have escalated this to our payment operations team to review the | | duplicate transaction details. We will update you here as soon as this | | review is complete. | | | | [ Accept Draft (Tab) ] [ Edit Draft ] [ Reject ] | +-------------------------------------------------------------------------+ | [ Agent Response Box ] | +-------------------------------------------------------------------------+ ```

---

6. ML & Data Engineering Specifications

6.1 Data Preparation (2-Year Historical Corpus)

1. Sanitization: Strip all historical PII (tax identifiers, SSNs, credit card numbers, passwords) using Microsoft Presidio or an equivalent NER pipeline before training/indexing. 2. Filtering: * Drop all historical tickets that required more than 4 re-routes (noisy labels). * Filter out tickets closed with negative customer satisfaction (CSAT $\le 2$). * Exclude responses superseded by outdated accounting rules or old pricing tiers (Ops team to define date cutoffs). 3. Macro Ground-Truth Alignment: Map the 40 canonical macros against historical agent responses to serve as gold-standard reference pairs.

6.2 Model Specifications

• Routing Classifier:
• Architecture: Fine-tuned lightweight encoder (e.g., `modern-bert-base` or `RoBERTa-base`) or an optimized classification endpoint.
• Input: Ticket Subject + Body.
• Output: Softmax distribution over `[Billing, Technical, Account Access, General]`.
• Latency Target: $< 200\text{ ms}$.
• Generative Drafting Model:
• Architecture: Hosted LLM (e.g., Claude 3.5 Sonnet or GPT-4o-mini) via secure enterprise API with zero-data-retention agreements.
• Prompt Design: System prompt containing Ledgerly tone guidelines, user context JSON, the selected macro instructions, and strict instructions forbidding financial promises.
• Temperature: `0.1` (low variability, high determinism).

6.3 Telemetry & Event Logging

The frontend and backend must emit the following events to the analytics lakehouse: * `ticketrouted`: `{ ticketid, predictedqueue, confidence, autoassigned: bool }` * `ticketrerouted`: `{ ticketid, oldqueue, newqueue, agentid }` * `draftgenerated`: `{ ticketid, macroid, promptversion, modelid, generationtimems }` * `draftactioned`: * `action`: `ACCEPTEDASIS` | `EDITED` | `REJECTED` * `originaldraft`: string * `finalsenttext`: string * `levenshteindistance`: int * `agentid`: string

---

7. Safety, Compliance, and Security

1. Refund Protection Hard Filter: As specified in Section 5.3, any draft containing automated refund promises is blocked from the UI by an air-gapped deterministic validation check that runs after LLM generation and cannot be bypassed by prompt engineering.
2. Zero Training on Customer Data: LLM vendors must have explicit Data Processing Agreements (DPAs) confirming customer ticket content is not used to train foundation models.
3. Data Storage & SOC 2: Masked inputs and outputs stored for model evaluation must adhere to standard data retention policies (auto-purged after 90 days).

---

8. Rollout Plan & Milestones

``` Phase 0: Offline Validation & Guardrail Auditing (Weeks 1-3) ├── Train routing model on historical data; evaluate against held-out test set. ├── Run 10,000 synthetic adversarial prompts targeting the refund guardrail. └── Success Gate: Routing accuracy > 93%, 0 adversarial refund breaches.

Phase 1: Shadow Mode (Weeks 4-5) ├── Pipeline processes live tickets; outputs are logged to Databricks/Snowflake. ├── No UI visible to agents. └── Evaluate: Model inference latency, real-time routing accuracy vs. manual triage.

Phase 2: Single-Queue Canary Pilot (Weeks 6-7) ├── Deploy exclusively to the Billing Queue (10 designated agents). ├── Daily sync with Support Operations leads to review draft quality. └── Success Gate: Draft utilization > 50%, no severe agent blockers.

Phase 3: General Availability (GA) & Iteration (Weeks 8+) ├── Enable routing and drafting across all 42 agents and 4 queues. ├── Implement weekly retraining cadence for classifier based on override logs. └── Track progress toward < 2.0 hour FRT target. ```

---

9. Risks & Mitigations

PRD: AI First-Response Drafting and Ticket Routing

Status: Proposed Product: Ledgerly Support Audience: Engineering, ML, and Support Operations leads

1. Summary

Ledgerly receives about 9,000 support tickets per week. Tickets are handled by 42 agents across Billing, Technical, Account access, and General. Average first response time is seven hours, against a target of under two hours. About 60% of tickets receive one of roughly 40 saved replies.

This feature will recommend a queue and prepare a first-response draft when a new ticket arrives. An agent must review and approve every draft before it is sent. The feature is intended to reduce time spent triaging and composing routine responses, while leaving decisions and customer communication under agent control. It must never send a response autonomously or make a refund commitment on Ledgerly’s behalf.

2. Goals and non-goals

Goals - Reduce median time from ticket creation to first response, with an operational goal of under two hours. - Reduce agent effort on routine first responses by using appropriate saved replies and ticket-specific details. - Route tickets to the best-fit queue, while making uncertain recommendations visible and easy to correct. - Preserve agent review and control over every customer-facing response.

Non-goals - Automatically send, resolve, or close tickets. - Decide refund eligibility, promise a refund, or commit to refund timing. - Draft replies to later messages in an existing conversation in the initial release. - Replace queue ownership, escalation policies, or agents’ judgment.

3. Users and workflow

The primary user is a support agent reviewing new tickets in their assigned queue. Support Operations owns queue definitions, saved replies, and handling guidance. ML and Engineering own model quality, serving, integrations, and monitoring.

For each new ticket, the system will: 1. Read the ticket’s permitted content and available support context. 2. Recommend one of the four queues, with a confidence indicator and brief reason. 3. Generate a first-response draft, preferentially based on a relevant approved saved reply where appropriate. 4. Display the recommendation and draft in the agent workflow. 5. Let the agent edit, discard, or approve. Approval sends the response through the existing support system; no response is sent before approval.<sup>1</sup> 6. Record the final queue, draft disposition, edits, and outcome for evaluation.

Agents may change the queue before or after reviewing the draft. A draft must remain clearly marked as AI-generated until approved.

4. Functional requirements

Routing

• Classify each ticket as Billing, Technical, Account access, or General.
• Show the recommended queue, confidence, and a short explanation grounded in ticket content.
• Allow agents to override the recommendation; the override becomes an evaluation signal, not an automatic training label.
• Route low-confidence or ambiguous cases to General and flag them for review. Support Operations must set and approve the launch confidence threshold using offline and pilot results.
• Do not infer urgency or bypass existing escalation rules in the initial release.

Drafting

• Generate a concise, relevant first response based on the ticket and approved support materials available to the system.
• Use a saved reply when it fits, adapting only with verified ticket details. Do not fabricate account status, actions taken, policy, or resolution.
• When information is insufficient, ask a clear clarifying question or provide a safe acknowledgement rather than guessing.
• Support agents can edit, discard, or approve the draft. No draft may be sent without an explicit agent action.
• If the request concerns a refund, the draft must not promise, confirm, or imply a refund or refund timing. It should use approved noncommittal wording and leave eligibility and timing to an agent. Refund-related tickets should be visibly flagged for agent attention.

5. Data, model, and system requirements

Ledgerly has two years of resolved tickets and agent replies. ML should assess data coverage and quality before training: queue labels, duplicate or reopened cases, outdated answers, agent-specific wording, and tickets whose resolution depended on account information unavailable at ticket creation. Historical agent replies are examples, not policy; approved saved replies and current support guidance take precedence.

Create a time-based training, validation, and test split to reduce leakage from repeated tickets or changing policies. Evaluate routing and draft quality separately, including by queue, ticket type, and relevant risk category. Do not train on post-response information when evaluating first-response behavior. Remove or protect unnecessary sensitive data, and use only the customer and account context needed for support.

The serving path must retrieve only context the agent is authorized to see. Engineering must confirm integration points with the ticketing system, queue assignment, saved replies, and send/approval controls before implementation. Log model/version, recommendation, confidence, draft, agent edits, approval or discard, final queue, and response timestamps, subject to Ledgerly’s retention and access policies. Restrict access to these logs and avoid storing secrets or unnecessary personal data in model prompts.

6. Quality, safety, and success measures

Establish a pre-launch baseline and report weekly: - Median and 90th-percentile time to first response; proportion answered within two hours. - Queue recommendation accuracy and agent override rate, overall and by queue. - Draft approval rate, edit rate, discard rate, and time from ticket opening to send. - Agent-rated usefulness and sampled quality review results. - Refund-policy violations, fabricated claims, and other serious customer-impacting errors.

Suggested launch gates, to be confirmed by Support Operations and Legal: routing performance must be no worse than the current human baseline on the held-out set; pilot draft samples must meet an agreed quality rubric; and no refund commitment may appear in an approved draft during safety testing. Any refund commitment, unauthorized send, or material fabricated claim triggers immediate disablement of the affected capability and incident review. Do not optimize approval rate alone: agents must be able to reject drafts without penalty.

7. Rollout and operations

1. Discovery and offline evaluation: Validate data, current queue definitions, approved saved replies, policy ownership, and integration feasibility. Build a labeled test set reviewed by Support Operations, with explicit refund and other high-risk examples.
2. Shadow mode: Run recommendations without showing or applying them. Compare predicted queues and drafts with agent outcomes; inspect errors and subgroup performance.
3. Limited pilot: Enable the workflow for a small, representative group of agents and ticket types. Keep agent approval mandatory. Support Operations reviews a daily sample and gathers structured agent feedback.
4. Expansion: Expand to all 42 agents only after launch gates are met and Legal and Support Operations approve. Roll back by disabling drafting or routing independently if either degrades.

Support Operations owns the queue taxonomy, approved reply content, quality rubric, and agent training. ML owns evaluation, thresholds, drift monitoring, and model updates. Engineering owns access controls, reliable delivery, audit logs, feature flags, and rollback. A named on-call owner must be assigned before pilot launch. Review quality and performance weekly during rollout and after material model, policy, or saved-reply changes. Re-evaluate against a fresh sample regularly; do not promote agent edits to training data without review and policy checks.

8. Open decisions before build

Confirm the ticketing platform and integration surfaces; which customer/account fields are available at ticket creation; the authoritative source for current support policy; Legal’s approved refund language and review process; the minimum acceptable routing and draft-quality thresholds; and the retention period for prompts, drafts, and audit logs. These decisions are launch prerequisites, not reasons to relax agent approval or refund safeguards.