Write a PRD

PRD: AI First-Response Drafting and Ticket Routing

Status: Proposed Product: Ledgerly Support Audience: Engineering, ML, and Support Operations leads

1. Summary

Ledgerly receives about 9,000 support tickets per week. Tickets are handled by 42 agents across Billing, Technical, Account access, and General. Average first response time is seven hours, against a target of under two hours. About 60% of tickets receive one of roughly 40 saved replies.

This feature will recommend a queue and prepare a first-response draft when a new ticket arrives. An agent must review and approve every draft before it is sent. The feature is intended to reduce time spent triaging and composing routine responses, while leaving decisions and customer communication under agent control. It must never send a response autonomously or make a refund commitment on Ledgerly’s behalf.

2. Goals and non-goals

Goals - Reduce median time from ticket creation to first response, with an operational goal of under two hours. - Reduce agent effort on routine first responses by using appropriate saved replies and ticket-specific details. - Route tickets to the best-fit queue, while making uncertain recommendations visible and easy to correct. - Preserve agent review and control over every customer-facing response.

Non-goals - Automatically send, resolve, or close tickets. - Decide refund eligibility, promise a refund, or commit to refund timing. - Draft replies to later messages in an existing conversation in the initial release. - Replace queue ownership, escalation policies, or agents’ judgment.

3. Users and workflow

The primary user is a support agent reviewing new tickets in their assigned queue. Support Operations owns queue definitions, saved replies, and handling guidance. ML and Engineering own model quality, serving, integrations, and monitoring.

For each new ticket, the system will: 1. Read the ticket’s permitted content and available support context. 2. Recommend one of the four queues, with a confidence indicator and brief reason. 3. Generate a first-response draft, preferentially based on a relevant approved saved reply where appropriate. 4. Display the recommendation and draft in the agent workflow. 5. Let the agent edit, discard, or approve. Approval sends the response through the existing support system; no response is sent before approval.¹ 6. Record the final queue, draft disposition, edits, and outcome for evaluation.

Agents may change the queue before or after reviewing the draft. A draft must remain clearly marked as AI-generated until approved.

4. Functional requirements

Routing

• Classify each ticket as Billing, Technical, Account access, or General.
• Show the recommended queue, confidence, and a short explanation grounded in ticket content.
• Allow agents to override the recommendation; the override becomes an evaluation signal, not an automatic training label.
• Route low-confidence or ambiguous cases to General and flag them for review. Support Operations must set and approve the launch confidence threshold using offline and pilot results.
• Do not infer urgency or bypass existing escalation rules in the initial release.

Drafting

• Generate a concise, relevant first response based on the ticket and approved support materials available to the system.
• Use a saved reply when it fits, adapting only with verified ticket details. Do not fabricate account status, actions taken, policy, or resolution.
• When information is insufficient, ask a clear clarifying question or provide a safe acknowledgement rather than guessing.
• Support agents can edit, discard, or approve the draft. No draft may be sent without an explicit agent action.
• If the request concerns a refund, the draft must not promise, confirm, or imply a refund or refund timing. It should use approved noncommittal wording and leave eligibility and timing to an agent. Refund-related tickets should be visibly flagged for agent attention.

5. Data, model, and system requirements

Ledgerly has two years of resolved tickets and agent replies. ML should assess data coverage and quality before training: queue labels, duplicate or reopened cases, outdated answers, agent-specific wording, and tickets whose resolution depended on account information unavailable at ticket creation. Historical agent replies are examples, not policy; approved saved replies and current support guidance take precedence.

Create a time-based training, validation, and test split to reduce leakage from repeated tickets or changing policies. Evaluate routing and draft quality separately, including by queue, ticket type, and relevant risk category. Do not train on post-response information when evaluating first-response behavior. Remove or protect unnecessary sensitive data, and use only the customer and account context needed for support.

The serving path must retrieve only context the agent is authorized to see. Engineering must confirm integration points with the ticketing system, queue assignment, saved replies, and send/approval controls before implementation. Log model/version, recommendation, confidence, draft, agent edits, approval or discard, final queue, and response timestamps, subject to Ledgerly’s retention and access policies. Restrict access to these logs and avoid storing secrets or unnecessary personal data in model prompts.

6. Quality, safety, and success measures

Establish a pre-launch baseline and report weekly: - Median and 90th-percentile time to first response; proportion answered within two hours. - Queue recommendation accuracy and agent override rate, overall and by queue. - Draft approval rate, edit rate, discard rate, and time from ticket opening to send. - Agent-rated usefulness and sampled quality review results. - Refund-policy violations, fabricated claims, and other serious customer-impacting errors.

Suggested launch gates, to be confirmed by Support Operations and Legal: routing performance must be no worse than the current human baseline on the held-out set; pilot draft samples must meet an agreed quality rubric; and no refund commitment may appear in an approved draft during safety testing. Any refund commitment, unauthorized send, or material fabricated claim triggers immediate disablement of the affected capability and incident review. Do not optimize approval rate alone: agents must be able to reject drafts without penalty.

7. Rollout and operations

1. Discovery and offline evaluation: Validate data, current queue definitions, approved saved replies, policy ownership, and integration feasibility. Build a labeled test set reviewed by Support Operations, with explicit refund and other high-risk examples.
2. Shadow mode: Run recommendations without showing or applying them. Compare predicted queues and drafts with agent outcomes; inspect errors and subgroup performance.
3. Limited pilot: Enable the workflow for a small, representative group of agents and ticket types. Keep agent approval mandatory. Support Operations reviews a daily sample and gathers structured agent feedback.
4. Expansion: Expand to all 42 agents only after launch gates are met and Legal and Support Operations approve. Roll back by disabling drafting or routing independently if either degrades.

Support Operations owns the queue taxonomy, approved reply content, quality rubric, and agent training. ML owns evaluation, thresholds, drift monitoring, and model updates. Engineering owns access controls, reliable delivery, audit logs, feature flags, and rollback. A named on-call owner must be assigned before pilot launch. Review quality and performance weekly during rollout and after material model, policy, or saved-reply changes. Re-evaluate against a fresh sample regularly; do not promote agent edits to training data without review and policy checks.

8. Open decisions before build

Confirm the ticketing platform and integration surfaces; which customer/account fields are available at ticket creation; the authoritative source for current support policy; Legal’s approved refund language and review process; the minimum acceptable routing and draft-quality thresholds; and the retention period for prompts, drafts, and audit logs. These decisions are launch prerequisites, not reasons to relax agent approval or refund safeguards.

PRD: AI-Assisted First Response and Ticket Routing

Product: Ledgerly Support Assist Audience: Engineering, ML and Support Operations leads Status: Proposed MVP; thresholds below are launch targets, not measured results Accountable owner: Support Operations lead, with Engineering and ML leads owning delivery and model quality

1. Problem and objective

Ledgerly’s 42 support agents handle 9,000 tickets per week across Billing, Technical, Account access and General. Billing represents 38% of volume—approximately 3,420 tickets weekly. Average first-response time is seven hours, against a target of under two hours. Approximately 60% of tickets receive one of roughly 40 saved replies.

The feature will classify new tickets into the appropriate queue and prepare a grounded first-response draft for an agent to review. It should reduce queue-selection work and repetitive writing without delegating customer communication or financial commitments to the model.

Every outbound response requires explicit agent approval. Legal prohibits automated sending of refund commitments. The MVP will have no automated-send path and will not generate refund promises; agents must author any commitment using the existing authorized refund process.

2. Goals and success measures

Primary goals:

• Reduce average time to the first substantive human-approved response to under two hours during a staffed pilot.
• Reduce median active agent time spent preparing first responses by at least 30%.
• Route tickets accurately while avoiding hidden misroutes.
• Preserve response correctness, account security and customer satisfaction.

Measure first-response time from ticket creation to the first substantive response sent by an agent. Automated receipts do not count. Report calendar-hour and staffed-hour results separately, plus median and p90, to prevent averages hiding long waits.

Additional proposed launch and pilot thresholds:

Zero observed errors is a release gate, not proof of zero production risk. ML must report sample sizes and confidence intervals. Support Operations must confirm staffing coverage: drafting improvements alone cannot guarantee the response-time target.

3. Scope

MVP includes:

• Classification into Billing, Technical, Account access or General.
• Automatic routing only above validated, queue-specific confidence thresholds.
• Retrieval of approved saved replies and current support documentation.
• A first-response draft with internal source references, risk flags and suggested clarifying questions.
• Agent controls to edit, discard, regenerate, change queue and approve/send.
• Audit logs, monitoring and immediate disable controls.

Excluded: automatic sending, subsequent-turn assistance, ticket resolution, refunds, account changes, security verification decisions and customer-facing AI chat.

Ticket channels, languages and attachment formats are not specified. Support Operations will inventory them before implementation. MVP supports only validated text channels and languages; unsupported inputs receive normal human triage without a generated draft.

4. User workflow and functional requirements

1. Ingest: On creation of a new eligible ticket, capture its text, subject and approved metadata. Use only authorized customer/account context available to the assigned support role.
2. Assess: Predict a queue, calibrated confidence and risk flags such as refund request, account compromise or insufficient information.
3. Route: Assign high-confidence tickets to the predicted queue. Send uncertain cases to the existing manual-triage destination, provisionally General. Surface uncertainty prominently; do not treat General as a confident classification.
4. Draft: Retrieve relevant approved material and produce a concise response addressing the request. Prefer adapting an applicable saved reply over open-ended generation.
5. Review: Show the draft, suggested queue, risk flags and source links in the agent workspace. Clearly label the text “AI draft—not sent.”
6. Approve/send: The existing send action requires an explicit authenticated agent action. No background job, timeout or model output may trigger sending.
7. Learn: Record queue corrections, edits, rejection reasons and quality reviews for controlled evaluation and future retraining.

Drafts must not invent transactions, troubleshooting outcomes, refund eligibility or account status. Where evidence is missing, ask for necessary information or acknowledge that an agent must investigate. Avoid requesting passwords, full payment-card details or other unnecessary sensitive data.

Refund-related drafts may acknowledge the request and explain approved next steps, but must not promise an amount, eligibility or processing date. Flag these tickets for Billing review. Agents may add commitments only after authorized verification.

A draft becomes stale if relevant ticket content or account context changes. Disable approval until it is refreshed or explicitly reviewed against the updated context. Manual queue changes take precedence over later model results.

5. Data and ML approach

Two years of resolved tickets and agent replies are available. Historical replies are examples, not authoritative policy: outdated guidance and unauthorized commitments must not be reproduced.

ML and Support Operations will:

• Define queue labels and rules for mixed-intent tickets. Account-security concerns take precedence over routine billing questions; Support Operations must approve the complete precedence matrix.
• Audit final queue labels and sample ambiguous cases for expert adjudication.
• Remove duplicates, signatures and irrelevant quoted history; redact unnecessary personal and payment information.
• Preserve ticket/thread/customer grouping across splits to reduce leakage.
• Use a chronological training, validation and held-out test split, with the newest period reserved for testing.
• Evaluate only information available when the ticket arrived. Later replies and resolutions may provide labels but must never become runtime inputs.

Benchmark saved-reply retrieval and a conventional classifier before introducing more complex models. Select the simplest approach meeting quality, latency and operational requirements.

Generation will use current, versioned approved content. Historical replies may support offline training subject to privacy and quality approval, but must not serve as an unrestricted runtime knowledge source. Conflicting or missing sources trigger a clarification or human-investigation draft rather than a guessed answer.

Customer text and retrieved content are untrusted inputs. Instructions embedded in tickets must not change system rules, authorize actions or bypass review.

6. Engineering design and controls

Implement an asynchronous pipeline behind feature flags:

Ticket event → eligibility check → classifier → routing decision → retrieval → generation → policy validation → draft storage → agent UI.

Persist ticket ID/version, model and prompt versions, retrieved document versions, queue scores, draft state, risk flags and agent actions. Use idempotency keys to prevent duplicate processing and concurrency controls to avoid overwriting agent work.

The AI service must have no credential or permission to send messages, issue refunds or modify accounts. Routing permissions must be limited to approved queues. Outbound messages remain controlled by the existing authenticated support application.

Post-generation checks will block prohibited promises and unsupported sensitive claims. Failed checks suppress the draft and display a reason; they must not merely append a disclaimer.

Apply role-based access, encryption and existing retention rules. Any external model provider requires Security and Legal approval, including contractual restrictions on retention and training use. Log identifiers and operational metadata where possible, not unrestricted ticket bodies.

On timeout, provider outage or validation failure, preserve normal ticket handling. Keep the ticket visible, apply manual triage where necessary and show “Draft unavailable.” Never delay ticket intake while waiting for AI.

7. Evaluation and release gates

Build an adjudicated test set covering all queues, common saved-reply cases, mixed intent, sparse descriptions, refund disputes, account compromise, outdated-policy examples and prompt-injection attempts. Report overall and per-queue performance; oversampled risk cases must also be reported separately from production-weighted results.

Two support reviewers will score drafts for correctness, relevance, completeness, tone and policy compliance, with disagreements adjudicated. Distinguish cosmetic edits from substantive corrections. Track source support and appropriate abstention, not acceptance rate alone.

Required acceptance tests include:

• A refund-request draft contains no commitment.
• No model or pipeline component can invoke sending.
• Unsupported facts cause omission or escalation.
• Ticket updates invalidate stale drafts.
• Human rerouting is not overwritten.
• Provider failure leaves tickets accessible.
• Duplicate events create neither duplicate drafts nor duplicate sends.
• Agent permissions also constrain retrieved context.

Production expansion requires meeting quality thresholds and passing all safety-critical tests. Any unauthorized financial promise in evaluation blocks launch pending remediation.

8. Rollout, ownership and operations

Indicative eight-week plan:

• Weeks 1–2: Support Operations approves taxonomy, saved replies, triage rules and baseline measurements; ML audits data; Engineering completes integration and security design.
• Weeks 3–4: Build the pipeline, UI, evaluation harness and operational controls.
• Weeks 5–6: Run offline evaluation and shadow mode without changing queues or showing drafts.
• Weeks 7–8: Pilot with selected agents, then expand by queue only after gates pass.

Use a randomized holdout where practical, balanced by queue and shift, to compare handling time, response time, corrections and customer outcomes.

Engineering owns availability, access controls and rollback. ML owns evaluation, calibration and drift monitoring. Support Operations owns policy content, QA, agent training and staffing. Legal approves refund wording and provider terms.

Provide separate kill switches for routing and drafting. Suspected unauthorized commitments, data exposure or account-security mishandling trigger immediate suspension of the affected capability and incident review. Review corrections and rejected drafts weekly; retraining requires fresh holdout evaluation, not automatic ingestion of agent edits.