Write a PRD

PRD: AI-Assisted First Response and Ticket Routing

Product: Ledgerly Support Assist Audience: Engineering, ML and Support Operations leads Status: Proposed MVP; thresholds below are launch targets, not measured results Accountable owner: Support Operations lead, with Engineering and ML leads owning delivery and model quality

1. Problem and objective

Ledgerly’s 42 support agents handle 9,000 tickets per week across Billing, Technical, Account access and General. Billing represents 38% of volume—approximately 3,420 tickets weekly. Average first-response time is seven hours, against a target of under two hours. Approximately 60% of tickets receive one of roughly 40 saved replies.

The feature will classify new tickets into the appropriate queue and prepare a grounded first-response draft for an agent to review. It should reduce queue-selection work and repetitive writing without delegating customer communication or financial commitments to the model.

Every outbound response requires explicit agent approval. Legal prohibits automated sending of refund commitments. The MVP will have no automated-send path and will not generate refund promises; agents must author any commitment using the existing authorized refund process.

2. Goals and success measures

Primary goals:

• Reduce average time to the first substantive human-approved response to under two hours during a staffed pilot.
• Reduce median active agent time spent preparing first responses by at least 30%.
• Route tickets accurately while avoiding hidden misroutes.
• Preserve response correctness, account security and customer satisfaction.

Measure first-response time from ticket creation to the first substantive response sent by an agent. Automated receipts do not count. Report calendar-hour and staffed-hour results separately, plus median and p90, to prevent averages hiding long waits.

Additional proposed launch and pilot thresholds:

Zero observed errors is a release gate, not proof of zero production risk. ML must report sample sizes and confidence intervals. Support Operations must confirm staffing coverage: drafting improvements alone cannot guarantee the response-time target.

3. Scope

MVP includes:

• Classification into Billing, Technical, Account access or General.
• Automatic routing only above validated, queue-specific confidence thresholds.
• Retrieval of approved saved replies and current support documentation.
• A first-response draft with internal source references, risk flags and suggested clarifying questions.
• Agent controls to edit, discard, regenerate, change queue and approve/send.
• Audit logs, monitoring and immediate disable controls.

Excluded: automatic sending, subsequent-turn assistance, ticket resolution, refunds, account changes, security verification decisions and customer-facing AI chat.

Ticket channels, languages and attachment formats are not specified. Support Operations will inventory them before implementation. MVP supports only validated text channels and languages; unsupported inputs receive normal human triage without a generated draft.

4. User workflow and functional requirements

1. Ingest: On creation of a new eligible ticket, capture its text, subject and approved metadata. Use only authorized customer/account context available to the assigned support role.
2. Assess: Predict a queue, calibrated confidence and risk flags such as refund request, account compromise or insufficient information.
3. Route: Assign high-confidence tickets to the predicted queue. Send uncertain cases to the existing manual-triage destination, provisionally General. Surface uncertainty prominently; do not treat General as a confident classification.
4. Draft: Retrieve relevant approved material and produce a concise response addressing the request. Prefer adapting an applicable saved reply over open-ended generation.
5. Review: Show the draft, suggested queue, risk flags and source links in the agent workspace. Clearly label the text “AI draft—not sent.”
6. Approve/send: The existing send action requires an explicit authenticated agent action. No background job, timeout or model output may trigger sending.
7. Learn: Record queue corrections, edits, rejection reasons and quality reviews for controlled evaluation and future retraining.

Drafts must not invent transactions, troubleshooting outcomes, refund eligibility or account status. Where evidence is missing, ask for necessary information or acknowledge that an agent must investigate. Avoid requesting passwords, full payment-card details or other unnecessary sensitive data.

Refund-related drafts may acknowledge the request and explain approved next steps, but must not promise an amount, eligibility or processing date. Flag these tickets for Billing review. Agents may add commitments only after authorized verification.

A draft becomes stale if relevant ticket content or account context changes. Disable approval until it is refreshed or explicitly reviewed against the updated context. Manual queue changes take precedence over later model results.

5. Data and ML approach

Two years of resolved tickets and agent replies are available. Historical replies are examples, not authoritative policy: outdated guidance and unauthorized commitments must not be reproduced.

ML and Support Operations will:

• Define queue labels and rules for mixed-intent tickets. Account-security concerns take precedence over routine billing questions; Support Operations must approve the complete precedence matrix.
• Audit final queue labels and sample ambiguous cases for expert adjudication.
• Remove duplicates, signatures and irrelevant quoted history; redact unnecessary personal and payment information.
• Preserve ticket/thread/customer grouping across splits to reduce leakage.
• Use a chronological training, validation and held-out test split, with the newest period reserved for testing.
• Evaluate only information available when the ticket arrived. Later replies and resolutions may provide labels but must never become runtime inputs.

Benchmark saved-reply retrieval and a conventional classifier before introducing more complex models. Select the simplest approach meeting quality, latency and operational requirements.

Generation will use current, versioned approved content. Historical replies may support offline training subject to privacy and quality approval, but must not serve as an unrestricted runtime knowledge source. Conflicting or missing sources trigger a clarification or human-investigation draft rather than a guessed answer.

Customer text and retrieved content are untrusted inputs. Instructions embedded in tickets must not change system rules, authorize actions or bypass review.

6. Engineering design and controls

Implement an asynchronous pipeline behind feature flags:

Ticket event → eligibility check → classifier → routing decision → retrieval → generation → policy validation → draft storage → agent UI.

Persist ticket ID/version, model and prompt versions, retrieved document versions, queue scores, draft state, risk flags and agent actions. Use idempotency keys to prevent duplicate processing and concurrency controls to avoid overwriting agent work.

The AI service must have no credential or permission to send messages, issue refunds or modify accounts. Routing permissions must be limited to approved queues. Outbound messages remain controlled by the existing authenticated support application.

Post-generation checks will block prohibited promises and unsupported sensitive claims. Failed checks suppress the draft and display a reason; they must not merely append a disclaimer.

Apply role-based access, encryption and existing retention rules. Any external model provider requires Security and Legal approval, including contractual restrictions on retention and training use. Log identifiers and operational metadata where possible, not unrestricted ticket bodies.

On timeout, provider outage or validation failure, preserve normal ticket handling. Keep the ticket visible, apply manual triage where necessary and show “Draft unavailable.” Never delay ticket intake while waiting for AI.

7. Evaluation and release gates

Build an adjudicated test set covering all queues, common saved-reply cases, mixed intent, sparse descriptions, refund disputes, account compromise, outdated-policy examples and prompt-injection attempts. Report overall and per-queue performance; oversampled risk cases must also be reported separately from production-weighted results.

Two support reviewers will score drafts for correctness, relevance, completeness, tone and policy compliance, with disagreements adjudicated. Distinguish cosmetic edits from substantive corrections. Track source support and appropriate abstention, not acceptance rate alone.

Required acceptance tests include:

• A refund-request draft contains no commitment.
• No model or pipeline component can invoke sending.
• Unsupported facts cause omission or escalation.
• Ticket updates invalidate stale drafts.
• Human rerouting is not overwritten.
• Provider failure leaves tickets accessible.
• Duplicate events create neither duplicate drafts nor duplicate sends.
• Agent permissions also constrain retrieved context.

Production expansion requires meeting quality thresholds and passing all safety-critical tests. Any unauthorized financial promise in evaluation blocks launch pending remediation.

8. Rollout, ownership and operations

Indicative eight-week plan:

• Weeks 1–2: Support Operations approves taxonomy, saved replies, triage rules and baseline measurements; ML audits data; Engineering completes integration and security design.
• Weeks 3–4: Build the pipeline, UI, evaluation harness and operational controls.
• Weeks 5–6: Run offline evaluation and shadow mode without changing queues or showing drafts.
• Weeks 7–8: Pilot with selected agents, then expand by queue only after gates pass.

Use a randomized holdout where practical, balanced by queue and shift, to compare handling time, response time, corrections and customer outcomes.

Engineering owns availability, access controls and rollback. ML owns evaluation, calibration and drift monitoring. Support Operations owns policy content, QA, agent training and staffing. Legal approves refund wording and provider terms.

Provide separate kill switches for routing and drafting. Suspected unauthorized commitments, data exposure or account-security mishandling trigger immediate suspension of the affected capability and incident review. Review corrections and rejected drafts weekly; retraining requires fresh holdout evaluation, not automatic ingestion of agent edits.

Ledgerly: AI first-response drafts and ticket routing

Status: Proposed MVP requirements. Numerical gates below are proposed targets, not observed performance. Product owns scope; Engineering owns workflow integrity; ML owns model quality; Support Operations owns queue policy, content and review quality.

1. Problem and outcome

Ledgerly’s 42 support agents handle 9,000 tickets weekly across Billing, Technical, Account access and General. Billing represents 38%, approximately 3,420 tickets weekly. Average first response is seven hours against a target below two. Approximately 60% of tickets use one of 40 saved replies.

The feature should reduce triage and composition time without shifting effort into correcting poor drafts or recovering misrouted tickets. Every outgoing draft requires human approval. Legal prohibits automated sending of refund commitments.

Faster drafting alone may not close the five-hour gap. Before piloting, Support Operations must measure arrival-to-assignment, assignment-to-review and review-to-send delays by queue and staffed hours, then identify any coverage or staffing changes needed.

2. Scope and operating boundary

MVP processes newly created English-language, text-based tickets in the existing support workspace. It assigns a queue and prepares a first-response draft. Threads, ticket metadata and approved support content are inputs. Attachments are not interpreted; attachment-dependent or unsupported-language tickets remain available for manual handling.

Excluded: follow-up generation, autonomous sending, refunds, account changes, payment actions and staffing optimisation. Existing spam, security and priority rules run first and cannot be overridden by the model.²

Routing and drafting operate independently. A drafting failure must not prevent routing or agent access. An uncertain route must not prevent a useful draft.

3. Agent workflow

1. Ticket creation starts asynchronous processing. The ticket is immediately visible and its response clock starts at original receipt.
2. The system records a route, reason and confidence tier, then prepares a draft where supported.
3. The agent sees the original message, assigned queue, editable draft, source references and warnings. Sources and warnings are internal only.
4. The agent can change queue, edit, regenerate, discard or write manually. Regeneration never overwrites unsaved edits without confirmation.
5. Selecting Approve and send authorises the exact visible text. There is no bulk approval.

Draft states are pending, ready, unavailable, stale and sent. Queue changes preserve receipt time and draft history. New customer messages or changes to relevant ticket context mark drafts stale and require renewed review. Show actionable failure messages and keep the manual composer available.

4. Routing requirements

Support Operations owns this initial taxonomy:

For multiple intents, suspected account compromise takes precedence, followed by access-blocking issues; otherwise route by the customer’s main requested resolution. Record secondary intents for the receiving agent.

Automatically assign only when a queue-specific threshold meets the evaluation gate below. Confidence must be calibrated against labelled examples, not taken from a model’s self-reported certainty.

Below threshold, assign to General with a distinct Needs triage status and show the leading suggestions. This is a fallback assignment, not a successful classification. Support Operations assigns a named triage owner each shift and reviews these tickets at least every 30 minutes during staffed hours. Existing out-of-hours escalation remains in force.

Agents can override any route and optionally record a reason. Never automatically reroute after an agent takes ownership. Log overrides for review, not immediate retraining.

5. Draft content and refund controls

Start with retrieval from the approximately 40 saved replies and current, Support Operations-approved help and policy content. Adapt an applicable reply before attempting a novel answer. Each source has an owner, version and review date; withdrawn content becomes unavailable immediately.

Drafts must answer the stated question, request necessary missing details and cite supporting sources internally. They must not invent account facts, troubleshooting outcomes, eligibility, amounts or dates. Account-specific claims require authorised, current account context. Without it, draft a clarification or indicate that an agent must investigate.

For refund requests, MVP drafts may acknowledge the request and explain approved review steps, but must not promise eligibility, payment amounts or payment dates. Agents may manually add commitments only under Ledgerly’s existing refund authority policy. Flag refund-related tickets and require an explicit acknowledgement when the final text contains a detected commitment. This check assists reviewers; it is not the legal enforcement boundary.

The enforcement boundary is the sending service: generation workers have no send credentials. Every AI-assisted send requires an authenticated agent approval bound to ticket ID, recipient and exact message version. Any subsequent edit invalidates approval. Retries cannot send duplicate messages. Existing automation must not consume AI drafts as sendable replies. Therefore, missed refund detection cannot trigger autonomous sending.

6. Data and ML approach

Use the two-year resolved-ticket archive to learn routing patterns and evaluate drafting. Historical agent replies are examples, not authoritative policy. Resolution does not establish correctness or refund permission.

Support Operations must relabel a representative sample against today’s taxonomy, with two reviewers and adjudication for disagreements. Redact credentials, payment details and unnecessary personal information. Preserve tenant boundaries and restrict access to approved training personnel and services.

Split chronologically into training, validation and a locked recent test set. Keep complete conversations and duplicate ticket clusters in one split. At inference and evaluation, expose only information available before the first response; later replies and final queue labels must not leak into inputs.

Begin with a classifier plus retrieval-grounded generation. Fine-tuning is optional and requires measurable improvement over this baseline. No automatic learning from approvals. Provider data use and retention must be approved before production data leaves Ledgerly.

Treat customer text and retrieved content as untrusted data: embedded instructions cannot change policies, access other tenants or invoke privileged actions.

7. Evaluation and launch gates

Build a locked test set of at least 1,000 tickets, stratified across queues, with at least 150 per queue. Include saved-reply matches, ambiguous intents, refunds, outdated policies, missing context, prompt injection and sensitive-data cases. Report volume-weighted results and per-queue results separately.

Required gates:

• Routing: at least 95% precision among automatically assigned tickets in each queue. Report confidence intervals, recall, coverage and the confusion matrix. Target at least 60% overall automatic-routing coverage; do not lower precision to achieve coverage.
• Drafts: at least 90% rated sendable unchanged or with minor stylistic edits by support reviewers.¹ Score factual accuracy, policy compliance, completeness and tone separately.
• Critical errors: zero observed cross-tenant disclosures, unsupported refund commitments in generated drafts or account-security instructions that bypass verification. Any occurrence blocks release pending correction and regression testing.
• Workflow: all permission, stale-approval, recipient-change, duplicate-event and retry tests pass. Attempts to send without valid approval must fail.

Use blinded human review and adjudicate disagreements. Automated grading can assist sampling but cannot determine safety gates alone. Zero observed errors is a test result, not proof of zero production risk.

8. Pilot, measurement and operations

Run one week in shadow mode across all queues, followed by a two-week pilot with agents from every queue. Randomise eligible tickets within queue and shift between assisted and existing workflows; account for shared-agent effects when interpreting results.

Primary pilot gate: at least 20% lower mean receipt-to-first-human-response time versus control. Track progress towards the under-two-hour target across all incoming tickets, including unsupported and fallback cases. Also report median, p90, percentage answered within two hours, review time, queue transfers, correction severity, backlog age and customer satisfaction. Draft creation is not a first response.

Expand only if offline gates hold, human handling time improves and transfer rates and customer satisfaction show no material deterioration. Define tolerances and adequate sample sizes before the pilot; extend measurement when inconclusive.

Engineering provides independent routing and drafting kill switches, audit logs and alerts. Target p95 draft readiness within 60 seconds; after timeout, mark unavailable and retain manual handling. Retries must not overwrite agent work.

Support Operations samples 50 assisted tickets weekly, oversampling refunds and overrides. ML monitors quality and drift by queue and content version. Any unauthorised send or cross-tenant disclosure triggers immediate suspension of the affected feature and incident review.

Before pilot launch, Engineering verifies send-path enforcement and integration contracts; ML publishes evaluation results; Support Operations approves taxonomy, sources and shift ownership; Legal confirms the refund workflow.

PRD: AI-Drafted First Responses and Ticket Routing

Status: Draft for build planning Audience: Engineering, ML, and Support Operations leads Product: Ledgerly Support

1. Problem

Ledgerly receives about 9,000 support tickets a week, handled by 42 agents across four queues: Billing, Technical, Account access, and General. Billing is 38% of volume (about 3,400 tickets a week). Average first response is 7 hours against a target of under 2.

Two things likely drive the delay, and we don't yet know their split:

• Wait time. Tickets sit in a queue, or in the wrong queue, before anyone opens them.
• Handle time. Agents spend time writing or assembling a first reply. About 60% of tickets are already answered with one of roughly 40 saved replies, so much of this work is repetitive.

This feature has two parts. It routes each new ticket to the right queue, and it drafts a first response for the agent to review, edit, and send. Nothing is ever sent without an agent's approval.

2. Goals and non-goals

Goals 1. Bring mean first response time under 2 hours within 90 days of full rollout. 2. Route at least 92% of tickets to the correct queue on first assignment. 3. Have agents send at least 50% of drafts with no or light edits. 4. Ensure no draft or sent message contains an unreviewed refund commitment (see §5).

Non-goals - Auto-sending any message, in any queue, at any confidence level. - Resolving tickets, or taking actions such as issuing refunds, changing plans, or resetting access. - Replacing follow-up replies after the first response (later phase). - Changing queue structure or staffing.

3. Users

• Support agents (42): review, edit, approve, or discard drafts, and correct routing.
• Support-ops leads: own saved replies, policy content, thresholds, and quality review.
• Legal/compliance: consulted on the refund guardrail and audit logs.

4. Functional requirements

4.1 Routing

• R1. On ticket creation, classify into one of the four queues and assign it automatically.
• R2. Store a confidence score with each assignment. Below a threshold set by support ops (initially tuned so about 10% of tickets fall below it), assign to General with a "needs triage" tag and show the top two suggested queues.
• R3. Agents can re-route in one click and give an optional reason. Every re-route is logged as a labeled correction.
• R4. Also emit tags used by drafting: a topic label (for example, invoice question, failed payment, login lockout) and a refund-related flag.

4.2 Draft generation

• R5. A draft must be ready in the agent's ticket view within 60 seconds of ticket creation, and never block the ticket from appearing. If drafting fails or times out, the ticket appears as it does today.
• R6. Use two drafting paths:
• Saved-reply path. If a ticket matches one of the ~40 saved replies above a similarity threshold, use that reply with variables filled in (name, plan, invoice number). The wording stays as approved by support ops.
• Generated path. Otherwise, generate a draft grounded in retrieved help-center articles and similar past resolved tickets. Every factual claim about policy, pricing, or product behavior must come from retrieved content. If the content is insufficient, the draft asks the customer a clarifying question or says an agent is investigating, rather than guessing.
• R7. Each draft shows which path produced it and its sources (saved reply ID or article links) so agents can verify quickly.
• R8. Match the tone and format of the best-rated historical agent replies. Support ops maintains the style guide.

4.3 Agent workflow

• R9. Drafts appear pre-filled in the reply composer with three actions: Send (after optional edits), Discard, and Regenerate. Discarding takes an optional reason from a short list.
• R10. The system has no auto-send code path. Sending requires an authenticated agent action, and this is enforced in the sending service, not just the UI.
• R11. Log for each ticket: draft text, final sent text, edit distance, agent ID, time from ticket open to send, and discard or re-route reasons.

4.4 Account-access safeguards

• R12. Drafts for Account access tickets must never confirm whether an account exists, reveal account details, or state that access was restored or changed. They may give standard verification instructions from approved content only.

5. Refund guardrail (legal requirement)

Legal requires no automated sending of refund commitments. Because agents approve every draft, the primary control is R10. We add layered controls so the model never puts a commitment in front of an agent as if it were approved policy.

• G1. Prompt and template rules. Drafts may acknowledge a refund request and say it is being reviewed. They must not promise, imply, or estimate a refund, credit, waiver, or timeline, and no saved reply used by this feature may contain one.
• G2. Output classifier. Every draft passes a commitment detector (rules plus model) before display. If it fires, replace the offending sentence with a neutral placeholder, such as "[Agent: refund decision needed]", and tag the ticket.
• G3. Send-time check. The detector also runs on the final edited text. If an agent's own text contains a refund commitment, Send requires an explicit confirmation ("This message commits to a refund"), which is logged. Agents may make commitments in their own words. The system may not.
• G4. Audit. Retain drafts and final messages for the period Legal specifies. Support ops reviews a weekly sample of 100 refund-related tickets.
• G5. Release bar. The detector must reach at least 98% recall on a labeled set of refund-commitment phrasings, including implicit ones like "we'll take care of that charge", before Billing goes live. Legal signs off on the test set.

6. Data and evaluation

Available: two years of resolved tickets with agent replies.

Data preparation (ML lead owns) - Use the queue where a ticket was resolved as the routing label, not where it first landed, since misroutes are the problem we're fixing. Also keep the initial queue to measure the historical misroute rate. - Remove PII before training or indexing. Get security review of the vendor and hosting setup if any external model is used. - Filter out replies that are outdated (old pricing, retired features, superseded policies). Support ops flags policy change dates so the pipeline can exclude earlier replies. - Mark replies that used a saved reply. This gives the saved-reply matching set and a baseline for the other 40%. - Have support ops label about 1,500 recent tickets for refund-commitment presence, routing, and draft quality. This set doubles as the gold evaluation set.

Evaluation - Split by time, not randomly: train on the earliest ~21 months, test on the most recent 3. Random splits will overstate performance because of seasonality and policy drift. - Routing: report accuracy and per-queue precision and recall, plus confusion between Billing and General, and Account access and Technical, which are likely weak spots. - Drafts: blind human review by senior agents on a 5-point rubric (accuracy, policy correctness, tone, completeness), with automatic policy-violation checks. Compare against the original agent reply. - Slice results by queue, topic, and saved-reply versus generated path.

7. Success metrics

First task: instrument the current split of first-response time into queue wait and agent handle time. Drafting mainly cuts handle time. If wait time dominates, we also need queue-ordering changes, such as surfacing the oldest tickets first, or the 2-hour target will be missed even with high draft quality. Throughput is about 43 tickets per agent per day, so handle time matters, but it is not necessarily the bottleneck.

8. Rollout

1. Shadow mode (2–3 weeks). Run routing and drafting on live tickets without showing anything to agents. Compare with actual outcomes, set thresholds, and validate the refund detector.
2. Pilot (3 weeks). Enable routing for all tickets. Show drafts to about 12 agents across General, Technical, and Account access.
3. Expand. Enable drafts for all agents in those three queues once the pilot meets the quality targets.
4. Billing. Enable only after G5 is met and Legal has signed off. Start with 6–8 agents, then expand.

Any queue can be switched off by a feature flag within minutes. Rollback triggers: any refund-commitment incident, a policy-incorrect rate above 5% in weekly review, or a CSAT drop of more than 3 points.

9. Risks and open questions

• Automation bias. Agents may approve drafts without reading them. Mitigations: show sources, sample-audit sent messages, and track the edit rate by agent. A near-zero edit rate on generated-path drafts is a warning sign.
• Stale content. Saved replies and help articles drift from policy. Support ops needs a named owner and a review cadence.
• Language. We don't know the share of non-English tickets. Confirm before launch. If it is significant, scope it out or handle it separately.
• Legal definition. Does "refund commitment" include credits, prorations, or fee waivers? Legal to define by the end of shadow mode, since it determines G1–G3.
• Hosting. Confirm data-residency and vendor constraints for ticket content.
• Threshold ownership. Decide who can change confidence thresholds in production (proposed: support-ops lead, with ML review).

10. Ownership

• Engineering: integration with the ticketing system, sending-service enforcement (R10), logging, feature flags, latency.
• ML: data preparation, routing model, drafting pipeline, refund detector, evaluation and monitoring.
• Support ops: labeling, saved-reply and style-guide upkeep, pilot agents, weekly quality review, and thresholds.
• Legal: the refund definition, the detector test set sign-off, and audit retention.