Humanity’s Last PRD
Latest drop: Haiku 5.5 · 8 Oct
Tasks / Product Craft

Write a PRD

Can the model turn a brief into a spec engineers could build from, including how an AI feature behaves when it is wrong?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 25% were usable with at most a quick edit.

Reliably right

  1. Specifies AI failure behaviour100% pass
    It specifies what happens when uncertain or wrong: block ownership, show retry states, disable assignments on violation, and never fall back to calendar data.
    GPT-6 Astra · ChatGPT · Meeting summaries with action items
  2. Builds the regulator's rules in100% pass
    All three rules (2-business-day acknowledge, 15-business-day resolve or explain, no harder than paying) appear as testable requirements R8, R10, R13.
    Sonnet 5.5 · API · Disputes that aren't disputes
  3. Produces the required deliverable94% pass
    The PRD is complete, written for engineering and design leads, within the 1,000–1,500 word range, and directly actionable.
    GPT-6 Astra · ChatGPT · Meeting summaries with action items

Where it slips

  1. Proposes tests that could fail36% pass
    Launch gates have numeric thresholds and actions, but lack explicit measurement windows (e.g., pilot duration, number of meetings) needed to read out results.
    GPT-6.1 Sol · API · Meeting summaries with action items
  2. Avoids unsupported claims42% pass
    Presents the join log and identity tiers as established facts without labelling them as assumptions, when the context provides no support.
    Opus 5.5 · Claude · Meeting summaries with action items
  3. Uses the supplied evidence correctly44% pass
    The output invents a join log and identity tiers (Verified, Guest, Room) that are not in the supplied context, making unsupported claims about the current system.
    Opus 5.5 · Claude · Meeting summaries with action items

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Success settles trade-offs

    Does the PRD define success as a measurable outcome with a target and time frame, and say which goal wins when two of them conflict?

    Passes when Names a success metric with a target and time frame, and gives at least one explicit trade-off rule (for example, accepting lower coverage to keep precision above a stated level).

    Product and growth leadersOn Lenny’s Podcast

  2. Says who it's for, and who it isn't

    Does the PRD name the specific users it is for, say explicitly who it is not for, and make at least one scope decision that follows from that choice?

    Passes when Names the target users and at least one group it deliberately leaves out, and a scope decision visibly follows from the exclusion.

    Matt LeMayLenny’s Podcast

  3. Plans how people will find it

    Does the PRD say how the feature reaches people: how it rolls out, how users discover it, and who needs to be ready for it (support, sales), rather than stopping at what to build?

    Passes when Gives a staged rollout, a way users will discover the feature, and the teams that need to be ready, all fitted to this feature.

    Ami VoraLenny’s PodcastKarri SaarinenLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Proposes tests that could fail · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for card servicing at Clearbank. Write the PRD for self-serve disputes in the app, as Imogen asks below. It goes to the squad (engineering, design, support operations and compliance) who'll build it next quarter. 1,000 to 1,400 words. The dispute export and the background are attached.

What the model was given6 items: About Clearbank, imogen_note.md, how_disputes_work_today.md, regulator_rules.md, disputes_sample_last_quarter.csv (a random sample of 400 of last quarter's 2,400 disputes), merchant_data_option.md
About ClearbankA digital bank for individuals: current accounts and debit cards, run through a mobile app.
imogen_note.md1 lines · Download
From Imogen (Head of Product): "Write the PRD for self-serve disputes in the app. Support says it's their biggest cost after card freezes. The PRD goes to the squad (engineering, design, support ops and compliance) who will build it next quarter. 1,000 to 1,400 words."
how_disputes_work_today.md5 lines · Download
# How disputes work today

Customers raise a dispute by chatting to support. An agent collects details, then either explains the payment or opens a chargeback. Agents spend about 11 minutes per dispute. 2,400 disputes last quarter; 14% of them were resolved later than the regulator's deadline.

Card transactions show the merchant's billing name (for example "SQ *BRKLYN BAGEL 0412"), not its trading name or logo.
regulator_rules.md5 lines · Download
# Regulator's rules (summary from Legal)

- Acknowledge a dispute within 2 business days.
- Resolve genuine disputes within 15 business days, or tell the customer why it's taking longer.
- Don't make it harder to raise a dispute than to make a payment.
disputes_sample_last_quarter.csv (a random sample of 400 of last quarter's 2,400 disputes)dispute,customer_said,days_to_resolve,outcome CB-40000,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40006,I didn't receive what I paid for,17,Chargeback lost CB-40012,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40018,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40024,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40030,I didn't receive what I paid for,11,Chargeback lost CB-40036,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40042,I didn't receive what I paid for,16,Chargeback lost CB-40048,I was charged twice,5,Not a duplicate: two separate purchases CB-40054,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40060,I was charged twice,10,Not a duplicate: two separate purchases CB-40066,I was charged twice,7,Not a duplicate: two separate purchases CB-40072,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40078,I don't recognise this payment,5,Fraud confirmed CB-40084,I don't recognise this payment,7,Fraud confirmed CB-40090,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40096,I was charged twice,6,Not a duplicate: two separate purchases CB-40102,I didn't receive what I paid for,7,Chargeback won CB-40108,I don't recognise this payment,6,Fraud confirmed CB-40114,I was charged twice,9,Duplicate refunded CB-40120,I was charged twice,7,Not a duplicate: two separate purchases CB-40126,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40132,I was charged twice,9,Duplicate refunded CB-40138,Subscription I cancelled,18,Chargeback won CB-40144,I didn't receive what I paid for,21,Chargeback won CB-40150,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40156,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40162,I don't recognise this payment,6,Fraud confirmed CB-40168,I was charged twice,4,Not a duplicate: two separate purchases CB-40174,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40180,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-40186,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40192,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-40198,I don't recognise this payment,1,Fraud confirmed CB-40204,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40210,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40216,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40222,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40228,I didn't receive what I paid for,29,Chargeback won CB-40234,I don't recognise this payment,8,Fraud confirmed CB-40240,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40246,I didn't receive what I paid for,10,Merchant refunded CB-40252,I don't recognise this payment,4,Fraud confirmed CB-40258,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40264,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40270,I didn't receive what I paid for,21,Chargeback won CB-40276,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40282,I didn't receive what I paid for,17,Merchant refunded CB-40288,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40294,I don't recognise this payment,1,Fraud confirmed CB-40300,I don't recognise this payment,7,Fraud confirmed CB-40306,I don't recognise this payment,8,Fraud confirmed CB-40312,I didn't receive what I paid for,20,Merchant refunded CB-40318,I didn't receive what I paid for,26,Chargeback lost CB-40324,Subscription I cancelled,10,Merchant refunded CB-40330,I don't recognise this payment,1,Fraud confirmed CB-40336,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-40342,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40348,Fraud: I didn't make this payment,9,"Fraud confirmed, refunded" CB-40354,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40360,I was charged twice,9,Not a duplicate: two separate purchases CB-40366,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40372,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40378,I don't recognise this payment,8,Fraud confirmed CB-40384,I didn't receive what I paid for,14,Merchant refunded CB-40390,I don't recognise this payment,7,Fraud confirmed CB-40396,I don't recognise this payment,8,Fraud confirmed CB-40402,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40408,I was charged twice,5,Not a duplicate: two separate purchases CB-40414,I didn't receive what I paid for,13,Chargeback lost CB-40420,Subscription I cancelled,11,Chargeback won CB-40426,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-40432,I didn't receive what I paid for,13,Chargeback won CB-40438,Fraud: I didn't make this payment,4,"Fraud confirmed, refunded" CB-40444,I was charged twice,9,Duplicate refunded CB-40450,I didn't receive what I paid for,18,Chargeback won CB-40456,I don't recognise this payment,1,Fraud confirmed CB-40462,I don't recognise this payment,4,Fraud confirmed CB-40468,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40474,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40480,Fraud: I didn't make this payment,8,"Fraud confirmed, refunded" CB-40486,I didn't receive what I paid for,27,Chargeback lost CB-40492,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40498,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40504,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40510,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40516,I don't recognise this payment,5,Fraud confirmed CB-40522,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40528,Fraud: I didn't make this payment,8,"Fraud confirmed, refunded" CB-40534,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40540,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40546,I don't recognise this payment,2,Fraud confirmed CB-40552,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40558,I was charged twice,5,Duplicate refunded CB-40564,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40570,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40576,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40582,I didn't receive what I paid for,14,Chargeback lost CB-40588,I didn't receive what I paid for,18,Chargeback lost CB-40594,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40600,Fraud: I didn't make this payment,2,"Fraud confirmed, refunded" CB-40606,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40612,I don't recognise this payment,3,Fraud confirmed CB-40618,Fraud: I didn't make this payment,5,"Fraud confirmed, refunded" CB-40624,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40630,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40636,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40642,Fraud: I didn't make this payment,11,"Fraud confirmed, refunded" CB-40648,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40654,I don't recognise this payment,5,Fraud confirmed CB-40660,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40666,I don't recognise this payment,6,Fraud confirmed CB-40672,I was charged twice,5,Not a duplicate: two separate purchases CB-40678,Subscription I cancelled,10,Chargeback won CB-40684,Fraud: I didn't make this payment,9,"Fraud confirmed, refunded" CB-40690,Subscription I cancelled,7,Chargeback won CB-40696,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40702,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40708,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40714,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40720,I was charged twice,13,Not a duplicate: two separate purchases CB-40726,Subscription I cancelled,13,Chargeback won CB-40732,I don't recognise this payment,3,Fraud confirmed CB-40738,I didn't receive what I paid for,18,Chargeback lost CB-40744,Fraud: I didn't make this payment,8,"Fraud confirmed, refunded" CB-40750,I was charged twice,11,Not a duplicate: two separate purchases CB-40756,Subscription I cancelled,10,Merchant refunded CB-40762,I was charged twice,7,Duplicate refunded CB-40768,Subscription I cancelled,18,Chargeback won CB-40774,I didn't receive what I paid for,23,Chargeback won CB-40780,I didn't receive what I paid for,7,Chargeback won CB-40786,Subscription I cancelled,5,Chargeback won CB-40792,I didn't receive what I paid for,10,Merchant refunded CB-40798,I was charged twice,9,Duplicate refunded CB-40804,I didn't receive what I paid for,24,Merchant refunded CB-40810,I don't recognise this payment,2,Fraud confirmed CB-40816,Fraud: I didn't make this payment,11,"Fraud confirmed, refunded" CB-40822,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40828,Subscription I cancelled,13,Chargeback won CB-40834,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40840,I don't recognise this payment,7,Fraud confirmed CB-40846,Subscription I cancelled,7,Merchant refunded CB-40852,I don't recognise this payment,3,Fraud confirmed CB-40858,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40864,I was charged twice,4,Not a duplicate: two separate purchases CB-40870,I don't recognise this payment,8,Fraud confirmed CB-40876,I was charged twice,6,Not a duplicate: two separate purchases CB-40882,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-40888,I don't recognise this payment,8,Fraud confirmed CB-40894,I don't recognise this payment,6,Fraud confirmed CB-40900,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40906,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40912,I don't recognise this payment,8,Fraud confirmed CB-40918,I didn't receive what I paid for,21,Chargeback won CB-40924,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-40930,Fraud: I didn't make this payment,8,"Fraud confirmed, refunded" CB-40936,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40942,I was charged twice,6,Duplicate refunded CB-40948,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40954,I didn't receive what I paid for,14,Chargeback won CB-40960,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40966,I was charged twice,4,Not a duplicate: two separate purchases CB-40972,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40978,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40984,I didn't receive what I paid for,9,Merchant refunded CB-40990,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40996,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41002,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41008,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41014,Fraud: I didn't make this payment,2,"Fraud confirmed, refunded" CB-41020,I didn't receive what I paid for,27,Chargeback won CB-41026,I didn't receive what I paid for,28,Chargeback won CB-41032,I was charged twice,7,Duplicate refunded CB-41038,I don't recognise this payment,2,Fraud confirmed CB-41044,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41050,I don't recognise this payment,4,Fraud confirmed CB-41056,Fraud: I didn't make this payment,11,"Fraud confirmed, refunded" CB-41062,I was charged twice,3,Duplicate refunded CB-41068,I don't recognise this payment,1,Fraud confirmed CB-41074,I don't recognise this payment,1,Fraud confirmed CB-41080,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41086,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41092,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41098,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41104,I didn't receive what I paid for,29,Merchant refunded CB-41110,Subscription I cancelled,16,Chargeback won CB-41116,I didn't receive what I paid for,22,Chargeback won CB-41122,I didn't receive what I paid for,13,Chargeback lost CB-41128,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41134,I was charged twice,10,Not a duplicate: two separate purchases CB-41140,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41146,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41152,I was charged twice,5,Duplicate refunded CB-41158,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41164,I didn't receive what I paid for,12,Chargeback lost CB-41170,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41176,Fraud: I didn't make this payment,2,"Fraud confirmed, refunded" CB-41182,I don't recognise this payment,1,Fraud confirmed CB-41188,I don't recognise this payment,3,Fraud confirmed CB-41194,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41200,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41206,Fraud: I didn't make this payment,5,"Fraud confirmed, refunded" CB-41212,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41218,Fraud: I didn't make this payment,6,"Fraud confirmed, refunded" CB-41224,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41230,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41236,I don't recognise this payment,3,Fraud confirmed CB-41242,Subscription I cancelled,9,Merchant refunded CB-41248,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41254,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41260,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41266,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41272,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41278,I didn't receive what I paid for,11,Chargeback lost CB-41284,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41290,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41296,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-41302,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41308,I was charged twice,11,Duplicate refunded CB-41314,I didn't receive what I paid for,27,Chargeback won CB-41320,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41326,Fraud: I didn't make this payment,5,"Fraud confirmed, refunded" CB-41332,I was charged twice,12,Not a duplicate: two separate purchases CB-41338,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-41344,I didn't receive what I paid for,17,Merchant refunded CB-41350,I didn't receive what I paid for,21,Chargeback won CB-41356,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41362,Subscription I cancelled,14,Merchant refunded CB-41368,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41374,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41380,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41386,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41392,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41398,I don't recognise this payment,2,Fraud confirmed CB-41404,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41410,I don't recognise this payment,3,Fraud confirmed CB-41416,I don't recognise this payment,5,Fraud confirmed CB-41422,I was charged twice,12,Duplicate refunded CB-41428,Subscription I cancelled,9,Merchant refunded CB-41434,I was charged twice,6,Duplicate refunded CB-41440,Subscription I cancelled,15,Merchant refunded CB-41446,I don't recognise this payment,5,Fraud confirmed CB-41452,I don't recognise this payment,7,Fraud confirmed CB-41458,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41464,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41470,I was charged twice,6,Not a duplicate: two separate purchases CB-41476,I don't recognise this payment,5,Fraud confirmed CB-41482,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41488,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41494,I didn't receive what I paid for,26,Merchant refunded CB-41500,I didn't receive what I paid for,24,Chargeback won CB-41506,Subscription I cancelled,20,Chargeback won CB-41512,I didn't receive what I paid for,29,Merchant refunded CB-41518,Subscription I cancelled,20,Chargeback won CB-41524,I don't recognise this payment,8,Fraud confirmed CB-41530,I was charged twice,3,Not a duplicate: two separate purchases CB-41536,I didn't receive what I paid for,8,Chargeback won CB-41542,I don't recognise this payment,2,Fraud confirmed CB-41548,I don't recognise this payment,2,Fraud confirmed CB-41554,I was charged twice,12,Not a duplicate: two separate purchases CB-41560,Subscription I cancelled,6,Chargeback won CB-41566,I was charged twice,3,Not a duplicate: two separate purchases CB-41572,I don't recognise this payment,1,Fraud confirmed CB-41578,Fraud: I didn't make this payment,7,"Fraud confirmed, refunded" CB-41584,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41590,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-41596,Subscription I cancelled,16,Merchant refunded CB-41602,Subscription I cancelled,15,Merchant refunded CB-41608,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41614,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41620,I was charged twice,5,Duplicate refunded CB-41626,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41632,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41638,Fraud: I didn't make this payment,3,"Fraud confirmed, refunded" CB-41644,I don't recognise this payment,3,Fraud confirmed CB-41650,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41656,I don't recognise this payment,6,Fraud confirmed CB-41662,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41668,I didn't receive what I paid for,12,Chargeback won CB-41674,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41680,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41686,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-41692,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41698,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41704,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41710,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41716,Fraud: I didn't make this payment,8,"Fraud confirmed, refunded" CB-41722,I don't recognise this payment,3,Fraud confirmed CB-41728,I was charged twice,12,Duplicate refunded CB-41734,Fraud: I didn't make this payment,5,"Fraud confirmed, refunded" CB-41740,I don't recognise this payment,6,Fraud confirmed CB-41746,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41752,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41758,I was charged twice,13,Duplicate refunded CB-41764,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41770,I didn't receive what I paid for,23,Chargeback won CB-41776,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41782,I didn't receive what I paid for,27,Chargeback won CB-41788,I didn't receive what I paid for,23,Chargeback lost CB-41794,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41800,I don't recognise this payment,3,Fraud confirmed CB-41806,I don't recognise this payment,1,Fraud confirmed CB-41812,I was charged twice,4,Not a duplicate: two separate purchases CB-41818,I don't recognise this payment,8,Fraud confirmed CB-41824,I didn't receive what I paid for,18,Chargeback won CB-41830,Subscription I cancelled,13,Chargeback won CB-41836,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41842,I don't recognise this payment,7,Fraud confirmed CB-41848,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41854,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41860,I was charged twice,5,Duplicate refunded CB-41866,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41872,I don't recognise this payment,8,Fraud confirmed CB-41878,I don't recognise this payment,2,Fraud confirmed CB-41884,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41890,I was charged twice,7,Not a duplicate: two separate purchases CB-41896,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41902,Fraud: I didn't make this payment,9,"Fraud confirmed, refunded" CB-41908,I don't recognise this payment,4,Fraud confirmed CB-41914,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41920,I didn't receive what I paid for,24,Merchant refunded CB-41926,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41932,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-41938,Subscription I cancelled,15,Chargeback won CB-41944,I didn't receive what I paid for,13,Chargeback won CB-41950,I don't recognise this payment,2,Fraud confirmed CB-41956,I didn't receive what I paid for,24,Chargeback lost CB-41962,I don't recognise this payment,4,Fraud confirmed CB-41968,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41974,Subscription I cancelled,15,Chargeback won CB-41980,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41986,Fraud: I didn't make this payment,2,"Fraud confirmed, refunded" CB-41992,Fraud: I didn't make this payment,6,"Fraud confirmed, refunded" CB-41998,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-42004,I didn't receive what I paid for,12,Merchant refunded CB-42010,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42016,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42022,I was charged twice,11,Duplicate refunded CB-42028,I didn't receive what I paid for,25,Merchant refunded CB-42034,I didn't receive what I paid for,10,Merchant refunded CB-42040,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42046,I was charged twice,8,Duplicate refunded CB-42052,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-42058,I don't recognise this payment,7,Fraud confirmed CB-42064,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-42070,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42076,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-42082,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-42088,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-42094,Subscription I cancelled,12,Chargeback won CB-42100,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-42106,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-42112,Subscription I cancelled,18,Chargeback won CB-42118,Fraud: I didn't make this payment,6,"Fraud confirmed, refunded" CB-42124,I don't recognise this payment,5,Fraud confirmed CB-42130,I didn't receive what I paid for,14,Chargeback lost CB-42136,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-42142,Fraud: I didn't make this payment,2,"Fraud confirmed, refunded" CB-42148,I was charged twice,12,Not a duplicate: two separate purchases CB-42154,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-42160,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-42166,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42172,Fraud: I didn't make this payment,9,"Fraud confirmed, refunded" CB-42178,I don't recognise this payment,3,Fraud confirmed CB-42184,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-42190,I was charged twice,7,Duplicate refunded CB-42196,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-42202,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-42208,I didn't receive what I paid for,21,Chargeback won CB-42214,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-42220,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-42226,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-42232,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-42238,I was charged twice,10,Duplicate refunded CB-42244,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-42250,Subscription I cancelled,11,Chargeback won CB-42256,Fraud: I didn't make this payment,3,"Fraud confirmed, refunded" CB-42262,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-42268,I didn't receive what I paid for,16,Merchant refunded CB-42274,I was charged twice,3,Duplicate refunded CB-42280,Fraud: I didn't make this payment,9,"Fraud confirmed, refunded" CB-42286,I was charged twice,11,Duplicate refunded CB-42292,I was charged twice,9,Not a duplicate: two separate purchases CB-42298,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-42304,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-42310,Subscription I cancelled,16,Merchant refunded CB-42316,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-42322,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-42328,Fraud: I didn't make this payment,3,"Fraud confirmed, refunded" CB-42334,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-42340,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-42346,I didn't receive what I paid for,19,Merchant refunded CB-42352,I was charged twice,12,Duplicate refunded CB-42358,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-42364,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-42370,I don't recognise this payment,3,Fraud confirmed CB-42376,I didn't receive what I paid for,17,Chargeback won CB-42382,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-42388,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42394,I don't recognise this payment,5,Customer recognised it once shown the merchant's details
merchant_data_option.md3 lines · Download
# Merchant data we could use

Our card processor offers an enriched merchant feed (trading name, logo, location, website) for 94% of transactions, at $0.0008 per transaction. We process about 9 million card transactions a month.
What a strong answer doesThe answer key the graders mark against

Starts from the data: in the sample, 232 of 400 disputes (58%, so about 1,400 of the quarter's 2,400) were 'I don't recognise this payment', and 169 of those (about three in four) were resolved simply by showing the merchant's details; so the first thing to build is recognisable transactions (trading name, logo, location from the enriched feed, about $7,200 a month for 9 million transactions), shown before someone raises a dispute, with self-serve disputes for the genuine cases. Defines success with targets (disputes per 1,000 transactions, share resolved without an agent, regulator deadlines met against 14% late today) and says which wins when they conflict (never make disputing harder than paying, per the regulator). Covers the flows for genuine disputes (charged twice, not received, cancelled subscription, fraud) with acknowledgement within 2 business days and resolution within 15, what happens when the merchant data is missing (6% of transactions), and how the rollout reaches people, with support ready. Says who it's not for (for example business accounts, or disputes over a set amount that still go to an agent) and what's out of scope.

Critical failures (cap the score at 40)
  • Adds steps that make raising a dispute harder than making a payment
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Sonnet 5.5API83Usable with a quick editGPT-6 AstraChatGPT73Needs more workGPT-6 LunaAPI69Needs more workHaiku 5.5API69Needs more workOpus 5.5Claude60Needs more workGPT-6.1 SolAPI58Needs more workGemini 3.5 Flash-LiteGemini40Needs more workGemini 3.8 FlashAPI37Needs more work
Haiku 5.5API

Needs more than a quick edit8 of 12 checks passed

The PRD lacks user exclusion, discovery plan, and test thresholds, requiring rework on those sections before a squad could act.

Output

PRD: Self-serve disputes in the app

Owner: Card servicing PM Audience: Engineering, Design, Support Operations, Compliance Status: Draft for squad review Target: Next quarter

1. Problem

Customers raise disputes by chatting with support. An agent collects the details, then either explains the payment or opens a chargeback. Each dispute takes about 11 agent minutes. Support tells us disputes are their biggest cost after card freezes. Last quarter we handled 2,400 disputes, and 14% were resolved after the regulator's deadline.

2. What the data says

I reviewed a random sample of 400 of last quarter's disputes. Percentages are accurate to about ±5 points at 95% confidence.

What the customer saidShare of sampleWhat happened
"I don't recognise this payment"58% (233)171 ended with the customer recognising the payment once shown the merchant's details. 62 were confirmed fraud. All closed within 8 days.
"I was charged twice"12% (48)24 were two separate purchases. 24 were true duplicates, refunded.
"Fraud: I didn't make this payment"9% (35)All confirmed fraud and refunded, in 2 to 11 days.
"I didn't receive what I paid for"14% (56)Merchant-dependent: 24 chargebacks won, 17 merchant refunds, 15 chargebacks lost.
"Subscription I cancelled"7% (28)Merchant-dependent: mostly won or refunded.

The main finding: the largest group is a recognition problem, not a dispute problem. Card transactions show the merchant's billing name (for example, "SQ \*BRKLYN BAGEL 0412"). Customers don't recognise it, contact us, and an agent spends time showing them the merchant details the app should have shown in the first place. Roughly half of "charged twice" cases are the same confusion: two separate purchases.

The late cases are a different problem. In the sample, 43 of 400 disputes took more than 15 days as recorded in the export. 36 of those were "didn't receive" disputes, which depend on merchant responses. Self-serve intake won't fix these by itself. They need proactive status updates so customers aren't left waiting. (Legal to confirm whether the export counts business or calendar days.)

Fraud is a minority but time-critical. About 24% of the sample was confirmed fraud. These customers need to freeze the card first, and the dispute should follow from that.

3. Goals

  1. Customers can raise, track, and close any card dispute in the app without starting a chat.
  2. Recognition and duplicate cases close without an agent wherever possible.
  3. Fewer disputes miss the regulator's deadline. Proposed target: under 5%, down from 14%.
  4. Disputing is never harder than paying. This is a regulatory requirement, not just a design goal.

Non-goals for this release: - Changing chargeback decision rules or card scheme processes. - Self-serve approval of chargebacks. Agents still decide outcomes for "didn't receive" and subscription disputes, which need merchant contact. Customers can still submit the case in-app. - Redesigning card freezes. That is a separate PRD, although fraud disputes link into it.

4. Requirements

R1: Show the real merchant on every card transaction. Use the processor's enriched feed (trading name, logo, location, website). It covers 94% of transactions at $0.0008 each, about $7,200 a month at our 9 million transactions. For the remaining 6%, show the billing name with a plain-English note that the merchant name couldn't be confirmed.

R2: "Do you recognise this?" as the first step. Before the dispute form, show the enriched merchant details and ask whether the customer recognises the payment. If yes, close the dispute with a confirmation and no agent involvement. If no, one tap continues to the dispute. There must be no dead end, and the wording must be approved by Compliance, because a recognition screen could discourage valid disputes.

R3: Structured intake for each dispute type. Customers choose from the five categories. Each category asks only the questions it needs. For "charged twice," show the matching transactions side by side and ask whether they're for the same thing.

R4: Automatic duplicate handling (conditional). If two charges for the same amount at the same merchant appear within a short window, offer an automatic refund request. This is conditional on Support Ops and Compliance agreeing the rules and thresholds.

R5: Fraud goes to freeze first. "I didn't make this payment" offers an immediate card freeze and replacement, then starts the dispute with the transaction already filled in. Customers must never have to chat before they can freeze their card.

R6: Automatic acknowledgement. Confirm receipt in-app and by push the moment a dispute is submitted. This comfortably meets the 2-business-day rule.

R7: Status and deadlines. Show a timeline for each dispute. Proactively notify the customer at 10 business days (proposed) if it's still open. If it passes 15 business days, show the reason and the expected date. This is what the regulator's "or tell the customer why" requirement looks like in practice.

R8: Evidence upload. Let customers attach receipts, screenshots, or messages to "didn't receive" and subscription disputes.

R9: Chat is always available. "Talk to someone" stays visible at every step. Any hand-off carries the structured answers, so the agent never asks the customer to repeat themselves.

5. Regulatory constraints

  • Acknowledge within 2 business days; resolve within 15 business days or explain the delay. R6 and R7 cover both.
  • Disputing must not be harder than paying. Compliance should define a benchmark, such as the number of taps and screens for a typical card payment, and we test the dispute flow against it. R2 must always offer a one-tap route forward.
  • The recognition screen must not discourage valid disputes. Compliance signs off on the copy and the layout before launch.

6. Success metrics

MetricBaselineProposed target
Disputes closed without an agent0%40% of disputes within one quarter of launch
Agent minutes per agent-handled dispute~11Under 5
Disputes resolved after deadline14%Under 5%
Repeat disputes on the same transaction within 30 daysTo measureNot increasing
Time from fraud report to card freezeTo measureUnder 1 minute
Dispute-related complaintsTo measureNot increasing

Guardrails: payment completion rate and card-freeze volume must not drop. Engineering should instrument the baselines marked "to measure" before the build starts.

7. Rollout

  • Phase 1: Merchant details (R1) and the recognition screen (R2), behind a feature flag. Run an A/B test with a holdout group. The 171 recognition closures in the sample were only observed after an agent showed the merchant details. The test checks whether the app-only version produces the same result.
  • Phase 2: Structured intake for all categories (R3, R5, R8), automatic acknowledgement and tracking (R6, R7), and the agent hand-off (R9).
  • Phase 3: Automatic duplicate handling (R4), if approved.

8. Open questions and risks

  1. Does the 43% recognition rate hold? The sample shows what happened after agents explained the payments. It doesn't show what customers do when they see the merchant details themselves. Phase 1 tests this.
  2. Will self-serve increase dispute volume? Some customers who used to ask a question may now file a dispute. That is acceptable, but we need to watch cost per dispute, not just the count.
  3. Feed quality. Support should check a sample of enriched merchant names for accuracy before launch. An incorrect trading name is worse than the billing name.
  4. Logo use. Legal should confirm we can display merchant logos.
  5. Card scheme deadlines. Our pack doesn't include scheme chargeback timeframes. Legal and Ops need to confirm them, as they may limit how long R7 can promise.
  6. Days field. Confirm whether `daystoresolve` counts business or calendar days. It changes how we read the late-case figures.

Grades and run details

Decision model 75 · LLM judge 69
Decision model checks
  • failUses the supplied evidence correctly18%
  • passAddresses the actual decision70%
  • partialRespects explicit constraints49%
  • passIdentifies material uncertainty97%
  • partialAvoids unsupported claims31%
  • passProduces the required deliverable35%
  • passFinds that most disputes are recognition100%
  • passBuilds the regulator's rules in98%
  • passSuccess settles trade-offs97%
  • partialProposes tests that could fail74%
  • partialSays who it's for, and who it isn't62%
  • passPlans how people will find it31%
Run
Run
#1
API response time
78 s
Submitted
8 Oct 2026
Opus 5.5Claude

Needs more than a quick edit6 of 12 checks passed

The PRD contains an inconsistent statistic (42% vs 48%) and lacks explicit trade-off rules, user definition, and rollout plan, requiring more than a quick edit.

Output

PRD: Self-serve card disputes in the app

Owner: PM, Card Servicing · Squad: Engineering, Design, Support Ops, Compliance · Target: next quarter

1. Summary

Customers can only raise a card dispute today by chatting with support. An agent spends about 11 minutes on each one. Last quarter there were 2,400 disputes, which is roughly 440 agent hours, and 14% were resolved after the regulator's deadline.

This project lets customers raise and track disputes from the transaction screen. It answers the most common question ("what is this payment?") before a dispute is needed. Every case that does need an agent arrives fully documented.

The most important finding: 42% of disputes aren't really disputes. The customer didn't recognise a merchant billing name. However, the late-resolution problem sits in a different group of cases, so the project has to solve both.

2. What the data says

These figures come from a random sample of 400 of last quarter's 2,400 disputes.

Customer saidShareOutcomesResolved in >15 days
I don't recognise this payment58% (232)169 recognised it once shown merchant details; 63 fraud0
I didn't receive what I paid for14% (56)24 chargeback won, 15 lost, 17 merchant refunded36 (64%)
I was charged twice12% (48)24 genuine duplicates, 24 separate purchases0
Fraud: I didn't make this payment9% (36)All confirmed and refunded0
Subscription I cancelled7% (28)18 chargeback won, 10 merchant refunded8 (29%)

What this means:

  1. Most disputes can be answered with information. 169 "don't recognise" cases plus 24 "not a duplicate" cases make 48% of disputes. These ended with the customer understanding a payment they had made. The cause is that we show billing names like "SQ \*BRKLYN BAGEL 0412" instead of the trading name.
  2. "Don't recognise" is also a real fraud channel. 27% of those cases were fraud. Fraud from both reasons combined is about 25% of all disputes. Any feature that explains payments must never slow down a customer who is genuinely a victim.
  3. Late resolutions are all chargebacks. Every sample case that took more than 15 days was a "not received" or "subscription" dispute. Deflecting unrecognised payments will cut agent time, but it won't fix the deadline problem. That needs its own work (section 4.5).
  4. Many merchant disputes end with the merchant refunding. This happened in 30% of "not received" cases and 36% of subscription cases.

3. Goals and non-goals

Goals - Customers can raise any dispute in the app, with no chat required. - Fewer disputes raised for payments the customer actually made. - Every chargeback case is either resolved within 15 business days or the customer is told why not, with no exceptions. - Less agent time per dispute.

Non-goals (v1) - Automated chargeback decisions or automatic refunds. Agents still decide every case. - Disputes on non-card payments. - Changing how chargebacks are filed with the card scheme.

4. Requirements

4.1 Entry point

  • R1. Every card transaction's detail screen has a "Problem with this payment?" action.
  • R2. The customer picks one of five reasons, matching today's categories: don't recognise it, I didn't make it (fraud), charged twice, didn't receive it, cancelled subscription.
  • R3. Compliance constraint. Raising a dispute must never take more steps than making a payment. Design will count taps for each path against our card payment and transfer flows. Compliance must sign off on the counts before launch. Every explanation screen below is shown alongside the dispute action, never in front of it.

4.2 "I don't recognise this payment"

  • R4. Show enriched merchant details: trading name, logo, location and website. Also show the customer's previous payments to the same merchant.
  • R5. The same screen offers two equally weighted buttons: "I recognise it now" and "I still don't recognise it".
  • R6. "I still don't recognise it" goes straight to the fraud path (4.3). It asks no further questions.
  • R7. If no enrichment exists (about 6% of transactions), show the raw billing name, merchant category and location from the transaction record. The flow is otherwise unchanged.

4.3 Fraud

  • R8. Offer to freeze the card on the first screen of the fraud path. The customer can decline.
  • R9. Let the customer flag other transactions on the same card as unauthorised in the same submission.
  • R10. The case goes to the fraud queue at high priority.

4.4 Charged twice / not received / subscription

  • R11. Charged twice. Show the two transactions side by side, with time, amount, location and card-present status. Half of these cases were separate purchases. The customer can still continue to raise the dispute.
  • R12. Not received / subscription. Ask structured questions instead of using a chat:
  • expected delivery date or cancellation date
  • whether the customer has contacted the merchant
  • optional uploads (cancellation email, receipt, screenshots)

These questions should capture everything an agent needs for a chargeback. - R13. Show the merchant's website and suggest contacting them, since merchants refunded in about a third of cases. This is optional and must not delay submission.

4.5 Acknowledgement, tracking and deadlines

  • R14. Submitting a dispute triggers an in-app and email acknowledgement immediately, with a case reference. This meets the 2-business-day rule automatically.
  • R15. The customer gets a dispute status screen showing:
  • the reason they gave
  • current stage
  • expected resolution date
  • any temporary credit
  • R16. The system counts business days from submission. At business day 10, any open case gets flagged in the agent queue. By business day 14, the agent must choose a delay reason, for example "awaiting merchant response to chargeback". The customer is then notified with that reason in plain language. If no reason is chosen, the case escalates to the team lead.
  • R17. Every notification is logged against the case for audit.

4.6 Support operations

  • R18. In-app cases enter the existing dispute queue already populated with:
  • the reason
  • the answers to the structured questions
  • attachments
  • which explanation screens the customer saw
  • R19. Chat stays available. Agents can start the same structured flow on a customer's behalf, so every case looks the same in the queue.

5. Merchant data feed: cost needs resolving before build

The processor's enriched feed covers 94% of transactions at $0.0008 each. On all ~9M monthly transactions that is $7,200/month, or $21,600/quarter.

The feed is very unlikely to pay for itself through dispute savings alone. Using the sample rates, about 1,014 disputes a quarter were "recognised once shown details". At 11 minutes each, that is around 186 agent hours. To save $21,600, an agent hour would have to cost over $115, and that assumes every one of those disputes is prevented.

The feed may still be worth it. Clearer names on every transaction could reduce "what is this?" chats that never became disputes, and that number isn't in this data. But the full-feed price needs a business case of its own.

Action: Engineering and the PM will ask the processor whether enrichment can be bought per lookup. We would only call it when a customer opens a transaction or starts a dispute. At that volume the cost would be trivial. If per-lookup isn't available, I'll bring the full-feed decision to Imogen with Support's numbers on unrecognised-payment chats.

6. Success metrics

MetricBaselineTarget (one quarter after launch)
Disputes raised via app (vs chat)0%≥70%
"Don't recognise" disputes ending as "customer recognised it"73% of that reason<25% (resolved earlier by R4/R5 instead)
Chargeback cases past 15 business days with no delay noticeUnknown0
Disputes resolved after the deadline14%<5%
Agent minutes per dispute11≤5

Guardrails. These protect customers from being deflected when they shouldn't be: - Fraud reported late. Track fraud reported within 60 days on transactions where the customer tapped "I recognise it now". Any increase triggers a design review. - Abandoned flows. Track the rate of dispute flows started but not submitted, broken down by reason. - Tap counts. Keep tap-count parity with the payment flow (R3) and recheck it on every release.

7. Open questions

  1. Units of `daystoresolve`. Are these business or calendar days? The analysis above assumes business days. If they are calendar days, fewer cases breach the deadline.
  2. Gap between 11% and 14%. The sample shows 11% of cases over 15 days, while the background doc says 14%. This could be sampling noise. It could also mean cases are being counted late when a delay notice was sent, or not sent. Support Ops, please confirm how the 14% is calculated.
  3. Refunds on "don't recognise" fraud cases. These are recorded as "Fraud confirmed", while direct fraud claims are recorded as "Fraud confirmed, refunded". Compliance, please confirm all of those customers were refunded.
  4. Feed pricing. Is per-lookup pricing for the merchant feed available (section 5)?
  5. Temporary credit policy. Should we credit the customer while a chargeback is pending, and when? Compliance and Finance to decide before design finalises R15.

Claims the judge couldn’t find in the brief

  • 42% of disputes aren't really disputes.

Grades and run details

Decision model 79 · LLM judge 46
Decision model checks
  • passUses the supplied evidence correctly19%
  • passAddresses the actual decision36%
  • partialRespects explicit constraints58%
  • passIdentifies material uncertainty98%
  • partialAvoids unsupported claims36%
  • passProduces the required deliverable28%
  • passFinds that most disputes are recognition86%
  • passBuilds the regulator's rules in95%
  • passSuccess settles trade-offs99%
  • partialProposes tests that could fail77%
  • partialSays who it's for, and who it isn't36%
  • partialPlans how people will find it77%
Run
Run
#1
Time to output
1.9 min
Submitted
6 Oct 2026
Gemini 3.8 FlashAPI

Needs more than a quick edit3 of 12 checks passed

The PRD contains factual errors in the sample analysis, omits key scope exclusions, lacks a rollout plan, and does not address trade-offs—requiring more than light edits before a squad could act on it.

Output

Product Requirements Document (PRD): Self-Serve Card Disputes

Author: PM, Card Servicing Audience: Squad (Engineering, Product Design, Support Operations, Compliance) Target Delivery: Q3 Status: Ready for Review

---

1. Executive Summary & Problem Context

At Clearbank, disputing a card transaction currently requires initiating an in-app chat with Support. Agents spend an average of 11 minutes per dispute manually gathering transaction details, explaining billing descriptors, or submitting chargebacks. Last quarter, Support handled 2,400 disputes (~440 agent hours), making disputes the single largest operating cost after card freezes. Crucially, 14% of disputes breached the regulator’s 15-business-day resolution window, exposing Clearbank to regulatory enforcement.

An analysis of 400 sampled disputes from last quarter reveals four distinct problem drivers: 1. Unrecognised Merchant Descriptors (48.5% of sample): Customers dispute cryptic billing names (e.g., `SQ *BRKLYN BAGEL 0412`). In nearly every case, the customer abandoned the dispute once the agent clarified the merchant’s trading name and location. 2. Duplicate Charges (11.5% of sample): Split between genuine duplicate settlements and two separate purchases made in quick succession. 3. Merchant & Subscription Issues (19.8% of sample): Goods not received, defective services, or cancelled subscriptions. These drive the longest resolution cycles (often 18–29 days), causing the bulk of regulatory SLA breaches because customers were not notified when investigations ran past 15 business days. 4. Confirmed Fraud (20.2% of sample): Unauthorised card use requiring immediate card compromise handling (freeze/cancel) and immediate provisional credit or refund.

Strategic Objectives

  • Deflect ~45% of dispute volume before submission by surfacing clear merchant data and transaction context.
  • Eliminate manual intake time by replacing chat conversations with a structured, automated in-app submission flow.
  • Achieve 100% regulatory compliance across all three mandates: sub-2-day acknowledgement, automated interim updates before day 15, and an intake friction level lower than making a payment.

---

2. Regulatory Compliance Framework

Compliance is a non-negotiable architectural constraint. Every user journey and automated trigger must map directly to regulatory rules:

Regulatory RuleProduct ImplementationAcceptance Criteria
:---:---:---
1. Acknowledge within 2 business daysAutomated push notification, in-app inbox message, and email confirmation triggered immediately upon submission.Sent within < 60 seconds of dispute submission (Target: 100%).
2. Resolve within 15 business days or explain delayAutomated tracking engine monitors business days. At Day 10, if uncompleted, an interim update is automatically dispatched explaining why the merchant review is pending.0% of disputes cross Day 15 without a formal written status update.
3. Friction parity (Not harder to dispute than to pay)1-tap entry directly from the transaction detail screen; pre-filled details; maximum 3 guided screens to submit.Submission flow completable in < 90 seconds without entering chat.

---

3. Merchant Data Feed Integration: Decision & Business Case

Clearbank’s card processor offers an enriched merchant feed (clean trading name, consumer-facing logo, geolocation, and website) covering 94% of card transactions at $0.0008 per transaction.

  • Monthly Transaction Volume: ~9,000,000
  • Monthly Feed Cost: $7,200 ($21,600/quarter; $86,400/year)
  • Dispute Deflection Value: ~1,160 disputes/quarter avoided (48.5% of 2,400). At 11 minutes per dispute, this saves 213 agent hours quarterly. Beyond direct support savings, this reduces card reissuance costs from panicked users, improves customer trust, and lifts app engagement across all 9 million monthly transactions.
  • Recommendation: Adopt the feed platform-wide. Integrate enriched data into the core transaction feed as well as the dispute intake screens.

---

4. User Journeys & Functional Requirements

``` [Transaction Detail Screen] │ ▼ ["Dispute this payment"] │ ▼ ┌────────────────────────────────────────┐ │ Step 1: Enriched Context & Deflection │ │ - Clean name, logo, map, website │ │ - "Does this look familiar?" │ └───────────────────┬────────────────────┘ │ Still unrecognised / issue ▼ ┌────────────────────────────────────────┐ │ Step 2: Dynamic Dispute Categorisation│ │ 1. I don't recognise this (Fraud) │ │ 2. Charged twice / wrong amount │ │ 3. Issue with merchant/order │ │ 4. Cancelled subscription │ └───────────────────┬────────────────────┘ │ ▼ ┌────────────────────────────────────────┐ │ Step 3: Tailored Evidence & Actions │ │ - Fraud: Auto-freeze card toggle │ │ - Duplicate: Select other charge │ │ - Merchant: Date cancelled / receipts │ └───────────────────┬────────────────────┘ │ ▼ ┌────────────────────────────────────────┐ │ Step 4: Instant Confirmation & Tracker│ │ - Immediate legal acknowledgment │ │ - Live in-app SLA status timeline │ └────────────────────────────────────────┘ ```

4.1 Entry Point: Transaction Details

  • Located on every settled and pending card transaction detail view via a clear secondary action button: "Report a problem / Dispute".
  • Available only to the primary cardholder. Disabled for internal account transfers.

4.2 Step 1: Pre-Dispute Deflection Screen

  • Displays enriched merchant trading name, Google Maps pin of physical terminal or merchant domain, verified logo, and official contact phone/web details.
  • For transactions where the descriptor differs significantly from the trading name, display a helper badge: "Billing descriptor on file: [Raw String]".
  • Action buttons: "Yes, I recognise this" (exits flow) and "Continue dispute".

4.3 Step 2: Categorisation & Guided Dynamic Intake

If the customer continues, the flow asks: "What went wrong?"

#### Category A: "I suspect fraud / I didn't authorise this" * Requirements: * System prompts customer to confirm whether they possess the physical debit card. * Auto-recommends immediate card freeze via a pre-checked toggle: "Freeze this card now to prevent further unauthorized charges". * Collects date/time customer noticed the charge and any travel context. * System flags case as `FRAUDHIGHPRIORITY`.

#### Category B: "I was charged twice" * Requirements: * System queries transaction history for identical or near-identical amounts at the same merchant within ±7 days. * Presents potential matching duplicates. Customer selects the corresponding charge. * If the system detects two authorizations with distinct timestamps and receipt identifiers, it displays an educational alert: "We found two distinct purchases on this date. Did you intend to buy both?"

#### Category C: "Problem with merchant or order" (Goods not received / damaged) * Requirements: * Form requires: Expected delivery date, summary of merchant contact attempts, and merchant response. * Document upload capability (photos, delivery tracking screenshots, cancellation receipts). Accepted formats: PDF, JPG, PNG (up to 10MB).

#### Category D: "Cancelled subscription" * Requirements: * Form collects: Date subscription was cancelled, cancellation confirmation number or email screenshot, and merchant terms agreed to.

4.4 Step 3: Submission & Automated Regulatory Acknowledgement

  • Customer reviews structured summary and taps "Submit Dispute".
  • Engine Action:
  • Generates unique Dispute ID (e.g., `CB-DISP-XXXXX`).
  • Dispatches instant Push Notification, In-App Inbox confirmation, and transactional Email stating: receipt confirmation, regulatory 15-day timeline, and next steps.
  • Sets status to `ACKNOWLEDGED`.

4.5 Step 4: Self-Service Dispute Tracker

* A dedicated "Disputes & Claims" hub in the app profile and linked directly from the disputed transaction. * Features a visual 4-stage stepper: 1. Submitted & Acknowledged 2. Under Review by Clearbank 3. Awaiting Merchant Response (Chargeback Filed) 4. Resolved * Displays estimated resolution deadline (target: 15 business days).

---

5. Automated Regulatory SLA & Delay Notification Engine

To eradicate the 14% SLA breach rate, the backend service will run an automated business-day tracker against every active case:

``` Day 0: Dispute Submitted ────────> Instant Legal Acknowledgment Sent │ Day 10 (Business Days): Case Open? ───┴──> Automated Trigger: - Push Notification + Email - Explains ongoing merchant inquiry - Gives updated completion estimate │ Day 15 (Business Days): Case Open? ──────> SLA Escalation to Ops Lead + Final Legal Extension Notice ```

* Day 10 Trigger: If a case status is not `RESOLVED` by 10 business days, the engine triggers an automated update: > "Update on your dispute (ID: [X]): We are awaiting evidence from [Merchant Trading Name]. We anticipate concluding our review by [Date]. You do not need to take any action." * Day 15 Escalation: Any dispute approaching Day 14 without an outcome automatically escalates to a priority queue for Support Ops Leads.

---

6. Support Operations & Back-Office Integration

  • Automated Dossier Generation: The self-serve flow compiles customer inputs, transaction metadata, card telemetry (3DS status, chip vs. contactless), and uploaded files into a standardized chargeback packet in the Support CRM.
  • Straight-Through Processing (STP):
  • Claims under $25 classified as first-time fraud or verified double-billing auto-credit without manual agent intervention (subject to risk scoring).
  • Agent Impact: Reduces average handling time per dispute from 11 minutes to < 2 minutes (handling only non-STP reviews and representments).

---

7. Squad Implementation Plan & Technical Considerations

Engineering (Backend & Mobile)

  • Integrations: Card processor enriched merchant feed API, S3 secure document upload bucket, and event-driven notification service.
  • State Machine: Implement explicit state transitions: `SUBMITTED` -> `ACKNOWLEDGED` -> `OPSREVIEW` -> `CHARGEBACKOPEN` -> `RESOLVEDWON` / `RESOLVEDLOST` / `MERCHANT_REFUNDED`.
  • SLA Cron Job: Compute business days based on the bank calendar, excluding weekends and national holidays.

Product Design

  • Maintain cognitive ease: Progressive disclosure on evidence upload.
  • Mobile-first responsive camera capture for receipts.
  • Reassuring visual tone during the fraud journey.

Compliance & Legal Sign-Off

  • Mandatory review of all automated customer-facing notification templates.
  • Bi-weekly audit logging to prove sub-2-day acknowledgement and interim delay notices for regulatory inspection.

---

8. Success Metrics & KPIs

MetricCurrent BaselineTarget (Q3 Post-Launch)
:---:---:---
Disputes Requiring Agent Chat100% (2,400/qtr)< 15% (all direct self-serve)
Pre-Dispute Deflection (Unrecognised)0%≥ 40% deflection via enriched feed
Average Agent Handling Time11 minutes≤ 2 minutes (structured packets)
Regulatory Acknowledgment (<2 Days)Manual (~92%)100% (Automated <60 sec)
Regulatory SLA Breaches (>15 Days)14%0% (Automated delay notices)
Dispute Submission Time (Friction)~8–15 min chat< 90 seconds

Claims the judge couldn’t find in the brief

  • Unrecognised Merchant Descriptors represent 48.5% of the sampled disputes.
  • Duplicate Charges represent 11.5% of the sampled disputes.
  • Confirmed Fraud represents 20.2% of the sampled disputes.
  • The baseline regulatory acknowledgment rate is ~92% (manual).
  • Current dispute submission time (customer) is ~8–15 minutes in chat.

Grades and run details

Decision model 54 · LLM judge 23
Decision model checks
  • failUses the supplied evidence correctly73%
  • passAddresses the actual decision34%
  • partialRespects explicit constraints53%
  • failIdentifies material uncertainty54%
  • failAvoids unsupported claims65%
  • partialProduces the required deliverable60%
  • passFinds that most disputes are recognition40%
  • passBuilds the regulator's rules in99%
  • passSuccess settles trade-offs59%
  • partialProposes tests that could fail25%
  • partialSays who it's for, and who it isn't26%
  • partialPlans how people will find it87%
Run
Run
#1
API response time
42 s
Submitted
6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 7

Uses the supplied evidence correctlyMixedMixedWrong
Haiku 5.5 · API

All facts about the current situation are from the brief or derived from the sample; no invented facts.

Opus 5.5 · Claude

The output claims 42% of disputes aren't really disputes, but the sample data yields 48%; this fact about the current situation is incorrect and unsupported.

Gemini 3.8 Flash · API

Several facts about the sample (unrecognised 48.5% instead of 58%, duplicate 11.5% instead of 12%, fraud 20.2% instead of 25%) and a made-up baseline acknowledgment rate (~92%) are incorrect or unsupported.

Addresses the actual decisionRightRightMixed
Haiku 5.5 · API

Commits to building self-serve disputes with merchant recognition first, and says Phase 1 test will validate the approach.

Opus 5.5 · Claude

The PRD commits to building self-serve disputes with merchant recognition first, and says what would change the feed decision (per-lookup pricing).

Gemini 3.8 Flash · API

The PRD commits to building self-serve with the enriched feed, but never states what would change that decision (e.g., if feed quality drops, cost exceeds threshold, or deflection proves lower than expected).

Identifies material uncertaintyRightRightWrong
Haiku 5.5 · API

Names specific unknowns (recognition rate in-app, volume increase, feed quality, logo use, scheme deadlines, day counting) and how to resolve them.

Opus 5.5 · Claude

Open questions name specific unknowns (days_to_resolve units, 11% vs 14% gap, feed pricing, temporary credit) and how to resolve them.

Gemini 3.8 Flash · API

No material unknowns are named, bounded, or resolved; the PRD does not discuss what happens when enriched data is wrong, false positives in duplicate detection, or how risk scoring thresholds would be adjusted.

Avoids unsupported claimsRightWrongWrong
Haiku 5.5 · API

Interpretations are labelled as findings, and no unsupported causal claims are presented as fact.

Opus 5.5 · Claude

The '42% of disputes aren't really disputes' claim is presented as a confident finding but is not supported by the data (actual is 48%).

Gemini 3.8 Flash · API

Causal claims like 'causing the bulk of regulatory SLA breaches because customers were not notified' are presented as fact without evidence, and the sample percentages are incorrect.

Finds that most disputes are recognitionRightRightMixed
Haiku 5.5 · API

Computes 58% recognition share and 171 resolved by showing merchant details, and makes enriched merchant display the first requirement.

Opus 5.5 · Claude

The PRD computes the 58% share and 73% resolved by showing merchant details, and makes recognisable transactions the first scope item (R4/R5).

Gemini 3.8 Flash · API

The PRD incorrectly states 48.5% of disputes are unrecognised and does not compute the share (58%) or how many (169/232) were resolved by showing merchant details, failing to put recognition first.

Success settles trade-offsRightMixedMixed
Haiku 5.5 · API

Defines success metrics with targets and timeframe, and states that the regulatory requirement 'disputing never harder than paying' overrides other goals.

Opus 5.5 · Claude

Success metrics have targets and time frame, but no explicit trade-off rule is given (e.g., which goal wins when they conflict).

Gemini 3.8 Flash · API

The PRD defines success metrics with targets and a timeframe, but gives no rule for which goal wins when they conflict (e.g., speed vs completeness, or deflection vs false negatives).

Plans how people will find itMixedWrongWrong
Haiku 5.5 · API

Rollout is staged but does not describe how users will discover the feature, and support readiness is only partially addressed.

Opus 5.5 · Claude

No staged rollout, user discovery method, or support readiness plan is described beyond agents being able to start the flow.

Gemini 3.8 Flash · API

There is no rollout plan, no mention of how users will discover the feature, and no statement of which support teams need to be ready, beyond a general agent-impact note.

All got wrong 2

Proposes tests that could failWrongWrongWrong
Haiku 5.5 · API

The A/B test in Phase 1 lacks a numeric threshold, measurement window, and action for each outcome; guardrails lack thresholds.

Opus 5.5 · Claude

Only one guardrail (fraud reported late) has a threshold and action; other metrics lack thresholds, measurement windows, and actions for each outcome.

Gemini 3.8 Flash · API

No test, pilot, or gate proposed has a numeric threshold, a measurement window, and the action each result would trigger; the SLA engine defines actions but not a kill criterion.

Says who it's for, and who it isn'tWrongWrongWrong
Haiku 5.5 · API

Does not name the target users (e.g., individual current account holders) or explicitly exclude any user group, and no scope decision follows from a user exclusion.

Opus 5.5 · Claude

The PRD does not name the target users (individuals) or explicitly exclude any user group (e.g., business accounts), and no scope decision visibly follows from such an exclusion.

Gemini 3.8 Flash · API

The PRD names primary cardholders as the target, but never explicitly says who it is not for (e.g., joint account holders, business accounts, disputes above a threshold) and no scope decision follows from an exclusion.

All got right 3

Respects explicit constraintsRightRightRight
Haiku 5.5 · API

The PRD is within word count, addresses the squad, and incorporates all regulatory constraints with enforcement mechanisms.

Opus 5.5 · Claude

The PRD is within 1,000-1,400 words, addresses the squad, and includes R3 to enforce that disputing is never harder than paying.

Gemini 3.8 Flash · API

The output is a PRD, addressed to the squad, and the word count is approximately 1,343 words, within the 1,000–1,400 range.

Produces the required deliverableRightRightRight
Haiku 5.5 · API

Complete PRD with problem, data, goals, requirements, metrics, rollout, and open questions, usable by the squad.

Opus 5.5 · Claude

The PRD is complete, in the right form, within length, and the squad could act on it with the open questions resolved.

Gemini 3.8 Flash · API

The document is a PRD in the right form, reads as a PRD for a squad, fits the word limit, and is largely actionable with light edits.

Builds the regulator's rules inRightRightRight
Haiku 5.5 · API

All three rules (acknowledge 2 days, resolve 15 days or explain, disputing no harder than paying) appear as testable requirements R6, R7, and regulatory constraints section.

Opus 5.5 · Claude

All three rules appear as testable requirements: R14 (acknowledge within 2 days), R16 (resolve within 15 or explain), R3 (no harder than paying).

Gemini 3.8 Flash · API

The three rules (acknowledge within 2 business days, resolve/explain within 15, not harder to dispute than to pay) appear with specific, testable implementations and acceptance criteria.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 70% of 81 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstraChatGPT87.184.62None
2Sonnet 5.5API75.979.82None
3GPT-6.1 SolAPI82.660.92None
4Haiku 5.5API76.163.82None
5Opus 5.5Claude75.960.62None
6GPT-6 LunaAPI75.859.92None
7Gemini 3.8 FlashAPI49.828.22None
8Gemini 3.5 Flash-LiteGemini49.832.12None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 84, 106× the cheapest (1 of 2 outputs estimated)Sonnet 5.5 · API: task score 77, 35× the cheapestGPT-6.1 Sol · API: task score 71, 14× the cheapestHaiku 5.5 · API: task score 67, the cheapestOpus 5.5 · Claude: task score 65, 85× the cheapest (1 of 2 outputs estimated)GPT-6 Luna · API: task score 65, the cheapestGemini 3.8 Flash · API: task score 39, 9.9× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 39, 4.4× the cheapest (1 of 2 outputs estimated)GPT-6 Astra · ChatGPTSonnet 5.5 · APIGPT-6.1 Sol · APIHaiku 5.5 · APIOpus 5.5 · ClaudeGPT-6 Luna · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Writing the requirements document a team will build and test against.

Why it matters

A PRD is where ambiguity becomes either a decision or a bug. For AI products it must also say what happens when the model is uncertain or wrong. Most generated PRDs skip that part.

What good looks like

  • States the user problem and the decision the PRD enables
  • Specifies behaviour under uncertainty, failure and refusal
  • Names eval criteria and a launch bar
  • Defines success precisely enough to settle trade-offs
  • Says what is out of scope

Deliberately not measured

  • Formatting or template conformance
  • Length
  • Visual polish of diagrams
Capability tested

Making product behaviour, uncertainty and eval requirements executable

The failure we’re looking for

A generic feature spec that ignores AI failure behaviour

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

AI product PRD (core) · Conventional product PRD