Humanity’s Last PRD
Latest drop: Haiku 5.5 · 8 Oct
Tasks / Product Craft

Write a PRD

Can the model turn a brief into a spec engineers could build from, including how an AI feature behaves when it is wrong?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 25% were usable with at most a quick edit.

Reliably right

  1. Specifies AI failure behaviour100% pass
    It specifies what happens when uncertain or wrong: block ownership, show retry states, disable assignments on violation, and never fall back to calendar data.
    GPT-6 Astra · ChatGPT · Meeting summaries with action items
  2. Builds the regulator's rules in100% pass
    All three rules (2-business-day acknowledge, 15-business-day resolve or explain, no harder than paying) appear as testable requirements R8, R10, R13.
    Sonnet 5.5 · API · Disputes that aren't disputes
  3. Produces the required deliverable94% pass
    The PRD is complete, written for engineering and design leads, within the 1,000–1,500 word range, and directly actionable.
    GPT-6 Astra · ChatGPT · Meeting summaries with action items

Where it slips

  1. Proposes tests that could fail36% pass
    Launch gates have numeric thresholds and actions, but lack explicit measurement windows (e.g., pilot duration, number of meetings) needed to read out results.
    GPT-6.1 Sol · API · Meeting summaries with action items
  2. Avoids unsupported claims42% pass
    Presents the join log and identity tiers as established facts without labelling them as assumptions, when the context provides no support.
    Opus 5.5 · Claude · Meeting summaries with action items
  3. Uses the supplied evidence correctly44% pass
    The output invents a join log and identity tiers (Verified, Guest, Room) that are not in the supplied context, making unsupported claims about the current system.
    Opus 5.5 · Claude · Meeting summaries with action items

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Success settles trade-offs

    Does the PRD define success as a measurable outcome with a target and time frame, and say which goal wins when two of them conflict?

    Passes when Names a success metric with a target and time frame, and gives at least one explicit trade-off rule (for example, accepting lower coverage to keep precision above a stated level).

    Product and growth leadersOn Lenny’s Podcast

  2. Says who it's for, and who it isn't

    Does the PRD name the specific users it is for, say explicitly who it is not for, and make at least one scope decision that follows from that choice?

    Passes when Names the target users and at least one group it deliberately leaves out, and a scope decision visibly follows from the exclusion.

    Matt LeMayLenny’s Podcast

  3. Plans how people will find it

    Does the PRD say how the feature reaches people: how it rolls out, how users discover it, and who needs to be ready for it (support, sales), rather than stopping at what to build?

    Passes when Gives a staged rollout, a way users will discover the feature, and the teams that need to be ready, all fitted to this feature.

    Ami VoraLenny’s PodcastKarri SaarinenLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Proposes tests that could fail · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for card servicing at Clearbank. Write the PRD for self-serve disputes in the app, as Imogen asks below. It goes to the squad (engineering, design, support operations and compliance) who'll build it next quarter. 1,000 to 1,400 words. The dispute export and the background are attached.

What the model was given6 items: About Clearbank, imogen_note.md, how_disputes_work_today.md, regulator_rules.md, disputes_sample_last_quarter.csv (a random sample of 400 of last quarter's 2,400 disputes), merchant_data_option.md
About ClearbankA digital bank for individuals: current accounts and debit cards, run through a mobile app.
imogen_note.md1 lines · Download
From Imogen (Head of Product): "Write the PRD for self-serve disputes in the app. Support says it's their biggest cost after card freezes. The PRD goes to the squad (engineering, design, support ops and compliance) who will build it next quarter. 1,000 to 1,400 words."
how_disputes_work_today.md5 lines · Download
# How disputes work today

Customers raise a dispute by chatting to support. An agent collects details, then either explains the payment or opens a chargeback. Agents spend about 11 minutes per dispute. 2,400 disputes last quarter; 14% of them were resolved later than the regulator's deadline.

Card transactions show the merchant's billing name (for example "SQ *BRKLYN BAGEL 0412"), not its trading name or logo.
regulator_rules.md5 lines · Download
# Regulator's rules (summary from Legal)

- Acknowledge a dispute within 2 business days.
- Resolve genuine disputes within 15 business days, or tell the customer why it's taking longer.
- Don't make it harder to raise a dispute than to make a payment.
disputes_sample_last_quarter.csv (a random sample of 400 of last quarter's 2,400 disputes)dispute,customer_said,days_to_resolve,outcome CB-40000,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40006,I didn't receive what I paid for,17,Chargeback lost CB-40012,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40018,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40024,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40030,I didn't receive what I paid for,11,Chargeback lost CB-40036,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40042,I didn't receive what I paid for,16,Chargeback lost CB-40048,I was charged twice,5,Not a duplicate: two separate purchases CB-40054,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40060,I was charged twice,10,Not a duplicate: two separate purchases CB-40066,I was charged twice,7,Not a duplicate: two separate purchases CB-40072,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40078,I don't recognise this payment,5,Fraud confirmed CB-40084,I don't recognise this payment,7,Fraud confirmed CB-40090,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40096,I was charged twice,6,Not a duplicate: two separate purchases CB-40102,I didn't receive what I paid for,7,Chargeback won CB-40108,I don't recognise this payment,6,Fraud confirmed CB-40114,I was charged twice,9,Duplicate refunded CB-40120,I was charged twice,7,Not a duplicate: two separate purchases CB-40126,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40132,I was charged twice,9,Duplicate refunded CB-40138,Subscription I cancelled,18,Chargeback won CB-40144,I didn't receive what I paid for,21,Chargeback won CB-40150,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40156,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40162,I don't recognise this payment,6,Fraud confirmed CB-40168,I was charged twice,4,Not a duplicate: two separate purchases CB-40174,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40180,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-40186,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40192,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-40198,I don't recognise this payment,1,Fraud confirmed CB-40204,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40210,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40216,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40222,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40228,I didn't receive what I paid for,29,Chargeback won CB-40234,I don't recognise this payment,8,Fraud confirmed CB-40240,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40246,I didn't receive what I paid for,10,Merchant refunded CB-40252,I don't recognise this payment,4,Fraud confirmed CB-40258,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40264,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40270,I didn't receive what I paid for,21,Chargeback won CB-40276,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40282,I didn't receive what I paid for,17,Merchant refunded CB-40288,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40294,I don't recognise this payment,1,Fraud confirmed CB-40300,I don't recognise this payment,7,Fraud confirmed CB-40306,I don't recognise this payment,8,Fraud confirmed CB-40312,I didn't receive what I paid for,20,Merchant refunded CB-40318,I didn't receive what I paid for,26,Chargeback lost CB-40324,Subscription I cancelled,10,Merchant refunded CB-40330,I don't recognise this payment,1,Fraud confirmed CB-40336,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-40342,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40348,Fraud: I didn't make this payment,9,"Fraud confirmed, refunded" CB-40354,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40360,I was charged twice,9,Not a duplicate: two separate purchases CB-40366,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40372,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40378,I don't recognise this payment,8,Fraud confirmed CB-40384,I didn't receive what I paid for,14,Merchant refunded CB-40390,I don't recognise this payment,7,Fraud confirmed CB-40396,I don't recognise this payment,8,Fraud confirmed CB-40402,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40408,I was charged twice,5,Not a duplicate: two separate purchases CB-40414,I didn't receive what I paid for,13,Chargeback lost CB-40420,Subscription I cancelled,11,Chargeback won CB-40426,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-40432,I didn't receive what I paid for,13,Chargeback won CB-40438,Fraud: I didn't make this payment,4,"Fraud confirmed, refunded" CB-40444,I was charged twice,9,Duplicate refunded CB-40450,I didn't receive what I paid for,18,Chargeback won CB-40456,I don't recognise this payment,1,Fraud confirmed CB-40462,I don't recognise this payment,4,Fraud confirmed CB-40468,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40474,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40480,Fraud: I didn't make this payment,8,"Fraud confirmed, refunded" CB-40486,I didn't receive what I paid for,27,Chargeback lost CB-40492,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40498,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40504,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40510,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40516,I don't recognise this payment,5,Fraud confirmed CB-40522,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40528,Fraud: I didn't make this payment,8,"Fraud confirmed, refunded" CB-40534,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40540,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40546,I don't recognise this payment,2,Fraud confirmed CB-40552,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40558,I was charged twice,5,Duplicate refunded CB-40564,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40570,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40576,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40582,I didn't receive what I paid for,14,Chargeback lost CB-40588,I didn't receive what I paid for,18,Chargeback lost CB-40594,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40600,Fraud: I didn't make this payment,2,"Fraud confirmed, refunded" CB-40606,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40612,I don't recognise this payment,3,Fraud confirmed CB-40618,Fraud: I didn't make this payment,5,"Fraud confirmed, refunded" CB-40624,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40630,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40636,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40642,Fraud: I didn't make this payment,11,"Fraud confirmed, refunded" CB-40648,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40654,I don't recognise this payment,5,Fraud confirmed CB-40660,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40666,I don't recognise this payment,6,Fraud confirmed CB-40672,I was charged twice,5,Not a duplicate: two separate purchases CB-40678,Subscription I cancelled,10,Chargeback won CB-40684,Fraud: I didn't make this payment,9,"Fraud confirmed, refunded" CB-40690,Subscription I cancelled,7,Chargeback won CB-40696,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40702,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40708,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40714,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40720,I was charged twice,13,Not a duplicate: two separate purchases CB-40726,Subscription I cancelled,13,Chargeback won CB-40732,I don't recognise this payment,3,Fraud confirmed CB-40738,I didn't receive what I paid for,18,Chargeback lost CB-40744,Fraud: I didn't make this payment,8,"Fraud confirmed, refunded" CB-40750,I was charged twice,11,Not a duplicate: two separate purchases CB-40756,Subscription I cancelled,10,Merchant refunded CB-40762,I was charged twice,7,Duplicate refunded CB-40768,Subscription I cancelled,18,Chargeback won CB-40774,I didn't receive what I paid for,23,Chargeback won CB-40780,I didn't receive what I paid for,7,Chargeback won CB-40786,Subscription I cancelled,5,Chargeback won CB-40792,I didn't receive what I paid for,10,Merchant refunded CB-40798,I was charged twice,9,Duplicate refunded CB-40804,I didn't receive what I paid for,24,Merchant refunded CB-40810,I don't recognise this payment,2,Fraud confirmed CB-40816,Fraud: I didn't make this payment,11,"Fraud confirmed, refunded" CB-40822,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40828,Subscription I cancelled,13,Chargeback won CB-40834,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40840,I don't recognise this payment,7,Fraud confirmed CB-40846,Subscription I cancelled,7,Merchant refunded CB-40852,I don't recognise this payment,3,Fraud confirmed CB-40858,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40864,I was charged twice,4,Not a duplicate: two separate purchases CB-40870,I don't recognise this payment,8,Fraud confirmed CB-40876,I was charged twice,6,Not a duplicate: two separate purchases CB-40882,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-40888,I don't recognise this payment,8,Fraud confirmed CB-40894,I don't recognise this payment,6,Fraud confirmed CB-40900,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40906,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40912,I don't recognise this payment,8,Fraud confirmed CB-40918,I didn't receive what I paid for,21,Chargeback won CB-40924,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-40930,Fraud: I didn't make this payment,8,"Fraud confirmed, refunded" CB-40936,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40942,I was charged twice,6,Duplicate refunded CB-40948,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40954,I didn't receive what I paid for,14,Chargeback won CB-40960,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40966,I was charged twice,4,Not a duplicate: two separate purchases CB-40972,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40978,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40984,I didn't receive what I paid for,9,Merchant refunded CB-40990,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40996,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41002,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41008,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41014,Fraud: I didn't make this payment,2,"Fraud confirmed, refunded" CB-41020,I didn't receive what I paid for,27,Chargeback won CB-41026,I didn't receive what I paid for,28,Chargeback won CB-41032,I was charged twice,7,Duplicate refunded CB-41038,I don't recognise this payment,2,Fraud confirmed CB-41044,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41050,I don't recognise this payment,4,Fraud confirmed CB-41056,Fraud: I didn't make this payment,11,"Fraud confirmed, refunded" CB-41062,I was charged twice,3,Duplicate refunded CB-41068,I don't recognise this payment,1,Fraud confirmed CB-41074,I don't recognise this payment,1,Fraud confirmed CB-41080,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41086,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41092,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41098,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41104,I didn't receive what I paid for,29,Merchant refunded CB-41110,Subscription I cancelled,16,Chargeback won CB-41116,I didn't receive what I paid for,22,Chargeback won CB-41122,I didn't receive what I paid for,13,Chargeback lost CB-41128,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41134,I was charged twice,10,Not a duplicate: two separate purchases CB-41140,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41146,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41152,I was charged twice,5,Duplicate refunded CB-41158,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41164,I didn't receive what I paid for,12,Chargeback lost CB-41170,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41176,Fraud: I didn't make this payment,2,"Fraud confirmed, refunded" CB-41182,I don't recognise this payment,1,Fraud confirmed CB-41188,I don't recognise this payment,3,Fraud confirmed CB-41194,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41200,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41206,Fraud: I didn't make this payment,5,"Fraud confirmed, refunded" CB-41212,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41218,Fraud: I didn't make this payment,6,"Fraud confirmed, refunded" CB-41224,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41230,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41236,I don't recognise this payment,3,Fraud confirmed CB-41242,Subscription I cancelled,9,Merchant refunded CB-41248,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41254,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41260,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41266,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41272,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41278,I didn't receive what I paid for,11,Chargeback lost CB-41284,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41290,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41296,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-41302,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41308,I was charged twice,11,Duplicate refunded CB-41314,I didn't receive what I paid for,27,Chargeback won CB-41320,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41326,Fraud: I didn't make this payment,5,"Fraud confirmed, refunded" CB-41332,I was charged twice,12,Not a duplicate: two separate purchases CB-41338,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-41344,I didn't receive what I paid for,17,Merchant refunded CB-41350,I didn't receive what I paid for,21,Chargeback won CB-41356,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41362,Subscription I cancelled,14,Merchant refunded CB-41368,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41374,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41380,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41386,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41392,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41398,I don't recognise this payment,2,Fraud confirmed CB-41404,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41410,I don't recognise this payment,3,Fraud confirmed CB-41416,I don't recognise this payment,5,Fraud confirmed CB-41422,I was charged twice,12,Duplicate refunded CB-41428,Subscription I cancelled,9,Merchant refunded CB-41434,I was charged twice,6,Duplicate refunded CB-41440,Subscription I cancelled,15,Merchant refunded CB-41446,I don't recognise this payment,5,Fraud confirmed CB-41452,I don't recognise this payment,7,Fraud confirmed CB-41458,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41464,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41470,I was charged twice,6,Not a duplicate: two separate purchases CB-41476,I don't recognise this payment,5,Fraud confirmed CB-41482,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41488,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41494,I didn't receive what I paid for,26,Merchant refunded CB-41500,I didn't receive what I paid for,24,Chargeback won CB-41506,Subscription I cancelled,20,Chargeback won CB-41512,I didn't receive what I paid for,29,Merchant refunded CB-41518,Subscription I cancelled,20,Chargeback won CB-41524,I don't recognise this payment,8,Fraud confirmed CB-41530,I was charged twice,3,Not a duplicate: two separate purchases CB-41536,I didn't receive what I paid for,8,Chargeback won CB-41542,I don't recognise this payment,2,Fraud confirmed CB-41548,I don't recognise this payment,2,Fraud confirmed CB-41554,I was charged twice,12,Not a duplicate: two separate purchases CB-41560,Subscription I cancelled,6,Chargeback won CB-41566,I was charged twice,3,Not a duplicate: two separate purchases CB-41572,I don't recognise this payment,1,Fraud confirmed CB-41578,Fraud: I didn't make this payment,7,"Fraud confirmed, refunded" CB-41584,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41590,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-41596,Subscription I cancelled,16,Merchant refunded CB-41602,Subscription I cancelled,15,Merchant refunded CB-41608,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41614,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41620,I was charged twice,5,Duplicate refunded CB-41626,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41632,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41638,Fraud: I didn't make this payment,3,"Fraud confirmed, refunded" CB-41644,I don't recognise this payment,3,Fraud confirmed CB-41650,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41656,I don't recognise this payment,6,Fraud confirmed CB-41662,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41668,I didn't receive what I paid for,12,Chargeback won CB-41674,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41680,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41686,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-41692,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41698,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41704,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41710,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41716,Fraud: I didn't make this payment,8,"Fraud confirmed, refunded" CB-41722,I don't recognise this payment,3,Fraud confirmed CB-41728,I was charged twice,12,Duplicate refunded CB-41734,Fraud: I didn't make this payment,5,"Fraud confirmed, refunded" CB-41740,I don't recognise this payment,6,Fraud confirmed CB-41746,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41752,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41758,I was charged twice,13,Duplicate refunded CB-41764,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41770,I didn't receive what I paid for,23,Chargeback won CB-41776,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41782,I didn't receive what I paid for,27,Chargeback won CB-41788,I didn't receive what I paid for,23,Chargeback lost CB-41794,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41800,I don't recognise this payment,3,Fraud confirmed CB-41806,I don't recognise this payment,1,Fraud confirmed CB-41812,I was charged twice,4,Not a duplicate: two separate purchases CB-41818,I don't recognise this payment,8,Fraud confirmed CB-41824,I didn't receive what I paid for,18,Chargeback won CB-41830,Subscription I cancelled,13,Chargeback won CB-41836,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41842,I don't recognise this payment,7,Fraud confirmed CB-41848,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41854,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41860,I was charged twice,5,Duplicate refunded CB-41866,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41872,I don't recognise this payment,8,Fraud confirmed CB-41878,I don't recognise this payment,2,Fraud confirmed CB-41884,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41890,I was charged twice,7,Not a duplicate: two separate purchases CB-41896,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41902,Fraud: I didn't make this payment,9,"Fraud confirmed, refunded" CB-41908,I don't recognise this payment,4,Fraud confirmed CB-41914,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41920,I didn't receive what I paid for,24,Merchant refunded CB-41926,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41932,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-41938,Subscription I cancelled,15,Chargeback won CB-41944,I didn't receive what I paid for,13,Chargeback won CB-41950,I don't recognise this payment,2,Fraud confirmed CB-41956,I didn't receive what I paid for,24,Chargeback lost CB-41962,I don't recognise this payment,4,Fraud confirmed CB-41968,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41974,Subscription I cancelled,15,Chargeback won CB-41980,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41986,Fraud: I didn't make this payment,2,"Fraud confirmed, refunded" CB-41992,Fraud: I didn't make this payment,6,"Fraud confirmed, refunded" CB-41998,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-42004,I didn't receive what I paid for,12,Merchant refunded CB-42010,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42016,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42022,I was charged twice,11,Duplicate refunded CB-42028,I didn't receive what I paid for,25,Merchant refunded CB-42034,I didn't receive what I paid for,10,Merchant refunded CB-42040,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42046,I was charged twice,8,Duplicate refunded CB-42052,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-42058,I don't recognise this payment,7,Fraud confirmed CB-42064,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-42070,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42076,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-42082,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-42088,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-42094,Subscription I cancelled,12,Chargeback won CB-42100,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-42106,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-42112,Subscription I cancelled,18,Chargeback won CB-42118,Fraud: I didn't make this payment,6,"Fraud confirmed, refunded" CB-42124,I don't recognise this payment,5,Fraud confirmed CB-42130,I didn't receive what I paid for,14,Chargeback lost CB-42136,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-42142,Fraud: I didn't make this payment,2,"Fraud confirmed, refunded" CB-42148,I was charged twice,12,Not a duplicate: two separate purchases CB-42154,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-42160,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-42166,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42172,Fraud: I didn't make this payment,9,"Fraud confirmed, refunded" CB-42178,I don't recognise this payment,3,Fraud confirmed CB-42184,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-42190,I was charged twice,7,Duplicate refunded CB-42196,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-42202,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-42208,I didn't receive what I paid for,21,Chargeback won CB-42214,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-42220,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-42226,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-42232,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-42238,I was charged twice,10,Duplicate refunded CB-42244,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-42250,Subscription I cancelled,11,Chargeback won CB-42256,Fraud: I didn't make this payment,3,"Fraud confirmed, refunded" CB-42262,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-42268,I didn't receive what I paid for,16,Merchant refunded CB-42274,I was charged twice,3,Duplicate refunded CB-42280,Fraud: I didn't make this payment,9,"Fraud confirmed, refunded" CB-42286,I was charged twice,11,Duplicate refunded CB-42292,I was charged twice,9,Not a duplicate: two separate purchases CB-42298,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-42304,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-42310,Subscription I cancelled,16,Merchant refunded CB-42316,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-42322,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-42328,Fraud: I didn't make this payment,3,"Fraud confirmed, refunded" CB-42334,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-42340,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-42346,I didn't receive what I paid for,19,Merchant refunded CB-42352,I was charged twice,12,Duplicate refunded CB-42358,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-42364,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-42370,I don't recognise this payment,3,Fraud confirmed CB-42376,I didn't receive what I paid for,17,Chargeback won CB-42382,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-42388,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42394,I don't recognise this payment,5,Customer recognised it once shown the merchant's details
merchant_data_option.md3 lines · Download
# Merchant data we could use

Our card processor offers an enriched merchant feed (trading name, logo, location, website) for 94% of transactions, at $0.0008 per transaction. We process about 9 million card transactions a month.
What a strong answer doesThe answer key the graders mark against

Starts from the data: in the sample, 232 of 400 disputes (58%, so about 1,400 of the quarter's 2,400) were 'I don't recognise this payment', and 169 of those (about three in four) were resolved simply by showing the merchant's details; so the first thing to build is recognisable transactions (trading name, logo, location from the enriched feed, about $7,200 a month for 9 million transactions), shown before someone raises a dispute, with self-serve disputes for the genuine cases. Defines success with targets (disputes per 1,000 transactions, share resolved without an agent, regulator deadlines met against 14% late today) and says which wins when they conflict (never make disputing harder than paying, per the regulator). Covers the flows for genuine disputes (charged twice, not received, cancelled subscription, fraud) with acknowledgement within 2 business days and resolution within 15, what happens when the merchant data is missing (6% of transactions), and how the rollout reaches people, with support ready. Says who it's not for (for example business accounts, or disputes over a set amount that still go to an agent) and what's out of scope.

Critical failures (cap the score at 40)
  • Adds steps that make raising a dispute harder than making a payment
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Sonnet 5.5API83Usable with a quick editGPT-6 AstraChatGPT73Needs more workGPT-6 LunaAPI69Needs more workHaiku 5.5API69Needs more workOpus 5.5Claude60Needs more workGPT-6.1 SolAPI58Needs more workGemini 3.5 Flash-LiteGemini40Needs more workGemini 3.8 FlashAPI37Needs more work
GPT-6 LunaAPI

Needs more than a quick edit8 of 12 checks passed

The PRD lacks explicit computation of the recognition share, an explicit trade‑off rule, and numeric pilot kill criteria, so a PM would need to rework these sections before handing it off.

Output

PRD: Self-serve card disputes in the Clearbank app

Owner: Card Servicing Delivery: Next quarter Audience: Engineering, Design, Support Operations, Compliance

Summary

Let customers understand an unfamiliar card transaction and, if they still believe it is wrong, raise and track a dispute in the app without first contacting support. The flow should help customers recognise legitimate payments while keeping the route to report a genuine issue clear and easy.

Today, customers raise disputes in chat. Agents collect details, explain the payment or open a chargeback; each dispute takes about 11 minutes. Last quarter there were 2,400 disputes—around 440 agent hours—and 14% were resolved later than the regulator’s deadline. The sample of 400 disputes also shows a recurring opportunity: many customers who said they did not recognise a payment later recognised it when shown merchant details. The current transaction descriptor may show a billing name, such as “SQ *BRKLYN BAGEL 0412”, rather than the trading name a customer knows.

Problem and opportunity

Customers cannot reliably identify a payment from its billing descriptor, so some contact support unnecessarily. Customers with genuine disputes face a chat-based intake and may not receive timely updates. Support must collect information manually and manage deadlines across cases.

The processor’s enriched merchant feed provides a trading name, logo, location and website for 94% of transactions. At $0.0008 per transaction, applying it to the eligible share of approximately 9 million monthly transactions would cost about $6,768 per month (subject to the vendor’s billing basis). This is a material ongoing cost; we should measure whether the enrichment improves recognition and reduces support demand.

Goals and success measures

  1. Reduce avoidable agent work while preserving a simple route to dispute.
  2. Give customers a clear, trackable way to report a genuine card-payment problem.
  3. Meet the regulator’s acknowledgement and resolution-or-delay-notice requirements.

Proposed launch targets, measured over the first 90 days against a pre-launch baseline: - Reduce support-handled dispute contacts per 1,000 card transactions by 30%. - Acknowledge at least 95% of in-app disputes within 2 business days. - For at least 95% of genuine disputes, resolve within 15 business days or send a clear explanation that the case is taking longer, with the next update expectation. - No material increase in complaints, reopened cases, confirmed fraud loss, or customers abandoning a genuine dispute compared with baseline.

Track the number of customers who recognise a payment, continue to raise a dispute, abandon at each step, contact support after submitting, and the cost and coverage of enriched merchant data. Segment outcomes by dispute reason and whether merchant enrichment was available. The sample is useful evidence of the recognition problem, but it is not a substitute for a live baseline or a causal estimate of savings.

Users and scope

Primary users: Clearbank personal current-account customers with a debit-card transaction they do not understand or believe is incorrect.

In scope for MVP: A dispute journey launched from an eligible card transaction; merchant context; guided reason selection and information capture; case submission and acknowledgement; in-app status and updates; operational case routing and SLA monitoring.

Out of scope: Replacing the chargeback process, guaranteeing refunds, automated denial of claims, disputes for non-card payment types, and a general-purpose support-chat redesign.

Proposed experience

  1. Start from the transaction. Put a visible “Get help with this payment” action on eligible card transaction details. Do not require customers to find a separate disputes area or contact chat. Eligibility must be determined by Compliance and Operations using applicable scheme and processing rules. If a transaction cannot be disputed in-app, explain why and provide a clear alternative support route; never leave the customer at a dead end.
  1. Help identify the merchant. Show the enriched trading name and, when available and reliable, logo, location and website alongside the original billing descriptor, amount and date. Do not imply that a logo or website proves the customer made the purchase. If data is unavailable, show the billing descriptor and explain that the merchant may use a different name. Do not make merchant lookup a prerequisite to reporting.
  1. Let the customer choose what happened. Offer plain-language reasons, including “I don’t recognise this”, “I was charged twice”, “I didn’t receive what I paid for”, “I cancelled a subscription”, and “I didn’t make this payment”. Keep “I don’t recognise this” distinct from confirmed fraud. Ask only the follow-up questions needed for the selected reason—for example, whether the customer contacted the merchant or when they cancelled. Explain why requested information helps, and allow submission without optional evidence.
  1. Offer relevant help without blocking a claim. If the customer recognises the merchant, offer a simple way to exit without opening a case. For suspected fraud, offer an immediate card-freeze action and explain that freezing the card is separate from reporting the transaction. Never require a freeze, merchant contact, or troubleshooting step before the customer can submit.
  1. Confirm and track. On successful submission, show a case reference, the reported transaction and reason, confirmation that the dispute has been received, and what happens next. Provide a case status and updates in the app, with notifications where enabled. If further information is required, explain what is needed and how to provide it. Do not suggest a refund is guaranteed or that a chargeback has already been won.

Operational and compliance requirements

  • The system must record submission and acknowledgement timestamps, customer-provided details, case status, decisions, communications and audit history.
  • An in-app receipt should serve as acknowledgement only if Compliance confirms it meets the applicable acknowledgement requirement. Otherwise, send a compliant acknowledgement within 2 business days.
  • Operations needs a queue with reason, age, owner, priority and deadlines, plus alerts before SLA breaches. Cases should not silently age in a queue.
  • Genuine disputes must be resolved within 15 business days, or the customer must be told why it is taking longer. The message must include the reason and a meaningful next update expectation; Operations must continue to monitor the case.
  • Any classification or routing automation may assist staff but must not automatically reject a claim. A customer who disagrees with a suggested explanation must still be able to proceed.
  • The journey must be accessible, use clear language, and not be harder to reach or complete than making a payment. Store and share customer and merchant data only for appropriate servicing purposes, with vendor use, security and retention reviewed by Compliance.

Dependencies and open decisions

Before build completion, Compliance and Operations must confirm eligible transaction types, claim deadlines, reason-specific evidence needs, the definition of “genuine dispute” for reporting, and the compliant wording for acknowledgements and delay notices. Engineering must confirm processor-feed availability, fallback behaviour, case-system integration, and reliable business-day SLA calculation. Product and Finance should agree an enrichment pilot or rollout threshold using recognition, contact reduction, coverage and vendor cost; the feed’s 6% coverage gap needs a tested fallback.

Rollout and risks

Pilot with a limited customer cohort after internal testing of each dispute reason, merchant-data failures, duplicate submissions, accessibility and SLA alerts. Compare pilot outcomes with a pre-launch baseline before expanding. Key risks are customers mistaking a legitimate merchant for fraud, poor enrichment data, confusing a submitted claim with a guaranteed refund, and moving work from chat into an unmanaged queue. Mitigate through clear descriptors and caveats, an always-available dispute route, explicit case status, operational ownership and deadline monitoring.

Grades and run details

Decision model 83 · LLM judge 62
Decision model checks
  • passUses the supplied evidence correctly47%
  • passAddresses the actual decision39%
  • partialRespects explicit constraints57%
  • passIdentifies material uncertainty68%
  • passAvoids unsupported claims50%
  • partialProduces the required deliverable34%
  • partialFinds that most disputes are recognition71%
  • passBuilds the regulator's rules in97%
  • passSuccess settles trade-offs92%
  • partialProposes tests that could fail76%
  • passSays who it's for, and who it isn't30%
  • passPlans how people will find it64%
Run
Run
#1
API response time
32 s
Submitted
6 Oct 2026
Sonnet 5.5API

Usable with a quick edit10 of 12 checks passed

The PRD is comprehensive and actionable. Adding an explicit user exclusion and a numeric guardrail threshold are minor edits a PM can make quickly.

Output

PRD: Self-serve disputes in the app

Owner: PM, Card Servicing · Audience: Engineering, Design, Support Ops, Compliance · Build: next quarter

1. Summary

Disputes are currently raised only through chat. Agents spend about 11 minutes on each of the 2,400 disputes a quarter (roughly 440 hours), and 14% were resolved after the regulator's deadline.

The sample shows that most disputes aren't really disputes. Customers can't recognise the merchant name on the statement. The late cases are a different problem. So this PRD covers two things: stop avoidable disputes at the source, and give genuine disputes a structured, timed path.

2. What the data says

I analysed the 400-dispute random sample (of 2,400). With n=400, shares are accurate to roughly ±5 points.

Customer saidShareOutcomes
I don't recognise this payment58% (232)169 recognised it once shown merchant details; 63 confirmed fraud
Fraud: I didn't make this9% (36)All confirmed and refunded
I didn't receive what I paid for14% (56)24 chargebacks won, 15 lost, 17 merchant refunded
I was charged twice12% (48)24 genuine duplicates, 24 two separate purchases
Subscription I cancelled7% (28)18 chargebacks won, 10 merchant refunded

Four findings shape the design:

  1. 42% of all disputes (169 of 400) end when the customer sees who the merchant is. Card transactions show billing names like "SQ *BRKLYN BAGEL 0412". This is the biggest lever, and it needs better information, not a dispute flow.
  2. About a quarter of disputes are fraud (63 + 36 = 99 of 400). A customer with a stolen card must not wait in a queue, so the flow has to offer a card freeze at once.
  3. Half of "charged twice" claims are two real purchases. Showing both transactions side by side resolves these.
  4. Every case over 15 days is a "not received" or "subscription" dispute (44 of 400, 11%; reported figure is 14%). None of the other types exceeded 13 days. Deflecting lookups will free agent time, but it will not fix lateness by itself. Late cases need structured evidence intake and deadline timers. Support Ops to confirm whether `daystoresolve` is business or calendar days.

3. Goals and metrics

Targets below are my proposals, to be agreed with Support Ops.

  • Reduce agent-handled disputes by 25% within 90 days of launch. That is about 600 a quarter, or ~110 agent-hours. The ceiling is roughly 40%: the 42% "recognised" share, less the ~6% of transactions with no enriched data.
  • Late resolutions from 14% to under 5%.
  • 100% of disputes acknowledged within 2 business days. In-app, this is instant.
  • Guardrail: every suspected-fraud case reaches the fraud queue the same day. Track "I recognise this" taps later reversed into a dispute on the same transaction. Fraud and Compliance to set the acceptable threshold.

Non-goals: changing chargeback decisioning, scheme rules, or provisional-credit policy. Merchant logos across the whole transaction feed are also out of scope.

4. Requirements

Entry and merchant details

  • R1. Card transaction detail shows trading name, logo, location and website where available. It always shows date, time, amount and the raw billing name.
  • R2. Where no enriched data exists (~6%), show the raw billing name with obvious prefixes and store codes stripped, plus the raw string.
  • R3. Every card transaction has a visible "Something wrong?" button, one tap to start. Reasons: I don't recognise this, Charged twice, Didn't receive it, Cancelled subscription, I didn't make this payment.

Paths by reason

  • R4. Don't recognise. Show the merchant details with two equal-weight buttons: "I remember it" and "I still don't recognise this". The second continues the dispute. It is never hidden or delayed. It then offers a one-tap card freeze (optional) and creates a case flagged suspected fraud, routed to the fraud queue.
  • R5. "I didn't make this payment". Skip the merchant step and go straight to the freeze offer and the fraud queue.
  • R6. Charged twice. Show the two transactions side by side. If same merchant, same amount, within a window Support Ops defines, treat as a likely duplicate and create the case. Otherwise say "these look like separate purchases", with a prominent "Dispute anyway" button.
  • R7. Not received / subscription. Collect structured evidence: order date, what was bought, whether the merchant was contacted, and for subscriptions the cancellation date and proof upload. Ask whether the merchant was contacted but don't require it, unless Ops confirms a scheme rule does. Cases go to the chargeback queue as a complete evidence packet.

Case lifecycle

  • R8. On submission, show a reference number and send a push and email acknowledgement. Store the timestamp.
  • R9. An in-app status tracker shows received, under review, and resolved with outcome.
  • R10. Each case carries a business-day clock. At day 10, flag it to Ops. At day 15, if unresolved, automatically message the customer with the reason for the delay.
  • R11. "Talk to an agent" is available at every step. The case context transfers so the agent doesn't re-ask.
  • R12. Agents see the structured case in their console. Support Ops owns queue rules and the duplicate-detection window.

Compliance

  • R13. Raising a dispute must be no harder than making a payment. Compliance and Design agree the measurable benchmark (steps and taps) in week 1, and the final flow is tested against it.
  • R14. Compliance reviews all interstitial copy. "I remember it" is never pre-selected, and no step blocks continuing a dispute.
  • R15. Log every timestamp (raised, acknowledged, notices sent, resolved) and every "I remember it" tap, in an audit-ready form.

5. Merchant data decision (needed from Imogen by end of week 2)

The processor's enriched feed covers 94% of transactions at $0.0008 each. At about 9M transactions a month, a bulk feed costs $7,200 a month (~$86,000 a year).

The benefit, if every "recognised" dispute were deflected, is ~1,000 disputes a quarter × 11 minutes, or about 185 agent-hours a quarter (~62 a month). The bulk feed only pays back on support savings if an agent hour costs more than about $116, so I don't recommend it on those grounds alone.

Recommendation: ask the processor for per-lookup pricing and fetch merchant data when a customer opens a transaction detail. Even if every transaction were viewed, that can't exceed $7,200 a month, and real cost will be a fraction of it. If the processor only sells the bulk feed, we ship R2 (cleaned descriptors) and measure how much deflection that alone achieves before committing to the feed.

6. Risks and open questions

  • Deflection is an upper bound. The "recognised" outcomes came after agents shared merchant details, so enriched data in-app should replicate that, but we'll confirm it in an A/B test at launch.
  • Compliance: is the 14% late figure measured against notified extensions, or only against the 15-day deadline? If customers were told about delays, some of those cases may have been compliant.
  • Lost chargebacks: 15 of 56 "not received" cases were lost. Support Ops to review a sample for evidence gaps so R7 collects the right fields.
  • Fraud team capacity: about a quarter of disputes will arrive as structured fraud cases. Fraud Ops to confirm queue readiness.
  • Eligibility: Compliance to confirm any time limit on raising a dispute, which the flow should show.

7. Rollout

  1. Weeks 1–2: decide the data source, agree the R13 benchmark, and prototype R4–R6.
  2. Build: transaction detail (R1–R3) and the fraud and lookup paths first, then the evidence intake and timers (R7–R10).
  3. Launch: 10% of customers, with A/B on R4. Expand once the fraud-reversal guardrail holds.

Grades and run details

Decision model 79 · LLM judge 85
Decision model checks
  • passUses the supplied evidence correctly31%
  • passAddresses the actual decision48%
  • partialRespects explicit constraints49%
  • passIdentifies material uncertainty99%
  • partialAvoids unsupported claims26%
  • partialProduces the required deliverable38%
  • passFinds that most disputes are recognition100%
  • passBuilds the regulator's rules in98%
  • passSuccess settles trade-offs98%
  • partialProposes tests that could fail74%
  • partialSays who it's for, and who it isn't75%
  • passPlans how people will find it78%
Run
Run
#1
API response time
1.8 min
Submitted
6 Oct 2026
Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit4 of 12 checks passed

The PRD contains an incorrect data point and lacks essential scope, trade-off rules, and discovery planning; a PM would need to rework the evidence and exclusions before using it.

Output

Product Requirements Document (PRD): Self-Serve Card Disputes & Merchant Enrichment

Author: PM, Card Servicing Target Audience: Engineering, Design, Support Operations, Compliance Timeline: Build scheduled for Q3

---

1. Executive Summary & Background

At Clearbank, customer card servicing is foundational to trust. Currently, raising a debit card dispute requires customers to engage our support team via in-app chat. Agents spend an average of 11 minutes per dispute collecting information, diagnosing issues, and routing chargebacks. Last quarter alone, we handled 2,400 disputes.

This process creates two major organizational pain points: 1. High Operational Costs: Support operations report that disputes represent their second-largest cost center, trailing only card freezes. 2. Regulatory Exposure: Last quarter, 14% of disputes were resolved later than the regulator’s deadline (acknowledgement within 2 business days; resolution within 15 business days).

Furthermore, an analysis of last quarter’s dispute data reveals a systemic UX failure: 52% of all disputes raised under "I don't recognise this payment" were instantly resolved the moment a support agent displayed the merchant's actual trading name and location. For example, a charge appearing on a bank statement as `SQ *BRKLYN BAGEL 0412` causes immediate confusion because raw payment gateway billing names obscure true merchant identities.

This PRD outlines a two-part solution for next quarter: 1. In-app Self-Serve Disputes: Allowing customers to independently file, track, and manage card disputes directly within the Clearbank mobile application, meeting all regulatory timelines effortlessly. 2. Merchant Data Enrichment: Integrating an enriched merchant feed (trading names, logos, maps, and websites) for 94% of transactions, neutralizing over half of unrecognized payment disputes before they ever begin.

---

2. Product Goals & Success Metrics

Goals

  • Empower customers to resolve transaction ambiguities and file disputes without human intervention.
  • Drastically reduce support agent handling time and operational expenses.
  • Ensure 100% compliance with regulatory response and resolution deadlines.
  • Improve statement clarity to eliminate unnecessary friction.

Key Performance Indicators (KPIs)

  • Support Cost Reduction: Decrease support-handled dispute volume by 70% within 3 months of launch.
  • Regulatory Compliance: Achieve 0% breach rate on 2-day acknowledgement and 15-day resolution deadlines.
  • Deflection via Enrichment: Reduce "I don't recognise this payment" inbound queries by 45% simply by surfacing clean merchant data on transaction rows.
  • Time to Resolution: Lower average dispute resolution time from ~8 days (human-driven) to under 3 days (automated intake and routing).

---

3. Regulatory Framework & Legal Guardrails

Per Clearbank’s Legal and Compliance team, the automated dispute flow must strictly adhere to the following mandates: * Acknowledgement: The system must automatically issue a formal acknowledgement to the customer within 2 business days of submission. * Resolution Window: Genuine disputes must be resolved within 15 business days. If an investigation requires more time (e.g., complex chargebacks), the system must automatically notify the customer with an explanation before the window closes. * Friction Parity: The regulator explicitly mandates that “it must not be harder to raise a dispute than it is to make a payment.” The self-serve flow must be intuitive, frictionless, and accessible in 3 taps or fewer from any transaction details screen.

---

4. Product Scope & User Experience

Phase 1: Merchant Data Enrichment (The Deflection Layer)

To prevent unnecessary dispute creation, we will integrate our card processor’s enriched merchant feed for 94% of transactions at a cost of $0.0008 per transaction (~$7,200/month for 9M monthly transactions).

  • Transaction View Upgrade: When a user taps a transaction, instead of seeing raw billing strings like `SQ *BRKLYN BAGEL 0412`, they will see:
  • Clean Trading Name ("Brooklyn Bagel Co.")
  • Verified Merchant Logo
  • Interactive Map Pin / Location
  • Merchant Website / Support Link
  • Impact: This immediately addresses the root cause of our most frequent dispute category ("I don't recognise this payment"), allowing customers to instantly self-authenticate transactions.

Phase 2: In-App Self-Serve Dispute Flow

For transactions that genuinely require a dispute, customers can access a streamlined, step-by-step wizard directly from the transaction detail view ("Report an issue with this payment").

#### Step 1: Reason Selection The customer selects from four standardized dispute categories (aligned with our historical data taxonomy): 1. I don't recognise this payment (Includes suspected fraud). 2. I didn't receive what I paid for (Goods/services never delivered). 3. Subscription I cancelled (Charged after formal cancellation). 4. I was charged twice (Duplicate processing).

#### Step 2: Contextual Interactive Questionnaire Depending on the selected reason, the app dynamically collects required evidence: * Unrecognized/Fraud: Instant card freeze toggle (if not already frozen), option to report card stolen, and automated device fingerprinting capture. * Non-receipt / Cancelled Subscriptions: Prompt to upload supporting documents (e.g., cancellation emails, screenshots of merchant terms) and confirmation of date contact was attempted with the merchant. * Duplicate Charges: Automatic cross-referencing of adjacent ledger entries to check for matching amounts within a 48-hour window.

#### Step 3: Submission & Instant Acknowledgement * Upon final confirmation, the app generates a unique dispute reference (e.g., `CB-50123`). * An immediate in-app confirmation screen and notification are triggered, satisfying the 2-day regulatory acknowledgement rule instantly.

#### Step 4: Dispute Tracker Hub * A dedicated "Disputes" section in the account settings allows users to track the real-time status of their claim (e.g., Submitted ➔ Under Review ➔ Chargeback Filed ➔ Resolved / Refunded). * Customers can upload additional evidence or message support directly within the specific dispute thread if further documentation is requested by operations.

---

5. Technical & Operational Requirements

Engineering (Core App & Backend)

  • Transaction Feed Integration: API integration with the card processor's enriched merchant database, ensuring asynchronous caching so transaction lists load without latency.
  • Dispute State Machine: Build a robust backend state machine handling dispute lifecycles, SLA timers (2-day/15-day alerts), and automated notifications.
  • Document Storage: Secure, encrypted document upload pipeline in AWS S3 for customer-submitted receipts and screenshots.

Design

  • Frictionless UI: Ensure mobile components comply with accessibility standards (WCAG 2.1 AA) and fit within standard iOS and Android design patterns.
  • Clear Microcopy: Avoid legal jargon; use plain language explaining chargeback timelines and merchant response windows.

Support Operations

  • Agent Dashboard Integration: Update the internal CRM/servicing tool so agents can review customer-submitted structured data, reducing average handling time from 11 minutes to under 3 minutes.
  • SLA Escalation Queues: Implement automated pager/dashboard alerts for disputes approaching the 12-day mark to prevent regulatory breaches.

Compliance

  • Audit Logging: Every state change, customer submission, and agent interaction must maintain an immutable audit trail for regulatory inspections.
  • Policy Rule Updates: Sign-off on automated disclosure text presented to users during the dispute submission flow.

---

6. Milestones & Timeline (Q3 Build)

  • Sprint 1–2 (Weeks 1–4):
  • Backend API integration for Merchant Enrichment feed.
  • Database schema updates for dispute management and state machine architecture.
  • Sprint 3–4 (Weeks 5–8):
  • Mobile UI/UX implementation for transaction detail enhancement and self-serve dispute wizard.
  • Internal support operations tool integration and agent training.
  • Sprint 5 (Weeks 9–10):
  • End-to-end QA testing, compliance audit, and closed beta testing with 500 internal Clearbank employees.
  • Sprint 6 (Weeks 11–12):
  • Phased rollout to 10% ➔ 50% ➔ 100% of the customer base. Post-launch monitoring against key KPIs.

Claims the judge couldn’t find in the brief

  • 52% of all disputes raised under 'I don't recognise this payment' were instantly resolved by showing the merchant's details.
  • Current average dispute resolution time is ~8 days (human-driven).

Grades and run details

Decision model 54 · LLM judge 31
Decision model checks
  • failUses the supplied evidence correctly79%
  • passAddresses the actual decision20%
  • partialRespects explicit constraints32%
  • failIdentifies material uncertainty36%
  • failAvoids unsupported claims62%
  • partialProduces the required deliverable78%
  • passFinds that most disputes are recognition41%
  • passBuilds the regulator's rules in98%
  • passSuccess settles trade-offs36%
  • failProposes tests that could fail30%
  • partialSays who it's for, and who it isn't73%
  • passPlans how people will find it58%
Run
Run
#1
Time to output
9 s
Submitted
6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 7

Uses the supplied evidence correctlyRightRightWrong
GPT-6 Luna · API

All factual statements about the current situation are taken from the brief or context, or follow directly from arithmetic.

Sonnet 5.5 · API

All factual claims about current situation come directly from the supplied context or are correct arithmetic on the sample data.

Gemini 3.5 Flash-Lite · Gemini

Uses an incorrect statistic (52% instead of ~73%) and an unsupported average resolution time (~8 days) not found in the evidence.

Identifies material uncertaintyRightRightWrong
GPT-6 Luna · API

It lists open compliance, engineering, and product decisions, acknowledges the sample is not a causal baseline, and calls for a pilot to resolve key uncertainties.

Sonnet 5.5 · API

The output lists specific unknowns (days interpretation, late-case compliance, evidence gaps, fraud capacity) and says how to resolve them.

Gemini 3.5 Flash-Lite · Gemini

Does not name unknown unknowns (e.g., impact of missing 6% merchant data, customer behaviour) or conditions that would change the decision.

Avoids unsupported claimsRightRightWrong
GPT-6 Luna · API

Interpretations and forecasts are presented as proposals or measurements to be taken, not as established facts.

Sonnet 5.5 · API

Findings and hypotheses are clearly labelled as derived from data; no unsupported causes or motivations are presented as established fact.

Gemini 3.5 Flash-Lite · Gemini

Presents the 52% and ~8-day resolution figures as established facts without labelling them as assumptions, and they are not supported by the evidence.

Finds that most disputes are recognitionWrongRightMixed
GPT-6 Luna · API

The PRD does not compute the exact share (≈58%) or the number resolved by showing merchant details, only describes it as 'many'.

Sonnet 5.5 · API

Correctly computes the 58% recognition share and makes enriched merchant details the primary solution, placed before the dispute flow.

Gemini 3.5 Flash-Lite · Gemini

Does not compute that 58% of all disputes are 'I don't recognise' or state how many of those were resolved by showing details; the cited 52% is wrong, and the share of total disputes is missing.

Success settles trade-offsMixedRightMixed
GPT-6 Luna · API

Success metrics with targets are set, but no explicit trade‑off rule states which goal wins when two conflict.

Sonnet 5.5 · API

Metrics include targets and timeframe, and the fraud-routing guardrail ('every suspected-fraud case reaches the fraud queue the same day') provides an explicit trade-off.

Gemini 3.5 Flash-Lite · Gemini

Sets measurable targets with time frames but does not state which goal wins when they conflict (e.g., cost reduction vs. regulatory compliance).

Says who it's for, and who it isn'tMixedWrongWrong
GPT-6 Luna · API

The PRD names target users (personal current‑account customers) but does not explicitly exclude a user group like business accounts; 'non‑card payment types' is a payment type exclusion, not a user group.

Sonnet 5.5 · API

Does not name the target users (individual account holders) or explicitly exclude a group (e.g., business accounts), missing a required scope decision.

Gemini 3.5 Flash-Lite · Gemini

Does not name the specific users it is for or explicitly say who it is not for (e.g., business accounts), and no scope decision visibly follows from an exclusion.

Plans how people will find itRightRightMixed
GPT-6 Luna · API

It gives a staged rollout (pilot then expand), a discovery method (button on transaction), and names support operations as needing readiness.

Sonnet 5.5 · API

Describes a staged rollout, discovery via the transaction detail button, and identifies support, fraud, and compliance teams that must be ready.

Gemini 3.5 Flash-Lite · Gemini

Mentions a phased rollout but does not describe how users will discover the feature or identify which teams (beyond support) need to be ready.

All got wrong 1

Proposes tests that could failWrongWrongWrong
GPT-6 Luna · API

The pilot is described without a numeric kill threshold, measurement window, or the action each outcome would trigger.

Sonnet 5.5 · API

The A/B expansion gate relies on a fraud-reversal guardrail but does not specify a numeric threshold, leaving the test incomplete.

Gemini 3.5 Flash-Lite · Gemini

The beta test and phased rollout lack numeric thresholds, pass/fail criteria, or what action each outcome would trigger.

All got right 4

Addresses the actual decisionRightRightRight
GPT-6 Luna · API

The output commits early to building self‑serve disputes with enriched merchant recognition, framed for the squad, and implies the pilot outcome will decide expansion.

Sonnet 5.5 · API

The PRD commits to self-serve disputes with a clear design and a data-driven recommendation on merchant data, stating what would change the call.

Gemini 3.5 Flash-Lite · Gemini

Commits unambiguously to building in-app self-serve disputes and merchant enrichment, as requested, and names the readers.

Respects explicit constraintsRightRightRight
GPT-6 Luna · API

The PRD is within length, addresses the named squad, never makes disputing harder than paying, and respects all given constraints.

Sonnet 5.5 · API

Word count fits the range, reader is the squad, and every explicit constraint (including the regulator's rules) is turned into a testable requirement.

Gemini 3.5 Flash-Lite · Gemini

Fits within 1,000-1,400 words, addresses the named squad, and embeds the regulatory rules as enforceable requirements.

Produces the required deliverableRightRightRight
GPT-6 Luna · API

The output is a complete PRD addressed to the squad, within 1,000–1,400 words, and provides enough substance for the team to act on with light refinement.

Sonnet 5.5 · API

The PRD is complete, well-structured, and provides everything the squad needs to start building next quarter.

Gemini 3.5 Flash-Lite · Gemini

Provides a complete PRD in the right form for the squad, within the word limit, and could be acted on with light editing.

Builds the regulator's rules inRightRightRight
GPT-6 Luna · API

All three regulatory rules (acknowledge ≤2 days, resolve ≤15 days or explain, no harder than paying) appear as testable requirements.

Sonnet 5.5 · API

All three rules (2-business-day acknowledge, 15-business-day resolve or explain, no harder than paying) appear as testable requirements R8, R10, R13.

Gemini 3.5 Flash-Lite · Gemini

Turns all three regulatory mandates into testable requirements (2‑day acknowledgement, 15‑day resolution or explanation, friction parity with payments).

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 70% of 81 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstraChatGPT87.184.62None
2Sonnet 5.5API75.979.82None
3GPT-6.1 SolAPI82.660.92None
4Haiku 5.5API76.163.82None
5Opus 5.5Claude75.960.62None
6GPT-6 LunaAPI75.859.92None
7Gemini 3.8 FlashAPI49.828.22None
8Gemini 3.5 Flash-LiteGemini49.832.12None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 84, 106× the cheapest (1 of 2 outputs estimated)Sonnet 5.5 · API: task score 77, 35× the cheapestGPT-6.1 Sol · API: task score 71, 14× the cheapestHaiku 5.5 · API: task score 67, the cheapestOpus 5.5 · Claude: task score 65, 85× the cheapest (1 of 2 outputs estimated)GPT-6 Luna · API: task score 65, the cheapestGemini 3.8 Flash · API: task score 39, 9.9× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 39, 4.4× the cheapest (1 of 2 outputs estimated)GPT-6 Astra · ChatGPTSonnet 5.5 · APIGPT-6.1 Sol · APIHaiku 5.5 · APIOpus 5.5 · ClaudeGPT-6 Luna · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Writing the requirements document a team will build and test against.

Why it matters

A PRD is where ambiguity becomes either a decision or a bug. For AI products it must also say what happens when the model is uncertain or wrong. Most generated PRDs skip that part.

What good looks like

  • States the user problem and the decision the PRD enables
  • Specifies behaviour under uncertainty, failure and refusal
  • Names eval criteria and a launch bar
  • Defines success precisely enough to settle trade-offs
  • Says what is out of scope

Deliberately not measured

  • Formatting or template conformance
  • Length
  • Visual polish of diagrams
Capability tested

Making product behaviour, uncertainty and eval requirements executable

The failure we’re looking for

A generic feature spec that ignores AI failure behaviour

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

AI product PRD (core) · Conventional product PRD