Humanity’s Last PRD
Latest drop: Haiku 5.5 · 8 Oct
Tasks / Product Craft

Write a PRD

Can the model turn a brief into a spec engineers could build from, including how an AI feature behaves when it is wrong?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 25% were usable with at most a quick edit.

Reliably right

  1. Specifies AI failure behaviour100% pass
    It specifies what happens when uncertain or wrong: block ownership, show retry states, disable assignments on violation, and never fall back to calendar data.
    GPT-6 Astra · ChatGPT · Meeting summaries with action items
  2. Builds the regulator's rules in100% pass
    All three rules (2-business-day acknowledge, 15-business-day resolve or explain, no harder than paying) appear as testable requirements R8, R10, R13.
    Sonnet 5.5 · API · Disputes that aren't disputes
  3. Produces the required deliverable94% pass
    The PRD is complete, written for engineering and design leads, within the 1,000–1,500 word range, and directly actionable.
    GPT-6 Astra · ChatGPT · Meeting summaries with action items

Where it slips

  1. Proposes tests that could fail36% pass
    Launch gates have numeric thresholds and actions, but lack explicit measurement windows (e.g., pilot duration, number of meetings) needed to read out results.
    GPT-6.1 Sol · API · Meeting summaries with action items
  2. Avoids unsupported claims42% pass
    Presents the join log and identity tiers as established facts without labelling them as assumptions, when the context provides no support.
    Opus 5.5 · Claude · Meeting summaries with action items
  3. Uses the supplied evidence correctly44% pass
    The output invents a join log and identity tiers (Verified, Guest, Room) that are not in the supplied context, making unsupported claims about the current system.
    Opus 5.5 · Claude · Meeting summaries with action items

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Success settles trade-offs

    Does the PRD define success as a measurable outcome with a target and time frame, and say which goal wins when two of them conflict?

    Passes when Names a success metric with a target and time frame, and gives at least one explicit trade-off rule (for example, accepting lower coverage to keep precision above a stated level).

    Product and growth leadersOn Lenny’s Podcast

  2. Says who it's for, and who it isn't

    Does the PRD name the specific users it is for, say explicitly who it is not for, and make at least one scope decision that follows from that choice?

    Passes when Names the target users and at least one group it deliberately leaves out, and a scope decision visibly follows from the exclusion.

    Matt LeMayLenny’s Podcast

  3. Plans how people will find it

    Does the PRD say how the feature reaches people: how it rolls out, how users discover it, and who needs to be ready for it (support, sales), rather than stopping at what to build?

    Passes when Gives a staged rollout, a way users will discover the feature, and the teams that need to be ready, all fitted to this feature.

    Ami VoraLenny’s PodcastKarri SaarinenLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Proposes tests that could fail · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for card servicing at Clearbank. Write the PRD for self-serve disputes in the app, as Imogen asks below. It goes to the squad (engineering, design, support operations and compliance) who'll build it next quarter. 1,000 to 1,400 words. The dispute export and the background are attached.

What the model was given6 items: About Clearbank, imogen_note.md, how_disputes_work_today.md, regulator_rules.md, disputes_sample_last_quarter.csv (a random sample of 400 of last quarter's 2,400 disputes), merchant_data_option.md
About ClearbankA digital bank for individuals: current accounts and debit cards, run through a mobile app.
imogen_note.md1 lines · Download
From Imogen (Head of Product): "Write the PRD for self-serve disputes in the app. Support says it's their biggest cost after card freezes. The PRD goes to the squad (engineering, design, support ops and compliance) who will build it next quarter. 1,000 to 1,400 words."
how_disputes_work_today.md5 lines · Download
# How disputes work today

Customers raise a dispute by chatting to support. An agent collects details, then either explains the payment or opens a chargeback. Agents spend about 11 minutes per dispute. 2,400 disputes last quarter; 14% of them were resolved later than the regulator's deadline.

Card transactions show the merchant's billing name (for example "SQ *BRKLYN BAGEL 0412"), not its trading name or logo.
regulator_rules.md5 lines · Download
# Regulator's rules (summary from Legal)

- Acknowledge a dispute within 2 business days.
- Resolve genuine disputes within 15 business days, or tell the customer why it's taking longer.
- Don't make it harder to raise a dispute than to make a payment.
disputes_sample_last_quarter.csv (a random sample of 400 of last quarter's 2,400 disputes)dispute,customer_said,days_to_resolve,outcome CB-40000,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40006,I didn't receive what I paid for,17,Chargeback lost CB-40012,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40018,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40024,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40030,I didn't receive what I paid for,11,Chargeback lost CB-40036,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40042,I didn't receive what I paid for,16,Chargeback lost CB-40048,I was charged twice,5,Not a duplicate: two separate purchases CB-40054,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40060,I was charged twice,10,Not a duplicate: two separate purchases CB-40066,I was charged twice,7,Not a duplicate: two separate purchases CB-40072,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40078,I don't recognise this payment,5,Fraud confirmed CB-40084,I don't recognise this payment,7,Fraud confirmed CB-40090,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40096,I was charged twice,6,Not a duplicate: two separate purchases CB-40102,I didn't receive what I paid for,7,Chargeback won CB-40108,I don't recognise this payment,6,Fraud confirmed CB-40114,I was charged twice,9,Duplicate refunded CB-40120,I was charged twice,7,Not a duplicate: two separate purchases CB-40126,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40132,I was charged twice,9,Duplicate refunded CB-40138,Subscription I cancelled,18,Chargeback won CB-40144,I didn't receive what I paid for,21,Chargeback won CB-40150,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40156,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40162,I don't recognise this payment,6,Fraud confirmed CB-40168,I was charged twice,4,Not a duplicate: two separate purchases CB-40174,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40180,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-40186,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40192,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-40198,I don't recognise this payment,1,Fraud confirmed CB-40204,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40210,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40216,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40222,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40228,I didn't receive what I paid for,29,Chargeback won CB-40234,I don't recognise this payment,8,Fraud confirmed CB-40240,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40246,I didn't receive what I paid for,10,Merchant refunded CB-40252,I don't recognise this payment,4,Fraud confirmed CB-40258,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40264,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40270,I didn't receive what I paid for,21,Chargeback won CB-40276,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40282,I didn't receive what I paid for,17,Merchant refunded CB-40288,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40294,I don't recognise this payment,1,Fraud confirmed CB-40300,I don't recognise this payment,7,Fraud confirmed CB-40306,I don't recognise this payment,8,Fraud confirmed CB-40312,I didn't receive what I paid for,20,Merchant refunded CB-40318,I didn't receive what I paid for,26,Chargeback lost CB-40324,Subscription I cancelled,10,Merchant refunded CB-40330,I don't recognise this payment,1,Fraud confirmed CB-40336,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-40342,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40348,Fraud: I didn't make this payment,9,"Fraud confirmed, refunded" CB-40354,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40360,I was charged twice,9,Not a duplicate: two separate purchases CB-40366,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40372,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40378,I don't recognise this payment,8,Fraud confirmed CB-40384,I didn't receive what I paid for,14,Merchant refunded CB-40390,I don't recognise this payment,7,Fraud confirmed CB-40396,I don't recognise this payment,8,Fraud confirmed CB-40402,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40408,I was charged twice,5,Not a duplicate: two separate purchases CB-40414,I didn't receive what I paid for,13,Chargeback lost CB-40420,Subscription I cancelled,11,Chargeback won CB-40426,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-40432,I didn't receive what I paid for,13,Chargeback won CB-40438,Fraud: I didn't make this payment,4,"Fraud confirmed, refunded" CB-40444,I was charged twice,9,Duplicate refunded CB-40450,I didn't receive what I paid for,18,Chargeback won CB-40456,I don't recognise this payment,1,Fraud confirmed CB-40462,I don't recognise this payment,4,Fraud confirmed CB-40468,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40474,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40480,Fraud: I didn't make this payment,8,"Fraud confirmed, refunded" CB-40486,I didn't receive what I paid for,27,Chargeback lost CB-40492,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40498,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40504,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40510,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40516,I don't recognise this payment,5,Fraud confirmed CB-40522,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40528,Fraud: I didn't make this payment,8,"Fraud confirmed, refunded" CB-40534,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40540,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40546,I don't recognise this payment,2,Fraud confirmed CB-40552,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40558,I was charged twice,5,Duplicate refunded CB-40564,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40570,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40576,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40582,I didn't receive what I paid for,14,Chargeback lost CB-40588,I didn't receive what I paid for,18,Chargeback lost CB-40594,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40600,Fraud: I didn't make this payment,2,"Fraud confirmed, refunded" CB-40606,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40612,I don't recognise this payment,3,Fraud confirmed CB-40618,Fraud: I didn't make this payment,5,"Fraud confirmed, refunded" CB-40624,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40630,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40636,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40642,Fraud: I didn't make this payment,11,"Fraud confirmed, refunded" CB-40648,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40654,I don't recognise this payment,5,Fraud confirmed CB-40660,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40666,I don't recognise this payment,6,Fraud confirmed CB-40672,I was charged twice,5,Not a duplicate: two separate purchases CB-40678,Subscription I cancelled,10,Chargeback won CB-40684,Fraud: I didn't make this payment,9,"Fraud confirmed, refunded" CB-40690,Subscription I cancelled,7,Chargeback won CB-40696,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40702,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40708,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40714,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40720,I was charged twice,13,Not a duplicate: two separate purchases CB-40726,Subscription I cancelled,13,Chargeback won CB-40732,I don't recognise this payment,3,Fraud confirmed CB-40738,I didn't receive what I paid for,18,Chargeback lost CB-40744,Fraud: I didn't make this payment,8,"Fraud confirmed, refunded" CB-40750,I was charged twice,11,Not a duplicate: two separate purchases CB-40756,Subscription I cancelled,10,Merchant refunded CB-40762,I was charged twice,7,Duplicate refunded CB-40768,Subscription I cancelled,18,Chargeback won CB-40774,I didn't receive what I paid for,23,Chargeback won CB-40780,I didn't receive what I paid for,7,Chargeback won CB-40786,Subscription I cancelled,5,Chargeback won CB-40792,I didn't receive what I paid for,10,Merchant refunded CB-40798,I was charged twice,9,Duplicate refunded CB-40804,I didn't receive what I paid for,24,Merchant refunded CB-40810,I don't recognise this payment,2,Fraud confirmed CB-40816,Fraud: I didn't make this payment,11,"Fraud confirmed, refunded" CB-40822,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40828,Subscription I cancelled,13,Chargeback won CB-40834,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40840,I don't recognise this payment,7,Fraud confirmed CB-40846,Subscription I cancelled,7,Merchant refunded CB-40852,I don't recognise this payment,3,Fraud confirmed CB-40858,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40864,I was charged twice,4,Not a duplicate: two separate purchases CB-40870,I don't recognise this payment,8,Fraud confirmed CB-40876,I was charged twice,6,Not a duplicate: two separate purchases CB-40882,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-40888,I don't recognise this payment,8,Fraud confirmed CB-40894,I don't recognise this payment,6,Fraud confirmed CB-40900,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40906,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40912,I don't recognise this payment,8,Fraud confirmed CB-40918,I didn't receive what I paid for,21,Chargeback won CB-40924,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-40930,Fraud: I didn't make this payment,8,"Fraud confirmed, refunded" CB-40936,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40942,I was charged twice,6,Duplicate refunded CB-40948,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40954,I didn't receive what I paid for,14,Chargeback won CB-40960,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40966,I was charged twice,4,Not a duplicate: two separate purchases CB-40972,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40978,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40984,I didn't receive what I paid for,9,Merchant refunded CB-40990,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40996,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41002,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41008,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41014,Fraud: I didn't make this payment,2,"Fraud confirmed, refunded" CB-41020,I didn't receive what I paid for,27,Chargeback won CB-41026,I didn't receive what I paid for,28,Chargeback won CB-41032,I was charged twice,7,Duplicate refunded CB-41038,I don't recognise this payment,2,Fraud confirmed CB-41044,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41050,I don't recognise this payment,4,Fraud confirmed CB-41056,Fraud: I didn't make this payment,11,"Fraud confirmed, refunded" CB-41062,I was charged twice,3,Duplicate refunded CB-41068,I don't recognise this payment,1,Fraud confirmed CB-41074,I don't recognise this payment,1,Fraud confirmed CB-41080,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41086,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41092,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41098,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41104,I didn't receive what I paid for,29,Merchant refunded CB-41110,Subscription I cancelled,16,Chargeback won CB-41116,I didn't receive what I paid for,22,Chargeback won CB-41122,I didn't receive what I paid for,13,Chargeback lost CB-41128,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41134,I was charged twice,10,Not a duplicate: two separate purchases CB-41140,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41146,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41152,I was charged twice,5,Duplicate refunded CB-41158,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41164,I didn't receive what I paid for,12,Chargeback lost CB-41170,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41176,Fraud: I didn't make this payment,2,"Fraud confirmed, refunded" CB-41182,I don't recognise this payment,1,Fraud confirmed CB-41188,I don't recognise this payment,3,Fraud confirmed CB-41194,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41200,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41206,Fraud: I didn't make this payment,5,"Fraud confirmed, refunded" CB-41212,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41218,Fraud: I didn't make this payment,6,"Fraud confirmed, refunded" CB-41224,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41230,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41236,I don't recognise this payment,3,Fraud confirmed CB-41242,Subscription I cancelled,9,Merchant refunded CB-41248,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41254,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41260,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41266,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41272,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41278,I didn't receive what I paid for,11,Chargeback lost CB-41284,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41290,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41296,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-41302,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41308,I was charged twice,11,Duplicate refunded CB-41314,I didn't receive what I paid for,27,Chargeback won CB-41320,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41326,Fraud: I didn't make this payment,5,"Fraud confirmed, refunded" CB-41332,I was charged twice,12,Not a duplicate: two separate purchases CB-41338,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-41344,I didn't receive what I paid for,17,Merchant refunded CB-41350,I didn't receive what I paid for,21,Chargeback won CB-41356,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41362,Subscription I cancelled,14,Merchant refunded CB-41368,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41374,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41380,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41386,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41392,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41398,I don't recognise this payment,2,Fraud confirmed CB-41404,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41410,I don't recognise this payment,3,Fraud confirmed CB-41416,I don't recognise this payment,5,Fraud confirmed CB-41422,I was charged twice,12,Duplicate refunded CB-41428,Subscription I cancelled,9,Merchant refunded CB-41434,I was charged twice,6,Duplicate refunded CB-41440,Subscription I cancelled,15,Merchant refunded CB-41446,I don't recognise this payment,5,Fraud confirmed CB-41452,I don't recognise this payment,7,Fraud confirmed CB-41458,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41464,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41470,I was charged twice,6,Not a duplicate: two separate purchases CB-41476,I don't recognise this payment,5,Fraud confirmed CB-41482,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41488,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41494,I didn't receive what I paid for,26,Merchant refunded CB-41500,I didn't receive what I paid for,24,Chargeback won CB-41506,Subscription I cancelled,20,Chargeback won CB-41512,I didn't receive what I paid for,29,Merchant refunded CB-41518,Subscription I cancelled,20,Chargeback won CB-41524,I don't recognise this payment,8,Fraud confirmed CB-41530,I was charged twice,3,Not a duplicate: two separate purchases CB-41536,I didn't receive what I paid for,8,Chargeback won CB-41542,I don't recognise this payment,2,Fraud confirmed CB-41548,I don't recognise this payment,2,Fraud confirmed CB-41554,I was charged twice,12,Not a duplicate: two separate purchases CB-41560,Subscription I cancelled,6,Chargeback won CB-41566,I was charged twice,3,Not a duplicate: two separate purchases CB-41572,I don't recognise this payment,1,Fraud confirmed CB-41578,Fraud: I didn't make this payment,7,"Fraud confirmed, refunded" CB-41584,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41590,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-41596,Subscription I cancelled,16,Merchant refunded CB-41602,Subscription I cancelled,15,Merchant refunded CB-41608,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41614,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41620,I was charged twice,5,Duplicate refunded CB-41626,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41632,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41638,Fraud: I didn't make this payment,3,"Fraud confirmed, refunded" CB-41644,I don't recognise this payment,3,Fraud confirmed CB-41650,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41656,I don't recognise this payment,6,Fraud confirmed CB-41662,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41668,I didn't receive what I paid for,12,Chargeback won CB-41674,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41680,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41686,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-41692,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41698,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41704,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41710,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41716,Fraud: I didn't make this payment,8,"Fraud confirmed, refunded" CB-41722,I don't recognise this payment,3,Fraud confirmed CB-41728,I was charged twice,12,Duplicate refunded CB-41734,Fraud: I didn't make this payment,5,"Fraud confirmed, refunded" CB-41740,I don't recognise this payment,6,Fraud confirmed CB-41746,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41752,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41758,I was charged twice,13,Duplicate refunded CB-41764,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41770,I didn't receive what I paid for,23,Chargeback won CB-41776,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41782,I didn't receive what I paid for,27,Chargeback won CB-41788,I didn't receive what I paid for,23,Chargeback lost CB-41794,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41800,I don't recognise this payment,3,Fraud confirmed CB-41806,I don't recognise this payment,1,Fraud confirmed CB-41812,I was charged twice,4,Not a duplicate: two separate purchases CB-41818,I don't recognise this payment,8,Fraud confirmed CB-41824,I didn't receive what I paid for,18,Chargeback won CB-41830,Subscription I cancelled,13,Chargeback won CB-41836,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41842,I don't recognise this payment,7,Fraud confirmed CB-41848,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41854,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41860,I was charged twice,5,Duplicate refunded CB-41866,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41872,I don't recognise this payment,8,Fraud confirmed CB-41878,I don't recognise this payment,2,Fraud confirmed CB-41884,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41890,I was charged twice,7,Not a duplicate: two separate purchases CB-41896,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41902,Fraud: I didn't make this payment,9,"Fraud confirmed, refunded" CB-41908,I don't recognise this payment,4,Fraud confirmed CB-41914,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41920,I didn't receive what I paid for,24,Merchant refunded CB-41926,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41932,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-41938,Subscription I cancelled,15,Chargeback won CB-41944,I didn't receive what I paid for,13,Chargeback won CB-41950,I don't recognise this payment,2,Fraud confirmed CB-41956,I didn't receive what I paid for,24,Chargeback lost CB-41962,I don't recognise this payment,4,Fraud confirmed CB-41968,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41974,Subscription I cancelled,15,Chargeback won CB-41980,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41986,Fraud: I didn't make this payment,2,"Fraud confirmed, refunded" CB-41992,Fraud: I didn't make this payment,6,"Fraud confirmed, refunded" CB-41998,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-42004,I didn't receive what I paid for,12,Merchant refunded CB-42010,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42016,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42022,I was charged twice,11,Duplicate refunded CB-42028,I didn't receive what I paid for,25,Merchant refunded CB-42034,I didn't receive what I paid for,10,Merchant refunded CB-42040,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42046,I was charged twice,8,Duplicate refunded CB-42052,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-42058,I don't recognise this payment,7,Fraud confirmed CB-42064,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-42070,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42076,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-42082,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-42088,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-42094,Subscription I cancelled,12,Chargeback won CB-42100,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-42106,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-42112,Subscription I cancelled,18,Chargeback won CB-42118,Fraud: I didn't make this payment,6,"Fraud confirmed, refunded" CB-42124,I don't recognise this payment,5,Fraud confirmed CB-42130,I didn't receive what I paid for,14,Chargeback lost CB-42136,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-42142,Fraud: I didn't make this payment,2,"Fraud confirmed, refunded" CB-42148,I was charged twice,12,Not a duplicate: two separate purchases CB-42154,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-42160,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-42166,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42172,Fraud: I didn't make this payment,9,"Fraud confirmed, refunded" CB-42178,I don't recognise this payment,3,Fraud confirmed CB-42184,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-42190,I was charged twice,7,Duplicate refunded CB-42196,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-42202,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-42208,I didn't receive what I paid for,21,Chargeback won CB-42214,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-42220,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-42226,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-42232,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-42238,I was charged twice,10,Duplicate refunded CB-42244,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-42250,Subscription I cancelled,11,Chargeback won CB-42256,Fraud: I didn't make this payment,3,"Fraud confirmed, refunded" CB-42262,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-42268,I didn't receive what I paid for,16,Merchant refunded CB-42274,I was charged twice,3,Duplicate refunded CB-42280,Fraud: I didn't make this payment,9,"Fraud confirmed, refunded" CB-42286,I was charged twice,11,Duplicate refunded CB-42292,I was charged twice,9,Not a duplicate: two separate purchases CB-42298,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-42304,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-42310,Subscription I cancelled,16,Merchant refunded CB-42316,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-42322,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-42328,Fraud: I didn't make this payment,3,"Fraud confirmed, refunded" CB-42334,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-42340,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-42346,I didn't receive what I paid for,19,Merchant refunded CB-42352,I was charged twice,12,Duplicate refunded CB-42358,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-42364,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-42370,I don't recognise this payment,3,Fraud confirmed CB-42376,I didn't receive what I paid for,17,Chargeback won CB-42382,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-42388,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42394,I don't recognise this payment,5,Customer recognised it once shown the merchant's details
merchant_data_option.md3 lines · Download
# Merchant data we could use

Our card processor offers an enriched merchant feed (trading name, logo, location, website) for 94% of transactions, at $0.0008 per transaction. We process about 9 million card transactions a month.
What a strong answer doesThe answer key the graders mark against

Starts from the data: in the sample, 232 of 400 disputes (58%, so about 1,400 of the quarter's 2,400) were 'I don't recognise this payment', and 169 of those (about three in four) were resolved simply by showing the merchant's details; so the first thing to build is recognisable transactions (trading name, logo, location from the enriched feed, about $7,200 a month for 9 million transactions), shown before someone raises a dispute, with self-serve disputes for the genuine cases. Defines success with targets (disputes per 1,000 transactions, share resolved without an agent, regulator deadlines met against 14% late today) and says which wins when they conflict (never make disputing harder than paying, per the regulator). Covers the flows for genuine disputes (charged twice, not received, cancelled subscription, fraud) with acknowledgement within 2 business days and resolution within 15, what happens when the merchant data is missing (6% of transactions), and how the rollout reaches people, with support ready. Says who it's not for (for example business accounts, or disputes over a set amount that still go to an agent) and what's out of scope.

Critical failures (cap the score at 40)
  • Adds steps that make raising a dispute harder than making a payment
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Sonnet 5.5API83Usable with a quick editGPT-6 AstraChatGPT73Needs more workGPT-6 LunaAPI69Needs more workHaiku 5.5API69Needs more workOpus 5.5Claude60Needs more workGPT-6.1 SolAPI58Needs more workGemini 3.5 Flash-LiteGemini40Needs more workGemini 3.8 FlashAPI37Needs more work
GPT-6 LunaAPI

Needs more than a quick edit8 of 12 checks passed

The PRD lacks explicit computation of the recognition share, an explicit trade‑off rule, and numeric pilot kill criteria, so a PM would need to rework these sections before handing it off.

Output

PRD: Self-serve card disputes in the Clearbank app

Owner: Card Servicing Delivery: Next quarter Audience: Engineering, Design, Support Operations, Compliance

Summary

Let customers understand an unfamiliar card transaction and, if they still believe it is wrong, raise and track a dispute in the app without first contacting support. The flow should help customers recognise legitimate payments while keeping the route to report a genuine issue clear and easy.

Today, customers raise disputes in chat. Agents collect details, explain the payment or open a chargeback; each dispute takes about 11 minutes. Last quarter there were 2,400 disputes—around 440 agent hours—and 14% were resolved later than the regulator’s deadline. The sample of 400 disputes also shows a recurring opportunity: many customers who said they did not recognise a payment later recognised it when shown merchant details. The current transaction descriptor may show a billing name, such as “SQ *BRKLYN BAGEL 0412”, rather than the trading name a customer knows.

Problem and opportunity

Customers cannot reliably identify a payment from its billing descriptor, so some contact support unnecessarily. Customers with genuine disputes face a chat-based intake and may not receive timely updates. Support must collect information manually and manage deadlines across cases.

The processor’s enriched merchant feed provides a trading name, logo, location and website for 94% of transactions. At $0.0008 per transaction, applying it to the eligible share of approximately 9 million monthly transactions would cost about $6,768 per month (subject to the vendor’s billing basis). This is a material ongoing cost; we should measure whether the enrichment improves recognition and reduces support demand.

Goals and success measures

  1. Reduce avoidable agent work while preserving a simple route to dispute.
  2. Give customers a clear, trackable way to report a genuine card-payment problem.
  3. Meet the regulator’s acknowledgement and resolution-or-delay-notice requirements.

Proposed launch targets, measured over the first 90 days against a pre-launch baseline: - Reduce support-handled dispute contacts per 1,000 card transactions by 30%. - Acknowledge at least 95% of in-app disputes within 2 business days. - For at least 95% of genuine disputes, resolve within 15 business days or send a clear explanation that the case is taking longer, with the next update expectation. - No material increase in complaints, reopened cases, confirmed fraud loss, or customers abandoning a genuine dispute compared with baseline.

Track the number of customers who recognise a payment, continue to raise a dispute, abandon at each step, contact support after submitting, and the cost and coverage of enriched merchant data. Segment outcomes by dispute reason and whether merchant enrichment was available. The sample is useful evidence of the recognition problem, but it is not a substitute for a live baseline or a causal estimate of savings.

Users and scope

Primary users: Clearbank personal current-account customers with a debit-card transaction they do not understand or believe is incorrect.

In scope for MVP: A dispute journey launched from an eligible card transaction; merchant context; guided reason selection and information capture; case submission and acknowledgement; in-app status and updates; operational case routing and SLA monitoring.

Out of scope: Replacing the chargeback process, guaranteeing refunds, automated denial of claims, disputes for non-card payment types, and a general-purpose support-chat redesign.

Proposed experience

  1. Start from the transaction. Put a visible “Get help with this payment” action on eligible card transaction details. Do not require customers to find a separate disputes area or contact chat. Eligibility must be determined by Compliance and Operations using applicable scheme and processing rules. If a transaction cannot be disputed in-app, explain why and provide a clear alternative support route; never leave the customer at a dead end.
  1. Help identify the merchant. Show the enriched trading name and, when available and reliable, logo, location and website alongside the original billing descriptor, amount and date. Do not imply that a logo or website proves the customer made the purchase. If data is unavailable, show the billing descriptor and explain that the merchant may use a different name. Do not make merchant lookup a prerequisite to reporting.
  1. Let the customer choose what happened. Offer plain-language reasons, including “I don’t recognise this”, “I was charged twice”, “I didn’t receive what I paid for”, “I cancelled a subscription”, and “I didn’t make this payment”. Keep “I don’t recognise this” distinct from confirmed fraud. Ask only the follow-up questions needed for the selected reason—for example, whether the customer contacted the merchant or when they cancelled. Explain why requested information helps, and allow submission without optional evidence.
  1. Offer relevant help without blocking a claim. If the customer recognises the merchant, offer a simple way to exit without opening a case. For suspected fraud, offer an immediate card-freeze action and explain that freezing the card is separate from reporting the transaction. Never require a freeze, merchant contact, or troubleshooting step before the customer can submit.
  1. Confirm and track. On successful submission, show a case reference, the reported transaction and reason, confirmation that the dispute has been received, and what happens next. Provide a case status and updates in the app, with notifications where enabled. If further information is required, explain what is needed and how to provide it. Do not suggest a refund is guaranteed or that a chargeback has already been won.

Operational and compliance requirements

  • The system must record submission and acknowledgement timestamps, customer-provided details, case status, decisions, communications and audit history.
  • An in-app receipt should serve as acknowledgement only if Compliance confirms it meets the applicable acknowledgement requirement. Otherwise, send a compliant acknowledgement within 2 business days.
  • Operations needs a queue with reason, age, owner, priority and deadlines, plus alerts before SLA breaches. Cases should not silently age in a queue.
  • Genuine disputes must be resolved within 15 business days, or the customer must be told why it is taking longer. The message must include the reason and a meaningful next update expectation; Operations must continue to monitor the case.
  • Any classification or routing automation may assist staff but must not automatically reject a claim. A customer who disagrees with a suggested explanation must still be able to proceed.
  • The journey must be accessible, use clear language, and not be harder to reach or complete than making a payment. Store and share customer and merchant data only for appropriate servicing purposes, with vendor use, security and retention reviewed by Compliance.

Dependencies and open decisions

Before build completion, Compliance and Operations must confirm eligible transaction types, claim deadlines, reason-specific evidence needs, the definition of “genuine dispute” for reporting, and the compliant wording for acknowledgements and delay notices. Engineering must confirm processor-feed availability, fallback behaviour, case-system integration, and reliable business-day SLA calculation. Product and Finance should agree an enrichment pilot or rollout threshold using recognition, contact reduction, coverage and vendor cost; the feed’s 6% coverage gap needs a tested fallback.

Rollout and risks

Pilot with a limited customer cohort after internal testing of each dispute reason, merchant-data failures, duplicate submissions, accessibility and SLA alerts. Compare pilot outcomes with a pre-launch baseline before expanding. Key risks are customers mistaking a legitimate merchant for fraud, poor enrichment data, confusing a submitted claim with a guaranteed refund, and moving work from chat into an unmanaged queue. Mitigate through clear descriptors and caveats, an always-available dispute route, explicit case status, operational ownership and deadline monitoring.

Grades and run details

Decision model 83 · LLM judge 62
Decision model checks
  • passUses the supplied evidence correctly47%
  • passAddresses the actual decision39%
  • partialRespects explicit constraints57%
  • passIdentifies material uncertainty68%
  • passAvoids unsupported claims50%
  • partialProduces the required deliverable34%
  • partialFinds that most disputes are recognition71%
  • passBuilds the regulator's rules in97%
  • passSuccess settles trade-offs92%
  • partialProposes tests that could fail76%
  • passSays who it's for, and who it isn't30%
  • passPlans how people will find it64%
Run
Run
#1
API response time
32 s
Submitted
6 Oct 2026
GPT-6 AstraChatGPT

Needs more than a quick edit9 of 12 checks passed

Missing an explicit user discovery plan and a non-target user segment, and pause criteria need numeric thresholds. These gaps require more than a quick wording edit.

Output

PRD: Self-serve card disputes

Owner: Card Servicing PM Delivery: Next quarter Squad: Engineering, Design, Support Operations, Compliance Status: Proposed; merchant-data procurement and legal implementation require approval

1. Problem and product decision

Today, customers must chat with support to question a card payment. Agents collect information, explain payments or open chargebacks, spending approximately 11 minutes per dispute. Last quarter’s 2,400 disputes consumed approximately 440 agent hours; 14% were resolved later than the regulator’s deadline.

The opportunity is not simply to replace chat with a chargeback form. Many customers need help identifying a payment; others need fast access to a genuine dispute or fraud investigation.

We will build a transaction-level journey that helps customers understand payments, submit unresolved concerns without an agent, and track progress. Identification help must remain optional: it must never become a barrier to reporting fraud or disputing a payment.

2. Evidence and economics

The random sample of 400 disputes shows:

FindingProduct implication
233 began “I don’t recognise this payment.” Of these, 170 ended with customer recognition and 63 with confirmed fraud.Show useful merchant information, but never assume an unfamiliar payment is legitimate.
Recognition-only cases represent 42.5% of the sample.Preventing confusion is a major opportunity, not a secondary feature.
Of 47 duplicate complaints, 24 were separate purchases and 23 were refunded duplicates.Offer comparison information without automatically dismissing claims.
57 concerned non-receipt, 27 cancelled subscriptions, and 36 explicitly reported fraud.Provide reason-specific intake and operational routing.

Extrapolating recognition-only cases suggests approximately 1,020 cases, or 187 agent hours, per quarter could theoretically be avoided. This is an optimistic ceiling, not a savings forecast.

The processor’s enriched merchant feed covers 94% of transactions. At nine million transactions monthly, it costs $7,200/month or $21,600/quarter. Recognition-only savings would require approximately $116 per avoided agent hour just to cover that fee, before development costs and incomplete coverage.

Therefore, we will not approve blanket feed purchasing on support savings alone. Engineering and Procurement must confirm selective-lookup availability, pricing and data quality. The squad will build an enrichment-compatible detail view, using existing verified information initially if necessary. Full-feed adoption requires a separate business case, including measured recognition improvements.

The export does not identify business versus calendar days or extension notices. We cannot use it alone to reconstruct regulatory breaches.

3. Goals, measures and guardrails

Targets apply to the first full quarter after rollout and are hypotheses unless regulatory:

  • Reduce agent handling hours per 1,000 active card customers by 25%, against a matched baseline; include follow-up contacts and operational review.
  • Achieve 70% self-serve submission among submitted in-scope disputes.
  • Acknowledge 100% within two business days.
  • Resolve genuine disputes, or send a substantive explanation of delay, within 15 business days in 100% of cases.
  • No deterioration in confirmed-fraud reporting time, repeat complaints or accessibility versus baseline.

Track median and p90 resolution time separately from deadline compliance. An extension notice is not a resolution.

Instrument detail views, help entry, recognition confirmations, reason selection, submission, abandonment, support handoff, case updates and outcomes. Report by reason and enrichment availability. Only explicit recognition counts as recognition success; exits do not.

4. Scope

Included: Mobile-app debit-card payment identification, intake for the five sampled reasons plus “Something else,” evidence upload, case acknowledgement, status tracking, operational routing and deadline controls.

Excluded: Automated liability decisions, automatic chargeback adjudication, rebuilding card-scheme integrations, non-card disputes, and a new refund policy. Existing specialists retain responsibility for investigation, refunds and chargebacks.

Pending payments must have an accessible reporting route, even if a chargeback cannot yet proceed. Explain the distinction and route suspected fraud immediately. Support remains available throughout.

5. Customer experience and requirements

A. Understand the payment

Transaction details display the original billing descriptor, amount, date and available verified merchant information. Enriched trading name, logo, location and website appear when available and sufficiently reliable; retain the descriptor for traceability. Missing or uncertain matches must not block the journey or imply legitimacy.

Provide prominent “Report a problem” and “I didn’t make this payment” actions. Identification guidance is skippable. Only an explicit “I recognise this payment” ends the identification journey; customers can return later to dispute it.

B. Raise a concern

Prefill transaction and customer details. Let customers choose:

  • Unrecognised / not authorised: Submit directly to the fraud team; offer the existing card-freeze action without making freezing a prerequisite.
  • Charged twice: Show potentially related transactions and let customers select the second payment or continue without finding it. Similar amounts do not prove duplication.
  • Goods or services not received: Ask what was expected and the expected delivery date.
  • Cancelled subscription: Ask when cancellation occurred and invite supporting evidence.
  • Something else: Accept a short description and route for triage.

Collect a concise account of the problem. Request receipts, cancellation confirmations and merchant correspondence where useful, but permit evidence later. Merchant contact is not a universal submission prerequisite.

Customers review and submit without re-entering known information. Save drafts, support screen readers and dynamic text, and provide contextual support with entered details preserved.

C. Acknowledge and track

After durable case creation, immediately display a case reference, receipt timestamp, summary and next-update date. Deliver an in-app acknowledgement and notification; do not rely solely on push delivery.

The case screen shows understandable states: Received, Under review, Information needed, Awaiting external response, Resolved. Show outstanding requests, update history and any refund status separately.

Resolution includes the decision, explanation and support/escalation route. Never promise a refund or successful chargeback.

6. Operational and technical requirements

Create one case record shared by app and support, containing the transaction, reported reason, narrative, attachments, owner, timestamps, deadlines and communication history.

Submission must be idempotent. Retries cannot create duplicate cases; an existing case should be surfaced with an option to add information. Confirm receipt only after persistence. Failed submissions retain entered data and expose a support fallback.

Support Operations owns reason-based queues, fraud prioritisation, daily deadline monitoring and capacity planning. Agents can request evidence, change routing, record decisions and send approved updates without asking customers to repeat information.

Engineering must implement:

  • Configurable business-day calendars and deadline calculation.
  • Alerts before deadlines, including day-one acknowledgement and day-ten/day-fourteen resolution warnings.
  • Escalation to a named operational owner before breach.
  • Reconciliation between app submissions and operational queues.
  • Encryption, role-based access, attachment validation and auditable state changes.

Compliance must approve clock-start rules, holidays, retention, extension wording and any additional fraud/refund obligations. Preserve the earliest report timestamp across channels; asking for evidence must not silently reset deadlines. Delay notices must explain the actual reason and next update, not merely mark cases compliant.

7. Delivery, acceptance and rollout

Weeks 1–2: Validate merchant-data options; audit existing case-system integration; agree legal rules; baseline reason-level handling time. Design tests prototypes with customers, including accessibility needs.

Weeks 3–8: Build intake, case integration, tracking and deadline controls. Develop merchant-detail improvements in parallel without making reporting dependent on enrichment.

Weeks 9–10: Run end-to-end testing, operational rehearsal and a limited pilot. Test missing merchant data, uncertain matches, fraud bypass, pending payments, failed uploads, repeat taps, holiday deadlines and cross-channel reporting.

Weeks 11–12: Expand progressively after Engineering, Support Operations and Compliance sign-off.

Launch requires zero lost or duplicate cases in acceptance tests, correct deadline calculations, accessible submission, and verified acknowledgement delivery. Design and Compliance must demonstrate that raising a dispute is no harder than making a payment, using comparative journey testing rather than assuming a form satisfies the rule.

Pause expansion for lost submissions, misleading merchant matches, increased fraud-reporting delay or deteriorating deadline compliance. Rollback must preserve existing cases, clocks and customer access to support.

Grades and run details

Decision model 83 · LLM judge 69
Decision model checks
  • passUses the supplied evidence correctly13%
  • passAddresses the actual decision71%
  • partialRespects explicit constraints46%
  • passIdentifies material uncertainty98%
  • passAvoids unsupported claims69%
  • passProduces the required deliverable51%
  • passFinds that most disputes are recognition70%
  • passBuilds the regulator's rules in100%
  • passSuccess settles trade-offs99%
  • partialProposes tests that could fail62%
  • partialSays who it's for, and who it isn't38%
  • partialPlans how people will find it32%
Run
Run
#1
Time to output
1.5 min
Submitted
6 Oct 2026
Opus 5.5Claude

Needs more than a quick edit6 of 12 checks passed

The PRD contains an inconsistent statistic (42% vs 48%) and lacks explicit trade-off rules, user definition, and rollout plan, requiring more than a quick edit.

Output

PRD: Self-serve card disputes in the app

Owner: PM, Card Servicing · Squad: Engineering, Design, Support Ops, Compliance · Target: next quarter

1. Summary

Customers can only raise a card dispute today by chatting with support. An agent spends about 11 minutes on each one. Last quarter there were 2,400 disputes, which is roughly 440 agent hours, and 14% were resolved after the regulator's deadline.

This project lets customers raise and track disputes from the transaction screen. It answers the most common question ("what is this payment?") before a dispute is needed. Every case that does need an agent arrives fully documented.

The most important finding: 42% of disputes aren't really disputes. The customer didn't recognise a merchant billing name. However, the late-resolution problem sits in a different group of cases, so the project has to solve both.

2. What the data says

These figures come from a random sample of 400 of last quarter's 2,400 disputes.

Customer saidShareOutcomesResolved in >15 days
I don't recognise this payment58% (232)169 recognised it once shown merchant details; 63 fraud0
I didn't receive what I paid for14% (56)24 chargeback won, 15 lost, 17 merchant refunded36 (64%)
I was charged twice12% (48)24 genuine duplicates, 24 separate purchases0
Fraud: I didn't make this payment9% (36)All confirmed and refunded0
Subscription I cancelled7% (28)18 chargeback won, 10 merchant refunded8 (29%)

What this means:

  1. Most disputes can be answered with information. 169 "don't recognise" cases plus 24 "not a duplicate" cases make 48% of disputes. These ended with the customer understanding a payment they had made. The cause is that we show billing names like "SQ \*BRKLYN BAGEL 0412" instead of the trading name.
  2. "Don't recognise" is also a real fraud channel. 27% of those cases were fraud. Fraud from both reasons combined is about 25% of all disputes. Any feature that explains payments must never slow down a customer who is genuinely a victim.
  3. Late resolutions are all chargebacks. Every sample case that took more than 15 days was a "not received" or "subscription" dispute. Deflecting unrecognised payments will cut agent time, but it won't fix the deadline problem. That needs its own work (section 4.5).
  4. Many merchant disputes end with the merchant refunding. This happened in 30% of "not received" cases and 36% of subscription cases.

3. Goals and non-goals

Goals - Customers can raise any dispute in the app, with no chat required. - Fewer disputes raised for payments the customer actually made. - Every chargeback case is either resolved within 15 business days or the customer is told why not, with no exceptions. - Less agent time per dispute.

Non-goals (v1) - Automated chargeback decisions or automatic refunds. Agents still decide every case. - Disputes on non-card payments. - Changing how chargebacks are filed with the card scheme.

4. Requirements

4.1 Entry point

  • R1. Every card transaction's detail screen has a "Problem with this payment?" action.
  • R2. The customer picks one of five reasons, matching today's categories: don't recognise it, I didn't make it (fraud), charged twice, didn't receive it, cancelled subscription.
  • R3. Compliance constraint. Raising a dispute must never take more steps than making a payment. Design will count taps for each path against our card payment and transfer flows. Compliance must sign off on the counts before launch. Every explanation screen below is shown alongside the dispute action, never in front of it.

4.2 "I don't recognise this payment"

  • R4. Show enriched merchant details: trading name, logo, location and website. Also show the customer's previous payments to the same merchant.
  • R5. The same screen offers two equally weighted buttons: "I recognise it now" and "I still don't recognise it".
  • R6. "I still don't recognise it" goes straight to the fraud path (4.3). It asks no further questions.
  • R7. If no enrichment exists (about 6% of transactions), show the raw billing name, merchant category and location from the transaction record. The flow is otherwise unchanged.

4.3 Fraud

  • R8. Offer to freeze the card on the first screen of the fraud path. The customer can decline.
  • R9. Let the customer flag other transactions on the same card as unauthorised in the same submission.
  • R10. The case goes to the fraud queue at high priority.

4.4 Charged twice / not received / subscription

  • R11. Charged twice. Show the two transactions side by side, with time, amount, location and card-present status. Half of these cases were separate purchases. The customer can still continue to raise the dispute.
  • R12. Not received / subscription. Ask structured questions instead of using a chat:
  • expected delivery date or cancellation date
  • whether the customer has contacted the merchant
  • optional uploads (cancellation email, receipt, screenshots)

These questions should capture everything an agent needs for a chargeback. - R13. Show the merchant's website and suggest contacting them, since merchants refunded in about a third of cases. This is optional and must not delay submission.

4.5 Acknowledgement, tracking and deadlines

  • R14. Submitting a dispute triggers an in-app and email acknowledgement immediately, with a case reference. This meets the 2-business-day rule automatically.
  • R15. The customer gets a dispute status screen showing:
  • the reason they gave
  • current stage
  • expected resolution date
  • any temporary credit
  • R16. The system counts business days from submission. At business day 10, any open case gets flagged in the agent queue. By business day 14, the agent must choose a delay reason, for example "awaiting merchant response to chargeback". The customer is then notified with that reason in plain language. If no reason is chosen, the case escalates to the team lead.
  • R17. Every notification is logged against the case for audit.

4.6 Support operations

  • R18. In-app cases enter the existing dispute queue already populated with:
  • the reason
  • the answers to the structured questions
  • attachments
  • which explanation screens the customer saw
  • R19. Chat stays available. Agents can start the same structured flow on a customer's behalf, so every case looks the same in the queue.

5. Merchant data feed: cost needs resolving before build

The processor's enriched feed covers 94% of transactions at $0.0008 each. On all ~9M monthly transactions that is $7,200/month, or $21,600/quarter.

The feed is very unlikely to pay for itself through dispute savings alone. Using the sample rates, about 1,014 disputes a quarter were "recognised once shown details". At 11 minutes each, that is around 186 agent hours. To save $21,600, an agent hour would have to cost over $115, and that assumes every one of those disputes is prevented.

The feed may still be worth it. Clearer names on every transaction could reduce "what is this?" chats that never became disputes, and that number isn't in this data. But the full-feed price needs a business case of its own.

Action: Engineering and the PM will ask the processor whether enrichment can be bought per lookup. We would only call it when a customer opens a transaction or starts a dispute. At that volume the cost would be trivial. If per-lookup isn't available, I'll bring the full-feed decision to Imogen with Support's numbers on unrecognised-payment chats.

6. Success metrics

MetricBaselineTarget (one quarter after launch)
Disputes raised via app (vs chat)0%≥70%
"Don't recognise" disputes ending as "customer recognised it"73% of that reason<25% (resolved earlier by R4/R5 instead)
Chargeback cases past 15 business days with no delay noticeUnknown0
Disputes resolved after the deadline14%<5%
Agent minutes per dispute11≤5

Guardrails. These protect customers from being deflected when they shouldn't be: - Fraud reported late. Track fraud reported within 60 days on transactions where the customer tapped "I recognise it now". Any increase triggers a design review. - Abandoned flows. Track the rate of dispute flows started but not submitted, broken down by reason. - Tap counts. Keep tap-count parity with the payment flow (R3) and recheck it on every release.

7. Open questions

  1. Units of `daystoresolve`. Are these business or calendar days? The analysis above assumes business days. If they are calendar days, fewer cases breach the deadline.
  2. Gap between 11% and 14%. The sample shows 11% of cases over 15 days, while the background doc says 14%. This could be sampling noise. It could also mean cases are being counted late when a delay notice was sent, or not sent. Support Ops, please confirm how the 14% is calculated.
  3. Refunds on "don't recognise" fraud cases. These are recorded as "Fraud confirmed", while direct fraud claims are recorded as "Fraud confirmed, refunded". Compliance, please confirm all of those customers were refunded.
  4. Feed pricing. Is per-lookup pricing for the merchant feed available (section 5)?
  5. Temporary credit policy. Should we credit the customer while a chargeback is pending, and when? Compliance and Finance to decide before design finalises R15.

Claims the judge couldn’t find in the brief

  • 42% of disputes aren't really disputes.

Grades and run details

Decision model 79 · LLM judge 46
Decision model checks
  • passUses the supplied evidence correctly19%
  • passAddresses the actual decision36%
  • partialRespects explicit constraints58%
  • passIdentifies material uncertainty98%
  • partialAvoids unsupported claims36%
  • passProduces the required deliverable28%
  • passFinds that most disputes are recognition86%
  • passBuilds the regulator's rules in95%
  • passSuccess settles trade-offs99%
  • partialProposes tests that could fail77%
  • partialSays who it's for, and who it isn't36%
  • partialPlans how people will find it77%
Run
Run
#1
Time to output
1.9 min
Submitted
6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 6

Uses the supplied evidence correctlyRightRightMixed
GPT-6 Luna · API

All factual statements about the current situation are taken from the brief or context, or follow directly from arithmetic.

GPT-6 Astra · ChatGPT

Every statement about the current situation is directly from the supplied context or follows by arithmetic.

Opus 5.5 · Claude

The output claims 42% of disputes aren't really disputes, but the sample data yields 48%; this fact about the current situation is incorrect and unsupported.

Avoids unsupported claimsRightRightWrong
GPT-6 Luna · API

Interpretations and forecasts are presented as proposals or measurements to be taken, not as established facts.

GPT-6 Astra · ChatGPT

Labels extrapolations as an 'optimistic ceiling, not a savings forecast' and distinguishes sample-based findings from forward-looking assumptions.

Opus 5.5 · Claude

The '42% of disputes aren't really disputes' claim is presented as a confident finding but is not supported by the data (actual is 48%).

Finds that most disputes are recognitionWrongRightRight
GPT-6 Luna · API

The PRD does not compute the exact share (≈58%) or the number resolved by showing merchant details, only describes it as 'many'.

GPT-6 Astra · ChatGPT

Computes that 58% of disputes are unrecognised payments and that most resolve by showing merchant details, then makes transaction recognition the first-scope item.

Opus 5.5 · Claude

The PRD computes the 58% share and 73% resolved by showing merchant details, and makes recognisable transactions the first scope item (R4/R5).

Success settles trade-offsMixedRightMixed
GPT-6 Luna · API

Success metrics with targets are set, but no explicit trade‑off rule states which goal wins when two conflict.

GPT-6 Astra · ChatGPT

Sets metrics with targets (25% reduction in handling hours, 70% self-serve) and establishes that the regulatory rule against making disputing harder wins over other goals.

Opus 5.5 · Claude

Success metrics have targets and time frame, but no explicit trade-off rule is given (e.g., which goal wins when they conflict).

Says who it's for, and who it isn'tMixedWrongWrong
GPT-6 Luna · API

The PRD names target users (personal current‑account customers) but does not explicitly exclude a user group like business accounts; 'non‑card payment types' is a payment type exclusion, not a user group.

GPT-6 Astra · ChatGPT

Does not explicitly name a user group it's not for (e.g., disputes above a set amount) and no scope decision visibly follows from such an exclusion.

Opus 5.5 · Claude

The PRD does not name the target users (individuals) or explicitly exclude any user group (e.g., business accounts), and no scope decision visibly follows from such an exclusion.

Plans how people will find itRightWrongWrong
GPT-6 Luna · API

It gives a staged rollout (pilot then expand), a discovery method (button on transaction), and names support operations as needing readiness.

GPT-6 Astra · ChatGPT

Describes a staged rollout but never says how users will discover the feature (e.g., in-app notification, help entry) or who needs to be ready beyond support ops.

Opus 5.5 · Claude

No staged rollout, user discovery method, or support readiness plan is described beyond agents being able to start the flow.

All got wrong 1

Proposes tests that could failWrongWrongWrong
GPT-6 Luna · API

The pilot is described without a numeric kill threshold, measurement window, or the action each outcome would trigger.

GPT-6 Astra · ChatGPT

Pause triggers like 'misleading merchant matches' or 'increased fraud-reporting delay' lack numeric thresholds and defined measurement windows.

Opus 5.5 · Claude

Only one guardrail (fraud reported late) has a threshold and action; other metrics lack thresholds, measurement windows, and actions for each outcome.

All got right 5

Addresses the actual decisionRightRightRight
GPT-6 Luna · API

The output commits early to building self‑serve disputes with enriched merchant recognition, framed for the squad, and implies the pilot outcome will decide expansion.

GPT-6 Astra · ChatGPT

Commits to building a transaction-level journey with merchant detail view and self-serve disputes, deferring the full-feed purchase to a later business case.

Opus 5.5 · Claude

The PRD commits to building self-serve disputes with merchant recognition first, and says what would change the feed decision (per-lookup pricing).

Respects explicit constraintsRightRightRight
GPT-6 Luna · API

The PRD is within length, addresses the named squad, never makes disputing harder than paying, and respects all given constraints.

GPT-6 Astra · ChatGPT

Delivers a PRD within 1,000–1,400 words, addresses the named squad, and includes enforceable checks for the regulator's rule on dispute difficulty.

Opus 5.5 · Claude

The PRD is within 1,000-1,400 words, addresses the squad, and includes R3 to enforce that disputing is never harder than paying.

Identifies material uncertaintyRightRightRight
GPT-6 Luna · API

It lists open compliance, engineering, and product decisions, acknowledges the sample is not a causal baseline, and calls for a pilot to resolve key uncertainties.

GPT-6 Astra · ChatGPT

Names unknowns in selective merchant data availability, pricing, data quality, and the export's day-type clarity, and ties them to actions that would resolve them.

Opus 5.5 · Claude

Open questions name specific unknowns (days_to_resolve units, 11% vs 14% gap, feed pricing, temporary credit) and how to resolve them.

Produces the required deliverableRightRightRight
GPT-6 Luna · API

The output is a complete PRD addressed to the squad, within 1,000–1,400 words, and provides enough substance for the team to act on with light refinement.

GPT-6 Astra · ChatGPT

Provides a complete PRD with scope, requirements, operational needs, and acceptance criteria that the squad could build from.

Opus 5.5 · Claude

The PRD is complete, in the right form, within length, and the squad could act on it with the open questions resolved.

Builds the regulator's rules inRightRightRight
GPT-6 Luna · API

All three regulatory rules (acknowledge ≤2 days, resolve ≤15 days or explain, no harder than paying) appear as testable requirements.

GPT-6 Astra · ChatGPT

Includes 2-business-day acknowledgement, 15-business-day resolution/explanation, and 'no harder than paying' as testable requirements.

Opus 5.5 · Claude

All three rules appear as testable requirements: R14 (acknowledge within 2 days), R16 (resolve within 15 or explain), R3 (no harder than paying).

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 70% of 81 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstraChatGPT87.184.62None
2Sonnet 5.5API75.979.82None
3GPT-6.1 SolAPI82.660.92None
4Haiku 5.5API76.163.82None
5Opus 5.5Claude75.960.62None
6GPT-6 LunaAPI75.859.92None
7Gemini 3.8 FlashAPI49.828.22None
8Gemini 3.5 Flash-LiteGemini49.832.12None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 84, 106× the cheapest (1 of 2 outputs estimated)Sonnet 5.5 · API: task score 77, 35× the cheapestGPT-6.1 Sol · API: task score 71, 14× the cheapestHaiku 5.5 · API: task score 67, the cheapestOpus 5.5 · Claude: task score 65, 85× the cheapest (1 of 2 outputs estimated)GPT-6 Luna · API: task score 65, the cheapestGemini 3.8 Flash · API: task score 39, 9.9× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 39, 4.4× the cheapest (1 of 2 outputs estimated)GPT-6 Astra · ChatGPTSonnet 5.5 · APIGPT-6.1 Sol · APIHaiku 5.5 · APIOpus 5.5 · ClaudeGPT-6 Luna · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Writing the requirements document a team will build and test against.

Why it matters

A PRD is where ambiguity becomes either a decision or a bug. For AI products it must also say what happens when the model is uncertain or wrong. Most generated PRDs skip that part.

What good looks like

  • States the user problem and the decision the PRD enables
  • Specifies behaviour under uncertainty, failure and refusal
  • Names eval criteria and a launch bar
  • Defines success precisely enough to settle trade-offs
  • Says what is out of scope

Deliberately not measured

  • Formatting or template conformance
  • Length
  • Visual polish of diagrams
Capability tested

Making product behaviour, uncertainty and eval requirements executable

The failure we’re looking for

A generic feature spec that ignores AI failure behaviour

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

AI product PRD (core) · Conventional product PRD