Humanity’s Last PRD
Latest drop: Gemini 4 Argon · 13 Oct
Tasks / Product Craft

Write a PRD

Can the model turn a brief into a spec engineers could build from, including how an AI feature behaves when it is wrong?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 28% were usable with at most a quick edit.

Reliably right

  1. Specifies AI failure behaviour100% pass
    It specifies what happens when uncertain or wrong: block ownership, show retry states, disable assignments on violation, and never fall back to calendar data.
    GPT-6 Astra · ChatGPT · Meeting summaries with action items
  2. Builds the regulator's rules in100% pass
    All three rules (2-business-day acknowledge, 15-business-day resolve or explain, no harder than paying) appear as testable requirements R8, R10, R13.
    Sonnet 5.5 · API · Disputes that aren't disputes
  3. Produces the required deliverable96% pass
    The PRD is complete, written for engineering and design leads, within the 1,000–1,500 word range, and directly actionable.
    GPT-6 Astra · ChatGPT · Meeting summaries with action items

Where it slips

  1. Says who it's for, and who it isn't35% pass
    Does not name the target users (individual account holders) or explicitly exclude a group (e.g., business accounts), missing a required scope decision.
    Sonnet 5.5 · API · Disputes that aren't disputes
  2. Proposes tests that could fail36% pass
    The A/B expansion gate relies on a fraud-reversal guardrail but does not specify a numeric threshold, leaving the test incomplete.
    Sonnet 5.5 · API · Disputes that aren't disputes
  3. Plans how people will find it44% pass
    Gives a staged rollout and discovery method (reviewer notification), but does not name the teams that need to be ready (e.g., support, sales) beyond technical dependencies.
    GPT-6.1 Sol · API · Meeting summaries with action items

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Success settles trade-offs

    Does the PRD define success as a measurable outcome with a target and time frame, and say which goal wins when two of them conflict?

    Passes when Names a success metric with a target and time frame, and gives at least one explicit trade-off rule (for example, accepting lower coverage to keep precision above a stated level).

    Product and growth leadersOn Lenny’s Podcast

  2. Says who it's for, and who it isn't

    Does the PRD name the specific users it is for, say explicitly who it is not for, and make at least one scope decision that follows from that choice?

    Passes when Names the target users and at least one group it deliberately leaves out, and a scope decision visibly follows from the exclusion.

    Matt LeMayLenny’s Podcast

  3. Plans how people will find it

    Does the PRD say how the feature reaches people: how it rolls out, how users discover it, and who needs to be ready for it (support, sales), rather than stopping at what to build?

    Passes when Gives a staged rollout, a way users will discover the feature, and the teams that need to be ready, all fitted to this feature.

    Ami VoraLenny’s PodcastKarri SaarinenLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Proposes tests that could fail · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for card servicing at Clearbank. Write the PRD for self-serve disputes in the app, as Imogen asks below. It goes to the squad (engineering, design, support operations and compliance) who'll build it next quarter. 1,000 to 1,400 words. The dispute export and the background are attached.

What the model was given6 items: About Clearbank, imogen_note.md, how_disputes_work_today.md, regulator_rules.md, disputes_sample_last_quarter.csv (a random sample of 400 of last quarter's 2,400 disputes), merchant_data_option.md
About ClearbankA digital bank for individuals: current accounts and debit cards, run through a mobile app.
imogen_note.md1 lines · Download
From Imogen (Head of Product): "Write the PRD for self-serve disputes in the app. Support says it's their biggest cost after card freezes. The PRD goes to the squad (engineering, design, support ops and compliance) who will build it next quarter. 1,000 to 1,400 words."
how_disputes_work_today.md5 lines · Download
# How disputes work today

Customers raise a dispute by chatting to support. An agent collects details, then either explains the payment or opens a chargeback. Agents spend about 11 minutes per dispute. 2,400 disputes last quarter; 14% of them were resolved later than the regulator's deadline.

Card transactions show the merchant's billing name (for example "SQ *BRKLYN BAGEL 0412"), not its trading name or logo.
regulator_rules.md5 lines · Download
# Regulator's rules (summary from Legal)

- Acknowledge a dispute within 2 business days.
- Resolve genuine disputes within 15 business days, or tell the customer why it's taking longer.
- Don't make it harder to raise a dispute than to make a payment.
disputes_sample_last_quarter.csv (a random sample of 400 of last quarter's 2,400 disputes)dispute,customer_said,days_to_resolve,outcome CB-40000,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40006,I didn't receive what I paid for,17,Chargeback lost CB-40012,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40018,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40024,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40030,I didn't receive what I paid for,11,Chargeback lost CB-40036,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40042,I didn't receive what I paid for,16,Chargeback lost CB-40048,I was charged twice,5,Not a duplicate: two separate purchases CB-40054,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40060,I was charged twice,10,Not a duplicate: two separate purchases CB-40066,I was charged twice,7,Not a duplicate: two separate purchases CB-40072,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40078,I don't recognise this payment,5,Fraud confirmed CB-40084,I don't recognise this payment,7,Fraud confirmed CB-40090,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40096,I was charged twice,6,Not a duplicate: two separate purchases CB-40102,I didn't receive what I paid for,7,Chargeback won CB-40108,I don't recognise this payment,6,Fraud confirmed CB-40114,I was charged twice,9,Duplicate refunded CB-40120,I was charged twice,7,Not a duplicate: two separate purchases CB-40126,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40132,I was charged twice,9,Duplicate refunded CB-40138,Subscription I cancelled,18,Chargeback won CB-40144,I didn't receive what I paid for,21,Chargeback won CB-40150,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40156,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40162,I don't recognise this payment,6,Fraud confirmed CB-40168,I was charged twice,4,Not a duplicate: two separate purchases CB-40174,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40180,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-40186,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40192,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-40198,I don't recognise this payment,1,Fraud confirmed CB-40204,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40210,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40216,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40222,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40228,I didn't receive what I paid for,29,Chargeback won CB-40234,I don't recognise this payment,8,Fraud confirmed CB-40240,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40246,I didn't receive what I paid for,10,Merchant refunded CB-40252,I don't recognise this payment,4,Fraud confirmed CB-40258,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40264,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40270,I didn't receive what I paid for,21,Chargeback won CB-40276,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40282,I didn't receive what I paid for,17,Merchant refunded CB-40288,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40294,I don't recognise this payment,1,Fraud confirmed CB-40300,I don't recognise this payment,7,Fraud confirmed CB-40306,I don't recognise this payment,8,Fraud confirmed CB-40312,I didn't receive what I paid for,20,Merchant refunded CB-40318,I didn't receive what I paid for,26,Chargeback lost CB-40324,Subscription I cancelled,10,Merchant refunded CB-40330,I don't recognise this payment,1,Fraud confirmed CB-40336,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-40342,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40348,Fraud: I didn't make this payment,9,"Fraud confirmed, refunded" CB-40354,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40360,I was charged twice,9,Not a duplicate: two separate purchases CB-40366,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40372,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40378,I don't recognise this payment,8,Fraud confirmed CB-40384,I didn't receive what I paid for,14,Merchant refunded CB-40390,I don't recognise this payment,7,Fraud confirmed CB-40396,I don't recognise this payment,8,Fraud confirmed CB-40402,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40408,I was charged twice,5,Not a duplicate: two separate purchases CB-40414,I didn't receive what I paid for,13,Chargeback lost CB-40420,Subscription I cancelled,11,Chargeback won CB-40426,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-40432,I didn't receive what I paid for,13,Chargeback won CB-40438,Fraud: I didn't make this payment,4,"Fraud confirmed, refunded" CB-40444,I was charged twice,9,Duplicate refunded CB-40450,I didn't receive what I paid for,18,Chargeback won CB-40456,I don't recognise this payment,1,Fraud confirmed CB-40462,I don't recognise this payment,4,Fraud confirmed CB-40468,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40474,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40480,Fraud: I didn't make this payment,8,"Fraud confirmed, refunded" CB-40486,I didn't receive what I paid for,27,Chargeback lost CB-40492,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40498,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40504,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40510,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40516,I don't recognise this payment,5,Fraud confirmed CB-40522,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40528,Fraud: I didn't make this payment,8,"Fraud confirmed, refunded" CB-40534,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40540,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40546,I don't recognise this payment,2,Fraud confirmed CB-40552,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40558,I was charged twice,5,Duplicate refunded CB-40564,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40570,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40576,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40582,I didn't receive what I paid for,14,Chargeback lost CB-40588,I didn't receive what I paid for,18,Chargeback lost CB-40594,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40600,Fraud: I didn't make this payment,2,"Fraud confirmed, refunded" CB-40606,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40612,I don't recognise this payment,3,Fraud confirmed CB-40618,Fraud: I didn't make this payment,5,"Fraud confirmed, refunded" CB-40624,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40630,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40636,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40642,Fraud: I didn't make this payment,11,"Fraud confirmed, refunded" CB-40648,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40654,I don't recognise this payment,5,Fraud confirmed CB-40660,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40666,I don't recognise this payment,6,Fraud confirmed CB-40672,I was charged twice,5,Not a duplicate: two separate purchases CB-40678,Subscription I cancelled,10,Chargeback won CB-40684,Fraud: I didn't make this payment,9,"Fraud confirmed, refunded" CB-40690,Subscription I cancelled,7,Chargeback won CB-40696,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40702,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40708,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40714,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40720,I was charged twice,13,Not a duplicate: two separate purchases CB-40726,Subscription I cancelled,13,Chargeback won CB-40732,I don't recognise this payment,3,Fraud confirmed CB-40738,I didn't receive what I paid for,18,Chargeback lost CB-40744,Fraud: I didn't make this payment,8,"Fraud confirmed, refunded" CB-40750,I was charged twice,11,Not a duplicate: two separate purchases CB-40756,Subscription I cancelled,10,Merchant refunded CB-40762,I was charged twice,7,Duplicate refunded CB-40768,Subscription I cancelled,18,Chargeback won CB-40774,I didn't receive what I paid for,23,Chargeback won CB-40780,I didn't receive what I paid for,7,Chargeback won CB-40786,Subscription I cancelled,5,Chargeback won CB-40792,I didn't receive what I paid for,10,Merchant refunded CB-40798,I was charged twice,9,Duplicate refunded CB-40804,I didn't receive what I paid for,24,Merchant refunded CB-40810,I don't recognise this payment,2,Fraud confirmed CB-40816,Fraud: I didn't make this payment,11,"Fraud confirmed, refunded" CB-40822,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40828,Subscription I cancelled,13,Chargeback won CB-40834,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40840,I don't recognise this payment,7,Fraud confirmed CB-40846,Subscription I cancelled,7,Merchant refunded CB-40852,I don't recognise this payment,3,Fraud confirmed CB-40858,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40864,I was charged twice,4,Not a duplicate: two separate purchases CB-40870,I don't recognise this payment,8,Fraud confirmed CB-40876,I was charged twice,6,Not a duplicate: two separate purchases CB-40882,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-40888,I don't recognise this payment,8,Fraud confirmed CB-40894,I don't recognise this payment,6,Fraud confirmed CB-40900,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40906,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40912,I don't recognise this payment,8,Fraud confirmed CB-40918,I didn't receive what I paid for,21,Chargeback won CB-40924,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-40930,Fraud: I didn't make this payment,8,"Fraud confirmed, refunded" CB-40936,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40942,I was charged twice,6,Duplicate refunded CB-40948,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40954,I didn't receive what I paid for,14,Chargeback won CB-40960,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40966,I was charged twice,4,Not a duplicate: two separate purchases CB-40972,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40978,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40984,I didn't receive what I paid for,9,Merchant refunded CB-40990,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40996,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41002,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41008,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41014,Fraud: I didn't make this payment,2,"Fraud confirmed, refunded" CB-41020,I didn't receive what I paid for,27,Chargeback won CB-41026,I didn't receive what I paid for,28,Chargeback won CB-41032,I was charged twice,7,Duplicate refunded CB-41038,I don't recognise this payment,2,Fraud confirmed CB-41044,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41050,I don't recognise this payment,4,Fraud confirmed CB-41056,Fraud: I didn't make this payment,11,"Fraud confirmed, refunded" CB-41062,I was charged twice,3,Duplicate refunded CB-41068,I don't recognise this payment,1,Fraud confirmed CB-41074,I don't recognise this payment,1,Fraud confirmed CB-41080,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41086,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41092,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41098,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41104,I didn't receive what I paid for,29,Merchant refunded CB-41110,Subscription I cancelled,16,Chargeback won CB-41116,I didn't receive what I paid for,22,Chargeback won CB-41122,I didn't receive what I paid for,13,Chargeback lost CB-41128,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41134,I was charged twice,10,Not a duplicate: two separate purchases CB-41140,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41146,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41152,I was charged twice,5,Duplicate refunded CB-41158,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41164,I didn't receive what I paid for,12,Chargeback lost CB-41170,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41176,Fraud: I didn't make this payment,2,"Fraud confirmed, refunded" CB-41182,I don't recognise this payment,1,Fraud confirmed CB-41188,I don't recognise this payment,3,Fraud confirmed CB-41194,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41200,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41206,Fraud: I didn't make this payment,5,"Fraud confirmed, refunded" CB-41212,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41218,Fraud: I didn't make this payment,6,"Fraud confirmed, refunded" CB-41224,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41230,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41236,I don't recognise this payment,3,Fraud confirmed CB-41242,Subscription I cancelled,9,Merchant refunded CB-41248,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41254,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41260,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41266,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41272,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41278,I didn't receive what I paid for,11,Chargeback lost CB-41284,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41290,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41296,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-41302,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41308,I was charged twice,11,Duplicate refunded CB-41314,I didn't receive what I paid for,27,Chargeback won CB-41320,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41326,Fraud: I didn't make this payment,5,"Fraud confirmed, refunded" CB-41332,I was charged twice,12,Not a duplicate: two separate purchases CB-41338,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-41344,I didn't receive what I paid for,17,Merchant refunded CB-41350,I didn't receive what I paid for,21,Chargeback won CB-41356,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41362,Subscription I cancelled,14,Merchant refunded CB-41368,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41374,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41380,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41386,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41392,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41398,I don't recognise this payment,2,Fraud confirmed CB-41404,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41410,I don't recognise this payment,3,Fraud confirmed CB-41416,I don't recognise this payment,5,Fraud confirmed CB-41422,I was charged twice,12,Duplicate refunded CB-41428,Subscription I cancelled,9,Merchant refunded CB-41434,I was charged twice,6,Duplicate refunded CB-41440,Subscription I cancelled,15,Merchant refunded CB-41446,I don't recognise this payment,5,Fraud confirmed CB-41452,I don't recognise this payment,7,Fraud confirmed CB-41458,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41464,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41470,I was charged twice,6,Not a duplicate: two separate purchases CB-41476,I don't recognise this payment,5,Fraud confirmed CB-41482,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41488,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41494,I didn't receive what I paid for,26,Merchant refunded CB-41500,I didn't receive what I paid for,24,Chargeback won CB-41506,Subscription I cancelled,20,Chargeback won CB-41512,I didn't receive what I paid for,29,Merchant refunded CB-41518,Subscription I cancelled,20,Chargeback won CB-41524,I don't recognise this payment,8,Fraud confirmed CB-41530,I was charged twice,3,Not a duplicate: two separate purchases CB-41536,I didn't receive what I paid for,8,Chargeback won CB-41542,I don't recognise this payment,2,Fraud confirmed CB-41548,I don't recognise this payment,2,Fraud confirmed CB-41554,I was charged twice,12,Not a duplicate: two separate purchases CB-41560,Subscription I cancelled,6,Chargeback won CB-41566,I was charged twice,3,Not a duplicate: two separate purchases CB-41572,I don't recognise this payment,1,Fraud confirmed CB-41578,Fraud: I didn't make this payment,7,"Fraud confirmed, refunded" CB-41584,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41590,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-41596,Subscription I cancelled,16,Merchant refunded CB-41602,Subscription I cancelled,15,Merchant refunded CB-41608,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41614,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41620,I was charged twice,5,Duplicate refunded CB-41626,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41632,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41638,Fraud: I didn't make this payment,3,"Fraud confirmed, refunded" CB-41644,I don't recognise this payment,3,Fraud confirmed CB-41650,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41656,I don't recognise this payment,6,Fraud confirmed CB-41662,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41668,I didn't receive what I paid for,12,Chargeback won CB-41674,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41680,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41686,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-41692,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41698,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41704,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41710,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41716,Fraud: I didn't make this payment,8,"Fraud confirmed, refunded" CB-41722,I don't recognise this payment,3,Fraud confirmed CB-41728,I was charged twice,12,Duplicate refunded CB-41734,Fraud: I didn't make this payment,5,"Fraud confirmed, refunded" CB-41740,I don't recognise this payment,6,Fraud confirmed CB-41746,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41752,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41758,I was charged twice,13,Duplicate refunded CB-41764,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41770,I didn't receive what I paid for,23,Chargeback won CB-41776,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41782,I didn't receive what I paid for,27,Chargeback won CB-41788,I didn't receive what I paid for,23,Chargeback lost CB-41794,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41800,I don't recognise this payment,3,Fraud confirmed CB-41806,I don't recognise this payment,1,Fraud confirmed CB-41812,I was charged twice,4,Not a duplicate: two separate purchases CB-41818,I don't recognise this payment,8,Fraud confirmed CB-41824,I didn't receive what I paid for,18,Chargeback won CB-41830,Subscription I cancelled,13,Chargeback won CB-41836,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41842,I don't recognise this payment,7,Fraud confirmed CB-41848,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41854,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41860,I was charged twice,5,Duplicate refunded CB-41866,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41872,I don't recognise this payment,8,Fraud confirmed CB-41878,I don't recognise this payment,2,Fraud confirmed CB-41884,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41890,I was charged twice,7,Not a duplicate: two separate purchases CB-41896,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41902,Fraud: I didn't make this payment,9,"Fraud confirmed, refunded" CB-41908,I don't recognise this payment,4,Fraud confirmed CB-41914,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41920,I didn't receive what I paid for,24,Merchant refunded CB-41926,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41932,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-41938,Subscription I cancelled,15,Chargeback won CB-41944,I didn't receive what I paid for,13,Chargeback won CB-41950,I don't recognise this payment,2,Fraud confirmed CB-41956,I didn't receive what I paid for,24,Chargeback lost CB-41962,I don't recognise this payment,4,Fraud confirmed CB-41968,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41974,Subscription I cancelled,15,Chargeback won CB-41980,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41986,Fraud: I didn't make this payment,2,"Fraud confirmed, refunded" CB-41992,Fraud: I didn't make this payment,6,"Fraud confirmed, refunded" CB-41998,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-42004,I didn't receive what I paid for,12,Merchant refunded CB-42010,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42016,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42022,I was charged twice,11,Duplicate refunded CB-42028,I didn't receive what I paid for,25,Merchant refunded CB-42034,I didn't receive what I paid for,10,Merchant refunded CB-42040,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42046,I was charged twice,8,Duplicate refunded CB-42052,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-42058,I don't recognise this payment,7,Fraud confirmed CB-42064,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-42070,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42076,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-42082,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-42088,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-42094,Subscription I cancelled,12,Chargeback won CB-42100,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-42106,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-42112,Subscription I cancelled,18,Chargeback won CB-42118,Fraud: I didn't make this payment,6,"Fraud confirmed, refunded" CB-42124,I don't recognise this payment,5,Fraud confirmed CB-42130,I didn't receive what I paid for,14,Chargeback lost CB-42136,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-42142,Fraud: I didn't make this payment,2,"Fraud confirmed, refunded" CB-42148,I was charged twice,12,Not a duplicate: two separate purchases CB-42154,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-42160,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-42166,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42172,Fraud: I didn't make this payment,9,"Fraud confirmed, refunded" CB-42178,I don't recognise this payment,3,Fraud confirmed CB-42184,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-42190,I was charged twice,7,Duplicate refunded CB-42196,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-42202,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-42208,I didn't receive what I paid for,21,Chargeback won CB-42214,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-42220,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-42226,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-42232,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-42238,I was charged twice,10,Duplicate refunded CB-42244,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-42250,Subscription I cancelled,11,Chargeback won CB-42256,Fraud: I didn't make this payment,3,"Fraud confirmed, refunded" CB-42262,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-42268,I didn't receive what I paid for,16,Merchant refunded CB-42274,I was charged twice,3,Duplicate refunded CB-42280,Fraud: I didn't make this payment,9,"Fraud confirmed, refunded" CB-42286,I was charged twice,11,Duplicate refunded CB-42292,I was charged twice,9,Not a duplicate: two separate purchases CB-42298,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-42304,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-42310,Subscription I cancelled,16,Merchant refunded CB-42316,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-42322,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-42328,Fraud: I didn't make this payment,3,"Fraud confirmed, refunded" CB-42334,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-42340,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-42346,I didn't receive what I paid for,19,Merchant refunded CB-42352,I was charged twice,12,Duplicate refunded CB-42358,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-42364,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-42370,I don't recognise this payment,3,Fraud confirmed CB-42376,I didn't receive what I paid for,17,Chargeback won CB-42382,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-42388,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42394,I don't recognise this payment,5,Customer recognised it once shown the merchant's details
merchant_data_option.md3 lines · Download
# Merchant data we could use

Our card processor offers an enriched merchant feed (trading name, logo, location, website) for 94% of transactions, at $0.0008 per transaction. We process about 9 million card transactions a month.
What a strong answer doesThe answer key the graders mark against

Starts from the data: in the sample, 232 of 400 disputes (58%, so about 1,400 of the quarter's 2,400) were 'I don't recognise this payment', and 169 of those (about three in four) were resolved simply by showing the merchant's details; so the first thing to build is recognisable transactions (trading name, logo, location from the enriched feed, about $7,200 a month for 9 million transactions), shown before someone raises a dispute, with self-serve disputes for the genuine cases. Defines success with targets (disputes per 1,000 transactions, share resolved without an agent, regulator deadlines met against 14% late today) and says which wins when they conflict (never make disputing harder than paying, per the regulator). Covers the flows for genuine disputes (charged twice, not received, cancelled subscription, fraud) with acknowledgement within 2 business days and resolution within 15, what happens when the merchant data is missing (6% of transactions), and how the rollout reaches people, with support ready. Says who it's not for (for example business accounts, or disputes over a set amount that still go to an agent) and what's out of scope.

Critical failures (cap the score at 40)
  • Adds steps that make raising a dispute harder than making a payment
Task

Staff-level · v1.1

The outputs

9 setups answered this brief.

Sonnet 5.5API87Usable with a quick editGPT-6 AstraChatGPT75Needs more workHaiku 5.5API67Needs more workGPT-6 LunaAPI65Needs more workOpus 5.5Claude60Needs more workGPT-6.1 SolAPI58Needs more workGemini 4 ArgonAPI40Critical failureGemini 3.5 Flash-LiteGemini37Needs more workGemini 3.8 FlashAPI31Needs more work
Gemini 4 ArgonAPI
Critical failure: Adds steps that make raising a dispute harder than making a payment. Score capped at 40.

Needs more than a quick edit5 of 12 checks passed

The sample data is miscounted leading to wrong baseline numbers, and missing elements (trade‑offs, rollout, user exclusion) would require rework.

Output

Product Requirements Document: Self-Serve Card Disputes

Author: PM, Card Servicing Target Release: Q2 Audience: Card Servicing Squad (Engineering, Design, Support Operations, Compliance) Status: Ready for Kickoff

---

1. Context & Problem Statement

Card disputes are currently Clearbank’s second-highest support cost driver after card freezes. Last quarter, customers raised 2,400 disputes via support chat. At an average handling time (AHT) of 11 minutes per ticket, disputes consume ~440 hours of agent time per quarter. Furthermore, manual triage bottlenecks caused us to miss the regulator’s 15-business-day resolution deadline on 14% of disputes last quarter, exposing Clearbank to compliance risk.

Today, raising a dispute requires starting a free-text support chat. An agent manually gathers transaction details, looks up merchant information, and either explains the payment to the customer or initiates a chargeback.

An analysis of a 400-dispute random sample from last quarter reveals that 49.5% of all disputes are false alarms (customers who recognize the payment once shown clear merchant details, or customers mistaking two separate purchases for a duplicate charge). Because our app only displays raw billing descriptors (e.g., `SQ *BRKLYN BAGEL 0412`), customers panic and message support.

Customer Stated Reason% of Volume (Sample)Breakdown of OutcomesAvg. Days to Resolve
:---:---:---:---
"I don't recognise this payment"61.75% (247)• 72% (44.5% of total): Recognised once shown merchant details<br>• 28% (17.25% of total): Fraud confirmed (no refund noted in log!)4.6 days
"I didn't receive what I paid for"13.25% (53)• 38%: Chargeback won<br>• 32%: Merchant refunded<br>• 30%: Chargeback lost18.8 days (68% > 15 days)
"I was charged twice"10.75% (43)• 53.5%: Duplicate refunded<br>• 46.5% (5% of total): Not a duplicate (two separate purchases)7.6 days
"Fraud: I didn't make this payment"8.25% (33)• 100%: Fraud confirmed, refunded6.8 days
"Subscription I cancelled"6.00% (24)• 62.5%: Chargeback won<br>• 37.5%: Merchant refunded13.4 days (33% > 15 days)

Critical Findings:

  1. Unnecessary Contact: Nearly half of all disputes can be deflected in-app by showing enriched merchant data and transaction comparisons before a ticket is ever created.
  2. SLA Breaches: 100% of our regulatory SLA breaches (>15 business days) occur in "Goods/Services not received" and "Cancelled subscriptions." Chargebacks take time, and we lack an automated mechanism to update customers at day 15 as required by the regulator.
  3. Compliance Gap on Unrecognised Fraud: In our sample, 100% of customers who explicitly claimed "Fraud" were refunded. However, for the 17.25% who said "I don't recognise this payment" where the outcome was Fraud confirmed, the outcome log shows `"Fraud confirmed"` rather than `"Fraud confirmed, refunded"`. Support Ops and Compliance must audit whether these 69 cases (~414 extrapolated quarterly) are failing to trigger automated refunds due to how agents categorise the initial ticket.

---

2. Goals & Success Metrics

Primary Goal: Reduce agent time spent on disputes while achieving 100% regulatory compliance.

  • Deflection Rate: Reduce total inbound dispute tickets by 40%+ by helping customers self-diagnose unrecognised and duplicate payments in-app.
  • Agent Efficiency: Reduce average agent handling time (AHT) on remaining disputes from 11 minutes to <4 minutes by collecting structured evidence upfront.
  • SLA Compliance: Reduce uncommunicated >15 business day resolutions from 14% to 0%. Achieve 100% automated acknowledgment within 2 business days (instant upon submission).
  • Regulatory Parity: Ensure raising a dispute takes no more taps/effort than making a payment (≤4 steps from the transaction screen).

---

3. Strategic Architecture Decision: On-Demand Merchant Enrichment

Our card processor offers an enriched merchant feed (trading name, logo, location, website) covering 94% of transactions at $0.0008 per transaction.

  • Option A (Enrich All): Enriching all 9M monthly transactions costs $7,200/month ($86,400/year). While nice for general UX, we cannot justify an $86k annual spend purely on support deflection (440 hours of agent time per quarter costs ~$9,000–$11,000/quarter, or ~$40k/year).
  • Option B (On-Demand / Selective Enrichment — Chosen): We will query the processor’s enrichment API only when a customer taps into a transaction detail screen or initiates the dispute flow (or cache enriched data by Merchant ID across our user base). Even if customers tap into 100,000 transactions a month, on-demand calls will cost <$80/month while still deflecting the 44.5% of disputes caused by cryptic billing descriptors. (Note for Eng: If the processor requires a bulk feed, we will negotiate a lookup API or build an internal merchant-mapping cache).

---

4. User Experience & Functional Requirements

Entry Point (Regulatory Requirement: Frictionless Access)

  • Customers tap any settled transaction in their feed and select "Something wrong? Get help" at the bottom of the screen.
  • Pending Transactions: If a transaction is still pending, explain that merchants cannot finalize or refund pending authorisations, and ask the customer to check back once it settles (unless they are reporting a lost/stolen card, which routes immediately to card freeze + fraud).

Step 1: The Enriched Context Interstitial (Deflection Layer)

Before asking why they are disputing, display an enriched breakdown of the transaction: * Trading Name & Logo (e.g., "Brooklyn Bagel" instead of `SQ BRKLYN BAGEL`). Map pin/location of the store (if physical) or website URL (if online). * Exact timestamp and authorisation method (e.g., Apple Pay, Chip & PIN, Contactless, Online). * Prompt: "Does this look familiar now?" -> `[Yes, all good]` (closes flow, logs deflection) / `[No, I still need help]`.

Step 2: Issue Categorisation & Smart Routing

If the customer continues, present four clear options:

#### Path A: "I don't recognise this payment / I didn't make it" (70% of volume) * Consolidation: Merges "Unrecognised" and "Fraud" into one path to eliminate the operational discrepancy where unrecognised payments confirmed as fraud missed the standard refund workflow. * Checklist: Ask customer to confirm they checked with family/household members and didn't forget a free trial. * Security Action: Because confirmed unrecognised payments are fraud, require the customer to freeze and replace their card automatically upon submission to prevent further compromise. * Routing: Routes directly to the Fraud Queue with device/IP telemetry and auth method (e.g., 3DS status) attached.

#### Path B: "I was charged twice" (10.75% of volume) * Smart Comparison: Automatically scan the customer’s transactions from the same merchant within ±7 days. * Deflection UI: Display the transactions side-by-side showing timestamps, exact amounts, and authorisation codes. * If different timestamps/items: Explain that these appear to be two separate purchases (e.g., buying two coffees 20 minutes apart). * If identical timestamp/amount: Pre-select the duplicate and allow 1-tap submission for a duplicate chargeback.

#### Path C: "I didn't receive what I paid for" (13.25% of volume) * Self-Help Check: Since 32% of these end in a merchant refund, ask: "Have you contacted [Merchant Name] yet? Merchants usually resolve delivery issues faster than a bank chargeback." Provide a clickable link to the merchant's website/contact page from the enrichment data. * Evidence Collection: If they have contacted the merchant, collect required chargeback evidence upfront: * Expected delivery date. * Description of goods/services. * Upload screenshot of merchant communication or receipt (optional but encouraged; reduces our 30% chargeback loss rate).

#### Path D: "I was charged for a cancelled subscription" (6% of volume) * Evidence Collection: Ask for date of cancellation and upload of cancellation confirmation email/screenshot. * Action: Offer a toggle to block future recurring payments from this Merchant ID.

---

5. Automated Compliance, SLAs & Tracking

To satisfy Legal and regulatory requirements without manual agent intervention:

1. Instant Acknowledgment (<2 Business Days Rule): * Submitting the flow immediately generates a case ID, sends an in-app notification/email confirming receipt, and adds a "Dispute in progress" status tracker directly onto the transaction detail screen. 2. Automated 15-Day Delay Notification (15 Business Days Rule): * Build a backend cron job tracking open disputes. At Business Day 13, if a dispute (typically Paths C and D) is still in `Awaiting Merchant / Scheme Response`, automatically notify the customer via push and email: "We're still working on your dispute for [Merchant]. Because card scheme chargeback rules give the merchant time to respond, this is taking longer than 15 business days. Your new estimated resolution date is [Date]." * Log this notification in the audit trail to guarantee 0% non-compliant SLA breaches.

---

6. Squad Responsibilities & Next Steps

  • Engineering: Investigate the processor’s merchant enrichment integration (API lookup vs. feed) to ensure we only pay per-lookup. Build the transaction-matching logic for duplicate detection and the Day-13 SLA notification cron job.
  • Design: Design the 4-step self-serve flow, ensuring tap-count parity with our payment flow. Design the in-app dispute status tracker on the transaction screen.
  • Support Operations: Map the structured JSON output from Paths A–D directly into our ticketing system fields so agents don't have to copy-paste. Urgent Action: Audit the 69 sample cases of `"I don't recognise this payment" -> "Fraud confirmed"` from Q1 to verify whether customers received their refunds, and unify the backend fraud resolution macro.
  • Compliance: Review and sign off on the automated Day-13 delay notification copy and verify that the step-count meets the "no harder than making a payment" mandate.

Claims the judge couldn’t find in the brief

  • In the 400-dispute sample, 247 (61.75%) were 'I don't recognise this payment' and 72% of those recognised it once shown merchant details.
  • We lack an automated mechanism to update customers at day 15 as required by the regulator.
  • Merchants cannot finalize or refund pending authorisations.
  • 49.5% of all disputes are false alarms (never needed an agent).

Grades and run details

Decision model 63 · LLM judge 38
Decision model checks
  • failUses the supplied evidence correctly70%
  • passAddresses the actual decision31%
  • partialRespects explicit constraints43%
  • partialIdentifies material uncertainty34%
  • partialAvoids unsupported claims28%
  • partialProduces the required deliverable48%
  • passFinds that most disputes are recognition10%
  • passBuilds the regulator's rules in95%
  • passSuccess settles trade-offs59%
  • partialProposes tests that could fail70%
  • partialSays who it's for, and who it isn't34%
  • partialPlans how people will find it85%
Run
Run
#1
API response time
33 s
Submitted
13 Oct 2026
Sonnet 5.5API

Usable with a quick edit10 of 12 checks passed

The PRD is comprehensive and actionable. Adding an explicit user exclusion and a numeric guardrail threshold are minor edits a PM can make quickly.

Output

PRD: Self-serve disputes in the app

Owner: PM, Card Servicing · Audience: Engineering, Design, Support Ops, Compliance · Build: next quarter

1. Summary

Disputes are currently raised only through chat. Agents spend about 11 minutes on each of the 2,400 disputes a quarter (roughly 440 hours), and 14% were resolved after the regulator's deadline.

The sample shows that most disputes aren't really disputes. Customers can't recognise the merchant name on the statement. The late cases are a different problem. So this PRD covers two things: stop avoidable disputes at the source, and give genuine disputes a structured, timed path.

2. What the data says

I analysed the 400-dispute random sample (of 2,400). With n=400, shares are accurate to roughly ±5 points.

Customer saidShareOutcomes
I don't recognise this payment58% (232)169 recognised it once shown merchant details; 63 confirmed fraud
Fraud: I didn't make this9% (36)All confirmed and refunded
I didn't receive what I paid for14% (56)24 chargebacks won, 15 lost, 17 merchant refunded
I was charged twice12% (48)24 genuine duplicates, 24 two separate purchases
Subscription I cancelled7% (28)18 chargebacks won, 10 merchant refunded

Four findings shape the design:

  1. 42% of all disputes (169 of 400) end when the customer sees who the merchant is. Card transactions show billing names like "SQ *BRKLYN BAGEL 0412". This is the biggest lever, and it needs better information, not a dispute flow.
  2. About a quarter of disputes are fraud (63 + 36 = 99 of 400). A customer with a stolen card must not wait in a queue, so the flow has to offer a card freeze at once.
  3. Half of "charged twice" claims are two real purchases. Showing both transactions side by side resolves these.
  4. Every case over 15 days is a "not received" or "subscription" dispute (44 of 400, 11%; reported figure is 14%). None of the other types exceeded 13 days. Deflecting lookups will free agent time, but it will not fix lateness by itself. Late cases need structured evidence intake and deadline timers. Support Ops to confirm whether `daystoresolve` is business or calendar days.

3. Goals and metrics

Targets below are my proposals, to be agreed with Support Ops.

  • Reduce agent-handled disputes by 25% within 90 days of launch. That is about 600 a quarter, or ~110 agent-hours. The ceiling is roughly 40%: the 42% "recognised" share, less the ~6% of transactions with no enriched data.
  • Late resolutions from 14% to under 5%.
  • 100% of disputes acknowledged within 2 business days. In-app, this is instant.
  • Guardrail: every suspected-fraud case reaches the fraud queue the same day. Track "I recognise this" taps later reversed into a dispute on the same transaction. Fraud and Compliance to set the acceptable threshold.

Non-goals: changing chargeback decisioning, scheme rules, or provisional-credit policy. Merchant logos across the whole transaction feed are also out of scope.

4. Requirements

Entry and merchant details

  • R1. Card transaction detail shows trading name, logo, location and website where available. It always shows date, time, amount and the raw billing name.
  • R2. Where no enriched data exists (~6%), show the raw billing name with obvious prefixes and store codes stripped, plus the raw string.
  • R3. Every card transaction has a visible "Something wrong?" button, one tap to start. Reasons: I don't recognise this, Charged twice, Didn't receive it, Cancelled subscription, I didn't make this payment.

Paths by reason

  • R4. Don't recognise. Show the merchant details with two equal-weight buttons: "I remember it" and "I still don't recognise this". The second continues the dispute. It is never hidden or delayed. It then offers a one-tap card freeze (optional) and creates a case flagged suspected fraud, routed to the fraud queue.
  • R5. "I didn't make this payment". Skip the merchant step and go straight to the freeze offer and the fraud queue.
  • R6. Charged twice. Show the two transactions side by side. If same merchant, same amount, within a window Support Ops defines, treat as a likely duplicate and create the case. Otherwise say "these look like separate purchases", with a prominent "Dispute anyway" button.
  • R7. Not received / subscription. Collect structured evidence: order date, what was bought, whether the merchant was contacted, and for subscriptions the cancellation date and proof upload. Ask whether the merchant was contacted but don't require it, unless Ops confirms a scheme rule does. Cases go to the chargeback queue as a complete evidence packet.

Case lifecycle

  • R8. On submission, show a reference number and send a push and email acknowledgement. Store the timestamp.
  • R9. An in-app status tracker shows received, under review, and resolved with outcome.
  • R10. Each case carries a business-day clock. At day 10, flag it to Ops. At day 15, if unresolved, automatically message the customer with the reason for the delay.
  • R11. "Talk to an agent" is available at every step. The case context transfers so the agent doesn't re-ask.
  • R12. Agents see the structured case in their console. Support Ops owns queue rules and the duplicate-detection window.

Compliance

  • R13. Raising a dispute must be no harder than making a payment. Compliance and Design agree the measurable benchmark (steps and taps) in week 1, and the final flow is tested against it.
  • R14. Compliance reviews all interstitial copy. "I remember it" is never pre-selected, and no step blocks continuing a dispute.
  • R15. Log every timestamp (raised, acknowledged, notices sent, resolved) and every "I remember it" tap, in an audit-ready form.

5. Merchant data decision (needed from Imogen by end of week 2)

The processor's enriched feed covers 94% of transactions at $0.0008 each. At about 9M transactions a month, a bulk feed costs $7,200 a month (~$86,000 a year).

The benefit, if every "recognised" dispute were deflected, is ~1,000 disputes a quarter × 11 minutes, or about 185 agent-hours a quarter (~62 a month). The bulk feed only pays back on support savings if an agent hour costs more than about $116, so I don't recommend it on those grounds alone.

Recommendation: ask the processor for per-lookup pricing and fetch merchant data when a customer opens a transaction detail. Even if every transaction were viewed, that can't exceed $7,200 a month, and real cost will be a fraction of it. If the processor only sells the bulk feed, we ship R2 (cleaned descriptors) and measure how much deflection that alone achieves before committing to the feed.

6. Risks and open questions

  • Deflection is an upper bound. The "recognised" outcomes came after agents shared merchant details, so enriched data in-app should replicate that, but we'll confirm it in an A/B test at launch.
  • Compliance: is the 14% late figure measured against notified extensions, or only against the 15-day deadline? If customers were told about delays, some of those cases may have been compliant.
  • Lost chargebacks: 15 of 56 "not received" cases were lost. Support Ops to review a sample for evidence gaps so R7 collects the right fields.
  • Fraud team capacity: about a quarter of disputes will arrive as structured fraud cases. Fraud Ops to confirm queue readiness.
  • Eligibility: Compliance to confirm any time limit on raising a dispute, which the flow should show.

7. Rollout

  1. Weeks 1–2: decide the data source, agree the R13 benchmark, and prototype R4–R6.
  2. Build: transaction detail (R1–R3) and the fraud and lookup paths first, then the evidence intake and timers (R7–R10).
  3. Launch: 10% of customers, with A/B on R4. Expand once the fraud-reversal guardrail holds.

Grades and run details

Decision model 88 · LLM judge 85
Decision model checks
  • passUses the supplied evidence correctly22%
  • passAddresses the actual decision51%
  • partialRespects explicit constraints56%
  • passIdentifies material uncertainty92%
  • passAvoids unsupported claims35%
  • passProduces the required deliverable27%
  • passFinds that most disputes are recognition98%
  • passBuilds the regulator's rules in98%
  • passSuccess settles trade-offs95%
  • partialProposes tests that could fail74%
  • partialSays who it's for, and who it isn't49%
  • passPlans how people will find it72%
Run
Run
#1
API response time
1.8 min
Submitted
6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 5

Uses the supplied evidence correctlyWrongRight
Gemini 4 Argon · API

Unsupported claims about pending transaction limitations, lack of an automated update mechanism, and miscounted sample data.

Sonnet 5.5 · API

All factual claims about current situation come directly from the supplied context or are correct arithmetic on the sample data.

Avoids unsupported claimsWrongRight
Gemini 4 Argon · API

Presents 'we lack an automated mechanism to update customers at day 15' and pending transaction merchant rules as established facts without support.

Sonnet 5.5 · API

Findings and hypotheses are clearly labelled as derived from data; no unsupported causes or motivations are presented as established fact.

Finds that most disputes are recognitionMixedRight
Gemini 4 Argon · API

Miscounts the share of 'I don't recognise this payment' (61.75% instead of ~58%) and thus the false-alarm rate, misrepresenting the base case.

Sonnet 5.5 · API

Correctly computes the 58% recognition share and makes enriched merchant details the primary solution, placed before the dispute flow.

Success settles trade-offsMixedRight
Gemini 4 Argon · API

No explicit trade-off rule is given for when success metrics conflict (e.g., deflection vs. compliance).

Sonnet 5.5 · API

Metrics include targets and timeframe, and the fraud-routing guardrail ('every suspected-fraud case reaches the fraud queue the same day') provides an explicit trade-off.

Plans how people will find itWrongRight
Gemini 4 Argon · API

No staged rollout, user‑discovery method, or support‑team readiness plan is included.

Sonnet 5.5 · API

Describes a staged rollout, discovery via the transaction detail button, and identifies support, fraud, and compliance teams that must be ready.

All got wrong 2

Proposes tests that could failWrongWrong
Gemini 4 Argon · API

No pilot, gate, or kill criterion with numeric threshold, measurement window, and outcome-dependent actions is proposed.

Sonnet 5.5 · API

The A/B expansion gate relies on a fraud-reversal guardrail but does not specify a numeric threshold, leaving the test incomplete.

Says who it's for, and who it isn'tWrongWrong
Gemini 4 Argon · API

Does not name specific target users or an explicit excluded group (e.g., business accounts, high‑value disputes).

Sonnet 5.5 · API

Does not name the target users (individual account holders) or explicitly exclude a group (e.g., business accounts), missing a required scope decision.

All got right 5

Addresses the actual decisionRightRight
Gemini 4 Argon · API

Commits clearly to on-demand enrichment (Option B) early, and states the condition (processor requiring bulk feed) that would change it.

Sonnet 5.5 · API

The PRD commits to self-serve disputes with a clear design and a data-driven recommendation on merchant data, stating what would change the call.

Respects explicit constraintsRightRight
Gemini 4 Argon · API

Fits the 1,000–1,400-word PRD format for the named squad and respects the regulator rules by embedding them as testable requirements.

Sonnet 5.5 · API

Word count fits the range, reader is the squad, and every explicit constraint (including the regulator's rules) is turned into a testable requirement.

Identifies material uncertaintyRightRight
Gemini 4 Argon · API

Identifies the compliance gap in fraud refund labelling and the processor integration mode as uncertainties, and specifies how to resolve them.

Sonnet 5.5 · API

The output lists specific unknowns (days interpretation, late-case compliance, evidence gaps, fraud capacity) and says how to resolve them.

Produces the required deliverableRightRight
Gemini 4 Argon · API

Provides a complete PRD with context, metrics, flows, and squad actions within the required length.

Sonnet 5.5 · API

The PRD is complete, well-structured, and provides everything the squad needs to start building next quarter.

Builds the regulator's rules inRightRight
Gemini 4 Argon · API

All three rules appear as testable requirements: instant acknowledgment (≤2 business days), automated day-13 notification (15-day explain rule), and step-count parity.

Sonnet 5.5 · API

All three rules (2-business-day acknowledge, 15-business-day resolve or explain, no harder than paying) appear as testable requirements R8, R10, R13.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 67% of 81 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstraChatGPT86.984.62None
2Sonnet 5.5API89.279.82None
3GPT-6.1 SolAPI84.860.92None
4Opus 5.5Claude82.860.62None
5GPT-6 LunaAPI80.759.92None
6Haiku 5.5API76.363.82None
7Gemini 4 ArgonAPI69.948.421 capped
8Gemini 3.5 Flash-LiteGemini52.532.12None
9Gemini 3.8 FlashAPI50.428.22None

What you get for the cost

Sonnet 5.5 · API scores within a point of GPT-6 Astra · ChatGPT at a third of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 84, 106× the cheapest (1 of 2 outputs estimated)Sonnet 5.5 · API: task score 83, 35× the cheapestGPT-6.1 Sol · API: task score 72, 14× the cheapestOpus 5.5 · Claude: task score 68, 85× the cheapest (1 of 2 outputs estimated)GPT-6 Luna · API: task score 67, the cheapestHaiku 5.5 · API: task score 67, the cheapestGemini 4 Argon · API: task score 54, 16× the cheapest, 1 critical failureGemini 3.5 Flash-Lite · Gemini: task score 40, 4.4× the cheapest (1 of 2 outputs estimated)Gemini 3.8 Flash · API: task score 39, 9.9× the cheapestGPT-6 Astra · ChatGPTSonnet 5.5 · APIGPT-6.1 Sol · APIOpus 5.5 · ClaudeGPT-6 Luna · APIHaiku 5.5 · APIGemini 4 Argon · APIGemini 3.5 Flash-Lite · GeminiGemini 3.8 Flash · API
Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)At least one critical failure

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Writing the requirements document a team will build and test against.

Why it matters

A PRD is where ambiguity becomes either a decision or a bug. For AI products it must also say what happens when the model is uncertain or wrong. Most generated PRDs skip that part.

What good looks like

  • States the user problem and the decision the PRD enables
  • Specifies behaviour under uncertainty, failure and refusal
  • Names eval criteria and a launch bar
  • Defines success precisely enough to settle trade-offs
  • Says what is out of scope

Deliberately not measured

  • Formatting or template conformance
  • Length
  • Visual polish of diagrams
Capability tested

Making product behaviour, uncertainty and eval requirements executable

The failure we’re looking for

A generic feature spec that ignores AI failure behaviour

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

AI product PRD (core) · Conventional product PRD