Humanity’s Last PRD
Latest drop: Haiku 5.5 · 8 Oct
Tasks / Product Craft

Write a PRD

Can the model turn a brief into a spec engineers could build from, including how an AI feature behaves when it is wrong?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 25% were usable with at most a quick edit.

Reliably right

  1. Specifies AI failure behaviour100% pass
    It specifies what happens when uncertain or wrong: block ownership, show retry states, disable assignments on violation, and never fall back to calendar data.
    GPT-6 Astra · ChatGPT · Meeting summaries with action items
  2. Builds the regulator's rules in100% pass
    All three rules (2-business-day acknowledge, 15-business-day resolve or explain, no harder than paying) appear as testable requirements R8, R10, R13.
    Sonnet 5.5 · API · Disputes that aren't disputes
  3. Produces the required deliverable94% pass
    The PRD is complete, written for engineering and design leads, within the 1,000–1,500 word range, and directly actionable.
    GPT-6 Astra · ChatGPT · Meeting summaries with action items

Where it slips

  1. Proposes tests that could fail36% pass
    Launch gates have numeric thresholds and actions, but lack explicit measurement windows (e.g., pilot duration, number of meetings) needed to read out results.
    GPT-6.1 Sol · API · Meeting summaries with action items
  2. Avoids unsupported claims42% pass
    Presents the join log and identity tiers as established facts without labelling them as assumptions, when the context provides no support.
    Opus 5.5 · Claude · Meeting summaries with action items
  3. Uses the supplied evidence correctly44% pass
    The output invents a join log and identity tiers (Verified, Guest, Room) that are not in the supplied context, making unsupported claims about the current system.
    Opus 5.5 · Claude · Meeting summaries with action items

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Success settles trade-offs

    Does the PRD define success as a measurable outcome with a target and time frame, and say which goal wins when two of them conflict?

    Passes when Names a success metric with a target and time frame, and gives at least one explicit trade-off rule (for example, accepting lower coverage to keep precision above a stated level).

    Product and growth leadersOn Lenny’s Podcast

  2. Says who it's for, and who it isn't

    Does the PRD name the specific users it is for, say explicitly who it is not for, and make at least one scope decision that follows from that choice?

    Passes when Names the target users and at least one group it deliberately leaves out, and a scope decision visibly follows from the exclusion.

    Matt LeMayLenny’s Podcast

  3. Plans how people will find it

    Does the PRD say how the feature reaches people: how it rolls out, how users discover it, and who needs to be ready for it (support, sales), rather than stopping at what to build?

    Passes when Gives a staged rollout, a way users will discover the feature, and the teams that need to be ready, all fitted to this feature.

    Ami VoraLenny’s PodcastKarri SaarinenLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Proposes tests that could fail · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for card servicing at Clearbank. Write the PRD for self-serve disputes in the app, as Imogen asks below. It goes to the squad (engineering, design, support operations and compliance) who'll build it next quarter. 1,000 to 1,400 words. The dispute export and the background are attached.

What the model was given6 items: About Clearbank, imogen_note.md, how_disputes_work_today.md, regulator_rules.md, disputes_sample_last_quarter.csv (a random sample of 400 of last quarter's 2,400 disputes), merchant_data_option.md
About ClearbankA digital bank for individuals: current accounts and debit cards, run through a mobile app.
imogen_note.md1 lines · Download
From Imogen (Head of Product): "Write the PRD for self-serve disputes in the app. Support says it's their biggest cost after card freezes. The PRD goes to the squad (engineering, design, support ops and compliance) who will build it next quarter. 1,000 to 1,400 words."
how_disputes_work_today.md5 lines · Download
# How disputes work today

Customers raise a dispute by chatting to support. An agent collects details, then either explains the payment or opens a chargeback. Agents spend about 11 minutes per dispute. 2,400 disputes last quarter; 14% of them were resolved later than the regulator's deadline.

Card transactions show the merchant's billing name (for example "SQ *BRKLYN BAGEL 0412"), not its trading name or logo.
regulator_rules.md5 lines · Download
# Regulator's rules (summary from Legal)

- Acknowledge a dispute within 2 business days.
- Resolve genuine disputes within 15 business days, or tell the customer why it's taking longer.
- Don't make it harder to raise a dispute than to make a payment.
disputes_sample_last_quarter.csv (a random sample of 400 of last quarter's 2,400 disputes)dispute,customer_said,days_to_resolve,outcome CB-40000,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40006,I didn't receive what I paid for,17,Chargeback lost CB-40012,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40018,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40024,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40030,I didn't receive what I paid for,11,Chargeback lost CB-40036,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40042,I didn't receive what I paid for,16,Chargeback lost CB-40048,I was charged twice,5,Not a duplicate: two separate purchases CB-40054,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40060,I was charged twice,10,Not a duplicate: two separate purchases CB-40066,I was charged twice,7,Not a duplicate: two separate purchases CB-40072,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40078,I don't recognise this payment,5,Fraud confirmed CB-40084,I don't recognise this payment,7,Fraud confirmed CB-40090,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40096,I was charged twice,6,Not a duplicate: two separate purchases CB-40102,I didn't receive what I paid for,7,Chargeback won CB-40108,I don't recognise this payment,6,Fraud confirmed CB-40114,I was charged twice,9,Duplicate refunded CB-40120,I was charged twice,7,Not a duplicate: two separate purchases CB-40126,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40132,I was charged twice,9,Duplicate refunded CB-40138,Subscription I cancelled,18,Chargeback won CB-40144,I didn't receive what I paid for,21,Chargeback won CB-40150,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40156,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40162,I don't recognise this payment,6,Fraud confirmed CB-40168,I was charged twice,4,Not a duplicate: two separate purchases CB-40174,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40180,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-40186,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40192,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-40198,I don't recognise this payment,1,Fraud confirmed CB-40204,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40210,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40216,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40222,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40228,I didn't receive what I paid for,29,Chargeback won CB-40234,I don't recognise this payment,8,Fraud confirmed CB-40240,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40246,I didn't receive what I paid for,10,Merchant refunded CB-40252,I don't recognise this payment,4,Fraud confirmed CB-40258,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40264,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40270,I didn't receive what I paid for,21,Chargeback won CB-40276,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40282,I didn't receive what I paid for,17,Merchant refunded CB-40288,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40294,I don't recognise this payment,1,Fraud confirmed CB-40300,I don't recognise this payment,7,Fraud confirmed CB-40306,I don't recognise this payment,8,Fraud confirmed CB-40312,I didn't receive what I paid for,20,Merchant refunded CB-40318,I didn't receive what I paid for,26,Chargeback lost CB-40324,Subscription I cancelled,10,Merchant refunded CB-40330,I don't recognise this payment,1,Fraud confirmed CB-40336,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-40342,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40348,Fraud: I didn't make this payment,9,"Fraud confirmed, refunded" CB-40354,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40360,I was charged twice,9,Not a duplicate: two separate purchases CB-40366,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40372,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40378,I don't recognise this payment,8,Fraud confirmed CB-40384,I didn't receive what I paid for,14,Merchant refunded CB-40390,I don't recognise this payment,7,Fraud confirmed CB-40396,I don't recognise this payment,8,Fraud confirmed CB-40402,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40408,I was charged twice,5,Not a duplicate: two separate purchases CB-40414,I didn't receive what I paid for,13,Chargeback lost CB-40420,Subscription I cancelled,11,Chargeback won CB-40426,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-40432,I didn't receive what I paid for,13,Chargeback won CB-40438,Fraud: I didn't make this payment,4,"Fraud confirmed, refunded" CB-40444,I was charged twice,9,Duplicate refunded CB-40450,I didn't receive what I paid for,18,Chargeback won CB-40456,I don't recognise this payment,1,Fraud confirmed CB-40462,I don't recognise this payment,4,Fraud confirmed CB-40468,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40474,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40480,Fraud: I didn't make this payment,8,"Fraud confirmed, refunded" CB-40486,I didn't receive what I paid for,27,Chargeback lost CB-40492,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40498,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40504,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40510,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40516,I don't recognise this payment,5,Fraud confirmed CB-40522,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40528,Fraud: I didn't make this payment,8,"Fraud confirmed, refunded" CB-40534,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40540,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40546,I don't recognise this payment,2,Fraud confirmed CB-40552,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40558,I was charged twice,5,Duplicate refunded CB-40564,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40570,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40576,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40582,I didn't receive what I paid for,14,Chargeback lost CB-40588,I didn't receive what I paid for,18,Chargeback lost CB-40594,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40600,Fraud: I didn't make this payment,2,"Fraud confirmed, refunded" CB-40606,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40612,I don't recognise this payment,3,Fraud confirmed CB-40618,Fraud: I didn't make this payment,5,"Fraud confirmed, refunded" CB-40624,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40630,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40636,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40642,Fraud: I didn't make this payment,11,"Fraud confirmed, refunded" CB-40648,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40654,I don't recognise this payment,5,Fraud confirmed CB-40660,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40666,I don't recognise this payment,6,Fraud confirmed CB-40672,I was charged twice,5,Not a duplicate: two separate purchases CB-40678,Subscription I cancelled,10,Chargeback won CB-40684,Fraud: I didn't make this payment,9,"Fraud confirmed, refunded" CB-40690,Subscription I cancelled,7,Chargeback won CB-40696,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40702,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-40708,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40714,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40720,I was charged twice,13,Not a duplicate: two separate purchases CB-40726,Subscription I cancelled,13,Chargeback won CB-40732,I don't recognise this payment,3,Fraud confirmed CB-40738,I didn't receive what I paid for,18,Chargeback lost CB-40744,Fraud: I didn't make this payment,8,"Fraud confirmed, refunded" CB-40750,I was charged twice,11,Not a duplicate: two separate purchases CB-40756,Subscription I cancelled,10,Merchant refunded CB-40762,I was charged twice,7,Duplicate refunded CB-40768,Subscription I cancelled,18,Chargeback won CB-40774,I didn't receive what I paid for,23,Chargeback won CB-40780,I didn't receive what I paid for,7,Chargeback won CB-40786,Subscription I cancelled,5,Chargeback won CB-40792,I didn't receive what I paid for,10,Merchant refunded CB-40798,I was charged twice,9,Duplicate refunded CB-40804,I didn't receive what I paid for,24,Merchant refunded CB-40810,I don't recognise this payment,2,Fraud confirmed CB-40816,Fraud: I didn't make this payment,11,"Fraud confirmed, refunded" CB-40822,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40828,Subscription I cancelled,13,Chargeback won CB-40834,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-40840,I don't recognise this payment,7,Fraud confirmed CB-40846,Subscription I cancelled,7,Merchant refunded CB-40852,I don't recognise this payment,3,Fraud confirmed CB-40858,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40864,I was charged twice,4,Not a duplicate: two separate purchases CB-40870,I don't recognise this payment,8,Fraud confirmed CB-40876,I was charged twice,6,Not a duplicate: two separate purchases CB-40882,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-40888,I don't recognise this payment,8,Fraud confirmed CB-40894,I don't recognise this payment,6,Fraud confirmed CB-40900,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40906,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40912,I don't recognise this payment,8,Fraud confirmed CB-40918,I didn't receive what I paid for,21,Chargeback won CB-40924,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-40930,Fraud: I didn't make this payment,8,"Fraud confirmed, refunded" CB-40936,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-40942,I was charged twice,6,Duplicate refunded CB-40948,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-40954,I didn't receive what I paid for,14,Chargeback won CB-40960,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40966,I was charged twice,4,Not a duplicate: two separate purchases CB-40972,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-40978,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-40984,I didn't receive what I paid for,9,Merchant refunded CB-40990,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-40996,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41002,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41008,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41014,Fraud: I didn't make this payment,2,"Fraud confirmed, refunded" CB-41020,I didn't receive what I paid for,27,Chargeback won CB-41026,I didn't receive what I paid for,28,Chargeback won CB-41032,I was charged twice,7,Duplicate refunded CB-41038,I don't recognise this payment,2,Fraud confirmed CB-41044,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41050,I don't recognise this payment,4,Fraud confirmed CB-41056,Fraud: I didn't make this payment,11,"Fraud confirmed, refunded" CB-41062,I was charged twice,3,Duplicate refunded CB-41068,I don't recognise this payment,1,Fraud confirmed CB-41074,I don't recognise this payment,1,Fraud confirmed CB-41080,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41086,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41092,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41098,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41104,I didn't receive what I paid for,29,Merchant refunded CB-41110,Subscription I cancelled,16,Chargeback won CB-41116,I didn't receive what I paid for,22,Chargeback won CB-41122,I didn't receive what I paid for,13,Chargeback lost CB-41128,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41134,I was charged twice,10,Not a duplicate: two separate purchases CB-41140,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41146,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41152,I was charged twice,5,Duplicate refunded CB-41158,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41164,I didn't receive what I paid for,12,Chargeback lost CB-41170,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41176,Fraud: I didn't make this payment,2,"Fraud confirmed, refunded" CB-41182,I don't recognise this payment,1,Fraud confirmed CB-41188,I don't recognise this payment,3,Fraud confirmed CB-41194,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41200,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41206,Fraud: I didn't make this payment,5,"Fraud confirmed, refunded" CB-41212,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41218,Fraud: I didn't make this payment,6,"Fraud confirmed, refunded" CB-41224,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41230,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41236,I don't recognise this payment,3,Fraud confirmed CB-41242,Subscription I cancelled,9,Merchant refunded CB-41248,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41254,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41260,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41266,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41272,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41278,I didn't receive what I paid for,11,Chargeback lost CB-41284,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41290,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41296,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-41302,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41308,I was charged twice,11,Duplicate refunded CB-41314,I didn't receive what I paid for,27,Chargeback won CB-41320,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41326,Fraud: I didn't make this payment,5,"Fraud confirmed, refunded" CB-41332,I was charged twice,12,Not a duplicate: two separate purchases CB-41338,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-41344,I didn't receive what I paid for,17,Merchant refunded CB-41350,I didn't receive what I paid for,21,Chargeback won CB-41356,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41362,Subscription I cancelled,14,Merchant refunded CB-41368,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41374,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41380,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41386,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41392,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41398,I don't recognise this payment,2,Fraud confirmed CB-41404,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41410,I don't recognise this payment,3,Fraud confirmed CB-41416,I don't recognise this payment,5,Fraud confirmed CB-41422,I was charged twice,12,Duplicate refunded CB-41428,Subscription I cancelled,9,Merchant refunded CB-41434,I was charged twice,6,Duplicate refunded CB-41440,Subscription I cancelled,15,Merchant refunded CB-41446,I don't recognise this payment,5,Fraud confirmed CB-41452,I don't recognise this payment,7,Fraud confirmed CB-41458,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41464,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41470,I was charged twice,6,Not a duplicate: two separate purchases CB-41476,I don't recognise this payment,5,Fraud confirmed CB-41482,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41488,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41494,I didn't receive what I paid for,26,Merchant refunded CB-41500,I didn't receive what I paid for,24,Chargeback won CB-41506,Subscription I cancelled,20,Chargeback won CB-41512,I didn't receive what I paid for,29,Merchant refunded CB-41518,Subscription I cancelled,20,Chargeback won CB-41524,I don't recognise this payment,8,Fraud confirmed CB-41530,I was charged twice,3,Not a duplicate: two separate purchases CB-41536,I didn't receive what I paid for,8,Chargeback won CB-41542,I don't recognise this payment,2,Fraud confirmed CB-41548,I don't recognise this payment,2,Fraud confirmed CB-41554,I was charged twice,12,Not a duplicate: two separate purchases CB-41560,Subscription I cancelled,6,Chargeback won CB-41566,I was charged twice,3,Not a duplicate: two separate purchases CB-41572,I don't recognise this payment,1,Fraud confirmed CB-41578,Fraud: I didn't make this payment,7,"Fraud confirmed, refunded" CB-41584,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41590,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-41596,Subscription I cancelled,16,Merchant refunded CB-41602,Subscription I cancelled,15,Merchant refunded CB-41608,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41614,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41620,I was charged twice,5,Duplicate refunded CB-41626,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41632,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41638,Fraud: I didn't make this payment,3,"Fraud confirmed, refunded" CB-41644,I don't recognise this payment,3,Fraud confirmed CB-41650,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41656,I don't recognise this payment,6,Fraud confirmed CB-41662,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41668,I didn't receive what I paid for,12,Chargeback won CB-41674,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41680,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41686,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-41692,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41698,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41704,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41710,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41716,Fraud: I didn't make this payment,8,"Fraud confirmed, refunded" CB-41722,I don't recognise this payment,3,Fraud confirmed CB-41728,I was charged twice,12,Duplicate refunded CB-41734,Fraud: I didn't make this payment,5,"Fraud confirmed, refunded" CB-41740,I don't recognise this payment,6,Fraud confirmed CB-41746,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41752,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41758,I was charged twice,13,Duplicate refunded CB-41764,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41770,I didn't receive what I paid for,23,Chargeback won CB-41776,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41782,I didn't receive what I paid for,27,Chargeback won CB-41788,I didn't receive what I paid for,23,Chargeback lost CB-41794,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41800,I don't recognise this payment,3,Fraud confirmed CB-41806,I don't recognise this payment,1,Fraud confirmed CB-41812,I was charged twice,4,Not a duplicate: two separate purchases CB-41818,I don't recognise this payment,8,Fraud confirmed CB-41824,I didn't receive what I paid for,18,Chargeback won CB-41830,Subscription I cancelled,13,Chargeback won CB-41836,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-41842,I don't recognise this payment,7,Fraud confirmed CB-41848,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41854,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-41860,I was charged twice,5,Duplicate refunded CB-41866,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41872,I don't recognise this payment,8,Fraud confirmed CB-41878,I don't recognise this payment,2,Fraud confirmed CB-41884,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41890,I was charged twice,7,Not a duplicate: two separate purchases CB-41896,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-41902,Fraud: I didn't make this payment,9,"Fraud confirmed, refunded" CB-41908,I don't recognise this payment,4,Fraud confirmed CB-41914,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-41920,I didn't receive what I paid for,24,Merchant refunded CB-41926,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-41932,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-41938,Subscription I cancelled,15,Chargeback won CB-41944,I didn't receive what I paid for,13,Chargeback won CB-41950,I don't recognise this payment,2,Fraud confirmed CB-41956,I didn't receive what I paid for,24,Chargeback lost CB-41962,I don't recognise this payment,4,Fraud confirmed CB-41968,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-41974,Subscription I cancelled,15,Chargeback won CB-41980,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-41986,Fraud: I didn't make this payment,2,"Fraud confirmed, refunded" CB-41992,Fraud: I didn't make this payment,6,"Fraud confirmed, refunded" CB-41998,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-42004,I didn't receive what I paid for,12,Merchant refunded CB-42010,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42016,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42022,I was charged twice,11,Duplicate refunded CB-42028,I didn't receive what I paid for,25,Merchant refunded CB-42034,I didn't receive what I paid for,10,Merchant refunded CB-42040,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42046,I was charged twice,8,Duplicate refunded CB-42052,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-42058,I don't recognise this payment,7,Fraud confirmed CB-42064,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-42070,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42076,Fraud: I didn't make this payment,10,"Fraud confirmed, refunded" CB-42082,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-42088,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-42094,Subscription I cancelled,12,Chargeback won CB-42100,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-42106,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-42112,Subscription I cancelled,18,Chargeback won CB-42118,Fraud: I didn't make this payment,6,"Fraud confirmed, refunded" CB-42124,I don't recognise this payment,5,Fraud confirmed CB-42130,I didn't receive what I paid for,14,Chargeback lost CB-42136,I don't recognise this payment,8,Customer recognised it once shown the merchant's details CB-42142,Fraud: I didn't make this payment,2,"Fraud confirmed, refunded" CB-42148,I was charged twice,12,Not a duplicate: two separate purchases CB-42154,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-42160,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-42166,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42172,Fraud: I didn't make this payment,9,"Fraud confirmed, refunded" CB-42178,I don't recognise this payment,3,Fraud confirmed CB-42184,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-42190,I was charged twice,7,Duplicate refunded CB-42196,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-42202,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-42208,I didn't receive what I paid for,21,Chargeback won CB-42214,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-42220,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-42226,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-42232,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-42238,I was charged twice,10,Duplicate refunded CB-42244,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-42250,Subscription I cancelled,11,Chargeback won CB-42256,Fraud: I didn't make this payment,3,"Fraud confirmed, refunded" CB-42262,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-42268,I didn't receive what I paid for,16,Merchant refunded CB-42274,I was charged twice,3,Duplicate refunded CB-42280,Fraud: I didn't make this payment,9,"Fraud confirmed, refunded" CB-42286,I was charged twice,11,Duplicate refunded CB-42292,I was charged twice,9,Not a duplicate: two separate purchases CB-42298,I don't recognise this payment,6,Customer recognised it once shown the merchant's details CB-42304,I don't recognise this payment,2,Customer recognised it once shown the merchant's details CB-42310,Subscription I cancelled,16,Merchant refunded CB-42316,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-42322,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-42328,Fraud: I didn't make this payment,3,"Fraud confirmed, refunded" CB-42334,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-42340,I don't recognise this payment,1,Customer recognised it once shown the merchant's details CB-42346,I didn't receive what I paid for,19,Merchant refunded CB-42352,I was charged twice,12,Duplicate refunded CB-42358,I don't recognise this payment,3,Customer recognised it once shown the merchant's details CB-42364,I don't recognise this payment,7,Customer recognised it once shown the merchant's details CB-42370,I don't recognise this payment,3,Fraud confirmed CB-42376,I didn't receive what I paid for,17,Chargeback won CB-42382,I don't recognise this payment,4,Customer recognised it once shown the merchant's details CB-42388,I don't recognise this payment,5,Customer recognised it once shown the merchant's details CB-42394,I don't recognise this payment,5,Customer recognised it once shown the merchant's details
merchant_data_option.md3 lines · Download
# Merchant data we could use

Our card processor offers an enriched merchant feed (trading name, logo, location, website) for 94% of transactions, at $0.0008 per transaction. We process about 9 million card transactions a month.
What a strong answer doesThe answer key the graders mark against

Starts from the data: in the sample, 232 of 400 disputes (58%, so about 1,400 of the quarter's 2,400) were 'I don't recognise this payment', and 169 of those (about three in four) were resolved simply by showing the merchant's details; so the first thing to build is recognisable transactions (trading name, logo, location from the enriched feed, about $7,200 a month for 9 million transactions), shown before someone raises a dispute, with self-serve disputes for the genuine cases. Defines success with targets (disputes per 1,000 transactions, share resolved without an agent, regulator deadlines met against 14% late today) and says which wins when they conflict (never make disputing harder than paying, per the regulator). Covers the flows for genuine disputes (charged twice, not received, cancelled subscription, fraud) with acknowledgement within 2 business days and resolution within 15, what happens when the merchant data is missing (6% of transactions), and how the rollout reaches people, with support ready. Says who it's not for (for example business accounts, or disputes over a set amount that still go to an agent) and what's out of scope.

Critical failures (cap the score at 40)
  • Adds steps that make raising a dispute harder than making a payment
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Sonnet 5.5API83Usable with a quick editGPT-6 AstraChatGPT73Needs more workGPT-6 LunaAPI69Needs more workHaiku 5.5API69Needs more workOpus 5.5Claude60Needs more workGPT-6.1 SolAPI58Needs more workGemini 3.5 Flash-LiteGemini40Needs more workGemini 3.8 FlashAPI37Needs more work
GPT-6 AstraChatGPT

Needs more than a quick edit9 of 12 checks passed

Missing an explicit user discovery plan and a non-target user segment, and pause criteria need numeric thresholds. These gaps require more than a quick wording edit.

Output

PRD: Self-serve card disputes

Owner: Card Servicing PM Delivery: Next quarter Squad: Engineering, Design, Support Operations, Compliance Status: Proposed; merchant-data procurement and legal implementation require approval

1. Problem and product decision

Today, customers must chat with support to question a card payment. Agents collect information, explain payments or open chargebacks, spending approximately 11 minutes per dispute. Last quarter’s 2,400 disputes consumed approximately 440 agent hours; 14% were resolved later than the regulator’s deadline.

The opportunity is not simply to replace chat with a chargeback form. Many customers need help identifying a payment; others need fast access to a genuine dispute or fraud investigation.

We will build a transaction-level journey that helps customers understand payments, submit unresolved concerns without an agent, and track progress. Identification help must remain optional: it must never become a barrier to reporting fraud or disputing a payment.

2. Evidence and economics

The random sample of 400 disputes shows:

FindingProduct implication
233 began “I don’t recognise this payment.” Of these, 170 ended with customer recognition and 63 with confirmed fraud.Show useful merchant information, but never assume an unfamiliar payment is legitimate.
Recognition-only cases represent 42.5% of the sample.Preventing confusion is a major opportunity, not a secondary feature.
Of 47 duplicate complaints, 24 were separate purchases and 23 were refunded duplicates.Offer comparison information without automatically dismissing claims.
57 concerned non-receipt, 27 cancelled subscriptions, and 36 explicitly reported fraud.Provide reason-specific intake and operational routing.

Extrapolating recognition-only cases suggests approximately 1,020 cases, or 187 agent hours, per quarter could theoretically be avoided. This is an optimistic ceiling, not a savings forecast.

The processor’s enriched merchant feed covers 94% of transactions. At nine million transactions monthly, it costs $7,200/month or $21,600/quarter. Recognition-only savings would require approximately $116 per avoided agent hour just to cover that fee, before development costs and incomplete coverage.

Therefore, we will not approve blanket feed purchasing on support savings alone. Engineering and Procurement must confirm selective-lookup availability, pricing and data quality. The squad will build an enrichment-compatible detail view, using existing verified information initially if necessary. Full-feed adoption requires a separate business case, including measured recognition improvements.

The export does not identify business versus calendar days or extension notices. We cannot use it alone to reconstruct regulatory breaches.

3. Goals, measures and guardrails

Targets apply to the first full quarter after rollout and are hypotheses unless regulatory:

  • Reduce agent handling hours per 1,000 active card customers by 25%, against a matched baseline; include follow-up contacts and operational review.
  • Achieve 70% self-serve submission among submitted in-scope disputes.
  • Acknowledge 100% within two business days.
  • Resolve genuine disputes, or send a substantive explanation of delay, within 15 business days in 100% of cases.
  • No deterioration in confirmed-fraud reporting time, repeat complaints or accessibility versus baseline.

Track median and p90 resolution time separately from deadline compliance. An extension notice is not a resolution.

Instrument detail views, help entry, recognition confirmations, reason selection, submission, abandonment, support handoff, case updates and outcomes. Report by reason and enrichment availability. Only explicit recognition counts as recognition success; exits do not.

4. Scope

Included: Mobile-app debit-card payment identification, intake for the five sampled reasons plus “Something else,” evidence upload, case acknowledgement, status tracking, operational routing and deadline controls.

Excluded: Automated liability decisions, automatic chargeback adjudication, rebuilding card-scheme integrations, non-card disputes, and a new refund policy. Existing specialists retain responsibility for investigation, refunds and chargebacks.

Pending payments must have an accessible reporting route, even if a chargeback cannot yet proceed. Explain the distinction and route suspected fraud immediately. Support remains available throughout.

5. Customer experience and requirements

A. Understand the payment

Transaction details display the original billing descriptor, amount, date and available verified merchant information. Enriched trading name, logo, location and website appear when available and sufficiently reliable; retain the descriptor for traceability. Missing or uncertain matches must not block the journey or imply legitimacy.

Provide prominent “Report a problem” and “I didn’t make this payment” actions. Identification guidance is skippable. Only an explicit “I recognise this payment” ends the identification journey; customers can return later to dispute it.

B. Raise a concern

Prefill transaction and customer details. Let customers choose:

  • Unrecognised / not authorised: Submit directly to the fraud team; offer the existing card-freeze action without making freezing a prerequisite.
  • Charged twice: Show potentially related transactions and let customers select the second payment or continue without finding it. Similar amounts do not prove duplication.
  • Goods or services not received: Ask what was expected and the expected delivery date.
  • Cancelled subscription: Ask when cancellation occurred and invite supporting evidence.
  • Something else: Accept a short description and route for triage.

Collect a concise account of the problem. Request receipts, cancellation confirmations and merchant correspondence where useful, but permit evidence later. Merchant contact is not a universal submission prerequisite.

Customers review and submit without re-entering known information. Save drafts, support screen readers and dynamic text, and provide contextual support with entered details preserved.

C. Acknowledge and track

After durable case creation, immediately display a case reference, receipt timestamp, summary and next-update date. Deliver an in-app acknowledgement and notification; do not rely solely on push delivery.

The case screen shows understandable states: Received, Under review, Information needed, Awaiting external response, Resolved. Show outstanding requests, update history and any refund status separately.

Resolution includes the decision, explanation and support/escalation route. Never promise a refund or successful chargeback.

6. Operational and technical requirements

Create one case record shared by app and support, containing the transaction, reported reason, narrative, attachments, owner, timestamps, deadlines and communication history.

Submission must be idempotent. Retries cannot create duplicate cases; an existing case should be surfaced with an option to add information. Confirm receipt only after persistence. Failed submissions retain entered data and expose a support fallback.

Support Operations owns reason-based queues, fraud prioritisation, daily deadline monitoring and capacity planning. Agents can request evidence, change routing, record decisions and send approved updates without asking customers to repeat information.

Engineering must implement:

  • Configurable business-day calendars and deadline calculation.
  • Alerts before deadlines, including day-one acknowledgement and day-ten/day-fourteen resolution warnings.
  • Escalation to a named operational owner before breach.
  • Reconciliation between app submissions and operational queues.
  • Encryption, role-based access, attachment validation and auditable state changes.

Compliance must approve clock-start rules, holidays, retention, extension wording and any additional fraud/refund obligations. Preserve the earliest report timestamp across channels; asking for evidence must not silently reset deadlines. Delay notices must explain the actual reason and next update, not merely mark cases compliant.

7. Delivery, acceptance and rollout

Weeks 1–2: Validate merchant-data options; audit existing case-system integration; agree legal rules; baseline reason-level handling time. Design tests prototypes with customers, including accessibility needs.

Weeks 3–8: Build intake, case integration, tracking and deadline controls. Develop merchant-detail improvements in parallel without making reporting dependent on enrichment.

Weeks 9–10: Run end-to-end testing, operational rehearsal and a limited pilot. Test missing merchant data, uncertain matches, fraud bypass, pending payments, failed uploads, repeat taps, holiday deadlines and cross-channel reporting.

Weeks 11–12: Expand progressively after Engineering, Support Operations and Compliance sign-off.

Launch requires zero lost or duplicate cases in acceptance tests, correct deadline calculations, accessible submission, and verified acknowledgement delivery. Design and Compliance must demonstrate that raising a dispute is no harder than making a payment, using comparative journey testing rather than assuming a form satisfies the rule.

Pause expansion for lost submissions, misleading merchant matches, increased fraud-reporting delay or deteriorating deadline compliance. Rollback must preserve existing cases, clocks and customer access to support.

Grades and run details

Decision model 83 · LLM judge 69
Decision model checks
  • passUses the supplied evidence correctly13%
  • passAddresses the actual decision71%
  • partialRespects explicit constraints46%
  • passIdentifies material uncertainty98%
  • passAvoids unsupported claims69%
  • passProduces the required deliverable51%
  • passFinds that most disputes are recognition70%
  • passBuilds the regulator's rules in100%
  • passSuccess settles trade-offs99%
  • partialProposes tests that could fail62%
  • partialSays who it's for, and who it isn't38%
  • partialPlans how people will find it32%
Run
Run
#1
Time to output
1.5 min
Submitted
6 Oct 2026
GPT-6.1 SolAPI

Needs more than a quick edit5 of 12 checks passed

The PRD lacks the exact 58% recognition share from the data and a discovery plan, requiring a PM to re-check the sample and fill those gaps before use.

Output

PRD: Self-serve card disputes

Owner: Card Servicing PM Delivery: Next quarter Audience: Engineering, Design, Support Operations, Compliance Status: Proposed; regulatory interpretation and merchant-data commercial approval required

1. Problem and product decision

Customers currently dispute card payments through support chat. Agents explain unfamiliar payments or collect information and open a chargeback. Last quarter, 2,400 disputes required approximately 440 agent hours at 11 minutes each. Support identifies disputes as its biggest cost after card freezes. More importantly, 14% were resolved after the regulatory deadline.

The 400-case random sample shows two distinct needs:

  • Understanding a payment: Unrecognised payments are the largest category, and many customers recognise them after seeing merchant details. Some suspected duplicates are actually separate purchases.
  • Getting a genuine problem investigated: Unrecognised payments also include confirmed fraud. Non-receipt and cancelled-subscription cases can take longer; several non-receipt cases took 17–29 days.

Today’s billing descriptors, such as “SQ *BRKLYN BAGEL 0412”, contribute to confusion.

Decision: Build a payment-detail-led experience that helps customers recognise legitimate transactions, then lets them report unresolved problems without contacting an agent. Pair this with case tracking and deadline controls. This is not an automated chargeback adjudication project.

2. Goals and success measures

Proposed targets for the first full quarter after rollout, subject to baseline validation:

  • Reduce total dispute-related agent minutes by 30%, normalised for card-transaction volume. At current volume, this represents approximately 132 hours per quarter.
  • Submit at least 60% of new card disputes through self-service, without requiring support chat.
  • Acknowledge 100% within two business days.
  • Resolve genuine disputes within 15 business days, or provide an individual explanation before that deadline; target zero missed obligations.
  • Reduce clarification-only contacts without increasing repeat reports, complaints or incorrectly closed cases.

Track completion time, abandonment by step, support handoffs, evidence completeness, repeat contacts and resolution outcomes by reason. Measure recognition using an explicit customer confirmation—not abandonment.

The existing 14% late-resolution figure is the operational baseline, not necessarily a measured regulatory-breach rate: the export does not show business-day calculations or extension communications.

3. Scope and priorities

Must ship

  • “Get help with this payment” on card-transaction details.
  • Merchant clarification with available enriched details.
  • Structured reporting for unrecognised payments, suspected fraud, duplicates, non-receipt and cancelled subscriptions.
  • “Something else” and accessible human-support routes.
  • Durable case creation, immediate acknowledgement, operational routing and deadline monitoring.
  • In-app case status, information requests and outcome communications.

Not in scope

Automated liability decisions, automatic chargeback submission, a replacement chargeback platform, bank-transfer disputes, or a broad redesign of card freezes. Use existing investigation, refund and card-security processes.

Customers with unsupported scenarios must still be able to raise a report; scope boundaries must not become silent rejection rules.

4. Customer experience requirements

A. Clarify the payment

Show the original billing descriptor, amount, date, payment status and, where available, merchant trading name, logo, location and website on transaction details.

For “I don’t recognise this payment”, offer two equally clear actions: “I recognise it now” and “Report a problem”. Clarification is optional, not a mandatory gate. An explicit fraud report bypasses recognition prompts.

Never imply that an enriched merchant match proves authorisation. Preserve original details, label uncertain information appropriately, and provide “These details look wrong”. If enrichment is unavailable or fails, reporting remains available.

For suspected duplicates, show relevant nearby transactions side by side, including pending versus completed status. Do not close a report because two purchases look similar; only the customer can confirm recognition.

B. Capture the problem

Prefill transaction and customer details. Ask only reason-specific questions:

  • Unrecognised/fraud: whether the customer authorised the payment and any useful context; route unresolved unrecognised reports for fraud assessment.
  • Duplicate: identify the other transaction, if known.
  • Non-receipt: expected delivery/service date and what was purchased.
  • Cancelled subscription: cancellation date and available confirmation.
  • Other: short free-text description.

Offer evidence uploads, but allow submission without documents and explain that further information may be requested. Record merchant-contact history where relevant; do not require customers to contact a merchant before reporting.

A short review screen leads to submission. Preserve answers during interruptions and support handoffs. Do not make customers repeat information already supplied.

C. Protect customers reporting fraud

Offer the existing card-freeze action and explain its effect. Reporting must not depend on freezing the card. Apply existing fraud-escalation procedures, including urgent human assistance where needed.

Submission must not promise a refund, imply that all fraud follows chargeback rules, or defer any applicable refund obligation until investigation completion. Compliance must approve this wording and routing.

D. Confirm and track

On successful durable receipt, show the case reference, reported transaction, acknowledgement, next step and expected update date. Provide an in-app message, with push notification as a supplement.

Customers can view status, add information and contact support from the case. Outcomes explain the decision, any refund and how to challenge it. A chargeback loss must not automatically determine the customer’s entitlement.

5. Operational and technical requirements

Create cases in the existing operational workflow with customer ID, transaction ID, reason, answers, attachments, reporting timestamp, owner and deadline fields. Support receives a readable summary rather than an unstructured transcript.

Requirements:

  • Reliable submission: retries must not create duplicate cases. Existing cases link customers to tracking while permitting additional information.
  • Cross-channel continuity: connect app and chat reports; preserve the earliest received report timestamp.
  • Deadline engine: calculate business days using Compliance-approved calendars and rules. Neither requests for evidence nor internal handoffs reset the clock.
  • Escalation: alert the responsible team before acknowledgement and resolution deadlines; escalate unowned or overdue cases.
  • Extension communications: require a meaningful reason and next update date before day 15 when unresolved. Generic status updates do not satisfy this requirement.
  • Auditability: retain report history, decisions, customer communications and delivery records under approved retention rules.
  • Security: enforce account ownership, secure attachments, malware scanning, least-privilege access and redaction of sensitive analytics.
  • Accessibility: support screen readers, scalable text and plain language; keep human assistance available throughout.

If submission fails, never show success. Retain entered information and offer support with context. Queue downstream processing only after durable receipt.

6. Merchant-data decision and economics

The processor offers enrichment for 94% of transactions at $0.0008 per transaction. Across nine million monthly transactions, a full feed costs approximately $7,200 monthly, or $21,600 quarterly, excluding integration costs.

Coverage is not accuracy. Engineering and Design must validate matching quality, latency, caching and fallback behaviour. Procurement must confirm whether selective lookup or a limited pilot is commercially possible; neither is assumed.

Do not justify the full feed solely through the 440-hour support baseline: we lack loaded hourly costs and evidence of achieved deflection. Approve full-feed spend only with a business case covering support savings and broader customer benefits. If approval slips, ship reporting and deadline controls using existing merchant information rather than delay safer dispute handling.

7. Delivery, validation and launch gates

Weeks 1–2: Design tests recognition, fraud and reporting journeys with customers; Support Operations validates routing and workload. Compliance approves clock rules, acknowledgement, extension wording, refund obligations and reporting-access requirements. Engineering confirms case-system and feed dependencies.

Weeks 3–8: Build transaction help, reason-based intake, case integration, tracking and deadline controls. Instrument journeys without capturing sensitive free text.

Weeks 9–12: Run end-to-end tests and a limited customer rollout, then expand if gates pass.

Release acceptance requires demonstrated:

  • Submission despite missing enrichment or evidence.
  • Accessible fraud reporting without recognition or merchant-contact gates.
  • Correct weekend/holiday deadlines and pre-deadline escalation.
  • Idempotent retries and cross-channel continuity.
  • Recoverable outages, secure attachments and usable support handoffs.

Support Operations owns queue coverage and escalation response; Compliance signs off controls; Engineering owns reliability; Design owns accessibility and comprehension.

Compare pilot and control cohorts on agent minutes, clarification contacts and repeat reports. Do not withhold required protections from controls. Pause expansion for deadline failures, misrouting, data exposure or evidence that customers cannot report genuine problems. A feature rollback must preserve submitted cases, monitoring and human-support access.

Grades and run details

Decision model 83 · LLM judge 38
Decision model checks
  • passUses the supplied evidence correctly50%
  • passAddresses the actual decision81%
  • partialRespects explicit constraints38%
  • passIdentifies material uncertainty99%
  • passAvoids unsupported claims59%
  • passProduces the required deliverable67%
  • partialFinds that most disputes are recognition31%
  • passBuilds the regulator's rules in96%
  • passSuccess settles trade-offs96%
  • partialProposes tests that could fail65%
  • partialSays who it's for, and who it isn't52%
  • passPlans how people will find it27%
Run
Run
#1
API response time
73 s
Submitted
6 Oct 2026
Haiku 5.5API

Needs more than a quick edit8 of 12 checks passed

The PRD lacks user exclusion, discovery plan, and test thresholds, requiring rework on those sections before a squad could act.

Output

PRD: Self-serve disputes in the app

Owner: Card servicing PM Audience: Engineering, Design, Support Operations, Compliance Status: Draft for squad review Target: Next quarter

1. Problem

Customers raise disputes by chatting with support. An agent collects the details, then either explains the payment or opens a chargeback. Each dispute takes about 11 agent minutes. Support tells us disputes are their biggest cost after card freezes. Last quarter we handled 2,400 disputes, and 14% were resolved after the regulator's deadline.

2. What the data says

I reviewed a random sample of 400 of last quarter's disputes. Percentages are accurate to about ±5 points at 95% confidence.

What the customer saidShare of sampleWhat happened
"I don't recognise this payment"58% (233)171 ended with the customer recognising the payment once shown the merchant's details. 62 were confirmed fraud. All closed within 8 days.
"I was charged twice"12% (48)24 were two separate purchases. 24 were true duplicates, refunded.
"Fraud: I didn't make this payment"9% (35)All confirmed fraud and refunded, in 2 to 11 days.
"I didn't receive what I paid for"14% (56)Merchant-dependent: 24 chargebacks won, 17 merchant refunds, 15 chargebacks lost.
"Subscription I cancelled"7% (28)Merchant-dependent: mostly won or refunded.

The main finding: the largest group is a recognition problem, not a dispute problem. Card transactions show the merchant's billing name (for example, "SQ \*BRKLYN BAGEL 0412"). Customers don't recognise it, contact us, and an agent spends time showing them the merchant details the app should have shown in the first place. Roughly half of "charged twice" cases are the same confusion: two separate purchases.

The late cases are a different problem. In the sample, 43 of 400 disputes took more than 15 days as recorded in the export. 36 of those were "didn't receive" disputes, which depend on merchant responses. Self-serve intake won't fix these by itself. They need proactive status updates so customers aren't left waiting. (Legal to confirm whether the export counts business or calendar days.)

Fraud is a minority but time-critical. About 24% of the sample was confirmed fraud. These customers need to freeze the card first, and the dispute should follow from that.

3. Goals

  1. Customers can raise, track, and close any card dispute in the app without starting a chat.
  2. Recognition and duplicate cases close without an agent wherever possible.
  3. Fewer disputes miss the regulator's deadline. Proposed target: under 5%, down from 14%.
  4. Disputing is never harder than paying. This is a regulatory requirement, not just a design goal.

Non-goals for this release: - Changing chargeback decision rules or card scheme processes. - Self-serve approval of chargebacks. Agents still decide outcomes for "didn't receive" and subscription disputes, which need merchant contact. Customers can still submit the case in-app. - Redesigning card freezes. That is a separate PRD, although fraud disputes link into it.

4. Requirements

R1: Show the real merchant on every card transaction. Use the processor's enriched feed (trading name, logo, location, website). It covers 94% of transactions at $0.0008 each, about $7,200 a month at our 9 million transactions. For the remaining 6%, show the billing name with a plain-English note that the merchant name couldn't be confirmed.

R2: "Do you recognise this?" as the first step. Before the dispute form, show the enriched merchant details and ask whether the customer recognises the payment. If yes, close the dispute with a confirmation and no agent involvement. If no, one tap continues to the dispute. There must be no dead end, and the wording must be approved by Compliance, because a recognition screen could discourage valid disputes.

R3: Structured intake for each dispute type. Customers choose from the five categories. Each category asks only the questions it needs. For "charged twice," show the matching transactions side by side and ask whether they're for the same thing.

R4: Automatic duplicate handling (conditional). If two charges for the same amount at the same merchant appear within a short window, offer an automatic refund request. This is conditional on Support Ops and Compliance agreeing the rules and thresholds.

R5: Fraud goes to freeze first. "I didn't make this payment" offers an immediate card freeze and replacement, then starts the dispute with the transaction already filled in. Customers must never have to chat before they can freeze their card.

R6: Automatic acknowledgement. Confirm receipt in-app and by push the moment a dispute is submitted. This comfortably meets the 2-business-day rule.

R7: Status and deadlines. Show a timeline for each dispute. Proactively notify the customer at 10 business days (proposed) if it's still open. If it passes 15 business days, show the reason and the expected date. This is what the regulator's "or tell the customer why" requirement looks like in practice.

R8: Evidence upload. Let customers attach receipts, screenshots, or messages to "didn't receive" and subscription disputes.

R9: Chat is always available. "Talk to someone" stays visible at every step. Any hand-off carries the structured answers, so the agent never asks the customer to repeat themselves.

5. Regulatory constraints

  • Acknowledge within 2 business days; resolve within 15 business days or explain the delay. R6 and R7 cover both.
  • Disputing must not be harder than paying. Compliance should define a benchmark, such as the number of taps and screens for a typical card payment, and we test the dispute flow against it. R2 must always offer a one-tap route forward.
  • The recognition screen must not discourage valid disputes. Compliance signs off on the copy and the layout before launch.

6. Success metrics

MetricBaselineProposed target
Disputes closed without an agent0%40% of disputes within one quarter of launch
Agent minutes per agent-handled dispute~11Under 5
Disputes resolved after deadline14%Under 5%
Repeat disputes on the same transaction within 30 daysTo measureNot increasing
Time from fraud report to card freezeTo measureUnder 1 minute
Dispute-related complaintsTo measureNot increasing

Guardrails: payment completion rate and card-freeze volume must not drop. Engineering should instrument the baselines marked "to measure" before the build starts.

7. Rollout

  • Phase 1: Merchant details (R1) and the recognition screen (R2), behind a feature flag. Run an A/B test with a holdout group. The 171 recognition closures in the sample were only observed after an agent showed the merchant details. The test checks whether the app-only version produces the same result.
  • Phase 2: Structured intake for all categories (R3, R5, R8), automatic acknowledgement and tracking (R6, R7), and the agent hand-off (R9).
  • Phase 3: Automatic duplicate handling (R4), if approved.

8. Open questions and risks

  1. Does the 43% recognition rate hold? The sample shows what happened after agents explained the payments. It doesn't show what customers do when they see the merchant details themselves. Phase 1 tests this.
  2. Will self-serve increase dispute volume? Some customers who used to ask a question may now file a dispute. That is acceptable, but we need to watch cost per dispute, not just the count.
  3. Feed quality. Support should check a sample of enriched merchant names for accuracy before launch. An incorrect trading name is worse than the billing name.
  4. Logo use. Legal should confirm we can display merchant logos.
  5. Card scheme deadlines. Our pack doesn't include scheme chargeback timeframes. Legal and Ops need to confirm them, as they may limit how long R7 can promise.
  6. Days field. Confirm whether `daystoresolve` counts business or calendar days. It changes how we read the late-case figures.

Grades and run details

Decision model 75 · LLM judge 69
Decision model checks
  • failUses the supplied evidence correctly18%
  • passAddresses the actual decision70%
  • partialRespects explicit constraints49%
  • passIdentifies material uncertainty97%
  • partialAvoids unsupported claims31%
  • passProduces the required deliverable35%
  • passFinds that most disputes are recognition100%
  • passBuilds the regulator's rules in98%
  • passSuccess settles trade-offs97%
  • partialProposes tests that could fail74%
  • partialSays who it's for, and who it isn't62%
  • passPlans how people will find it31%
Run
Run
#1
API response time
78 s
Submitted
8 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 5

Uses the supplied evidence correctlyRightMixedMixed
GPT-6 Astra · ChatGPT

Every statement about the current situation is directly from the supplied context or follows by arithmetic.

GPT-6.1 Sol · API

The output states 'Today’s billing descriptors ... contribute to confusion' as a fact about the current situation, but this causal claim is not present in the brief and is not labelled as an assumption; it is an unsupported interpretation.

Haiku 5.5 · API

All facts about the current situation are from the brief or derived from the sample; no invented facts.

Avoids unsupported claimsRightMixedRight
GPT-6 Astra · ChatGPT

Labels extrapolations as an 'optimistic ceiling, not a savings forecast' and distinguishes sample-based findings from forward-looking assumptions.

GPT-6.1 Sol · API

The causal claim that billing descriptors contribute to confusion is presented as an established fact without hedging, while the evidence only shows that many customers do not recognise payments.

Haiku 5.5 · API

Interpretations are labelled as findings, and no unsupported causal claims are presented as fact.

Finds that most disputes are recognitionRightWrongRight
GPT-6 Astra · ChatGPT

Computes that 58% of disputes are unrecognised payments and that most resolve by showing merchant details, then makes transaction recognition the first-scope item.

GPT-6.1 Sol · API

The PRD does not compute the exact share (58% = 232/400) or how many were resolved by showing merchant details, only stating that unrecognised payments are the largest category.

Haiku 5.5 · API

Computes 58% recognition share and 171 resolved by showing merchant details, and makes enriched merchant display the first requirement.

Success settles trade-offsRightMixedRight
GPT-6 Astra · ChatGPT

Sets metrics with targets (25% reduction in handling hours, 70% self-serve) and establishes that the regulatory rule against making disputing harder wins over other goals.

GPT-6.1 Sol · API

While success metrics with targets and timeframe are given, there is no explicit trade-off rule stating which goal wins when they conflict (e.g., self-service volume vs. resolution speed).

Haiku 5.5 · API

Defines success metrics with targets and timeframe, and states that the regulatory requirement 'disputing never harder than paying' overrides other goals.

Plans how people will find itWrongMixedMixed
GPT-6 Astra · ChatGPT

Describes a staged rollout but never says how users will discover the feature (e.g., in-app notification, help entry) or who needs to be ready beyond support ops.

GPT-6.1 Sol · API

It gives a staged rollout and identifies support ops, compliance, engineering, and design as ready teams, but does not state how users will discover the self-serve dispute feature in the app.

Haiku 5.5 · API

Rollout is staged but does not describe how users will discover the feature, and support readiness is only partially addressed.

All got wrong 2

Proposes tests that could failWrongWrongWrong
GPT-6 Astra · ChatGPT

Pause triggers like 'misleading merchant matches' or 'increased fraud-reporting delay' lack numeric thresholds and defined measurement windows.

GPT-6.1 Sol · API

Release acceptance criteria are qualitative (e.g., 'demonstrated submission despite missing enrichment'), with no numeric thresholds or measurement windows and only a vague expansion pause condition.

Haiku 5.5 · API

The A/B test in Phase 1 lacks a numeric threshold, measurement window, and action for each outcome; guardrails lack thresholds.

Says who it's for, and who it isn'tWrongWrongWrong
GPT-6 Astra · ChatGPT

Does not explicitly name a user group it's not for (e.g., disputes above a set amount) and no scope decision visibly follows from such an exclusion.

GPT-6.1 Sol · API

The PRD does not name the target users (e.g., individual current account holders) nor explicitly exclude a group (e.g., business accounts), and no scope decision visibly follows from such an exclusion.

Haiku 5.5 · API

Does not name the target users (e.g., individual current account holders) or explicitly exclude any user group, and no scope decision follows from a user exclusion.

All got right 5

Addresses the actual decisionRightRightRight
GPT-6 Astra · ChatGPT

Commits to building a transaction-level journey with merchant detail view and self-serve disputes, deferring the full-feed purchase to a later business case.

GPT-6.1 Sol · API

The output commits to a clear product decision (payment-detail-led experience, self-serve reporting, deadline controls) and specifies what would change the approach (merchant-data commercial approval).

Haiku 5.5 · API

Commits to building self-serve disputes with merchant recognition first, and says Phase 1 test will validate the approach.

Respects explicit constraintsRightRightRight
GPT-6 Astra · ChatGPT

Delivers a PRD within 1,000–1,400 words, addresses the named squad, and includes enforceable checks for the regulator's rule on dispute difficulty.

GPT-6.1 Sol · API

The PRD is within 1,000–1,400 words, addresses the squad (engineering, design, support ops, compliance), and builds in measures to keep disputing no harder than paying (optional clarification, no mandatory merchant contact).

Haiku 5.5 · API

The PRD is within word count, addresses the squad, and incorporates all regulatory constraints with enforcement mechanisms.

Identifies material uncertaintyRightRightRight
GPT-6 Astra · ChatGPT

Names unknowns in selective merchant data availability, pricing, data quality, and the export's day-type clarity, and ties them to actions that would resolve them.

GPT-6.1 Sol · API

The output names unknowns (regulatory interpretation needed, merchant-data commercial approval, coverage vs. accuracy, baseline validation) and states how they would affect the plan (e.g., ship without full feed if approval slips).

Haiku 5.5 · API

Names specific unknowns (recognition rate in-app, volume increase, feed quality, logo use, scheme deadlines, day counting) and how to resolve them.

Produces the required deliverableRightRightRight
GPT-6 Astra · ChatGPT

Provides a complete PRD with scope, requirements, operational needs, and acceptance criteria that the squad could build from.

GPT-6.1 Sol · API

A complete PRD is delivered for the named squad, containing problem, decision, scope, UX, operational requirements, and rollout plan, within the requested word count.

Haiku 5.5 · API

Complete PRD with problem, data, goals, requirements, metrics, rollout, and open questions, usable by the squad.

Builds the regulator's rules inRightRightRight
GPT-6 Astra · ChatGPT

Includes 2-business-day acknowledgement, 15-business-day resolution/explanation, and 'no harder than paying' as testable requirements.

GPT-6.1 Sol · API

All three regulator rules are turned into testable requirements: acknowledge within 2 business days, resolve within 15 or explain, and disputing not harder than paying (optional clarification, bypass, no forced merchant contact).

Haiku 5.5 · API

All three rules (acknowledge 2 days, resolve 15 days or explain, disputing no harder than paying) appear as testable requirements R6, R7, and regulatory constraints section.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 70% of 81 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstraChatGPT87.184.62None
2Sonnet 5.5API75.979.82None
3GPT-6.1 SolAPI82.660.92None
4Haiku 5.5API76.163.82None
5Opus 5.5Claude75.960.62None
6GPT-6 LunaAPI75.859.92None
7Gemini 3.8 FlashAPI49.828.22None
8Gemini 3.5 Flash-LiteGemini49.832.12None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 84, 106× the cheapest (1 of 2 outputs estimated)Sonnet 5.5 · API: task score 77, 35× the cheapestGPT-6.1 Sol · API: task score 71, 14× the cheapestHaiku 5.5 · API: task score 67, the cheapestOpus 5.5 · Claude: task score 65, 85× the cheapest (1 of 2 outputs estimated)GPT-6 Luna · API: task score 65, the cheapestGemini 3.8 Flash · API: task score 39, 9.9× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 39, 4.4× the cheapest (1 of 2 outputs estimated)GPT-6 Astra · ChatGPTSonnet 5.5 · APIGPT-6.1 Sol · APIHaiku 5.5 · APIOpus 5.5 · ClaudeGPT-6 Luna · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Writing the requirements document a team will build and test against.

Why it matters

A PRD is where ambiguity becomes either a decision or a bug. For AI products it must also say what happens when the model is uncertain or wrong. Most generated PRDs skip that part.

What good looks like

  • States the user problem and the decision the PRD enables
  • Specifies behaviour under uncertainty, failure and refusal
  • Names eval criteria and a launch bar
  • Defines success precisely enough to settle trade-offs
  • Says what is out of scope

Deliberately not measured

  • Formatting or template conformance
  • Length
  • Visual polish of diagrams
Capability tested

Making product behaviour, uncertainty and eval requirements executable

The failure we’re looking for

A generic feature spec that ignores AI failure behaviour

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

AI product PRD (core) · Conventional product PRD