Humanity’s Last PRD
Latest drop: Gemini 3.8 Flash · 1 Oct
Tasks / Operate

Make the launch call

Can the model make a clear go / no-go call from mixed launch evidence, checked against the criteria agreed up front?

Measures the modelTask v1.2 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 14 graded outputs by 7 models. 64% were usable with at most a quick edit.

Reliably right

  1. Finds the Scotland breach100% pass
    Clearly identifies Scotland's refund breach (66% above control, 143% after supplier change) and acceptance dip below 80%, correctly holds Scotland.
    Opus 5.5 · Claude · Go/no-go for AI substitutions, from the rollout data
  2. Produces the required deliverable98% pass
    The recommendation is in the requested form, for the meeting, within the length, and includes necessary steps a PM could act on.
    GPT-6 Astra · ChatGPT · Go/no-go for AI-drafted support replies
  3. Addresses the actual decision95% pass
    The output gives a clear, unambiguous 'no-go' call early, and states what would change it (fixes and retest).
    GPT-6 Astra · ChatGPT · Go/no-go for AI-drafted support replies

Where it slips

  1. Limits the downside of being wrong32% pass
    No specific post-launch monitoring signal or threshold is named; only generic 'monitoring' and a kill switch are mentioned.
    GPT-6 Astra · ChatGPT · Go/no-go for AI-drafted support replies
  2. Catches the duplicate rows50% pass
    Finds and removes the duplicate rows but never states whether they change the results.
    GPT-6.1 Sol · API · Go/no-go for AI substitutions, from the rollout data
  3. Uses the supplied evidence correctly54% pass
    The output states that pilot agents were hands-on, engaged and motivated, which is not supported by the supplied context (only that 7/8 wanted to keep it).
    Opus 5.5 · Claude · Go/no-go for AI-drafted support replies

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Staff PM for fulfilment at Basketful. On 30 October the launch meeting decides whether AI auto-substitutions (the model picks a replacement when an item is out of stock, instead of the picker) roll out to every region on Monday 2 November, ahead of the Christmas delivery-slot booking window that opens on 9 November. The rollout workbook is attached as two files: daily_metrics.csv (28 days of the regional test, by region and arm) and incidents.csv. Work from the data, not the summaries people have given you. Write your recommendation for the meeting: go, no-go or go with conditions, region by region, and why. Include a table that checks each launch criterion for each region with the figures. Keep the prose under 900 words.

What the model was given6 items: The test, Launch criteria (agreed in the PRD), What people have said, Engineering, daily_metrics.csv, incidents.csv
The test1–28 October, five regions. In each region orders were split between control (the picker chooses substitutes, as today) and treatment (AI auto-substitutions). The split was not 50/50 everywhere: treatment got 30% of orders in London, 50% in the South East and North West, and 70% in the Midlands and Scotland.
Launch criteria (agreed in the PRD)1. Substitution acceptance (subs_accepted ÷ subs_offered) of at least 80% in treatment, in every region. 2. Refund requests per 1,000 orders in treatment no more than 10% above control, in every region. 3. Zero dietary or allergen mismatches (a substitute that breaks a gluten-free, vegan, vegetarian, nut-free, halal or kosher attribute). 4. Average basket value not lower in treatment than control.
What people have saidHead of Commercial: “Acceptance is up 15 points and refunds are within the guardrail overall. Every day we wait costs us Christmas.” Head of Analytics: “Treatment baskets are £4 smaller across the test. That worries me.” Ops director (Scotland): “The supplier change caused some noise, but it's settling down.”
EngineeringThe dietary-attribute hard filter (never substitute across those attributes) is merged and scheduled to deploy on 4 November after regression tests. Scotland's catalogue re-map for the new dairy supplier is in progress with no date. Rollout can be switched on region by region.
daily_metrics.csv283 rows · Download
date,region,arm,orders,items_ordered,items_out_of_stock,subs_offered,subs_accepted,refund_requests,refund_value_gbp,complaints,avg_basket_gbp
2026-10-01,London,control,1757,52273,2351,1953,1387,32,160.71,6,72.89
2026-10-01,London,treatment,692,20455,871,820,708,12,62.05,2,72.42
2026-10-01,South East,control,853,22846,996,833,593,16,74.49,3,65.62
2026-10-01,South East,treatment,928,25223,1166,1106,952,15,75.93,3,66.59
2026-10-01,Midlands,control,467,10468,465,383,268,8,34.57,1,54.92
2026-10-01,Midlands,treatment,1113,25136,1166,1087,947,21,106.35,3,55.33
…
incidents.csv13 rows · Download
date,region,arm,type,severity,summary,status
2026-10-02,London,treatment,wrong_size,low,"4-pint milk substituted with 1-pint, customer accepted at door",closed
2026-10-04,Midlands,treatment,dietary_mismatch,high,"Gluten-free sliced loaf substituted with standard white loaf; customer coeliac, noticed at home",closed: refund and apology
2026-10-06,South East,control,wrong_item,low,"Picker substituted oat milk with soya milk; customer rejected at door",closed
2026-10-07,North West,treatment,data_export,low,"Daily export job re-ran after a timeout on 8, 9 and 10 Oct; analytics team flagged possible duplicate rows",open
2026-10-09,London,treatment,price,medium,"Substitute priced higher than original; customer charged the difference against policy",closed: policy fix deployed 11 Oct
2026-10-12,Scotland,control,late_delivery,low,"Van breakdown, 14 orders late",closed
…
What a strong answer doesThe answer key the graders mark against

No-go for a national launch on 2 November, because criterion 3 fails everywhere until the dietary hard filter ships: incidents.csv has three treatment dietary mismatches (4 Oct Midlands, 19 Oct South East, 23 Oct London, the last involving nuts and a child with an allergy). Go with conditions, region by region, once the filter is deployed and verified (planned 4 November, still before the 9 November booking window): London, South East, Midlands and North West pass the other criteria (acceptance about 86% against 71% control; refunds 2–5% below control). Hold Scotland: over the month its treatment refunds are about 66% above control, and from the 15 October supplier change about 143% above (43 against 18 per 1,000 orders), with acceptance falling to about 79%, below the 80% bar; it waits for the catalogue re-map and a clean re-test. The pooled refund rate (about 8% above control) hides this, and the criterion is per region. The pooled basket gap (about £65 against £61) is an artefact of the uneven split, since treatment baskets are equal or higher in every region (Simpson's paradox), so criterion 4 passes. Notices that North West treatment rows for 8–10 October are duplicated (an extra 2,194 orders), removes them, and says it doesn't change the result. States what would change the call: the filter slipping past 9 November, or Scotland's refunds not recovering.

Critical failures (cap the score at 40)
  • Recommends any rollout before the dietary hard filter is deployed
  • Recommends rolling out to Scotland while its refund guardrail is breached
Case

Staff-level · v1.8 · synthetic · e-commerce, AI product, data analysis, staff-level

The outputs

7 setups answered this brief. Pick up to three to read side by side.

Opus 5.5Claude91Usable with a quick editGPT-6.1 SolAPI86Usable with a quick editSonnet 5.5API84Usable with a quick editGPT-6 AstraChatGPT82Usable with a quick editGPT-6 LunaAPI64Needs more workGemini 3.5 Flash-LiteGemini50Needs more workGemini 3.8 FlashAPI40Critical failure
GPT-6 LunaAPI

Needs more than a quick edit9 of 13 checks passed

The refund figures are inaccurate, so a PM would need to re-check the data and recalculate before using the recommendation. The output is not usable as-is.

Output

Recommendation: No-go for the all-region rollout on 2 November

Do not enable auto-substitutions anywhere on 2 November: the dietary hard filter is not scheduled to deploy until 4 November, so the zero-mismatch safety criterion is not yet protected. After deployment and validation, North West is the only region I’d consider enabling first, with conditions. Hold the other regions for the reasons below.

Figures are for 1–28 October; refund rates are per 1,000 orders. Basket averages are order-weighted within each region. I excluded the three exact duplicate North West treatment rows on 8, 9 and 10 October.

RegionTreatment acceptance (≥80%)Refunds / 1,000 orders: treatment vs control (treatment ≤10% higher)Dietary/allergen mismatches (must be zero)Average basket: treatment vs control (must not be lower)Decision
------:---:------:---
London~86% Pass~17.9 vs ~18.4; Pass1; Fail — nut-containing bar substituted for nut-free item~£72.8 vs ~£72.5; PassNo-go
South East~85% Pass~17.8 vs ~17.9; Pass1; Fail — dairy cheese substituted for vegan cheese~£66.6 vs ~£66.3; PassNo-go
Midlands~86% Pass~17.6 vs ~18.0; Pass1; Fail — standard loaf substituted for gluten-free loaf~£55.6 vs ~£55.4; PassNo-go
North West~86% Pass~17.8 vs ~18.8; Pass0 observed; Pass in test~£57.5 vs ~£57.3; PassGo with conditions, after filter validation
Scotland~79% Fail1~29.8 vs ~18.4; Fail — about 62% higher0 observed; Pass in test~£53.0 vs ~£52.6; PassNo-go

Conditions and rationale

  • North West: After the hard filter deploys, pass regression tests across all six protected attributes (gluten-free, vegan, vegetarian, nut-free, halal and kosher); then enable regionally with close monitoring and a rollback switch. The export duplicates should also be fixed or guarded against before relying on daily reporting.
  • London, South East and Midlands: Do not launch until the filter is deployed and the dietary-mismatch failures are investigated and validated as fixed. Their good acceptance, refund and basket figures do not offset a safety-criterion failure.
  • Scotland: Hold until the supplier catalogue re-map is complete and verified, and a fresh test meets both acceptance and refund guardrails. The sharp rise in treatment refunds and lower acceptance coincide with the dairy supplier/catalogue change; the issue is open, with no completion date.

The pooled “£4 smaller” basket claim should not drive this decision: treatment exposure differed by region, and baskets vary substantially by region. Within each region, the treatment average is slightly higher in these data. Likewise, an overall acceptance or refund result cannot override region-level launch gates. The 9 November booking window is important, but it does not justify launching before the safety control is live.

What a PM had to fix

  1. 1Invented evidenceVerify or remove the claimTargeted repair

    What we’d change82.76%: Pass over 1–28 October. However, acceptance fell to 79.08% over 15–28 October after the supplier change, below the launch threshold.

Claims the judge couldn’t find in the brief

  • London treatment refunds per 1,000 orders is ~17.9.
  • London control refunds per 1,000 orders is ~18.4.
  • South East treatment refunds per 1,000 orders is ~17.8.
  • Midlands treatment refunds per 1,000 orders is ~17.6.
  • North West treatment refunds per 1,000 orders is ~17.8 (after excluding duplicates).
  • Scotland control refunds per 1,000 orders is ~18.4.

Grades and run details

Decision model 69 · LLM judge 9 of 14 checks
Decision model checks
  • failUses the supplied evidence correctly8%
  • passAddresses the actual decision47%
  • passRespects explicit constraints61%
  • partialIdentifies material uncertainty55%
  • passAvoids unsupported claims21%
  • passProduces the required deliverable63%
  • passChecks every criterion, region by region28%
  • passFinds the Scotland breach88%
  • passExplains the basket gap86%
  • partialCatches the duplicate rows69%
  • partialMakes a clear call62%
  • partialLimits the downside of being wrong96%
  • failGets the base of every number right45%
Run
Run
#1
API response time
44 s
Submitted
29 Sept 2026
Gemini 3.8 FlashAPI
Critical failure: Recommends any rollout before the dietary hard filter is deployed. Score capped at 40.

Needs more than a quick edit5 of 13 checks passed

A PM could not use it without rechecking evidence because the North West GO and duplicate-row figure are wrong. It also needs added rollback triggers and change conditions.

Output

Recommendation Summary: Go With Conditions

Do not execute a blanket rollout on Monday, 2 November.

  • North West: GO (2 November). Meets all four PRD launch criteria cleanly.
  • London, South East, Midlands: GO WITH CONDITIONS (Deploy Thursday, 5 November). Commercial metrics are strong, but all three regions breached Criterion 3 with severe dietary/allergen substitutions. Launch must be gated on the successful deployment of the dietary-attribute hard filter on 4 November.
  • Scotland: NO-GO. Fails Criterion 2 (+67.4% refund spike vs. control) and Criterion 1 post-supplier transition (76.6% acceptance). Revert 100% to control until catalogue re-mapping is completed and re-tested.

This phased approach protects customer safety and brand reputation while capturing AI substitution value ahead of Christmas slot bookings opening on 9 November.

---

PRD Launch Criteria Assessment

Note: North West treatment metrics reflect deduplication of export retries on 8–10 October (3,194 duplicate rows removed).

RegionArmOrders1. Subs Acceptance (Target: $\ge$80%)2. Refunds / 1k Orders (Target: $\le$+10% vs Ctrl)3. Allergen / Dietary Mismatches (Target: 0)4. Avg Basket Value (Target: $\ge$ Ctrl)Status
:---:---:---:---:---:---:---:---
LondonControl<br>Treatment49,159<br>21,05471.5%<br>87.4%18.06<br>17.53 (-2.9%)0<br>1 (FAIL)£72.52<br>£72.68 (+£0.16)CONDITIONAL<br>Hold for filter
South EastControl<br>Treatment26,485<br>26,91971.7%<br>86.5%17.86<br>17.35 (-2.9%)0<br>1 (FAIL)£66.27<br>£66.57 (+£0.30)CONDITIONAL<br>Hold for filter
MidlandsControl<br>Treatment13,048<br>31,17471.3%<br>85.2%17.86<br>17.58 (-1.6%)0<br>1 (FAIL)£55.37<br>£55.62 (+£0.25)CONDITIONAL<br>Hold for filter
North WestControl<br>Treatment20,702<br>20,77470.8%<br>85.8%18.26<br>17.47 (-4.3%)0<br>0 (PASS)£57.29<br>£57.53 (+£0.24)GO<br>Launch 2 Nov
ScotlandControl<br>Treatment7,853<br>18,37171.3%<br>81.5% (76.6% post-15 Oct)18.21<br>30.48 (+67.4% - FAIL)0<br>0 (PASS)£52.62<br>£52.89 (+£0.27)NO-GO<br>Catalogue fault

---

Analysis & Stakeholder Debunking

#### 1. Head of Analytics: “Treatment baskets are £4 smaller across the test.” Finding: False alarm caused by Simpson’s Paradox. Across all pooled orders, treatment average basket appears lower (£61.06 treatment vs. £65.18 control, a £4.12 deficit). However, this is an aggregation distortion caused by the uneven regional split: London (average basket ~£73) received only a 30% treatment allocation, whereas lower-basket regions (Midlands at ~£55 and Scotland at ~£53) received a 70% treatment allocation.

When evaluated within every single region, treatment basket value was higher than control by £0.16 to £0.30. Criterion 4 is fully satisfied across all regions.

#### 2. Head of Commercial: “Refunds are within the guardrail overall. Every day we wait costs Christmas.” Finding: Misleading pooled metric masking a severe regional failure. While aggregate refunds appear within limits (+8.1% overall), the PRD explicitly requires compliance in every region. Scotland completely breaches this guardrail (+67.4% above control). Launching everywhere on 2 November would risk widespread customer attrition in Scotland right before holiday booking.

#### 3. Ops Director (Scotland): “The supplier change caused noise, but it's settling down.” Finding: Factually incorrect; Scotland is failing and persistent. Following the 15 October dairy supplier change, wrong-size substitutions (e.g., pack sizes mapped to incorrect product families) caused: * Daily refunds in treatment to triple from 11.6/day (Oct 1–14) to 28.4/day (Oct 15–28), reaching 43.7 refunds per 1,000 orders (+140% vs. control). * Treatment acceptance dropped from 86.3% down to 76.6% (failing Criterion 1). * Daily refund value increased 4x to £180–£247/day. Performance has not stabilized. Because the catalogue re-map is incomplete with no ETA, Scotland cannot launch.

#### 4. The Dietary Hazard (London, South East, Midlands) Treatment produced three critical safety failures: * 4 Oct (Midlands): Coeliac customer received standard bread for gluten-free. * 19 Oct (South East): Vegan customer received dairy cheddar. * 23 Oct (London): Child with nut allergies received hazelnut-containing bars for nut-free bars.

Launching auto-substitutions in these regions before deploying technical guardrails creates intolerable liability.

---

Implementation Plan

1. Monday, 2 November: * Roll out North West to 100% treatment. (Clean test data, no allergen incidents, +15pp acceptance lift). * Revert Scotland treatment to 0% (100% control). Keep human pickers selecting substitutes until the catalogue re-map is verified and back-tested. * Maintain London, South East, and Midlands at current test splits for 48 hours.

2. Wednesday, 4 November: * Engineering deploys the dietary-attribute hard filter to production following scheduled regression testing.

3. Thursday, 5 November: * Verify filter telemetry on live orders. * Roll out London, South East, and Midlands to 100% treatment.

4. Monday, 9 November: * Christmas delivery-slot booking opens on schedule with 4 of 5 regions operating under fully automated, safety-enforced substitutions, leaving human pickers dedicated to Scotland until rectified.

Claims the judge couldn’t find in the brief

  • North West: GO (2 November).
  • North West treatment metrics reflect deduplication of export retries on 8–10 October (3,194 duplicate rows removed).

Grades and run details

Decision model 54 · LLM judge 5 of 14 checks
Decision model checks
  • failUses the supplied evidence correctly75%
  • passAddresses the actual decision51%
  • partialRespects explicit constraints23%
  • partialIdentifies material uncertainty39%
  • failAvoids unsupported claims43%
  • passProduces the required deliverable53%
  • partialChecks every criterion, region by region50%
  • passFinds the Scotland breach100%
  • passExplains the basket gap99%
  • partialCatches the duplicate rows84%
  • passMakes a clear call42%
  • failLimits the downside of being wrong30%
  • failGets the base of every number right69%
Run
Run
#1
API response time
85 s
Submitted
1 Oct 2026
Opus 5.5Claude

Usable with a quick edit11 of 13 checks passed

The output is evidence-based, fully meets the brief, and can be used directly by a product manager without re-checking data or redoing analysis.

Output

AI auto-substitutions: launch recommendation

Oct 29, 2026 · @Martin Slaney · For the 30 October launch meeting

Recommendation

No-go for Monday 2 November in any region. The dietary hard filter isn't live until 4 November, and the test produced three allergen and dietary mismatches.

Go with conditions for London, South East, Midlands and North West from Thursday 5 November, once the filter is live and verified. That is still four days before the Christmas booking window opens on 9 November.

No-go for Scotland until the dairy catalogue re-map ships and a fresh week of test data passes the criteria.

Launch criteria by region

Figures are for 1–28 October from daily\_metrics.csv, with duplicate rows removed. T = treatment, C = control.

Region1. Acceptance (T), ≥80%2. Refunds per 1k orders, T vs C, ≤+10%3. Dietary mismatches (T), 04. Avg basket, T vs CVerdict
London86.7% ✅17.4 vs 18.0 (−3%) ✅1: nut-free to hazelnut, 23 Oct, still open ❌£72.70 vs £72.49 ✅Go 5 Nov, after filter
South East86.4% ✅17.5 vs 17.9 (−2%) ✅1: vegan to dairy, 19 Oct ❌£66.64 vs £66.20 ✅Go 5 Nov, after filter
Midlands86.4% ✅17.5 vs 17.9 (−2%) ✅1: gluten-free to standard, 4 Oct ❌£55.66 vs £55.35 ✅Go 5 Nov, after filter
North West86.3% ✅17.6 vs 18.5 (−5%) ✅0 ✅ (same model, so not evidence of safety)£57.51 vs £57.29 ✅Go 5 Nov, after filter
Scotland82.8% overall; 79.1% since 15 Oct ❌30.2 vs 18.2 (+66%); since 15 Oct 43.2 vs 17.8 (+143%) ❌0 ✅£52.98 vs £52.62 ✅No-go until re-map and re-test

Why not 2 November

Criterion 3 requires zero dietary or allergen mismatches. Treatment had three, all rated high severity: a gluten-free loaf swapped for a standard one (the customer is coeliac), vegan cheese swapped for dairy cheddar (the customer complained on social media), and a nut-free multipack swapped for one containing hazelnuts, sent to a child with a nut allergy. The nut incident is still open. Control had none.

The fix is the dietary hard filter, which is merged and deploys on 4 November after regression tests. Launching on 2 November would put every order in every region onto the unfiltered model for two days. North West's zero is luck, because it runs the same model.

Strictly, all four "go" regions fail criterion 3 as it's written. I'm asking the meeting to accept the filter as the remedy, on condition that we show it would have blocked all three incidents. If the meeting won't accept that, the honest answer is no-go everywhere.

Waiting three days costs nothing for Christmas, because bookings don't open until 9 November.

What the summaries get wrong

Head of Commercial: "refunds within the guardrail overall." Pooled, treatment refunds are 7.9% above control (19.5 vs 18.1 per 1,000 orders), which is inside 10%. But the criterion applies to every region, and the pooled figure hides Scotland at +66%. In the other four regions, treatment refunds are below control. The 15-point acceptance lift is real in those four regions (+15.0 to +15.4 points). In Scotland it's +11.7 overall and falling.

Head of Analytics: "treatment baskets are £4 smaller." Pooled, the gap is £61.09 vs £65.15, but it comes from how orders were split, not from the feature. Treatment got 70% of orders in Midlands and Scotland, which have the smallest baskets (£52–56), and only 30% in London, which has the largest (£72). Within every region, treatment baskets are £0.21–£0.44 larger, so criterion 4 passes everywhere.

Ops director, Scotland: "it's settling down." It isn't. Before the supplier switch on 15 October, Scotland's treatment arm matched the other regions (86.3% acceptance, 17.3 refunds per 1,000 vs 18.6 in control). Since then, treatment refunds have run at 25–33 a day with no downward trend: 28.1 a day on average from 15 to 21 October and 28.7 from 22 to 28 October. Refund value per order has tripled, from £0.09 to £0.30. Scotland's control arm is unaffected at 17.8 per 1,000. So the problem is specific to the model reading a mis-mapped catalogue: pickers can see the real pack sizes, and the model can't. The re-map has no date.

Data quality

The North West treatment rows for 8, 9 and 10 October each appear twice, because the export job re-ran (the open incident from 7 October). I removed the duplicates. They don't change any verdict: North West acceptance moves from 86.2% to 86.3%.

Conditions for the 5 November go

  1. The dietary filter deploys on 4 November and passes regression tests. We replay the three incident orders and confirm the filter blocks each of them.2
  2. The London nut-allergen incident is closed with a root cause.
  3. We switch on region by region and keep a 10% control holdout in each region through Christmas, so criteria 2 and 4 stay measurable.
  4. Kill switch: any single dietary or allergen mismatch turns the region off, as does a treatment refund rate more than 10% above its holdout over a rolling 3 days.1
  5. Scotland: ship the catalogue re-map, then run 7 days of test and re-check the criteria. If it can't be ready by 9 November, Scotland takes Christmas orders with picker substitutions.

What a PM had to fix

  1. 1Test or gate too weakTighten the testQuick edit

    What we’d changeSet the refund trigger outside day-to-day noise, with a bigger gap or a longer window, or it will switch regions off for no reason.

  2. 2Test or gate too weakTighten the testQuick edit

    What we’d changeThree replays is thin evidence the filter works. Test it across every protected attribute, including missing or conflicting ones.

Grades and run details

Decision model 81 · LLM judge 14 of 14 checks
Decision model checks
  • failUses the supplied evidence correctly23%
  • passAddresses the actual decision82%
  • passRespects explicit constraints27%
  • passIdentifies material uncertainty97%
  • partialAvoids unsupported claims38%
  • passProduces the required deliverable71%
  • passChecks every criterion, region by region53%
  • passFinds the Scotland breach100%
  • passExplains the basket gap98%
  • passCatches the duplicate rows100%
  • passMakes a clear call96%
  • passLimits the downside of being wrong86%
  • failGets the base of every number right40%
Run
Run
#1
Time to output
2.6 min
Submitted
25 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 8

Uses the supplied evidence correctlyWrongWrongMixed
GPT-6 Luna · API

Several refund-per-1000 figures are incorrect (e.g., London treatment 17.9 vs actual 17.4, London control 18.4 vs 18.0), so the output does not use the supplied evidence correctly.

Gemini 3.8 Flash · API

It contains material unsupported or incorrect current-situation claims, including North West GO despite a dietary incident and a wrong duplicate-row figure.

Opus 5.5 · Claude

Every factual claim about the current situation is drawn directly from the supplied files or computed correctly from them.

Identifies material uncertaintyRightWrongRight
GPT-6 Luna · API

Names the filter deployment, duplicate fix, and Scotland catalogue re-map as unknowns that could change the decision.

Gemini 3.8 Flash · API

It does not clearly name the key unknowns that would change the call, such as the hard filter slipping past 9 November or Scotland failing to recover after re-map.

Opus 5.5 · Claude

Names the unknowns (filter deployment date, Scotland re-map readiness, acceptance of filter as remedy) and says each would change the decision.

Avoids unsupported claimsRightWrongRight
GPT-6 Luna · API

Causal claims like the supplier change coinciding with refunds are presented as observations, not unsupported facts.

Gemini 3.8 Flash · API

It presents North West as clean and safe to launch despite an open dietary incident, and asserts duplicate-row removal with an incorrect figure.

Opus 5.5 · Claude

Hypotheses (e.g. 'luck' for NW zero) are clearly labelled as interpretation; otherwise claims are grounded in the data.

Checks every criterion, region by regionMixedWrongRight
GPT-6 Luna · API

The table checks each criterion per region but the refund figures are incorrect, so it does not present correct figures from the files.

Gemini 3.8 Flash · API

The table checks all four criteria by region, but the North West dietary result is wrong and the duplicate handling figure is wrong.

Opus 5.5 · Claude

All four launch criteria are checked for each region with correct computed figures from the files.

Catches the duplicate rowsRightWrongRight
GPT-6 Luna · API

Catches the duplicate North West rows, excludes them, and recommends fixing the export job.

Gemini 3.8 Flash · API

It notices the North West duplicates but misstates them as 3,194 duplicate rows rather than 2,194 duplicate orders across three rows.

Opus 5.5 · Claude

Notices the duplicated North West treatment rows for 8–10 October, removes them, and confirms the verdict doesn't change.

Makes a clear callRightMixedRight
GPT-6 Luna · API

Makes a clear call per region with dates and conditions, and says what would change it.

Gemini 3.8 Flash · API

It gives dates and conditions, but the North West call is wrong and it does not state what would change the overall call.

Opus 5.5 · Claude

Gives a clear decision per region with dates (no-go 2 Nov, go with conditions 5 Nov for four regions, no-go for Scotland) and states what would change it.

Limits the downside of being wrongWrongWrongRight
GPT-6 Luna · API

Does not name a specific post-launch signal, threshold, or trigger for rollback; only says 'close monitoring and a rollback switch' without specifics.

Gemini 3.8 Flash · API

It lacks post-launch monitoring signals, pause/rollback thresholds, and irreversible-risk triggers.

Opus 5.5 · Claude

Specifies post-launch kill switch (any dietary mismatch or refund rate >10% above holdout over rolling 3 days), a 10% control holdout, and conditions for Scotland re-test.

Gets the base of every number rightWrongWrongMixed
GPT-6 Luna · API

Multiple refund-per-1000 figures are miscalculated (e.g., London treatment 17.9 vs 17.4, London control 18.4 vs 18.0), so the base numbers are wrong.

Gemini 3.8 Flash · API

Several derived figures are correct, but the duplicate-row count and North West dietary status are wrong.

Opus 5.5 · Claude

All derived figures (acceptance, refunds per 1k, basket averages) are computed correctly from the supplied data with appropriate denominators.

All got right 5

Addresses the actual decisionRightRightRight
GPT-6 Luna · API

Commits to a clear no-go for all-region on 2 Nov, then conditional go for North West after filter, with conditions and what would change it.

Gemini 3.8 Flash · API

It commits early to a clear region-by-region call and states conditions and dates.

Opus 5.5 · Claude

Commits unequivocally to a no-go for 2 November, go with conditions from 5 November for four regions, and no-go for Scotland, and states what would change each call.

Respects explicit constraintsRightRightRight
GPT-6 Luna · API

Includes the required region-by-region table, prose under 900 words, and addresses the launch meeting.

Gemini 3.8 Flash · API

It is under 900 words, includes a region-by-criterion table, and respects the launch-criteria framing.

Opus 5.5 · Claude

Respects the 900-word prose limit, includes the required region-by-criterion table, and is addressed to the launch meeting as requested.

Produces the required deliverableRightRightRight
GPT-6 Luna · API

Provides a complete recommendation with table, region-by-region, usable for the meeting.

Gemini 3.8 Flash · API

The requested recommendation and table are present and usable in form, though some figures need correction.

Opus 5.5 · Claude

The output is a complete, actionable recommendation with a criterion-check table, suitable for the meeting with minimal editing.

Finds the Scotland breachRightRightRight
GPT-6 Luna · API

Identifies Scotland's refund breach, ties it to the supplier change, notes acceptance dip, and holds Scotland.

Gemini 3.8 Flash · API

It identifies Scotland's refund breach, ties it to the supplier change, notes acceptance dip, and holds Scotland.

Opus 5.5 · Claude

Clearly identifies Scotland's refund breach (66% above control, 143% after supplier change) and acceptance dip below 80%, correctly holds Scotland.

Explains the basket gapRightRightRight
GPT-6 Luna · API

Explains the pooled basket gap as a mix effect, noting treatment baskets are equal or higher within each region.

Gemini 3.8 Flash · API

It compares within-region baskets and explains the pooled gap as a Simpson's Paradox mix effect.

Opus 5.5 · Claude

Explicitly explains the pooled basket gap as a traffic-mix artefact from the uneven regional split, with within-region comparisons showing treatment baskets are higher.

Results

Every setup we’ve tested on this task, across all cases and repeats, graded on the current checklist. Calibrated: the graders match our PM on 87% of checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6.1 SolAPI87.388.32None
2GPT-6 AstraChatGPT86.088.32None
3Sonnet 5.5API85.488.32None
4Opus 5.5Claude80.486.42None
5GPT-6 LunaAPI82.177.62None
6Gemini 3.8 FlashAPI71.931.521 capped
7Gemini 3.5 Flash-LiteGemini50.642.22None

The PM job

Deciding whether a feature ships on the planned date, and on what conditions.

Why it matters

Launch meetings reward optimism. A good call checks each agreed criterion, weighs a real risk against a real gain, and says exactly what would change the answer. A weak one rubber-stamps the launch or blocks it on noise.

What good looks like

  • Checks each agreed criterion against the evidence
  • Makes one clear call, with conditions if needed
  • Separates risks that block launch from risks that can be managed
  • Says what would change the call
  • Says how to limit the damage if the call is wrong

Deliberately not measured

  • Rollout engineering detail
  • Project-plan formatting
Capability tested

Deciding against agreed launch criteria

The failure we’re looking for

Rubber-stamps a launch that misses an agreed bar, or blocks it on noise

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

A launch memo from supplied evidence · Staff level: a regional call from an attached data workbook