Tasks / Product Craft

Make the launch call

Can the model make a clear go / no-go call from mixed launch evidence, checked against the criteria agreed up front?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 69% were usable with at most a quick edit.

Reliably right

Finds the Scotland breach100% pass
Finds Scotland's refund breach (+66% overall, +143% since 15 Oct), ties it to the supplier change, notes acceptance dip, and holds Scotland.
Sonnet 5.5 · API · Go/no-go for AI substitutions, from the rollout data
Treats the overwrites as data loss100% pass
Treats the overwritten notes on Android 11 as a breach of the no-data-loss criterion and makes fixing the storage-API issue a condition for that platform.
Opus 5.5 · Claude · Offline, on older phones
Addresses the actual decision98% pass
Commits to a clear no-go for 2 Nov, conditional go from 4 Nov for four regions, and no-go for Scotland, with conditions that would change it.
Sonnet 5.5 · API · Go/no-go for AI substitutions, from the rollout data

Where it slips

Limits the downside of being wrong63% pass
It does not name a specific post-launch signal, a threshold that would trigger pause or rollback, or flag any irreversible actions.
GPT-6 Luna · API · Go/no-go for AI substitutions, from the rollout data
Judges the launch against a baseline set now66% pass
Does not name a specific review point or when the team will review results to decide next steps; only gives a pause trigger without a scheduled evaluation.
GPT-6.1 Sol · API · Offline, on older phones
Checks every criterion for every platform69% pass
Android 13's error-free sync is 98.9%, below the 99% threshold, yet the table marks it as 'GO WITH CONDITIONS' instead of 'NO-GO' for that criterion, misapplying the hard launch criteria.
Gemini 3.5 Flash-Lite · Gemini · Offline, on older phones

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Limits the downside of being wrong
Does the call say how exposure is limited if it's wrong: what is watched after launch, the threshold that would pause or roll back, and which parts can't be undone?
Passes when Names the post-launch signal, a threshold or trigger, and the action, and flags anything irreversible (customer communications, pricing, migrations).
Karri SaarinenLenny’s Podcast
Judges the launch against a baseline set now
Does the call name the metric and the current baseline the launch will be judged against, and when the team will review the result to decide what's next?
Passes when Names the success metric, its baseline from the material, and a review point with what each result would lead to.
Todd JacksonLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 5 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Staff PM for fulfilment at Basketful. On 30 October the launch meeting decides whether AI auto-substitutions (the model picks a replacement when an item is out of stock, instead of the picker) roll out to every region on Monday 2 November, ahead of the Christmas delivery-slot booking window that opens on 9 November. The rollout workbook is attached as two files: daily_metrics.csv (28 days of the regional test, by region and arm) and incidents.csv. Work from the data, not the summaries people have given you. Write your recommendation for the meeting: go, no-go or go with conditions, region by region, and why. Include a table that checks each launch criterion for each region with the figures. Keep the prose under 900 words.

What the model was given6 items: The test, Launch criteria (agreed in the PRD), What people have said, Engineering, daily_metrics.csv, incidents.csv

The test1–28 October, five regions. In each region orders were split between control (the picker chooses substitutes, as today) and treatment (AI auto-substitutions). The split was not 50/50 everywhere: treatment got 30% of orders in London, 50% in the South East and North West, and 70% in the Midlands and Scotland.

Launch criteria (agreed in the PRD)1. Substitution acceptance (subs_accepted ÷ subs_offered) of at least 80% in treatment, in every region. 2. Refund requests per 1,000 orders in treatment no more than 10% above control, in every region. 3. Zero dietary or allergen mismatches (a substitute that breaks a gluten-free, vegan, vegetarian, nut-free, halal or kosher attribute). 4. Average basket value not lower in treatment than control.

What people have saidHead of Commercial: “Acceptance is up 15 points and refunds are within the guardrail overall. Every day we wait costs us Christmas.” Head of Analytics: “Treatment baskets are £4 smaller across the test. That worries me.” Ops director (Scotland): “The supplier change caused some noise, but it's settling down.”

EngineeringThe dietary-attribute hard filter (never substitute across those attributes) is merged and scheduled to deploy on 4 November after regression tests. Scotland's catalogue re-map for the new dairy supplier is in progress with no date. Rollout can be switched on region by region.

daily_metrics.csv283 rows · Download

date,region,arm,orders,items_ordered,items_out_of_stock,subs_offered,subs_accepted,refund_requests,refund_value_gbp,complaints,avg_basket_gbp
2026-10-01,London,control,1757,52273,2351,1953,1387,32,160.71,6,72.89
2026-10-01,London,treatment,692,20455,871,820,708,12,62.05,2,72.42
2026-10-01,South East,control,853,22846,996,833,593,16,74.49,3,65.62
2026-10-01,South East,treatment,928,25223,1166,1106,952,15,75.93,3,66.59
2026-10-01,Midlands,control,467,10468,465,383,268,8,34.57,1,54.92
2026-10-01,Midlands,treatment,1113,25136,1166,1087,947,21,106.35,3,55.33
…

incidents.csv13 rows · Download

date,region,arm,type,severity,summary,status
2026-10-02,London,treatment,wrong_size,low,"4-pint milk substituted with 1-pint, customer accepted at door",closed
2026-10-04,Midlands,treatment,dietary_mismatch,high,"Gluten-free sliced loaf substituted with standard white loaf; customer coeliac, noticed at home",closed: refund and apology
2026-10-06,South East,control,wrong_item,low,"Picker substituted oat milk with soya milk; customer rejected at door",closed
2026-10-07,North West,treatment,data_export,low,"Daily export job re-ran after a timeout on 8, 9 and 10 Oct; analytics team flagged possible duplicate rows",open
2026-10-09,London,treatment,price,medium,"Substitute priced higher than original; customer charged the difference against policy",closed: policy fix deployed 11 Oct
2026-10-12,Scotland,control,late_delivery,low,"Van breakdown, 14 orders late",closed
…

What a strong answer doesThe answer key the graders mark against

No-go for a national launch on 2 November, because criterion 3 fails everywhere until the dietary hard filter ships: incidents.csv has three treatment dietary mismatches (4 Oct Midlands, 19 Oct South East, 23 Oct London, the last involving nuts and a child with an allergy). Go with conditions, region by region, once the filter is deployed and verified (planned 4 November, still before the 9 November booking window): London, South East, Midlands and North West pass the other criteria (acceptance about 86% against 71% control; refunds 2–5% below control). Hold Scotland: over the month its treatment refunds are about 66% above control, and from the 15 October supplier change about 143% above (43 against 18 per 1,000 orders), with acceptance falling to about 79%, below the 80% bar; it waits for the catalogue re-map and a clean re-test. The pooled refund rate (about 8% above control) hides this, and the criterion is per region. The pooled basket gap (about £65 against £61) is an artefact of the uneven split, since treatment baskets are equal or higher in every region (Simpson's paradox), so criterion 4 passes. Notices that North West treatment rows for 8–10 October are duplicated (an extra 2,194 orders), removes them, and says it doesn't change the result. States what would change the call: the filter slipping past 9 November, or Scotland's refunds not recovering.

Critical failures (cap the score at 40)

Recommends any rollout before the dietary hard filter is deployed
Recommends rolling out to Scotland while its refund guardrail is breached

Task

Staff-level · v1.12

The outputs

8 setups answered this brief.

Sonnet 5.5API95Usable with a quick edit GPT-6.1 SolAPI93Usable with a quick edit GPT-6 AstraChatGPT90Usable with a quick edit Haiku 5.5API90Usable with a quick edit Opus 5.5Claude88Usable with a quick edit GPT-6 LunaAPI78Usable with a quick edit Gemini 3.5 Flash-LiteGemini52Needs more work Gemini 3.8 FlashAPI40Critical failure

Haiku 5.5API

Usable with a quick edit12 of 14 checks passed

The output gives a complete, evidence‑based recommendation with a criteria table, region‑specific calls, and clear conditions. A PM could use it with minor edits without re‑checking the data or reasoning.

Output

Launch recommendation: AI auto-substitutions, 2 November

Recommendation: do not launch to all regions on 2 November. Switch on North West once the dietary hard filter is confirmed live (target 5 November). Hold London, South East and Midlands, and re-decide on 6 November. Keep Scotland on picker substitutes.

Criterion 3 is a hard PRD gate, and the test recorded three high-severity dietary breaks in treatment. The hard filter does not deploy until 4 November, so a 2 November launch would go live without the control that would have prevented them.

Data notes

Duplicates removed. North West treatment rows for 8, 9 and 10 October are duplicated (the export re-ran). I kept one copy of each. Including them barely moves the result (acceptance 86.2%, refunds 17.5 per 1,000), but the incident is still open and the export should be fixed.
Split is uneven, so I used rates. Acceptance, refunds per 1,000 orders and basket are computed per region and arm, not from totals.
Basket: the £4 gap isn't in the file. Daily average basket is higher in treatment in every region (+£0.24 to +£0.46 per order on simple averages). A £4 gap would need a different definition, such as totals, which differ because arm volumes differ. Analytics should share its query before anyone quotes the figure. I haven't checked order-weighted averages, but I don't expect them to reverse the sign.
Pooled figures hide the problem. Pooled, treatment refunds are 19.5 per 1,000 against 18.1 for control (+8%), which passes. That is only because four regions are under control and Scotland is far over it.

Criteria check (treatment vs control)

Region	1. Acceptance (≥80%)	2. Refunds per 1,000 (≤10% above control)	3. Dietary breaks (zero)	4. Basket (not lower)	Call
London	86.7% vs 71.4% ✔	17.4 vs 18.0 (−3%) ✔	1 high: nut-free → hazelnut, 23 Oct, open ✘	£72.72 vs £72.48 ✔	No-go
South East	86.4% vs 71.0% ✔	17.5 vs 17.9 (−2%) ✔	1 high: vegan → dairy cheddar, 19 Oct ✘	£66.66 vs £66.20 ✔	No-go
Midlands	86.3% vs 71.1% ✔	17.5 vs 17.9 (−2%) ✔	1 high: gluten-free → standard loaf, 4 Oct ✘	£55.66 vs £55.35 ✔	No-go
North West	86.3% vs 71.2% ✔	17.6 vs 18.5 (−5%) ✔	0 logged ✔	£57.52 vs £57.28 ✔	Go with conditions
Scotland	82.8% overall ✔; 79.1% since 15 Oct ✘	30.2 vs 18.2 (+66%) ✘	0 logged ✔	£52.98 vs £52.62 ✔	No-go

Region by region

North West: go with conditions. It is the only region that passes all four criteria on clean data. Its zero is partly luck, though. The three dietary breaks show the model crosses attribute boundaries with no hard filter, so the filter is a precondition here as well. Conditions: 1. The hard filter is deployed and regression-tested (4 November), and North West's catalogue is checked against it. 2. The duplicate-row export issue is closed and the North West figures are re-run on the fixed export. 3. Switch on around 5 November with the kill switch ready. Review refunds and dietary complaints at one week.

London, South East, Midlands: no-go for 2 November. Acceptance and refunds are fine, and basket is up. But each region has a high-severity dietary break in treatment, and the nut-free case (London, 23 October) is still open, with a child with a nut allergy involved. Before we re-decide on 6 November, we need: 1. The hard filter live and regression-tested. 2. A retrospective audit of every treatment substitution in these regions against product attributes. The logged incidents are only the ones customers reported. 3. Root cause for the London nut-free break and confirmation the price-policy fix (11 October) holds. London's basket gain may partly reflect higher-priced substitutes, which is the issue that prompted that fix. 4. A short ramp (for example 10% of treatment orders) from 4–6 November, with a zero-mismatch check, before any wider switch-on. If the audit finds further breaks, these regions stay off.

Scotland: no-go. The ops director's "settling down" is not what the data shows. Before the supplier change on 15 October, treatment acceptance was 86.3% and refunds were 17.2 per 1,000, the same as control (17.3). Since 15 October, acceptance has been flat at about 79% (79.0% in week two, 79.2% in week three) and refunds at 43 per 1,000 against 19 for control. Refund value since 15 October is about £2.8k. Control's refund rate barely moved, so the cause is the treatment engine running on the new catalogue: the open wrong-size dairy incident matches the mapping error described in the log. The re-map has no date. - Keep Scotland at 0% treatment. - Re-enter only when the re-map is validated, and treatment acceptance is at least 80% and refunds within 10% of control over two weeks at a small share.

Asks for the meeting

Agree that no region goes live on 2 November, and that criterion 3 is not waived.
Approve North West for go-on-filter (target 5 November), subject to the conditions above.
Set the 6 November gate for London, South East and Midlands, with the audit and ramp as entry conditions.
Keep Scotland on control. Commercial should not plan Scotland into Christmas slots until the re-map is validated.
Analytics to reconcile the £4 basket claim before it goes into any further material.

The Christmas booking window opens on 9 November. The North West switch-on and the 6 November gate both land before then, so the main cost of this recommendation is that London, South East and Midlands may not have a validated launch for the first booking week.

Grades and run details

Decision model 79 · LLM judge 100

Decision model checks

failUses the supplied evidence correctly25%
passAddresses the actual decision72%
passRespects explicit constraints23%
passIdentifies material uncertainty85%
partialAvoids unsupported claims38%
passProduces the required deliverable61%
passChecks every criterion, region by region34%
passFinds the Scotland breach100%
passExplains the basket gap70%
passCatches the duplicate rows100%
passMakes a clear call85%
partialLimits the downside of being wrong95%
failGets the base of every number right48%
passJudges the launch against a baseline set now92%

Run

Run: #1
API response time: 1.9 min
Submitted: 8 Oct 2026

Sonnet 5.5API

Usable with a quick edit13 of 14 checks passed

The output is complete, well-structured, and directly usable by a PM with at most light editing for tone or emphasis.

Output

Recommendation: AI auto-substitutions, launch meeting 30 October

Decision: no-go for Monday 2 November. Go with conditions from 4 November for London, South East, Midlands and North West. Scotland stays on picker substitutions until it re-passes.

Criteria by region

Figures are recomputed from the daily rows. Treatment (T) and control (C) are pooled over the 28 days.

Region	1. Acceptance (T) ≥80%	2. Refunds per 1,000 orders, T vs C (≤ +10%)	3. Dietary mismatches (0)	4. Basket T vs C (not lower)	Verdict
London	86.7% ✅	17.4 vs 18.0 (−3%) ✅	1 (nut allergen, 23 Oct, open) ❌	£72.72 vs £72.48 (+£0.24) ✅	Conditional
South East	86.4% ✅	17.5 vs 17.9 (−2%) ✅	1 (vegan cheese, 19 Oct) ❌	£66.66 vs £66.20 (+£0.46) ✅	Conditional
Midlands	86.3% ✅	17.5 vs 17.9 (−2%) ✅	1 (gluten-free, 4 Oct) ❌	£55.66 vs £55.35 (+£0.31) ✅	Conditional
North West	86.3% ✅	17.6 vs 18.5 (−5%) ✅	0 logged ✅	£57.52 vs £57.28 (+£0.24) ✅	Conditional
Scotland	82.8% overall, 79.1% since 15 Oct ❌	30.2 vs 18.2 (+66%); since 15 Oct 43.3 vs 17.8 (+143%) ❌	0 logged ✅	£52.98 vs £52.62 (+£0.36) ✅	No-go

Baskets are means of daily averages. The differences are small and I have not tested them for significance, but the point estimate is positive everywhere.

Checking what people have said

Head of Commercial. The pooled numbers are right: acceptance is 85.9% against 71.2% (+14.7 points), and refunds are 19.5 against 18.1 per 1,000 (+8%). But the pooled refund figure hides Scotland, which is failing badly. Four regions are 2–5% better than control on refunds.
Head of Analytics. The £4 gap is real in the pooled data (£61.10 vs £65.15) but it is a mix effect, not a treatment effect. London, with the highest baskets (about £72), was 42% of control orders but only 18% of treatment orders. The Midlands and Scotland (about £53–56) were 18% of control but 42% of treatment. Within every region, treatment baskets are £0.24–£0.46 higher. Criterion 4 passes. The uneven split did not bias the regional comparisons.
Ops director (Scotland). The data does not support "settling down". Before the supplier switch on 15 October, Scotland treatment was healthy (86.3% acceptance, 17.3 refunds per 1,000). From 15 October to 28 October, acceptance has been flat at 78–81%, and daily refunds have been 25–33 against a pre-switch level of 10–14. The cause is the open incident: the new supplier's pack sizes are mapped to the wrong product family, so the model makes wrong-size dairy substitutions. That is a catalogue defect with no fix date, not noise. Scotland control is unaffected.

Why the date moves

Criterion 3 fails in three regions. The three mismatches include a nut-allergy case and a coeliac case. The hard filter that prevents them deploys on 4 November. Until it is live, every region has the same exposure. North West and Scotland show zero only because none were reported, and the incidents come from customer complaints, so the true rate is unknown. I would not switch on a known allergen risk two days before the fix.

Waiting is cheap. The booking window opens on 9 November, not 2 November. Deploying the filter on 4 November and switching regions on 4–5 November leaves several days before the window. Only Scotland has no route to launch before then.

Conditions for the four regions

The filter deploys on 4 November and passes regression tests. Before any region goes live, it is tested in production against the three incident SKUs (gluten-free loaf, vegan cheese, nut-free bar) and the other attributes: vegetarian, halal and kosher. The London incident is still open with the filter "in review", so confirm the filter would have blocked it.
Regions are switched on region by region, North West and South East first, then London and the Midlands. This gives a day of live data before the larger volumes go on.
Any dietary or allergen mismatch after launch switches that region back to picker substitution immediately, pending review. That includes any mismatch in the first week.
Daily monitoring of acceptance (80% floor) and refunds (10% guardrail) per region against a small held-back control. Delegate the switch-on decision so it does not need another meeting.

Scotland

Keep Scotland on picker substitutions. Retest only after the catalogue re-map ships, on a small treatment share. It should show at least three consecutive days at 80% or better acceptance, and refunds within 10% of control, before it goes to full volume. Engineering has no date, so Scotland should not be in the Christmas plan and should be treated as upside.

Data notes

North West treatment has three duplicated rows (8, 9 and 10 October), matching the open export-job incident. I removed them. This restores the intended 50% split, and the North West rates barely move either way.
Incident logs are a floor on dietary failures, not a measure of them.
This is 28 days of data in a low-volume region (Scotland control is about 265 orders a day). Even so, the Scotland gap is far outside noise.

Grades and run details

Decision model 89 · LLM judge 100

Decision model checks

passUses the supplied evidence correctly10%
passAddresses the actual decision98%
passRespects explicit constraints37%
passIdentifies material uncertainty91%
partialAvoids unsupported claims31%
passProduces the required deliverable84%
passChecks every criterion, region by region54%
passFinds the Scotland breach100%
passExplains the basket gap99%
passCatches the duplicate rows100%
passMakes a clear call85%
passLimits the downside of being wrong69%
failGets the base of every number right28%
passJudges the launch against a baseline set now89%

Run

Run: #1
API response time: 2.3 min
Submitted: 29 Sept 2026

Gemini 3.8 FlashAPI

Needs more than a quick edit9 of 14 checks passed

The output contains a critical failure (rollout before dietary filter) and factual errors in the duplicate count and Scotland acceptance, requiring rework before use.

Output

Recommendation Summary: Go With Conditions

Do not execute a blanket rollout on Monday, 2 November.

North West: GO (2 November). Meets all four PRD launch criteria cleanly.
London, South East, Midlands: GO WITH CONDITIONS (Deploy Thursday, 5 November). Commercial metrics are strong, but all three regions breached Criterion 3 with severe dietary/allergen substitutions. Launch must be gated on the successful deployment of the dietary-attribute hard filter on 4 November.
Scotland: NO-GO. Fails Criterion 2 (+67.4% refund spike vs. control) and Criterion 1 post-supplier transition (76.6% acceptance). Revert 100% to control until catalogue re-mapping is completed and re-tested.

This phased approach protects customer safety and brand reputation while capturing AI substitution value ahead of Christmas slot bookings opening on 9 November.

---

PRD Launch Criteria Assessment

Note: North West treatment metrics reflect deduplication of export retries on 8–10 October (3,194 duplicate rows removed).

Region	Arm	Orders	1. Subs Acceptance (Target: $\ge$80%)	2. Refunds / 1k Orders (Target: $\le$+10% vs Ctrl)	3. Allergen / Dietary Mismatches (Target: 0)	4. Avg Basket Value (Target: $\ge$ Ctrl)	Status
:---	:---	:---	:---	:---	:---	:---	:---
London	Control<br>Treatment	49,159<br>21,054	71.5%<br>87.4%	18.06<br>17.53 (-2.9%)	0<br>1 (FAIL)	£72.52<br>£72.68 (+£0.16)	CONDITIONAL<br>Hold for filter
South East	Control<br>Treatment	26,485<br>26,919	71.7%<br>86.5%	17.86<br>17.35 (-2.9%)	0<br>1 (FAIL)	£66.27<br>£66.57 (+£0.30)	CONDITIONAL<br>Hold for filter
Midlands	Control<br>Treatment	13,048<br>31,174	71.3%<br>85.2%	17.86<br>17.58 (-1.6%)	0<br>1 (FAIL)	£55.37<br>£55.62 (+£0.25)	CONDITIONAL<br>Hold for filter
North West	Control<br>Treatment	20,702<br>20,774	70.8%<br>85.8%	18.26<br>17.47 (-4.3%)	0<br>0 (PASS)	£57.29<br>£57.53 (+£0.24)	GO<br>Launch 2 Nov
Scotland	Control<br>Treatment	7,853<br>18,371	71.3%<br>81.5% (76.6% post-15 Oct)	18.21<br>30.48 (+67.4% - FAIL)	0<br>0 (PASS)	£52.62<br>£52.89 (+£0.27)	NO-GO<br>Catalogue fault

---

Analysis & Stakeholder Debunking

#### 1. Head of Analytics: “Treatment baskets are £4 smaller across the test.” Finding: False alarm caused by Simpson’s Paradox. Across all pooled orders, treatment average basket appears lower (£61.06 treatment vs. £65.18 control, a £4.12 deficit). However, this is an aggregation distortion caused by the uneven regional split: London (average basket ~£73) received only a 30% treatment allocation, whereas lower-basket regions (Midlands at ~£55 and Scotland at ~£53) received a 70% treatment allocation.

When evaluated within every single region, treatment basket value was higher than control by £0.16 to £0.30. Criterion 4 is fully satisfied across all regions.

#### 2. Head of Commercial: “Refunds are within the guardrail overall. Every day we wait costs Christmas.” Finding: Misleading pooled metric masking a severe regional failure. While aggregate refunds appear within limits (+8.1% overall), the PRD explicitly requires compliance in every region. Scotland completely breaches this guardrail (+67.4% above control). Launching everywhere on 2 November would risk widespread customer attrition in Scotland right before holiday booking.

#### 3. Ops Director (Scotland): “The supplier change caused noise, but it's settling down.” Finding: Factually incorrect; Scotland is failing and persistent. Following the 15 October dairy supplier change, wrong-size substitutions (e.g., pack sizes mapped to incorrect product families) caused: * Daily refunds in treatment to triple from 11.6/day (Oct 1–14) to 28.4/day (Oct 15–28), reaching 43.7 refunds per 1,000 orders (+140% vs. control). * Treatment acceptance dropped from 86.3% down to 76.6% (failing Criterion 1). * Daily refund value increased 4x to £180–£247/day. Performance has not stabilized. Because the catalogue re-map is incomplete with no ETA, Scotland cannot launch.

#### 4. The Dietary Hazard (London, South East, Midlands) Treatment produced three critical safety failures: * 4 Oct (Midlands): Coeliac customer received standard bread for gluten-free. * 19 Oct (South East): Vegan customer received dairy cheddar. * 23 Oct (London): Child with nut allergies received hazelnut-containing bars for nut-free bars.

Launching auto-substitutions in these regions before deploying technical guardrails creates intolerable liability.

---

Implementation Plan

1. Monday, 2 November: * Roll out North West to 100% treatment. (Clean test data, no allergen incidents, +15pp acceptance lift). * Revert Scotland treatment to 0% (100% control). Keep human pickers selecting substitutes until the catalogue re-map is verified and back-tested. * Maintain London, South East, and Midlands at current test splits for 48 hours.

2. Wednesday, 4 November: * Engineering deploys the dietary-attribute hard filter to production following scheduled regression testing.

3. Thursday, 5 November: * Verify filter telemetry on live orders. * Roll out London, South East, and Midlands to 100% treatment.

4. Monday, 9 November: * Christmas delivery-slot booking opens on schedule with 4 of 5 regions operating under fully automated, safety-enforced substitutions, leaving human pickers dedicated to Scotland until rectified.

Claims the judge couldn’t find in the brief

North West treatment metrics reflect deduplication of export retries on 8–10 October (3,194 duplicate rows removed).
Scotland fails Criterion 2 (+67.4% refund spike vs. control) and Criterion 1 post-supplier transition (76.6% acceptance).
Treatment acceptance dropped from 86.3% down to 76.6% (failing Criterion 1).
Daily refund value increased 4x to £180–£247/day.

Grades and run details

Decision model 57 · LLM judge 60

Decision model checks

failUses the supplied evidence correctly63%
passAddresses the actual decision51%
partialRespects explicit constraints24%
partialIdentifies material uncertainty51%
failAvoids unsupported claims34%
passProduces the required deliverable62%
partialChecks every criterion, region by region45%
passFinds the Scotland breach99%
passExplains the basket gap99%
partialCatches the duplicate rows83%
passMakes a clear call40%
failLimits the downside of being wrong43%
failGets the base of every number right67%
passJudges the launch against a baseline set now66%

Run

Run: #1
API response time: 85 s
Submitted: 1 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 5

Uses the supplied evidence correctlyMixedRightWrong

Haiku 5.5 · API

Every factual statement about the current situation is drawn from the brief or the attached files, with no invented facts.

Sonnet 5.5 · API

All factual claims about the current situation are taken directly from the supplied data or computed correctly from it; no invented facts.

Gemini 3.8 Flash · API

The output states 3,194 duplicate rows were removed, but the correct number is 2,194; it also claims Scotland's post-15 Oct acceptance is 76.6% when it is 79.1%.

Avoids unsupported claimsRightRightWrong

Haiku 5.5 · API

Speculation about causes is labelled with 'may' and 'partly', and assertions are backed by the data.

Sonnet 5.5 · API

Causal claims are supported by incident logs; interpretations are clearly analytical, not presented as established fact.

Gemini 3.8 Flash · API

Presents the 76.6% acceptance figure and the 4x refund value increase as fact when the data does not support those exact numbers.

Limits the downside of being wrongRightRightWrong

Haiku 5.5 · API

Describes kill switch, ramp with zero‑mismatch check, and review of refunds/complaints at one week, limiting downside if things go wrong.

Sonnet 5.5 · API

Specifies post-launch monitoring (acceptance, refunds, dietary mismatches), thresholds (80%, 10%, any mismatch), and immediate rollback action.

Gemini 3.8 Flash · API

Does not name a post-launch signal with a threshold that would pause or roll back, nor flag any irreversible actions.

Gets the base of every number rightMixedMixedWrong

Haiku 5.5 · API

All percentages and rates (acceptance, refunds per 1 000) are computed from the correct totals, and basket figures are checked within region.

Sonnet 5.5 · API

All derived figures (acceptance, refunds per 1,000, basket comparisons) are computed correctly from the supplied data with appropriate denominators.

Gemini 3.8 Flash · API

The duplicate row count is wrong (3,194 vs 2,194) and the Scotland post-15 Oct acceptance is misstated (76.6% vs 79.1%).

Judges the launch against a baseline set nowRightRightMixed

Haiku 5.5 · API

The launch criteria themselves serve as the baseline, and specific review points (5 Nov for NW, 6 Nov for others) with conditions are set.

Sonnet 5.5 · API

Names success metrics (acceptance ≥80%, refunds ≤+10%, zero dietary mismatches), sets baseline from test data, and defines daily monitoring with triggers for action.

Gemini 3.8 Flash · API

Does not name a success metric with its current baseline or a review point with what each result would lead to.

All got right 9

Addresses the actual decisionRightRightRight

Haiku 5.5 · API

Commits to a clear, early call (no‑go for all on 2 Nov, conditional go for NW, hold others, no‑go for Scotland) and states what would change it.

Sonnet 5.5 · API

Commits to a clear no-go for 2 Nov, conditional go from 4 Nov for four regions, and no-go for Scotland, with conditions that would change it.

Gemini 3.8 Flash · API

Commits to a clear region-by-region call with conditions and says what would change it (filter deployment, Scotland refunds recovering).

Respects explicit constraintsRightRightRight

Haiku 5.5 · API

Respects length (under 900 words), includes the required region‑by‑region criteria table, and addresses the launch meeting audience.

Sonnet 5.5 · API

Respects the brief: region-by-region recommendation, criteria table, prose under 900 words, addressed to the launch meeting.

Gemini 3.8 Flash · API

Includes a region-by-criterion table with figures, prose under 900 words, and addresses the launch meeting.

Identifies material uncertaintyRightRightRight

Haiku 5.5 · API

Identifies unknowns (duplicate export, audit may find further breaks, catalogue re‑map date, basket metric definition) and says how they would be resolved.

Sonnet 5.5 · API

Identifies unknowns (filter deployment date, Scotland catalogue re-map date, true dietary mismatch rate) and says what would change the call.

Gemini 3.8 Flash · API

Names the filter slipping past 9 November and Scotland's refunds not recovering as unknowns that would change the call.

Produces the required deliverableRightRightRight

Haiku 5.5 · API

Delivers a complete recommendation with a criteria table, region‑by‑region calls, conditions, and actions for the meeting.

Sonnet 5.5 · API

Delivers a complete recommendation with table, region-by-region verdict, conditions, and data notes, usable as-is.

Gemini 3.8 Flash · API

Provides a complete recommendation with table and implementation plan, usable for the meeting with light edits.

Checks every criterion, region by regionRightRightRight

Haiku 5.5 · API

The table checks all four criteria for each region with figures computed from the files (acceptance, refunds per 1,000, dietary incidents, basket).

Sonnet 5.5 · API

Table checks all four criteria for each region with figures computed from the files.

Gemini 3.8 Flash · API

Table checks all four criteria for each region with figures computed from the data.

Finds the Scotland breachRightRightRight

Haiku 5.5 · API

Finds Scotland's refund breach (30.2 vs 18.2, +66%), ties it to the supplier change, notes acceptance dip to 79%, and holds Scotland.

Sonnet 5.5 · API

Finds Scotland's refund breach (+66% overall, +143% since 15 Oct), ties it to the supplier change, notes acceptance dip, and holds Scotland.

Gemini 3.8 Flash · API

Identifies Scotland's refund breach, ties it to the supplier change, notes acceptance dip, and holds Scotland.

Explains the basket gapRightRightRight

Haiku 5.5 · API

Shows that within each region treatment baskets equal or exceed control, explains the pooled £4 gap as an artefact of the uneven split.

Sonnet 5.5 · API

Shows treatment baskets are higher in every region and explains the pooled gap as a mix effect from the uneven split.

Gemini 3.8 Flash · API

Explains the pooled basket gap as a Simpson's paradox artefact and shows treatment baskets are higher within each region.

Catches the duplicate rowsRightRightRight

Haiku 5.5 · API

Notices the duplicated North‑West rows for 8–10 October, removes them, and states that including them barely changes the result.

Sonnet 5.5 · API

Catches the duplicated North West treatment rows for 8–10 Oct, removes them, and states they barely change the rates.

Gemini 3.8 Flash · API

Notices the duplicate North West rows, removes them, and states the metrics reflect deduplication; the exact count error is a separate factual mistake.

Makes a clear callRightRightRight

Haiku 5.5 · API

Makes one clear call per region with dates (NW go 5 Nov, L/SE/Mid re‑decide 6 Nov, Scotland no‑go) and states what would change each call.

Sonnet 5.5 · API

Makes a clear call per region with dates and conditions, and says what would change it (filter deploy, Scotland re-test).

Gemini 3.8 Flash · API

Gives a clear decision per region with dates and conditions, and states what would change it.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 63% of 68 checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	Sonnet 5.5withAPI	87.8	100.0	2	None
2	Haiku 5.5withAPI	82.5	100.0	2	None
3	GPT-6.1 SolwithAPI	86.0	95.8	2	None
4	Opus 5.5withClaude	84.3	96.7	2	None
5	GPT-6 AstrawithChatGPT	83.8	92.5	2	None
6	GPT-6 LunawithAPI	74.8	80.8	2	None
7	Gemini 3.8 FlashwithAPI	64.9	67.5	2	1 capped
8	Gemini 3.5 Flash-LitewithGemini	48.2	59.2	2	None

What you get for the cost

Haiku 5.5 · API scores within 2 points of Sonnet 5.5 · API at about a tenth of the cost.

Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)At least one critical failure

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
GPT-6 Luna · API	77	1.0×
Gemini 3.5 Flash-Lite · Gemini	52	2.7× (partly estimated)
Haiku 5.5 · API	92	2.8×
Gemini 3.8 Flash · API	55	11×
GPT-6.1 Sol · API	91	12×
Sonnet 5.5 · API	94	37×
GPT-6 Astra · ChatGPT	89	46× (partly estimated)
Opus 5.5 · Claude	91	59× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Deciding whether a feature ships on the planned date, and on what conditions.

Why it matters

Launch meetings reward optimism. A good call checks each agreed criterion, weighs a real risk against a real gain, and says exactly what would change the answer. A weak one rubber-stamps the launch or blocks it on noise.

What good looks like

Checks each agreed criterion against the evidence
Makes one clear call, with conditions if needed
Separates risks that block launch from risks that can be managed
Says what would change the call
Says how to limit the damage if the call is wrong

Deliberately not measured

Rollout engineering detail
Project-plan formatting

Capability tested

Deciding against agreed launch criteria

The failure we’re looking for

Rubber-stamps a launch that misses an agreed bar, or blocks it on noise

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

A launch memo from supplied evidence · Staff level: a regional call from an attached data workbook

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

Limits the downside of being wrong

Judges the launch against a baseline set now

Plus our standard checks

The tasks

The outputs

Output

Launch recommendation: AI auto-substitutions, 2 November

Data notes

Criteria check (treatment vs control)

Region by region

Asks for the meeting

Grades and run details

Decision model checks

Run

Output

Recommendation: AI auto-substitutions, launch meeting 30 October

Criteria by region

Checking what people have said

Why the date moves

Conditions for the four regions

Scotland

Data notes

Grades and run details

Decision model checks

Run

Output

Recommendation Summary: Go With Conditions

PRD Launch Criteria Assessment

Analysis & Stakeholder Debunking

Implementation Plan

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Run

Check by check

Where they differ 5

All got right 9

Results

What you get for the cost

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured