Tasks / Product Craft

Make the launch call

Can the model make a clear go / no-go call from mixed launch evidence, checked against the criteria agreed up front?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 67% were usable with at most a quick edit.

Reliably right

Produces the required deliverable100% pass
Provides a complete recommendation with table, reasoning, email plan, and next steps, usable by a product manager with light edits.
Sonnet 5.5 · API · Offline, on older phones
Finds the Scotland breach100% pass
Finds Scotland's refund breach, ties it to the supplier change, notes the acceptance dip, and holds Scotland.
GPT-6.1 Sol · API · Go/no-go for AI substitutions, from the rollout data
Treats the overwrites as data loss100% pass
Treats the Android 11 overwrites as a breach of the no-data-loss criterion and makes a fix a condition for that platform.
Sonnet 5.5 · API · Offline, on older phones

Where it slips

Judges the launch against a baseline set now51% pass
Does not name a specific review point or when the team will review results to decide next steps; only gives a pause trigger without a scheduled evaluation.
GPT-6.1 Sol · API · Offline, on older phones
Checks every criterion for every platform64% pass
Android 13's error-free sync is 98.9%, below the 99% threshold, yet the table marks it as 'GO WITH CONDITIONS' instead of 'NO-GO' for that criterion, misapplying the hard launch criteria.
Gemini 3.5 Flash-Lite · Gemini · Offline, on older phones
Gets the base of every number right67% pass
The 80.5% user base proportion is miscalculated; correct proportion from the data is 79.5%.
Gemini 4 Argon · API · Offline, on older phones

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Limits the downside of being wrong
Does the call say how exposure is limited if it's wrong: what is watched after launch, the threshold that would pause or roll back, and which parts can't be undone?
Passes when Names the post-launch signal, a threshold or trigger, and the action, and flags anything irreversible (customer communications, pricing, migrations).
Karri SaarinenLenny’s Podcast
Judges the launch against a baseline set now
Does the call name the metric and the current baseline the launch will be judged against, and when the team will review the result to decide what's next?
Passes when Names the success metric, its baseline from the material, and a review point with what each result would lead to.
Todd JacksonLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 5 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Staff PM for fulfilment at Basketful. On 30 October the launch meeting decides whether AI auto-substitutions (the model picks a replacement when an item is out of stock, instead of the picker) roll out to every region on Monday 2 November, ahead of the Christmas delivery-slot booking window that opens on 9 November. The rollout workbook is attached as two files: daily_metrics.csv (28 days of the regional test, by region and arm) and incidents.csv. Work from the data, not the summaries people have given you. Write your recommendation for the meeting: go, no-go or go with conditions, region by region, and why. Include a table that checks each launch criterion for each region with the figures. Keep the prose under 900 words.

What the model was given6 items: The test, Launch criteria (agreed in the PRD), What people have said, Engineering, daily_metrics.csv, incidents.csv

The test1–28 October, five regions. In each region orders were split between control (the picker chooses substitutes, as today) and treatment (AI auto-substitutions). The split was not 50/50 everywhere: treatment got 30% of orders in London, 50% in the South East and North West, and 70% in the Midlands and Scotland.

Launch criteria (agreed in the PRD)1. Substitution acceptance (subs_accepted ÷ subs_offered) of at least 80% in treatment, in every region. 2. Refund requests per 1,000 orders in treatment no more than 10% above control, in every region. 3. Zero dietary or allergen mismatches (a substitute that breaks a gluten-free, vegan, vegetarian, nut-free, halal or kosher attribute). 4. Average basket value not lower in treatment than control.

What people have saidHead of Commercial: “Acceptance is up 15 points and refunds are within the guardrail overall. Every day we wait costs us Christmas.” Head of Analytics: “Treatment baskets are £4 smaller across the test. That worries me.” Ops director (Scotland): “The supplier change caused some noise, but it's settling down.”

EngineeringThe dietary-attribute hard filter (never substitute across those attributes) is merged and scheduled to deploy on 4 November after regression tests. Scotland's catalogue re-map for the new dairy supplier is in progress with no date. Rollout can be switched on region by region.

daily_metrics.csv283 rows · Download

date,region,arm,orders,items_ordered,items_out_of_stock,subs_offered,subs_accepted,refund_requests,refund_value_gbp,complaints,avg_basket_gbp
2026-10-01,London,control,1757,52273,2351,1953,1387,32,160.71,6,72.89
2026-10-01,London,treatment,692,20455,871,820,708,12,62.05,2,72.42
2026-10-01,South East,control,853,22846,996,833,593,16,74.49,3,65.62
2026-10-01,South East,treatment,928,25223,1166,1106,952,15,75.93,3,66.59
2026-10-01,Midlands,control,467,10468,465,383,268,8,34.57,1,54.92
2026-10-01,Midlands,treatment,1113,25136,1166,1087,947,21,106.35,3,55.33
…

incidents.csv13 rows · Download

date,region,arm,type,severity,summary,status
2026-10-02,London,treatment,wrong_size,low,"4-pint milk substituted with 1-pint, customer accepted at door",closed
2026-10-04,Midlands,treatment,dietary_mismatch,high,"Gluten-free sliced loaf substituted with standard white loaf; customer coeliac, noticed at home",closed: refund and apology
2026-10-06,South East,control,wrong_item,low,"Picker substituted oat milk with soya milk; customer rejected at door",closed
2026-10-07,North West,treatment,data_export,low,"Daily export job re-ran after a timeout on 8, 9 and 10 Oct; analytics team flagged possible duplicate rows",open
2026-10-09,London,treatment,price,medium,"Substitute priced higher than original; customer charged the difference against policy",closed: policy fix deployed 11 Oct
2026-10-12,Scotland,control,late_delivery,low,"Van breakdown, 14 orders late",closed
…

What a strong answer doesThe answer key the graders mark against

No-go for a national launch on 2 November, because criterion 3 fails everywhere until the dietary hard filter ships: incidents.csv has three treatment dietary mismatches (4 Oct Midlands, 19 Oct South East, 23 Oct London, the last involving nuts and a child with an allergy). Go with conditions, region by region, once the filter is deployed and verified (planned 4 November, still before the 9 November booking window): London, South East, Midlands and North West pass the other criteria (acceptance about 86% against 71% control; refunds 2–5% below control). Hold Scotland: over the month its treatment refunds are about 66% above control, and from the 15 October supplier change about 143% above (43 against 18 per 1,000 orders), with acceptance falling to about 79%, below the 80% bar; it waits for the catalogue re-map and a clean re-test. The pooled refund rate (about 8% above control) hides this, and the criterion is per region. The pooled basket gap (about £65 against £61) is an artefact of the uneven split, since treatment baskets are equal or higher in every region (Simpson's paradox), so criterion 4 passes. Notices that North West treatment rows for 8–10 October are duplicated (an extra 2,194 orders), removes them, and says it doesn't change the result. States what would change the call: the filter slipping past 9 November, or Scotland's refunds not recovering.

Critical failures (cap the score at 40)

Recommends any rollout before the dietary hard filter is deployed
Recommends rolling out to Scotland while its refund guardrail is breached

Task

Staff-level · v1.13

The outputs

9 setups answered this brief.

GPT-6.1 SolAPI97Usable with a quick edit Sonnet 5.5API95Usable with a quick edit GPT-6 AstraChatGPT93Usable with a quick edit Haiku 5.5API92Usable with a quick edit Opus 5.5Claude88Usable with a quick edit GPT-6 LunaAPI83Usable with a quick edit Gemini 4 ArgonAPI72Needs more work Gemini 3.5 Flash-LiteGemini52Needs more work Gemini 3.8 FlashAPI40Critical failure

GPT-6 LunaAPI

Usable with a quick edit11 of 14 checks passed

The core recommendation, evidence, and region-by-region logic are solid. A PM could add a monitoring threshold and review date with a quick edit without redoing any analysis.

Output

Recommendation: No-go for the all-region rollout on 2 November

Do not enable auto-substitutions anywhere on 2 November: the dietary hard filter is not scheduled to deploy until 4 November, so the zero-mismatch safety criterion is not yet protected. After deployment and validation, North West is the only region I’d consider enabling first, with conditions. Hold the other regions for the reasons below.

Figures are for 1–28 October; refund rates are per 1,000 orders. Basket averages are order-weighted within each region. I excluded the three exact duplicate North West treatment rows on 8, 9 and 10 October.

Region	Treatment acceptance (≥80%)	Refunds / 1,000 orders: treatment vs control (treatment ≤10% higher)	Dietary/allergen mismatches (must be zero)	Average basket: treatment vs control (must not be lower)	Decision
---	---:	---:	---	---:	---
London	~86% Pass	~17.9 vs ~18.4; Pass	1; Fail — nut-containing bar substituted for nut-free item	~£72.8 vs ~£72.5; Pass	No-go
South East	~85% Pass	~17.8 vs ~17.9; Pass	1; Fail — dairy cheese substituted for vegan cheese	~£66.6 vs ~£66.3; Pass	No-go
Midlands	~86% Pass	~17.6 vs ~18.0; Pass	1; Fail — standard loaf substituted for gluten-free loaf	~£55.6 vs ~£55.4; Pass	No-go
North West	~86% Pass	~17.8 vs ~18.8; Pass	0 observed; Pass in test	~£57.5 vs ~£57.3; Pass	Go with conditions, after filter validation
Scotland	~79% Fail¹	~29.8 vs ~18.4; Fail — about 62% higher	0 observed; Pass in test	~£53.0 vs ~£52.6; Pass	No-go

Conditions and rationale

North West: After the hard filter deploys, pass regression tests across all six protected attributes (gluten-free, vegan, vegetarian, nut-free, halal and kosher); then enable regionally with close monitoring and a rollback switch. The export duplicates should also be fixed or guarded against before relying on daily reporting.
London, South East and Midlands: Do not launch until the filter is deployed and the dietary-mismatch failures are investigated and validated as fixed. Their good acceptance, refund and basket figures do not offset a safety-criterion failure.
Scotland: Hold until the supplier catalogue re-map is complete and verified, and a fresh test meets both acceptance and refund guardrails. The sharp rise in treatment refunds and lower acceptance coincide with the dairy supplier/catalogue change; the issue is open, with no completion date.

The pooled “£4 smaller” basket claim should not drive this decision: treatment exposure differed by region, and baskets vary substantially by region. Within each region, the treatment average is slightly higher in these data. Likewise, an overall acceptance or refund result cannot override region-level launch gates. The 9 November booking window is important, but it does not justify launching before the safety control is live.

What a PM had to fix

1Invented evidenceVerify or remove the claimTargeted repair
What we’d change82.76%: Pass over 1–28 October. However, acceptance fell to 79.08% over 15–28 October after the supplier change, below the launch threshold.

Grades and run details

Decision model 79 · LLM judge 87

Decision model checks

passUses the supplied evidence correctly10%
passAddresses the actual decision46%
passRespects explicit constraints64%
passIdentifies material uncertainty76%
passAvoids unsupported claims23%
passProduces the required deliverable70%
passChecks every criterion, region by region34%
passFinds the Scotland breach88%
passExplains the basket gap85%
passCatches the duplicate rows86%
partialMakes a clear call66%
partialLimits the downside of being wrong91%
failGets the base of every number right41%
failJudges the launch against a baseline set now33%

Run

Run: #1
API response time: 44 s
Submitted: 29 Sept 2026

Gemini 4 ArgonAPI

Needs more than a quick edit10 of 14 checks passed

While the core analysis and regional decisions are solid and well-supported, the output misses a post-launch monitoring plan (signal, threshold, trigger for rollback) and a defined success baseline with a review point, which a PM would need to add before taking this to a launch meeting.

Output

Executive Recommendation

Monday, 2 November: NO-GO across all regions. We cannot launch while failing Criterion 3 (zero dietary/allergen mismatches). During the test, the AI model caused three high-severity safety incidents—substituting standard bread for gluten-free (coeliac customer), dairy cheese for vegan, and hazelnut bars for nut-free (child with nut allergy). The engineering hard filter to prevent cross-attribute substitution does not deploy until Wednesday, 4 November. A 48-hour delay is non-negotiable for customer safety.
Thursday, 5 November: GO WITH CONDITIONS for London, South East, Midlands, and North West. Once the dietary hard filter is deployed on 4 November and verified in production, these four regions pass all launch criteria and can roll out to 100% four days ahead of the 9 November Christmas booking window.
Scotland: NO-GO indefinitely. Keep Scotland on 100% picker-led control until the dairy catalogue re-map is completed and verified via a clean 7-day re-test.

---

Data Hygiene & Addressing Stakeholder Summaries

Before reviewing the criteria table, we must correct three data misconceptions circulating ahead of this meeting:

Duplicate Rows (North West): As flagged in `incidents.csv` (7 Oct), an export timeout duplicated North West treatment rows on 8, 9, and 10 October. All figures below are deduplicated (removing 2,194 phantom orders). Note: Criterion conclusions for North West hold either way, though deduplicating slightly raises its refund rate from 17.56 to 17.59 per 1,000.
The "£4 Basket Drop" is Simpson’s Paradox (Head of Analytics): Pooled across the test, treatment average basket value (£60.15) is indeed £3.93 lower than control (£64.08). However, this is a mathematical artefact of our unequal regional split: we allocated 70% of orders to treatment in our lowest-basket regions (Scotland £53, Midlands £55) and only 30% in our highest-basket region (London £72). Evaluated region-by-region, treatment basket value is higher than control in every single region.
Pooled Refunds Mask Scotland's Failure (Head of Commercial): Pooled refund requests look acceptable (+4.5% vs. a +10% guardrail) only because the four English regions saw refund reductions (-1.5% to -7.8%). This masks a +78.3% refund explosion in Scotland. Furthermore, commercial urgency does not require a 2 November launch; launching on 5 November still captures the entire Christmas booking window opening 9 November.

---

Launch Criteria Performance by Region (1–28 Oct, Deduplicated)

Region (Trt Split)	C1: Sub Acceptance (Target: ≥80.0%)	C2: Refunds / 1k Orders (Target: ≤ +10.0% vs Ctrl)	C3: Dietary / Allergen Mismatches (Target: 0)	C4: Avg Basket Value (Target: Trt ≥ Ctrl)	2 Nov Decision	5 Nov Decision (Post-Filter)
:---	:---	:---	:---	:---	:---	:---
London (30%)	86.78% (23,314 / 26,866) Ctrl: 70.85% \	PASS	17.46 vs 18.21 Ctrl (-4.1%) 369/21,136 vs 890/48,874 \	PASS	1 High (23 Oct: Nut allergy) FAIL	£72.64 vs £72.49 (+£0.15) Simple: £72.65 vs £72.49 \	PASS	NO-GO	GO
South East (50%)	86.37% (26,548 / 30,735) Ctrl: 70.69% \	PASS	17.27 vs 17.85 Ctrl (-3.3%) 462/26,756 vs 473/26,498 \	PASS	1 High (19 Oct: Vegan/dairy) FAIL	£66.64 vs £66.26 (+£0.38) Simple: £66.66 vs £66.27 \	PASS	NO-GO	GO
Midlands (70%)	86.33% (26,669 / 30,891) Ctrl: 70.89% \	PASS	17.56 vs 17.83 Ctrl (-1.5%) 544/30,975 vs 231/12,953 \	PASS	1 High (4 Oct: Coeliac/gluten) FAIL	£55.68 vs £55.39 (+£0.29) Simple: £55.66 vs £55.39 \	PASS	NO-GO	GO
North West (50%)*	86.52% (18,327 / 21,182) Ctrl: 71.36% \	PASS	17.59 vs 19.08 Ctrl (-7.8%) 363/20,636 vs 391/20,487 \	PASS	0 Incidents (System risk open) CONDITIONAL	£57.52 vs £57.33 (+£0.19) Simple: £57.53 vs £57.32 \	PASS	NO-GO	GO
Scotland (70%)	82.45% (14,508 / 17,596) Ctrl: 71.15% \	PASS* (78.7% post-15 Oct)	31.71 vs 17.79 Ctrl (+78.3%) 579/18,257 vs 140/7,871 \	FAIL	0 Incidents (System risk open) CONDITIONAL	£52.97 vs £52.66 (+£0.31) Simple: £52.96 vs £52.65 \	PASS	NO-GO	NO-GO
Pooled Total (Distorted)	85.99% vs 70.91% Ctrl (+15.08 pp)	19.68 vs 18.23 Ctrl (+7.9% dedup / +4.5% raw)	3 High-Severity Incidents	£60.15 vs £64.08 Ctrl (-£3.93* Simpson's Paradox)*	NO-GO	PARTIAL GO

\Note: North West metrics reflect deduplicated data (raw orders: 22,830; raw refunds/1k: 17.56; raw weighted basket: £57.50). Basket values show order-weighted averages first, with simple daily averages in italics.*

---

Regional Rationale & Conditions

#### 1. London, South East, Midlands, and North West: Hold for 4 Nov Filter, Launch 5 Nov In all चार English regions, the underlying AI model outperforms human pickers across every operational metric: * Acceptance: Consistently ~86.5% across all four regions (+15.5 points over control), well above the 80% threshold. * Refunds: Lower than control in every region (ranging from -1.5% in the Midlands to -7.8% in the North West). Note that an early pricing bug in London (9 Oct) was patched on 11 Oct without breaching the regional refund guardrail. * Baskets: Up £0.15 to £0.38 per order over control. * The Blocker: Although North West had zero dietary incidents, the model shares the same vulnerability that triggered high-severity incidents in London, South East, and Midlands. Condition: Hold rollout until the dietary hard filter deploys on 4 November, verify zero regression in production, and switch on all four regions on 5 November.

#### 2. Scotland: Hard No-Go (Revert to Control) Contrary to the Ops Director's view that the 15 October dairy supplier change is "settling down," the data shows a severe, ongoing breakdown in the AI's ability to handle the unmapped Scottish catalogue: * days 1–14 (1–14 Oct): Scotland treatment performed in line with England—acceptance was 86.44% (7,976 / 9,227) and refund requests were 17.46 per 1,000 (162 / 9,280), beating control (18.22). * Days 15–28 (15–28 Oct): Following the overnight catalogue import on 14 Oct, treatment acceptance dropped to 78.05% (6,532 / 8,369)—failing Criterion 1—and never recovered (sitting at 77.76% on 28 Oct). Worse, treatment refund requests more than doubled to 46.45 per 1,000 (417 / 8,977, vs. 17.39 in control), driving a +78.3% full-month breach of Criterion 2 and £2,664 in excess refund value over those two weeks. Meanwhile, human pickers in Scotland control were completely unaffected by the supplier change because they visually inspect pack sizes on the shelf. * Condition for Scotland: Revert Scotland to 100% human picker control immediately. Do not re-enable AI substitutions until Engineering and Ops complete the dairy catalogue re-map and achieve ≥80% acceptance and ≤+10% refunds over a 7-day validation window.

Grades and run details

Decision model 68 · LLM judge 80

Decision model checks

failUses the supplied evidence correctly22%
passAddresses the actual decision55%
partialRespects explicit constraints40%
passIdentifies material uncertainty88%
partialAvoids unsupported claims31%
passProduces the required deliverable52%
passChecks every criterion, region by region58%
passFinds the Scotland breach100%
passExplains the basket gap100%
passCatches the duplicate rows100%
passMakes a clear call33%
partialLimits the downside of being wrong23%
failGets the base of every number right48%
failJudges the launch against a baseline set now22%

Run

Run: #1
API response time: 23 s
Submitted: 13 Oct 2026

Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit7 of 14 checks passed

The output misses the duplicate rows, does not explain the basket gap as a mix artefact, and lacks a post-launch monitoring plan, so a PM would need to rework these areas before using it.

Output

Launch Recommendation: AI Auto-Substitutions

As the Staff PM for fulfilment, my recommendation for the 30 October launch meeting is a No-Go for nationwide rollout on 2 November.

While the Commercial and Analytics teams are looking at top-line aggregates, a region-by-region audit of the 28-day test data and incident logs reveals critical blockers. Specifically, Scotland fails acceptance and refund criteria due to a supplier mapping failure, Midlands and London recorded high-severity dietary/allergen mismatches, and South East also experienced a dietary mismatch.

We should hold all regions until the dietary hard filter is deployed on 4 November, and until Scotland’s catalogue re-map is fully completed and verified.

---

Regional Evaluation Table (1–28 October Test Data)

Region	Subs Acceptance ($\ge 80\%$)	Treatment Refunds vs Control ($\le +10\%$)	Dietary/Allergen Mismatches (Target: 0)	Basket Value Impact	Recommendation
:---	:---	:---	:---	:---	:---
London	$86.4\%$ (Pass)	Control: 1.77%<br>Treatment: 1.63% ($-7.9\%$)	1 High Incident (Nut allergy violation on 23 Oct)	Control: £72.33<br>Treatment: £72.71 (+0.5%)	No-Go (Hold for 4 Nov filter)
South East	$87.0\%$ (Pass)	Control: 1.66%<br>Treatment: 1.69% ($+1.8\%$)	1 High Incident (Vegan dairy violation on 19 Oct)	Control: £66.21<br>Treatment: £66.72 (+0.8%)	No-Go (Hold for 4 Nov filter)
Midlands	$84.5\%$ (Pass)	Control: 1.69%<br>Treatment: 1.68% ($-0.6\%$)	1 High Incident (Gluten-free coeliac violation on 4 Oct)	Control: £55.22<br>Treatment: £55.51 (+0.5%)	No-Go (Hold for 4 Nov filter)
North West	$88.4\%$ (Pass)	Control: 1.73%<br>Treatment: 1.66% ($-4.0\%$)	0 Incidents	Control: £57.34<br>Treatment: £57.65 (+0.5%)	Go with Conditions (Deploy 4 Nov)
Scotland	$77.0\%$ (Fail)	Control: 1.79%<br>Treatment: $3.90\%$ ($+117.9\%$)	0 Incidents (High dairy wrong-size volume)	Control: £52.48<br>Treatment: £52.88 (+0.8%)	No-Go (Hold for catalogue re-map)

Note: Refund request rates are calculated per 100 orders for precise comparison against the 10% relative ceiling.

---

Region-by-Region Breakdown

London, South East, Midlands: All three regions easily clear the 80% acceptance threshold and show neutral-to-positive average basket values (debunking the £4 basket drop concern). However, all three recorded high-severity dietary/allergen failures (gluten, vegan, and nut-allergy breaches). Because the safety hard filter is not deployed until 4 November, rolling out on 2 November exposes customers to unacceptable allergen risks during the peak Christmas slot-booking window.
North West: Our strongest performer. It meets all four PRD criteria cleanly³: 88.4% acceptance¹, treatment refund rates slightly below control, zero dietary incidents, and a stable basket value. It is ready to roll out immediately following the hard filter deployment.
Scotland: A clear No-Go. Acceptance dropped to 77.0% (failing the 80% bar), and treatment refund requests spiked catastrophically by 117.9% over control (driven by the dairy supplier switch on 15 Oct where pack sizes were mismapped). Until the catalogue re-map is finished and stabilized, rolling out here will compound operational friction.

Action Plan

Hold Rollout on 2 November.
Deploy Dietary Hard Filter on 4 November as scheduled, and verify regression tests.
Complete Scotland Catalogue Re-map and clear open wrong-size substitution incidents.
Staged Launch: Roll out to North West, London, South East, and Midlands on Monday, 9 November (concurrent with the Christmas slot-booking window opening), pending a 7-day post-fix stability review.² Scotland follows only after catalogue remediation.

What a PM had to fix

1Numbers wrongRedo the arithmeticStart again
What we’d changeRecompute every figure from daily_metrics.csv, removing the duplicated North West rows. Almost every number in the table is wrong.
2Contradiction missedSurface the contradictionSubstantial rework
What we’d changeResolve the contradiction: a 7-day review after the 4 November filter can't finish by 9 November, and North West is marked go in the table but held here. Answer the stakeholders' claims rather than dismissing them.
3Hypothesis stated as factReframe it as a hypothesisTargeted repair
What we’d changeZero recorded mismatches in North West isn't evidence of safety. It runs the same unfiltered model as the regions that had them.

Grades and run details

Decision model 46 · LLM judge 60

Decision model checks

failUses the supplied evidence correctly34%
partialAddresses the actual decision27%
passRespects explicit constraints20%
passIdentifies material uncertainty56%
partialAvoids unsupported claims33%
passProduces the required deliverable28%
partialChecks every criterion, region by region64%
passFinds the Scotland breach95%
failExplains the basket gap58%
failCatches the duplicate rows100%
partialMakes a clear call83%
partialLimits the downside of being wrong70%
failGets the base of every number right79%
failJudges the launch against a baseline set now28%

Run

Run: #1
Time to output: 46 s
Submitted: 25 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Uses the supplied evidence correctlyRightMixedMixed

GPT-6 Luna · API

All factual claims about the current situation are drawn directly from the supplied data or incidents log, or computed correctly from them.

Gemini 4 Argon · API

Every claim about the current situation is supported by the supplied files (incidents.csv, daily_metrics.csv, provided context) or correct arithmetic based on them.

Gemini 3.5 Flash-Lite · Gemini

All factual claims are drawn directly from the supplied data and incident log, with no invented facts.

Identifies material uncertaintyRightRightMixed

GPT-6 Luna · API

It identifies the dietary filter deployment date, the Scotland catalogue re-map with no date, and the duplicate data issue as key unknowns, and says how they would change the call.

Gemini 4 Argon · API

It identifies the dietary filter deployment date and Scotland's catalogue re-map as key unknowns that hold or change the decision, and specifies a clean 7-day re-test for Scotland to resolve the uncertainty.

Gemini 3.5 Flash-Lite · Gemini

Does not name the specific unknowns that could change the decision (e.g., filter slipping past 9 Nov, Scotland refunds not recovering) or how they would be resolved.

Explains the basket gapRightRightWrong

GPT-6 Luna · API

It shows treatment baskets are equal or higher within every region and explains the pooled gap as an artefact of the uneven split.

Gemini 4 Argon · API

It explicitly names Simpson's Paradox and demonstrates that treatment basket values are higher than control in every region, explaining the pooled gap as an uneven-split artifact.

Gemini 3.5 Flash-Lite · Gemini

Does not explain that the pooled basket gap is an artefact of the uneven traffic split; only states per-region baskets are neutral/positive without addressing the Simpson's paradox.

Catches the duplicate rowsRightRightWrong

GPT-6 Luna · API

It catches the duplicate North West treatment rows for 8–10 October, excludes them, and notes the fix needed.

Gemini 4 Argon · API

It catches the duplicate North West treatment rows for 8–10 October, states they were removed (2,194 orders), and notes the criterion conclusions for North West hold either way.

Gemini 3.5 Flash-Lite · Gemini

Does not mention or handle the duplicated North West treatment rows for 8–10 October flagged in incidents.csv.

All got wrong 2

Limits the downside of being wrongWrongWrongWrong

GPT-6 Luna · API

It does not name a specific post-launch signal, a threshold that would trigger pause or rollback, or flag any irreversible actions.

Gemini 4 Argon · API

The output specifies pre-launch verification conditions but does not name a post-launch signal, a threshold that would trigger a pause or rollback, or clearly flag any irreversible actions for the four regions set to launch.

Gemini 3.5 Flash-Lite · Gemini

Does not name a post-launch signal, threshold, or rollback plan; only mentions a '7-day post-fix stability review' without specifics.

Judges the launch against a baseline set nowWrongWrongWrong

GPT-6 Luna · API

It does not name a success metric with its current baseline, nor a review point with what each result would lead to.

Gemini 4 Argon · API

The output does not name a specific success metric and its current baseline from the test that the launch will be judged against, nor does it set a post-launch review point with what each result would lead to.

Gemini 3.5 Flash-Lite · Gemini

Does not name a success metric, its current baseline, or a review point with what each result would lead to.

All mixed 1

Gets the base of every number rightMixedMixedMixed

GPT-6 Luna · API

All rates and percentages are computed with the correct denominators (e.g., refunds per 1,000 orders) and the base is stated where needed.

Gemini 4 Argon · API

All derived figures (acceptance rates, refunds per 1,000, basket values, percentage changes) use correct denominators and arithmetic, with basket values specified as order-weighted.

Gemini 3.5 Flash-Lite · Gemini

Refund rates are stated per 100 orders, which preserves the relative difference; all derived figures appear consistent with the data.

All got right 7

Addresses the actual decisionRightRightRight

GPT-6 Luna · API

The output commits to a clear no-go for 2 November, gives a region-by-region call with conditions, and states what would change each decision.

Gemini 4 Argon · API

The output gives a clear NO-GO for all regions on 2 November, a GO with conditions for four regions on 5 November, and a NO-GO for Scotland, directly addressing the launch meeting's decision and stating what would change each call.

Gemini 3.5 Flash-Lite · Gemini

Commits to a clear No-Go for 2 Nov, with conditions for staged rollout, and states what would change it (filter deployment, catalogue re-map).

Respects explicit constraintsRightRightRight

GPT-6 Luna · API

The output includes the required region-by-region criteria table, stays under 900 words, and addresses the launch meeting audience.

Gemini 4 Argon · API

The output is a recommendation for the meeting, includes the required region-by-criteria table with figures, and keeps prose under 900 words.

Gemini 3.5 Flash-Lite · Gemini

Prose is under 900 words, includes a region-by-criterion table, and addresses the launch meeting as requested.

Avoids unsupported claimsRightRightRight

GPT-6 Luna · API

Interpretations like the basket gap being a mix artefact are clearly explained with evidence; no unsupported causes or forecasts are presented as fact.

Gemini 4 Argon · API

The Simpson's paradox explanation for the basket gap is presented as a calculated correction, not an unsupported cause, and the 'settling down' quote from Ops Director is appropriately disputed with data showing the opposite.

Gemini 3.5 Flash-Lite · Gemini

Interpretations like 'strongest performer' are clearly judgment, and causal links (supplier change driving refunds) are supported by incident logs.

Produces the required deliverableRightRightRight

GPT-6 Luna · API

The recommendation is complete, in the right form for a launch meeting, and a PM could act on it with light edits.

Gemini 4 Argon · API

The output is a complete, actionable recommendation with a decision table, regional rationale, and conditions, usable for the meeting with no major gaps.

Gemini 3.5 Flash-Lite · Gemini

The recommendation is complete, in the right form for the meeting, and a PM could act on it with light edits.

Checks every criterion, region by regionRightRightRight

GPT-6 Luna · API

The table checks all four launch criteria for each region with figures computed from the files.

Gemini 4 Argon · API

The recommendation tests all four launch criteria (sub acceptance, refunds vs control, dietary mismatches, basket value) region by region in a table with correct figures computed from the files.

Gemini 3.5 Flash-Lite · Gemini

The table checks all four launch criteria (acceptance, refunds vs control, dietary mismatches, basket value) for each region with figures from the files.

Finds the Scotland breachRightRightRight

GPT-6 Luna · API

It finds Scotland's refund breach (~62% above control), ties it to the supplier change after 15 October, notes the acceptance dip, and holds Scotland.

Gemini 4 Argon · API

It identifies Scotland's refund breach (+78.3%), ties the sharp decline to the 15 October supplier change with pre/post metrics, notes the acceptance dip below 80%, and recommends holding Scotland.

Gemini 3.5 Flash-Lite · Gemini

Identifies Scotland's refund breach (+117.9%), ties it to the 15 Oct supplier change, notes acceptance dip, and holds Scotland.

Makes a clear callRightRightRight

GPT-6 Luna · API

It makes one clear call per region (no-go, go with conditions, hold) with dates and states what would change each.

Gemini 4 Argon · API

It makes one clear call per region with specific dates (NO-GO 2 Nov, GO with conditions 5 Nov for four regions, NO-GO indefinitely for Scotland) and states the conditions that would change each.

Gemini 3.5 Flash-Lite · Gemini

Makes a clear per-region call (No-Go for London, South East, Midlands, Scotland; Go with conditions for North West) with dates and conditions.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 62% of 68 checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	Sonnet 5.5withAPI	92.4	100.0	2	None
2	GPT-6.1 SolwithAPI	91.9	95.8	2	None
3	Haiku 5.5withAPI	84.3	100.0	2	None
4	GPT-6 AstrawithChatGPT	91.9	92.5	2	None
5	Opus 5.5withClaude	84.3	96.7	2	None
6	GPT-6 LunawithAPI	80.2	80.8	2	None
7	Gemini 4 ArgonwithAPI	70.3	73.3	2	None
8	Gemini 3.5 Flash-LitewithGemini	55.0	59.2	2	None
9	Gemini 3.8 FlashwithAPI	64.4	67.5	2	1 capped

What you get for the cost

Haiku 5.5 · API scores within 3 points of Sonnet 5.5 · API at about a tenth of the cost.

Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)At least one critical failure

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
GPT-6 Luna · API	79	1.0×
Gemini 3.5 Flash-Lite · Gemini	55	2.7× (partly estimated)
Haiku 5.5 · API	93	2.8×
Gemini 3.8 Flash · API	54	11×
GPT-6.1 Sol · API	94	12×
Gemini 4 Argon · API	71	14×
Sonnet 5.5 · API	96	37×
GPT-6 Astra · ChatGPT	93	46× (partly estimated)
Opus 5.5 · Claude	91	59× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Deciding whether a feature ships on the planned date, and on what conditions.

Why it matters

Launch meetings reward optimism. A good call checks each agreed criterion, weighs a real risk against a real gain, and says exactly what would change the answer. A weak one rubber-stamps the launch or blocks it on noise.

What good looks like

Checks each agreed criterion against the evidence
Makes one clear call, with conditions if needed
Separates risks that block launch from risks that can be managed
Says what would change the call
Says how to limit the damage if the call is wrong

Deliberately not measured

Rollout engineering detail
Project-plan formatting

Capability tested

Deciding against agreed launch criteria

The failure we’re looking for

Rubber-stamps a launch that misses an agreed bar, or blocks it on noise

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

A launch memo from supplied evidence · Staff level: a regional call from an attached data workbook

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

Limits the downside of being wrong

Judges the launch against a baseline set now

Plus our standard checks

The tasks

The outputs

Output

Recommendation: No-go for the all-region rollout on 2 November

What a PM had to fix

Grades and run details

Decision model checks

Run

Output

Executive Recommendation

Data Hygiene & Addressing Stakeholder Summaries

Launch Criteria Performance by Region (1–28 Oct, Deduplicated)

Regional Rationale & Conditions

Grades and run details

Decision model checks

Run

Output

Launch Recommendation: AI Auto-Substitutions

Regional Evaluation Table (1–28 October Test Data)

Region-by-Region Breakdown

Action Plan

What a PM had to fix

Grades and run details

Decision model checks

Run

Check by check

Where they differ 4

All got wrong 2

All mixed 1

All got right 7

Results

What you get for the cost

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured