Tasks / Operate

Make the launch call

Can the model make a clear go / no-go call from mixed launch evidence, checked against the criteria agreed up front?

Measures the modelTask v1.2 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 14 graded outputs by 7 models. 64% were usable with at most a quick edit.

Reliably right

Finds the Scotland breach100% pass
Clearly identifies Scotland's refund breach (66% above control, 143% after supplier change) and acceptance dip below 80%, correctly holds Scotland.
Opus 5.5 · Claude · Go/no-go for AI substitutions, from the rollout data
Produces the required deliverable98% pass
The recommendation is in the requested form, for the meeting, within the length, and includes necessary steps a PM could act on.
GPT-6 Astra · ChatGPT · Go/no-go for AI-drafted support replies
Addresses the actual decision95% pass
The output gives a clear, unambiguous 'no-go' call early, and states what would change it (fixes and retest).
GPT-6 Astra · ChatGPT · Go/no-go for AI-drafted support replies

Where it slips

Limits the downside of being wrong32% pass
No specific post-launch monitoring signal or threshold is named; only generic 'monitoring' and a kill switch are mentioned.
GPT-6 Astra · ChatGPT · Go/no-go for AI-drafted support replies
Catches the duplicate rows50% pass
Finds and removes the duplicate rows but never states whether they change the results.
GPT-6.1 Sol · API · Go/no-go for AI substitutions, from the rollout data
Uses the supplied evidence correctly54% pass
The output states that pilot agents were hands-on, engaged and motivated, which is not supported by the supplied context (only that 7/8 wanted to keep it).
Opus 5.5 · Claude · Go/no-go for AI-drafted support replies

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Staff PM for fulfilment at Basketful. On 30 October the launch meeting decides whether AI auto-substitutions (the model picks a replacement when an item is out of stock, instead of the picker) roll out to every region on Monday 2 November, ahead of the Christmas delivery-slot booking window that opens on 9 November. The rollout workbook is attached as two files: daily_metrics.csv (28 days of the regional test, by region and arm) and incidents.csv. Work from the data, not the summaries people have given you. Write your recommendation for the meeting: go, no-go or go with conditions, region by region, and why. Include a table that checks each launch criterion for each region with the figures. Keep the prose under 900 words.

What the model was given6 items: The test, Launch criteria (agreed in the PRD), What people have said, Engineering, daily_metrics.csv, incidents.csv

The test1–28 October, five regions. In each region orders were split between control (the picker chooses substitutes, as today) and treatment (AI auto-substitutions). The split was not 50/50 everywhere: treatment got 30% of orders in London, 50% in the South East and North West, and 70% in the Midlands and Scotland.

Launch criteria (agreed in the PRD)1. Substitution acceptance (subs_accepted ÷ subs_offered) of at least 80% in treatment, in every region. 2. Refund requests per 1,000 orders in treatment no more than 10% above control, in every region. 3. Zero dietary or allergen mismatches (a substitute that breaks a gluten-free, vegan, vegetarian, nut-free, halal or kosher attribute). 4. Average basket value not lower in treatment than control.

What people have saidHead of Commercial: “Acceptance is up 15 points and refunds are within the guardrail overall. Every day we wait costs us Christmas.” Head of Analytics: “Treatment baskets are £4 smaller across the test. That worries me.” Ops director (Scotland): “The supplier change caused some noise, but it's settling down.”

EngineeringThe dietary-attribute hard filter (never substitute across those attributes) is merged and scheduled to deploy on 4 November after regression tests. Scotland's catalogue re-map for the new dairy supplier is in progress with no date. Rollout can be switched on region by region.

daily_metrics.csv283 rows · Download

date,region,arm,orders,items_ordered,items_out_of_stock,subs_offered,subs_accepted,refund_requests,refund_value_gbp,complaints,avg_basket_gbp
2026-10-01,London,control,1757,52273,2351,1953,1387,32,160.71,6,72.89
2026-10-01,London,treatment,692,20455,871,820,708,12,62.05,2,72.42
2026-10-01,South East,control,853,22846,996,833,593,16,74.49,3,65.62
2026-10-01,South East,treatment,928,25223,1166,1106,952,15,75.93,3,66.59
2026-10-01,Midlands,control,467,10468,465,383,268,8,34.57,1,54.92
2026-10-01,Midlands,treatment,1113,25136,1166,1087,947,21,106.35,3,55.33
…

incidents.csv13 rows · Download

date,region,arm,type,severity,summary,status
2026-10-02,London,treatment,wrong_size,low,"4-pint milk substituted with 1-pint, customer accepted at door",closed
2026-10-04,Midlands,treatment,dietary_mismatch,high,"Gluten-free sliced loaf substituted with standard white loaf; customer coeliac, noticed at home",closed: refund and apology
2026-10-06,South East,control,wrong_item,low,"Picker substituted oat milk with soya milk; customer rejected at door",closed
2026-10-07,North West,treatment,data_export,low,"Daily export job re-ran after a timeout on 8, 9 and 10 Oct; analytics team flagged possible duplicate rows",open
2026-10-09,London,treatment,price,medium,"Substitute priced higher than original; customer charged the difference against policy",closed: policy fix deployed 11 Oct
2026-10-12,Scotland,control,late_delivery,low,"Van breakdown, 14 orders late",closed
…

What a strong answer doesThe answer key the graders mark against

No-go for a national launch on 2 November, because criterion 3 fails everywhere until the dietary hard filter ships: incidents.csv has three treatment dietary mismatches (4 Oct Midlands, 19 Oct South East, 23 Oct London, the last involving nuts and a child with an allergy). Go with conditions, region by region, once the filter is deployed and verified (planned 4 November, still before the 9 November booking window): London, South East, Midlands and North West pass the other criteria (acceptance about 86% against 71% control; refunds 2–5% below control). Hold Scotland: over the month its treatment refunds are about 66% above control, and from the 15 October supplier change about 143% above (43 against 18 per 1,000 orders), with acceptance falling to about 79%, below the 80% bar; it waits for the catalogue re-map and a clean re-test. The pooled refund rate (about 8% above control) hides this, and the criterion is per region. The pooled basket gap (about £65 against £61) is an artefact of the uneven split, since treatment baskets are equal or higher in every region (Simpson's paradox), so criterion 4 passes. Notices that North West treatment rows for 8–10 October are duplicated (an extra 2,194 orders), removes them, and says it doesn't change the result. States what would change the call: the filter slipping past 9 November, or Scotland's refunds not recovering.

Critical failures (cap the score at 40)

Recommends any rollout before the dietary hard filter is deployed
Recommends rolling out to Scotland while its refund guardrail is breached

Case

Staff-level · v1.8 · synthetic · e-commerce, AI product, data analysis, staff-level

The outputs

7 setups answered this brief. Pick up to three to read side by side.

Opus 5.5Claude91Usable with a quick edit GPT-6.1 SolAPI86Usable with a quick edit Sonnet 5.5API84Usable with a quick edit GPT-6 AstraChatGPT82Usable with a quick edit GPT-6 LunaAPI64Needs more work Gemini 3.5 Flash-LiteGemini50Needs more work Gemini 3.8 FlashAPI40Critical failure

Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit6 of 13 checks passed

It fails to address the known duplicate data rows, doesn't specify monitoring triggers or rollback criteria, and makes a causal claim about Scotland without hedging, so a PM would need to add those before using it.

Output

Launch Recommendation: AI Auto-Substitutions

As the Staff PM for fulfilment, my recommendation for the 30 October launch meeting is a No-Go for nationwide rollout on 2 November.

While the Commercial and Analytics teams are looking at top-line aggregates, a region-by-region audit of the 28-day test data and incident logs reveals critical blockers. Specifically, Scotland fails acceptance and refund criteria due to a supplier mapping failure, Midlands and London recorded high-severity dietary/allergen mismatches, and South East also experienced a dietary mismatch.

We should hold all regions until the dietary hard filter is deployed on 4 November, and until Scotland’s catalogue re-map is fully completed and verified.

---

Regional Evaluation Table (1–28 October Test Data)

Region	Subs Acceptance ($\ge 80\%$)	Treatment Refunds vs Control ($\le +10\%$)	Dietary/Allergen Mismatches (Target: 0)	Basket Value Impact	Recommendation
:---	:---	:---	:---	:---	:---
London	$86.4\%$ (Pass)	Control: 1.77%<br>Treatment: 1.63% ($-7.9\%$)	1 High Incident (Nut allergy violation on 23 Oct)	Control: £72.33<br>Treatment: £72.71 (+0.5%)	No-Go (Hold for 4 Nov filter)
South East	$87.0\%$ (Pass)	Control: 1.66%<br>Treatment: 1.69% ($+1.8\%$)	1 High Incident (Vegan dairy violation on 19 Oct)	Control: £66.21<br>Treatment: £66.72 (+0.8%)	No-Go (Hold for 4 Nov filter)
Midlands	$84.5\%$ (Pass)	Control: 1.69%<br>Treatment: 1.68% ($-0.6\%$)	1 High Incident (Gluten-free coeliac violation on 4 Oct)	Control: £55.22<br>Treatment: £55.51 (+0.5%)	No-Go (Hold for 4 Nov filter)
North West	$88.4\%$ (Pass)	Control: 1.73%<br>Treatment: 1.66% ($-4.0\%$)	0 Incidents	Control: £57.34<br>Treatment: £57.65 (+0.5%)	Go with Conditions (Deploy 4 Nov)
Scotland	$77.0\%$ (Fail)	Control: 1.79%<br>Treatment: $3.90\%$ ($+117.9\%$)	0 Incidents (High dairy wrong-size volume)	Control: £52.48<br>Treatment: £52.88 (+0.8%)	No-Go (Hold for catalogue re-map)

Note: Refund request rates are calculated per 100 orders for precise comparison against the 10% relative ceiling.

---

Region-by-Region Breakdown

London, South East, Midlands: All three regions easily clear the 80% acceptance threshold and show neutral-to-positive average basket values (debunking the £4 basket drop concern). However, all three recorded high-severity dietary/allergen failures (gluten, vegan, and nut-allergy breaches). Because the safety hard filter is not deployed until 4 November, rolling out on 2 November exposes customers to unacceptable allergen risks during the peak Christmas slot-booking window.
North West: Our strongest performer. It meets all four PRD criteria cleanly³: 88.4% acceptance¹, treatment refund rates slightly below control, zero dietary incidents, and a stable basket value. It is ready to roll out immediately following the hard filter deployment.
Scotland: A clear No-Go. Acceptance dropped to 77.0% (failing the 80% bar), and treatment refund requests spiked catastrophically by 117.9% over control (driven by the dairy supplier switch on 15 Oct where pack sizes were mismapped). Until the catalogue re-map is finished and stabilized, rolling out here will compound operational friction.

Action Plan

Hold Rollout on 2 November.
Deploy Dietary Hard Filter on 4 November as scheduled, and verify regression tests.
Complete Scotland Catalogue Re-map and clear open wrong-size substitution incidents.
Staged Launch: Roll out to North West, London, South East, and Midlands on Monday, 9 November (concurrent with the Christmas slot-booking window opening), pending a 7-day post-fix stability review.² Scotland follows only after catalogue remediation.

What a PM had to fix

1Numbers wrongRedo the arithmeticStart again
What we’d changeRecompute every figure from daily_metrics.csv, removing the duplicated North West rows. Almost every number in the table is wrong.
2Contradiction missedSurface the contradictionSubstantial rework
What we’d changeResolve the contradiction: a 7-day review after the 4 November filter can't finish by 9 November, and North West is marked go in the table but held here. Answer the stakeholders' claims rather than dismissing them.
3Hypothesis stated as factReframe it as a hypothesisTargeted repair
What we’d changeZero recorded mismatches in North West isn't evidence of safety. It runs the same unfiltered model as the regions that had them.

Grades and run details

Decision model 46 · LLM judge 8 of 14 checks

Decision model checks

failUses the supplied evidence correctly49%
passAddresses the actual decision32%
passRespects explicit constraints8%
partialIdentifies material uncertainty63%
partialAvoids unsupported claims25%
passProduces the required deliverable26%
partialChecks every criterion, region by region64%
passFinds the Scotland breach96%
failExplains the basket gap48%
failCatches the duplicate rows100%
partialMakes a clear call83%
failLimits the downside of being wrong41%
failGets the base of every number right77%

Run

Run: #1
Time to output: 46 s
Submitted: 25 Sept 2026

Gemini 3.8 FlashAPI

Needs more than a quick edit5 of 13 checks passed

A PM could not use it without rechecking evidence because the North West GO and duplicate-row figure are wrong. It also needs added rollback triggers and change conditions.

Output

Recommendation Summary: Go With Conditions

Do not execute a blanket rollout on Monday, 2 November.

North West: GO (2 November). Meets all four PRD launch criteria cleanly.
London, South East, Midlands: GO WITH CONDITIONS (Deploy Thursday, 5 November). Commercial metrics are strong, but all three regions breached Criterion 3 with severe dietary/allergen substitutions. Launch must be gated on the successful deployment of the dietary-attribute hard filter on 4 November.
Scotland: NO-GO. Fails Criterion 2 (+67.4% refund spike vs. control) and Criterion 1 post-supplier transition (76.6% acceptance). Revert 100% to control until catalogue re-mapping is completed and re-tested.

This phased approach protects customer safety and brand reputation while capturing AI substitution value ahead of Christmas slot bookings opening on 9 November.

---

PRD Launch Criteria Assessment

Note: North West treatment metrics reflect deduplication of export retries on 8–10 October (3,194 duplicate rows removed).

Region	Arm	Orders	1. Subs Acceptance (Target: $\ge$80%)	2. Refunds / 1k Orders (Target: $\le$+10% vs Ctrl)	3. Allergen / Dietary Mismatches (Target: 0)	4. Avg Basket Value (Target: $\ge$ Ctrl)	Status
:---	:---	:---	:---	:---	:---	:---	:---
London	Control<br>Treatment	49,159<br>21,054	71.5%<br>87.4%	18.06<br>17.53 (-2.9%)	0<br>1 (FAIL)	£72.52<br>£72.68 (+£0.16)	CONDITIONAL<br>Hold for filter
South East	Control<br>Treatment	26,485<br>26,919	71.7%<br>86.5%	17.86<br>17.35 (-2.9%)	0<br>1 (FAIL)	£66.27<br>£66.57 (+£0.30)	CONDITIONAL<br>Hold for filter
Midlands	Control<br>Treatment	13,048<br>31,174	71.3%<br>85.2%	17.86<br>17.58 (-1.6%)	0<br>1 (FAIL)	£55.37<br>£55.62 (+£0.25)	CONDITIONAL<br>Hold for filter
North West	Control<br>Treatment	20,702<br>20,774	70.8%<br>85.8%	18.26<br>17.47 (-4.3%)	0<br>0 (PASS)	£57.29<br>£57.53 (+£0.24)	GO<br>Launch 2 Nov
Scotland	Control<br>Treatment	7,853<br>18,371	71.3%<br>81.5% (76.6% post-15 Oct)	18.21<br>30.48 (+67.4% - FAIL)	0<br>0 (PASS)	£52.62<br>£52.89 (+£0.27)	NO-GO<br>Catalogue fault

---

Analysis & Stakeholder Debunking

#### 1. Head of Analytics: “Treatment baskets are £4 smaller across the test.” Finding: False alarm caused by Simpson’s Paradox. Across all pooled orders, treatment average basket appears lower (£61.06 treatment vs. £65.18 control, a £4.12 deficit). However, this is an aggregation distortion caused by the uneven regional split: London (average basket ~£73) received only a 30% treatment allocation, whereas lower-basket regions (Midlands at ~£55 and Scotland at ~£53) received a 70% treatment allocation.

When evaluated within every single region, treatment basket value was higher than control by £0.16 to £0.30. Criterion 4 is fully satisfied across all regions.

#### 2. Head of Commercial: “Refunds are within the guardrail overall. Every day we wait costs Christmas.” Finding: Misleading pooled metric masking a severe regional failure. While aggregate refunds appear within limits (+8.1% overall), the PRD explicitly requires compliance in every region. Scotland completely breaches this guardrail (+67.4% above control). Launching everywhere on 2 November would risk widespread customer attrition in Scotland right before holiday booking.

#### 3. Ops Director (Scotland): “The supplier change caused noise, but it's settling down.” Finding: Factually incorrect; Scotland is failing and persistent. Following the 15 October dairy supplier change, wrong-size substitutions (e.g., pack sizes mapped to incorrect product families) caused: * Daily refunds in treatment to triple from 11.6/day (Oct 1–14) to 28.4/day (Oct 15–28), reaching 43.7 refunds per 1,000 orders (+140% vs. control). * Treatment acceptance dropped from 86.3% down to 76.6% (failing Criterion 1). * Daily refund value increased 4x to £180–£247/day. Performance has not stabilized. Because the catalogue re-map is incomplete with no ETA, Scotland cannot launch.

#### 4. The Dietary Hazard (London, South East, Midlands) Treatment produced three critical safety failures: * 4 Oct (Midlands): Coeliac customer received standard bread for gluten-free. * 19 Oct (South East): Vegan customer received dairy cheddar. * 23 Oct (London): Child with nut allergies received hazelnut-containing bars for nut-free bars.

Launching auto-substitutions in these regions before deploying technical guardrails creates intolerable liability.

---

Implementation Plan

1. Monday, 2 November: * Roll out North West to 100% treatment. (Clean test data, no allergen incidents, +15pp acceptance lift). * Revert Scotland treatment to 0% (100% control). Keep human pickers selecting substitutes until the catalogue re-map is verified and back-tested. * Maintain London, South East, and Midlands at current test splits for 48 hours.

2. Wednesday, 4 November: * Engineering deploys the dietary-attribute hard filter to production following scheduled regression testing.

3. Thursday, 5 November: * Verify filter telemetry on live orders. * Roll out London, South East, and Midlands to 100% treatment.

4. Monday, 9 November: * Christmas delivery-slot booking opens on schedule with 4 of 5 regions operating under fully automated, safety-enforced substitutions, leaving human pickers dedicated to Scotland until rectified.

Claims the judge couldn’t find in the brief

North West: GO (2 November).
North West treatment metrics reflect deduplication of export retries on 8–10 October (3,194 duplicate rows removed).

Grades and run details

Decision model 54 · LLM judge 5 of 14 checks

Decision model checks

failUses the supplied evidence correctly75%
passAddresses the actual decision51%
partialRespects explicit constraints23%
partialIdentifies material uncertainty39%
failAvoids unsupported claims43%
passProduces the required deliverable53%
partialChecks every criterion, region by region50%
passFinds the Scotland breach100%
passExplains the basket gap99%
partialCatches the duplicate rows84%
passMakes a clear call42%
failLimits the downside of being wrong30%
failGets the base of every number right69%

Run

Run: #1
API response time: 85 s
Submitted: 1 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.