Humanity’s Last PRD
Latest drop: Haiku 5.5 · 8 Oct
Tasks / Product Craft

Make the launch call

Can the model make a clear go / no-go call from mixed launch evidence, checked against the criteria agreed up front?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 69% were usable with at most a quick edit.

Reliably right

  1. Finds the Scotland breach100% pass
    Finds Scotland's refund breach (+66% overall, +143% since 15 Oct), ties it to the supplier change, notes acceptance dip, and holds Scotland.
    Sonnet 5.5 · API · Go/no-go for AI substitutions, from the rollout data
  2. Treats the overwrites as data loss100% pass
    Treats the overwritten notes on Android 11 as a breach of the no-data-loss criterion and makes fixing the storage-API issue a condition for that platform.
    Opus 5.5 · Claude · Offline, on older phones
  3. Addresses the actual decision98% pass
    Commits to a clear no-go for 2 Nov, conditional go from 4 Nov for four regions, and no-go for Scotland, with conditions that would change it.
    Sonnet 5.5 · API · Go/no-go for AI substitutions, from the rollout data

Where it slips

  1. Limits the downside of being wrong63% pass
    It does not name a specific post-launch signal, a threshold that would trigger pause or rollback, or flag any irreversible actions.
    GPT-6 Luna · API · Go/no-go for AI substitutions, from the rollout data
  2. Judges the launch against a baseline set now66% pass
    Does not name a specific review point or when the team will review results to decide next steps; only gives a pause trigger without a scheduled evaluation.
    GPT-6.1 Sol · API · Offline, on older phones
  3. Checks every criterion for every platform69% pass
    Android 13's error-free sync is 98.9%, below the 99% threshold, yet the table marks it as 'GO WITH CONDITIONS' instead of 'NO-GO' for that criterion, misapplying the hard launch criteria.
    Gemini 3.5 Flash-Lite · Gemini · Offline, on older phones

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Limits the downside of being wrong

    Does the call say how exposure is limited if it's wrong: what is watched after launch, the threshold that would pause or roll back, and which parts can't be undone?

    Passes when Names the post-launch signal, a threshold or trigger, and the action, and flags anything irreversible (customer communications, pricing, migrations).

    Karri SaarinenLenny’s Podcast

  2. Judges the launch against a baseline set now

    Does the call name the metric and the current baseline the launch will be judged against, and when the team will review the result to decide what's next?

    Passes when Names the success metric, its baseline from the material, and a review point with what each result would lead to.

    Todd JacksonLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 5 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Staff PM for fulfilment at Basketful. On 30 October the launch meeting decides whether AI auto-substitutions (the model picks a replacement when an item is out of stock, instead of the picker) roll out to every region on Monday 2 November, ahead of the Christmas delivery-slot booking window that opens on 9 November. The rollout workbook is attached as two files: daily_metrics.csv (28 days of the regional test, by region and arm) and incidents.csv. Work from the data, not the summaries people have given you. Write your recommendation for the meeting: go, no-go or go with conditions, region by region, and why. Include a table that checks each launch criterion for each region with the figures. Keep the prose under 900 words.

What the model was given6 items: The test, Launch criteria (agreed in the PRD), What people have said, Engineering, daily_metrics.csv, incidents.csv
The test1–28 October, five regions. In each region orders were split between control (the picker chooses substitutes, as today) and treatment (AI auto-substitutions). The split was not 50/50 everywhere: treatment got 30% of orders in London, 50% in the South East and North West, and 70% in the Midlands and Scotland.
Launch criteria (agreed in the PRD)1. Substitution acceptance (subs_accepted ÷ subs_offered) of at least 80% in treatment, in every region. 2. Refund requests per 1,000 orders in treatment no more than 10% above control, in every region. 3. Zero dietary or allergen mismatches (a substitute that breaks a gluten-free, vegan, vegetarian, nut-free, halal or kosher attribute). 4. Average basket value not lower in treatment than control.
What people have saidHead of Commercial: “Acceptance is up 15 points and refunds are within the guardrail overall. Every day we wait costs us Christmas.” Head of Analytics: “Treatment baskets are £4 smaller across the test. That worries me.” Ops director (Scotland): “The supplier change caused some noise, but it's settling down.”
EngineeringThe dietary-attribute hard filter (never substitute across those attributes) is merged and scheduled to deploy on 4 November after regression tests. Scotland's catalogue re-map for the new dairy supplier is in progress with no date. Rollout can be switched on region by region.
daily_metrics.csv283 rows · Download
date,region,arm,orders,items_ordered,items_out_of_stock,subs_offered,subs_accepted,refund_requests,refund_value_gbp,complaints,avg_basket_gbp
2026-10-01,London,control,1757,52273,2351,1953,1387,32,160.71,6,72.89
2026-10-01,London,treatment,692,20455,871,820,708,12,62.05,2,72.42
2026-10-01,South East,control,853,22846,996,833,593,16,74.49,3,65.62
2026-10-01,South East,treatment,928,25223,1166,1106,952,15,75.93,3,66.59
2026-10-01,Midlands,control,467,10468,465,383,268,8,34.57,1,54.92
2026-10-01,Midlands,treatment,1113,25136,1166,1087,947,21,106.35,3,55.33
…
incidents.csv13 rows · Download
date,region,arm,type,severity,summary,status
2026-10-02,London,treatment,wrong_size,low,"4-pint milk substituted with 1-pint, customer accepted at door",closed
2026-10-04,Midlands,treatment,dietary_mismatch,high,"Gluten-free sliced loaf substituted with standard white loaf; customer coeliac, noticed at home",closed: refund and apology
2026-10-06,South East,control,wrong_item,low,"Picker substituted oat milk with soya milk; customer rejected at door",closed
2026-10-07,North West,treatment,data_export,low,"Daily export job re-ran after a timeout on 8, 9 and 10 Oct; analytics team flagged possible duplicate rows",open
2026-10-09,London,treatment,price,medium,"Substitute priced higher than original; customer charged the difference against policy",closed: policy fix deployed 11 Oct
2026-10-12,Scotland,control,late_delivery,low,"Van breakdown, 14 orders late",closed
…
What a strong answer doesThe answer key the graders mark against

No-go for a national launch on 2 November, because criterion 3 fails everywhere until the dietary hard filter ships: incidents.csv has three treatment dietary mismatches (4 Oct Midlands, 19 Oct South East, 23 Oct London, the last involving nuts and a child with an allergy). Go with conditions, region by region, once the filter is deployed and verified (planned 4 November, still before the 9 November booking window): London, South East, Midlands and North West pass the other criteria (acceptance about 86% against 71% control; refunds 2–5% below control). Hold Scotland: over the month its treatment refunds are about 66% above control, and from the 15 October supplier change about 143% above (43 against 18 per 1,000 orders), with acceptance falling to about 79%, below the 80% bar; it waits for the catalogue re-map and a clean re-test. The pooled refund rate (about 8% above control) hides this, and the criterion is per region. The pooled basket gap (about £65 against £61) is an artefact of the uneven split, since treatment baskets are equal or higher in every region (Simpson's paradox), so criterion 4 passes. Notices that North West treatment rows for 8–10 October are duplicated (an extra 2,194 orders), removes them, and says it doesn't change the result. States what would change the call: the filter slipping past 9 November, or Scotland's refunds not recovering.

Critical failures (cap the score at 40)
  • Recommends any rollout before the dietary hard filter is deployed
  • Recommends rolling out to Scotland while its refund guardrail is breached
Task

Staff-level · v1.12

The outputs

8 setups answered this brief.

Sonnet 5.5API95Usable with a quick editGPT-6.1 SolAPI93Usable with a quick editGPT-6 AstraChatGPT90Usable with a quick editHaiku 5.5API90Usable with a quick editOpus 5.5Claude88Usable with a quick editGPT-6 LunaAPI78Usable with a quick editGemini 3.5 Flash-LiteGemini52Needs more workGemini 3.8 FlashAPI40Critical failure
Haiku 5.5API

Usable with a quick edit12 of 14 checks passed

The output gives a complete, evidence‑based recommendation with a criteria table, region‑specific calls, and clear conditions. A PM could use it with minor edits without re‑checking the data or reasoning.

Output

Launch recommendation: AI auto-substitutions, 2 November

Recommendation: do not launch to all regions on 2 November. Switch on North West once the dietary hard filter is confirmed live (target 5 November). Hold London, South East and Midlands, and re-decide on 6 November. Keep Scotland on picker substitutes.

Criterion 3 is a hard PRD gate, and the test recorded three high-severity dietary breaks in treatment. The hard filter does not deploy until 4 November, so a 2 November launch would go live without the control that would have prevented them.

Data notes

  • Duplicates removed. North West treatment rows for 8, 9 and 10 October are duplicated (the export re-ran). I kept one copy of each. Including them barely moves the result (acceptance 86.2%, refunds 17.5 per 1,000), but the incident is still open and the export should be fixed.
  • Split is uneven, so I used rates. Acceptance, refunds per 1,000 orders and basket are computed per region and arm, not from totals.
  • Basket: the £4 gap isn't in the file. Daily average basket is higher in treatment in every region (+£0.24 to +£0.46 per order on simple averages). A £4 gap would need a different definition, such as totals, which differ because arm volumes differ. Analytics should share its query before anyone quotes the figure. I haven't checked order-weighted averages, but I don't expect them to reverse the sign.
  • Pooled figures hide the problem. Pooled, treatment refunds are 19.5 per 1,000 against 18.1 for control (+8%), which passes. That is only because four regions are under control and Scotland is far over it.

Criteria check (treatment vs control)

Region1. Acceptance (≥80%)2. Refunds per 1,000 (≤10% above control)3. Dietary breaks (zero)4. Basket (not lower)Call
London86.7% vs 71.4% ✔17.4 vs 18.0 (−3%) ✔1 high: nut-free → hazelnut, 23 Oct, open ✘£72.72 vs £72.48 ✔No-go
South East86.4% vs 71.0% ✔17.5 vs 17.9 (−2%) ✔1 high: vegan → dairy cheddar, 19 Oct ✘£66.66 vs £66.20 ✔No-go
Midlands86.3% vs 71.1% ✔17.5 vs 17.9 (−2%) ✔1 high: gluten-free → standard loaf, 4 Oct ✘£55.66 vs £55.35 ✔No-go
North West86.3% vs 71.2% ✔17.6 vs 18.5 (−5%) ✔0 logged ✔£57.52 vs £57.28 ✔Go with conditions
Scotland82.8% overall ✔; 79.1% since 15 Oct ✘30.2 vs 18.2 (+66%) ✘0 logged ✔£52.98 vs £52.62 ✔No-go

Region by region

North West: go with conditions. It is the only region that passes all four criteria on clean data. Its zero is partly luck, though. The three dietary breaks show the model crosses attribute boundaries with no hard filter, so the filter is a precondition here as well. Conditions: 1. The hard filter is deployed and regression-tested (4 November), and North West's catalogue is checked against it. 2. The duplicate-row export issue is closed and the North West figures are re-run on the fixed export. 3. Switch on around 5 November with the kill switch ready. Review refunds and dietary complaints at one week.

London, South East, Midlands: no-go for 2 November. Acceptance and refunds are fine, and basket is up. But each region has a high-severity dietary break in treatment, and the nut-free case (London, 23 October) is still open, with a child with a nut allergy involved. Before we re-decide on 6 November, we need: 1. The hard filter live and regression-tested. 2. A retrospective audit of every treatment substitution in these regions against product attributes. The logged incidents are only the ones customers reported. 3. Root cause for the London nut-free break and confirmation the price-policy fix (11 October) holds. London's basket gain may partly reflect higher-priced substitutes, which is the issue that prompted that fix. 4. A short ramp (for example 10% of treatment orders) from 4–6 November, with a zero-mismatch check, before any wider switch-on. If the audit finds further breaks, these regions stay off.

Scotland: no-go. The ops director's "settling down" is not what the data shows. Before the supplier change on 15 October, treatment acceptance was 86.3% and refunds were 17.2 per 1,000, the same as control (17.3). Since 15 October, acceptance has been flat at about 79% (79.0% in week two, 79.2% in week three) and refunds at 43 per 1,000 against 19 for control. Refund value since 15 October is about £2.8k. Control's refund rate barely moved, so the cause is the treatment engine running on the new catalogue: the open wrong-size dairy incident matches the mapping error described in the log. The re-map has no date. - Keep Scotland at 0% treatment. - Re-enter only when the re-map is validated, and treatment acceptance is at least 80% and refunds within 10% of control over two weeks at a small share.

Asks for the meeting

  1. Agree that no region goes live on 2 November, and that criterion 3 is not waived.
  2. Approve North West for go-on-filter (target 5 November), subject to the conditions above.
  3. Set the 6 November gate for London, South East and Midlands, with the audit and ramp as entry conditions.
  4. Keep Scotland on control. Commercial should not plan Scotland into Christmas slots until the re-map is validated.
  5. Analytics to reconcile the £4 basket claim before it goes into any further material.

The Christmas booking window opens on 9 November. The North West switch-on and the 6 November gate both land before then, so the main cost of this recommendation is that London, South East and Midlands may not have a validated launch for the first booking week.

Grades and run details

Decision model 79 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly25%
  • passAddresses the actual decision72%
  • passRespects explicit constraints23%
  • passIdentifies material uncertainty85%
  • partialAvoids unsupported claims38%
  • passProduces the required deliverable61%
  • passChecks every criterion, region by region34%
  • passFinds the Scotland breach100%
  • passExplains the basket gap70%
  • passCatches the duplicate rows100%
  • passMakes a clear call85%
  • partialLimits the downside of being wrong95%
  • failGets the base of every number right48%
  • passJudges the launch against a baseline set now92%
Run
Run
#1
API response time
1.9 min
Submitted
8 Oct 2026
GPT-6 LunaAPI

Usable with a quick edit10 of 14 checks passed

The core recommendation, evidence, and region-by-region logic are solid. A PM could add a monitoring threshold and review date with a quick edit without redoing any analysis.

Output

Recommendation: No-go for the all-region rollout on 2 November

Do not enable auto-substitutions anywhere on 2 November: the dietary hard filter is not scheduled to deploy until 4 November, so the zero-mismatch safety criterion is not yet protected. After deployment and validation, North West is the only region I’d consider enabling first, with conditions. Hold the other regions for the reasons below.

Figures are for 1–28 October; refund rates are per 1,000 orders. Basket averages are order-weighted within each region. I excluded the three exact duplicate North West treatment rows on 8, 9 and 10 October.

RegionTreatment acceptance (≥80%)Refunds / 1,000 orders: treatment vs control (treatment ≤10% higher)Dietary/allergen mismatches (must be zero)Average basket: treatment vs control (must not be lower)Decision
------:---:------:---
London~86% Pass~17.9 vs ~18.4; Pass1; Fail — nut-containing bar substituted for nut-free item~£72.8 vs ~£72.5; PassNo-go
South East~85% Pass~17.8 vs ~17.9; Pass1; Fail — dairy cheese substituted for vegan cheese~£66.6 vs ~£66.3; PassNo-go
Midlands~86% Pass~17.6 vs ~18.0; Pass1; Fail — standard loaf substituted for gluten-free loaf~£55.6 vs ~£55.4; PassNo-go
North West~86% Pass~17.8 vs ~18.8; Pass0 observed; Pass in test~£57.5 vs ~£57.3; PassGo with conditions, after filter validation
Scotland~79% Fail1~29.8 vs ~18.4; Fail — about 62% higher0 observed; Pass in test~£53.0 vs ~£52.6; PassNo-go

Conditions and rationale

  • North West: After the hard filter deploys, pass regression tests across all six protected attributes (gluten-free, vegan, vegetarian, nut-free, halal and kosher); then enable regionally with close monitoring and a rollback switch. The export duplicates should also be fixed or guarded against before relying on daily reporting.
  • London, South East and Midlands: Do not launch until the filter is deployed and the dietary-mismatch failures are investigated and validated as fixed. Their good acceptance, refund and basket figures do not offset a safety-criterion failure.
  • Scotland: Hold until the supplier catalogue re-map is complete and verified, and a fresh test meets both acceptance and refund guardrails. The sharp rise in treatment refunds and lower acceptance coincide with the dairy supplier/catalogue change; the issue is open, with no completion date.

The pooled “£4 smaller” basket claim should not drive this decision: treatment exposure differed by region, and baskets vary substantially by region. Within each region, the treatment average is slightly higher in these data. Likewise, an overall acceptance or refund result cannot override region-level launch gates. The 9 November booking window is important, but it does not justify launching before the safety control is live.

What a PM had to fix

  1. 1Invented evidenceVerify or remove the claimTargeted repair

    What we’d change82.76%: Pass over 1–28 October. However, acceptance fell to 79.08% over 15–28 October after the supplier change, below the launch threshold.

Grades and run details

Decision model 68 · LLM judge 87
Decision model checks
  • failUses the supplied evidence correctly10%
  • passAddresses the actual decision48%
  • passRespects explicit constraints58%
  • partialIdentifies material uncertainty52%
  • passAvoids unsupported claims31%
  • passProduces the required deliverable69%
  • passChecks every criterion, region by region27%
  • passFinds the Scotland breach88%
  • passExplains the basket gap85%
  • partialCatches the duplicate rows80%
  • partialMakes a clear call72%
  • partialLimits the downside of being wrong97%
  • failGets the base of every number right40%
  • partialJudges the launch against a baseline set now31%
Run
Run
#1
API response time
44 s
Submitted
29 Sept 2026
GPT-6 AstraChatGPT

Usable with a quick edit12 of 14 checks passed

The output is well-structured, catches duplicates, explains the mix artefact, and provides a clear call. A PM could use it with minor edits, such as adding a note that the duplicates do not change the result.

Output

Recommendation for 30 October: no-go in every region on 2 November. Conditional phased rollout thereafter in London, South East, Midlands and North West. Scotland remains blocked pending catalogue repair and fresh evidence.

The dietary filter is scheduled for 4 November, two days after the proposed launch. Three treatment substitutions breached dietary requirements, including gluten-free bread replaced with standard bread and nut-free bars replaced with hazelnut bars. The shared system needs this control before expansion, including in regions with no recorded mismatch.

I would pause existing auto-substitution exposure and return to picker selection until the filter is deployed and verified.

Regional checks, 1–28 October

I removed three exact duplicate North West treatment rows dated 8–10 October4, consistent with the export incident, leaving 280 unique region–arm–day records. Acceptance and refund rates use summed numerators and denominators. Basket values are weighted by orders.

T = treatment; C = control. Refund changes are relative changes in requests per 1,000 orders.

RegionAcceptance T ≥80%Refunds/1,000: T vs C; change ≤+10%Dietary mismatches: zeroBasket T vs C; difference ≥£0Decision for 2 Nov
London86.68%: pass17.44 vs 18.04; −3.33%: pass1: fail£72.70 vs £72.49; +£0.21: passNo-go
South East86.40%: pass17.54 vs 17.89; −1.93%: pass1: fail£66.64 vs £66.20; +£0.44: passNo-go
Midlands86.35%: pass17.48 vs 17.89; −2.25%: pass1: fail£55.66 vs £55.35; +£0.31: passNo-go
North West86.28%: pass17.58 vs 18.47; −4.80%: pass0 recorded: observed pass£57.51 vs £57.29; +£0.22: passNo-go: shared safety gap
Scotland82.76%: pass over full test230.16 vs 18.20; +65.70%: fail0 recorded: observed pass£52.98 vs £52.62; +£0.36: passNo-go: safety and catalogue

Zero recorded incidents does not establish that an unprotected region is safe. Refunding a customer closes the complaint, not the underlying failure.

The aggregate claims conceal different problems.

Commercial’s acceptance claim is broadly correct: pooled acceptance rose from 71.20% to 85.93%, a 14.74 percentage-point improvement. Pooled refunds increased 7.86%, inside the overall guardrail. But the agreed criterion applies to every region. Scotland fails substantially.

Analytics’ £4 concern is also numerically correct but misleading as a treatment comparison: pooled baskets are £61.09 in treatment versus £65.15 in control. Treatment contains proportionally fewer high-value London orders and more lower-value Midlands and Scotland orders because allocation varies by region. Within every region, treatment baskets are higher across the full test. Applying the same combined regional order mix to both arms gives £63.26 versus £62.96, approximately £0.30 higher in treatment. This resolves the composition distortion; it does not establish a statistically reliable basket uplift.

Scotland is not settling down.

After the supplier switch on 15 October, treatment acceptance fell from 86.34% to 79.08%, below the launch threshold. Refunds rose from 17.25 to 43.25 per 1,000 orders, versus 17.79 in concurrent control, approximately 143% higher.

The final seven days still show 79.17% acceptance and 43.64 refunds per 1,000. The full-period acceptance pass masks a persistent failure in the current catalogue. The open pack-size mapping incident provides a plausible mechanism, and the repair has no date.

Conditions for reconsideration

  • Engineering and QA, by 4 November: deploy and verify the dietary filter across all six protected attributes. Replay the three incident cases, test missing or conflicting attributes and confirm unsafe or unverified candidates cannot be automatically selected. A merged change is insufficient.
  • PM, Analytics and regional Ops, review on 6 November:3 assess a limited post-fix pilot in London, South East, Midlands and North West, retaining randomised controls. Recheck every regional criterion, audit dietary compliance and verify London’s pricing fix. Expand only where the evidence supports it; insufficient evidence means continued limited exposure.1
  • Scotland Ops and catalogue Engineering, owner and repair date agreed on 30 October: complete and validate the re-map before restarting a controlled pilot. Judge readiness on post-repair results, not the earlier healthy fortnight.
  • During any restart: immediately disable auto-substitutions for a dietary mismatch and pause affected regions for guardrail breaches, with daily regional monitoring.

The 9 November booking window is a commercial deadline. It supplies no evidence that either unresolved defect will be fixed in time.

What a PM had to fix

  1. 1Decision deferredMake the callTargeted repair

    What we’d changeGive Commercial something to plan around: say whether the four clean regions can go by 9 November, set a threshold for pausing a region, and give Scotland a Christmas fallback.

  2. 2Contradiction missedSurface the contradictionQuick edit

    What we’d changeShow Scotland failing acceptance since the supplier switch in the table, as the text does, rather than passing.

  3. 3Test or gate too weakTighten the testQuick edit

    What we’d changeA 6 November review would have only a day or two of post-filter data. Allow enough to judge.

  4. 4OtherFix itQuick edit

    What we’d changeSay whether removing them changes any result. The brief asks for it.

Grades and run details

Decision model 86 · LLM judge 93
Decision model checks
  • passUses the supplied evidence correctly16%
  • passAddresses the actual decision93%
  • passRespects explicit constraints30%
  • passIdentifies material uncertainty87%
  • passAvoids unsupported claims28%
  • passProduces the required deliverable82%
  • passChecks every criterion, region by region67%
  • passFinds the Scotland breach100%
  • passExplains the basket gap98%
  • partialCatches the duplicate rows50%
  • passMakes a clear call68%
  • partialLimits the downside of being wrong62%
  • failGets the base of every number right22%
  • passJudges the launch against a baseline set now96%
Run
Run
#1
Time to output
71 s
Submitted
25 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Uses the supplied evidence correctlyMixedMixedRight
Haiku 5.5 · API

Every factual statement about the current situation is drawn from the brief or the attached files, with no invented facts.

GPT-6 Luna · API

All factual claims about the current situation are drawn directly from the supplied data or incidents log, or computed correctly from them.

GPT-6 Astra · ChatGPT

All factual claims are taken directly from the brief or derived correctly from the supplied data.

Catches the duplicate rowsRightRightWrong
Haiku 5.5 · API

Notices the duplicated North‑West rows for 8–10 October, removes them, and states that including them barely changes the result.

GPT-6 Luna · API

It catches the duplicate North West treatment rows for 8–10 October, excludes them, and notes the fix needed.

GPT-6 Astra · ChatGPT

Output removes the duplicate North West rows but does not state whether they change the results, as the criterion requires.

Limits the downside of being wrongRightWrongRight
Haiku 5.5 · API

Describes kill switch, ramp with zero‑mismatch check, and review of refunds/complaints at one week, limiting downside if things go wrong.

GPT-6 Luna · API

It does not name a specific post-launch signal, a threshold that would trigger pause or rollback, or flag any irreversible actions.

GPT-6 Astra · ChatGPT

Specifies post-launch monitoring (daily regional), triggers (dietary mismatch, guardrail breach), and action (disable, pause).

Judges the launch against a baseline set nowRightWrongRight
Haiku 5.5 · API

The launch criteria themselves serve as the baseline, and specific review points (5 Nov for NW, 6 Nov for others) with conditions are set.

GPT-6 Luna · API

It does not name a success metric with its current baseline, nor a review point with what each result would lead to.

GPT-6 Astra · ChatGPT

Names a review point (6 Nov), the criteria to recheck, and what each outcome leads to (expand or keep limited).

All mixed 1

Gets the base of every number rightMixedMixedMixed
Haiku 5.5 · API

All percentages and rates (acceptance, refunds per 1 000) are computed from the correct totals, and basket figures are checked within region.

GPT-6 Luna · API

All rates and percentages are computed with the correct denominators (e.g., refunds per 1,000 orders) and the base is stated where needed.

GPT-6 Astra · ChatGPT

All derived percentages, rates, and basket values check out against the supplied data with correct denominators.

All got right 9

Addresses the actual decisionRightRightRight
Haiku 5.5 · API

Commits to a clear, early call (no‑go for all on 2 Nov, conditional go for NW, hold others, no‑go for Scotland) and states what would change it.

GPT-6 Luna · API

The output commits to a clear no-go for 2 November, gives a region-by-region call with conditions, and states what would change each decision.

GPT-6 Astra · ChatGPT

Commits to a clear no-go on 2 Nov, conditional phased rollout thereafter, and states what would change the call.

Respects explicit constraintsRightRightRight
Haiku 5.5 · API

Respects length (under 900 words), includes the required region‑by‑region criteria table, and addresses the launch meeting audience.

GPT-6 Luna · API

The output includes the required region-by-region criteria table, stays under 900 words, and addresses the launch meeting audience.

GPT-6 Astra · ChatGPT

Prose is under 900 words, includes the required region-by-region criteria table, and addresses the launch meeting.

Identifies material uncertaintyRightRightRight
Haiku 5.5 · API

Identifies unknowns (duplicate export, audit may find further breaks, catalogue re‑map date, basket metric definition) and says how they would be resolved.

GPT-6 Luna · API

It identifies the dietary filter deployment date, the Scotland catalogue re-map with no date, and the duplicate data issue as key unknowns, and says how they would change the call.

GPT-6 Astra · ChatGPT

Names the dietary filter deployment, Scotland catalogue repair, and post-fix pilot results as unknowns, with conditions for reconsideration.

Avoids unsupported claimsRightRightRight
Haiku 5.5 · API

Speculation about causes is labelled with 'may' and 'partly', and assertions are backed by the data.

GPT-6 Luna · API

Interpretations like the basket gap being a mix artefact are clearly explained with evidence; no unsupported causes or forecasts are presented as fact.

GPT-6 Astra · ChatGPT

Interpretations like the mix effect and Scotland's mechanism are labelled as such; no confident claims go beyond the evidence.

Produces the required deliverableRightRightRight
Haiku 5.5 · API

Delivers a complete recommendation with a criteria table, region‑by‑region calls, conditions, and actions for the meeting.

GPT-6 Luna · API

The recommendation is complete, in the right form for a launch meeting, and a PM could act on it with light edits.

GPT-6 Astra · ChatGPT

The recommendation is complete, in the right form for a senior PM, and could be acted on with light edits.

Checks every criterion, region by regionRightRightRight
Haiku 5.5 · API

The table checks all four criteria for each region with figures computed from the files (acceptance, refunds per 1,000, dietary incidents, basket).

GPT-6 Luna · API

The table checks all four launch criteria for each region with figures computed from the files.

GPT-6 Astra · ChatGPT

Table checks all four launch criteria for each region with figures computed from the files.

Finds the Scotland breachRightRightRight
Haiku 5.5 · API

Finds Scotland's refund breach (30.2 vs 18.2, +66%), ties it to the supplier change, notes acceptance dip to 79%, and holds Scotland.

GPT-6 Luna · API

It finds Scotland's refund breach (~62% above control), ties it to the supplier change after 15 October, notes the acceptance dip, and holds Scotland.

GPT-6 Astra · ChatGPT

Identifies Scotland's refund breach, ties it to the supplier change, notes the acceptance dip, and holds Scotland.

Explains the basket gapRightRightRight
Haiku 5.5 · API

Shows that within each region treatment baskets equal or exceed control, explains the pooled £4 gap as an artefact of the uneven split.

GPT-6 Luna · API

It shows treatment baskets are equal or higher within every region and explains the pooled gap as an artefact of the uneven split.

GPT-6 Astra · ChatGPT

Shows treatment baskets are higher in every region and explains the pooled gap as a mix effect from the uneven split.

Makes a clear callRightRightRight
Haiku 5.5 · API

Makes one clear call per region with dates (NW go 5 Nov, L/SE/Mid re‑decide 6 Nov, Scotland no‑go) and states what would change each call.

GPT-6 Luna · API

It makes one clear call per region (no-go, go with conditions, hold) with dates and states what would change each.

GPT-6 Astra · ChatGPT

Makes one clear call per region with dates and conditions, and says what would change it.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 63% of 68 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5API87.8100.02None
2Haiku 5.5API82.5100.02None
3GPT-6.1 SolAPI86.095.82None
4Opus 5.5Claude84.396.72None
5GPT-6 AstraChatGPT83.892.52None
6GPT-6 LunaAPI74.880.82None
7Gemini 3.8 FlashAPI64.967.521 capped
8Gemini 3.5 Flash-LiteGemini48.259.22None

What you get for the cost

Haiku 5.5 · API scores within 2 points of Sonnet 5.5 · API at about a tenth of the cost.

Task score against relative cost for each setup. The table below has the same figures.4050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreSonnet 5.5 · API: task score 94, 37× the cheapestHaiku 5.5 · API: task score 92, 2.8× the cheapestGPT-6.1 Sol · API: task score 91, 12× the cheapestOpus 5.5 · Claude: task score 91, 59× the cheapest (1 of 2 outputs estimated)GPT-6 Astra · ChatGPT: task score 89, 46× the cheapest (1 of 2 outputs estimated)GPT-6 Luna · API: task score 77, the cheapestGemini 3.8 Flash · API: task score 55, 11× the cheapest, 1 critical failureGemini 3.5 Flash-Lite · Gemini: task score 52, 2.7× the cheapest (1 of 2 outputs estimated)Sonnet 5.5 · APIHaiku 5.5 · APIGPT-6.1 Sol · APIOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTGPT-6 Luna · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)At least one critical failure

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Deciding whether a feature ships on the planned date, and on what conditions.

Why it matters

Launch meetings reward optimism. A good call checks each agreed criterion, weighs a real risk against a real gain, and says exactly what would change the answer. A weak one rubber-stamps the launch or blocks it on noise.

What good looks like

  • Checks each agreed criterion against the evidence
  • Makes one clear call, with conditions if needed
  • Separates risks that block launch from risks that can be managed
  • Says what would change the call
  • Says how to limit the damage if the call is wrong

Deliberately not measured

  • Rollout engineering detail
  • Project-plan formatting
Capability tested

Deciding against agreed launch criteria

The failure we’re looking for

Rubber-stamps a launch that misses an agreed bar, or blocks it on noise

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

A launch memo from supplied evidence · Staff level: a regional call from an attached data workbook