Usable with a quick edit11 of 13 checks passed
A PM could use it with light edits, mainly correcting minor arithmetic and removing the unsupported incident-reporting inference. The core decision, table, and conditions are actionable.
Output
Recommendation: AI auto-substitutions, launch meeting 30 October
Decision: no-go for Monday 2 November. Go with conditions from 4 November for London, South East, Midlands and North West. Scotland stays on picker substitutions until it re-passes.
Criteria by region
Figures are recomputed from the daily rows. Treatment (T) and control (C) are pooled over the 28 days.
| Region | 1. Acceptance (T) ≥80% | 2. Refunds per 1,000 orders, T vs C (≤ +10%) | 3. Dietary mismatches (0) | 4. Basket T vs C (not lower) | Verdict |
|---|---|---|---|---|---|
| London | 86.7% ✅ | 17.4 vs 18.0 (−3%) ✅ | 1 (nut allergen, 23 Oct, open) ❌ | £72.72 vs £72.48 (+£0.24) ✅ | Conditional |
| South East | 86.4% ✅ | 17.5 vs 17.9 (−2%) ✅ | 1 (vegan cheese, 19 Oct) ❌ | £66.66 vs £66.20 (+£0.46) ✅ | Conditional |
| Midlands | 86.3% ✅ | 17.5 vs 17.9 (−2%) ✅ | 1 (gluten-free, 4 Oct) ❌ | £55.66 vs £55.35 (+£0.31) ✅ | Conditional |
| North West | 86.3% ✅ | 17.6 vs 18.5 (−5%) ✅ | 0 logged ✅ | £57.52 vs £57.28 (+£0.24) ✅ | Conditional |
| Scotland | 82.8% overall, 79.1% since 15 Oct ❌ | 30.2 vs 18.2 (+66%); since 15 Oct 43.3 vs 17.8 (+143%) ❌ | 0 logged ✅ | £52.98 vs £52.62 (+£0.36) ✅ | No-go |
Baskets are means of daily averages. The differences are small and I have not tested them for significance, but the point estimate is positive everywhere.
Checking what people have said
- Head of Commercial. The pooled numbers are right: acceptance is 85.9% against 71.2% (+14.7 points), and refunds are 19.5 against 18.1 per 1,000 (+8%). But the pooled refund figure hides Scotland, which is failing badly. Four regions are 2–5% better than control on refunds.
- Head of Analytics. The £4 gap is real in the pooled data (£61.10 vs £65.15) but it is a mix effect, not a treatment effect. London, with the highest baskets (about £72), was 42% of control orders but only 18% of treatment orders. The Midlands and Scotland (about £53–56) were 18% of control but 42% of treatment. Within every region, treatment baskets are £0.24–£0.46 higher. Criterion 4 passes. The uneven split did not bias the regional comparisons.
- Ops director (Scotland). The data does not support "settling down". Before the supplier switch on 15 October, Scotland treatment was healthy (86.3% acceptance, 17.3 refunds per 1,000). From 15 October to 28 October, acceptance has been flat at 78–81%, and daily refunds have been 25–33 against a pre-switch level of 10–14. The cause is the open incident: the new supplier's pack sizes are mapped to the wrong product family, so the model makes wrong-size dairy substitutions. That is a catalogue defect with no fix date, not noise. Scotland control is unaffected.
Why the date moves
Criterion 3 fails in three regions. The three mismatches include a nut-allergy case and a coeliac case. The hard filter that prevents them deploys on 4 November. Until it is live, every region has the same exposure. North West and Scotland show zero only because none were reported, and the incidents come from customer complaints, so the true rate is unknown. I would not switch on a known allergen risk two days before the fix.
Waiting is cheap. The booking window opens on 9 November, not 2 November. Deploying the filter on 4 November and switching regions on 4–5 November leaves several days before the window. Only Scotland has no route to launch before then.
Conditions for the four regions
- The filter deploys on 4 November and passes regression tests. Before any region goes live, it is tested in production against the three incident SKUs (gluten-free loaf, vegan cheese, nut-free bar) and the other attributes: vegetarian, halal and kosher. The London incident is still open with the filter "in review", so confirm the filter would have blocked it.
- Regions are switched on region by region, North West and South East first, then London and the Midlands. This gives a day of live data before the larger volumes go on.
- Any dietary or allergen mismatch after launch switches that region back to picker substitution immediately, pending review. That includes any mismatch in the first week.
- Daily monitoring of acceptance (80% floor) and refunds (10% guardrail) per region against a small held-back control. Delegate the switch-on decision so it does not need another meeting.
Scotland
Keep Scotland on picker substitutions. Retest only after the catalogue re-map ships, on a small treatment share. It should show at least three consecutive days at 80% or better acceptance, and refunds within 10% of control, before it goes to full volume. Engineering has no date, so Scotland should not be in the Christmas plan and should be treated as upside.
Data notes
- North West treatment has three duplicated rows (8, 9 and 10 October), matching the open export-job incident. I removed them. This restores the intended 50% split, and the North West rates barely move either way.
- Incident logs are a floor on dietary failures, not a measure of them.
- This is 28 days of data in a low-volume region (Scotland control is about 265 orders a day). Even so, the Scotland gap is far outside noise.
Check by check
Got wrong · 1
- Gets the base of every number rightSome derived figures are off, including the pooled basket gap and the control-order share for London, and the basket base is stated as means of daily averages rather than order-weighted averages.
Mixed · 1
- Avoids unsupported claimsIt asserts that zero logged dietary incidents in North West and Scotland are only because none were reported and that incidents come from customer complaints, which is not established by the supplied evidence.The two graders disagreed on this one.
Got right · 11
- Uses the supplied evidence correctlyThe output's factual claims are drawn from the brief and supplied CSVs, with only minor rounding and one unsupported inference about incident reporting.
- Addresses the actual decisionIt commits early to no-go on 2 November, conditional go from 4 November for four regions, hold Scotland, and states what would change the call.
- Respects explicit constraintsIt provides a region-by-region recommendation with a launch-criteria table and keeps prose under 900 words.
- Identifies material uncertaintyIt names material unknowns such as filter deployment, Scotland re-map timing, duplicate handling, and incident under-reporting, and gives monitoring/rollback conditions.
- Produces the required deliverableThe requested recommendation and table are present, usable for the launch meeting, and within the length limit.
- Checks every criterion, region by regionIt checks all four launch criteria for each region with figures computed from the files, including the duplicate correction.
- Finds the Scotland breachIt identifies Scotland's refund breach, ties it to the 15 October supplier change, notes the acceptance dip, and holds Scotland.
- Explains the basket gapIt compares baskets within each region and explains the pooled gap as an artefact of the uneven regional split.
- Catches the duplicate rowsIt finds the duplicated North West treatment rows, removes them, and says the result barely changes.
- Makes a clear callIt gives one clear call per region with dates, conditions, and triggers for change.
- Limits the downside of being wrongIt limits downside with staged switch-on, daily monitoring, immediate rollback on any dietary mismatch, and Scotland held out.
Grades and run details
Decision model 92 · LLM judge 12 of 14 checks
Decision model checks
- passUses the supplied evidence correctly14%
- passAddresses the actual decision99%
- passRespects explicit constraints29%
- passIdentifies material uncertainty90%
- passAvoids unsupported claims23%
- passProduces the required deliverable80%
- passChecks every criterion, region by region55%
- passFinds the Scotland breach100%
- passExplains the basket gap99%
- passCatches the duplicate rows100%
- passMakes a clear call84%
- passLimits the downside of being wrong72%
- failGets the base of every number right36%
Run
- Run
- #1
- API response time
- 2.3 min
- Submitted
- 29 Sept 2026