Usable with a quick edit10 of 13 checks passed
A PM could use it with light edits, mainly correcting minor arithmetic and removing the unsupported incident-reporting inference. The core decision, table, and conditions are actionable.
Output
Recommendation: AI auto-substitutions, launch meeting 30 October
Decision: no-go for Monday 2 November. Go with conditions from 4 November for London, South East, Midlands and North West. Scotland stays on picker substitutions until it re-passes.
Criteria by region
Figures are recomputed from the daily rows. Treatment (T) and control (C) are pooled over the 28 days.
| Region | 1. Acceptance (T) ≥80% | 2. Refunds per 1,000 orders, T vs C (≤ +10%) | 3. Dietary mismatches (0) | 4. Basket T vs C (not lower) | Verdict |
|---|---|---|---|---|---|
| London | 86.7% ✅ | 17.4 vs 18.0 (−3%) ✅ | 1 (nut allergen, 23 Oct, open) ❌ | £72.72 vs £72.48 (+£0.24) ✅ | Conditional |
| South East | 86.4% ✅ | 17.5 vs 17.9 (−2%) ✅ | 1 (vegan cheese, 19 Oct) ❌ | £66.66 vs £66.20 (+£0.46) ✅ | Conditional |
| Midlands | 86.3% ✅ | 17.5 vs 17.9 (−2%) ✅ | 1 (gluten-free, 4 Oct) ❌ | £55.66 vs £55.35 (+£0.31) ✅ | Conditional |
| North West | 86.3% ✅ | 17.6 vs 18.5 (−5%) ✅ | 0 logged ✅ | £57.52 vs £57.28 (+£0.24) ✅ | Conditional |
| Scotland | 82.8% overall, 79.1% since 15 Oct ❌ | 30.2 vs 18.2 (+66%); since 15 Oct 43.3 vs 17.8 (+143%) ❌ | 0 logged ✅ | £52.98 vs £52.62 (+£0.36) ✅ | No-go |
Baskets are means of daily averages. The differences are small and I have not tested them for significance, but the point estimate is positive everywhere.
Checking what people have said
- Head of Commercial. The pooled numbers are right: acceptance is 85.9% against 71.2% (+14.7 points), and refunds are 19.5 against 18.1 per 1,000 (+8%). But the pooled refund figure hides Scotland, which is failing badly. Four regions are 2–5% better than control on refunds.
- Head of Analytics. The £4 gap is real in the pooled data (£61.10 vs £65.15) but it is a mix effect, not a treatment effect. London, with the highest baskets (about £72), was 42% of control orders but only 18% of treatment orders. The Midlands and Scotland (about £53–56) were 18% of control but 42% of treatment. Within every region, treatment baskets are £0.24–£0.46 higher. Criterion 4 passes. The uneven split did not bias the regional comparisons.
- Ops director (Scotland). The data does not support "settling down". Before the supplier switch on 15 October, Scotland treatment was healthy (86.3% acceptance, 17.3 refunds per 1,000). From 15 October to 28 October, acceptance has been flat at 78–81%, and daily refunds have been 25–33 against a pre-switch level of 10–14. The cause is the open incident: the new supplier's pack sizes are mapped to the wrong product family, so the model makes wrong-size dairy substitutions. That is a catalogue defect with no fix date, not noise. Scotland control is unaffected.
Why the date moves
Criterion 3 fails in three regions. The three mismatches include a nut-allergy case and a coeliac case. The hard filter that prevents them deploys on 4 November. Until it is live, every region has the same exposure. North West and Scotland show zero only because none were reported, and the incidents come from customer complaints, so the true rate is unknown. I would not switch on a known allergen risk two days before the fix.
Waiting is cheap. The booking window opens on 9 November, not 2 November. Deploying the filter on 4 November and switching regions on 4–5 November leaves several days before the window. Only Scotland has no route to launch before then.
Conditions for the four regions
- The filter deploys on 4 November and passes regression tests. Before any region goes live, it is tested in production against the three incident SKUs (gluten-free loaf, vegan cheese, nut-free bar) and the other attributes: vegetarian, halal and kosher. The London incident is still open with the filter "in review", so confirm the filter would have blocked it.
- Regions are switched on region by region, North West and South East first, then London and the Midlands. This gives a day of live data before the larger volumes go on.
- Any dietary or allergen mismatch after launch switches that region back to picker substitution immediately, pending review. That includes any mismatch in the first week.
- Daily monitoring of acceptance (80% floor) and refunds (10% guardrail) per region against a small held-back control. Delegate the switch-on decision so it does not need another meeting.
Scotland
Keep Scotland on picker substitutions. Retest only after the catalogue re-map ships, on a small treatment share. It should show at least three consecutive days at 80% or better acceptance, and refunds within 10% of control, before it goes to full volume. Engineering has no date, so Scotland should not be in the Christmas plan and should be treated as upside.
Data notes
- North West treatment has three duplicated rows (8, 9 and 10 October), matching the open export-job incident. I removed them. This restores the intended 50% split, and the North West rates barely move either way.
- Incident logs are a floor on dietary failures, not a measure of them.
- This is 28 days of data in a low-volume region (Scotland control is about 265 orders a day). Even so, the Scotland gap is far outside noise.
Grades and run details
Decision model 81 · LLM judge 12 of 14 checks
Decision model checks
- failUses the supplied evidence correctly9%
- passAddresses the actual decision98%
- passRespects explicit constraints40%
- passIdentifies material uncertainty92%
- partialAvoids unsupported claims36%
- passProduces the required deliverable79%
- passChecks every criterion, region by region56%
- passFinds the Scotland breach100%
- passExplains the basket gap99%
- passCatches the duplicate rows100%
- passMakes a clear call85%
- passLimits the downside of being wrong76%
- failGets the base of every number right29%
Run
- Run
- #1
- API response time
- 2.3 min
- Submitted
- 29 Sept 2026