Make the launch call

Executive Recommendation

• Monday, 2 November: NO-GO across all regions. We cannot launch while failing Criterion 3 (zero dietary/allergen mismatches). During the test, the AI model caused three high-severity safety incidents—substituting standard bread for gluten-free (coeliac customer), dairy cheese for vegan, and hazelnut bars for nut-free (child with nut allergy). The engineering hard filter to prevent cross-attribute substitution does not deploy until Wednesday, 4 November. A 48-hour delay is non-negotiable for customer safety.
• Thursday, 5 November: GO WITH CONDITIONS for London, South East, Midlands, and North West. Once the dietary hard filter is deployed on 4 November and verified in production, these four regions pass all launch criteria and can roll out to 100% four days ahead of the 9 November Christmas booking window.
• Scotland: NO-GO indefinitely. Keep Scotland on 100% picker-led control until the dairy catalogue re-map is completed and verified via a clean 7-day re-test.

---

Data Hygiene & Addressing Stakeholder Summaries

Before reviewing the criteria table, we must correct three data misconceptions circulating ahead of this meeting:

1. Duplicate Rows (North West): As flagged in `incidents.csv` (7 Oct), an export timeout duplicated North West treatment rows on 8, 9, and 10 October. All figures below are deduplicated (removing 2,194 phantom orders). Note: Criterion conclusions for North West hold either way, though deduplicating slightly raises its refund rate from 17.56 to 17.59 per 1,000.
2. The "£4 Basket Drop" is Simpson’s Paradox (Head of Analytics): Pooled across the test, treatment average basket value (£60.15) is indeed £3.93 lower than control (£64.08). However, this is a mathematical artefact of our unequal regional split: we allocated 70% of orders to treatment in our lowest-basket regions (Scotland £53, Midlands £55) and only 30% in our highest-basket region (London £72). Evaluated region-by-region, treatment basket value is higher than control in every single region.
3. Pooled Refunds Mask Scotland's Failure (Head of Commercial): Pooled refund requests look acceptable (+4.5% vs. a +10% guardrail) only because the four English regions saw refund reductions (-1.5% to -7.8%). This masks a +78.3% refund explosion in Scotland. Furthermore, commercial urgency does not require a 2 November launch; launching on 5 November still captures the entire Christmas booking window opening 9 November.

---

Launch Criteria Performance by Region (1–28 Oct, Deduplicated)

\Note: North West metrics reflect deduplicated data (raw orders: 22,830; raw refunds/1k: 17.56; raw weighted basket: £57.50). Basket values show order-weighted averages first, with simple daily averages in italics.*

---

Regional Rationale & Conditions

#### 1. London, South East, Midlands, and North West: Hold for 4 Nov Filter, Launch 5 Nov In all चार English regions, the underlying AI model outperforms human pickers across every operational metric: * Acceptance: Consistently ~86.5% across all four regions (+15.5 points over control), well above the 80% threshold. * Refunds: Lower than control in every region (ranging from -1.5% in the Midlands to -7.8% in the North West). Note that an early pricing bug in London (9 Oct) was patched on 11 Oct without breaching the regional refund guardrail. * Baskets: Up £0.15 to £0.38 per order over control. * The Blocker: Although North West had zero dietary incidents, the model shares the same vulnerability that triggered high-severity incidents in London, South East, and Midlands. Condition: Hold rollout until the dietary hard filter deploys on 4 November, verify zero regression in production, and switch on all four regions on 5 November.

#### 2. Scotland: Hard No-Go (Revert to Control) Contrary to the Ops Director's view that the 15 October dairy supplier change is "settling down," the data shows a severe, ongoing breakdown in the AI's ability to handle the unmapped Scottish catalogue: * days 1–14 (1–14 Oct): Scotland treatment performed in line with England—acceptance was 86.44% (7,976 / 9,227) and refund requests were 17.46 per 1,000 (162 / 9,280), beating control (18.22). * Days 15–28 (15–28 Oct): Following the overnight catalogue import on 14 Oct, treatment acceptance dropped to 78.05% (6,532 / 8,369)—failing Criterion 1—and never recovered (sitting at 77.76% on 28 Oct). Worse, treatment refund requests more than doubled to 46.45 per 1,000 (417 / 8,977, vs. 17.39 in control), driving a +78.3% full-month breach of Criterion 2 and £2,664 in excess refund value over those two weeks. Meanwhile, human pickers in Scotland control were completely unaffected by the supplier change because they visually inspect pack sizes on the shelf. * Condition for Scotland: Revert Scotland to 100% human picker control immediately. Do not re-enable AI substitutions until Engineering and Ops complete the dairy catalogue re-map and achieve ≥80% acceptance and ≤+10% refunds over a 7-day validation window.

AI auto-substitutions: launch recommendation

Oct 29, 2026 · @Martin Slaney · For the 30 October launch meeting

Recommendation

No-go for Monday 2 November in any region. The dietary hard filter isn't live until 4 November, and the test produced three allergen and dietary mismatches.

Go with conditions for London, South East, Midlands and North West from Thursday 5 November, once the filter is live and verified. That is still four days before the Christmas booking window opens on 9 November.

No-go for Scotland until the dairy catalogue re-map ships and a fresh week of test data passes the criteria.

Launch criteria by region

Figures are for 1–28 October from daily\_metrics.csv, with duplicate rows removed. T = treatment, C = control.

Why not 2 November

Criterion 3 requires zero dietary or allergen mismatches. Treatment had three, all rated high severity: a gluten-free loaf swapped for a standard one (the customer is coeliac), vegan cheese swapped for dairy cheddar (the customer complained on social media), and a nut-free multipack swapped for one containing hazelnuts, sent to a child with a nut allergy. The nut incident is still open. Control had none.

The fix is the dietary hard filter, which is merged and deploys on 4 November after regression tests. Launching on 2 November would put every order in every region onto the unfiltered model for two days. North West's zero is luck, because it runs the same model.

Strictly, all four "go" regions fail criterion 3 as it's written. I'm asking the meeting to accept the filter as the remedy, on condition that we show it would have blocked all three incidents. If the meeting won't accept that, the honest answer is no-go everywhere.

Waiting three days costs nothing for Christmas, because bookings don't open until 9 November.

What the summaries get wrong

Head of Commercial: "refunds within the guardrail overall." Pooled, treatment refunds are 7.9% above control (19.5 vs 18.1 per 1,000 orders), which is inside 10%. But the criterion applies to every region, and the pooled figure hides Scotland at +66%. In the other four regions, treatment refunds are below control. The 15-point acceptance lift is real in those four regions (+15.0 to +15.4 points). In Scotland it's +11.7 overall and falling.

Head of Analytics: "treatment baskets are £4 smaller." Pooled, the gap is £61.09 vs £65.15, but it comes from how orders were split, not from the feature. Treatment got 70% of orders in Midlands and Scotland, which have the smallest baskets (£52–56), and only 30% in London, which has the largest (£72). Within every region, treatment baskets are £0.21–£0.44 larger, so criterion 4 passes everywhere.

Ops director, Scotland: "it's settling down." It isn't. Before the supplier switch on 15 October, Scotland's treatment arm matched the other regions (86.3% acceptance, 17.3 refunds per 1,000 vs 18.6 in control). Since then, treatment refunds have run at 25–33 a day with no downward trend: 28.1 a day on average from 15 to 21 October and 28.7 from 22 to 28 October. Refund value per order has tripled, from £0.09 to £0.30. Scotland's control arm is unaffected at 17.8 per 1,000. So the problem is specific to the model reading a mis-mapped catalogue: pickers can see the real pack sizes, and the model can't. The re-map has no date.

Data quality

The North West treatment rows for 8, 9 and 10 October each appear twice, because the export job re-ran (the open incident from 7 October). I removed the duplicates. They don't change any verdict: North West acceptance moves from 86.2% to 86.3%.

Conditions for the 5 November go

1. The dietary filter deploys on 4 November and passes regression tests. We replay the three incident orders and confirm the filter blocks each of them.²
2. The London nut-allergen incident is closed with a root cause.
3. We switch on region by region and keep a 10% control holdout in each region through Christmas, so criteria 2 and 4 stay measurable.
4. Kill switch: any single dietary or allergen mismatch turns the region off, as does a treatment refund rate more than 10% above its holdout over a rolling 3 days.¹
5. Scotland: ship the catalogue re-map, then run 7 days of test and re-check the criteria. If it can't be ready by 9 November, Scotland takes Christmas orders with picker substitutions.

Recommendation: No-go for the all-region rollout on 2 November

Do not enable auto-substitutions anywhere on 2 November: the dietary hard filter is not scheduled to deploy until 4 November, so the zero-mismatch safety criterion is not yet protected. After deployment and validation, North West is the only region I’d consider enabling first, with conditions. Hold the other regions for the reasons below.

Figures are for 1–28 October; refund rates are per 1,000 orders. Basket averages are order-weighted within each region. I excluded the three exact duplicate North West treatment rows on 8, 9 and 10 October.

Conditions and rationale

• North West: After the hard filter deploys, pass regression tests across all six protected attributes (gluten-free, vegan, vegetarian, nut-free, halal and kosher); then enable regionally with close monitoring and a rollback switch. The export duplicates should also be fixed or guarded against before relying on daily reporting.
• London, South East and Midlands: Do not launch until the filter is deployed and the dietary-mismatch failures are investigated and validated as fixed. Their good acceptance, refund and basket figures do not offset a safety-criterion failure.
• Scotland: Hold until the supplier catalogue re-map is complete and verified, and a fresh test meets both acceptance and refund guardrails. The sharp rise in treatment refunds and lower acceptance coincide with the dairy supplier/catalogue change; the issue is open, with no completion date.

The pooled “£4 smaller” basket claim should not drive this decision: treatment exposure differed by region, and baskets vary substantially by region. Within each region, the treatment average is slightly higher in these data. Likewise, an overall acceptance or refund result cannot override region-level launch gates. The 9 November booking window is important, but it does not justify launching before the safety control is live.