Make the launch call

Recommendation for 30 October: no-go in every region on 2 November. Conditional phased rollout thereafter in London, South East, Midlands and North West. Scotland remains blocked pending catalogue repair and fresh evidence.

The dietary filter is scheduled for 4 November, two days after the proposed launch. Three treatment substitutions breached dietary requirements, including gluten-free bread replaced with standard bread and nut-free bars replaced with hazelnut bars. The shared system needs this control before expansion, including in regions with no recorded mismatch.

I would pause existing auto-substitution exposure and return to picker selection until the filter is deployed and verified.

Regional checks, 1–28 October

I removed three exact duplicate North West treatment rows dated 8–10 October⁴, consistent with the export incident, leaving 280 unique region–arm–day records. Acceptance and refund rates use summed numerators and denominators. Basket values are weighted by orders.

T = treatment; C = control. Refund changes are relative changes in requests per 1,000 orders.

Zero recorded incidents does not establish that an unprotected region is safe. Refunding a customer closes the complaint, not the underlying failure.

The aggregate claims conceal different problems.

Commercial’s acceptance claim is broadly correct: pooled acceptance rose from 71.20% to 85.93%, a 14.74 percentage-point improvement. Pooled refunds increased 7.86%, inside the overall guardrail. But the agreed criterion applies to every region. Scotland fails substantially.

Analytics’ £4 concern is also numerically correct but misleading as a treatment comparison: pooled baskets are £61.09 in treatment versus £65.15 in control. Treatment contains proportionally fewer high-value London orders and more lower-value Midlands and Scotland orders because allocation varies by region. Within every region, treatment baskets are higher across the full test. Applying the same combined regional order mix to both arms gives £63.26 versus £62.96, approximately £0.30 higher in treatment. This resolves the composition distortion; it does not establish a statistically reliable basket uplift.

Scotland is not settling down.

After the supplier switch on 15 October, treatment acceptance fell from 86.34% to 79.08%, below the launch threshold. Refunds rose from 17.25 to 43.25 per 1,000 orders, versus 17.79 in concurrent control, approximately 143% higher.

The final seven days still show 79.17% acceptance and 43.64 refunds per 1,000. The full-period acceptance pass masks a persistent failure in the current catalogue. The open pack-size mapping incident provides a plausible mechanism, and the repair has no date.

Conditions for reconsideration

• Engineering and QA, by 4 November: deploy and verify the dietary filter across all six protected attributes. Replay the three incident cases, test missing or conflicting attributes and confirm unsafe or unverified candidates cannot be automatically selected. A merged change is insufficient.
• PM, Analytics and regional Ops, review on 6 November:³ assess a limited post-fix pilot in London, South East, Midlands and North West, retaining randomised controls. Recheck every regional criterion, audit dietary compliance and verify London’s pricing fix. Expand only where the evidence supports it; insufficient evidence means continued limited exposure.¹
• Scotland Ops and catalogue Engineering, owner and repair date agreed on 30 October: complete and validate the re-map before restarting a controlled pilot. Judge readiness on post-repair results, not the earlier healthy fortnight.
• During any restart: immediately disable auto-substitutions for a dietary mismatch and pause affected regions for guardrail breaches, with daily regional monitoring.

The 9 November booking window is a commercial deadline. It supplies no evidence that either unresolved defect will be fixed in time.

AI auto-substitutions: launch recommendation

Oct 29, 2026 · @Martin Slaney · For the 30 October launch meeting

Recommendation

No-go for Monday 2 November in any region. The dietary hard filter isn't live until 4 November, and the test produced three allergen and dietary mismatches.

Go with conditions for London, South East, Midlands and North West from Thursday 5 November, once the filter is live and verified. That is still four days before the Christmas booking window opens on 9 November.

No-go for Scotland until the dairy catalogue re-map ships and a fresh week of test data passes the criteria.

Launch criteria by region

Figures are for 1–28 October from daily\_metrics.csv, with duplicate rows removed. T = treatment, C = control.

Why not 2 November

Criterion 3 requires zero dietary or allergen mismatches. Treatment had three, all rated high severity: a gluten-free loaf swapped for a standard one (the customer is coeliac), vegan cheese swapped for dairy cheddar (the customer complained on social media), and a nut-free multipack swapped for one containing hazelnuts, sent to a child with a nut allergy. The nut incident is still open. Control had none.

The fix is the dietary hard filter, which is merged and deploys on 4 November after regression tests. Launching on 2 November would put every order in every region onto the unfiltered model for two days. North West's zero is luck, because it runs the same model.

Strictly, all four "go" regions fail criterion 3 as it's written. I'm asking the meeting to accept the filter as the remedy, on condition that we show it would have blocked all three incidents. If the meeting won't accept that, the honest answer is no-go everywhere.

Waiting three days costs nothing for Christmas, because bookings don't open until 9 November.

What the summaries get wrong

Head of Commercial: "refunds within the guardrail overall." Pooled, treatment refunds are 7.9% above control (19.5 vs 18.1 per 1,000 orders), which is inside 10%. But the criterion applies to every region, and the pooled figure hides Scotland at +66%. In the other four regions, treatment refunds are below control. The 15-point acceptance lift is real in those four regions (+15.0 to +15.4 points). In Scotland it's +11.7 overall and falling.

Head of Analytics: "treatment baskets are £4 smaller." Pooled, the gap is £61.09 vs £65.15, but it comes from how orders were split, not from the feature. Treatment got 70% of orders in Midlands and Scotland, which have the smallest baskets (£52–56), and only 30% in London, which has the largest (£72). Within every region, treatment baskets are £0.21–£0.44 larger, so criterion 4 passes everywhere.

Ops director, Scotland: "it's settling down." It isn't. Before the supplier switch on 15 October, Scotland's treatment arm matched the other regions (86.3% acceptance, 17.3 refunds per 1,000 vs 18.6 in control). Since then, treatment refunds have run at 25–33 a day with no downward trend: 28.1 a day on average from 15 to 21 October and 28.7 from 22 to 28 October. Refund value per order has tripled, from £0.09 to £0.30. Scotland's control arm is unaffected at 17.8 per 1,000. So the problem is specific to the model reading a mis-mapped catalogue: pickers can see the real pack sizes, and the model can't. The re-map has no date.

Data quality

The North West treatment rows for 8, 9 and 10 October each appear twice, because the export job re-ran (the open incident from 7 October). I removed the duplicates. They don't change any verdict: North West acceptance moves from 86.2% to 86.3%.

Conditions for the 5 November go

1. The dietary filter deploys on 4 November and passes regression tests. We replay the three incident orders and confirm the filter blocks each of them.²
2. The London nut-allergen incident is closed with a root cause.
3. We switch on region by region and keep a 10% control holdout in each region through Christmas, so criteria 2 and 4 stay measurable.
4. Kill switch: any single dietary or allergen mismatch turns the region off, as does a treatment refund rate more than 10% above its holdout over a rolling 3 days.¹
5. Scotland: ship the catalogue re-map, then run 7 days of test and re-check the criteria. If it can't be ready by 9 November, Scotland takes Christmas orders with picker substitutions.

Recommendation: AI auto-substitutions, launch meeting 30 October

Decision: no-go for Monday 2 November. Go with conditions from 4 November for London, South East, Midlands and North West. Scotland stays on picker substitutions until it re-passes.

Criteria by region

Figures are recomputed from the daily rows. Treatment (T) and control (C) are pooled over the 28 days.

Baskets are means of daily averages. The differences are small and I have not tested them for significance, but the point estimate is positive everywhere.

Checking what people have said

• Head of Commercial. The pooled numbers are right: acceptance is 85.9% against 71.2% (+14.7 points), and refunds are 19.5 against 18.1 per 1,000 (+8%). But the pooled refund figure hides Scotland, which is failing badly. Four regions are 2–5% better than control on refunds.
• Head of Analytics. The £4 gap is real in the pooled data (£61.10 vs £65.15) but it is a mix effect, not a treatment effect. London, with the highest baskets (about £72), was 42% of control orders but only 18% of treatment orders. The Midlands and Scotland (about £53–56) were 18% of control but 42% of treatment. Within every region, treatment baskets are £0.24–£0.46 higher. Criterion 4 passes. The uneven split did not bias the regional comparisons.
• Ops director (Scotland). The data does not support "settling down". Before the supplier switch on 15 October, Scotland treatment was healthy (86.3% acceptance, 17.3 refunds per 1,000). From 15 October to 28 October, acceptance has been flat at 78–81%, and daily refunds have been 25–33 against a pre-switch level of 10–14. The cause is the open incident: the new supplier's pack sizes are mapped to the wrong product family, so the model makes wrong-size dairy substitutions. That is a catalogue defect with no fix date, not noise. Scotland control is unaffected.

Why the date moves

Criterion 3 fails in three regions. The three mismatches include a nut-allergy case and a coeliac case. The hard filter that prevents them deploys on 4 November. Until it is live, every region has the same exposure. North West and Scotland show zero only because none were reported, and the incidents come from customer complaints, so the true rate is unknown. I would not switch on a known allergen risk two days before the fix.

Waiting is cheap. The booking window opens on 9 November, not 2 November. Deploying the filter on 4 November and switching regions on 4–5 November leaves several days before the window. Only Scotland has no route to launch before then.

Conditions for the four regions

1. The filter deploys on 4 November and passes regression tests. Before any region goes live, it is tested in production against the three incident SKUs (gluten-free loaf, vegan cheese, nut-free bar) and the other attributes: vegetarian, halal and kosher. The London incident is still open with the filter "in review", so confirm the filter would have blocked it.
2. Regions are switched on region by region, North West and South East first, then London and the Midlands. This gives a day of live data before the larger volumes go on.
3. Any dietary or allergen mismatch after launch switches that region back to picker substitution immediately, pending review. That includes any mismatch in the first week.
4. Daily monitoring of acceptance (80% floor) and refunds (10% guardrail) per region against a small held-back control. Delegate the switch-on decision so it does not need another meeting.

Scotland

Keep Scotland on picker substitutions. Retest only after the catalogue re-map ships, on a small treatment share. It should show at least three consecutive days at 80% or better acceptance, and refunds within 10% of control, before it goes to full volume. Engineering has no date, so Scotland should not be in the Christmas plan and should be treated as upside.

Data notes

• North West treatment has three duplicated rows (8, 9 and 10 October), matching the open export-job incident. I removed them. This restores the intended 50% split, and the North West rates barely move either way.
• Incident logs are a floor on dietary failures, not a measure of them.
• This is 28 days of data in a low-volume region (Scotland control is about 265 orders a day). Even so, the Scotland gap is far outside noise.