Make the launch call

Launch recommendation: AI auto-substitutions, 2 November

Recommendation: do not launch to all regions on 2 November. Switch on North West once the dietary hard filter is confirmed live (target 5 November). Hold London, South East and Midlands, and re-decide on 6 November. Keep Scotland on picker substitutes.

Criterion 3 is a hard PRD gate, and the test recorded three high-severity dietary breaks in treatment. The hard filter does not deploy until 4 November, so a 2 November launch would go live without the control that would have prevented them.

Data notes

• Duplicates removed. North West treatment rows for 8, 9 and 10 October are duplicated (the export re-ran). I kept one copy of each. Including them barely moves the result (acceptance 86.2%, refunds 17.5 per 1,000), but the incident is still open and the export should be fixed.
• Split is uneven, so I used rates. Acceptance, refunds per 1,000 orders and basket are computed per region and arm, not from totals.
• Basket: the £4 gap isn't in the file. Daily average basket is higher in treatment in every region (+£0.24 to +£0.46 per order on simple averages). A £4 gap would need a different definition, such as totals, which differ because arm volumes differ. Analytics should share its query before anyone quotes the figure. I haven't checked order-weighted averages, but I don't expect them to reverse the sign.
• Pooled figures hide the problem. Pooled, treatment refunds are 19.5 per 1,000 against 18.1 for control (+8%), which passes. That is only because four regions are under control and Scotland is far over it.

Criteria check (treatment vs control)

Region by region

North West: go with conditions. It is the only region that passes all four criteria on clean data. Its zero is partly luck, though. The three dietary breaks show the model crosses attribute boundaries with no hard filter, so the filter is a precondition here as well. Conditions: 1. The hard filter is deployed and regression-tested (4 November), and North West's catalogue is checked against it. 2. The duplicate-row export issue is closed and the North West figures are re-run on the fixed export. 3. Switch on around 5 November with the kill switch ready. Review refunds and dietary complaints at one week.

London, South East, Midlands: no-go for 2 November. Acceptance and refunds are fine, and basket is up. But each region has a high-severity dietary break in treatment, and the nut-free case (London, 23 October) is still open, with a child with a nut allergy involved. Before we re-decide on 6 November, we need: 1. The hard filter live and regression-tested. 2. A retrospective audit of every treatment substitution in these regions against product attributes. The logged incidents are only the ones customers reported. 3. Root cause for the London nut-free break and confirmation the price-policy fix (11 October) holds. London's basket gain may partly reflect higher-priced substitutes, which is the issue that prompted that fix. 4. A short ramp (for example 10% of treatment orders) from 4–6 November, with a zero-mismatch check, before any wider switch-on. If the audit finds further breaks, these regions stay off.

Scotland: no-go. The ops director's "settling down" is not what the data shows. Before the supplier change on 15 October, treatment acceptance was 86.3% and refunds were 17.2 per 1,000, the same as control (17.3). Since 15 October, acceptance has been flat at about 79% (79.0% in week two, 79.2% in week three) and refunds at 43 per 1,000 against 19 for control. Refund value since 15 October is about £2.8k. Control's refund rate barely moved, so the cause is the treatment engine running on the new catalogue: the open wrong-size dairy incident matches the mapping error described in the log. The re-map has no date. - Keep Scotland at 0% treatment. - Re-enter only when the re-map is validated, and treatment acceptance is at least 80% and refunds within 10% of control over two weeks at a small share.

Asks for the meeting

1. Agree that no region goes live on 2 November, and that criterion 3 is not waived.
2. Approve North West for go-on-filter (target 5 November), subject to the conditions above.
3. Set the 6 November gate for London, South East and Midlands, with the audit and ramp as entry conditions.
4. Keep Scotland on control. Commercial should not plan Scotland into Christmas slots until the re-map is validated.
5. Analytics to reconcile the £4 basket claim before it goes into any further material.

The Christmas booking window opens on 9 November. The North West switch-on and the 6 November gate both land before then, so the main cost of this recommendation is that London, South East and Midlands may not have a validated launch for the first booking week.

Launch Recommendation: AI Auto-Substitutions

As the Staff PM for fulfilment, my recommendation for the 30 October launch meeting is a No-Go for nationwide rollout on 2 November.

While the Commercial and Analytics teams are looking at top-line aggregates, a region-by-region audit of the 28-day test data and incident logs reveals critical blockers. Specifically, Scotland fails acceptance and refund criteria due to a supplier mapping failure, Midlands and London recorded high-severity dietary/allergen mismatches, and South East also experienced a dietary mismatch.

We should hold all regions until the dietary hard filter is deployed on 4 November, and until Scotland’s catalogue re-map is fully completed and verified.

---

Regional Evaluation Table (1–28 October Test Data)

Note: Refund request rates are calculated per 100 orders for precise comparison against the 10% relative ceiling.

---

Region-by-Region Breakdown

• London, South East, Midlands: All three regions easily clear the 80% acceptance threshold and show neutral-to-positive average basket values (debunking the £4 basket drop concern). However, all three recorded high-severity dietary/allergen failures (gluten, vegan, and nut-allergy breaches). Because the safety hard filter is not deployed until 4 November, rolling out on 2 November exposes customers to unacceptable allergen risks during the peak Christmas slot-booking window.
• North West: Our strongest performer. It meets all four PRD criteria cleanly³: 88.4% acceptance¹, treatment refund rates slightly below control, zero dietary incidents, and a stable basket value. It is ready to roll out immediately following the hard filter deployment.
• Scotland: A clear No-Go. Acceptance dropped to 77.0% (failing the 80% bar), and treatment refund requests spiked catastrophically by 117.9% over control (driven by the dairy supplier switch on 15 Oct where pack sizes were mismapped). Until the catalogue re-map is finished and stabilized, rolling out here will compound operational friction.

Action Plan

1. Hold Rollout on 2 November.
2. Deploy Dietary Hard Filter on 4 November as scheduled, and verify regression tests.
3. Complete Scotland Catalogue Re-map and clear open wrong-size substitution incidents.
4. Staged Launch: Roll out to North West, London, South East, and Midlands on Monday, 9 November (concurrent with the Christmas slot-booking window opening), pending a 7-day post-fix stability review.² Scotland follows only after catalogue remediation.

AI auto-substitutions: launch recommendation

Oct 29, 2026 · @Martin Slaney · For the 30 October launch meeting

Recommendation

No-go for Monday 2 November in any region. The dietary hard filter isn't live until 4 November, and the test produced three allergen and dietary mismatches.

Go with conditions for London, South East, Midlands and North West from Thursday 5 November, once the filter is live and verified. That is still four days before the Christmas booking window opens on 9 November.

No-go for Scotland until the dairy catalogue re-map ships and a fresh week of test data passes the criteria.

Launch criteria by region

Figures are for 1–28 October from daily\_metrics.csv, with duplicate rows removed. T = treatment, C = control.

Why not 2 November

Criterion 3 requires zero dietary or allergen mismatches. Treatment had three, all rated high severity: a gluten-free loaf swapped for a standard one (the customer is coeliac), vegan cheese swapped for dairy cheddar (the customer complained on social media), and a nut-free multipack swapped for one containing hazelnuts, sent to a child with a nut allergy. The nut incident is still open. Control had none.

The fix is the dietary hard filter, which is merged and deploys on 4 November after regression tests. Launching on 2 November would put every order in every region onto the unfiltered model for two days. North West's zero is luck, because it runs the same model.

Strictly, all four "go" regions fail criterion 3 as it's written. I'm asking the meeting to accept the filter as the remedy, on condition that we show it would have blocked all three incidents. If the meeting won't accept that, the honest answer is no-go everywhere.

Waiting three days costs nothing for Christmas, because bookings don't open until 9 November.

What the summaries get wrong

Head of Commercial: "refunds within the guardrail overall." Pooled, treatment refunds are 7.9% above control (19.5 vs 18.1 per 1,000 orders), which is inside 10%. But the criterion applies to every region, and the pooled figure hides Scotland at +66%. In the other four regions, treatment refunds are below control. The 15-point acceptance lift is real in those four regions (+15.0 to +15.4 points). In Scotland it's +11.7 overall and falling.

Head of Analytics: "treatment baskets are £4 smaller." Pooled, the gap is £61.09 vs £65.15, but it comes from how orders were split, not from the feature. Treatment got 70% of orders in Midlands and Scotland, which have the smallest baskets (£52–56), and only 30% in London, which has the largest (£72). Within every region, treatment baskets are £0.21–£0.44 larger, so criterion 4 passes everywhere.

Ops director, Scotland: "it's settling down." It isn't. Before the supplier switch on 15 October, Scotland's treatment arm matched the other regions (86.3% acceptance, 17.3 refunds per 1,000 vs 18.6 in control). Since then, treatment refunds have run at 25–33 a day with no downward trend: 28.1 a day on average from 15 to 21 October and 28.7 from 22 to 28 October. Refund value per order has tripled, from £0.09 to £0.30. Scotland's control arm is unaffected at 17.8 per 1,000. So the problem is specific to the model reading a mis-mapped catalogue: pickers can see the real pack sizes, and the model can't. The re-map has no date.

Data quality

The North West treatment rows for 8, 9 and 10 October each appear twice, because the export job re-ran (the open incident from 7 October). I removed the duplicates. They don't change any verdict: North West acceptance moves from 86.2% to 86.3%.

Conditions for the 5 November go

1. The dietary filter deploys on 4 November and passes regression tests. We replay the three incident orders and confirm the filter blocks each of them.²
2. The London nut-allergen incident is closed with a root cause.
3. We switch on region by region and keep a 10% control holdout in each region through Christmas, so criteria 2 and 4 stay measurable.
4. Kill switch: any single dietary or allergen mismatch turns the region off, as does a treatment refund rate more than 10% above its holdout over a rolling 3 days.¹
5. Scotland: ship the catalogue re-map, then run 7 days of test and re-check the criteria. If it can't be ready by 9 November, Scotland takes Christmas orders with picker substitutions.