Make the launch call

Launch Recommendation: AI Auto-Substitutions

As the Staff PM for fulfilment, my recommendation for the 30 October launch meeting is a No-Go for nationwide rollout on 2 November.

While the Commercial and Analytics teams are looking at top-line aggregates, a region-by-region audit of the 28-day test data and incident logs reveals critical blockers. Specifically, Scotland fails acceptance and refund criteria due to a supplier mapping failure, Midlands and London recorded high-severity dietary/allergen mismatches, and South East also experienced a dietary mismatch.

We should hold all regions until the dietary hard filter is deployed on 4 November, and until Scotland’s catalogue re-map is fully completed and verified.

---

Regional Evaluation Table (1–28 October Test Data)

Note: Refund request rates are calculated per 100 orders for precise comparison against the 10% relative ceiling.

---

Region-by-Region Breakdown

• London, South East, Midlands: All three regions easily clear the 80% acceptance threshold and show neutral-to-positive average basket values (debunking the £4 basket drop concern). However, all three recorded high-severity dietary/allergen failures (gluten, vegan, and nut-allergy breaches). Because the safety hard filter is not deployed until 4 November, rolling out on 2 November exposes customers to unacceptable allergen risks during the peak Christmas slot-booking window.
• North West: Our strongest performer. It meets all four PRD criteria cleanly³: 88.4% acceptance¹, treatment refund rates slightly below control, zero dietary incidents, and a stable basket value. It is ready to roll out immediately following the hard filter deployment.
• Scotland: A clear No-Go. Acceptance dropped to 77.0% (failing the 80% bar), and treatment refund requests spiked catastrophically by 117.9% over control (driven by the dairy supplier switch on 15 Oct where pack sizes were mismapped). Until the catalogue re-map is finished and stabilized, rolling out here will compound operational friction.

Action Plan

1. Hold Rollout on 2 November.
2. Deploy Dietary Hard Filter on 4 November as scheduled, and verify regression tests.
3. Complete Scotland Catalogue Re-map and clear open wrong-size substitution incidents.
4. Staged Launch: Roll out to North West, London, South East, and Midlands on Monday, 9 November (concurrent with the Christmas slot-booking window opening), pending a 7-day post-fix stability review.² Scotland follows only after catalogue remediation.

Launch recommendation: AI auto-substitutions, 2 November

Recommendation: do not launch to all regions on 2 November. Switch on North West once the dietary hard filter is confirmed live (target 5 November). Hold London, South East and Midlands, and re-decide on 6 November. Keep Scotland on picker substitutes.

Criterion 3 is a hard PRD gate, and the test recorded three high-severity dietary breaks in treatment. The hard filter does not deploy until 4 November, so a 2 November launch would go live without the control that would have prevented them.

Data notes

• Duplicates removed. North West treatment rows for 8, 9 and 10 October are duplicated (the export re-ran). I kept one copy of each. Including them barely moves the result (acceptance 86.2%, refunds 17.5 per 1,000), but the incident is still open and the export should be fixed.
• Split is uneven, so I used rates. Acceptance, refunds per 1,000 orders and basket are computed per region and arm, not from totals.
• Basket: the £4 gap isn't in the file. Daily average basket is higher in treatment in every region (+£0.24 to +£0.46 per order on simple averages). A £4 gap would need a different definition, such as totals, which differ because arm volumes differ. Analytics should share its query before anyone quotes the figure. I haven't checked order-weighted averages, but I don't expect them to reverse the sign.
• Pooled figures hide the problem. Pooled, treatment refunds are 19.5 per 1,000 against 18.1 for control (+8%), which passes. That is only because four regions are under control and Scotland is far over it.

Criteria check (treatment vs control)

Region by region

North West: go with conditions. It is the only region that passes all four criteria on clean data. Its zero is partly luck, though. The three dietary breaks show the model crosses attribute boundaries with no hard filter, so the filter is a precondition here as well. Conditions: 1. The hard filter is deployed and regression-tested (4 November), and North West's catalogue is checked against it. 2. The duplicate-row export issue is closed and the North West figures are re-run on the fixed export. 3. Switch on around 5 November with the kill switch ready. Review refunds and dietary complaints at one week.

London, South East, Midlands: no-go for 2 November. Acceptance and refunds are fine, and basket is up. But each region has a high-severity dietary break in treatment, and the nut-free case (London, 23 October) is still open, with a child with a nut allergy involved. Before we re-decide on 6 November, we need: 1. The hard filter live and regression-tested. 2. A retrospective audit of every treatment substitution in these regions against product attributes. The logged incidents are only the ones customers reported. 3. Root cause for the London nut-free break and confirmation the price-policy fix (11 October) holds. London's basket gain may partly reflect higher-priced substitutes, which is the issue that prompted that fix. 4. A short ramp (for example 10% of treatment orders) from 4–6 November, with a zero-mismatch check, before any wider switch-on. If the audit finds further breaks, these regions stay off.

Scotland: no-go. The ops director's "settling down" is not what the data shows. Before the supplier change on 15 October, treatment acceptance was 86.3% and refunds were 17.2 per 1,000, the same as control (17.3). Since 15 October, acceptance has been flat at about 79% (79.0% in week two, 79.2% in week three) and refunds at 43 per 1,000 against 19 for control. Refund value since 15 October is about £2.8k. Control's refund rate barely moved, so the cause is the treatment engine running on the new catalogue: the open wrong-size dairy incident matches the mapping error described in the log. The re-map has no date. - Keep Scotland at 0% treatment. - Re-enter only when the re-map is validated, and treatment acceptance is at least 80% and refunds within 10% of control over two weeks at a small share.

Asks for the meeting

1. Agree that no region goes live on 2 November, and that criterion 3 is not waived.
2. Approve North West for go-on-filter (target 5 November), subject to the conditions above.
3. Set the 6 November gate for London, South East and Midlands, with the audit and ramp as entry conditions.
4. Keep Scotland on control. Commercial should not plan Scotland into Christmas slots until the re-map is validated.
5. Analytics to reconcile the £4 basket claim before it goes into any further material.

The Christmas booking window opens on 9 November. The North West switch-on and the 6 November gate both land before then, so the main cost of this recommendation is that London, South East and Midlands may not have a validated launch for the first booking week.

Executive Recommendation

• Monday, 2 November: NO-GO across all regions. We cannot launch while failing Criterion 3 (zero dietary/allergen mismatches). During the test, the AI model caused three high-severity safety incidents—substituting standard bread for gluten-free (coeliac customer), dairy cheese for vegan, and hazelnut bars for nut-free (child with nut allergy). The engineering hard filter to prevent cross-attribute substitution does not deploy until Wednesday, 4 November. A 48-hour delay is non-negotiable for customer safety.
• Thursday, 5 November: GO WITH CONDITIONS for London, South East, Midlands, and North West. Once the dietary hard filter is deployed on 4 November and verified in production, these four regions pass all launch criteria and can roll out to 100% four days ahead of the 9 November Christmas booking window.
• Scotland: NO-GO indefinitely. Keep Scotland on 100% picker-led control until the dairy catalogue re-map is completed and verified via a clean 7-day re-test.

---

Data Hygiene & Addressing Stakeholder Summaries

Before reviewing the criteria table, we must correct three data misconceptions circulating ahead of this meeting:

1. Duplicate Rows (North West): As flagged in `incidents.csv` (7 Oct), an export timeout duplicated North West treatment rows on 8, 9, and 10 October. All figures below are deduplicated (removing 2,194 phantom orders). Note: Criterion conclusions for North West hold either way, though deduplicating slightly raises its refund rate from 17.56 to 17.59 per 1,000.
2. The "£4 Basket Drop" is Simpson’s Paradox (Head of Analytics): Pooled across the test, treatment average basket value (£60.15) is indeed £3.93 lower than control (£64.08). However, this is a mathematical artefact of our unequal regional split: we allocated 70% of orders to treatment in our lowest-basket regions (Scotland £53, Midlands £55) and only 30% in our highest-basket region (London £72). Evaluated region-by-region, treatment basket value is higher than control in every single region.
3. Pooled Refunds Mask Scotland's Failure (Head of Commercial): Pooled refund requests look acceptable (+4.5% vs. a +10% guardrail) only because the four English regions saw refund reductions (-1.5% to -7.8%). This masks a +78.3% refund explosion in Scotland. Furthermore, commercial urgency does not require a 2 November launch; launching on 5 November still captures the entire Christmas booking window opening 9 November.

---

Launch Criteria Performance by Region (1–28 Oct, Deduplicated)

\Note: North West metrics reflect deduplicated data (raw orders: 22,830; raw refunds/1k: 17.56; raw weighted basket: £57.50). Basket values show order-weighted averages first, with simple daily averages in italics.*

---

Regional Rationale & Conditions

#### 1. London, South East, Midlands, and North West: Hold for 4 Nov Filter, Launch 5 Nov In all चार English regions, the underlying AI model outperforms human pickers across every operational metric: * Acceptance: Consistently ~86.5% across all four regions (+15.5 points over control), well above the 80% threshold. * Refunds: Lower than control in every region (ranging from -1.5% in the Midlands to -7.8% in the North West). Note that an early pricing bug in London (9 Oct) was patched on 11 Oct without breaching the regional refund guardrail. * Baskets: Up £0.15 to £0.38 per order over control. * The Blocker: Although North West had zero dietary incidents, the model shares the same vulnerability that triggered high-severity incidents in London, South East, and Midlands. Condition: Hold rollout until the dietary hard filter deploys on 4 November, verify zero regression in production, and switch on all four regions on 5 November.

#### 2. Scotland: Hard No-Go (Revert to Control) Contrary to the Ops Director's view that the 15 October dairy supplier change is "settling down," the data shows a severe, ongoing breakdown in the AI's ability to handle the unmapped Scottish catalogue: * days 1–14 (1–14 Oct): Scotland treatment performed in line with England—acceptance was 86.44% (7,976 / 9,227) and refund requests were 17.46 per 1,000 (162 / 9,280), beating control (18.22). * Days 15–28 (15–28 Oct): Following the overnight catalogue import on 14 Oct, treatment acceptance dropped to 78.05% (6,532 / 8,369)—failing Criterion 1—and never recovered (sitting at 77.76% on 28 Oct). Worse, treatment refund requests more than doubled to 46.45 per 1,000 (417 / 8,977, vs. 17.39 in control), driving a +78.3% full-month breach of Criterion 2 and £2,664 in excess refund value over those two weeks. Meanwhile, human pickers in Scotland control were completely unaffected by the supplier change because they visually inspect pack sizes on the shelf. * Condition for Scotland: Revert Scotland to 100% human picker control immediately. Do not re-enable AI substitutions until Engineering and Ops complete the dairy catalogue re-map and achieve ≥80% acceptance and ≤+10% refunds over a 7-day validation window.