Make the launch call

Executive Recommendation

• Monday, 2 November: NO-GO across all regions. We cannot launch while failing Criterion 3 (zero dietary/allergen mismatches). During the test, the AI model caused three high-severity safety incidents—substituting standard bread for gluten-free (coeliac customer), dairy cheese for vegan, and hazelnut bars for nut-free (child with nut allergy). The engineering hard filter to prevent cross-attribute substitution does not deploy until Wednesday, 4 November. A 48-hour delay is non-negotiable for customer safety.
• Thursday, 5 November: GO WITH CONDITIONS for London, South East, Midlands, and North West. Once the dietary hard filter is deployed on 4 November and verified in production, these four regions pass all launch criteria and can roll out to 100% four days ahead of the 9 November Christmas booking window.
• Scotland: NO-GO indefinitely. Keep Scotland on 100% picker-led control until the dairy catalogue re-map is completed and verified via a clean 7-day re-test.

---

Data Hygiene & Addressing Stakeholder Summaries

Before reviewing the criteria table, we must correct three data misconceptions circulating ahead of this meeting:

1. Duplicate Rows (North West): As flagged in `incidents.csv` (7 Oct), an export timeout duplicated North West treatment rows on 8, 9, and 10 October. All figures below are deduplicated (removing 2,194 phantom orders). Note: Criterion conclusions for North West hold either way, though deduplicating slightly raises its refund rate from 17.56 to 17.59 per 1,000.
2. The "£4 Basket Drop" is Simpson’s Paradox (Head of Analytics): Pooled across the test, treatment average basket value (£60.15) is indeed £3.93 lower than control (£64.08). However, this is a mathematical artefact of our unequal regional split: we allocated 70% of orders to treatment in our lowest-basket regions (Scotland £53, Midlands £55) and only 30% in our highest-basket region (London £72). Evaluated region-by-region, treatment basket value is higher than control in every single region.
3. Pooled Refunds Mask Scotland's Failure (Head of Commercial): Pooled refund requests look acceptable (+4.5% vs. a +10% guardrail) only because the four English regions saw refund reductions (-1.5% to -7.8%). This masks a +78.3% refund explosion in Scotland. Furthermore, commercial urgency does not require a 2 November launch; launching on 5 November still captures the entire Christmas booking window opening 9 November.

---

Launch Criteria Performance by Region (1–28 Oct, Deduplicated)

\Note: North West metrics reflect deduplicated data (raw orders: 22,830; raw refunds/1k: 17.56; raw weighted basket: £57.50). Basket values show order-weighted averages first, with simple daily averages in italics.*

---

Regional Rationale & Conditions

#### 1. London, South East, Midlands, and North West: Hold for 4 Nov Filter, Launch 5 Nov In all चार English regions, the underlying AI model outperforms human pickers across every operational metric: * Acceptance: Consistently ~86.5% across all four regions (+15.5 points over control), well above the 80% threshold. * Refunds: Lower than control in every region (ranging from -1.5% in the Midlands to -7.8% in the North West). Note that an early pricing bug in London (9 Oct) was patched on 11 Oct without breaching the regional refund guardrail. * Baskets: Up £0.15 to £0.38 per order over control. * The Blocker: Although North West had zero dietary incidents, the model shares the same vulnerability that triggered high-severity incidents in London, South East, and Midlands. Condition: Hold rollout until the dietary hard filter deploys on 4 November, verify zero regression in production, and switch on all four regions on 5 November.

#### 2. Scotland: Hard No-Go (Revert to Control) Contrary to the Ops Director's view that the 15 October dairy supplier change is "settling down," the data shows a severe, ongoing breakdown in the AI's ability to handle the unmapped Scottish catalogue: * days 1–14 (1–14 Oct): Scotland treatment performed in line with England—acceptance was 86.44% (7,976 / 9,227) and refund requests were 17.46 per 1,000 (162 / 9,280), beating control (18.22). * Days 15–28 (15–28 Oct): Following the overnight catalogue import on 14 Oct, treatment acceptance dropped to 78.05% (6,532 / 8,369)—failing Criterion 1—and never recovered (sitting at 77.76% on 28 Oct). Worse, treatment refund requests more than doubled to 46.45 per 1,000 (417 / 8,977, vs. 17.39 in control), driving a +78.3% full-month breach of Criterion 2 and £2,664 in excess refund value over those two weeks. Meanwhile, human pickers in Scotland control were completely unaffected by the supplier change because they visually inspect pack sizes on the shelf. * Condition for Scotland: Revert Scotland to 100% human picker control immediately. Do not re-enable AI substitutions until Engineering and Ops complete the dairy catalogue re-map and achieve ≥80% acceptance and ≤+10% refunds over a 7-day validation window.

Recommendation Summary: Go With Conditions

Do not execute a blanket rollout on Monday, 2 November.

• North West: GO (2 November). Meets all four PRD launch criteria cleanly.
• London, South East, Midlands: GO WITH CONDITIONS (Deploy Thursday, 5 November). Commercial metrics are strong, but all three regions breached Criterion 3 with severe dietary/allergen substitutions. Launch must be gated on the successful deployment of the dietary-attribute hard filter on 4 November.
• Scotland: NO-GO. Fails Criterion 2 (+67.4% refund spike vs. control) and Criterion 1 post-supplier transition (76.6% acceptance). Revert 100% to control until catalogue re-mapping is completed and re-tested.

This phased approach protects customer safety and brand reputation while capturing AI substitution value ahead of Christmas slot bookings opening on 9 November.

---

PRD Launch Criteria Assessment

Note: North West treatment metrics reflect deduplication of export retries on 8–10 October (3,194 duplicate rows removed).

---

Analysis & Stakeholder Debunking

#### 1. Head of Analytics: “Treatment baskets are £4 smaller across the test.” Finding: False alarm caused by Simpson’s Paradox. Across all pooled orders, treatment average basket appears lower (£61.06 treatment vs. £65.18 control, a £4.12 deficit). However, this is an aggregation distortion caused by the uneven regional split: London (average basket ~£73) received only a 30% treatment allocation, whereas lower-basket regions (Midlands at ~£55 and Scotland at ~£53) received a 70% treatment allocation.

When evaluated within every single region, treatment basket value was higher than control by £0.16 to £0.30. Criterion 4 is fully satisfied across all regions.

#### 2. Head of Commercial: “Refunds are within the guardrail overall. Every day we wait costs Christmas.” Finding: Misleading pooled metric masking a severe regional failure. While aggregate refunds appear within limits (+8.1% overall), the PRD explicitly requires compliance in every region. Scotland completely breaches this guardrail (+67.4% above control). Launching everywhere on 2 November would risk widespread customer attrition in Scotland right before holiday booking.

#### 3. Ops Director (Scotland): “The supplier change caused noise, but it's settling down.” Finding: Factually incorrect; Scotland is failing and persistent. Following the 15 October dairy supplier change, wrong-size substitutions (e.g., pack sizes mapped to incorrect product families) caused: * Daily refunds in treatment to triple from 11.6/day (Oct 1–14) to 28.4/day (Oct 15–28), reaching 43.7 refunds per 1,000 orders (+140% vs. control). * Treatment acceptance dropped from 86.3% down to 76.6% (failing Criterion 1). * Daily refund value increased 4x to £180–£247/day. Performance has not stabilized. Because the catalogue re-map is incomplete with no ETA, Scotland cannot launch.

#### 4. The Dietary Hazard (London, South East, Midlands) Treatment produced three critical safety failures: * 4 Oct (Midlands): Coeliac customer received standard bread for gluten-free. * 19 Oct (South East): Vegan customer received dairy cheddar. * 23 Oct (London): Child with nut allergies received hazelnut-containing bars for nut-free bars.

Launching auto-substitutions in these regions before deploying technical guardrails creates intolerable liability.

---

Implementation Plan

1. Monday, 2 November: * Roll out North West to 100% treatment. (Clean test data, no allergen incidents, +15pp acceptance lift). * Revert Scotland treatment to 0% (100% control). Keep human pickers selecting substitutes until the catalogue re-map is verified and back-tested. * Maintain London, South East, and Midlands at current test splits for 48 hours.

2. Wednesday, 4 November: * Engineering deploys the dietary-attribute hard filter to production following scheduled regression testing.

3. Thursday, 5 November: * Verify filter telemetry on live orders. * Roll out London, South East, and Midlands to 100% treatment.

4. Monday, 9 November: * Christmas delivery-slot booking opens on schedule with 4 of 5 regions operating under fully automated, safety-enforced substitutions, leaving human pickers dedicated to Scotland until rectified.

Recommendation: AI auto-substitutions, launch meeting 30 October

Decision: no-go for Monday 2 November. Go with conditions from 4 November for London, South East, Midlands and North West. Scotland stays on picker substitutions until it re-passes.

Criteria by region

Figures are recomputed from the daily rows. Treatment (T) and control (C) are pooled over the 28 days.

Baskets are means of daily averages. The differences are small and I have not tested them for significance, but the point estimate is positive everywhere.

Checking what people have said

• Head of Commercial. The pooled numbers are right: acceptance is 85.9% against 71.2% (+14.7 points), and refunds are 19.5 against 18.1 per 1,000 (+8%). But the pooled refund figure hides Scotland, which is failing badly. Four regions are 2–5% better than control on refunds.
• Head of Analytics. The £4 gap is real in the pooled data (£61.10 vs £65.15) but it is a mix effect, not a treatment effect. London, with the highest baskets (about £72), was 42% of control orders but only 18% of treatment orders. The Midlands and Scotland (about £53–56) were 18% of control but 42% of treatment. Within every region, treatment baskets are £0.24–£0.46 higher. Criterion 4 passes. The uneven split did not bias the regional comparisons.
• Ops director (Scotland). The data does not support "settling down". Before the supplier switch on 15 October, Scotland treatment was healthy (86.3% acceptance, 17.3 refunds per 1,000). From 15 October to 28 October, acceptance has been flat at 78–81%, and daily refunds have been 25–33 against a pre-switch level of 10–14. The cause is the open incident: the new supplier's pack sizes are mapped to the wrong product family, so the model makes wrong-size dairy substitutions. That is a catalogue defect with no fix date, not noise. Scotland control is unaffected.

Why the date moves

Criterion 3 fails in three regions. The three mismatches include a nut-allergy case and a coeliac case. The hard filter that prevents them deploys on 4 November. Until it is live, every region has the same exposure. North West and Scotland show zero only because none were reported, and the incidents come from customer complaints, so the true rate is unknown. I would not switch on a known allergen risk two days before the fix.

Waiting is cheap. The booking window opens on 9 November, not 2 November. Deploying the filter on 4 November and switching regions on 4–5 November leaves several days before the window. Only Scotland has no route to launch before then.

Conditions for the four regions

1. The filter deploys on 4 November and passes regression tests. Before any region goes live, it is tested in production against the three incident SKUs (gluten-free loaf, vegan cheese, nut-free bar) and the other attributes: vegetarian, halal and kosher. The London incident is still open with the filter "in review", so confirm the filter would have blocked it.
2. Regions are switched on region by region, North West and South East first, then London and the Midlands. This gives a day of live data before the larger volumes go on.
3. Any dietary or allergen mismatch after launch switches that region back to picker substitution immediately, pending review. That includes any mismatch in the first week.
4. Daily monitoring of acceptance (80% floor) and refunds (10% guardrail) per region against a small held-back control. Delegate the switch-on decision so it does not need another meeting.

Scotland

Keep Scotland on picker substitutions. Retest only after the catalogue re-map ships, on a small treatment share. It should show at least three consecutive days at 80% or better acceptance, and refunds within 10% of control, before it goes to full volume. Engineering has no date, so Scotland should not be in the Christmas plan and should be treated as upside.

Data notes

• North West treatment has three duplicated rows (8, 9 and 10 October), matching the open export-job incident. I removed them. This restores the intended 50% split, and the North West rates barely move either way.
• Incident logs are a floor on dietary failures, not a measure of them.
• This is 28 days of data in a low-volume region (Scotland control is about 265 orders a day). Even so, the Scotland gap is far outside noise.