Make the launch call

Recommendation for 30 October: go with conditions in four regions; no-go in Scotland

Do not expand any region on Monday 2 November. London, South East, Midlands and North West are candidates for a gated rollout after the dietary hard filter is deployed and validated—not before its scheduled 4 November deployment. Scotland must additionally resolve and re-test its catalogue mapping.

Christmas urgency does not justify launching ahead of a known safety fix. The 9 November booking window should guide sequencing, not override the PRD.

Method and launch-criterion checks

I removed three exact duplicate North West treatment rows, dated 8–10 October, consistent with the export incident. This leaves 280 unique date–region–arm records.

Acceptance and refunds below use ratios of summed counts, not averages of daily rates. Basket values are order-weighted. T = treatment; C = control. Dietary counts are recorded treatment incidents; zero recorded incidents is not proof that none occurred.

What the data actually say

Commercial’s overall refund claim hides a regional failure. After deduplication, pooled refunds are 19.50 per 1,000 treatment orders versus 18.08 control: +7.9%, within the overall guardrail. But the PRD requires compliance in every region, and Scotland fails badly. Strong acceptance cannot compensate for dietary breaches.

Analytics’ £4 basket concern is real in the pooled arithmetic, but not within regions. Pooled baskets are £61.09 treatment versus £65.15 control (−£4.06). Treatment disproportionately includes lower-basket Midlands and Scotland orders, while control disproportionately includes higher-basket London orders. Every within-region comparison is positive. This is a composition effect, not evidence here that AI reduces basket value. These are point estimates; the daily averages alone do not establish statistical certainty.

Scotland is not settling down. Following the supplier switch on 15 October:

• Acceptance falls from 86.3% on 1–14 October to 79.1% on 15–28 October, below the launch threshold.
• Treatment refunds rise from 17.25 to 43.25 per 1,000 orders. Post-switch control is 17.79, making treatment 143.1% worse.
• The final week still has 79.2% acceptance and 43.64 refunds per 1,000.

The open pack-size mapping incident is consistent with this sustained deterioration. The full-period acceptance pass masks the current operating state.

Conditions and release plan

For London, South East, Midlands and North West, require:

1. Deployed, validated safety protection: successful regression tests across gluten-free, vegan, vegetarian, nut-free, halal and kosher attributes, including missing/incorrect metadata and the three observed failures. “Merged” is not “protected.”
2. Fresh post-fix evidence: a controlled validation window with zero dietary mismatches and all regional numerical criteria met, with explicit launch sign-off. Refunds and apologies do not erase historical criterion failures.
3. Staged regional expansion: monitor acceptance, refunds and dietary incidents daily; stop immediately for any dietary mismatch and roll back for guardrail breaches. Reconcile the North West export before reporting.

For Scotland, also require a completed catalogue re-map, pack-size validation and fresh post-remediation evidence. With no completion date and continuing deterioration, make no Christmas rollout commitment.

Meeting decision: reject the all-region 2 November launch; approve a gated path for four regions, while keeping Scotland blocked.

AI auto-substitutions: launch recommendation

Oct 29, 2026 · @Martin Slaney · For the 30 October launch meeting

Recommendation

No-go for Monday 2 November in any region. The dietary hard filter isn't live until 4 November, and the test produced three allergen and dietary mismatches.

Go with conditions for London, South East, Midlands and North West from Thursday 5 November, once the filter is live and verified. That is still four days before the Christmas booking window opens on 9 November.

No-go for Scotland until the dairy catalogue re-map ships and a fresh week of test data passes the criteria.

Launch criteria by region

Figures are for 1–28 October from daily\_metrics.csv, with duplicate rows removed. T = treatment, C = control.

Why not 2 November

Criterion 3 requires zero dietary or allergen mismatches. Treatment had three, all rated high severity: a gluten-free loaf swapped for a standard one (the customer is coeliac), vegan cheese swapped for dairy cheddar (the customer complained on social media), and a nut-free multipack swapped for one containing hazelnuts, sent to a child with a nut allergy. The nut incident is still open. Control had none.

The fix is the dietary hard filter, which is merged and deploys on 4 November after regression tests. Launching on 2 November would put every order in every region onto the unfiltered model for two days. North West's zero is luck, because it runs the same model.

Strictly, all four "go" regions fail criterion 3 as it's written. I'm asking the meeting to accept the filter as the remedy, on condition that we show it would have blocked all three incidents. If the meeting won't accept that, the honest answer is no-go everywhere.

Waiting three days costs nothing for Christmas, because bookings don't open until 9 November.

What the summaries get wrong

Head of Commercial: "refunds within the guardrail overall." Pooled, treatment refunds are 7.9% above control (19.5 vs 18.1 per 1,000 orders), which is inside 10%. But the criterion applies to every region, and the pooled figure hides Scotland at +66%. In the other four regions, treatment refunds are below control. The 15-point acceptance lift is real in those four regions (+15.0 to +15.4 points). In Scotland it's +11.7 overall and falling.

Head of Analytics: "treatment baskets are £4 smaller." Pooled, the gap is £61.09 vs £65.15, but it comes from how orders were split, not from the feature. Treatment got 70% of orders in Midlands and Scotland, which have the smallest baskets (£52–56), and only 30% in London, which has the largest (£72). Within every region, treatment baskets are £0.21–£0.44 larger, so criterion 4 passes everywhere.

Ops director, Scotland: "it's settling down." It isn't. Before the supplier switch on 15 October, Scotland's treatment arm matched the other regions (86.3% acceptance, 17.3 refunds per 1,000 vs 18.6 in control). Since then, treatment refunds have run at 25–33 a day with no downward trend: 28.1 a day on average from 15 to 21 October and 28.7 from 22 to 28 October. Refund value per order has tripled, from £0.09 to £0.30. Scotland's control arm is unaffected at 17.8 per 1,000. So the problem is specific to the model reading a mis-mapped catalogue: pickers can see the real pack sizes, and the model can't. The re-map has no date.

Data quality

The North West treatment rows for 8, 9 and 10 October each appear twice, because the export job re-ran (the open incident from 7 October). I removed the duplicates. They don't change any verdict: North West acceptance moves from 86.2% to 86.3%.

Conditions for the 5 November go

1. The dietary filter deploys on 4 November and passes regression tests. We replay the three incident orders and confirm the filter blocks each of them.²
2. The London nut-allergen incident is closed with a root cause.
3. We switch on region by region and keep a 10% control holdout in each region through Christmas, so criteria 2 and 4 stay measurable.
4. Kill switch: any single dietary or allergen mismatch turns the region off, as does a treatment refund rate more than 10% above its holdout over a rolling 3 days.¹
5. Scotland: ship the catalogue re-map, then run 7 days of test and re-check the criteria. If it can't be ready by 9 November, Scotland takes Christmas orders with picker substitutions.

Recommendation for 30 October: no-go in every region on 2 November. Conditional phased rollout thereafter in London, South East, Midlands and North West. Scotland remains blocked pending catalogue repair and fresh evidence.

The dietary filter is scheduled for 4 November, two days after the proposed launch. Three treatment substitutions breached dietary requirements, including gluten-free bread replaced with standard bread and nut-free bars replaced with hazelnut bars. The shared system needs this control before expansion, including in regions with no recorded mismatch.

I would pause existing auto-substitution exposure and return to picker selection until the filter is deployed and verified.

Regional checks, 1–28 October

I removed three exact duplicate North West treatment rows dated 8–10 October⁴, consistent with the export incident, leaving 280 unique region–arm–day records. Acceptance and refund rates use summed numerators and denominators. Basket values are weighted by orders.

T = treatment; C = control. Refund changes are relative changes in requests per 1,000 orders.

Zero recorded incidents does not establish that an unprotected region is safe. Refunding a customer closes the complaint, not the underlying failure.

The aggregate claims conceal different problems.

Commercial’s acceptance claim is broadly correct: pooled acceptance rose from 71.20% to 85.93%, a 14.74 percentage-point improvement. Pooled refunds increased 7.86%, inside the overall guardrail. But the agreed criterion applies to every region. Scotland fails substantially.

Analytics’ £4 concern is also numerically correct but misleading as a treatment comparison: pooled baskets are £61.09 in treatment versus £65.15 in control. Treatment contains proportionally fewer high-value London orders and more lower-value Midlands and Scotland orders because allocation varies by region. Within every region, treatment baskets are higher across the full test. Applying the same combined regional order mix to both arms gives £63.26 versus £62.96, approximately £0.30 higher in treatment. This resolves the composition distortion; it does not establish a statistically reliable basket uplift.

Scotland is not settling down.

After the supplier switch on 15 October, treatment acceptance fell from 86.34% to 79.08%, below the launch threshold. Refunds rose from 17.25 to 43.25 per 1,000 orders, versus 17.79 in concurrent control, approximately 143% higher.

The final seven days still show 79.17% acceptance and 43.64 refunds per 1,000. The full-period acceptance pass masks a persistent failure in the current catalogue. The open pack-size mapping incident provides a plausible mechanism, and the repair has no date.

Conditions for reconsideration

• Engineering and QA, by 4 November: deploy and verify the dietary filter across all six protected attributes. Replay the three incident cases, test missing or conflicting attributes and confirm unsafe or unverified candidates cannot be automatically selected. A merged change is insufficient.
• PM, Analytics and regional Ops, review on 6 November:³ assess a limited post-fix pilot in London, South East, Midlands and North West, retaining randomised controls. Recheck every regional criterion, audit dietary compliance and verify London’s pricing fix. Expand only where the evidence supports it; insufficient evidence means continued limited exposure.¹
• Scotland Ops and catalogue Engineering, owner and repair date agreed on 30 October: complete and validate the re-map before restarting a controlled pilot. Judge readiness on post-repair results, not the earlier healthy fortnight.
• During any restart: immediately disable auto-substitutions for a dietary mismatch and pause affected regions for guardrail breaches, with daily regional monitoring.

The 9 November booking window is a commercial deadline. It supplies no evidence that either unresolved defect will be fixed in time.