Usable with a quick edit10 of 13 checks passed
The recommendation, table, and calculations are solid; only minor wording additions to state duplicate impact and flag irreversible dietary harm are needed.
Output
Recommendation for 30 October: go with conditions in four regions; no-go in Scotland
Do not expand any region on Monday 2 November. London, South East, Midlands and North West are candidates for a gated rollout after the dietary hard filter is deployed and validated—not before its scheduled 4 November deployment. Scotland must additionally resolve and re-test its catalogue mapping.
Christmas urgency does not justify launching ahead of a known safety fix. The 9 November booking window should guide sequencing, not override the PRD.
Method and launch-criterion checks
I removed three exact duplicate North West treatment rows, dated 8–10 October, consistent with the export incident. This leaves 280 unique date–region–arm records.
Acceptance and refunds below use ratios of summed counts, not averages of daily rates. Basket values are order-weighted. T = treatment; C = control. Dietary counts are recorded treatment incidents; zero recorded incidents is not proof that none occurred.
| Region | Treatment acceptance ≥80% | Refunds/1,000 orders: T vs C; relative change ≤+10% | Zero dietary/allergen mismatches | Average basket: T vs C; difference ≥£0 | Recommendation |
|---|---|---|---|---|---|
| London | Pass: 86.7% (23,607/27,234) | Pass: 17.44 vs 18.04; −3.3% | Fail: 1, nut-free → hazelnuts, 23 Oct | Pass: £72.70 vs £72.49; +£0.21 | Go with conditions; not 2 Nov |
| South East | Pass: 86.4% (26,586/30,770) | Pass: 17.54 vs 17.89; −1.9% | Fail: 1, vegan → dairy, 19 Oct | Pass: £66.64 vs £66.20; +£0.44 | Go with conditions; not 2 Nov |
| Midlands | Pass: 86.3% (26,732/30,958) | Pass: 17.48 vs 17.89; −2.3% | Fail: 1, gluten-free → standard bread, 4 Oct | Pass: £55.66 vs £55.35; +£0.31 | Go with conditions; not 2 Nov |
| North West | Pass: 86.3% (18,153/21,039) | Pass: 17.58 vs 18.47; −4.8% | Pass on incident log: 0 recorded | Pass: £57.51 vs £57.29; +£0.22 | Go with conditions; not 2 Nov |
| Scotland | Pass over 28 days: 82.8% (14,333/17,319) | Fail: 30.16 vs 18.20; +65.7% | Pass on incident log: 0 recorded | Pass: £52.98 vs £52.62; +£0.36 | No-go |
What the data actually say
Commercial’s overall refund claim hides a regional failure. After deduplication, pooled refunds are 19.50 per 1,000 treatment orders versus 18.08 control: +7.9%, within the overall guardrail. But the PRD requires compliance in every region, and Scotland fails badly. Strong acceptance cannot compensate for dietary breaches.
Analytics’ £4 basket concern is real in the pooled arithmetic, but not within regions. Pooled baskets are £61.09 treatment versus £65.15 control (−£4.06). Treatment disproportionately includes lower-basket Midlands and Scotland orders, while control disproportionately includes higher-basket London orders. Every within-region comparison is positive. This is a composition effect, not evidence here that AI reduces basket value. These are point estimates; the daily averages alone do not establish statistical certainty.
Scotland is not settling down. Following the supplier switch on 15 October:
- Acceptance falls from 86.3% on 1–14 October to 79.1% on 15–28 October, below the launch threshold.
- Treatment refunds rise from 17.25 to 43.25 per 1,000 orders. Post-switch control is 17.79, making treatment 143.1% worse.
- The final week still has 79.2% acceptance and 43.64 refunds per 1,000.
The open pack-size mapping incident is consistent with this sustained deterioration. The full-period acceptance pass masks the current operating state.
Conditions and release plan
For London, South East, Midlands and North West, require:
- Deployed, validated safety protection: successful regression tests across gluten-free, vegan, vegetarian, nut-free, halal and kosher attributes, including missing/incorrect metadata and the three observed failures. “Merged” is not “protected.”
- Fresh post-fix evidence: a controlled validation window with zero dietary mismatches and all regional numerical criteria met, with explicit launch sign-off. Refunds and apologies do not erase historical criterion failures.
- Staged regional expansion: monitor acceptance, refunds and dietary incidents daily; stop immediately for any dietary mismatch and roll back for guardrail breaches. Reconcile the North West export before reporting.
For Scotland, also require a completed catalogue re-map, pack-size validation and fresh post-remediation evidence. With no completion date and continuing deterioration, make no Christmas rollout commitment.
Meeting decision: reject the all-region 2 November launch; approve a gated path for four regions, while keeping Scotland blocked.
Check by check
Got wrong · 2
- Catches the duplicate rowsFinds and removes the duplicate rows but never states whether they change the results.
- Limits the downside of being wrongGives monitoring and rollback thresholds but does not flag irreversible consequences such as customer-facing dietary substitutions or communications.
Mixed · 1
- Gets the base of every number rightAll percentages, per-1,000 rates, and basket differences use correct summed or order-weighted bases.The two graders disagreed on this one.
Got right · 10
- Uses the supplied evidence correctlyAll current-state numbers and facts trace to the brief, incidents.csv, or arithmetic from daily_metrics.csv.
- Addresses the actual decisionCommits to a clear region-by-region go/no-go with dates and conditions for the launch meeting.
- Respects explicit constraintsIncludes the required region-by-criterion table and keeps prose within the length while respecting the no-early-launch safety constraint.
- Identifies material uncertaintyNames filter deployment/validation and Scotland catalogue remap as unresolved and ties go decisions to fresh evidence.
- Avoids unsupported claimsCausal claims are hedged ('consistent with') and interpretations are labelled as composition effects or point estimates.
- Produces the required deliverableProvides a usable recommendation, table, and release conditions that the reader could act on with light edits.
- Checks every criterion, region by regionRegion-by-criterion table covers acceptance, refunds vs control, dietary incidents, and basket values with computed figures.
- Finds the Scotland breachFinds Scotland's refund breach, ties it to the supplier change with post-15 Oct numbers, and holds Scotland.
- Explains the basket gapExplains the pooled basket gap as a traffic-mix effect and shows all within-region treatment baskets positive.
- Makes a clear callMakes a clear per-region call with dates and conditions, and Scotland is blocked.
Grades and run details
Decision model 85 · LLM judge 12 of 14 checks
Decision model checks
- passUses the supplied evidence correctly26%
- passAddresses the actual decision81%
- passRespects explicit constraints41%
- passIdentifies material uncertainty46%
- passAvoids unsupported claims42%
- passProduces the required deliverable79%
- passChecks every criterion, region by region71%
- passFinds the Scotland breach100%
- passExplains the basket gap99%
- partialCatches the duplicate rows49%
- passMakes a clear call34%
- partialLimits the downside of being wrong43%
- failGets the base of every number right22%
Run
- Run
- #1
- API response time
- 1.9 min
- Submitted
- 29 Sept 2026