Usable with a quick edit10 of 13 checks passed
The output is fully reasoned, includes the required table and conditions, and could be used with a quick edit to remove the one unsupported sentence.
Output
Recommendation for 30 October: no-go in every region on 2 November. Conditional phased rollout thereafter in London, South East, Midlands and North West. Scotland remains blocked pending catalogue repair and fresh evidence.
The dietary filter is scheduled for 4 November, two days after the proposed launch. Three treatment substitutions breached dietary requirements, including gluten-free bread replaced with standard bread and nut-free bars replaced with hazelnut bars. The shared system needs this control before expansion, including in regions with no recorded mismatch.
I would pause existing auto-substitution exposure and return to picker selection until the filter is deployed and verified.
Regional checks, 1–28 October
I removed three exact duplicate North West treatment rows dated 8–10 October4, consistent with the export incident, leaving 280 unique region–arm–day records. Acceptance and refund rates use summed numerators and denominators. Basket values are weighted by orders.
T = treatment; C = control. Refund changes are relative changes in requests per 1,000 orders.
| Region | Acceptance T ≥80% | Refunds/1,000: T vs C; change ≤+10% | Dietary mismatches: zero | Basket T vs C; difference ≥£0 | Decision for 2 Nov |
|---|---|---|---|---|---|
| London | 86.68%: pass | 17.44 vs 18.04; −3.33%: pass | 1: fail | £72.70 vs £72.49; +£0.21: pass | No-go |
| South East | 86.40%: pass | 17.54 vs 17.89; −1.93%: pass | 1: fail | £66.64 vs £66.20; +£0.44: pass | No-go |
| Midlands | 86.35%: pass | 17.48 vs 17.89; −2.25%: pass | 1: fail | £55.66 vs £55.35; +£0.31: pass | No-go |
| North West | 86.28%: pass | 17.58 vs 18.47; −4.80%: pass | 0 recorded: observed pass | £57.51 vs £57.29; +£0.22: pass | No-go: shared safety gap |
| Scotland | 82.76%: pass over full test2 | 30.16 vs 18.20; +65.70%: fail | 0 recorded: observed pass | £52.98 vs £52.62; +£0.36: pass | No-go: safety and catalogue |
Zero recorded incidents does not establish that an unprotected region is safe. Refunding a customer closes the complaint, not the underlying failure.
The aggregate claims conceal different problems.
Commercial’s acceptance claim is broadly correct: pooled acceptance rose from 71.20% to 85.93%, a 14.74 percentage-point improvement. Pooled refunds increased 7.86%, inside the overall guardrail. But the agreed criterion applies to every region. Scotland fails substantially.
Analytics’ £4 concern is also numerically correct but misleading as a treatment comparison: pooled baskets are £61.09 in treatment versus £65.15 in control. Treatment contains proportionally fewer high-value London orders and more lower-value Midlands and Scotland orders because allocation varies by region. Within every region, treatment baskets are higher across the full test. Applying the same combined regional order mix to both arms gives £63.26 versus £62.96, approximately £0.30 higher in treatment. This resolves the composition distortion; it does not establish a statistically reliable basket uplift.
Scotland is not settling down.
After the supplier switch on 15 October, treatment acceptance fell from 86.34% to 79.08%, below the launch threshold. Refunds rose from 17.25 to 43.25 per 1,000 orders, versus 17.79 in concurrent control, approximately 143% higher.
The final seven days still show 79.17% acceptance and 43.64 refunds per 1,000. The full-period acceptance pass masks a persistent failure in the current catalogue. The open pack-size mapping incident provides a plausible mechanism, and the repair has no date.
Conditions for reconsideration
- Engineering and QA, by 4 November: deploy and verify the dietary filter across all six protected attributes. Replay the three incident cases, test missing or conflicting attributes and confirm unsafe or unverified candidates cannot be automatically selected. A merged change is insufficient.
- PM, Analytics and regional Ops, review on 6 November:3 assess a limited post-fix pilot in London, South East, Midlands and North West, retaining randomised controls. Recheck every regional criterion, audit dietary compliance and verify London’s pricing fix. Expand only where the evidence supports it; insufficient evidence means continued limited exposure.1
- Scotland Ops and catalogue Engineering, owner and repair date agreed on 30 October: complete and validate the re-map before restarting a controlled pilot. Judge readiness on post-repair results, not the earlier healthy fortnight.
- During any restart: immediately disable auto-substitutions for a dietary mismatch and pause affected regions for guardrail breaches, with daily regional monitoring.
The 9 November booking window is a commercial deadline. It supplies no evidence that either unresolved defect will be fixed in time.
What a PM had to fix
1Decision deferredMake the callTargeted repair
What we’d changeGive Commercial something to plan around: say whether the four clean regions can go by 9 November, set a threshold for pausing a region, and give Scotland a Christmas fallback.
2Contradiction missedSurface the contradictionQuick edit
What we’d changeShow Scotland failing acceptance since the supplier switch in the table, as the text does, rather than passing.
3Test or gate too weakTighten the testQuick edit
What we’d changeA 6 November review would have only a day or two of post-filter data. Allow enough to judge.
4OtherFix itQuick edit
What we’d changeSay whether removing them changes any result. The brief asks for it.
Check by check
Got wrong · 1
- Catches the duplicate rowsFinds and removes the duplicate North West rows but does not explicitly state whether they change the overall assessment.
Mixed · 2
- Avoids unsupported claimsPresents 'Refunding a customer closes the complaint, not the underlying failure' as fact without support from the supplied context.The two graders disagreed on this one.
- Gets the base of every number rightAll derived percentages and rates use correct numerators and denominators, and basket values are weighted by orders.The two graders disagreed on this one.
Got right · 10
- Uses the supplied evidence correctlyAll factual statements about the current situation are supported by the brief or correct arithmetic; no invented facts.
- Addresses the actual decisionCommits to a clear no-go on 2 Nov, then conditional rollout by region, and states what would change the call.
- Respects explicit constraintsRespects the requested form, reader, word limit, and uses the data rather than stakeholder summaries.
- Identifies material uncertaintyNames the unknowns (filter delivery, Scotland catalogue repair, post-fix evidence) and says what would change the call.
- Produces the required deliverableA complete recommendation with a criteria table, under 900 words, usable for the launch meeting.
- Checks every criterion, region by regionTable checks every criterion for each region using figures computed from the attached files.
- Finds the Scotland breachIdentifies the Scotland refund breach, ties it to the 15 October supplier change, notes acceptance dip, and blocks Scotland.
- Explains the basket gapShows treatment baskets are higher within every region and explains the pooled gap as a mix artefact.
- Makes a clear callOne clear call per region, with conditions and dates, and says what would change it.
- Limits the downside of being wrongProposes daily regional monitoring, a trigger (dietary mismatch or guardrail breach) and action (disable, pause).
Claims the judge couldn’t find in the brief
- Refunding a customer closes the complaint, not the underlying failure.
Grades and run details
Decision model 85 · LLM judge 12 of 14 checks
Decision model checks
- passUses the supplied evidence correctly18%
- passAddresses the actual decision96%
- passRespects explicit constraints29%
- passIdentifies material uncertainty87%
- passAvoids unsupported claims51%
- passProduces the required deliverable80%
- passChecks every criterion, region by region61%
- passFinds the Scotland breach100%
- passExplains the basket gap99%
- partialCatches the duplicate rows31%
- passMakes a clear call75%
- partialLimits the downside of being wrong61%
- failGets the base of every number right28%
Run
- Run
- #1
- Time to output
- 71 s
- Submitted
- 25 Sept 2026