Needs more than a quick edit3 of 12 checks passed
A PM would need to redo the sizing and duration, remove or redesign the early kill rule, add a sample-ratio check, and complete guardrail thresholds. These are substantive reasoning changes, not quick wording edits.
Output
Experiment Spec: Delivery Fee Transparency in Basket
To: Chloe (Approval), Ravi (Build), Ines (Analysis) From: Checkout PM
---
1. Objective & Hypothesis
- Context: 12% of support contacts complain about surprise delivery fees at final checkout.
- Hypothesis: Displaying the delivery fee (£3.99, or "Free" if >£60) upfront in the basket will set accurate expectations, reduce support contacts, and build customer trust without reducing net orders.
- Risk (Chloe’s concern): Upfront cost shock could cause drop-off before checkout begins.
---
2. Corrections to the Draft Plan
1. Primary Metric Changed: Draft used Checkout Completion Rate (Orders ÷ Starts). Correction: Use Basket-to-Order Conversion Rate (Orders ÷ Basket Viewers). Measuring only checkout starts introduces severe selection bias—if fee visibility deters low-intent users at the basket, checkout completion would artificially increase while overall orders plummet. 2. Unit of Randomization Changed: Draft used session-level. Correction: Randomize by User ID (falling back to persistent anonymous visitor ID). 50% of weekly viewers are repeat shoppers; session-level randomization would expose returning users to conflicting experiences. 3. No Peeking / Fixed Duration: Draft suggested stopping once "significant." Correction: Continuous monitoring with early stopping inflates false-positive rates to >30%. We will fix sample size upfront and run for complete weekly cycles. 4. Scope Control (Chloe’s Banner): Cut the promotional free-delivery banner from this test. Adding a promotional banner introduces a confounding variable, making it impossible to isolate whether fee transparency helped or hurt. We will test the banner in a follow-up experiment.
---
3. Variants (For Ravi)
- Allocation: 50/50 split at User/Persistent Device ID level. Triggered on basket view.
- Control (A): Current basket. No delivery fee shown (fee appears on final checkout step as today).
- Variant (B): Add a delivery fee line item to the Basket Order Summary:
- Basket < £60: `Delivery: £3.99`
- Basket ≥ £60: `Delivery: FREE`
- Include delivery fee in the displayed Basket Total.
- Note: Do not add promotional progress bars or marketing banners.
---
4. Metrics Framework (For Ines)
- Primary Metric: Basket-to-Order Conversion Rate (Total Unique Ordering Users ÷ Total Unique Basket Viewers).
- Guardrail Metrics:
- Average Order Value (AOV baseline: £47).
- Basket-to-Checkout Start Rate (tracks where abandonment occurs).
- Total Net Revenue (Orders × AOV).
- Secondary / Diagnostic Metrics:
- Delivery-fee-related customer support ticket rate per 1,000 orders.
- Checkout Completion Rate (Checkout Starts → Orders).
---
5. Sample Size & Run Time (For Ines & Chloe)
- Baseline Conversion: 8.0% of basket viewers place an order.
- Minimum Detectable Effect (MDE): Absolute ±0.5 percentage points (7.5% or 8.5%, a 6.25% relative change), per Chloe's threshold.
- Statistical Parameters: $\alpha = 0.05$ (two-tailed), Power ($1 - \beta$) = 80%.
- Required Sample: ~47,000 unique users per variant (~94,000 total unique basket viewers).
- Duration: 3 full weeks (21 days).
- With 40,000 viewers/week (~20k new, ~20k repeat), 3 weeks yields ~80,000–90,000 unique basket viewers across three full grocery-shopping cycles.
- Ines will run the read-out strictly after the 21-day window closes.
---
6. Decision & Rollout Criteria (For Chloe)
- Ship Variant: Primary conversion does not drop by more than Chloe’s threshold of -0.5pp (non-inferiority) AND delivery-related support contacts decrease significantly.
- Iterate / Re-test: Basket-to-order conversion drops by >0.5pp. In this scenario, early fee disclosure hurts conversion, justifying Chloe’s concern; we then test the £60 threshold banner/progress bar to offset drop-off.
- Kill Immediately: Overall revenue drops >5% statistically significantly at the end of Week 1 (safety check only; not an early win stop).
Claims the judge couldn’t find in the brief
- Three weeks yields about 80,000–90,000 unique basket viewers.
- Early stopping inflates false-positive rates to more than 30%.
- The variant's free-delivery threshold is £60 or more.
Grades and run details
Decision model 67 · LLM judge 3 of 13 checks
Decision model checks
- failUses the supplied evidence correctly55%
- passAddresses the actual decision63%
- partialRespects explicit constraints20%
- passIdentifies material uncertainty65%
- partialAvoids unsupported claims49%
- partialProduces the required deliverable32%
- passTests one change at a time100%
- passFixes the duration up front66%
- passAn unambiguous primary metric28%
- partialDecision rule written before the test79%
- partialSized from the real traffic94%
- partialGuardrails with thresholds90%
Run
- Run
- #1
- API response time
- 19 s
- Submitted
- 1 Oct 2026