Needs more than a quick edit9 of 14 checks passed
The spec lacks a valid sample size and a basket drop-off guardrail, so a PM would need to redo sizing and add a guardrail before using it.
Output
Experiment: delivery transparency on the basket
Status: Proposed; launch requires a basket-level sizing check and agreement with Payments. The supplied traffic supports a planning estimate, not a defensible final user-level sample size.
Hypothesis and treatment
Showing the next-day delivery date and fee on the basket increases completed orders by removing a late surprise. It may instead move abandonment earlier, so success must be measured from basket exposure, not checkout entry.
- Control: Date and fee first appear at the final checkout step.
- Treatment: Show the same date and fee on the basket, before the checkout button. Retain the final-step confirmation.
- No changes to fees, delivery availability, payment options or other copy. Recalculate both displays consistently if the basket or address changes.
Eligibility and randomisation
Randomise eligible signed-in customers 50:50 at their first basket view, before rendering either experience. Eligibility requires an address sufficient to quote delivery accurately.
Use a persistent customer-ID assignment shared across app and web. Session/device randomisation would contaminate exposure: 31% of buyers use both platforms. Anonymous customers are excluded unless we can reliably preserve and reconcile assignment.
Use an independent experiment hash, balanced within existing Payments assignment and first-exposure platform. Persist assignment throughout the experiment.
Analyse intention-to-treat: include everyone assigned, including customers who never enter checkout. Each customer contributes once to the primary outcome.
Metrics
Primary: Percentage of assigned customers placing at least one order within 24 hours of first eligible basket exposure, on either platform.
Secondary: - Checkout entry within 24 hours. - Basket-to-order conversion and final-step abandonment, with explicit funnel denominators. - Orders and revenue per assigned customer within 24 hours. - Order value, descriptive only: treatment can change who buys. - Repeat ordering within 30 days of assignment, reported after maturation. The supplied 38% is buyer-conditional, not this metric’s baseline.
Guardrails, precommitted proposed tolerances: - Revenue per assigned customer: no more than 1% relative deterioration. - Cancellation/refund incidence per assigned customer: no more than 0.2 percentage-point deterioration. - Checkout technical-error incidence per assigned customer: no more than 0.1 percentage-point deterioration. - Basket p95 latency: no more than 100 ms deterioration. - Any incorrect fee/date presentation is a stop-and-fix issue.
Instrument assignment, basket exposure, checkout steps, displayed quote and orders. Check sample-ratio mismatch, cross-platform consistency and event completeness before interpreting results.
Sample size and duration: sums
Across the supplied 56 days:
| Platform | Checkout sessions | Orders | Orders/session |
|---|---|---|---|
| --- | ---: | ---: | ---: |
| App | 1,206,537 | 85,526 | 7.09% |
| Web | 876,853 | 46,670 | 5.32% |
| Total | 2,083,390 | 132,196 | 6.35% |
Average checkout traffic:
\[ 2,083,390/56=37,203\text{ sessions/day}. \]
Planning assumptions: two-sided α=5%, 80% power, 5% relative lift as the minimum detectable effect—not an observed or commercially validated threshold.
Using checkout conversion only as a provisional proxy:
\[ p=132,196/2,083,390=0.063452,\quad \Delta=0.05p=0.003173. \]
Approximate equal-arm sample:
\[ n=\frac{2(1.96+0.84)^2p(1-p)}{\Delta^2} \approx92,600\text{ per arm}. \]
Thus approximately 185,200 independent observations. Naively:
\[ 185,200/37,203=4.98\text{ days}. \]
Five days is not a valid duration. These are checkout sessions, not unique eligible basket visitors; checkout entry is affected by treatment. The 3.8 sessions/month figure applies only to buyers, so dividing total sessions by 3.8 would not repair the denominator. Nor do 412,000 monthly buyers establish eligible traffic.
Pre-launch requirement: Measure deduplicated eligible basket visitors, their 24-hour conversion, and expected new enrolments over the planned window. Recalculate using that conversion rate; separately check guardrail power. Lock the largest required sample before launch.
Proposed maximum enrolment is 28 days, representing:
\[ 28\times37,203\approx1,041,695 \]
checkout sessions as a volume benchmark—not a count of independent participants. Launch only if forecast eligible customers support the locked sample within that window.
Calendar
Propose 12 October–8 November 2026, four complete weeks, with primary outcomes matured by 9 November.
This overlaps Pay Later. Obtain Payments agreement for independent allocation, retain its assignment in analysis, and estimate the treatment-by-payment interaction. Otherwise, do not use this window.
Search ends 30 October; log its assignment and account for calendar-date changes.
The uncontaminated pre-promotion window is only 10–19 November: ten days. The 16 November freeze requires any running experiment and kill switch to be preapproved; it is not automatically a shutdown. Use that alternative only if revised sizing supports it. Otherwise postpone until a confirmed post-promotion window.
Decision rule
At the locked endpoint, ship if the primary lift’s two-sided 95% confidence interval excludes zero positively and all guardrails pass their precommitted non-inferiority checks. Review platform and Payments interactions before broad rollout.
Reject on clear conversion harm or a guardrail breach. Otherwise call it inconclusive—not “no effect.” No efficacy peeking or opportunistic extension; safety monitoring remains continuous.
Claims the judge couldn’t find in the brief
- The supplied traffic supports a planning estimate, not a defensible final user-level sample size.
- The 16 November freeze requires any running experiment and kill switch to be preapproved; it is not automatically a shutdown.
- Nor do 412,000 monthly buyers establish eligible traffic.
Grades and run details
Decision model 89 · LLM judge 60
Decision model checks
- passUses the supplied evidence correctly31%
- passAddresses the actual decision14%
- partialRespects explicit constraints29%
- passIdentifies material uncertainty87%
- passAvoids unsupported claims70%
- partialProduces the required deliverable84%
- passRandomises by user93%
- passWorks around the calendar98%
- passDoesn't let the metric flatter the change82%
- passAn unambiguous primary metric65%
- passDecision rule written before the test77%
- partialSized from the real traffic85%
- passGuardrails with thresholds61%
- passSays why a test is worth it here32%
Run
- Run
- #1
- API response time
- 1.6 min
- Submitted
- 6 Oct 2026