Needs more than a quick edit11 of 14 checks passed
The spec lacks a final sample size and duration, requiring additional data and recalculation, so a PM cannot act on it without major rework.
Output
Experiment spec: earlier delivery-date and fee disclosure
Status: Conditional launch approval. The supplied traffic supports a planning calculation, but not a defensible final sample size for the correct primary metric.
Hypothesis and experience
Showing the next-day delivery date and actual delivery fee on the basket page increases completed purchases by reducing late surprises. Earlier abandonment may increase; the test must measure the net effect from basket entry, not just conversion among people who proceed to checkout.
- Control: Current disclosure at the final checkout step.
- Treatment: Disclose on the basket page; retain an accurate fee/date in the final order summary. No changes to prices, delivery eligibility or payment options.
- Eligibility: identifiable, signed-in customers opening a non-empty basket for which we can calculate the date and fee. Do not introduce a login requirement. Results initially apply only to this population.
Randomisation and analysis
Assign eligible customers 50:50 by persistent account-ID hash, immediately before their first eligible basket renders. Keep assignment sticky across sessions, app and web.
Cross-platform consistency matters: \(412,000 × 31\% = 127,720\) recent buyers used both platforms. Device- or session-level assignment would expose many people to both variants.
Analyse intention-to-treat: everyone assigned remains in their arm, including failed renders and immediate exits. Each customer contributes one primary observation. Pre-specify adjustment for entry platform and enrolment date; report app/web effects as secondary, not independently powered launch decisions.
Validate allocation, identity stitching, exposure events and order attribution before starting.
Metrics
Primary: Percentage of assigned customers placing a paid order within 24 hours of first eligible basket entry, on either platform. This captures both earlier exits and downstream gains without a treatment-selected checkout denominator.
Secondary: - Basket-to-checkout progression and final-step abandonment, as funnel diagnostics. - Orders and net revenue per assigned customer within 24 hours. - Order value among purchasers, labelled a conditional diagnostic. - Thirty-day repeat purchase; the supplied 38% is context, not necessarily this cohort’s baseline. This needs a separate mature readout.
Guardrails — proposed tolerances, requiring sign-off: - Net revenue/customer: no worse than −1% relative. - Payment-error and cancellation/refund incidence per assigned customer: no worse than +0.1 percentage points, using fixed, pre-registered observation windows. - Zero tolerance for confirmed incorrect fee/date disclosure or unintended charges. - Monitor basket latency, crashes and sample-ratio mismatch; material regressions trigger investigation or rollback.
The $64 average order value alone cannot establish revenue variance or power these guardrails.
Sample size and duration: sums
Across the supplied 56 days:
| Platform | Checkout sessions | Orders |
|---|---|---|
| --- | ---: | ---: |
| App | 1,206,537 | 85,526 |
| Web | 876,853 | 46,670 |
| Total | 2,083,390 | 132,196 |
Thus:
\[ p_{\text{checkout}}=132,196/2,083,390=6.345\% \]
\[ \text{sessions/day}=2,083,390/56=37,203 \]
Propose 80% power, two-sided 5% significance, 3% relative minimum detectable uplift, subject to commercial sign-off.
Illustrative checkout-session calculation only:
\[ \Delta=0.06345×0.03=0.0019035 \]
That is 0.190 percentage points.
\[ n_{\text{arm}}\approx \frac{2(1.96+0.84)^2p(1-p)}{\Delta^2} \approx257,200 \]
\[ N\approx514,400;\quad514,400/37,203=13.83\text{ days} \]
Two complete weeks would provide approximately:
\[ 2,083,390/8×2=520,848\text{ checkout sessions} \]
This does not prove the proposed experiment is powered. The intervention precedes checkout, and checkout sessions are neither eligible basket visitors nor independent customers. The 412,000 figure counts buyers only; 3.8 sessions/month describes buyers, not all eligible visitors. Dividing session traffic by 3.8 would be invalid.
Launch-blocking data request: Extract deduplicated eligible basket entrants, their 24-hour purchase rate and cumulative unique entrants over candidate windows. Recalculate:
\[ n{\text{arm}}= \frac{15.68pb(1-pb)}{(0.03pb)^2} \]
Choose a fixed enrolment duration covering at least two complete weeks and enough unique customers, then add 24 hours for primary-outcome maturation.
Calendar
There is no unconditionally clean two-week pre-promotion slot after Payments ends:
- Payments runs through 9 November.
- 10–15 November offers only six days before freeze.
- Even if running tests may continue during freeze, 10–19 November offers only ten days before promotions.
Conditional slot: 31 October–13 November, outcomes mature 14 November, readout 15 November. Search has ended. This requires Payments to hold its configuration through follow-up and agree independent, orthogonal assignment, balanced payment strata and interaction analysis. It also requires the recalculated customer sample to fit.
Otherwise, negotiate isolation earlier or defer until after promotions and freeze; their end dates are not supplied.
Decision rule
No efficacy peeking or early “winner” calls. At the fixed endpoint, ship only if the primary 95% confidence interval excludes zero positively, the estimated uplift meets the agreed 3% business threshold, and guardrail confidence bounds exclude their harm margins. Require valid instrumentation and no material platform/payment interaction harm.
Otherwise retain control: distinguish evidence of harm from an inconclusive result. Missing power, immature guardrails or calendar approval means no launch, not an assumed pass.
Grades and run details
Decision model 89 · LLM judge 73
Decision model checks
- passUses the supplied evidence correctly45%
- passAddresses the actual decision51%
- partialRespects explicit constraints24%
- passIdentifies material uncertainty96%
- passAvoids unsupported claims64%
- partialProduces the required deliverable88%
- passRandomises by user98%
- passWorks around the calendar100%
- passDoesn't let the metric flatter the change81%
- passAn unambiguous primary metric83%
- passDecision rule written before the test82%
- partialSized from the real traffic78%
- passGuardrails with thresholds58%
- passSays why a test is worth it here85%
Run
- Run
- #1
- Time to output
- 73 s
- Submitted
- 5 Oct 2026