Experiment specification

Experiment Specification: Early Delivery Fee & Date Preview

Author: PM, Checkout Squad Date: October 2026

---

1. Hypothesis

Showing the delivery date and delivery fee on the basket page (pre-checkout) will reduce final-step checkout abandonment by setting upfront expectations. * Core Hypothesis: The reduction in late-stage drop-off outweighs any early-stage basket abandonment caused by friction, resulting in a net positive impact on overall checkout conversion.

2. Randomisation Strategy & Unit

• Unit of Randomization: User ID (sticky across app and web platforms, given 31% multi-platform usage).
• Allocation: 50% Control (status quo: fee shown on final checkout step), 50% Treatment (fee and date visible on the basket page).

3. Metrics

• Primary Metric: Checkout Conversion Rate (Completed Orders / Unique Checkout Sessions).
• Secondary Metrics:
• Basket-to-Checkout Drop-off Rate (users leaving before entering checkout).
• Average Order Value (AOV).
• Guardrail Metrics:
• Total Orders / Gross Merchandise Value (GMV).
• 30-day Repeat Purchase Rate.

---

4. Traffic Analysis & Sample Size Sizing (The Sums)

Real Traffic Baseline (from past 8 weeks data): * Summing the 8-week dataset (56 days): * Total App Checkout Sessions = 1,304,784 (~23,300/day) * Total Web Checkout Sessions = 959,183 (~17,128/day) * Total Combined Sessions (8 weeks) = 2,263,967 * Daily Average Traffic = 40,428 checkout sessions/day * Total Orders = 155,752 * Baseline Conversion Rate ($p$) = $155,752 / 2,263,967 \approx 6.88\%$

Sample Size Calculations: To detect a relative change of 2% in conversion rate (e.g., from 6.88% to 7.02%) with $\alpha = 0.05$ (two-tailed) and power $(1-\beta) = 0.80$: * Using standard sample size formulas for proportions, we require approximately 375,000 users per variant (~750,000 total sessions). * At our baseline traffic of ~40,428 sessions/day (split 50/50 = ~20,214 sessions per variant daily): * Required Duration = $750,000 / 40,428 \approx 18.5$ days. * We will round up to 3 full weeks (21 days) to cleanly account for day-of-week seasonality (grocery ordering behaves very differently on Saturdays vs. Mondays).

---

5. Calendar & Timing

• Overlaps & Conflicts:
• Search squad: Runs exclusively on search results pages. Zero overlap with checkout/basket flows. Safe to run concurrently.
• Payments squad ("Pay later"): Runs on the checkout page (affecting 50% of users) until 9 November. Because the checkout page is downstream of our intervention (basket page), running both creates a multi-variate interaction risk if not accounted for.
• Holiday/Promo Constraints:
• Black Friday promotions start November 20.
• Code freeze begins November 16.
• Proposed Window:
• Start: Monday, October 12, 2026
• End: Monday, November 2, 2026 (Exactly 21 days).
• Note: This completely clears the Payments squad end-date (Nov 9) by launching earlier, and sits safely clear of the November 16 code freeze.

---

6. Decision Rule

We will evaluate the experiment after exactly 21 days of data collection.

1. Ship (Rollout 100%): If the Primary Metric (Checkout Conversion Rate) shows a statistically significant positive lift ($p < 0.05$), and no guardrail metrics (GMV, 30-day repeat rate) drop significantly.
2. Iterate / Inconclusive: If the primary metric shows no statistically significant change ($p \ge 0.05$), but secondary metrics show positive intent shifts, review basket abandonment drop-offs and redesign the UI placement.
3. Rollback: If the Primary Metric drops significantly ($p < 0.05$) due to early basket abandonment outweighing final-step savings, or if guardrail metrics degrade, immediately halt the experiment and revert to control.

Experiment spec: earlier delivery-date and fee disclosure

Status: Conditional launch approval. The supplied traffic supports a planning calculation, but not a defensible final sample size for the correct primary metric.

Hypothesis and experience

Showing the next-day delivery date and actual delivery fee on the basket page increases completed purchases by reducing late surprises. Earlier abandonment may increase; the test must measure the net effect from basket entry, not just conversion among people who proceed to checkout.

• Control: Current disclosure at the final checkout step.
• Treatment: Disclose on the basket page; retain an accurate fee/date in the final order summary. No changes to prices, delivery eligibility or payment options.
• Eligibility: identifiable, signed-in customers opening a non-empty basket for which we can calculate the date and fee. Do not introduce a login requirement. Results initially apply only to this population.

Randomisation and analysis

Assign eligible customers 50:50 by persistent account-ID hash, immediately before their first eligible basket renders. Keep assignment sticky across sessions, app and web.

Cross-platform consistency matters: \(412,000 × 31\% = 127,720\) recent buyers used both platforms. Device- or session-level assignment would expose many people to both variants.

Analyse intention-to-treat: everyone assigned remains in their arm, including failed renders and immediate exits. Each customer contributes one primary observation. Pre-specify adjustment for entry platform and enrolment date; report app/web effects as secondary, not independently powered launch decisions.

Validate allocation, identity stitching, exposure events and order attribution before starting.

Metrics

Primary: Percentage of assigned customers placing a paid order within 24 hours of first eligible basket entry, on either platform. This captures both earlier exits and downstream gains without a treatment-selected checkout denominator.

Secondary: - Basket-to-checkout progression and final-step abandonment, as funnel diagnostics. - Orders and net revenue per assigned customer within 24 hours. - Order value among purchasers, labelled a conditional diagnostic. - Thirty-day repeat purchase; the supplied 38% is context, not necessarily this cohort’s baseline. This needs a separate mature readout.

Guardrails — proposed tolerances, requiring sign-off: - Net revenue/customer: no worse than −1% relative. - Payment-error and cancellation/refund incidence per assigned customer: no worse than +0.1 percentage points, using fixed, pre-registered observation windows. - Zero tolerance for confirmed incorrect fee/date disclosure or unintended charges. - Monitor basket latency, crashes and sample-ratio mismatch; material regressions trigger investigation or rollback.

The $64 average order value alone cannot establish revenue variance or power these guardrails.

Sample size and duration: sums

Across the supplied 56 days:

Thus:

\[ p_{\text{checkout}}=132,196/2,083,390=6.345\% \]

\[ \text{sessions/day}=2,083,390/56=37,203 \]

Propose 80% power, two-sided 5% significance, 3% relative minimum detectable uplift, subject to commercial sign-off.

Illustrative checkout-session calculation only:

\[ \Delta=0.06345×0.03=0.0019035 \]

That is 0.190 percentage points.

\[ n_{\text{arm}}\approx \frac{2(1.96+0.84)^2p(1-p)}{\Delta^2} \approx257,200 \]

\[ N\approx514,400;\quad514,400/37,203=13.83\text{ days} \]

Two complete weeks would provide approximately:

\[ 2,083,390/8×2=520,848\text{ checkout sessions} \]

This does not prove the proposed experiment is powered. The intervention precedes checkout, and checkout sessions are neither eligible basket visitors nor independent customers. The 412,000 figure counts buyers only; 3.8 sessions/month describes buyers, not all eligible visitors. Dividing session traffic by 3.8 would be invalid.

Launch-blocking data request: Extract deduplicated eligible basket entrants, their 24-hour purchase rate and cumulative unique entrants over candidate windows. Recalculate:

\[ n{\text{arm}}= \frac{15.68pb(1-pb)}{(0.03pb)^2} \]

Choose a fixed enrolment duration covering at least two complete weeks and enough unique customers, then add 24 hours for primary-outcome maturation.

Calendar

There is no unconditionally clean two-week pre-promotion slot after Payments ends:

• Payments runs through 9 November.
• 10–15 November offers only six days before freeze.
• Even if running tests may continue during freeze, 10–19 November offers only ten days before promotions.

Conditional slot: 31 October–13 November, outcomes mature 14 November, readout 15 November. Search has ended. This requires Payments to hold its configuration through follow-up and agree independent, orthogonal assignment, balanced payment strata and interaction analysis. It also requires the recalculated customer sample to fit.

Otherwise, negotiate isolation earlier or defer until after promotions and freeze; their end dates are not supplied.

Decision rule

No efficacy peeking or early “winner” calls. At the fixed endpoint, ship only if the primary 95% confidence interval excludes zero positively, the estimated uplift meets the agreed 3% business threshold, and guardrail confidence bounds exclude their harm margins. Require valid instrumentation and no material platform/payment interaction harm.

Otherwise retain control: distinguish evidence of harm from an inconclusive result. Missing power, immature guardrails or calendar approval means no launch, not an assumed pass.