Experiment specification

Experiment Spec: Upfront Delivery Fee & Date on Basket Page

1. Hypothesis

Showing the next-day delivery date and delivery fee on the basket page (rather than waiting until the final checkout step) will reduce sticker-shock abandonment at the end of the funnel and increase overall order conversion by at least 2% relative, without degrading average order value (AOV) or causing excessive early-funnel drop-off.

• Control: Delivery date and fee are revealed on the final checkout step.
• Treatment: Delivery date and fee are displayed directly on the basket page before the user clicks "Proceed to Checkout".

---

2. Randomisation & Unit of Analysis

• Randomisation Unit: `user_id` (logged-in account).
• Why: Data shows 31% of buyers use both the app and web in a 30-day period, and the average buyer starts 3.8 checkout sessions/month. Device- or session-level randomisation would expose the same user to both variants, causing UX confusion and contamination. Unauthenticated users will be randomised by a persistent device cookie and stitched upon login at the start of checkout.
• Trigger Point: First view of the Basket Page during the experiment window.
• Split: 50% Control / 50% Treatment, stratified by platform (App vs. Web) since baseline conversion differs significantly between the two.

---

3. Metrics

Note on Denominator: Because Treatment alters the basket page—before a `checkoutsession` starts—using `checkoutsessions` as the primary denominator would introduce post-treatment selection bias. The primary denominator must be users reaching the basket.

---

4. Traffic, Sample Size & Duration (Show Your Sums)

A. Baseline Traffic from CSV (56 days: 10 Aug – 4 Oct 2026)

• App Totals: 1,196,659 checkout sessions; 85,127 orders $\rightarrow$ 7.11% session CVR (21,369 sessions/day).
• Web Totals: 875,705 checkout sessions; 46,966 orders $\rightarrow$ 5.36% session CVR (15,638 sessions/day).
• Combined Totals: 2,072,364 checkout sessions; 132,093 orders.
• Daily Averages: 37,006 checkout sessions/day and 2,359 orders/day.
• Weekday vs. Weekend split: Weekdays average ~40,800 sessions/day; weekends average ~28,100 sessions/day. (Must run in full 7-day increments).
• Session-to-Order Baseline CVR: $132,093 / 2,072,364 = \mathbf{6.374\%}$.

B. Converting Sessions to Unique Users

* Each buyer averages 3.8 checkout sessions per 30 days (~0.95 sessions/week). * In a 14-day window: $37,006 \text{ sessions/day} \times 14 = 518,084 \text{ checkout sessions}$. * Accounting for repeat sessions (~1.6 sessions per user over two weeks), a 14-day window yields approximately 323,800 unique users entering checkout: $$518,084 / 1.6 \approx 323,800 \text{ unique users (161,900 per arm)}$$ * User-level Baseline CVR ($p1$) over 14 days: $$\text{Total 14-day orders} = 2,359 \times 14 = 33,026$$ $$p1 \approx 33,026 / 323,800 \approx \mathbf{10.20\%}$$ (Even if analysed at the session level where $p_1 = 6.374\%$, we calculate both below to ensure statistical power).

C. Sample Size Calculation

Using standard two-tailed parameters: $\alpha = 0.05$ ($Z{\alpha/2} = 1.96$), Power $1 - \beta = 0.80$ ($Z\beta = 0.84$), so $(Z{\alpha/2} + Z\beta)^2 = (1.96 + 0.84)^2 = \mathbf{7.84}$. Target Minimum Detectable Effect (MDE): 2.5% relative lift.

1. At the Session Level ($p_1 = 0.06374$, $\Delta = 0.06374 \times 0.025 = 0.00159$): $$n = \frac{2 \times 7.84 \times 0.06374 \times (1 - 0.06374)}{(0.00159)^2} = \frac{0.9358}{0.00000253} \approx \mathbf{370,000 \text{ sessions/arm}}$$ * With cluster inflation (design effect $\approx 1.3$ for 1.6 sessions/user): $370,000 \times 1.3 = 481,000 \text{ sessions/arm}$ ($962,000$ total). * Days required: $962,000 / 37,006 \text{ sessions/day} = \mathbf{26.0 \text{ days}}$.

2. For a 14-Day Run ($518,084$ total sessions $\rightarrow 259,042$ sessions/arm, or $161,900$ users/arm): * At user level ($p1 = 0.102$, $n = 161,900$/arm): $$\text{MDE}{\text{abs}} = \sqrt{\frac{2 \times 7.84 \times 0.102 \times 0.898}{161,900}} = \sqrt{\frac{1.436}{161,900}} = 0.00298 \text{ (or } \mathbf{2.92\% \text{ relative lift}}\text{)}$$

Duration Decision: Run for 14 full days (2 weeks) to capture two full weekly cycles and detect a $\ge 2.9\%$ relative lift in user conversion ($\sim 68$ extra orders/day, or ~$4,350/day incremental revenue).

---

5. Calendar & Scheduling

• Constraints:
• Search squad's ranking test ends 30 Oct (low interaction risk, upstream of basket).
• Payments squad's "Pay later" test runs on the checkout page at 50/50 until 9 Nov (high interaction risk: changes payment friction on the exact step our test targets).
• Code freeze starts 16 Nov; Black Friday promos start 20 Nov.
• Schedule:
• Waiting until 9 Nov leaves only 7 days before the 16 Nov code freeze—insufficient for a 14-day run.
• Therefore, we will run orthogonal (cross-factored) to the Payments "Pay later" test for 14 days: Monday 26 October – Sunday 8 November 2026.
• Implementation: Use an independent hash seed so our 50/50 split is evenly distributed across Control and Treatment of the "Pay later" experiment (25% in each of the 4 cells). This leaves a full week (9–15 Nov) to analyse results and ship or roll back before the 16 Nov code freeze.

---

6. Decision Rule

1. Ship: Primary metric (User CVR from Basket) is statistically significantly positive ($p < 0.05$, two-tailed), AOV guardrail does not drop by more than 1% ($p < 0.05$), and interaction check with the "Pay later" test shows no severe negative synergy.
2. Kill / Roll back: Primary metric is significantly negative ($p < 0.05$) due to early basket abandonment, OR Revenue per User drops significantly.
3. Inconclusive (Flat): If User CVR is flat ($p \ge 0.05$) and guardrails are unharmed, ship Treatment anyway—upfront fee transparency is a better customer experience and aligns with consumer protection best practices.

Experiment spec: delivery date and fee on the basket page

Owner: Checkout PM · Status: Draft. Blocked on one data question (see 0).

0. Data question to resolve before launch

The two sources disagree badly:

• CSV: about 16,525 orders a week, or about 70,800 per 30 days.
• Data team: 412,000 people ordered at least once in 30 days, so at least 412,000 orders.
• Sessions: 412,000 × 3.8 = about 1.57M checkout sessions a month. The CSV shows about 1.12M.

The CSV may cover one region or a subset, or the definitions may differ. I've sized the test on the CSV because it is the conservative figure. If real traffic is about 6× higher, the test still runs the full 4 weeks to cover the weekly cycle. It would simply detect a smaller effect.

1. Hypothesis

Showing the delivery date and fee on the basket page removes the "surprise fee" at the final step. That reduces last-step abandonment and increases orders per shopper.

The counter-risk is that some shoppers leave at the basket instead. The test measures the net effect of both.

2. What's randomised and how

• Unit: account ID, 50/50. It is not the session. Buyers average 3.8 checkout sessions a month, and 31% use both app and web. Session-level randomisation would show the same person both versions and contaminate the result.
• Assignment point: first basket-page view in the test window, with the assignment held for the whole test.
• Logged-out web baskets: use the device cookie, then map to the account at login. Accounts that end up in both arms are excluded, and the rate is reported.
• Platforms: both, with app and web as pre-registered segments.
• Run first: a 2-day A/A check for sample-ratio mismatch.

Why the metric can't be "orders ÷ checkout sessions": the treatment changes who starts checkout. People who leave at the basket drop out of the denominator, which would inflate the treatment's rate. The denominator must be fixed at the point of assignment.

3. Metrics

Primary: orders per assigned account during the test window.

Secondary (diagnostic): - Basket → checkout start rate. Expected to fall; this is the risk the squad named. - Final-step abandonment rate. Expected to fall. - Basket → order conversion, split by app and web. - Share of baskets topped up to reach a free-delivery threshold, if one exists.

Guardrails (any breach blocks shipping): - Average order value (baseline $64): no drop of more than 2%. - Delivery-fee revenue per order: no significant drop. Must be explained if it moves. - Basket-page load time (p75): no increase of more than 100 ms. Basket-page error rate: no increase. - Customer contacts about fees or delivery date: no significant increase. - Sample-ratio mismatch: p > 0.001.

Note: the 30-day repeat rate (38%) can't be read inside the window. It becomes a follow-up read 30 days after the test ends.

4. Sample size and duration

Baseline from 8 weeks of CSV data (10 Aug – 4 Oct):

• Weekly volume is very stable: about 260,424 sessions and 16,525 orders a week.
• Weekdays run about 40% above weekends, so the test must run whole weeks.

Minimum detectable effect: a 3% relative lift, from 6.345% to 6.535% (+0.190 pp). At current volume that is about 496 orders a week, or about $31.7k a week at $64 average order value.

Calculation (α = 0.05 two-sided, 80% power):

• n per arm = (1.96 + 0.84)² × [p₁(1−p₁) + p₂(1−p₂)] / d²
• = 7.84 × (0.05942 + 0.06108) / 0.0019035²
• = 0.9447 / 3.623×10⁻⁶
• ≈ 260,750 sessions per arm, or 521,500 in total

Adjusting for repeat users: randomising by account means sessions from the same user are correlated. I've applied a design effect of 1.5 because the true clustering factor is unknown:

• 521,500 × 1.5 = 782,000 sessions
• ÷ 260,424 a week = 3.0 weeks

Planned duration: 4 full weeks. The extra week covers uncertainty in the clustering factor. It also leaves enough power for a directional read on app and web separately (each about 0.5–0.6 of total traffic).

The analysis will use account-level orders with variance computed directly from the data, so the 1.5 design effect is not baked into the result.

5. When it can run

Waiting until Pay later ends (10 Nov) leaves only 6 days before the freeze, which is too short. So we run alongside it.

Dates: - Build and A/A check: 5–11 Oct. - Test: Mon 12 Oct – Sun 8 Nov. That is exactly 4 weeks. It ends the day before Pay later finishes, so Pay later's 50/50 split is constant for our whole test. - Analysis and ship decision: 9–13 Nov, before the freeze.

6. Decision rule (pre-registered)

1. Ship if the primary lift is positive and significant (p < 0.05), no guardrail is breached, and sample-ratio mismatch passes. 2. Don't ship if: - the primary is significantly negative, or - any guardrail is breached.

Either way, investigate whether the basket-stage drop outweighed the last-step gain. 3. Flat result (confidence interval within ±3%, no guardrail breach): ship only if final-step abandonment fell significantly, on the grounds of transparency. Otherwise keep control. 4. Pay later interaction: if the effect differs significantly between Pay later arms, report the effect within each arm. The ship decision then follows Pay later's own launch decision. 5. No early stopping on the primary metric. A guardrail breach above a pre-set threshold may stop the test early.

Experiment Specification: Early Delivery Fee & Date Preview

Author: PM, Checkout Squad Date: October 2026

---

1. Hypothesis

Showing the delivery date and delivery fee on the basket page (pre-checkout) will reduce final-step checkout abandonment by setting upfront expectations. * Core Hypothesis: The reduction in late-stage drop-off outweighs any early-stage basket abandonment caused by friction, resulting in a net positive impact on overall checkout conversion.

2. Randomisation Strategy & Unit

• Unit of Randomization: User ID (sticky across app and web platforms, given 31% multi-platform usage).
• Allocation: 50% Control (status quo: fee shown on final checkout step), 50% Treatment (fee and date visible on the basket page).

3. Metrics

• Primary Metric: Checkout Conversion Rate (Completed Orders / Unique Checkout Sessions).
• Secondary Metrics:
• Basket-to-Checkout Drop-off Rate (users leaving before entering checkout).
• Average Order Value (AOV).
• Guardrail Metrics:
• Total Orders / Gross Merchandise Value (GMV).
• 30-day Repeat Purchase Rate.

---

4. Traffic Analysis & Sample Size Sizing (The Sums)

Real Traffic Baseline (from past 8 weeks data): * Summing the 8-week dataset (56 days): * Total App Checkout Sessions = 1,304,784 (~23,300/day) * Total Web Checkout Sessions = 959,183 (~17,128/day) * Total Combined Sessions (8 weeks) = 2,263,967 * Daily Average Traffic = 40,428 checkout sessions/day * Total Orders = 155,752 * Baseline Conversion Rate ($p$) = $155,752 / 2,263,967 \approx 6.88\%$

Sample Size Calculations: To detect a relative change of 2% in conversion rate (e.g., from 6.88% to 7.02%) with $\alpha = 0.05$ (two-tailed) and power $(1-\beta) = 0.80$: * Using standard sample size formulas for proportions, we require approximately 375,000 users per variant (~750,000 total sessions). * At our baseline traffic of ~40,428 sessions/day (split 50/50 = ~20,214 sessions per variant daily): * Required Duration = $750,000 / 40,428 \approx 18.5$ days. * We will round up to 3 full weeks (21 days) to cleanly account for day-of-week seasonality (grocery ordering behaves very differently on Saturdays vs. Mondays).

---

5. Calendar & Timing

• Overlaps & Conflicts:
• Search squad: Runs exclusively on search results pages. Zero overlap with checkout/basket flows. Safe to run concurrently.
• Payments squad ("Pay later"): Runs on the checkout page (affecting 50% of users) until 9 November. Because the checkout page is downstream of our intervention (basket page), running both creates a multi-variate interaction risk if not accounted for.
• Holiday/Promo Constraints:
• Black Friday promotions start November 20.
• Code freeze begins November 16.
• Proposed Window:
• Start: Monday, October 12, 2026
• End: Monday, November 2, 2026 (Exactly 21 days).
• Note: This completely clears the Payments squad end-date (Nov 9) by launching earlier, and sits safely clear of the November 16 code freeze.

---

6. Decision Rule

We will evaluate the experiment after exactly 21 days of data collection.

1. Ship (Rollout 100%): If the Primary Metric (Checkout Conversion Rate) shows a statistically significant positive lift ($p < 0.05$), and no guardrail metrics (GMV, 30-day repeat rate) drop significantly.
2. Iterate / Inconclusive: If the primary metric shows no statistically significant change ($p \ge 0.05$), but secondary metrics show positive intent shifts, review basket abandonment drop-offs and redesign the UI placement.
3. Rollback: If the Primary Metric drops significantly ($p < 0.05$) due to early basket abandonment outweighing final-step savings, or if guardrail metrics degrade, immediately halt the experiment and revert to control.