Experiment specification

Experiment spec: delivery date and fee on the basket page

Owner: Checkout PM · Squad: Checkout · Status: Draft for review

1. Hypothesis

Showing the delivery date and delivery fee on the basket page, before checkout, instead of only on the final checkout step, will increase orders per basket session (checkout completion) by at least +0.19 percentage points (about +3% relative, from 6.35% to 6.54%).

Mechanism: shoppers who learn the fee at the final step are surprised by it and leave. Seeing it early removes that surprise.

Counter-risk: some shoppers who would have paid the fee will leave at the basket page instead. The guardrails below test for this.

2. Population, unit and randomisation

• Population: all signed-in users who reach the basket page on app or web.
• Unit of randomisation: user, not session. Users average 3.8 checkout sessions a month, and 31% shop on both app and web. Session-level randomisation would show the same person both versions.
• Method: hash of the cross-platform user ID, salted per experiment, assigned 50% control / 50% treatment. The same assignment applies on app and web.
• Stratification: check balance by platform. App converts at 7.09% and web at 5.32% (see §4), and app is 57.9% of sessions.
• Dependency: this needs a cross-platform user ID. If we only have device or cookie IDs, cross-device contamination must be estimated before launch.

3. Metrics

Primary - Orders per basket session = orders ÷ basket sessions. Baseline 6.35%. This captures both early exits (at basket) and late exits (at checkout). Denominator needs confirming with the data team (see §8).

Secondary - Checkout-start rate from basket. - Average order value (baseline $64). - Orders per user over 14 days. - All of the above split by platform and by Pay later exposure (see §5).

Guardrails - Basket exit rate (leaving the basket without starting checkout). Must not rise by more than 0.5 percentage points absolute. This is the main test of the early-leave risk. - AOV must not fall by more than 2% ($1.28). - Delivery-fee contacts (support tickets mentioning fee) per 1,000 orders must not rise by more than 10%. - Order cancellations within 24 hours must not rise.

4. Sample size and duration

Baseline (from `checkouttrafficlast8weeks.csv`, 10 Aug to 4 Oct, 56 days): - Checkout sessions: 2,083,390 (about 260,000 a week, 37,000 a day) - Orders: 132,196 - Conversion: 132,196 ÷ 2,083,390 = 6.35% - Weekly range: 6.31% to 6.39% - App: 85,526 ÷ 1,206,537 = 7.09% (1,206,537 sessions) - Web: 46,670 ÷ 876,853 = 5.32% (876,853 sessions)

Power calculation (two-sided α = 0.05, power 80%, p₁ = 0.0635, p₂ = 0.0655): - Variance sum: 0.0635 × 0.9365 + 0.0655 × 0.9345 = 0.05946 + 0.06121 = 0.12067 - (z₀.₉₇₅ + z₀.₈₀)² = (1.96 + 0.8416)² = 7.849 - n per arm = 7.849 × 0.12067 ÷ (0.0020)² = 0.9472 ÷ 0.000004 = 236,800 sessions - Design effect for repeat sessions per user, assumed 1.2: 236,800 × 1.2 = 284,000 per arm, 568,000 total

Duration: - Traffic per arm: 260,000 ÷ 2 = 130,000 a week. - 284,000 ÷ 130,000 = 2.2 weeks. Two weeks is too short: it does not cover the full weekday cycle and gives no margin. - Run 3 full weeks (21 days). That gives 390,000 sessions per arm, which exceeds the 284,000 needed. Each weekday appears three times.

Sensitivity: detecting +0.10 pp needs 0.9472 ÷ 0.000001 × 1.2 ≈ 1.14M per arm, about 9 weeks. Not feasible, so +0.19 pp is the smallest effect this test can detect.

5. Calendar and overlaps

Proposed window: launch Mon 12 Oct, run 21 days, end Sun 1 Nov. Readout by 5 Nov, decision before the 16 Nov freeze.

• Must start by 26 Oct to finish before 16 Nov, so there is no slack for later starts.
• Pay later overlap: our test runs alongside Pay later until 9 Nov. Randomise independently (separate salt) so the two effects cross, and analyse the interaction. A 2×2 readout is enough. If Pay later changes how people respond to fees, the interaction term will show it.
• Search overlap: the search test runs until 30 Oct and changes which products people add to baskets. Monitor basket-value mix in both arms. Not a blocker.

6. Analysis

• Pre-registered, one readout at day 21. No interim peeking.
• Two-sided test for the primary metric; report 95% CI.
• SRM check: expected 50/50 split. Flag if the split is off by more than 0.5% and do not read results until resolved.
• Cluster-robust SEs by user.
• Pre-specified subgroups: platform, Pay later exposure. Report only; no decisions on them.

7. Decision rule

8. Open questions and data gaps

1. Basket session counts are missing. The CSV only has checkout sessions. Without basket sessions the primary metric and basket exit rate cannot be measured. Data team to supply by 8 Oct or we fall back to checkout-based conversion, which misses early exits.
2. Audience numbers do not reconcile. The data team says 412,000 people placed an order in the last 30 days, but the CSV shows about 71,000 orders a month (132,196 ÷ 56 × 30 ≈ 70,800). At 3.8 sessions a month, 412,000 buyers would imply about 1.57M checkout sessions a month, against about 1.13M in the CSV (260,000 × 4.33). One of these definitions is wrong. I used the CSV, which is daily and specific to checkout, for all sample sizes.
3. Cross-platform user ID. Needed for user-level randomisation (§2).
4. Confirm launch date and build owner. Build must be ready by 9 Oct for a 12 Oct launch.

Waymark checkout experiment spec

Hypothesis

Showing the delivery date and fee on the basket page will increase the share of eligible shoppers who place an order by setting expectations earlier and reducing last-step abandonment. It may also cause some shoppers to leave sooner; the primary metric must capture both effects.

Design and randomisation

Randomise eligible shoppers 50:50 when they first view a basket with delivery information available:

• Control: Current experience; date and fee appear at the final checkout step.
• Treatment: Date and fee appear on the basket page and remain visible through checkout.

Assign by stable customer ID across app and web, so shoppers who use both platforms see the same variant. Use a persistent anonymous ID for guests and merge it with the account on login. Stratify assignment by platform at first exposure.

Metrics

• Primary: Order conversion among eligible basket visitors: shoppers placing an order divided by shoppers randomised. This captures both earlier exits and improved final-step completion.
• Secondary: Checkout-start rate; completion among checkout starters; step-by-step abandonment; time to order; AOV; results by app/web.
• Guardrails: Basket-to-order conversion must not fall by more than 0.3 percentage points; monitor cancellations, fee-related support contacts, payment failures and site/app errors.

Sample size and duration

The traffic file covers 56 days. Summing its daily rows gives approximately 2.09m checkout sessions and 132k orders:

• Baseline checkout-session conversion: 132k ÷ 2.09m ≈ 6.3%
• Average traffic: 2.09m ÷ 56 ≈ 37.3k sessions/day, or 18.65k/arm/day at 50:50.

Power for a 5% relative lift (6.3% to 6.615%), with 80% power and two-sided 5% significance:

\[ n_{\text{per arm}}\approx \frac{2(1.96+0.84)^2(0.063)(0.937)}{(0.00315)^2} \approx 93{,}000 \]

Because shoppers can generate repeated sessions, conservatively inflate by the supplied 3.8 checkout sessions per buyer per month (worst-case design effect, assuming fully correlated sessions):

\[ 93{,}000 \times 3.8 \approx 353{,}000 \text{ sessions/arm} \]

\[ 353{,}000 \div 18{,}650 \approx 18.9 \text{ days} \]

Plan for at least 19 days, including full weeks and weekends, then allow time for order outcomes to mature. This is a provisional calculation: the file reports checkout sessions, not eligible basket visitors, and therefore cannot establish the primary metric’s baseline or power. Before launch, confirm basket-visitor volume and conversion, and validate the repeat-session assumption against deduplicated users. The supplied 412k monthly buyers and session-frequency figures should also be reconciled with observed traffic.

Calendar

Do not run concurrently with Payments’ checkout experiment, which ends 9 November. Starting 10 November would require about 19 days, crossing both the 16 November code freeze and 20 November Black Friday promotions. Defer until those changes and promotions are over and the freeze is lifted; the calendar does not specify those end dates. Confirm a stable window of at least three weeks before scheduling.

Decision rule

Ship only if the primary conversion lift is statistically significant at two-sided p < .05, the 95% confidence interval excludes zero, and the basket-to-order guardrail is not breached. Otherwise, do not ship; investigate platform and funnel-step results before proposing a follow-up. Don’t stop early for a positive result; stop only for a pre-agreed severe guardrail issue or data-quality failure.

Experiment spec: delivery date and fee on the basket page

Hypothesis

Showing the delivery date and fee on the basket page, rather than at the final checkout step, raises the share of basket sessions that end in an order. We expect fewer last-step abandons because the fee is no longer a surprise. The counter-risk is more exits at the basket, so the primary metric must count both.

Randomisation

• Unit: logged-in user ID, not session or device. 31% of buyers use both app and web, and buyers average 3.8 sessions a month. Session-level assignment would show one person both experiences and make observations non-independent.
• Split: 50/50 by salted hash of user ID, the same arm on app and web. Users are assigned on first basket view.
• Independence: use a different salt from the Payments squad's "Pay later" test, so every arm is about 50% Pay-later (see Calendar).
• Checks: run an A/A check in the week before launch. Run a sample-ratio-mismatch (SRM) test throughout, and pause if p < 0.001.

Metrics

Primary: order conversion = basket-page sessions that end in an order ÷ basket-page sessions, with standard errors clustered by user.

Do not use orders ÷ checkout sessions. If the fee scares people off at the basket, they never become checkout sessions, the denominator shrinks, and the treatment looks better than it is.

Secondary: - Basket→checkout-start rate (expected to fall, and we need to see by how much). - Checkout-start→order rate (the squad's mechanism). - Revenue per basket session. - Orders per user over the window. - Platform split (exploratory only).

Guardrails: - Average order value (AOV, baseline $64): fail if the lower CI bound is below −2% (−$1.28). - Revenue per basket session: fail if significantly negative. - Fee-related support contacts per 1,000 orders. - Basket-page error rate and latency. - 14-day repeat rate among exposed users. The 30-day repeat rate (38%) can't be read in a 3-week test.

Baseline from the traffic file

• Weekly sessions are about 260k and daily volumes are stable, so I use p = 6.35%.
• Platform split: app 57–58% of sessions at about 7.1% conversion; web 42% at about 5.3%.
• Weekends run at about 65% of weekday volume, so the test must cover whole weeks.

Sample size

• MDE: +3% relative, which is +0.19pp (6.35% → 6.54%). That is roughly 490 orders a week, or about $31k a week at $64 AOV, which is worth shipping for.
• Settings: two-sided α = 0.05, power 80%, so (1.96 + 0.84)² = 7.84.
• Per arm, ignoring clustering: n = 2 × 7.84 × p(1−p) / δ²
• p(1−p) = 0.0635 × 0.9365 = 0.0595
• δ = 0.00191, so δ² = 3.63×10⁻⁶
• n = 0.932 / 3.63×10⁻⁶ ≈ 257k sessions
• Clustering: users contribute about 2.7 sessions in 3 weeks. I assume an intra-user correlation of 0.2, giving a design effect of 1 + 1.7 × 0.2 ≈ 1.3. That puts the requirement at 257k × 1.3 ≈ 334k sessions per arm.
• Duration: each arm gets about 130k sessions a week (260k ÷ 2), so 334k ÷ 130k = 2.6 weeks. Run 3 full weeks (21 days), which gives about 390k per arm.
• For comparison: a 2% MDE needs 578k × 1.3 = 751k per arm, or 5.8 weeks, which the calendar can't fit. A 5% MDE needs 92k × 1.3 = 120k per arm, under 1 week.
• Subgroups: web alone (42% of traffic) can only detect about 4.6% relative (3% × √(1/0.42)). Platform cuts are exploratory.

Data issues to resolve before launch

1. Buyer count doesn't match the traffic file. The data team's 412k buyers in 30 days is inconsistent with about 70k orders a month in the file (16.4k a week × 30/7). Their 3.8 sessions per buyer implies about 1.57M sessions, against about 1.11M in the file. I sized from the traffic file. The data team needs to reconcile the two, since the user-level assumptions (the design effect) depend on it.
2. Check the definition of `checkout_sessions`. I assumed it equals basket-page sessions. If it only counts sessions that reached checkout, the true basket-level baseline is lower and the test is longer. For example, at 4% conversion, n scales by (0.96/0.04) ÷ (0.9365/0.0635) ≈ 1.6×, which is about 4.5 weeks.
3. Re-run the sums on the real basket-session baseline before launch, and fix the duration then (3 or 4 weeks).

Calendar

• Black Friday: promotions start 20 Nov and code freeze starts 16 Nov. Promo traffic isn't representative, and anything not shipped by 15 Nov waits until after the freeze.
• Waiting for Pay later to end (9 Nov) doesn't work. That leaves 10–15 Nov, which is 6 days and not enough.
• Pay later (until 9 Nov) sits on the checkout page and affects the same outcome, so it can't be ignored. I'll run the test concurrently with an independent split. Each arm then gets about the same Pay-later mix, so the contrast stays unbiased. I'll pre-register a treatment × Pay-later interaction check, and report the effect by Pay-later arm. The Payments squad must freeze their allocation for the duration.
• Search ranking (until 30 Oct) only touches search results pages. It's balanced by randomisation and needs no action, but I'll note it.
• Schedule:
• Build, QA and A/A: 5–11 Oct.
• Run 12 Oct – 1 Nov (3 weeks).
• Readout 2–4 Nov, decision before 9 Nov.
• Ship ramp before the 16 Nov freeze.
• Buffer: a 4-week run (to 8 Nov) still ends before Pay later does.

Decision rule

This is fixed in advance, with one analysis at the end of the run and no peeking.

• Ship if the primary lift is positive with the 95% CI excluding 0, and no guardrail fails, and revenue per basket session isn't significantly negative.
• Do not ship if the CI includes 0. We'd conclude "no detectable effect at +3%". Any wish to ship for fee-transparency reasons is a separate product decision, not an experiment win.
• Kill if the primary is significantly negative, or if AOV or revenue per basket session breaches its guardrail. Basket exits are the main thing we expect to see here.
• Extend: only if an outage or traffic shortfall leaves fewer than 334k sessions per arm, up to 28 days. Never extend because the result is "almost significant".
• Interaction: if the effect differs materially by Pay-later arm, ship only to the configuration that will be live after 9 Nov, and confirm with the Payments squad which one that is.