Experiment specification

Experiment Spec: Upfront Delivery Fee & Date on Basket

Squad: Checkout Author: PM, Checkout Date: October 5, 2026

---

1. Hypothesis & Context

• Hypothesis: Displaying the delivery date and delivery fee upfront on the basket page (instead of at the final checkout step) will reduce unexpected cost friction at checkout, decreasing last-step abandonment and increasing overall basket-to-order conversion.
• Risk: Showing fees earlier may induce immediate drop-off on the basket page, reducing the number of checkout sessions started.
• Core Question: Does the increase in final checkout completion compensate for any drop in basket-to-checkout progression?

---

2. Randomisation & Allocation

• Unit of Randomisation: User ID (with persistent Device/Visitor ID fallback for non-logged-in users).
• Why: Data shows 31% of monthly buyers shop on both the app and the website, and users average 3.8 checkout sessions/month. Randomising by session or device would cause severe cross-platform contamination and inconsistent user experiences.
• Split: 50% Control, 50% Treatment, stratified by platform (iOS, Android, Web).
• Variants:
• Control (A): Current flow. Delivery date and fee revealed on final checkout step.
• Treatment (B): Delivery date and calculated fee dynamically displayed on the basket page prior to entering checkout.

---

3. Metrics Framework

* Primary Metric: * Basket-to-Order Conversion Rate (Orders / Unique Basket Sessions): Note: We must not use Checkout-to-Order CVR alone as the primary metric, because filtering out users on the basket page could inflate downstream checkout conversion while reducing total orders. * Secondary Metrics (Funnel Diagnostics): * Basket-to-Checkout Initiation Rate: Measures upfront drop-off risk (Basket views $\to$ Checkout step 1). * Final-Step Checkout Abandonment Rate: Anticipated to drop as fee surprises are eliminated. * Average Order Value (AOV): Baseline is $64; monitors whether upfront fees alter basket composition. * Guardrail Metrics: * Total Orders & Revenue per User: Must not see a statistically significant decrease. * Basket Page P95 Latency: Fee lookup API must not add $>150\text{ ms}$ to basket load time. * CS Contact Rate: Delivery fee disputes/inquiries per 1,000 orders.

---

4. Traffic Sums, Sample Size & Duration

#### A. Baseline Traffic & Conversion (from 8-week data: Aug 10 – Oct 4, 2026) * Total 8-week volume: * App checkout sessions: $1,202,306$ | App orders: $85,466$ ($7.11\%$ CVR) * Web checkout sessions: $874,188$ | Web orders: $46,756$ ($5.35\%$ CVR) * Combined totals: $2,076,494$ checkout sessions | $132,222$ orders * Weekly averages: * Sessions: $\approx 259,560$ sessions/week ($\approx 37,080$ sessions/day) * Orders: $\approx 16,528$ orders/week ($\approx 2,360$ orders/day) * Baseline Checkout Conversion Rate ($p$): $$p = \frac{132,222}{2,076,494} = 6.37\%$$

#### B. Sample Size Calculation Using standard two-tailed hypothesis testing parameters: * Significance level ($\alpha$): $0.05$ ($Z{\alpha/2} = 1.96$) * Statistical power ($1 - \beta$): $80\%$ ($Z{\beta} = 0.84$) * Minimum Detectable Effect (MDE): $2.5\%$ relative lift ($\Delta = 6.37\% \times 0.025 = 0.159\%$ absolute $\to p_2 = 6.53\%$)

$$\text{Sample size per variant } n = \frac{(Z{\alpha/2} + Z\beta)^2 \cdot 2 \cdot p(1 - p)}{\Delta^2}$$ $$n = \frac{(1.96 + 0.84)^2 \cdot 2 \cdot (0.0637 \cdot 0.9363)}{(0.00159)^2} = \frac{7.84 \cdot 2 \cdot 0.05964}{0.000002528} \approx 369,980 \text{ sessions/variant}$$

• Total required sessions (Control + Treatment): $\approx 740,000$ checkout/basket sessions.

#### C. Duration * Total traffic per week $\approx 259,560$ sessions. * Required weeks: $740,000 / 259,560 \approx 2.85$ weeks. * To eliminate day-of-week seasonality (e.g., weekend drop to $\approx 28\text{k}$ sessions/day vs. weekday $\approx 41\text{k}$), we set the duration to exactly 3 full calendar weeks (21 days), providing $\approx 778,000$ sessions and achieving an MDE of $\approx 2.4\%$.

---

5. Calendar & Schedule

``` [Oct 5] Spec Approval & QA [Oct 12] Launch Experiment (Day 1) ├── Oct 12 – Nov 1: 3-Week Test Window └── Concurrently: Orthogonal to Payments "Pay later" [Nov 1] Conclude Test & Analysis [Nov 2 – Nov 6] Squad Review & Rollout Decision [Nov 9 – Nov 13] 100% Rollout (if successful) [Nov 16] CODE FREEZE for Black Friday ```

• Constraints & Conflicts:
• Code Freeze: Begins 16 November.
• Payments Squad ("Pay later" on checkout): Live for 50% of users until 9 November. We cannot wait until Nov 9, as that leaves only 7 days before code freeze.
• Search Ranking Test: Ends 30 October on search result pages only (no direct conflict).
• Interference Mitigation:
• Randomise our test on an orthogonal hash layer to Payments. Because assignment is independent, exposure to "Pay later" will balance evenly (50/50) across both our Control and Treatment groups.
• Check for an interaction effect via a $2 \times 2$ factorial ANOVA post-test.

---

6. Decision Rules

Experiment spec: earlier delivery-date and fee disclosure

Status: Conditional launch approval. The supplied traffic supports a planning calculation, but not a defensible final sample size for the correct primary metric.

Hypothesis and experience

Showing the next-day delivery date and actual delivery fee on the basket page increases completed purchases by reducing late surprises. Earlier abandonment may increase; the test must measure the net effect from basket entry, not just conversion among people who proceed to checkout.

• Control: Current disclosure at the final checkout step.
• Treatment: Disclose on the basket page; retain an accurate fee/date in the final order summary. No changes to prices, delivery eligibility or payment options.
• Eligibility: identifiable, signed-in customers opening a non-empty basket for which we can calculate the date and fee. Do not introduce a login requirement. Results initially apply only to this population.

Randomisation and analysis

Assign eligible customers 50:50 by persistent account-ID hash, immediately before their first eligible basket renders. Keep assignment sticky across sessions, app and web.

Cross-platform consistency matters: \(412,000 × 31\% = 127,720\) recent buyers used both platforms. Device- or session-level assignment would expose many people to both variants.

Analyse intention-to-treat: everyone assigned remains in their arm, including failed renders and immediate exits. Each customer contributes one primary observation. Pre-specify adjustment for entry platform and enrolment date; report app/web effects as secondary, not independently powered launch decisions.

Validate allocation, identity stitching, exposure events and order attribution before starting.

Metrics

Primary: Percentage of assigned customers placing a paid order within 24 hours of first eligible basket entry, on either platform. This captures both earlier exits and downstream gains without a treatment-selected checkout denominator.

Secondary: - Basket-to-checkout progression and final-step abandonment, as funnel diagnostics. - Orders and net revenue per assigned customer within 24 hours. - Order value among purchasers, labelled a conditional diagnostic. - Thirty-day repeat purchase; the supplied 38% is context, not necessarily this cohort’s baseline. This needs a separate mature readout.

Guardrails — proposed tolerances, requiring sign-off: - Net revenue/customer: no worse than −1% relative. - Payment-error and cancellation/refund incidence per assigned customer: no worse than +0.1 percentage points, using fixed, pre-registered observation windows. - Zero tolerance for confirmed incorrect fee/date disclosure or unintended charges. - Monitor basket latency, crashes and sample-ratio mismatch; material regressions trigger investigation or rollback.

The $64 average order value alone cannot establish revenue variance or power these guardrails.

Sample size and duration: sums

Across the supplied 56 days:

Thus:

\[ p_{\text{checkout}}=132,196/2,083,390=6.345\% \]

\[ \text{sessions/day}=2,083,390/56=37,203 \]

Propose 80% power, two-sided 5% significance, 3% relative minimum detectable uplift, subject to commercial sign-off.

Illustrative checkout-session calculation only:

\[ \Delta=0.06345×0.03=0.0019035 \]

That is 0.190 percentage points.

\[ n_{\text{arm}}\approx \frac{2(1.96+0.84)^2p(1-p)}{\Delta^2} \approx257,200 \]

\[ N\approx514,400;\quad514,400/37,203=13.83\text{ days} \]

Two complete weeks would provide approximately:

\[ 2,083,390/8×2=520,848\text{ checkout sessions} \]

This does not prove the proposed experiment is powered. The intervention precedes checkout, and checkout sessions are neither eligible basket visitors nor independent customers. The 412,000 figure counts buyers only; 3.8 sessions/month describes buyers, not all eligible visitors. Dividing session traffic by 3.8 would be invalid.

Launch-blocking data request: Extract deduplicated eligible basket entrants, their 24-hour purchase rate and cumulative unique entrants over candidate windows. Recalculate:

\[ n{\text{arm}}= \frac{15.68pb(1-pb)}{(0.03pb)^2} \]

Choose a fixed enrolment duration covering at least two complete weeks and enough unique customers, then add 24 hours for primary-outcome maturation.

Calendar

There is no unconditionally clean two-week pre-promotion slot after Payments ends:

• Payments runs through 9 November.
• 10–15 November offers only six days before freeze.
• Even if running tests may continue during freeze, 10–19 November offers only ten days before promotions.

Conditional slot: 31 October–13 November, outcomes mature 14 November, readout 15 November. Search has ended. This requires Payments to hold its configuration through follow-up and agree independent, orthogonal assignment, balanced payment strata and interaction analysis. It also requires the recalculated customer sample to fit.

Otherwise, negotiate isolation earlier or defer until after promotions and freeze; their end dates are not supplied.

Decision rule

No efficacy peeking or early “winner” calls. At the fixed endpoint, ship only if the primary 95% confidence interval excludes zero positively, the estimated uplift meets the agreed 3% business threshold, and guardrail confidence bounds exclude their harm margins. Require valid instrumentation and no material platform/payment interaction harm.

Otherwise retain control: distinguish evidence of harm from an inconclusive result. Missing power, immature guardrails or calendar approval means no launch, not an assumed pass.

Experiment spec: delivery date and fee on the basket page

Hypothesis

Showing the delivery date and fee on the basket page, rather than at the final checkout step, raises the share of basket sessions that end in an order. We expect fewer last-step abandons because the fee is no longer a surprise. The counter-risk is more exits at the basket, so the primary metric must count both.

Randomisation

• Unit: logged-in user ID, not session or device. 31% of buyers use both app and web, and buyers average 3.8 sessions a month. Session-level assignment would show one person both experiences and make observations non-independent.
• Split: 50/50 by salted hash of user ID, the same arm on app and web. Users are assigned on first basket view.
• Independence: use a different salt from the Payments squad's "Pay later" test, so every arm is about 50% Pay-later (see Calendar).
• Checks: run an A/A check in the week before launch. Run a sample-ratio-mismatch (SRM) test throughout, and pause if p < 0.001.

Metrics

Primary: order conversion = basket-page sessions that end in an order ÷ basket-page sessions, with standard errors clustered by user.

Do not use orders ÷ checkout sessions. If the fee scares people off at the basket, they never become checkout sessions, the denominator shrinks, and the treatment looks better than it is.

Secondary: - Basket→checkout-start rate (expected to fall, and we need to see by how much). - Checkout-start→order rate (the squad's mechanism). - Revenue per basket session. - Orders per user over the window. - Platform split (exploratory only).

Guardrails: - Average order value (AOV, baseline $64): fail if the lower CI bound is below −2% (−$1.28). - Revenue per basket session: fail if significantly negative. - Fee-related support contacts per 1,000 orders. - Basket-page error rate and latency. - 14-day repeat rate among exposed users. The 30-day repeat rate (38%) can't be read in a 3-week test.

Baseline from the traffic file

• Weekly sessions are about 260k and daily volumes are stable, so I use p = 6.35%.
• Platform split: app 57–58% of sessions at about 7.1% conversion; web 42% at about 5.3%.
• Weekends run at about 65% of weekday volume, so the test must cover whole weeks.

Sample size

• MDE: +3% relative, which is +0.19pp (6.35% → 6.54%). That is roughly 490 orders a week, or about $31k a week at $64 AOV, which is worth shipping for.
• Settings: two-sided α = 0.05, power 80%, so (1.96 + 0.84)² = 7.84.
• Per arm, ignoring clustering: n = 2 × 7.84 × p(1−p) / δ²
• p(1−p) = 0.0635 × 0.9365 = 0.0595
• δ = 0.00191, so δ² = 3.63×10⁻⁶
• n = 0.932 / 3.63×10⁻⁶ ≈ 257k sessions
• Clustering: users contribute about 2.7 sessions in 3 weeks. I assume an intra-user correlation of 0.2, giving a design effect of 1 + 1.7 × 0.2 ≈ 1.3. That puts the requirement at 257k × 1.3 ≈ 334k sessions per arm.
• Duration: each arm gets about 130k sessions a week (260k ÷ 2), so 334k ÷ 130k = 2.6 weeks. Run 3 full weeks (21 days), which gives about 390k per arm.
• For comparison: a 2% MDE needs 578k × 1.3 = 751k per arm, or 5.8 weeks, which the calendar can't fit. A 5% MDE needs 92k × 1.3 = 120k per arm, under 1 week.
• Subgroups: web alone (42% of traffic) can only detect about 4.6% relative (3% × √(1/0.42)). Platform cuts are exploratory.

Data issues to resolve before launch

1. Buyer count doesn't match the traffic file. The data team's 412k buyers in 30 days is inconsistent with about 70k orders a month in the file (16.4k a week × 30/7). Their 3.8 sessions per buyer implies about 1.57M sessions, against about 1.11M in the file. I sized from the traffic file. The data team needs to reconcile the two, since the user-level assumptions (the design effect) depend on it.
2. Check the definition of `checkout_sessions`. I assumed it equals basket-page sessions. If it only counts sessions that reached checkout, the true basket-level baseline is lower and the test is longer. For example, at 4% conversion, n scales by (0.96/0.04) ÷ (0.9365/0.0635) ≈ 1.6×, which is about 4.5 weeks.
3. Re-run the sums on the real basket-session baseline before launch, and fix the duration then (3 or 4 weeks).

Calendar

• Black Friday: promotions start 20 Nov and code freeze starts 16 Nov. Promo traffic isn't representative, and anything not shipped by 15 Nov waits until after the freeze.
• Waiting for Pay later to end (9 Nov) doesn't work. That leaves 10–15 Nov, which is 6 days and not enough.
• Pay later (until 9 Nov) sits on the checkout page and affects the same outcome, so it can't be ignored. I'll run the test concurrently with an independent split. Each arm then gets about the same Pay-later mix, so the contrast stays unbiased. I'll pre-register a treatment × Pay-later interaction check, and report the effect by Pay-later arm. The Payments squad must freeze their allocation for the duration.
• Search ranking (until 30 Oct) only touches search results pages. It's balanced by randomisation and needs no action, but I'll note it.
• Schedule:
• Build, QA and A/A: 5–11 Oct.
• Run 12 Oct – 1 Nov (3 weeks).
• Readout 2–4 Nov, decision before 9 Nov.
• Ship ramp before the 16 Nov freeze.
• Buffer: a 4-week run (to 8 Nov) still ends before Pay later does.

Decision rule

This is fixed in advance, with one analysis at the end of the run and no peeking.

• Ship if the primary lift is positive with the 95% CI excluding 0, and no guardrail fails, and revenue per basket session isn't significantly negative.
• Do not ship if the CI includes 0. We'd conclude "no detectable effect at +3%". Any wish to ship for fee-transparency reasons is a separate product decision, not an experiment win.
• Kill if the primary is significantly negative, or if AOV or revenue per basket session breaches its guardrail. Basket exits are the main thing we expect to see here.
• Extend: only if an outage or traffic shortfall leaves fewer than 334k sessions per arm, up to 28 days. Never extend because the result is "almost significant".
• Interaction: if the effect differs materially by Pay-later arm, ship only to the configuration that will be live after 9 Nov, and confirm with the Payments squad which one that is.