Experiment specification

Experiment Spec: Upfront Delivery Fee & Date on Basket

Squad: Checkout Author: PM, Checkout Date: October 5, 2026

---

1. Hypothesis & Context

• Hypothesis: Displaying the delivery date and delivery fee upfront on the basket page (instead of at the final checkout step) will reduce unexpected cost friction at checkout, decreasing last-step abandonment and increasing overall basket-to-order conversion.
• Risk: Showing fees earlier may induce immediate drop-off on the basket page, reducing the number of checkout sessions started.
• Core Question: Does the increase in final checkout completion compensate for any drop in basket-to-checkout progression?

---

2. Randomisation & Allocation

• Unit of Randomisation: User ID (with persistent Device/Visitor ID fallback for non-logged-in users).
• Why: Data shows 31% of monthly buyers shop on both the app and the website, and users average 3.8 checkout sessions/month. Randomising by session or device would cause severe cross-platform contamination and inconsistent user experiences.
• Split: 50% Control, 50% Treatment, stratified by platform (iOS, Android, Web).
• Variants:
• Control (A): Current flow. Delivery date and fee revealed on final checkout step.
• Treatment (B): Delivery date and calculated fee dynamically displayed on the basket page prior to entering checkout.

---

3. Metrics Framework

* Primary Metric: * Basket-to-Order Conversion Rate (Orders / Unique Basket Sessions): Note: We must not use Checkout-to-Order CVR alone as the primary metric, because filtering out users on the basket page could inflate downstream checkout conversion while reducing total orders. * Secondary Metrics (Funnel Diagnostics): * Basket-to-Checkout Initiation Rate: Measures upfront drop-off risk (Basket views $\to$ Checkout step 1). * Final-Step Checkout Abandonment Rate: Anticipated to drop as fee surprises are eliminated. * Average Order Value (AOV): Baseline is $64; monitors whether upfront fees alter basket composition. * Guardrail Metrics: * Total Orders & Revenue per User: Must not see a statistically significant decrease. * Basket Page P95 Latency: Fee lookup API must not add $>150\text{ ms}$ to basket load time. * CS Contact Rate: Delivery fee disputes/inquiries per 1,000 orders.

---

4. Traffic Sums, Sample Size & Duration

#### A. Baseline Traffic & Conversion (from 8-week data: Aug 10 – Oct 4, 2026) * Total 8-week volume: * App checkout sessions: $1,202,306$ | App orders: $85,466$ ($7.11\%$ CVR) * Web checkout sessions: $874,188$ | Web orders: $46,756$ ($5.35\%$ CVR) * Combined totals: $2,076,494$ checkout sessions | $132,222$ orders * Weekly averages: * Sessions: $\approx 259,560$ sessions/week ($\approx 37,080$ sessions/day) * Orders: $\approx 16,528$ orders/week ($\approx 2,360$ orders/day) * Baseline Checkout Conversion Rate ($p$): $$p = \frac{132,222}{2,076,494} = 6.37\%$$

#### B. Sample Size Calculation Using standard two-tailed hypothesis testing parameters: * Significance level ($\alpha$): $0.05$ ($Z{\alpha/2} = 1.96$) * Statistical power ($1 - \beta$): $80\%$ ($Z{\beta} = 0.84$) * Minimum Detectable Effect (MDE): $2.5\%$ relative lift ($\Delta = 6.37\% \times 0.025 = 0.159\%$ absolute $\to p_2 = 6.53\%$)

$$\text{Sample size per variant } n = \frac{(Z{\alpha/2} + Z\beta)^2 \cdot 2 \cdot p(1 - p)}{\Delta^2}$$ $$n = \frac{(1.96 + 0.84)^2 \cdot 2 \cdot (0.0637 \cdot 0.9363)}{(0.00159)^2} = \frac{7.84 \cdot 2 \cdot 0.05964}{0.000002528} \approx 369,980 \text{ sessions/variant}$$

• Total required sessions (Control + Treatment): $\approx 740,000$ checkout/basket sessions.

#### C. Duration * Total traffic per week $\approx 259,560$ sessions. * Required weeks: $740,000 / 259,560 \approx 2.85$ weeks. * To eliminate day-of-week seasonality (e.g., weekend drop to $\approx 28\text{k}$ sessions/day vs. weekday $\approx 41\text{k}$), we set the duration to exactly 3 full calendar weeks (21 days), providing $\approx 778,000$ sessions and achieving an MDE of $\approx 2.4\%$.

---

5. Calendar & Schedule

``` [Oct 5] Spec Approval & QA [Oct 12] Launch Experiment (Day 1) ├── Oct 12 – Nov 1: 3-Week Test Window └── Concurrently: Orthogonal to Payments "Pay later" [Nov 1] Conclude Test & Analysis [Nov 2 – Nov 6] Squad Review & Rollout Decision [Nov 9 – Nov 13] 100% Rollout (if successful) [Nov 16] CODE FREEZE for Black Friday ```

• Constraints & Conflicts:
• Code Freeze: Begins 16 November.
• Payments Squad ("Pay later" on checkout): Live for 50% of users until 9 November. We cannot wait until Nov 9, as that leaves only 7 days before code freeze.
• Search Ranking Test: Ends 30 October on search result pages only (no direct conflict).
• Interference Mitigation:
• Randomise our test on an orthogonal hash layer to Payments. Because assignment is independent, exposure to "Pay later" will balance evenly (50/50) across both our Control and Treatment groups.
• Check for an interaction effect via a $2 \times 2$ factorial ANOVA post-test.

---

6. Decision Rules

Experiment: delivery transparency on the basket

Status: Proposed; launch requires a basket-level sizing check and agreement with Payments. The supplied traffic supports a planning estimate, not a defensible final user-level sample size.

Hypothesis and treatment

Showing the next-day delivery date and fee on the basket increases completed orders by removing a late surprise. It may instead move abandonment earlier, so success must be measured from basket exposure, not checkout entry.

• Control: Date and fee first appear at the final checkout step.
• Treatment: Show the same date and fee on the basket, before the checkout button. Retain the final-step confirmation.
• No changes to fees, delivery availability, payment options or other copy. Recalculate both displays consistently if the basket or address changes.

Eligibility and randomisation

Randomise eligible signed-in customers 50:50 at their first basket view, before rendering either experience. Eligibility requires an address sufficient to quote delivery accurately.

Use a persistent customer-ID assignment shared across app and web. Session/device randomisation would contaminate exposure: 31% of buyers use both platforms. Anonymous customers are excluded unless we can reliably preserve and reconcile assignment.

Use an independent experiment hash, balanced within existing Payments assignment and first-exposure platform. Persist assignment throughout the experiment.

Analyse intention-to-treat: include everyone assigned, including customers who never enter checkout. Each customer contributes once to the primary outcome.

Metrics

Primary: Percentage of assigned customers placing at least one order within 24 hours of first eligible basket exposure, on either platform.

Secondary: - Checkout entry within 24 hours. - Basket-to-order conversion and final-step abandonment, with explicit funnel denominators. - Orders and revenue per assigned customer within 24 hours. - Order value, descriptive only: treatment can change who buys. - Repeat ordering within 30 days of assignment, reported after maturation. The supplied 38% is buyer-conditional, not this metric’s baseline.

Guardrails, precommitted proposed tolerances: - Revenue per assigned customer: no more than 1% relative deterioration. - Cancellation/refund incidence per assigned customer: no more than 0.2 percentage-point deterioration. - Checkout technical-error incidence per assigned customer: no more than 0.1 percentage-point deterioration. - Basket p95 latency: no more than 100 ms deterioration. - Any incorrect fee/date presentation is a stop-and-fix issue.

Instrument assignment, basket exposure, checkout steps, displayed quote and orders. Check sample-ratio mismatch, cross-platform consistency and event completeness before interpreting results.

Sample size and duration: sums

Across the supplied 56 days:

Average checkout traffic:

\[ 2,083,390/56=37,203\text{ sessions/day}. \]

Planning assumptions: two-sided α=5%, 80% power, 5% relative lift as the minimum detectable effect—not an observed or commercially validated threshold.

Using checkout conversion only as a provisional proxy:

\[ p=132,196/2,083,390=0.063452,\quad \Delta=0.05p=0.003173. \]

Approximate equal-arm sample:

\[ n=\frac{2(1.96+0.84)^2p(1-p)}{\Delta^2} \approx92,600\text{ per arm}. \]

Thus approximately 185,200 independent observations. Naively:

\[ 185,200/37,203=4.98\text{ days}. \]

Five days is not a valid duration. These are checkout sessions, not unique eligible basket visitors; checkout entry is affected by treatment. The 3.8 sessions/month figure applies only to buyers, so dividing total sessions by 3.8 would not repair the denominator. Nor do 412,000 monthly buyers establish eligible traffic.

Pre-launch requirement: Measure deduplicated eligible basket visitors, their 24-hour conversion, and expected new enrolments over the planned window. Recalculate using that conversion rate; separately check guardrail power. Lock the largest required sample before launch.

Proposed maximum enrolment is 28 days, representing:

\[ 28\times37,203\approx1,041,695 \]

checkout sessions as a volume benchmark—not a count of independent participants. Launch only if forecast eligible customers support the locked sample within that window.

Calendar

Propose 12 October–8 November 2026, four complete weeks, with primary outcomes matured by 9 November.

This overlaps Pay Later. Obtain Payments agreement for independent allocation, retain its assignment in analysis, and estimate the treatment-by-payment interaction. Otherwise, do not use this window.

Search ends 30 October; log its assignment and account for calendar-date changes.

The uncontaminated pre-promotion window is only 10–19 November: ten days. The 16 November freeze requires any running experiment and kill switch to be preapproved; it is not automatically a shutdown. Use that alternative only if revised sizing supports it. Otherwise postpone until a confirmed post-promotion window.

Decision rule

At the locked endpoint, ship if the primary lift’s two-sided 95% confidence interval excludes zero positively and all guardrails pass their precommitted non-inferiority checks. Review platform and Payments interactions before broad rollout.

Reject on clear conversion harm or a guardrail breach. Otherwise call it inconclusive—not “no effect.” No efficacy peeking or opportunistic extension; safety monitoring remains continuous.

Experiment Specification: Early Delivery Fee & Date Preview

Author: PM, Checkout Squad Date: October 2026

---

1. Hypothesis

Showing the delivery date and delivery fee on the basket page (pre-checkout) will reduce final-step checkout abandonment by setting upfront expectations. * Core Hypothesis: The reduction in late-stage drop-off outweighs any early-stage basket abandonment caused by friction, resulting in a net positive impact on overall checkout conversion.

2. Randomisation Strategy & Unit

• Unit of Randomization: User ID (sticky across app and web platforms, given 31% multi-platform usage).
• Allocation: 50% Control (status quo: fee shown on final checkout step), 50% Treatment (fee and date visible on the basket page).

3. Metrics

• Primary Metric: Checkout Conversion Rate (Completed Orders / Unique Checkout Sessions).
• Secondary Metrics:
• Basket-to-Checkout Drop-off Rate (users leaving before entering checkout).
• Average Order Value (AOV).
• Guardrail Metrics:
• Total Orders / Gross Merchandise Value (GMV).
• 30-day Repeat Purchase Rate.

---

4. Traffic Analysis & Sample Size Sizing (The Sums)

Real Traffic Baseline (from past 8 weeks data): * Summing the 8-week dataset (56 days): * Total App Checkout Sessions = 1,304,784 (~23,300/day) * Total Web Checkout Sessions = 959,183 (~17,128/day) * Total Combined Sessions (8 weeks) = 2,263,967 * Daily Average Traffic = 40,428 checkout sessions/day * Total Orders = 155,752 * Baseline Conversion Rate ($p$) = $155,752 / 2,263,967 \approx 6.88\%$

Sample Size Calculations: To detect a relative change of 2% in conversion rate (e.g., from 6.88% to 7.02%) with $\alpha = 0.05$ (two-tailed) and power $(1-\beta) = 0.80$: * Using standard sample size formulas for proportions, we require approximately 375,000 users per variant (~750,000 total sessions). * At our baseline traffic of ~40,428 sessions/day (split 50/50 = ~20,214 sessions per variant daily): * Required Duration = $750,000 / 40,428 \approx 18.5$ days. * We will round up to 3 full weeks (21 days) to cleanly account for day-of-week seasonality (grocery ordering behaves very differently on Saturdays vs. Mondays).

---

5. Calendar & Timing

• Overlaps & Conflicts:
• Search squad: Runs exclusively on search results pages. Zero overlap with checkout/basket flows. Safe to run concurrently.
• Payments squad ("Pay later"): Runs on the checkout page (affecting 50% of users) until 9 November. Because the checkout page is downstream of our intervention (basket page), running both creates a multi-variate interaction risk if not accounted for.
• Holiday/Promo Constraints:
• Black Friday promotions start November 20.
• Code freeze begins November 16.
• Proposed Window:
• Start: Monday, October 12, 2026
• End: Monday, November 2, 2026 (Exactly 21 days).
• Note: This completely clears the Payments squad end-date (Nov 9) by launching earlier, and sits safely clear of the November 16 code freeze.

---

6. Decision Rule

We will evaluate the experiment after exactly 21 days of data collection.

1. Ship (Rollout 100%): If the Primary Metric (Checkout Conversion Rate) shows a statistically significant positive lift ($p < 0.05$), and no guardrail metrics (GMV, 30-day repeat rate) drop significantly.
2. Iterate / Inconclusive: If the primary metric shows no statistically significant change ($p \ge 0.05$), but secondary metrics show positive intent shifts, review basket abandonment drop-offs and redesign the UI placement.
3. Rollback: If the Primary Metric drops significantly ($p < 0.05$) due to early basket abandonment outweighing final-step savings, or if guardrail metrics degrade, immediately halt the experiment and revert to control.