Experiment specification

Experiment Spec: Upfront Delivery Fee & Date on Basket Page

1. Hypothesis

Showing the next-day delivery date and delivery fee on the basket page (rather than waiting until the final checkout step) will reduce sticker-shock abandonment at the end of the funnel and increase overall order conversion by at least 2% relative, without degrading average order value (AOV) or causing excessive early-funnel drop-off.

• Control: Delivery date and fee are revealed on the final checkout step.
• Treatment: Delivery date and fee are displayed directly on the basket page before the user clicks "Proceed to Checkout".

---

2. Randomisation & Unit of Analysis

• Randomisation Unit: `user_id` (logged-in account).
• Why: Data shows 31% of buyers use both the app and web in a 30-day period, and the average buyer starts 3.8 checkout sessions/month. Device- or session-level randomisation would expose the same user to both variants, causing UX confusion and contamination. Unauthenticated users will be randomised by a persistent device cookie and stitched upon login at the start of checkout.
• Trigger Point: First view of the Basket Page during the experiment window.
• Split: 50% Control / 50% Treatment, stratified by platform (App vs. Web) since baseline conversion differs significantly between the two.

---

3. Metrics

Note on Denominator: Because Treatment alters the basket page—before a `checkoutsession` starts—using `checkoutsessions` as the primary denominator would introduce post-treatment selection bias. The primary denominator must be users reaching the basket.

---

4. Traffic, Sample Size & Duration (Show Your Sums)

A. Baseline Traffic from CSV (56 days: 10 Aug – 4 Oct 2026)

• App Totals: 1,196,659 checkout sessions; 85,127 orders $\rightarrow$ 7.11% session CVR (21,369 sessions/day).
• Web Totals: 875,705 checkout sessions; 46,966 orders $\rightarrow$ 5.36% session CVR (15,638 sessions/day).
• Combined Totals: 2,072,364 checkout sessions; 132,093 orders.
• Daily Averages: 37,006 checkout sessions/day and 2,359 orders/day.
• Weekday vs. Weekend split: Weekdays average ~40,800 sessions/day; weekends average ~28,100 sessions/day. (Must run in full 7-day increments).
• Session-to-Order Baseline CVR: $132,093 / 2,072,364 = \mathbf{6.374\%}$.

B. Converting Sessions to Unique Users

* Each buyer averages 3.8 checkout sessions per 30 days (~0.95 sessions/week). * In a 14-day window: $37,006 \text{ sessions/day} \times 14 = 518,084 \text{ checkout sessions}$. * Accounting for repeat sessions (~1.6 sessions per user over two weeks), a 14-day window yields approximately 323,800 unique users entering checkout: $$518,084 / 1.6 \approx 323,800 \text{ unique users (161,900 per arm)}$$ * User-level Baseline CVR ($p1$) over 14 days: $$\text{Total 14-day orders} = 2,359 \times 14 = 33,026$$ $$p1 \approx 33,026 / 323,800 \approx \mathbf{10.20\%}$$ (Even if analysed at the session level where $p_1 = 6.374\%$, we calculate both below to ensure statistical power).

C. Sample Size Calculation

Using standard two-tailed parameters: $\alpha = 0.05$ ($Z{\alpha/2} = 1.96$), Power $1 - \beta = 0.80$ ($Z\beta = 0.84$), so $(Z{\alpha/2} + Z\beta)^2 = (1.96 + 0.84)^2 = \mathbf{7.84}$. Target Minimum Detectable Effect (MDE): 2.5% relative lift.

1. At the Session Level ($p_1 = 0.06374$, $\Delta = 0.06374 \times 0.025 = 0.00159$): $$n = \frac{2 \times 7.84 \times 0.06374 \times (1 - 0.06374)}{(0.00159)^2} = \frac{0.9358}{0.00000253} \approx \mathbf{370,000 \text{ sessions/arm}}$$ * With cluster inflation (design effect $\approx 1.3$ for 1.6 sessions/user): $370,000 \times 1.3 = 481,000 \text{ sessions/arm}$ ($962,000$ total). * Days required: $962,000 / 37,006 \text{ sessions/day} = \mathbf{26.0 \text{ days}}$.

2. For a 14-Day Run ($518,084$ total sessions $\rightarrow 259,042$ sessions/arm, or $161,900$ users/arm): * At user level ($p1 = 0.102$, $n = 161,900$/arm): $$\text{MDE}{\text{abs}} = \sqrt{\frac{2 \times 7.84 \times 0.102 \times 0.898}{161,900}} = \sqrt{\frac{1.436}{161,900}} = 0.00298 \text{ (or } \mathbf{2.92\% \text{ relative lift}}\text{)}$$

Duration Decision: Run for 14 full days (2 weeks) to capture two full weekly cycles and detect a $\ge 2.9\%$ relative lift in user conversion ($\sim 68$ extra orders/day, or ~$4,350/day incremental revenue).

---

5. Calendar & Scheduling

• Constraints:
• Search squad's ranking test ends 30 Oct (low interaction risk, upstream of basket).
• Payments squad's "Pay later" test runs on the checkout page at 50/50 until 9 Nov (high interaction risk: changes payment friction on the exact step our test targets).
• Code freeze starts 16 Nov; Black Friday promos start 20 Nov.
• Schedule:
• Waiting until 9 Nov leaves only 7 days before the 16 Nov code freeze—insufficient for a 14-day run.
• Therefore, we will run orthogonal (cross-factored) to the Payments "Pay later" test for 14 days: Monday 26 October – Sunday 8 November 2026.
• Implementation: Use an independent hash seed so our 50/50 split is evenly distributed across Control and Treatment of the "Pay later" experiment (25% in each of the 4 cells). This leaves a full week (9–15 Nov) to analyse results and ship or roll back before the 16 Nov code freeze.

---

6. Decision Rule

1. Ship: Primary metric (User CVR from Basket) is statistically significantly positive ($p < 0.05$, two-tailed), AOV guardrail does not drop by more than 1% ($p < 0.05$), and interaction check with the "Pay later" test shows no severe negative synergy.
2. Kill / Roll back: Primary metric is significantly negative ($p < 0.05$) due to early basket abandonment, OR Revenue per User drops significantly.
3. Inconclusive (Flat): If User CVR is flat ($p \ge 0.05$) and guardrails are unharmed, ship Treatment anyway—upfront fee transparency is a better customer experience and aligns with consumer protection best practices.

Experiment: delivery transparency on the basket

Status: Proposed; launch requires a basket-level sizing check and agreement with Payments. The supplied traffic supports a planning estimate, not a defensible final user-level sample size.

Hypothesis and treatment

Showing the next-day delivery date and fee on the basket increases completed orders by removing a late surprise. It may instead move abandonment earlier, so success must be measured from basket exposure, not checkout entry.

• Control: Date and fee first appear at the final checkout step.
• Treatment: Show the same date and fee on the basket, before the checkout button. Retain the final-step confirmation.
• No changes to fees, delivery availability, payment options or other copy. Recalculate both displays consistently if the basket or address changes.

Eligibility and randomisation

Randomise eligible signed-in customers 50:50 at their first basket view, before rendering either experience. Eligibility requires an address sufficient to quote delivery accurately.

Use a persistent customer-ID assignment shared across app and web. Session/device randomisation would contaminate exposure: 31% of buyers use both platforms. Anonymous customers are excluded unless we can reliably preserve and reconcile assignment.

Use an independent experiment hash, balanced within existing Payments assignment and first-exposure platform. Persist assignment throughout the experiment.

Analyse intention-to-treat: include everyone assigned, including customers who never enter checkout. Each customer contributes once to the primary outcome.

Metrics

Primary: Percentage of assigned customers placing at least one order within 24 hours of first eligible basket exposure, on either platform.

Secondary: - Checkout entry within 24 hours. - Basket-to-order conversion and final-step abandonment, with explicit funnel denominators. - Orders and revenue per assigned customer within 24 hours. - Order value, descriptive only: treatment can change who buys. - Repeat ordering within 30 days of assignment, reported after maturation. The supplied 38% is buyer-conditional, not this metric’s baseline.

Guardrails, precommitted proposed tolerances: - Revenue per assigned customer: no more than 1% relative deterioration. - Cancellation/refund incidence per assigned customer: no more than 0.2 percentage-point deterioration. - Checkout technical-error incidence per assigned customer: no more than 0.1 percentage-point deterioration. - Basket p95 latency: no more than 100 ms deterioration. - Any incorrect fee/date presentation is a stop-and-fix issue.

Instrument assignment, basket exposure, checkout steps, displayed quote and orders. Check sample-ratio mismatch, cross-platform consistency and event completeness before interpreting results.

Sample size and duration: sums

Across the supplied 56 days:

Average checkout traffic:

\[ 2,083,390/56=37,203\text{ sessions/day}. \]

Planning assumptions: two-sided α=5%, 80% power, 5% relative lift as the minimum detectable effect—not an observed or commercially validated threshold.

Using checkout conversion only as a provisional proxy:

\[ p=132,196/2,083,390=0.063452,\quad \Delta=0.05p=0.003173. \]

Approximate equal-arm sample:

\[ n=\frac{2(1.96+0.84)^2p(1-p)}{\Delta^2} \approx92,600\text{ per arm}. \]

Thus approximately 185,200 independent observations. Naively:

\[ 185,200/37,203=4.98\text{ days}. \]

Five days is not a valid duration. These are checkout sessions, not unique eligible basket visitors; checkout entry is affected by treatment. The 3.8 sessions/month figure applies only to buyers, so dividing total sessions by 3.8 would not repair the denominator. Nor do 412,000 monthly buyers establish eligible traffic.

Pre-launch requirement: Measure deduplicated eligible basket visitors, their 24-hour conversion, and expected new enrolments over the planned window. Recalculate using that conversion rate; separately check guardrail power. Lock the largest required sample before launch.

Proposed maximum enrolment is 28 days, representing:

\[ 28\times37,203\approx1,041,695 \]

checkout sessions as a volume benchmark—not a count of independent participants. Launch only if forecast eligible customers support the locked sample within that window.

Calendar

Propose 12 October–8 November 2026, four complete weeks, with primary outcomes matured by 9 November.

This overlaps Pay Later. Obtain Payments agreement for independent allocation, retain its assignment in analysis, and estimate the treatment-by-payment interaction. Otherwise, do not use this window.

Search ends 30 October; log its assignment and account for calendar-date changes.

The uncontaminated pre-promotion window is only 10–19 November: ten days. The 16 November freeze requires any running experiment and kill switch to be preapproved; it is not automatically a shutdown. Use that alternative only if revised sizing supports it. Otherwise postpone until a confirmed post-promotion window.

Decision rule

At the locked endpoint, ship if the primary lift’s two-sided 95% confidence interval excludes zero positively and all guardrails pass their precommitted non-inferiority checks. Review platform and Payments interactions before broad rollout.

Reject on clear conversion harm or a guardrail breach. Otherwise call it inconclusive—not “no effect.” No efficacy peeking or opportunistic extension; safety monitoring remains continuous.

Experiment spec: earlier delivery-date and fee disclosure

Status: Conditional launch approval. The supplied traffic supports a planning calculation, but not a defensible final sample size for the correct primary metric.

Hypothesis and experience

Showing the next-day delivery date and actual delivery fee on the basket page increases completed purchases by reducing late surprises. Earlier abandonment may increase; the test must measure the net effect from basket entry, not just conversion among people who proceed to checkout.

• Control: Current disclosure at the final checkout step.
• Treatment: Disclose on the basket page; retain an accurate fee/date in the final order summary. No changes to prices, delivery eligibility or payment options.
• Eligibility: identifiable, signed-in customers opening a non-empty basket for which we can calculate the date and fee. Do not introduce a login requirement. Results initially apply only to this population.

Randomisation and analysis

Assign eligible customers 50:50 by persistent account-ID hash, immediately before their first eligible basket renders. Keep assignment sticky across sessions, app and web.

Cross-platform consistency matters: \(412,000 × 31\% = 127,720\) recent buyers used both platforms. Device- or session-level assignment would expose many people to both variants.

Analyse intention-to-treat: everyone assigned remains in their arm, including failed renders and immediate exits. Each customer contributes one primary observation. Pre-specify adjustment for entry platform and enrolment date; report app/web effects as secondary, not independently powered launch decisions.

Validate allocation, identity stitching, exposure events and order attribution before starting.

Metrics

Primary: Percentage of assigned customers placing a paid order within 24 hours of first eligible basket entry, on either platform. This captures both earlier exits and downstream gains without a treatment-selected checkout denominator.

Secondary: - Basket-to-checkout progression and final-step abandonment, as funnel diagnostics. - Orders and net revenue per assigned customer within 24 hours. - Order value among purchasers, labelled a conditional diagnostic. - Thirty-day repeat purchase; the supplied 38% is context, not necessarily this cohort’s baseline. This needs a separate mature readout.

Guardrails — proposed tolerances, requiring sign-off: - Net revenue/customer: no worse than −1% relative. - Payment-error and cancellation/refund incidence per assigned customer: no worse than +0.1 percentage points, using fixed, pre-registered observation windows. - Zero tolerance for confirmed incorrect fee/date disclosure or unintended charges. - Monitor basket latency, crashes and sample-ratio mismatch; material regressions trigger investigation or rollback.

The $64 average order value alone cannot establish revenue variance or power these guardrails.

Sample size and duration: sums

Across the supplied 56 days:

Thus:

\[ p_{\text{checkout}}=132,196/2,083,390=6.345\% \]

\[ \text{sessions/day}=2,083,390/56=37,203 \]

Propose 80% power, two-sided 5% significance, 3% relative minimum detectable uplift, subject to commercial sign-off.

Illustrative checkout-session calculation only:

\[ \Delta=0.06345×0.03=0.0019035 \]

That is 0.190 percentage points.

\[ n_{\text{arm}}\approx \frac{2(1.96+0.84)^2p(1-p)}{\Delta^2} \approx257,200 \]

\[ N\approx514,400;\quad514,400/37,203=13.83\text{ days} \]

Two complete weeks would provide approximately:

\[ 2,083,390/8×2=520,848\text{ checkout sessions} \]

This does not prove the proposed experiment is powered. The intervention precedes checkout, and checkout sessions are neither eligible basket visitors nor independent customers. The 412,000 figure counts buyers only; 3.8 sessions/month describes buyers, not all eligible visitors. Dividing session traffic by 3.8 would be invalid.

Launch-blocking data request: Extract deduplicated eligible basket entrants, their 24-hour purchase rate and cumulative unique entrants over candidate windows. Recalculate:

\[ n{\text{arm}}= \frac{15.68pb(1-pb)}{(0.03pb)^2} \]

Choose a fixed enrolment duration covering at least two complete weeks and enough unique customers, then add 24 hours for primary-outcome maturation.

Calendar

There is no unconditionally clean two-week pre-promotion slot after Payments ends:

• Payments runs through 9 November.
• 10–15 November offers only six days before freeze.
• Even if running tests may continue during freeze, 10–19 November offers only ten days before promotions.

Conditional slot: 31 October–13 November, outcomes mature 14 November, readout 15 November. Search has ended. This requires Payments to hold its configuration through follow-up and agree independent, orthogonal assignment, balanced payment strata and interaction analysis. It also requires the recalculated customer sample to fit.

Otherwise, negotiate isolation earlier or defer until after promotions and freeze; their end dates are not supplied.

Decision rule

No efficacy peeking or early “winner” calls. At the fixed endpoint, ship only if the primary 95% confidence interval excludes zero positively, the estimated uplift meets the agreed 3% business threshold, and guardrail confidence bounds exclude their harm margins. Require valid instrumentation and no material platform/payment interaction harm.

Otherwise retain control: distinguish evidence of harm from an inconclusive result. Missing power, immature guardrails or calendar approval means no launch, not an assumed pass.