Experiment specification

Experiment spec: delivery fee in basket

Owner: PM, Checkout | Readers: Ravi (build), Ines (analysis), Chloe (approval)

Hypothesis

Showing the delivery fee in the basket reduces surprise at checkout (12% of last quarter's support contacts) without reducing orders. Chloe's concern is that an early fee deters people before they start checkout, so the test measures that directly.

Changes from the team draft

Design

• Control: fee first shown at final checkout step (today).
• Variant: basket shows the delivery line: £3.99, or "Free" at £60+. Nothing else changes. The basket and checkout amounts must always match.
• Unit: customer ID, with a 50/50 split. Assignment is sticky for the whole test. Logged-out users are bucketed by a persistent device ID (Ravi to confirm how many there are).
• Enrolment: at first basket view during the test. Log an exposure event at that moment.

Metrics

Primary: share of enrolled customers who place at least one order within 7 days of first basket view. Baseline is about 8%.

Guardrails: - Checkout start rate (baseline about 12.9%, i.e. 8% ÷ 62%). - Revenue per enrolled customer (baseline about £3.76, i.e. 8% × £47). - Delivery-fee support contacts per 1,000 customers. This is the benefit metric, but volumes are low, so read it directionally.

Diagnostics (not decision metrics): - Checkout completion (orders ÷ checkout starts). Read it only alongside the primary metric, since it's conditional on starts. - New vs returning customers, and baskets under vs over £60. Both are exploratory.

Sample size and duration

• Smallest effect worth detecting: 0.5 percentage points on orders (8.0% → 8.5%), per Chloe.
• Requirement: 80% power, 5% two-sided significance. This needs about 46,000 customers per arm, roughly 92,000 in total.
• Time to reach it: Week 1 brings about 40,000 unique customers. After that, only about 20,000 new customers enrol per week, because returning customers are already assigned. That reaches about 100,000 by the end of week 4.
• Duration: enrol for 4 full weeks, plus 7 days of follow-up for the last cohort. Readout is at about 5 weeks, with no early stopping. Ines can monitor data quality but not results.
• Limit: the test can't reliably detect effects smaller than 0.5pp. A "no significant difference" result means "no effect larger than about 0.5pp", not "no effect".

Data quality checks (before readout)

• Sample ratio check: arms should be within normal variation of 50/50.
• Exposure is logged for every enrolled customer.
• Fee shown in basket equals fee charged, in all cases.

Decision rule (Chloe to confirm)

• Ship if the 95% confidence interval for the change in orders rules out a drop of 0.5pp or more, no guardrail is materially worse, and delivery-fee contacts do not rise.
• Don't ship if orders fall by 0.5pp or more, or revenue per customer drops meaningfully.
• Otherwise, treat as inconclusive and discuss. Do not extend the test to "find" significance.

Next

If the variant ships or is neutral, run a follow-up test of the free-delivery-over-£60 banner, where the fee is already visible. Its effect is likely concentrated in baskets just under £60.

Open items

• Ravi: logged-out identity handling; a flag to switch the variant off quickly.
• Chloe: confirm the decision rule and that the 0.5pp threshold means absolute percentage points on orders per basket viewer.
• Ines: confirm the 7-day order attribution and the support-contact tagging for fee-related contacts.

Experiment Spec: Delivery Fee Transparency in Basket

To: Chloe (Approval), Ravi (Build), Ines (Analysis) From: Checkout PM

---

1. Objective & Hypothesis

• Context: 12% of support contacts complain about surprise delivery fees at final checkout.
• Hypothesis: Displaying the delivery fee (£3.99, or "Free" if >£60) upfront in the basket will set accurate expectations, reduce support contacts, and build customer trust without reducing net orders.
• Risk (Chloe’s concern): Upfront cost shock could cause drop-off before checkout begins.

---

2. Corrections to the Draft Plan

1. Primary Metric Changed: Draft used Checkout Completion Rate (Orders ÷ Starts). Correction: Use Basket-to-Order Conversion Rate (Orders ÷ Basket Viewers). Measuring only checkout starts introduces severe selection bias—if fee visibility deters low-intent users at the basket, checkout completion would artificially increase while overall orders plummet. 2. Unit of Randomization Changed: Draft used session-level. Correction: Randomize by User ID (falling back to persistent anonymous visitor ID). 50% of weekly viewers are repeat shoppers; session-level randomization would expose returning users to conflicting experiences. 3. No Peeking / Fixed Duration: Draft suggested stopping once "significant." Correction: Continuous monitoring with early stopping inflates false-positive rates to >30%. We will fix sample size upfront and run for complete weekly cycles. 4. Scope Control (Chloe’s Banner): Cut the promotional free-delivery banner from this test. Adding a promotional banner introduces a confounding variable, making it impossible to isolate whether fee transparency helped or hurt. We will test the banner in a follow-up experiment.

---

3. Variants (For Ravi)

• Allocation: 50/50 split at User/Persistent Device ID level. Triggered on basket view.
• Control (A): Current basket. No delivery fee shown (fee appears on final checkout step as today).
• Variant (B): Add a delivery fee line item to the Basket Order Summary:
• Basket < £60: `Delivery: £3.99`
• Basket ≥ £60: `Delivery: FREE`
• Include delivery fee in the displayed Basket Total.
• Note: Do not add promotional progress bars or marketing banners.

---

4. Metrics Framework (For Ines)

• Primary Metric: Basket-to-Order Conversion Rate (Total Unique Ordering Users ÷ Total Unique Basket Viewers).
• Guardrail Metrics:
• Average Order Value (AOV baseline: £47).
• Basket-to-Checkout Start Rate (tracks where abandonment occurs).
• Total Net Revenue (Orders × AOV).
• Secondary / Diagnostic Metrics:
• Delivery-fee-related customer support ticket rate per 1,000 orders.
• Checkout Completion Rate (Checkout Starts → Orders).

---

5. Sample Size & Run Time (For Ines & Chloe)

• Baseline Conversion: 8.0% of basket viewers place an order.
• Minimum Detectable Effect (MDE): Absolute ±0.5 percentage points (7.5% or 8.5%, a 6.25% relative change), per Chloe's threshold.
• Statistical Parameters: $\alpha = 0.05$ (two-tailed), Power ($1 - \beta$) = 80%.
• Required Sample: ~47,000 unique users per variant (~94,000 total unique basket viewers).
• Duration: 3 full weeks (21 days).
• With 40,000 viewers/week (~20k new, ~20k repeat), 3 weeks yields ~80,000–90,000 unique basket viewers across three full grocery-shopping cycles.
• Ines will run the read-out strictly after the 21-day window closes.

---

6. Decision & Rollout Criteria (For Chloe)

• Ship Variant: Primary conversion does not drop by more than Chloe’s threshold of -0.5pp (non-inferiority) AND delivery-related support contacts decrease significantly.
• Iterate / Re-test: Basket-to-order conversion drops by >0.5pp. In this scenario, early fee disclosure hurts conversion, justifying Chloe’s concern; we then test the £60 threshold banner/progress bar to offset drop-off.
• Kill Immediately: Overall revenue drops >5% statistically significantly at the end of Week 1 (safety check only; not an early win stop).

Experiment Spec: Delivery Fee Shown in Basket

Owner: Checkout PM · Build: Ravi · Analysis: Ines · Approver: Chloe

Why we're testing this

Today the £3.99 delivery fee (free over £60) first appears on the final checkout step. Last quarter, 12% of support contacts were about unexpected delivery fees. Design expects that showing the fee earlier will build trust. Chloe's concern is that it will put people off before they start checkout. This test tells us which effect is bigger.

Hypothesis

Showing the delivery fee in the basket will change the share of basket viewers who place an order. We don't assume a direction. The test is two-sided because either outcome is plausible.

Variants

• Control: Current experience. The fee appears only at the final checkout step.
• Variant: The basket shows a "Delivery: £3.99" line, or "Free" when the basket is over £60. The same fee still appears at checkout.

Change from draft: The free-delivery-over-£60 banner is removed from this test. If both changes ship together, we can't tell which one caused any result. The banner could also raise orders while the fee line lowers them, so the two effects would cancel and we'd learn nothing. We'll test the banner as a follow-up once this result is in.

Primary metric

Order conversion: the share of customers who view their basket during the test and go on to place at least one order.

Change from draft: The draft used checkout completion (orders ÷ checkout starts). That metric would be misleading here. If the variant puts off fee-sensitive customers before checkout, the people who remain in checkout are the ones already willing to pay. Completion would then rise even if total orders fell. Order conversion counts everyone who saw the basket, so it catches losses at either stage.

Secondary and guardrail metrics

• Checkout start rate and checkout completion rate. These are diagnostic only and show where any change happens.
• Average order value. Seeing the fee may push some customers over £60.
• Revenue per basket viewer. This includes delivery fee income.
• Fee-related support contacts per 1,000 orders. This is the problem we set out to fix.

Randomisation

• Unit: customer (logged-in account ID). The draft split by session, which needs to change. About half of each week's basket viewers are returning customers. With a session split, they would see the fee in the basket on one visit and not on the next. That would contaminate both groups and confuse customers.
• Split: 50/50, sticky for the whole test.
• Ravi: Please log the assignment when a customer first views their basket, and confirm that logged-out traffic is either excluded or bucketed consistently.

Sample size and duration

• Assumptions: baseline 8% order conversion, two-sided test at α = 0.05, 80% power.
• Minimum effect we want to detect: 0.5 percentage points (8.0% → 8.5%). We're reading Chloe's threshold as absolute points on order conversion. That is roughly a 6% relative change.
• Customers needed: about 47,500 per arm, or about 95,000 total.

How long that takes: About 40,000 customers view their basket in week 1, then about 20,000 new customers each week after. That gives roughly 40k, 60k, 80k and 100k cumulative customers. The test therefore runs for 4 full weeks.

Caveat for Ines: The 8% baseline is per basket viewer per week. Over a 4-week window, each customer has more chances to order, so the per-customer baseline will be higher. At 12%, for example, we'd need about 135,000 customers, which means about 6 weeks. Before launch, Ines will calculate the 4-week per-customer baseline from historical data. The final duration will be fixed from that number and recorded here.

Stopping rule

Change from draft: We won't stop the test as soon as the dashboard shows significance. If we check every day and stop on the first significant result, the false-positive rate climbs well above 5%. The dashboard will often show early "wins" that aren't real.

• The test runs for the pre-set duration, in full weeks only.
• The dashboard can be watched for bugs and sample-ratio mismatch. Nobody makes ship decisions from it mid-test.
• Only reason to stop early: a guardrail failure. That means a broken experience, a sample-ratio mismatch (p < 0.001), or order conversion down by more than 1.5 points.

Decision rules

The confidence interval (CI) below is the interval around the difference in order conversion between the variant and control.

Sign-off

• ☐ Ravi: build plus assignment logging
• ☐ Ines: baseline check and final duration
• ☐ Chloe: approval