Experiment specification

Experiment Spec: Upfront Delivery Fee in Basket

To: Ravi (Eng), Ines (Analytics), Chloe (Head of Growth) From: Checkout PM Subject: Experiment Spec – Showing Delivery Fee in Basket

1. Background & Hypothesis

Last quarter, 12% of support contacts were complaints about unexpected delivery fees. Currently, our £3.99 fee (free on orders >£60) is hidden until the final checkout step.

• Hypothesis: Showing the delivery fee in the basket will build trust and reduce support contacts without hurting overall conversion, rather than scaring users away early.

2. Scope & Variants (Fixing the Draft)

Chloe suggested adding a "Free delivery over £60" banner to the variant. Recommendation: Hold this for a follow-up test. If we test the fee and the promotional banner together, we won’t know if a change in conversion came from fee transparency (friction) or the £60 threshold incentive (upsell). However, because our delivery fee is dynamic, the variant line item must accurately reflect the £60 rule so users aren't misled.

• Control: Current basket experience (no delivery fee shown until final checkout step).
• Variant: Add a single, standard-text line item to the basket summary:
• If basket < £60: “Estimated Delivery: £3.99 (Free over £60)”
• If basket ≥ £60: “Estimated Delivery: FREE”
• (Note for Chloe: If we want to test a high-visibility promotional banner/progress bar to drive AOV, let's fast-follow with that as V2).

3. Metrics (Fixing the Draft)

The draft plan used Checkout Completion Rate (Orders ÷ Checkout Starts) as the primary metric. We must change this. Moving the fee to the basket will likely cause sticker-shock drop-offs to happen in the basket rather than at the end of checkout. This will shrink the number of "Checkout Starts" and artificially inflate Checkout Completion Rate, even if total orders drop.

Because the intervention happens in the basket, our denominator must be basket viewers:

• Primary Metric: Basket-to-Order Conversion Rate (Unique customers placing an order ÷ Unique customers viewing the basket). Baseline: 8.0%.
• Secondary Metrics:
• Average Order Value (AOV): To see if the subtle text nudges baskets over £60. Baseline: £47.
• Basket-to-Checkout Start Rate & Checkout Completion Rate: To observe the shift in where drop-off occurs.
• Support Contacts (Delivery Fee): Tracked via contact form tags/UTMs where possible.
• Guardrail Metric: Net Revenue per Basket Viewer (ensuring total revenue doesn't drop even if conversion holds).

4. Randomization & Targeting (Fixing the Draft)

The draft plan splits traffic 50/50 by session. We must change this to a 50/50 split by Customer/User ID (or persistent device cookie for logged-out users). Groceries are a high-consideration purchase; 50% of our weekly traffic consists of returning users. Splitting by session means a single customer could see the fee in the morning, and not see it in the evening, causing confusion and polluting our data.

• Trigger: User lands on `/basket`.

5. Sample Size & Duration (Fixing the Draft)

The draft plan suggests checking daily and stopping when significant. We cannot do this ("peeking" skyrockets false-positive rates). We need a fixed horizon.

• Minimum Detectable Effect (MDE): Chloe set our threshold at 0.5 percentage points (an absolute shift from 8.0% to 8.5% or 7.5%).
• Sample Size Needed: At an 8% baseline, 80% power, and 95% significance (two-tailed), detecting a 0.5pp change requires ~46,200 unique users per variant (~92,400 total).
• Duration: 4 Weeks (Fixed).
• Traffic math: We get 40,000 unique basket viewers in Week 1, and ~20,000 net-new viewers each subsequent week (since 50% are repeat visitors). We will hit ~100,000 unique users across the test at the end of Week 4. Running in full weekly increments also accounts for weekend vs. weekday shopping cycles.

6. Decision Criteria

After 4 weeks, Ines will run the readout (segmented by New vs. Returning users): * Ship Variant: If Basket-to-Order CR is flat (within ±0.5pp) or significantly positive, AND AOV/Revenue guardrails do not significantly decline. * Rollback: If Basket-to-Order CR drops by ≥ 0.5pp.

Sign-offs Needed: * Chloe: Approve scope (holding the promo banner for V2) and 4-week runtime. * Ines: Confirm sample size math and user-level tracking. * Ravi: Confirm engineering effort for user-level bucketing on `/basket`.

To: Chloe (Head of Growth), Ravi (Growth Engineer), Ines (Data Analyst) From: PM, Checkout Subject: Experiment Spec: Early Delivery Fee Display in Basket

Team, here is the polished spec for our upcoming checkout experiment. I’ve incorporated Chloe’s threshold, fixed a few methodological traps in the draft, and scoped this for a clean, reliable read.

---

1. Objective & Hypothesis

• Current State: Delivery fee (£3.99, free over £60) is only revealed on the final checkout step. 12% of last quarter’s support contacts were fee-related surprises.
• Hypothesis: Showing the delivery fee (and progress to free delivery) directly in the basket increases customer trust and sets clear expectations upfront.
• Expected Impact: While Chloe rightly worries this might cause some upfront drop-off among sub-£60 baskets, we believe the resulting higher-intent checkout starts will lift overall conversion and reduce cart abandonment due to "sticker shock."

---

2. Variant Definition

• Control (A): Existing basket experience. Delivery fee hidden until final checkout step.
• Variant (B):
• Displays the delivery fee clearly in the basket summary.
• Note on scope: Per Chloe’s suggestion, we are bundling the "free delivery over £60" progress banner into Variant B. Since the fee and the threshold are conceptually tied, showing one without the context of the other would skew the test.

---

3. Traffic Allocation & Unit of Randomization

• Unit of Randomization: User-level (logged-in state or persistent cookie). Correction from draft: Session-level splitting would cause a jarring experience if a user refreshes or returns and sees fluctuating fee visibility.
• Split: 50/50 (Control / Variant).
• Audience: All customers viewing the basket (~40,000 unique users/week, split roughly 50/50 between new and returning).

---

4. Metrics

• Primary Metric (Guardrail for Business Value):
• Order Conversion Rate: Completed Orders ÷ Total Basket Views.
• (Correction from draft: The draft proposed "Checkout Completion Rate (Orders ÷ Checkout Starts)." That misses top-of-funnel drop-off. If Variant B scares people away before checkout, measuring only checkout-starts will falsely make the variant look better. We must measure from the basket view).
• Secondary Metrics:
• Checkout Start Rate: Checkout Starts ÷ Basket Views.
• Checkout Completion Rate: Orders ÷ Checkout Starts (to isolate downstream behavior).
• Average Order Value (AOV): To monitor if the £60 threshold banner successfully drives basket-building behavior (baseline: £47).
• Support Metric:
• Volume of delivery-fee-related support tickets (Ines, let’s pull this baseline post-launch).

---

5. Success Criteria & Statistical Guardrails

• Minimum Detectable Effect (MDE): Per Chloe, we care about a ≥ +0.5 percentage point absolute change in the Primary Metric (Order Conversion Rate from basket view, baseline ~8%).
• Duration & Peeking:
• Correction from draft: Do not stop the test the moment p < 0.05. Early peeking yields false positives.
• We will run the test for a fixed duration of 2 full weeks to capture day-of-week and new vs. returning user cycles (~80,000 total unique users), yielding adequate statistical power for our MDE.

---

Next Steps:

• Ravi: Let’s sync on tracking implementation for the basket-level fee and banner. Target build completion: [Insert Date].
• Ines: Please set up the tracking dashboard, ensuring we segment by New vs. Returning users and Basket Value (<£60 vs. >£60).
• Chloe: Good to approve?

Experiment Spec: Delivery Fee Shown in Basket

Owner: Checkout PM · Build: Ravi · Analysis: Ines · Approver: Chloe

Why we're testing this

Today the £3.99 delivery fee (free over £60) first appears on the final checkout step. Last quarter, 12% of support contacts were about unexpected delivery fees. Design expects that showing the fee earlier will build trust. Chloe's concern is that it will put people off before they start checkout. This test tells us which effect is bigger.

Hypothesis

Showing the delivery fee in the basket will change the share of basket viewers who place an order. We don't assume a direction. The test is two-sided because either outcome is plausible.

Variants

• Control: Current experience. The fee appears only at the final checkout step.
• Variant: The basket shows a "Delivery: £3.99" line, or "Free" when the basket is over £60. The same fee still appears at checkout.

Change from draft: The free-delivery-over-£60 banner is removed from this test. If both changes ship together, we can't tell which one caused any result. The banner could also raise orders while the fee line lowers them, so the two effects would cancel and we'd learn nothing. We'll test the banner as a follow-up once this result is in.

Primary metric

Order conversion: the share of customers who view their basket during the test and go on to place at least one order.

Change from draft: The draft used checkout completion (orders ÷ checkout starts). That metric would be misleading here. If the variant puts off fee-sensitive customers before checkout, the people who remain in checkout are the ones already willing to pay. Completion would then rise even if total orders fell. Order conversion counts everyone who saw the basket, so it catches losses at either stage.

Secondary and guardrail metrics

• Checkout start rate and checkout completion rate. These are diagnostic only and show where any change happens.
• Average order value. Seeing the fee may push some customers over £60.
• Revenue per basket viewer. This includes delivery fee income.
• Fee-related support contacts per 1,000 orders. This is the problem we set out to fix.

Randomisation

• Unit: customer (logged-in account ID). The draft split by session, which needs to change. About half of each week's basket viewers are returning customers. With a session split, they would see the fee in the basket on one visit and not on the next. That would contaminate both groups and confuse customers.
• Split: 50/50, sticky for the whole test.
• Ravi: Please log the assignment when a customer first views their basket, and confirm that logged-out traffic is either excluded or bucketed consistently.

Sample size and duration

• Assumptions: baseline 8% order conversion, two-sided test at α = 0.05, 80% power.
• Minimum effect we want to detect: 0.5 percentage points (8.0% → 8.5%). We're reading Chloe's threshold as absolute points on order conversion. That is roughly a 6% relative change.
• Customers needed: about 47,500 per arm, or about 95,000 total.

How long that takes: About 40,000 customers view their basket in week 1, then about 20,000 new customers each week after. That gives roughly 40k, 60k, 80k and 100k cumulative customers. The test therefore runs for 4 full weeks.

Caveat for Ines: The 8% baseline is per basket viewer per week. Over a 4-week window, each customer has more chances to order, so the per-customer baseline will be higher. At 12%, for example, we'd need about 135,000 customers, which means about 6 weeks. Before launch, Ines will calculate the 4-week per-customer baseline from historical data. The final duration will be fixed from that number and recorded here.

Stopping rule

Change from draft: We won't stop the test as soon as the dashboard shows significance. If we check every day and stop on the first significant result, the false-positive rate climbs well above 5%. The dashboard will often show early "wins" that aren't real.

• The test runs for the pre-set duration, in full weeks only.
• The dashboard can be watched for bugs and sample-ratio mismatch. Nobody makes ship decisions from it mid-test.
• Only reason to stop early: a guardrail failure. That means a broken experience, a sample-ratio mismatch (p < 0.001), or order conversion down by more than 1.5 points.

Decision rules

The confidence interval (CI) below is the interval around the difference in order conversion between the variant and control.

Sign-off

• ☐ Ravi: build plus assignment logging
• ☐ Ines: baseline check and final duration
• ☐ Chloe: approval