Experiment specification

Waymark checkout experiment spec

Hypothesis

Showing the delivery date and fee on the basket page will increase the share of eligible shoppers who place an order by setting expectations earlier and reducing last-step abandonment. It may also cause some shoppers to leave sooner; the primary metric must capture both effects.

Design and randomisation

Randomise eligible shoppers 50:50 when they first view a basket with delivery information available:

• Control: Current experience; date and fee appear at the final checkout step.
• Treatment: Date and fee appear on the basket page and remain visible through checkout.

Assign by stable customer ID across app and web, so shoppers who use both platforms see the same variant. Use a persistent anonymous ID for guests and merge it with the account on login. Stratify assignment by platform at first exposure.

Metrics

• Primary: Order conversion among eligible basket visitors: shoppers placing an order divided by shoppers randomised. This captures both earlier exits and improved final-step completion.
• Secondary: Checkout-start rate; completion among checkout starters; step-by-step abandonment; time to order; AOV; results by app/web.
• Guardrails: Basket-to-order conversion must not fall by more than 0.3 percentage points; monitor cancellations, fee-related support contacts, payment failures and site/app errors.

Sample size and duration

The traffic file covers 56 days. Summing its daily rows gives approximately 2.09m checkout sessions and 132k orders:

• Baseline checkout-session conversion: 132k ÷ 2.09m ≈ 6.3%
• Average traffic: 2.09m ÷ 56 ≈ 37.3k sessions/day, or 18.65k/arm/day at 50:50.

Power for a 5% relative lift (6.3% to 6.615%), with 80% power and two-sided 5% significance:

\[ n_{\text{per arm}}\approx \frac{2(1.96+0.84)^2(0.063)(0.937)}{(0.00315)^2} \approx 93{,}000 \]

Because shoppers can generate repeated sessions, conservatively inflate by the supplied 3.8 checkout sessions per buyer per month (worst-case design effect, assuming fully correlated sessions):

\[ 93{,}000 \times 3.8 \approx 353{,}000 \text{ sessions/arm} \]

\[ 353{,}000 \div 18{,}650 \approx 18.9 \text{ days} \]

Plan for at least 19 days, including full weeks and weekends, then allow time for order outcomes to mature. This is a provisional calculation: the file reports checkout sessions, not eligible basket visitors, and therefore cannot establish the primary metric’s baseline or power. Before launch, confirm basket-visitor volume and conversion, and validate the repeat-session assumption against deduplicated users. The supplied 412k monthly buyers and session-frequency figures should also be reconciled with observed traffic.

Calendar

Do not run concurrently with Payments’ checkout experiment, which ends 9 November. Starting 10 November would require about 19 days, crossing both the 16 November code freeze and 20 November Black Friday promotions. Defer until those changes and promotions are over and the freeze is lifted; the calendar does not specify those end dates. Confirm a stable window of at least three weeks before scheduling.

Decision rule

Ship only if the primary conversion lift is statistically significant at two-sided p < .05, the 95% confidence interval excludes zero, and the basket-to-order guardrail is not breached. Otherwise, do not ship; investigate platform and funnel-step results before proposing a follow-up. Don’t stop early for a positive result; stop only for a pre-agreed severe guardrail issue or data-quality failure.

Experiment spec: delivery date and fee on the basket page

Owner: Checkout PM · Status: Draft. Blocked on one data question (see 0).

0. Data question to resolve before launch

The two sources disagree badly:

• CSV: about 16,525 orders a week, or about 70,800 per 30 days.
• Data team: 412,000 people ordered at least once in 30 days, so at least 412,000 orders.
• Sessions: 412,000 × 3.8 = about 1.57M checkout sessions a month. The CSV shows about 1.12M.

The CSV may cover one region or a subset, or the definitions may differ. I've sized the test on the CSV because it is the conservative figure. If real traffic is about 6× higher, the test still runs the full 4 weeks to cover the weekly cycle. It would simply detect a smaller effect.

1. Hypothesis

Showing the delivery date and fee on the basket page removes the "surprise fee" at the final step. That reduces last-step abandonment and increases orders per shopper.

The counter-risk is that some shoppers leave at the basket instead. The test measures the net effect of both.

2. What's randomised and how

• Unit: account ID, 50/50. It is not the session. Buyers average 3.8 checkout sessions a month, and 31% use both app and web. Session-level randomisation would show the same person both versions and contaminate the result.
• Assignment point: first basket-page view in the test window, with the assignment held for the whole test.
• Logged-out web baskets: use the device cookie, then map to the account at login. Accounts that end up in both arms are excluded, and the rate is reported.
• Platforms: both, with app and web as pre-registered segments.
• Run first: a 2-day A/A check for sample-ratio mismatch.

Why the metric can't be "orders ÷ checkout sessions": the treatment changes who starts checkout. People who leave at the basket drop out of the denominator, which would inflate the treatment's rate. The denominator must be fixed at the point of assignment.

3. Metrics

Primary: orders per assigned account during the test window.

Secondary (diagnostic): - Basket → checkout start rate. Expected to fall; this is the risk the squad named. - Final-step abandonment rate. Expected to fall. - Basket → order conversion, split by app and web. - Share of baskets topped up to reach a free-delivery threshold, if one exists.

Guardrails (any breach blocks shipping): - Average order value (baseline $64): no drop of more than 2%. - Delivery-fee revenue per order: no significant drop. Must be explained if it moves. - Basket-page load time (p75): no increase of more than 100 ms. Basket-page error rate: no increase. - Customer contacts about fees or delivery date: no significant increase. - Sample-ratio mismatch: p > 0.001.

Note: the 30-day repeat rate (38%) can't be read inside the window. It becomes a follow-up read 30 days after the test ends.

4. Sample size and duration

Baseline from 8 weeks of CSV data (10 Aug – 4 Oct):

• Weekly volume is very stable: about 260,424 sessions and 16,525 orders a week.
• Weekdays run about 40% above weekends, so the test must run whole weeks.

Minimum detectable effect: a 3% relative lift, from 6.345% to 6.535% (+0.190 pp). At current volume that is about 496 orders a week, or about $31.7k a week at $64 average order value.

Calculation (α = 0.05 two-sided, 80% power):

• n per arm = (1.96 + 0.84)² × [p₁(1−p₁) + p₂(1−p₂)] / d²
• = 7.84 × (0.05942 + 0.06108) / 0.0019035²
• = 0.9447 / 3.623×10⁻⁶
• ≈ 260,750 sessions per arm, or 521,500 in total

Adjusting for repeat users: randomising by account means sessions from the same user are correlated. I've applied a design effect of 1.5 because the true clustering factor is unknown:

• 521,500 × 1.5 = 782,000 sessions
• ÷ 260,424 a week = 3.0 weeks

Planned duration: 4 full weeks. The extra week covers uncertainty in the clustering factor. It also leaves enough power for a directional read on app and web separately (each about 0.5–0.6 of total traffic).

The analysis will use account-level orders with variance computed directly from the data, so the 1.5 design effect is not baked into the result.

5. When it can run

Waiting until Pay later ends (10 Nov) leaves only 6 days before the freeze, which is too short. So we run alongside it.

Dates: - Build and A/A check: 5–11 Oct. - Test: Mon 12 Oct – Sun 8 Nov. That is exactly 4 weeks. It ends the day before Pay later finishes, so Pay later's 50/50 split is constant for our whole test. - Analysis and ship decision: 9–13 Nov, before the freeze.

6. Decision rule (pre-registered)

1. Ship if the primary lift is positive and significant (p < 0.05), no guardrail is breached, and sample-ratio mismatch passes. 2. Don't ship if: - the primary is significantly negative, or - any guardrail is breached.

Either way, investigate whether the basket-stage drop outweighed the last-step gain. 3. Flat result (confidence interval within ±3%, no guardrail breach): ship only if final-step abandonment fell significantly, on the grounds of transparency. Otherwise keep control. 4. Pay later interaction: if the effect differs significantly between Pay later arms, report the effect within each arm. The ship decision then follows Pay later's own launch decision. 5. No early stopping on the primary metric. A guardrail breach above a pre-set threshold may stop the test early.

Experiment: delivery transparency on the basket

Status: Proposed; launch requires a basket-level sizing check and agreement with Payments. The supplied traffic supports a planning estimate, not a defensible final user-level sample size.

Hypothesis and treatment

Showing the next-day delivery date and fee on the basket increases completed orders by removing a late surprise. It may instead move abandonment earlier, so success must be measured from basket exposure, not checkout entry.

• Control: Date and fee first appear at the final checkout step.
• Treatment: Show the same date and fee on the basket, before the checkout button. Retain the final-step confirmation.
• No changes to fees, delivery availability, payment options or other copy. Recalculate both displays consistently if the basket or address changes.

Eligibility and randomisation

Randomise eligible signed-in customers 50:50 at their first basket view, before rendering either experience. Eligibility requires an address sufficient to quote delivery accurately.

Use a persistent customer-ID assignment shared across app and web. Session/device randomisation would contaminate exposure: 31% of buyers use both platforms. Anonymous customers are excluded unless we can reliably preserve and reconcile assignment.

Use an independent experiment hash, balanced within existing Payments assignment and first-exposure platform. Persist assignment throughout the experiment.

Analyse intention-to-treat: include everyone assigned, including customers who never enter checkout. Each customer contributes once to the primary outcome.

Metrics

Primary: Percentage of assigned customers placing at least one order within 24 hours of first eligible basket exposure, on either platform.

Secondary: - Checkout entry within 24 hours. - Basket-to-order conversion and final-step abandonment, with explicit funnel denominators. - Orders and revenue per assigned customer within 24 hours. - Order value, descriptive only: treatment can change who buys. - Repeat ordering within 30 days of assignment, reported after maturation. The supplied 38% is buyer-conditional, not this metric’s baseline.

Guardrails, precommitted proposed tolerances: - Revenue per assigned customer: no more than 1% relative deterioration. - Cancellation/refund incidence per assigned customer: no more than 0.2 percentage-point deterioration. - Checkout technical-error incidence per assigned customer: no more than 0.1 percentage-point deterioration. - Basket p95 latency: no more than 100 ms deterioration. - Any incorrect fee/date presentation is a stop-and-fix issue.

Instrument assignment, basket exposure, checkout steps, displayed quote and orders. Check sample-ratio mismatch, cross-platform consistency and event completeness before interpreting results.

Sample size and duration: sums

Across the supplied 56 days:

Average checkout traffic:

\[ 2,083,390/56=37,203\text{ sessions/day}. \]

Planning assumptions: two-sided α=5%, 80% power, 5% relative lift as the minimum detectable effect—not an observed or commercially validated threshold.

Using checkout conversion only as a provisional proxy:

\[ p=132,196/2,083,390=0.063452,\quad \Delta=0.05p=0.003173. \]

Approximate equal-arm sample:

\[ n=\frac{2(1.96+0.84)^2p(1-p)}{\Delta^2} \approx92,600\text{ per arm}. \]

Thus approximately 185,200 independent observations. Naively:

\[ 185,200/37,203=4.98\text{ days}. \]

Five days is not a valid duration. These are checkout sessions, not unique eligible basket visitors; checkout entry is affected by treatment. The 3.8 sessions/month figure applies only to buyers, so dividing total sessions by 3.8 would not repair the denominator. Nor do 412,000 monthly buyers establish eligible traffic.

Pre-launch requirement: Measure deduplicated eligible basket visitors, their 24-hour conversion, and expected new enrolments over the planned window. Recalculate using that conversion rate; separately check guardrail power. Lock the largest required sample before launch.

Proposed maximum enrolment is 28 days, representing:

\[ 28\times37,203\approx1,041,695 \]

checkout sessions as a volume benchmark—not a count of independent participants. Launch only if forecast eligible customers support the locked sample within that window.

Calendar

Propose 12 October–8 November 2026, four complete weeks, with primary outcomes matured by 9 November.

This overlaps Pay Later. Obtain Payments agreement for independent allocation, retain its assignment in analysis, and estimate the treatment-by-payment interaction. Otherwise, do not use this window.

Search ends 30 October; log its assignment and account for calendar-date changes.

The uncontaminated pre-promotion window is only 10–19 November: ten days. The 16 November freeze requires any running experiment and kill switch to be preapproved; it is not automatically a shutdown. Use that alternative only if revised sizing supports it. Otherwise postpone until a confirmed post-promotion window.

Decision rule

At the locked endpoint, ship if the primary lift’s two-sided 95% confidence interval excludes zero positively and all guardrails pass their precommitted non-inferiority checks. Review platform and Payments interactions before broad rollout.

Reject on clear conversion harm or a guardrail breach. Otherwise call it inconclusive—not “no effect.” No efficacy peeking or opportunistic extension; safety monitoring remains continuous.