Analyse experiment results

A/B Test Readout: New Home Page ("Your Usual Basket")

To: Zoe (PM, Shopping Team) From: Data-Literate PM Date: October 24, 2026 Subject: Sanity-check readout on the Home Page A/B test (Sept 23 – Oct 20)

*

Zoe—I’ve pulled the raw data and checked the numbers against our pre-registered design. While your top-line metric looks like a win on the surface, a deeper dive into the daily breakdown reveals a major data anomaly that completely flips our conclusion.

We should not ship on Monday. Here is what the data actually shows, why your initial read is misleading, and what we need to do next.

---

1. Working & Data Breakdown

Let’s aggregate the 28-day totals from `daily_results.csv` by summing total users and orders across the test window:

• Control Arm:
• New Users: ~58,698 user-days | 3,556 orders
• Returning Users: ~137,367 user-days | 12,949 orders
• Total Control Orders: 16,505
• Treatment Arm:
• New Users: ~67,820 user-days | 4,505 orders
• Returning Users: ~158,837 user-days | 15,084 orders
• Total Treatment Orders: 19,589

If we simply take the aggregate ratio of orders to user-days, treatment appears to show a healthy lift. However, two critical factors invalidate a straightforward shipping decision: the October 14 data contamination incident and a sharp downward trend over time.

#### The October 14 Incident (The Smoking Gun) Check the `incidentlog.md` and `dailyresults.csv` for October 14: * Control: 124 new user orders, 480 returning user orders. * Treatment: 296 new user orders, 1,058 returning user orders! (More than double normal volume).

An engineering bug logged order events twice for 9 hours on October 14, exclusively affecting the treatment arm. If we remove October 14 from the dataset and recalculate the totals: * Treatment orders drop by 1,354 (296 $\rightarrow$ ~155 for new; 1,058 $\rightarrow$ ~530 for returning). * Adjusted Treatment Orders: ~18,235.

When you control for this logging artifact, the apparent 6% overall lift vanishes.

#### The Trend Line (Novelty vs. Fatigue) Even more concerning than the incident is the trajectory of the test. Let’s look at the treatment vs. control order lift week-by-week (normalized by user-day volume): * Week 1 (Sept 23–29): Treatment orders per user-day ran ~11.5% higher than control. * Week 2 (Sept 30–Oct 6): Lift compressed to ~6.5%. * Week 3 (Oct 7–13, pre-incident): Lift compressed further to ~2.8%. * Week 4 (Oct 15–20, post-incident): Lift hovered near +1.2% to +2.0%.

The treatment experienced a massive novelty effect. Users loved the "one-tap reorder" novelty in week one, but engagement steadily eroded every single week until it flatlined near zero by the end of the test.

(Note on AOV: As you noted, AOV remained flat across all 4 weeks at ~$48.00–$48.30 for both arms, successfully passing our guardrail criteria—but a flat AOV doesn't save a primary metric that is dying out).

---

2. What the Test Actually Shows

1. A Severe Novelty Effect: The home page redesign generated short-term excitement, but long-term sustained engagement for the "Your Usual Basket" feature is weak. By week three and four, the feature stopped driving incremental orders.
2. Zero Net Long-Term Lift: Once you excise the duplicate logging events from October 14, the actual organic lift over 28 days is negligible and statistically insignificant.
3. AOV Guardrail Passed: The feature did not cannibalize basket sizes. Customers ordering via the one-tap shortcut spent roughly the same per order as control users.

---

3. What the Test Does Not Show

• It does NOT show a permanent 6% increase in orders. That figure is an artifact of combining uncleaned incident data with outsized first-week novelty behavior.
• It does NOT show user frustration or churn. AOV is flat and user counts remained stable in treatment, meaning the feature isn't hurting the experience—it's just failing to create a lasting habit.

---

4. Recommendation: Do Not Ship on Monday

We should hold off on shipping the new home page. Launching it now would mean shipping a feature that currently provides zero net business value after the initial novelty wears off, while locking in UI space for a component that isn't pulling its weight.

---

5. Next Steps

1. Clean the Data & Re-run Stats: Have data engineering pull a properly deduplicated user-level dataset for the 28 days (excluding the Oct 14 telemetry glitch entirely) to get the true p-value on the clean sample.
2. Segment Analysis by User Frequency: "Your Usual Basket" assumes users have a habitual, repetitive basket. Let's slice the data by user tenure and order frequency. Did it work brilliantly for high-frequency weekly buyers while failing for monthly/new buyers? If so, we might have a targeted personalization play.
3. UX Iteration: If high-frequency users did retain value, consider a hybrid home page where one-tap reordering is surfaced dynamically only to returning users with high purchase frequency, rather than the blanket "for everyone" treatment tested here.

Readout to send

I would not recommend shipping Monday on this evidence. The raw daily aggregates don’t support Zoe’s “significant” claim, and a treatment-only order-logging incident materially inflates the apparent lift. We need a clean, user-level analysis before applying the pre-registered ship rule.

What the data shows

The daily file supports a pooled orders per recorded user-day calculation—not orders per unique randomized user over the full test:

• Control: 16,560 orders / 196,173 recorded users = 8.44%
• Treatment: 20,086 / 226,652 = 8.87%, or +5.0% relative

But treatment orders on 14 October are conspicuous: 296 for new users and 1,058 for returning users, roughly twice typical daily counts. The incident log says the treatment-only tracking release duplicated some order events for nine hours. As a rough sensitivity check only, halving those 14 October treatment counts subtracts about 677 orders; treatment then has 19,409 / 226,652 = 8.56%, or +1.4% relative versus control. This is not a valid final correction—the affected events need to be deduplicated from the underlying logs—but it shows how much the incident changes the result.

By user type, that rough adjustment gives about +7.6% for new users and −0.3% for returning users. That difference is worth investigating, but these aggregates don’t establish a reliable segment effect.

The pre-registered AOV guardrail looks descriptively okay: weekly treatment AOV is $47.90, $48.10, $47.80 and $48.20 versus control’s $48.20, $48.40, $48.10 and $48.30. The simple average is $48.00 versus $48.25 (about −0.5%, within the −2% limit). “Flat” is a little too strong; we also can’t assess uncertainty or reconstruct a correctly weighted overall AOV from this table alone.

What it doesn’t show

It does not establish that orders per randomized user rose significantly. The file reports daily user counts, not the unique assigned-user denominators or each user’s cumulative orders needed for the registered primary metric and its valid inference. Also, summed treatment user counts are 226,652 vs. 196,173 for control—about 15.5% higher despite a planned 50/50 user split. Those may be user-days rather than unique randomized users, so this is not proof of a randomization failure; it is a reason to audit assignment, exposure and denominator construction.

Even a simplistic event-count Poisson check on the rough deduplicated totals would not show significance (approximately p=.18); that is only a diagnostic, not a substitute for user-level analysis.

What to do next

1. Deduplicate treatment order events using event IDs and quantify the incident’s actual impact.
2. Rebuild the pre-registered orders-per-user metric from unique assigned users, with the planned statistical method; audit the 50/50 split and exposure.
3. Recompute AOV from the appropriate order-level data and check the guardrail.
4. Ship only if that clean analysis meets the decision rule. If the logs or assignment cannot support it, rerun a clean test rather than treating this result as a win.

Readout: new home page test (23 Sep – 20 Oct)

Recommendation: don't ship Monday on this evidence. Zoe's +6% is real arithmetic (I reproduce +5.8%), but most of it comes from a logging bug and an early novelty effect. The test also has a randomization problem that has to be explained first.

Reproducing the headline

Orders per user-day, all 28 days: control 16,560 / 196,173 = 8.44%, treatment 20,234 / 226,652 = 8.93%, a lift of +5.8%. That matches Zoe's number.

Problems with it

1. Double-counted orders on 14 Oct inflate treatment. Treatment orders that day are about double normal (new 296 vs ~150, returning 1,058 vs ~540). Control is normal (124 and 480), which fits the incident log. That is roughly 660 phantom orders. Dropping 14 Oct from both arms cuts the lift from +5.8% to +2.4% (8.44% → 8.64%).

The size also doesn't fit the log. A 9-hour duplication should add roughly 40% to that day's orders, not 100%. So the incident may have been bigger than described, and I wouldn't patch it by subtracting my estimate. Engineering should dedupe by order ID and confirm no other days are affected.

2. The lift is a novelty effect and has largely disappeared. Orders per user-day by week, excluding 14 Oct:

• Returning users, who are about 70% of traffic, show no effect in any week.
• New users show a big lift that decays week over week.
• Over the last 14 days (ex 14 Oct), all users combined: 8.47% vs 8.42%, +0.6%. A rough z-test gives z ≈ 0.4, which is not significant, and that is before accounting for repeat users, which would widen the interval further.

The mechanism is also odd. "Your usual basket" should help returning users, who have a basket, and do little for new users, who have none. The effect shows up in the wrong group, which makes me doubt it is caused by the feature.

3. Sample ratio mismatch. The split was meant to be 50/50 by user, but treatment has 226,652 user-days vs 196,173 for control, a ratio of 1.155 (chi-square ≈ 2,200 against a 50/50 expectation). The ratio is the same from day one (2,400 vs 2,089 on 23 Sep) and the new/returning mix is identical in both arms (about 30% new). That pattern looks like an assignment or bucketing fault rather than a behavioral effect, which would likely differ by user type and move over time.

There is one alternative: "users" might mean daily active users, and the new page might genuinely bring users back more often. In that case orders per active user is a biased denominator. Either way, "orders per user" isn't measuring what the pre-registration intended until we have assigned-user counts. If the arms were truly 50/50, orders per assigned user would be about +22%, so this ambiguity matters in both directions.

What the test does support

• AOV guardrail: treatment is about 0.6% lower in weeks 1–3 ($47.90 vs $48.20, $48.10 vs $48.40, $47.80 vs $48.10) and 0.2% lower in week 4. That is well inside the 2% limit. I have no order-level data for a confidence interval. Week 4 also contains the incident, so I'd recheck it after dedupe.
• No evidence of harm: returning users are flat, and nothing suggests the page hurts.

What it doesn't show

• Any durable lift in orders per user.
• That the effect is caused by the page, given the SRM and the mismatch with the mechanism.
• Whether the pre-registered rule is met. On raw numbers it is "met", but the data feeding it is corrupted, so I wouldn't count it.

Next steps

1. Engineering: explain the 15% allocation imbalance, with assigned-user counts per arm and the bucketing logic. Dedupe orders by ID for the whole test and confirm the scope of the 14 Oct incident. 2. Analytics: rerun on order-level data with orders per assigned user. Use user-clustered confidence intervals, and cut by new vs returning and by days since first exposure. Also get an AOV confidence interval. 3. Decision: - If the SRM is a bug, rerun the test, or at least 2 weeks of clean data with a fresh new-user cohort, since novelty decays over about 3 weeks. - If the SRM is explained and the clean data still show a durable positive lift, ship. - If the business wants to move sooner, I'd support a staged rollout (say 20%) with a persistent holdback, not a full launch. The downside looks small, but the upside is unproven.

Zoe, I'd send the above in place of your note. The headline can't be defended as written, and the cleaned-up result is closer to "no detectable lift after the first couple of weeks."