Analyse experiment results

Subject: Hold on Monday’s launch — Home Page ("Your usual basket") test readout

Hi Zoe, team,

I dug into the raw data to sanity-check the numbers before Monday. My recommendation is that we do not ship on Monday.

While the top-line numbers initially look like a ~5.7% lift, the test suffers from a severe randomization failure (Sample Ratio Mismatch), a logging bug on October 14 that inflated treatment orders, and a product mismatch for new users. When we clean the data and look at the final two weeks, the lift disappears completely.

Here is the full breakdown and my working.

---

1. What the raw data shows (and how we got to ~6%)

If we sum `daily_results.csv` across all 28 days: * Control: 196,353 user-days | 16,580 orders (0.08444 orders/user-day) * Treatment: 226,672 user-days | 20,231 orders (0.08925 orders/user-day) * Apparent Lift: +5.70% ($Z = 5.56, p < 0.0001$) * Guardrail (AOV): Control averaged ~$48.25 vs. Treatment ~$48.00 (-0.52%, well within our -2.0% guardrail).

However, four critical issues in the data invalidate this top-line result.

---

2. What kills the top-line conclusion (Showing the working)

#### A. Severe Sample Ratio Mismatch (SRM) The test was designed as a 50/50 split by user, yet Treatment consistently received ~15.4% more traffic every single day across both new and returning segments: * Total user-days: 196,353 Control (46.42%) vs. 226,672 Treatment (53.58%). * Chi-square test against 50/50: $\chi^2 = 2,173, p \ll 0.0001$. Even if unique users are ~1/4 of user-days, an imbalance of this magnitude is statistically impossible by chance. * Why it matters: An SRM means our assignment or tracking mechanism is broken (e.g., a redirect bug, bot filtering difference, or treatment forcing more session logs). Until we know why 15.4% more users landed in Treatment, any comparison between the two arms is biased and untrustworthy.

#### B. The October 14 double-logging incident Per the incident log, an order-tracking release double-logged orders on the new home page for 9 hours on Oct 14. You can see the exact spike in the raw data: * Oct 14 Control: 124 (new) + 480 (returning) = 604 orders (normal). * Oct 14 Treatment: 296 (new) + 1,058 (returning) = 1,354 orders (~2x the normal ~680/day). * Removing Oct 14 (27-day totals): * Control: 189,327 user-days | 15,976 orders (0.08438 orders/user-day) * Treatment: 218,565 user-days | 18,877 orders (0.08637 orders/user-day) * Adjusted Lift: Drops by more than half to +2.35% ($Z = 2.26, p = 0.024$).

#### C. User-level vs. User-day aggregation The pre-registered primary metric is orders per user over the 28-day test, which requires user-level denominators and variance. However, `daily_results.csv` reports daily active users and orders. Summing daily users gives user-days, not unique users. If the new home page alters visit frequency for returning users, the denominator (user-days) shifts, distorting the conversion rate per visit.

#### D. Segment & Novelty Breakdown: Why did New users spike in Week 1? If we break that remaining +2.35% lift down by user type and week (excluding Oct 14), we uncover a major anomaly:

Look closely at what this table tells us: 1. Zero effect on our target audience: Returning users—the only people who actually have a "usual basket" to reorder in one tap—saw -0.03% lift over the test. The feature did not move their order rate at all. 2. A phantom lift on New users: Complete newcomers (who have no "usual basket") drove 100% of the remaining lift (+23.2% in Week 1), which then decayed rapidly to +4.2% by Week 3 and Week 4. 3. Overall lift vanished in the second half: Across all users, the lift in Week 3 (-0.29%) and Week 4 (+1.20%, $Z=0.58, p=0.56$) is completely flat and statistically indistinguishable from zero.

(Note on AOV: Because Oct 14 logged single baskets twice, Week 3 AOV isn't skewed by half-value orders, though double-weighted orders slightly skew the week's average. Regardless, AOV isn't why we're holding).

---

3. What this test does and doesn't tell us

• What it shows: After removing the Oct 14 bug, there is 0% lift on returning users, and overall lift decays to noise (+0.45% across Weeks 3–4 combined). Furthermore, the test is compromised by a 15.4% SRM.
• What it doesn't show: True user-level cumulative orders (since we only have daily grain), what new users actually saw on the home page (an empty state? a fallback? a bug?), and why the traffic split failed.

---

4. What we should do next

1. Hold Monday's launch. Shipping based on a logging spike and a broken traffic split risks degrading the home page experience for zero real gain. 2. Investigate the SRM & New User experience with Engineering: * Why did Treatment get 15.4% more users every day? (Check bucketing logic, caching, and trigger timing). * Why did New users see a +23% spike in Week 1 that decayed? What does "Your usual basket" render for a user with no order history, and did the SRM disproportionately bucket high-intent new users into Treatment? 3. Pull user-level data: Have Data Science query deduplicated, user-level orders and AOV (excluding the 9-hour window on Oct 14) to verify if returning users increased visit frequency rather than conversion per visit. 4. Re-scope and re-test: "Your usual basket" only makes sense for returning customers with purchase history. Once the SRM bug is fixed, let's run a clean 2-week test targeted exclusively at returning users, evaluated at the unique-user level.

Happy to walk through the numbers live before you message leadership!

Readout: new home page test (23 Sep – 20 Oct)

Recommendation: don't ship Monday on this evidence. Zoe's +6% is real arithmetic (I reproduce +5.8%), but most of it comes from a logging bug and an early novelty effect. The test also has a randomization problem that has to be explained first.

Reproducing the headline

Orders per user-day, all 28 days: control 16,560 / 196,173 = 8.44%, treatment 20,234 / 226,652 = 8.93%, a lift of +5.8%. That matches Zoe's number.

Problems with it

1. Double-counted orders on 14 Oct inflate treatment. Treatment orders that day are about double normal (new 296 vs ~150, returning 1,058 vs ~540). Control is normal (124 and 480), which fits the incident log. That is roughly 660 phantom orders. Dropping 14 Oct from both arms cuts the lift from +5.8% to +2.4% (8.44% → 8.64%).

The size also doesn't fit the log. A 9-hour duplication should add roughly 40% to that day's orders, not 100%. So the incident may have been bigger than described, and I wouldn't patch it by subtracting my estimate. Engineering should dedupe by order ID and confirm no other days are affected.

2. The lift is a novelty effect and has largely disappeared. Orders per user-day by week, excluding 14 Oct:

• Returning users, who are about 70% of traffic, show no effect in any week.
• New users show a big lift that decays week over week.
• Over the last 14 days (ex 14 Oct), all users combined: 8.47% vs 8.42%, +0.6%. A rough z-test gives z ≈ 0.4, which is not significant, and that is before accounting for repeat users, which would widen the interval further.

The mechanism is also odd. "Your usual basket" should help returning users, who have a basket, and do little for new users, who have none. The effect shows up in the wrong group, which makes me doubt it is caused by the feature.

3. Sample ratio mismatch. The split was meant to be 50/50 by user, but treatment has 226,652 user-days vs 196,173 for control, a ratio of 1.155 (chi-square ≈ 2,200 against a 50/50 expectation). The ratio is the same from day one (2,400 vs 2,089 on 23 Sep) and the new/returning mix is identical in both arms (about 30% new). That pattern looks like an assignment or bucketing fault rather than a behavioral effect, which would likely differ by user type and move over time.

There is one alternative: "users" might mean daily active users, and the new page might genuinely bring users back more often. In that case orders per active user is a biased denominator. Either way, "orders per user" isn't measuring what the pre-registration intended until we have assigned-user counts. If the arms were truly 50/50, orders per assigned user would be about +22%, so this ambiguity matters in both directions.

What the test does support

• AOV guardrail: treatment is about 0.6% lower in weeks 1–3 ($47.90 vs $48.20, $48.10 vs $48.40, $47.80 vs $48.10) and 0.2% lower in week 4. That is well inside the 2% limit. I have no order-level data for a confidence interval. Week 4 also contains the incident, so I'd recheck it after dedupe.
• No evidence of harm: returning users are flat, and nothing suggests the page hurts.

What it doesn't show

• Any durable lift in orders per user.
• That the effect is caused by the page, given the SRM and the mismatch with the mechanism.
• Whether the pre-registered rule is met. On raw numbers it is "met", but the data feeding it is corrupted, so I wouldn't count it.

Next steps

1. Engineering: explain the 15% allocation imbalance, with assigned-user counts per arm and the bucketing logic. Dedupe orders by ID for the whole test and confirm the scope of the 14 Oct incident. 2. Analytics: rerun on order-level data with orders per assigned user. Use user-clustered confidence intervals, and cut by new vs returning and by days since first exposure. Also get an AOV confidence interval. 3. Decision: - If the SRM is a bug, rerun the test, or at least 2 weeks of clean data with a fresh new-user cohort, since novelty decays over about 3 weeks. - If the SRM is explained and the clean data still show a durable positive lift, ship. - If the business wants to move sooner, I'd support a staged rollout (say 20%) with a persistent holdback, not a full launch. The downside looks small, but the upside is unproven.

Zoe, I'd send the above in place of your note. The headline can't be defended as written, and the cleaned-up result is closer to "no detectable lift after the first couple of weeks."

Readout: new home page test (23 Sep – 20 Oct)

Recommendation: don't ship on Monday. The +6% headline comes from three things:

• A broken 50/50 split.
• A day of double-counted treatment orders.
• A new-user bump that faded to about zero by week 4.

Returning users are the people "Your usual basket" is built for, and they show no lift. None of the three problems is a reason to kill the page. They are reasons not to trust this result yet.

1. Reproducing Zoe's number

Totals across 28 days, using `daily_results.csv`, both user types:

The lift is +5.8%, which matches "about 6%". The arithmetic is right. The problems are in the data underneath it.

2. The split isn't 50/50 (sample ratio mismatch)

The test was pre-registered as 50/50. Treatment has 53.6% of users (226,652 of 422,825).

• A chi-square test against 50/50 gives χ² ≈ 2,200. The chance of that happening randomly is effectively zero.
• The gap is steady: treatment has about 15.5% more users every day, in both segments (new 1.154×, returning 1.156×).
• The new/returning mix is the same in both arms (about 30% new), so this isn't a mix-shift artefact.

A constant 15% surplus usually means assignment or exposure logging is broken. For example, treatment could be logging users that control drops, or redirects or caching could differ between arms. Until we know why, we can't assume the two groups are comparable. On its own, this fails the test's validity check, whatever the lift.

3. The 14 October incident doubled treatment orders

On 14 Oct, treatment orders roughly doubled while control stayed normal:

• New users: 296 orders in treatment, against about 150 on the days around it. Control: 124.
• Returning users: 1,058 orders in treatment, against about 530. Control: 480.

This matches the incident log: order events were logged twice, on the new home page only. That day adds about 680 phantom orders to treatment.

Excluding 14 Oct from both arms:

The overall lift drops from 5.8% to 2.4%. A naive z-test on that is about 2.3, but it treats each user-day as independent, which overstates confidence. The same user appears on many days, and the split itself is suspect.

4. The remaining lift is a fading new-user effect

New users, orders per user by week (week 4 excludes 14 Oct):

Returning users by week: +0.5%, −0.8%, −1.0%, +1.0%. That is flat, which is what noise looks like.

Overall lift went from +5.6% in week 1 to +1.1% in week 4. Two things stand out:

• The effect is decaying towards zero. That pattern looks like novelty, not a lasting change in behaviour.
• It shows up in new users, who mostly have no "usual basket" yet. It is absent in returning users, who are who the feature is for. That doesn't fit the mechanism we expected, and we should understand it before acting.

5. Average order value guardrail

Treatment AOV is $0.30 lower in weeks 1–3 (−0.6%) and $0.10 lower in week 4 (−0.2%). That is inside the 2% guardrail, so "flat" is fair.

Two caveats:

• Week 4 may include the duplicated orders.
• AOV only matters alongside order volume. With clean data, revenue per user is roughly +2.4% × 0.995 ≈ +1.8% overall, and about −0.7% for returning users.

What the test shows

• The new page probably raises early ordering among new users, at least for a couple of weeks.
• It doesn't hurt order value by enough to trip the guardrail.
• No breakage or disaster is visible.

What it doesn't show

• A valid comparison. The sample ratio mismatch means the arms may not be comparable.
• A lasting lift. By week 4 the effect is about 1%, inside the noise.
• Any effect on returning users, the target audience.
• The pre-registered metric. The test design specifies orders per user, but this file holds user-days. We need user-level data to compute it properly. The current significance claim relies on treating user-days as independent.
• That the incident only hit treatment. "Engineering believes" isn't confirmation. If control was also affected at a lower level, the 14 Oct correction above is off.

Should we ship?

No, not on this evidence.

The decision rule technically passes on the raw numbers. But it assumed a valid 50/50 split and clean order counts, and we have neither. Shipping now would lock in a 6% claim that is mostly a logging bug, an assignment bug and novelty.

Next steps

1. Find the cause of the 15% user surplus (owner: eng + analytics, this week). Compare assignment logs against exposure logs, and check bot filtering, redirects and caching per arm. Run an A/A check once it's fixed.
2. Deduplicate 14 Oct using order or event IDs, rather than dropping the day, and confirm control was unaffected.
3. Pull user-level data and recompute orders per user as pre-registered. Use proper per-user variance, or bootstrap by user.
4. Rerun once the split is fixed, for at least 4 weeks. Report results by week and by new vs returning users, with returning users as the key read. If returning users still show nothing, rethink the feature (e.g. show "usual basket" only once a user has one) rather than shipping it to everyone.
5. Investigate the new-user effect: what do new users see in the "usual basket" slot, and what are they ordering? If it's real and durable, a new-user variant could be worth its own test.

Zoe: the analysis was a reasonable first pass and the headline maths checks out. Happy to pair on the rerun plan so we can bring a clean answer back.