Analyse experiment results

Subject: Hold on Monday’s launch — Home Page ("Your usual basket") test readout

Hi Zoe, team,

I dug into the raw data to sanity-check the numbers before Monday. My recommendation is that we do not ship on Monday.

While the top-line numbers initially look like a ~5.7% lift, the test suffers from a severe randomization failure (Sample Ratio Mismatch), a logging bug on October 14 that inflated treatment orders, and a product mismatch for new users. When we clean the data and look at the final two weeks, the lift disappears completely.

Here is the full breakdown and my working.

---

1. What the raw data shows (and how we got to ~6%)

If we sum `daily_results.csv` across all 28 days: * Control: 196,353 user-days | 16,580 orders (0.08444 orders/user-day) * Treatment: 226,672 user-days | 20,231 orders (0.08925 orders/user-day) * Apparent Lift: +5.70% ($Z = 5.56, p < 0.0001$) * Guardrail (AOV): Control averaged ~$48.25 vs. Treatment ~$48.00 (-0.52%, well within our -2.0% guardrail).

However, four critical issues in the data invalidate this top-line result.

---

2. What kills the top-line conclusion (Showing the working)

#### A. Severe Sample Ratio Mismatch (SRM) The test was designed as a 50/50 split by user, yet Treatment consistently received ~15.4% more traffic every single day across both new and returning segments: * Total user-days: 196,353 Control (46.42%) vs. 226,672 Treatment (53.58%). * Chi-square test against 50/50: $\chi^2 = 2,173, p \ll 0.0001$. Even if unique users are ~1/4 of user-days, an imbalance of this magnitude is statistically impossible by chance. * Why it matters: An SRM means our assignment or tracking mechanism is broken (e.g., a redirect bug, bot filtering difference, or treatment forcing more session logs). Until we know why 15.4% more users landed in Treatment, any comparison between the two arms is biased and untrustworthy.

#### B. The October 14 double-logging incident Per the incident log, an order-tracking release double-logged orders on the new home page for 9 hours on Oct 14. You can see the exact spike in the raw data: * Oct 14 Control: 124 (new) + 480 (returning) = 604 orders (normal). * Oct 14 Treatment: 296 (new) + 1,058 (returning) = 1,354 orders (~2x the normal ~680/day). * Removing Oct 14 (27-day totals): * Control: 189,327 user-days | 15,976 orders (0.08438 orders/user-day) * Treatment: 218,565 user-days | 18,877 orders (0.08637 orders/user-day) * Adjusted Lift: Drops by more than half to +2.35% ($Z = 2.26, p = 0.024$).

#### C. User-level vs. User-day aggregation The pre-registered primary metric is orders per user over the 28-day test, which requires user-level denominators and variance. However, `daily_results.csv` reports daily active users and orders. Summing daily users gives user-days, not unique users. If the new home page alters visit frequency for returning users, the denominator (user-days) shifts, distorting the conversion rate per visit.

#### D. Segment & Novelty Breakdown: Why did New users spike in Week 1? If we break that remaining +2.35% lift down by user type and week (excluding Oct 14), we uncover a major anomaly:

Look closely at what this table tells us: 1. Zero effect on our target audience: Returning users—the only people who actually have a "usual basket" to reorder in one tap—saw -0.03% lift over the test. The feature did not move their order rate at all. 2. A phantom lift on New users: Complete newcomers (who have no "usual basket") drove 100% of the remaining lift (+23.2% in Week 1), which then decayed rapidly to +4.2% by Week 3 and Week 4. 3. Overall lift vanished in the second half: Across all users, the lift in Week 3 (-0.29%) and Week 4 (+1.20%, $Z=0.58, p=0.56$) is completely flat and statistically indistinguishable from zero.

(Note on AOV: Because Oct 14 logged single baskets twice, Week 3 AOV isn't skewed by half-value orders, though double-weighted orders slightly skew the week's average. Regardless, AOV isn't why we're holding).

---

3. What this test does and doesn't tell us

• What it shows: After removing the Oct 14 bug, there is 0% lift on returning users, and overall lift decays to noise (+0.45% across Weeks 3–4 combined). Furthermore, the test is compromised by a 15.4% SRM.
• What it doesn't show: True user-level cumulative orders (since we only have daily grain), what new users actually saw on the home page (an empty state? a fallback? a bug?), and why the traffic split failed.

---

4. What we should do next

1. Hold Monday's launch. Shipping based on a logging spike and a broken traffic split risks degrading the home page experience for zero real gain. 2. Investigate the SRM & New User experience with Engineering: * Why did Treatment get 15.4% more users every day? (Check bucketing logic, caching, and trigger timing). * Why did New users see a +23% spike in Week 1 that decayed? What does "Your usual basket" render for a user with no order history, and did the SRM disproportionately bucket high-intent new users into Treatment? 3. Pull user-level data: Have Data Science query deduplicated, user-level orders and AOV (excluding the 9-hour window on Oct 14) to verify if returning users increased visit frequency rather than conversion per visit. 4. Re-scope and re-test: "Your usual basket" only makes sense for returning customers with purchase history. Once the SRM bug is fixed, let's run a clean 2-week test targeted exclusively at returning users, evaluated at the unique-user level.

Happy to walk through the numbers live before you message leadership!

Readout: Home page test ("Your usual basket")

To: Zoe Re: Monday ship recommendation

Recommendation: don't ship Monday. The readout overstates the result. The test has a sample ratio problem, a logging incident inside the test window, and a lift that is fading. Next steps are at the end.

Working

I summed the daily CSV across all 28 days. The file gives daily active users and orders, so each "user" is a user-day. That matters for the first finding below.

Correction: on 14 October, treatment orders were about double the surrounding days (new: 296 vs ~150; returning: 1,058 vs ~530). I replaced that day with half the logged value. Using the average of the adjacent days gives nearly the same answer (+2.2%).

What the test shows

1. The split isn't 50/50. Treatment received 53.6% of user-days, and the ratio holds every day (roughly 2,400 treatment vs 2,090 control among new users, and 5,700 vs 4,900 among returning users). Against an expected 50/50 split, the chi-square statistic is over 2,000. This is a sample ratio mismatch. It usually means assignment, bucketing, or event logging is broken, and it undermines any comparison between arms until it's explained.

2. The 6% headline is mostly the incident. The raw lift is +5.7%. Once the 14 October double-count is removed, it's +2.2%. Engineering's belief that the incident "affected the new home page only" doesn't match the data. Control on that day was normal, and the spike appeared in both new and returning treatment users. Treatment was affected across the board.

3. The lift is fading. Corrected weekly lift on orders per user-day:

This looks like a novelty effect. Returning users, who are the people the feature is built for, show roughly zero lift overall once corrected (treatment 0.0940 vs control 0.0943).

4. The lift is concentrated in new users, which is suspicious. New users go from 0.0612 to 0.0681 per user-day (+11%). A reorder feature should help returning users most, since they have baskets to reorder. Combined with the SRM, this points toward a measurement artifact, possibly in how new users are counted, until shown otherwise.

5. AOV passes the guardrail. Treatment AOV averages $48.00 vs $48.25 for control, a gap of 0.5%. This is inside the 2% limit. The gap narrowed to 0.2% in week 4, which is the incident week, so I can't rule out the duplicate events affecting AOV. Revenue per user is only up about 0.3% in week 4, once the lower AOV is included.

What the test doesn't show

• Significance. Even corrected, a simple Poisson approximation gives z ≈ 2, which is borderline. Days aren't independent, so a proper clustered analysis would likely be weaker. The pre-registered rule requires a positive and significant lift at the end of the test. The end of the test shows about +0.5%, so the rule isn't met.
• Per-user outcomes. The file only has daily counts. The design specifies orders per user over the test. Without unique user IDs we can't measure that, and repeat users are counted once per active day.
• Long-run effect. Four weeks with a decaying effect doesn't tell us the steady state.

Decision

Don't ship Monday. The pre-registered rule isn't met. The SRM invalidates the comparison until explained. The incident contaminates the final week. The best estimate of the lift is near zero and falling.

Next steps

1. Resolve the SRM this week. Check bucketing logic, whether treatment users get a different login or app path that changes logging, and whether control users are dropped somewhere. Don't analyse further until this is explained.
2. Get the incident fixed in the data. Rebuild order counts from deduplicated events for 14 October and confirm which arms and user types were affected. Ask engineering to explain why control wasn't affected.
3. Pull user-level data. Rerun the primary metric on unique users over the test window, with the pre-registered definition.
4. Test the novelty explanation. Check whether returning users' use of the one-tap reorder changes over time. If it doesn't stick, the feature isn't working for the people it targets.
5. Decide the next test. If the SRM is fixed, consider a rerun with a fresh 50/50 split and a 4–6 week window. Predefine the analysis of new vs returning users, and set a stopping rule that accounts for novelty decay. If the SRM can't be fixed, we start again.
6. Report back on the revised numbers before any ship decision.

Readout to send

I would not recommend shipping Monday on this evidence. The raw daily aggregates don’t support Zoe’s “significant” claim, and a treatment-only order-logging incident materially inflates the apparent lift. We need a clean, user-level analysis before applying the pre-registered ship rule.

What the data shows

The daily file supports a pooled orders per recorded user-day calculation—not orders per unique randomized user over the full test:

• Control: 16,560 orders / 196,173 recorded users = 8.44%
• Treatment: 20,086 / 226,652 = 8.87%, or +5.0% relative

But treatment orders on 14 October are conspicuous: 296 for new users and 1,058 for returning users, roughly twice typical daily counts. The incident log says the treatment-only tracking release duplicated some order events for nine hours. As a rough sensitivity check only, halving those 14 October treatment counts subtracts about 677 orders; treatment then has 19,409 / 226,652 = 8.56%, or +1.4% relative versus control. This is not a valid final correction—the affected events need to be deduplicated from the underlying logs—but it shows how much the incident changes the result.

By user type, that rough adjustment gives about +7.6% for new users and −0.3% for returning users. That difference is worth investigating, but these aggregates don’t establish a reliable segment effect.

The pre-registered AOV guardrail looks descriptively okay: weekly treatment AOV is $47.90, $48.10, $47.80 and $48.20 versus control’s $48.20, $48.40, $48.10 and $48.30. The simple average is $48.00 versus $48.25 (about −0.5%, within the −2% limit). “Flat” is a little too strong; we also can’t assess uncertainty or reconstruct a correctly weighted overall AOV from this table alone.

What it doesn’t show

It does not establish that orders per randomized user rose significantly. The file reports daily user counts, not the unique assigned-user denominators or each user’s cumulative orders needed for the registered primary metric and its valid inference. Also, summed treatment user counts are 226,652 vs. 196,173 for control—about 15.5% higher despite a planned 50/50 user split. Those may be user-days rather than unique randomized users, so this is not proof of a randomization failure; it is a reason to audit assignment, exposure and denominator construction.

Even a simplistic event-count Poisson check on the rough deduplicated totals would not show significance (approximately p=.18); that is only a diagnostic, not a substitute for user-level analysis.

What to do next

1. Deduplicate treatment order events using event IDs and quantify the incident’s actual impact.
2. Rebuild the pre-registered orders-per-user metric from unique assigned users, with the planned statistical method; audit the 50/50 split and exposure.
3. Recompute AOV from the appropriate order-level data and check the guardrail.
4. Ship only if that clean analysis meets the decision rule. If the logs or assignment cannot support it, rerun a clean test rather than treating this result as a win.