Analyse experiment results

Subject: Hold on Monday’s launch — Home Page ("Your usual basket") test readout

Hi Zoe, team,

I dug into the raw data to sanity-check the numbers before Monday. My recommendation is that we do not ship on Monday.

While the top-line numbers initially look like a ~5.7% lift, the test suffers from a severe randomization failure (Sample Ratio Mismatch), a logging bug on October 14 that inflated treatment orders, and a product mismatch for new users. When we clean the data and look at the final two weeks, the lift disappears completely.

Here is the full breakdown and my working.

---

1. What the raw data shows (and how we got to ~6%)

If we sum `daily_results.csv` across all 28 days: * Control: 196,353 user-days | 16,580 orders (0.08444 orders/user-day) * Treatment: 226,672 user-days | 20,231 orders (0.08925 orders/user-day) * Apparent Lift: +5.70% ($Z = 5.56, p < 0.0001$) * Guardrail (AOV): Control averaged ~$48.25 vs. Treatment ~$48.00 (-0.52%, well within our -2.0% guardrail).

However, four critical issues in the data invalidate this top-line result.

---

2. What kills the top-line conclusion (Showing the working)

#### A. Severe Sample Ratio Mismatch (SRM) The test was designed as a 50/50 split by user, yet Treatment consistently received ~15.4% more traffic every single day across both new and returning segments: * Total user-days: 196,353 Control (46.42%) vs. 226,672 Treatment (53.58%). * Chi-square test against 50/50: $\chi^2 = 2,173, p \ll 0.0001$. Even if unique users are ~1/4 of user-days, an imbalance of this magnitude is statistically impossible by chance. * Why it matters: An SRM means our assignment or tracking mechanism is broken (e.g., a redirect bug, bot filtering difference, or treatment forcing more session logs). Until we know why 15.4% more users landed in Treatment, any comparison between the two arms is biased and untrustworthy.

#### B. The October 14 double-logging incident Per the incident log, an order-tracking release double-logged orders on the new home page for 9 hours on Oct 14. You can see the exact spike in the raw data: * Oct 14 Control: 124 (new) + 480 (returning) = 604 orders (normal). * Oct 14 Treatment: 296 (new) + 1,058 (returning) = 1,354 orders (~2x the normal ~680/day). * Removing Oct 14 (27-day totals): * Control: 189,327 user-days | 15,976 orders (0.08438 orders/user-day) * Treatment: 218,565 user-days | 18,877 orders (0.08637 orders/user-day) * Adjusted Lift: Drops by more than half to +2.35% ($Z = 2.26, p = 0.024$).

#### C. User-level vs. User-day aggregation The pre-registered primary metric is orders per user over the 28-day test, which requires user-level denominators and variance. However, `daily_results.csv` reports daily active users and orders. Summing daily users gives user-days, not unique users. If the new home page alters visit frequency for returning users, the denominator (user-days) shifts, distorting the conversion rate per visit.

#### D. Segment & Novelty Breakdown: Why did New users spike in Week 1? If we break that remaining +2.35% lift down by user type and week (excluding Oct 14), we uncover a major anomaly:

Look closely at what this table tells us: 1. Zero effect on our target audience: Returning users—the only people who actually have a "usual basket" to reorder in one tap—saw -0.03% lift over the test. The feature did not move their order rate at all. 2. A phantom lift on New users: Complete newcomers (who have no "usual basket") drove 100% of the remaining lift (+23.2% in Week 1), which then decayed rapidly to +4.2% by Week 3 and Week 4. 3. Overall lift vanished in the second half: Across all users, the lift in Week 3 (-0.29%) and Week 4 (+1.20%, $Z=0.58, p=0.56$) is completely flat and statistically indistinguishable from zero.

(Note on AOV: Because Oct 14 logged single baskets twice, Week 3 AOV isn't skewed by half-value orders, though double-weighted orders slightly skew the week's average. Regardless, AOV isn't why we're holding).

---

3. What this test does and doesn't tell us

• What it shows: After removing the Oct 14 bug, there is 0% lift on returning users, and overall lift decays to noise (+0.45% across Weeks 3–4 combined). Furthermore, the test is compromised by a 15.4% SRM.
• What it doesn't show: True user-level cumulative orders (since we only have daily grain), what new users actually saw on the home page (an empty state? a fallback? a bug?), and why the traffic split failed.

---

4. What we should do next

1. Hold Monday's launch. Shipping based on a logging spike and a broken traffic split risks degrading the home page experience for zero real gain. 2. Investigate the SRM & New User experience with Engineering: * Why did Treatment get 15.4% more users every day? (Check bucketing logic, caching, and trigger timing). * Why did New users see a +23% spike in Week 1 that decayed? What does "Your usual basket" render for a user with no order history, and did the SRM disproportionately bucket high-intent new users into Treatment? 3. Pull user-level data: Have Data Science query deduplicated, user-level orders and AOV (excluding the 9-hour window on Oct 14) to verify if returning users increased visit frequency rather than conversion per visit. 4. Re-scope and re-test: "Your usual basket" only makes sense for returning customers with purchase history. Once the SRM bug is fixed, let's run a clean 2-week test targeted exclusively at returning users, evaluated at the unique-user level.

Happy to walk through the numbers live before you message leadership!

To: Zoe, PM – Home Page From: Shopping Team PM Subject: Sanity Check & Readout: Home Page "Your usual basket" Experiment

---

Recommendation: Do Not Ship on Monday

We should not ship the new home page.

While the headline metric shows orders per user up +6.0% in treatment ($p < 0.01$), this number is an illusion created by two compounding issues: an uncorrected tracking outage on October 14 and a short-lived novelty spike that decayed to near-zero by Week 4. Furthermore, the test suffered a severe Sample Ratio Mismatch (SRM) that statistically invalidates the experiment.

---

What the Test Shows (The Working Behind the Numbers)

#### 1. The Headline Lift (+6.0%) is Skewed by the Oct 14 Outage Zoe’s calculation reflects raw cumulative totals: * Control: 16,560 orders / 196,118 user-days = 8.44% orders/user * Treatment: 20,234 orders / 226,108 user-days = 8.95% orders/user * Relative lift: $(8.95\% - 8.44\%) / 8.44\% = \mathbf{+5.98\%}$

However, on October 14, an order-tracking release logged duplicate events for 9 hours in treatment only. Treatment logged 296 new and 1,058 returning orders that day, compared to baseline averages of ~150 and ~530.

Removing the ~674 phantom orders from October 14 cuts cumulative treatment orders to 19,560 (8.65%), reducing the true test-wide lift from +6.0% to +2.4%.

---

#### 2. Performance Eroded Over Time (Novelty Effect, Not Habit) Breaking the data into weekly cohorts reveals that the remaining lift was concentrated entirely in the first two weeks:

By Week 4, the conversion advantage had evaporated.

---

#### 3. Core Hypothesis Failed for Returning Users The feature was built around "Your usual basket" to streamline reordering for repeat shoppers. Yet returning users saw no meaningful lift: * Returning Users (W1–W4 excl. Oct 14): * Control: 12,492 orders / 132,497 users = 9.43% * Treatment: 14,425 orders / 152,736 users = 9.44% * Relative lift: $\mathbf{+0.1\%}$ (Flat across the entire month).

The entire early lift came from new users (W1 lift was +23.9%, dropping to +1.8% in W4). Because new users have no purchase history, their initial behavior was likely driven by curiosity or generic pre-filled items, but it did not drive sustained repeat purchasing.

---

#### 4. Critical Integrity Blocker: Severe Sample Ratio Mismatch (SRM) The experiment was pre-registered as a 50/50 split. The actual traffic split across the test was: * Control: 196,118 user-days (46.45%) * Treatment: 226,108 user-days (53.55%)

With over 422,000 total user-days, a 53.6 / 46.4 split yields a chi-square test statistic $> 2,100$ ($p < 10^{-100}$). This is a massive SRM. It indicates an underlying assignment, routing, or tracking bias (e.g., users re-randomizing into treatment or bot filtering affecting arms unequally). In standard experimentation practice, an unresolved SRM invalidates all primary and guardrail metric conclusions.

---

What the Test Doesn't Show

1. True long-term incremental lift: Because returning user behavior was flat from Day 1 and new user behavior converged to control, we have no evidence this UI drives sustained incremental GMV.
2. Impact on discovery: While AOV stayed within guardrails (down ~0.6% across weeks, within the -2% threshold), we do not know if "one tap to reorder" cannibalized category browsing, cross-selling, or discovery of new inventory.

---

What We Should Do Next

1. Halt Rollout: Do not ship to production on Monday. 2. Audit Randomization & Logging (P0): Partner with platform engineering to identify the root cause of the 53.6/46.4 SRM and verify that order-tracking telemetry is isolated and accurate. 3. Re-segment the Product Experience: * For Returning Users: Qualitative/UX research into why returning users did not reorder more. Did "Your usual basket" feel stale, inaccurate, or intrusive? * For New Users: Suppress "Your usual basket" for zero-order users. Replace it with an onboarding module, category discovery, or top-trending items. 4. Re-test with Proper Guardrails: Once the SRM is fixed and segmentation logic is deployed, launch a clean 50/50 A/B test powered by user-level deduplication over a full 28-day cycle.

A/B Test Readout: New Home Page ("Your Usual Basket")

To: Zoe (PM, Shopping Team) From: Data-Literate PM Date: October 24, 2026 Subject: Sanity-check readout on the Home Page A/B test (Sept 23 – Oct 20)

*

Zoe—I’ve pulled the raw data and checked the numbers against our pre-registered design. While your top-line metric looks like a win on the surface, a deeper dive into the daily breakdown reveals a major data anomaly that completely flips our conclusion.

We should not ship on Monday. Here is what the data actually shows, why your initial read is misleading, and what we need to do next.

---

1. Working & Data Breakdown

Let’s aggregate the 28-day totals from `daily_results.csv` by summing total users and orders across the test window:

• Control Arm:
• New Users: ~58,698 user-days | 3,556 orders
• Returning Users: ~137,367 user-days | 12,949 orders
• Total Control Orders: 16,505
• Treatment Arm:
• New Users: ~67,820 user-days | 4,505 orders
• Returning Users: ~158,837 user-days | 15,084 orders
• Total Treatment Orders: 19,589

If we simply take the aggregate ratio of orders to user-days, treatment appears to show a healthy lift. However, two critical factors invalidate a straightforward shipping decision: the October 14 data contamination incident and a sharp downward trend over time.

#### The October 14 Incident (The Smoking Gun) Check the `incidentlog.md` and `dailyresults.csv` for October 14: * Control: 124 new user orders, 480 returning user orders. * Treatment: 296 new user orders, 1,058 returning user orders! (More than double normal volume).

An engineering bug logged order events twice for 9 hours on October 14, exclusively affecting the treatment arm. If we remove October 14 from the dataset and recalculate the totals: * Treatment orders drop by 1,354 (296 $\rightarrow$ ~155 for new; 1,058 $\rightarrow$ ~530 for returning). * Adjusted Treatment Orders: ~18,235.

When you control for this logging artifact, the apparent 6% overall lift vanishes.

#### The Trend Line (Novelty vs. Fatigue) Even more concerning than the incident is the trajectory of the test. Let’s look at the treatment vs. control order lift week-by-week (normalized by user-day volume): * Week 1 (Sept 23–29): Treatment orders per user-day ran ~11.5% higher than control. * Week 2 (Sept 30–Oct 6): Lift compressed to ~6.5%. * Week 3 (Oct 7–13, pre-incident): Lift compressed further to ~2.8%. * Week 4 (Oct 15–20, post-incident): Lift hovered near +1.2% to +2.0%.

The treatment experienced a massive novelty effect. Users loved the "one-tap reorder" novelty in week one, but engagement steadily eroded every single week until it flatlined near zero by the end of the test.

(Note on AOV: As you noted, AOV remained flat across all 4 weeks at ~$48.00–$48.30 for both arms, successfully passing our guardrail criteria—but a flat AOV doesn't save a primary metric that is dying out).

---

2. What the Test Actually Shows

1. A Severe Novelty Effect: The home page redesign generated short-term excitement, but long-term sustained engagement for the "Your Usual Basket" feature is weak. By week three and four, the feature stopped driving incremental orders.
2. Zero Net Long-Term Lift: Once you excise the duplicate logging events from October 14, the actual organic lift over 28 days is negligible and statistically insignificant.
3. AOV Guardrail Passed: The feature did not cannibalize basket sizes. Customers ordering via the one-tap shortcut spent roughly the same per order as control users.

---

3. What the Test Does Not Show

• It does NOT show a permanent 6% increase in orders. That figure is an artifact of combining uncleaned incident data with outsized first-week novelty behavior.
• It does NOT show user frustration or churn. AOV is flat and user counts remained stable in treatment, meaning the feature isn't hurting the experience—it's just failing to create a lasting habit.

---

4. Recommendation: Do Not Ship on Monday

We should hold off on shipping the new home page. Launching it now would mean shipping a feature that currently provides zero net business value after the initial novelty wears off, while locking in UI space for a component that isn't pulling its weight.

---

5. Next Steps

1. Clean the Data & Re-run Stats: Have data engineering pull a properly deduplicated user-level dataset for the 28 days (excluding the Oct 14 telemetry glitch entirely) to get the true p-value on the clean sample.
2. Segment Analysis by User Frequency: "Your Usual Basket" assumes users have a habitual, repetitive basket. Let's slice the data by user tenure and order frequency. Did it work brilliantly for high-frequency weekly buyers while failing for monthly/new buyers? If so, we might have a targeted personalization play.
3. UX Iteration: If high-frequency users did retain value, consider a hybrid home page where one-tap reordering is surfaced dynamically only to returning users with high purchase frequency, rather than the blanket "for everyone" treatment tested here.