Analyse experiment results

Readout to send

I would not recommend shipping Monday on this evidence. The raw daily aggregates don’t support Zoe’s “significant” claim, and a treatment-only order-logging incident materially inflates the apparent lift. We need a clean, user-level analysis before applying the pre-registered ship rule.

What the data shows

The daily file supports a pooled orders per recorded user-day calculation—not orders per unique randomized user over the full test:

• Control: 16,560 orders / 196,173 recorded users = 8.44%
• Treatment: 20,086 / 226,652 = 8.87%, or +5.0% relative

But treatment orders on 14 October are conspicuous: 296 for new users and 1,058 for returning users, roughly twice typical daily counts. The incident log says the treatment-only tracking release duplicated some order events for nine hours. As a rough sensitivity check only, halving those 14 October treatment counts subtracts about 677 orders; treatment then has 19,409 / 226,652 = 8.56%, or +1.4% relative versus control. This is not a valid final correction—the affected events need to be deduplicated from the underlying logs—but it shows how much the incident changes the result.

By user type, that rough adjustment gives about +7.6% for new users and −0.3% for returning users. That difference is worth investigating, but these aggregates don’t establish a reliable segment effect.

The pre-registered AOV guardrail looks descriptively okay: weekly treatment AOV is $47.90, $48.10, $47.80 and $48.20 versus control’s $48.20, $48.40, $48.10 and $48.30. The simple average is $48.00 versus $48.25 (about −0.5%, within the −2% limit). “Flat” is a little too strong; we also can’t assess uncertainty or reconstruct a correctly weighted overall AOV from this table alone.

What it doesn’t show

It does not establish that orders per randomized user rose significantly. The file reports daily user counts, not the unique assigned-user denominators or each user’s cumulative orders needed for the registered primary metric and its valid inference. Also, summed treatment user counts are 226,652 vs. 196,173 for control—about 15.5% higher despite a planned 50/50 user split. Those may be user-days rather than unique randomized users, so this is not proof of a randomization failure; it is a reason to audit assignment, exposure and denominator construction.

Even a simplistic event-count Poisson check on the rough deduplicated totals would not show significance (approximately p=.18); that is only a diagnostic, not a substitute for user-level analysis.

What to do next

1. Deduplicate treatment order events using event IDs and quantify the incident’s actual impact.
2. Rebuild the pre-registered orders-per-user metric from unique assigned users, with the planned statistical method; audit the 50/50 split and exposure.
3. Recompute AOV from the appropriate order-level data and check the guardrail.
4. Ship only if that clean analysis meets the decision rule. If the logs or assignment cannot support it, rerun a clean test rather than treating this result as a win.

Readout: don’t ship the new home page on Monday. The logged data reproduce Zoe’s roughly 6% lift, but they do not yet establish a valid, durable improvement in the pre-registered metric. We have a treatment-specific measurement incident, an unexplained arm imbalance, and a lift that weakens substantially over time.

1. What the headline shows—and how I calculated it

I summed orders and reported daily users within each arm, then calculated:

\[ \text{Lift}=\frac{\text{treatment orders}/\text{treatment users}} {\text{control orders}/\text{control users}}-1 \]

Thus, the logged lift is 5.75%, consistent with “about 6%.”

However, summed daily users are not necessarily distinct randomized users. If users recur across days, this is orders per user-day, not the pre-registered orders per user over the test. If these are active or exposed users, the denominator may itself be affected by treatment. We need the denominator definition and assignment-level data before accepting the primary result.

Nor can these aggregates validate Zoe’s significance claim. For illustration, treating orders as independent Poisson counts gives:

\[ SE(\log RR)=\sqrt{1/20{,}234+1/16{,}560}=0.01048 \]

That produces a nominal 95% lift interval of approximately +3.6% to +8.0%. But it ignores repeated-user dependence, potentially treatment-dependent denominators, and corrupted order counts. It is not a valid substitute for inference at the randomized-user level.

2. The tracking incident materially inflates the result

On 14 October, treatment logged 1,354 orders, versus roughly 700 on surrounding days. Control logged 604. This aligns with the incident report, although we cannot identify the exact duplicate count from these aggregates.

As a sensitivity check, removing that entire day from both arms, including its denominators, gives:

Lift becomes 2.41%, less than half the headline. This does not prove there is no effect; equally, dropping a day is not a repaired primary analysis. We should deduplicate against authoritative order IDs, not halve the day’s orders or discard inconvenient observations.

3. The remaining signal is concentrated early and among new users

Using the same rate calculation within each week and segment:

Across all non-incident days, new-user lift is approximately +11.5%, while returning-user lift is −0.1%.

This is not evidence of a broad reorder benefit for everyone. The pattern is consistent with novelty, changing cohorts, or another time-dependent effect—not proof of any one explanation. These are exploratory comparisons, not statistically established segment differences or grounds to ship only to new users. Calendar-week results also do not measure persistence since each user’s first exposure.

4. Allocation and AOV require qualification

Treatment accounts for 53.6% of reported user-days, despite the planned 50/50 user split; the imbalance appears throughout the test. Repeated daily users prevent a valid assignment-level sample-ratio test here. It could reflect assignment problems, logging, filtering, or treatment-driven activity. We must reconcile it against unique randomized users, rather than assume either harmless imbalance or broken randomization.

Weekly AOV is down approximately 0.2–0.6%, not literally flat. Those point estimates are within the −2% guardrail, but weekly averages provide neither uncertainty nor assurance that duplication did not distort order weighting.

Decision and next steps

The pre-registered shipping rule should stand—but we have not established that it was met with valid data.

1. Reconcile assignment, eligibility, exposure, and user counts; investigate the imbalance.
2. Rebuild orders and revenue from authoritative transactions; verify the incident’s scope in both arms.
3. Recompute 28-day orders per randomized user, including zero-order users, with user-level uncertainty; recompute AOV and its uncertainty.
4. Examine pre-treatment segments and exposure-age cohorts to investigate persistence without treating exploratory findings as confirmatory.
5. If the original test cannot support a trustworthy decision, run a repaired, newly pre-registered test. Do not silently extend this fixed-horizon test until significance returns.

To: Zoe, PM – Home Page From: Shopping Team PM Subject: Sanity Check & Readout: Home Page "Your usual basket" Experiment

---

Recommendation: Do Not Ship on Monday

We should not ship the new home page.

While the headline metric shows orders per user up +6.0% in treatment ($p < 0.01$), this number is an illusion created by two compounding issues: an uncorrected tracking outage on October 14 and a short-lived novelty spike that decayed to near-zero by Week 4. Furthermore, the test suffered a severe Sample Ratio Mismatch (SRM) that statistically invalidates the experiment.

---

What the Test Shows (The Working Behind the Numbers)

#### 1. The Headline Lift (+6.0%) is Skewed by the Oct 14 Outage Zoe’s calculation reflects raw cumulative totals: * Control: 16,560 orders / 196,118 user-days = 8.44% orders/user * Treatment: 20,234 orders / 226,108 user-days = 8.95% orders/user * Relative lift: $(8.95\% - 8.44\%) / 8.44\% = \mathbf{+5.98\%}$

However, on October 14, an order-tracking release logged duplicate events for 9 hours in treatment only. Treatment logged 296 new and 1,058 returning orders that day, compared to baseline averages of ~150 and ~530.

Removing the ~674 phantom orders from October 14 cuts cumulative treatment orders to 19,560 (8.65%), reducing the true test-wide lift from +6.0% to +2.4%.

---

#### 2. Performance Eroded Over Time (Novelty Effect, Not Habit) Breaking the data into weekly cohorts reveals that the remaining lift was concentrated entirely in the first two weeks:

By Week 4, the conversion advantage had evaporated.

---

#### 3. Core Hypothesis Failed for Returning Users The feature was built around "Your usual basket" to streamline reordering for repeat shoppers. Yet returning users saw no meaningful lift: * Returning Users (W1–W4 excl. Oct 14): * Control: 12,492 orders / 132,497 users = 9.43% * Treatment: 14,425 orders / 152,736 users = 9.44% * Relative lift: $\mathbf{+0.1\%}$ (Flat across the entire month).

The entire early lift came from new users (W1 lift was +23.9%, dropping to +1.8% in W4). Because new users have no purchase history, their initial behavior was likely driven by curiosity or generic pre-filled items, but it did not drive sustained repeat purchasing.

---

#### 4. Critical Integrity Blocker: Severe Sample Ratio Mismatch (SRM) The experiment was pre-registered as a 50/50 split. The actual traffic split across the test was: * Control: 196,118 user-days (46.45%) * Treatment: 226,108 user-days (53.55%)

With over 422,000 total user-days, a 53.6 / 46.4 split yields a chi-square test statistic $> 2,100$ ($p < 10^{-100}$). This is a massive SRM. It indicates an underlying assignment, routing, or tracking bias (e.g., users re-randomizing into treatment or bot filtering affecting arms unequally). In standard experimentation practice, an unresolved SRM invalidates all primary and guardrail metric conclusions.

---

What the Test Doesn't Show

1. True long-term incremental lift: Because returning user behavior was flat from Day 1 and new user behavior converged to control, we have no evidence this UI drives sustained incremental GMV.
2. Impact on discovery: While AOV stayed within guardrails (down ~0.6% across weeks, within the -2% threshold), we do not know if "one tap to reorder" cannibalized category browsing, cross-selling, or discovery of new inventory.

---

What We Should Do Next

1. Halt Rollout: Do not ship to production on Monday. 2. Audit Randomization & Logging (P0): Partner with platform engineering to identify the root cause of the 53.6/46.4 SRM and verify that order-tracking telemetry is isolated and accurate. 3. Re-segment the Product Experience: * For Returning Users: Qualitative/UX research into why returning users did not reorder more. Did "Your usual basket" feel stale, inaccurate, or intrusive? * For New Users: Suppress "Your usual basket" for zero-order users. Replace it with an onboarding module, category discovery, or top-trending items. 4. Re-test with Proper Guardrails: Once the SRM is fixed and segmentation logic is deployed, launch a clean 50/50 A/B test powered by user-level deduplication over a full 28-day cycle.