Analyse experiment results

A/B Test Readout: New Home Page ("Your Usual Basket")

To: Zoe (PM, Shopping Team) From: Data-Literate PM Date: October 24, 2026 Subject: Sanity-check readout on the Home Page A/B test (Sept 23 – Oct 20)

*

Zoe—I’ve pulled the raw data and checked the numbers against our pre-registered design. While your top-line metric looks like a win on the surface, a deeper dive into the daily breakdown reveals a major data anomaly that completely flips our conclusion.

We should not ship on Monday. Here is what the data actually shows, why your initial read is misleading, and what we need to do next.

---

1. Working & Data Breakdown

Let’s aggregate the 28-day totals from `daily_results.csv` by summing total users and orders across the test window:

• Control Arm:
• New Users: ~58,698 user-days | 3,556 orders
• Returning Users: ~137,367 user-days | 12,949 orders
• Total Control Orders: 16,505
• Treatment Arm:
• New Users: ~67,820 user-days | 4,505 orders
• Returning Users: ~158,837 user-days | 15,084 orders
• Total Treatment Orders: 19,589

If we simply take the aggregate ratio of orders to user-days, treatment appears to show a healthy lift. However, two critical factors invalidate a straightforward shipping decision: the October 14 data contamination incident and a sharp downward trend over time.

#### The October 14 Incident (The Smoking Gun) Check the `incidentlog.md` and `dailyresults.csv` for October 14: * Control: 124 new user orders, 480 returning user orders. * Treatment: 296 new user orders, 1,058 returning user orders! (More than double normal volume).

An engineering bug logged order events twice for 9 hours on October 14, exclusively affecting the treatment arm. If we remove October 14 from the dataset and recalculate the totals: * Treatment orders drop by 1,354 (296 $\rightarrow$ ~155 for new; 1,058 $\rightarrow$ ~530 for returning). * Adjusted Treatment Orders: ~18,235.

When you control for this logging artifact, the apparent 6% overall lift vanishes.

#### The Trend Line (Novelty vs. Fatigue) Even more concerning than the incident is the trajectory of the test. Let’s look at the treatment vs. control order lift week-by-week (normalized by user-day volume): * Week 1 (Sept 23–29): Treatment orders per user-day ran ~11.5% higher than control. * Week 2 (Sept 30–Oct 6): Lift compressed to ~6.5%. * Week 3 (Oct 7–13, pre-incident): Lift compressed further to ~2.8%. * Week 4 (Oct 15–20, post-incident): Lift hovered near +1.2% to +2.0%.

The treatment experienced a massive novelty effect. Users loved the "one-tap reorder" novelty in week one, but engagement steadily eroded every single week until it flatlined near zero by the end of the test.

(Note on AOV: As you noted, AOV remained flat across all 4 weeks at ~$48.00–$48.30 for both arms, successfully passing our guardrail criteria—but a flat AOV doesn't save a primary metric that is dying out).

---

2. What the Test Actually Shows

1. A Severe Novelty Effect: The home page redesign generated short-term excitement, but long-term sustained engagement for the "Your Usual Basket" feature is weak. By week three and four, the feature stopped driving incremental orders.
2. Zero Net Long-Term Lift: Once you excise the duplicate logging events from October 14, the actual organic lift over 28 days is negligible and statistically insignificant.
3. AOV Guardrail Passed: The feature did not cannibalize basket sizes. Customers ordering via the one-tap shortcut spent roughly the same per order as control users.

---

3. What the Test Does Not Show

• It does NOT show a permanent 6% increase in orders. That figure is an artifact of combining uncleaned incident data with outsized first-week novelty behavior.
• It does NOT show user frustration or churn. AOV is flat and user counts remained stable in treatment, meaning the feature isn't hurting the experience—it's just failing to create a lasting habit.

---

4. Recommendation: Do Not Ship on Monday

We should hold off on shipping the new home page. Launching it now would mean shipping a feature that currently provides zero net business value after the initial novelty wears off, while locking in UI space for a component that isn't pulling its weight.

---

5. Next Steps

1. Clean the Data & Re-run Stats: Have data engineering pull a properly deduplicated user-level dataset for the 28 days (excluding the Oct 14 telemetry glitch entirely) to get the true p-value on the clean sample.
2. Segment Analysis by User Frequency: "Your Usual Basket" assumes users have a habitual, repetitive basket. Let's slice the data by user tenure and order frequency. Did it work brilliantly for high-frequency weekly buyers while failing for monthly/new buyers? If so, we might have a targeted personalization play.
3. UX Iteration: If high-frequency users did retain value, consider a hybrid home page where one-tap reordering is surfaced dynamically only to returning users with high purchase frequency, rather than the blanket "for everyone" treatment tested here.

Readout: don’t ship the new home page on Monday. The logged data reproduce Zoe’s roughly 6% lift, but they do not yet establish a valid, durable improvement in the pre-registered metric. We have a treatment-specific measurement incident, an unexplained arm imbalance, and a lift that weakens substantially over time.

1. What the headline shows—and how I calculated it

I summed orders and reported daily users within each arm, then calculated:

\[ \text{Lift}=\frac{\text{treatment orders}/\text{treatment users}} {\text{control orders}/\text{control users}}-1 \]

Thus, the logged lift is 5.75%, consistent with “about 6%.”

However, summed daily users are not necessarily distinct randomized users. If users recur across days, this is orders per user-day, not the pre-registered orders per user over the test. If these are active or exposed users, the denominator may itself be affected by treatment. We need the denominator definition and assignment-level data before accepting the primary result.

Nor can these aggregates validate Zoe’s significance claim. For illustration, treating orders as independent Poisson counts gives:

\[ SE(\log RR)=\sqrt{1/20{,}234+1/16{,}560}=0.01048 \]

That produces a nominal 95% lift interval of approximately +3.6% to +8.0%. But it ignores repeated-user dependence, potentially treatment-dependent denominators, and corrupted order counts. It is not a valid substitute for inference at the randomized-user level.

2. The tracking incident materially inflates the result

On 14 October, treatment logged 1,354 orders, versus roughly 700 on surrounding days. Control logged 604. This aligns with the incident report, although we cannot identify the exact duplicate count from these aggregates.

As a sensitivity check, removing that entire day from both arms, including its denominators, gives:

Lift becomes 2.41%, less than half the headline. This does not prove there is no effect; equally, dropping a day is not a repaired primary analysis. We should deduplicate against authoritative order IDs, not halve the day’s orders or discard inconvenient observations.

3. The remaining signal is concentrated early and among new users

Using the same rate calculation within each week and segment:

Across all non-incident days, new-user lift is approximately +11.5%, while returning-user lift is −0.1%.

This is not evidence of a broad reorder benefit for everyone. The pattern is consistent with novelty, changing cohorts, or another time-dependent effect—not proof of any one explanation. These are exploratory comparisons, not statistically established segment differences or grounds to ship only to new users. Calendar-week results also do not measure persistence since each user’s first exposure.

4. Allocation and AOV require qualification

Treatment accounts for 53.6% of reported user-days, despite the planned 50/50 user split; the imbalance appears throughout the test. Repeated daily users prevent a valid assignment-level sample-ratio test here. It could reflect assignment problems, logging, filtering, or treatment-driven activity. We must reconcile it against unique randomized users, rather than assume either harmless imbalance or broken randomization.

Weekly AOV is down approximately 0.2–0.6%, not literally flat. Those point estimates are within the −2% guardrail, but weekly averages provide neither uncertainty nor assurance that duplication did not distort order weighting.

Decision and next steps

The pre-registered shipping rule should stand—but we have not established that it was met with valid data.

1. Reconcile assignment, eligibility, exposure, and user counts; investigate the imbalance.
2. Rebuild orders and revenue from authoritative transactions; verify the incident’s scope in both arms.
3. Recompute 28-day orders per randomized user, including zero-order users, with user-level uncertainty; recompute AOV and its uncertainty.
4. Examine pre-treatment segments and exposure-age cohorts to investigate persistence without treating exploratory findings as confirmatory.
5. If the original test cannot support a trustworthy decision, run a repaired, newly pre-registered test. Do not silently extend this fixed-horizon test until significance returns.

Readout: Home page test ("Your usual basket")

To: Zoe Re: Monday ship recommendation

Recommendation: don't ship Monday. The readout overstates the result. The test has a sample ratio problem, a logging incident inside the test window, and a lift that is fading. Next steps are at the end.

Working

I summed the daily CSV across all 28 days. The file gives daily active users and orders, so each "user" is a user-day. That matters for the first finding below.

Correction: on 14 October, treatment orders were about double the surrounding days (new: 296 vs ~150; returning: 1,058 vs ~530). I replaced that day with half the logged value. Using the average of the adjacent days gives nearly the same answer (+2.2%).

What the test shows

1. The split isn't 50/50. Treatment received 53.6% of user-days, and the ratio holds every day (roughly 2,400 treatment vs 2,090 control among new users, and 5,700 vs 4,900 among returning users). Against an expected 50/50 split, the chi-square statistic is over 2,000. This is a sample ratio mismatch. It usually means assignment, bucketing, or event logging is broken, and it undermines any comparison between arms until it's explained.

2. The 6% headline is mostly the incident. The raw lift is +5.7%. Once the 14 October double-count is removed, it's +2.2%. Engineering's belief that the incident "affected the new home page only" doesn't match the data. Control on that day was normal, and the spike appeared in both new and returning treatment users. Treatment was affected across the board.

3. The lift is fading. Corrected weekly lift on orders per user-day:

This looks like a novelty effect. Returning users, who are the people the feature is built for, show roughly zero lift overall once corrected (treatment 0.0940 vs control 0.0943).

4. The lift is concentrated in new users, which is suspicious. New users go from 0.0612 to 0.0681 per user-day (+11%). A reorder feature should help returning users most, since they have baskets to reorder. Combined with the SRM, this points toward a measurement artifact, possibly in how new users are counted, until shown otherwise.

5. AOV passes the guardrail. Treatment AOV averages $48.00 vs $48.25 for control, a gap of 0.5%. This is inside the 2% limit. The gap narrowed to 0.2% in week 4, which is the incident week, so I can't rule out the duplicate events affecting AOV. Revenue per user is only up about 0.3% in week 4, once the lower AOV is included.

What the test doesn't show

• Significance. Even corrected, a simple Poisson approximation gives z ≈ 2, which is borderline. Days aren't independent, so a proper clustered analysis would likely be weaker. The pre-registered rule requires a positive and significant lift at the end of the test. The end of the test shows about +0.5%, so the rule isn't met.
• Per-user outcomes. The file only has daily counts. The design specifies orders per user over the test. Without unique user IDs we can't measure that, and repeat users are counted once per active day.
• Long-run effect. Four weeks with a decaying effect doesn't tell us the steady state.

Decision

Don't ship Monday. The pre-registered rule isn't met. The SRM invalidates the comparison until explained. The incident contaminates the final week. The best estimate of the lift is near zero and falling.

Next steps

1. Resolve the SRM this week. Check bucketing logic, whether treatment users get a different login or app path that changes logging, and whether control users are dropped somewhere. Don't analyse further until this is explained.
2. Get the incident fixed in the data. Rebuild order counts from deduplicated events for 14 October and confirm which arms and user types were affected. Ask engineering to explain why control wasn't affected.
3. Pull user-level data. Rerun the primary metric on unique users over the test window, with the pre-registered definition.
4. Test the novelty explanation. Check whether returning users' use of the one-tap reorder changes over time. If it doesn't stick, the feature isn't working for the people it targets.
5. Decide the next test. If the SRM is fixed, consider a rerun with a fresh 50/50 split and a 4–6 week window. Predefine the analysis of new vs returning users, and set a stopping rule that accounts for novelty decay. If the SRM can't be fixed, we start again.
6. Report back on the revised numbers before any ship decision.