Analyse experiment results

Recommendation: don’t ship on Monday. The reported ~6% lift is reproducible, but it includes a treatment-specific tracking incident. The supplied aggregates also cannot establish significance for the pre-registered, user-level metric. We have a promising signal—not a reliable launch decision.

What the raw data shows

I summed orders and daily user counts across the 28 days:

The calculation is:

\[ \text{Lift}=\frac{20,234/226,652}{16,560/196,373}-1=5.86\%. \]

So Zoe’s arithmetic is right for this aggregate ratio. However, summed daily users are not necessarily unique randomized users. If people appear on multiple days, this is orders per user-day, not the pre-registered orders per user over the test.

The tracking incident materially changes the result

On 14 October, treatment recorded 1,354 orders versus 604 in control, with 8,107 versus 7,026 daily users. Treatment’s order rate was almost twice its normal level. That is consistent with the documented duplicate-event incident.

As a sensitivity check only, removing that entire day from both arms gives:

\[ \text{Lift excluding 14 October} =\frac{18,880/218,545}{15,956/189,347}-1 =2.52\%. \]

The remaining signal is positive, but much smaller. Dropping a whole day is not a valid substitute for recovering actual orders: the incident lasted nine hours, and that day also contains legitimate transactions.

The effect is not stable or broad-based

Using the same aggregate calculation:

Excluding the incident day, the full-period lift is +11.9% for new users and −0.14% for returning users. New-user weekly lifts decline from approximately 24.0% to 15.0%, 4.7%, and 2.5%.

That suggests the remaining aggregate gain is concentrated among new users and may be fading. It does not prove novelty: calendar effects, changing cohorts, or instrumentation could also explain the pattern. Nor does a positive new-user estimate alone establish a statistically reliable difference between segments.

What we cannot conclude yet

“Significant” is not established by these files. For the intended user-randomized analysis, we need unique assigned users—including users with zero orders—and each user’s deduplicated order count. Daily aggregates do not provide the user-level variance or account for repeated appearances. Treating user-days as independent could overstate precision.

The denominator needs investigation. Treatment has 15.4% more summed daily users than control, despite the planned 50/50 assignment, with an imbalance throughout the test. This is a warning, not proof of assignment failure: if these are active users, treatment could change activity. Check assignment counts separately from exposure/activity counts. An activity-conditioned denominator can change the estimand and introduce bias.

AOV is slightly down, not flat. Weekly declines range from approximately 0.2% to 0.6%, all within the 2% guardrail. Weighting the reported weekly averages by logged order counts gives approximately $48.25 control versus $48.01 treatment, or −0.51%. That is descriptively reassuring, but duplicate orders may distort the weights or AOV calculation. These summaries also lack uncertainty estimates.

What we should do next

1. Reconcile against canonical transactions. Deduplicate using order IDs and validate the incident’s actual scope rather than assuming only treatment was affected.
2. Audit randomization and denominators. Check unique assignments, eligibility, exposure logging, missing users, and sample-ratio mismatch.
3. Recompute the pre-registered analysis: intention-to-treat orders per unique assigned user over the full 28 days, with user-level confidence intervals; recompute AOV from valid orders and revenue.
4. Apply the original decision rule to that repaired result. If lift is positive and significant and the AOV guardrail passes, shipping becomes defensible. If trustworthy recovery is impossible, rerun with fixed instrumentation and a pre-specified duration and analysis.
5. Investigate durability and targeting separately. Pre-specify new/returning-user interactions and time-since-exposure analyses in follow-up work. Don’t use this post-hoc segment result to justify a targeted launch.

Bottom line: pause the launch, repair the measurement, and decide on the registered metric—not the contaminated headline.

Readout: Home page test ("Your usual basket")

To: Zoe Re: Monday ship recommendation

Recommendation: don't ship Monday. The readout overstates the result. The test has a sample ratio problem, a logging incident inside the test window, and a lift that is fading. Next steps are at the end.

Working

I summed the daily CSV across all 28 days. The file gives daily active users and orders, so each "user" is a user-day. That matters for the first finding below.

Correction: on 14 October, treatment orders were about double the surrounding days (new: 296 vs ~150; returning: 1,058 vs ~530). I replaced that day with half the logged value. Using the average of the adjacent days gives nearly the same answer (+2.2%).

What the test shows

1. The split isn't 50/50. Treatment received 53.6% of user-days, and the ratio holds every day (roughly 2,400 treatment vs 2,090 control among new users, and 5,700 vs 4,900 among returning users). Against an expected 50/50 split, the chi-square statistic is over 2,000. This is a sample ratio mismatch. It usually means assignment, bucketing, or event logging is broken, and it undermines any comparison between arms until it's explained.

2. The 6% headline is mostly the incident. The raw lift is +5.7%. Once the 14 October double-count is removed, it's +2.2%. Engineering's belief that the incident "affected the new home page only" doesn't match the data. Control on that day was normal, and the spike appeared in both new and returning treatment users. Treatment was affected across the board.

3. The lift is fading. Corrected weekly lift on orders per user-day:

This looks like a novelty effect. Returning users, who are the people the feature is built for, show roughly zero lift overall once corrected (treatment 0.0940 vs control 0.0943).

4. The lift is concentrated in new users, which is suspicious. New users go from 0.0612 to 0.0681 per user-day (+11%). A reorder feature should help returning users most, since they have baskets to reorder. Combined with the SRM, this points toward a measurement artifact, possibly in how new users are counted, until shown otherwise.

5. AOV passes the guardrail. Treatment AOV averages $48.00 vs $48.25 for control, a gap of 0.5%. This is inside the 2% limit. The gap narrowed to 0.2% in week 4, which is the incident week, so I can't rule out the duplicate events affecting AOV. Revenue per user is only up about 0.3% in week 4, once the lower AOV is included.

What the test doesn't show

• Significance. Even corrected, a simple Poisson approximation gives z ≈ 2, which is borderline. Days aren't independent, so a proper clustered analysis would likely be weaker. The pre-registered rule requires a positive and significant lift at the end of the test. The end of the test shows about +0.5%, so the rule isn't met.
• Per-user outcomes. The file only has daily counts. The design specifies orders per user over the test. Without unique user IDs we can't measure that, and repeat users are counted once per active day.
• Long-run effect. Four weeks with a decaying effect doesn't tell us the steady state.

Decision

Don't ship Monday. The pre-registered rule isn't met. The SRM invalidates the comparison until explained. The incident contaminates the final week. The best estimate of the lift is near zero and falling.

Next steps

1. Resolve the SRM this week. Check bucketing logic, whether treatment users get a different login or app path that changes logging, and whether control users are dropped somewhere. Don't analyse further until this is explained.
2. Get the incident fixed in the data. Rebuild order counts from deduplicated events for 14 October and confirm which arms and user types were affected. Ask engineering to explain why control wasn't affected.
3. Pull user-level data. Rerun the primary metric on unique users over the test window, with the pre-registered definition.
4. Test the novelty explanation. Check whether returning users' use of the one-tap reorder changes over time. If it doesn't stick, the feature isn't working for the people it targets.
5. Decide the next test. If the SRM is fixed, consider a rerun with a fresh 50/50 split and a 4–6 week window. Predefine the analysis of new vs returning users, and set a stopping rule that accounts for novelty decay. If the SRM can't be fixed, we start again.
6. Report back on the revised numbers before any ship decision.

Readout: new home page test (23 Sep – 20 Oct)

Recommendation: don't ship Monday on this evidence. Zoe's +6% is real arithmetic (I reproduce +5.8%), but most of it comes from a logging bug and an early novelty effect. The test also has a randomization problem that has to be explained first.

Reproducing the headline

Orders per user-day, all 28 days: control 16,560 / 196,173 = 8.44%, treatment 20,234 / 226,652 = 8.93%, a lift of +5.8%. That matches Zoe's number.

Problems with it

1. Double-counted orders on 14 Oct inflate treatment. Treatment orders that day are about double normal (new 296 vs ~150, returning 1,058 vs ~540). Control is normal (124 and 480), which fits the incident log. That is roughly 660 phantom orders. Dropping 14 Oct from both arms cuts the lift from +5.8% to +2.4% (8.44% → 8.64%).

The size also doesn't fit the log. A 9-hour duplication should add roughly 40% to that day's orders, not 100%. So the incident may have been bigger than described, and I wouldn't patch it by subtracting my estimate. Engineering should dedupe by order ID and confirm no other days are affected.

2. The lift is a novelty effect and has largely disappeared. Orders per user-day by week, excluding 14 Oct:

• Returning users, who are about 70% of traffic, show no effect in any week.
• New users show a big lift that decays week over week.
• Over the last 14 days (ex 14 Oct), all users combined: 8.47% vs 8.42%, +0.6%. A rough z-test gives z ≈ 0.4, which is not significant, and that is before accounting for repeat users, which would widen the interval further.

The mechanism is also odd. "Your usual basket" should help returning users, who have a basket, and do little for new users, who have none. The effect shows up in the wrong group, which makes me doubt it is caused by the feature.

3. Sample ratio mismatch. The split was meant to be 50/50 by user, but treatment has 226,652 user-days vs 196,173 for control, a ratio of 1.155 (chi-square ≈ 2,200 against a 50/50 expectation). The ratio is the same from day one (2,400 vs 2,089 on 23 Sep) and the new/returning mix is identical in both arms (about 30% new). That pattern looks like an assignment or bucketing fault rather than a behavioral effect, which would likely differ by user type and move over time.

There is one alternative: "users" might mean daily active users, and the new page might genuinely bring users back more often. In that case orders per active user is a biased denominator. Either way, "orders per user" isn't measuring what the pre-registration intended until we have assigned-user counts. If the arms were truly 50/50, orders per assigned user would be about +22%, so this ambiguity matters in both directions.

What the test does support

• AOV guardrail: treatment is about 0.6% lower in weeks 1–3 ($47.90 vs $48.20, $48.10 vs $48.40, $47.80 vs $48.10) and 0.2% lower in week 4. That is well inside the 2% limit. I have no order-level data for a confidence interval. Week 4 also contains the incident, so I'd recheck it after dedupe.
• No evidence of harm: returning users are flat, and nothing suggests the page hurts.

What it doesn't show

• Any durable lift in orders per user.
• That the effect is caused by the page, given the SRM and the mismatch with the mechanism.
• Whether the pre-registered rule is met. On raw numbers it is "met", but the data feeding it is corrupted, so I wouldn't count it.

Next steps

1. Engineering: explain the 15% allocation imbalance, with assigned-user counts per arm and the bucketing logic. Dedupe orders by ID for the whole test and confirm the scope of the 14 Oct incident. 2. Analytics: rerun on order-level data with orders per assigned user. Use user-clustered confidence intervals, and cut by new vs returning and by days since first exposure. Also get an AOV confidence interval. 3. Decision: - If the SRM is a bug, rerun the test, or at least 2 weeks of clean data with a fresh new-user cohort, since novelty decays over about 3 weeks. - If the SRM is explained and the clean data still show a durable positive lift, ship. - If the business wants to move sooner, I'd support a staged rollout (say 20%) with a persistent holdback, not a full launch. The downside looks small, but the upside is unproven.

Zoe, I'd send the above in place of your note. The headline can't be defended as written, and the cleaned-up result is closer to "no detectable lift after the first couple of weeks."