Analyse experiment results

Readout: new home page test (23 Sep – 20 Oct)

Recommendation: don't ship on Monday. The +6% headline comes from three things:

• A broken 50/50 split.
• A day of double-counted treatment orders.
• A new-user bump that faded to about zero by week 4.

Returning users are the people "Your usual basket" is built for, and they show no lift. None of the three problems is a reason to kill the page. They are reasons not to trust this result yet.

1. Reproducing Zoe's number

Totals across 28 days, using `daily_results.csv`, both user types:

The lift is +5.8%, which matches "about 6%". The arithmetic is right. The problems are in the data underneath it.

2. The split isn't 50/50 (sample ratio mismatch)

The test was pre-registered as 50/50. Treatment has 53.6% of users (226,652 of 422,825).

• A chi-square test against 50/50 gives χ² ≈ 2,200. The chance of that happening randomly is effectively zero.
• The gap is steady: treatment has about 15.5% more users every day, in both segments (new 1.154×, returning 1.156×).
• The new/returning mix is the same in both arms (about 30% new), so this isn't a mix-shift artefact.

A constant 15% surplus usually means assignment or exposure logging is broken. For example, treatment could be logging users that control drops, or redirects or caching could differ between arms. Until we know why, we can't assume the two groups are comparable. On its own, this fails the test's validity check, whatever the lift.

3. The 14 October incident doubled treatment orders

On 14 Oct, treatment orders roughly doubled while control stayed normal:

• New users: 296 orders in treatment, against about 150 on the days around it. Control: 124.
• Returning users: 1,058 orders in treatment, against about 530. Control: 480.

This matches the incident log: order events were logged twice, on the new home page only. That day adds about 680 phantom orders to treatment.

Excluding 14 Oct from both arms:

The overall lift drops from 5.8% to 2.4%. A naive z-test on that is about 2.3, but it treats each user-day as independent, which overstates confidence. The same user appears on many days, and the split itself is suspect.

4. The remaining lift is a fading new-user effect

New users, orders per user by week (week 4 excludes 14 Oct):

Returning users by week: +0.5%, −0.8%, −1.0%, +1.0%. That is flat, which is what noise looks like.

Overall lift went from +5.6% in week 1 to +1.1% in week 4. Two things stand out:

• The effect is decaying towards zero. That pattern looks like novelty, not a lasting change in behaviour.
• It shows up in new users, who mostly have no "usual basket" yet. It is absent in returning users, who are who the feature is for. That doesn't fit the mechanism we expected, and we should understand it before acting.

5. Average order value guardrail

Treatment AOV is $0.30 lower in weeks 1–3 (−0.6%) and $0.10 lower in week 4 (−0.2%). That is inside the 2% guardrail, so "flat" is fair.

Two caveats:

• Week 4 may include the duplicated orders.
• AOV only matters alongside order volume. With clean data, revenue per user is roughly +2.4% × 0.995 ≈ +1.8% overall, and about −0.7% for returning users.

What the test shows

• The new page probably raises early ordering among new users, at least for a couple of weeks.
• It doesn't hurt order value by enough to trip the guardrail.
• No breakage or disaster is visible.

What it doesn't show

• A valid comparison. The sample ratio mismatch means the arms may not be comparable.
• A lasting lift. By week 4 the effect is about 1%, inside the noise.
• Any effect on returning users, the target audience.
• The pre-registered metric. The test design specifies orders per user, but this file holds user-days. We need user-level data to compute it properly. The current significance claim relies on treating user-days as independent.
• That the incident only hit treatment. "Engineering believes" isn't confirmation. If control was also affected at a lower level, the 14 Oct correction above is off.

Should we ship?

No, not on this evidence.

The decision rule technically passes on the raw numbers. But it assumed a valid 50/50 split and clean order counts, and we have neither. Shipping now would lock in a 6% claim that is mostly a logging bug, an assignment bug and novelty.

Next steps

1. Find the cause of the 15% user surplus (owner: eng + analytics, this week). Compare assignment logs against exposure logs, and check bot filtering, redirects and caching per arm. Run an A/A check once it's fixed.
2. Deduplicate 14 Oct using order or event IDs, rather than dropping the day, and confirm control was unaffected.
3. Pull user-level data and recompute orders per user as pre-registered. Use proper per-user variance, or bootstrap by user.
4. Rerun once the split is fixed, for at least 4 weeks. Report results by week and by new vs returning users, with returning users as the key read. If returning users still show nothing, rethink the feature (e.g. show "usual basket" only once a user has one) rather than shipping it to everyone.
5. Investigate the new-user effect: what do new users see in the "usual basket" slot, and what are they ordering? If it's real and durable, a new-user variant could be worth its own test.

Zoe: the analysis was a reasonable first pass and the headline maths checks out. Happy to pair on the rerun plan so we can bring a clean answer back.

Readout to send

I would not recommend shipping Monday on this evidence. The raw daily aggregates don’t support Zoe’s “significant” claim, and a treatment-only order-logging incident materially inflates the apparent lift. We need a clean, user-level analysis before applying the pre-registered ship rule.

What the data shows

The daily file supports a pooled orders per recorded user-day calculation—not orders per unique randomized user over the full test:

• Control: 16,560 orders / 196,173 recorded users = 8.44%
• Treatment: 20,086 / 226,652 = 8.87%, or +5.0% relative

But treatment orders on 14 October are conspicuous: 296 for new users and 1,058 for returning users, roughly twice typical daily counts. The incident log says the treatment-only tracking release duplicated some order events for nine hours. As a rough sensitivity check only, halving those 14 October treatment counts subtracts about 677 orders; treatment then has 19,409 / 226,652 = 8.56%, or +1.4% relative versus control. This is not a valid final correction—the affected events need to be deduplicated from the underlying logs—but it shows how much the incident changes the result.

By user type, that rough adjustment gives about +7.6% for new users and −0.3% for returning users. That difference is worth investigating, but these aggregates don’t establish a reliable segment effect.

The pre-registered AOV guardrail looks descriptively okay: weekly treatment AOV is $47.90, $48.10, $47.80 and $48.20 versus control’s $48.20, $48.40, $48.10 and $48.30. The simple average is $48.00 versus $48.25 (about −0.5%, within the −2% limit). “Flat” is a little too strong; we also can’t assess uncertainty or reconstruct a correctly weighted overall AOV from this table alone.

What it doesn’t show

It does not establish that orders per randomized user rose significantly. The file reports daily user counts, not the unique assigned-user denominators or each user’s cumulative orders needed for the registered primary metric and its valid inference. Also, summed treatment user counts are 226,652 vs. 196,173 for control—about 15.5% higher despite a planned 50/50 user split. Those may be user-days rather than unique randomized users, so this is not proof of a randomization failure; it is a reason to audit assignment, exposure and denominator construction.

Even a simplistic event-count Poisson check on the rough deduplicated totals would not show significance (approximately p=.18); that is only a diagnostic, not a substitute for user-level analysis.

What to do next

1. Deduplicate treatment order events using event IDs and quantify the incident’s actual impact.
2. Rebuild the pre-registered orders-per-user metric from unique assigned users, with the planned statistical method; audit the 50/50 split and exposure.
3. Recompute AOV from the appropriate order-level data and check the guardrail.
4. Ship only if that clean analysis meets the decision rule. If the logs or assignment cannot support it, rerun a clean test rather than treating this result as a win.

Recommendation: don’t ship on Monday. The reported ~6% lift is reproducible, but it includes a treatment-specific tracking incident. The supplied aggregates also cannot establish significance for the pre-registered, user-level metric. We have a promising signal—not a reliable launch decision.

What the raw data shows

I summed orders and daily user counts across the 28 days:

The calculation is:

\[ \text{Lift}=\frac{20,234/226,652}{16,560/196,373}-1=5.86\%. \]

So Zoe’s arithmetic is right for this aggregate ratio. However, summed daily users are not necessarily unique randomized users. If people appear on multiple days, this is orders per user-day, not the pre-registered orders per user over the test.

The tracking incident materially changes the result

On 14 October, treatment recorded 1,354 orders versus 604 in control, with 8,107 versus 7,026 daily users. Treatment’s order rate was almost twice its normal level. That is consistent with the documented duplicate-event incident.

As a sensitivity check only, removing that entire day from both arms gives:

\[ \text{Lift excluding 14 October} =\frac{18,880/218,545}{15,956/189,347}-1 =2.52\%. \]

The remaining signal is positive, but much smaller. Dropping a whole day is not a valid substitute for recovering actual orders: the incident lasted nine hours, and that day also contains legitimate transactions.

The effect is not stable or broad-based

Using the same aggregate calculation:

Excluding the incident day, the full-period lift is +11.9% for new users and −0.14% for returning users. New-user weekly lifts decline from approximately 24.0% to 15.0%, 4.7%, and 2.5%.

That suggests the remaining aggregate gain is concentrated among new users and may be fading. It does not prove novelty: calendar effects, changing cohorts, or instrumentation could also explain the pattern. Nor does a positive new-user estimate alone establish a statistically reliable difference between segments.

What we cannot conclude yet

“Significant” is not established by these files. For the intended user-randomized analysis, we need unique assigned users—including users with zero orders—and each user’s deduplicated order count. Daily aggregates do not provide the user-level variance or account for repeated appearances. Treating user-days as independent could overstate precision.

The denominator needs investigation. Treatment has 15.4% more summed daily users than control, despite the planned 50/50 assignment, with an imbalance throughout the test. This is a warning, not proof of assignment failure: if these are active users, treatment could change activity. Check assignment counts separately from exposure/activity counts. An activity-conditioned denominator can change the estimand and introduce bias.

AOV is slightly down, not flat. Weekly declines range from approximately 0.2% to 0.6%, all within the 2% guardrail. Weighting the reported weekly averages by logged order counts gives approximately $48.25 control versus $48.01 treatment, or −0.51%. That is descriptively reassuring, but duplicate orders may distort the weights or AOV calculation. These summaries also lack uncertainty estimates.

What we should do next

1. Reconcile against canonical transactions. Deduplicate using order IDs and validate the incident’s actual scope rather than assuming only treatment was affected.
2. Audit randomization and denominators. Check unique assignments, eligibility, exposure logging, missing users, and sample-ratio mismatch.
3. Recompute the pre-registered analysis: intention-to-treat orders per unique assigned user over the full 28 days, with user-level confidence intervals; recompute AOV from valid orders and revenue.
4. Apply the original decision rule to that repaired result. If lift is positive and significant and the AOV guardrail passes, shipping becomes defensible. If trustworthy recovery is impossible, rerun with fixed instrumentation and a pre-specified duration and analysis.
5. Investigate durability and targeting separately. Pre-specify new/returning-user interactions and time-since-exposure analyses in follow-up work. Don’t use this post-hoc segment result to justify a targeted launch.

Bottom line: pause the launch, repair the measurement, and decide on the registered metric—not the contaminated headline.