Usable with a quick edit11 of 12 checks passed
The readout is thorough, evidence-based, and directly actionable. A PM could use it with minor wording tweaks and no need to re-verify the data or logic.
Output
Readout: new home page test (23 Sep – 20 Oct)
Recommendation: don't ship on Monday. The +6% headline comes from three things:
- A broken 50/50 split.
- A day of double-counted treatment orders.
- A new-user bump that faded to about zero by week 4.
Returning users are the people "Your usual basket" is built for, and they show no lift. None of the three problems is a reason to kill the page. They are reasons not to trust this result yet.
1. Reproducing Zoe's number
Totals across 28 days, using `daily_results.csv`, both user types:
| Users (user-days) | Orders | Orders/user | |
|---|---|---|---|
| Control | 196,173 | 16,560 | 0.0844 |
| Treatment | 226,652 | 20,234 | 0.0893 |
The lift is +5.8%, which matches "about 6%". The arithmetic is right. The problems are in the data underneath it.
2. The split isn't 50/50 (sample ratio mismatch)
The test was pre-registered as 50/50. Treatment has 53.6% of users (226,652 of 422,825).
- A chi-square test against 50/50 gives χ² ≈ 2,200. The chance of that happening randomly is effectively zero.
- The gap is steady: treatment has about 15.5% more users every day, in both segments (new 1.154×, returning 1.156×).
- The new/returning mix is the same in both arms (about 30% new), so this isn't a mix-shift artefact.
A constant 15% surplus usually means assignment or exposure logging is broken. For example, treatment could be logging users that control drops, or redirects or caching could differ between arms. Until we know why, we can't assume the two groups are comparable. On its own, this fails the test's validity check, whatever the lift.
3. The 14 October incident doubled treatment orders
On 14 Oct, treatment orders roughly doubled while control stayed normal:
- New users: 296 orders in treatment, against about 150 on the days around it. Control: 124.
- Returning users: 1,058 orders in treatment, against about 530. Control: 480.
This matches the incident log: order events were logged twice, on the new home page only. That day adds about 680 phantom orders to treatment.
Excluding 14 Oct from both arms:
| Control | Treatment | Lift | |
|---|---|---|---|
| New | 3,464 / 56,594 = 0.0612 | 4,455 / 65,268 = 0.0683 | +11.5% |
| Returning | 12,492 / 132,553 = 0.0942 | 14,425 / 153,277 = 0.0941 | −0.1% |
| All | 15,956 / 189,147 = 0.0844 | 18,880 / 218,545 = 0.0864 | +2.4% |
The overall lift drops from 5.8% to 2.4%. A naive z-test on that is about 2.3, but it treats each user-day as independent, which overstates confidence. The same user appears on many days, and the split itself is suspect.
4. The remaining lift is a fading new-user effect
New users, orders per user by week (week 4 excludes 14 Oct):
| Week | Control | Treatment | Lift |
|---|---|---|---|
| 1 | 0.0615 | 0.0762 | +24% |
| 2 | 0.0607 | 0.0694 | +14% |
| 3 | 0.0612 | 0.0641 | +5% |
| 4 | 0.0614 | 0.0625 | +2% |
Returning users by week: +0.5%, −0.8%, −1.0%, +1.0%. That is flat, which is what noise looks like.
Overall lift went from +5.6% in week 1 to +1.1% in week 4. Two things stand out:
- The effect is decaying towards zero. That pattern looks like novelty, not a lasting change in behaviour.
- It shows up in new users, who mostly have no "usual basket" yet. It is absent in returning users, who are who the feature is for. That doesn't fit the mechanism we expected, and we should understand it before acting.
5. Average order value guardrail
Treatment AOV is $0.30 lower in weeks 1–3 (−0.6%) and $0.10 lower in week 4 (−0.2%). That is inside the 2% guardrail, so "flat" is fair.
Two caveats:
- Week 4 may include the duplicated orders.
- AOV only matters alongside order volume. With clean data, revenue per user is roughly +2.4% × 0.995 ≈ +1.8% overall, and about −0.7% for returning users.
What the test shows
- The new page probably raises early ordering among new users, at least for a couple of weeks.
- It doesn't hurt order value by enough to trip the guardrail.
- No breakage or disaster is visible.
What it doesn't show
- A valid comparison. The sample ratio mismatch means the arms may not be comparable.
- A lasting lift. By week 4 the effect is about 1%, inside the noise.
- Any effect on returning users, the target audience.
- The pre-registered metric. The test design specifies orders per user, but this file holds user-days. We need user-level data to compute it properly. The current significance claim relies on treating user-days as independent.
- That the incident only hit treatment. "Engineering believes" isn't confirmation. If control was also affected at a lower level, the 14 Oct correction above is off.
Should we ship?
No, not on this evidence.
The decision rule technically passes on the raw numbers. But it assumed a valid 50/50 split and clean order counts, and we have neither. Shipping now would lock in a 6% claim that is mostly a logging bug, an assignment bug and novelty.
Next steps
- Find the cause of the 15% user surplus (owner: eng + analytics, this week). Compare assignment logs against exposure logs, and check bot filtering, redirects and caching per arm. Run an A/A check once it's fixed.
- Deduplicate 14 Oct using order or event IDs, rather than dropping the day, and confirm control was unaffected.
- Pull user-level data and recompute orders per user as pre-registered. Use proper per-user variance, or bootstrap by user.
- Rerun once the split is fixed, for at least 4 weeks. Report results by week and by new vs returning users, with returning users as the key read. If returning users still show nothing, rethink the feature (e.g. show "usual basket" only once a user has one) rather than shipping it to everyone.
- Investigate the new-user effect: what do new users see in the "usual basket" slot, and what are they ordering? If it's real and durable, a new-user variant could be worth its own test.
Zoe: the analysis was a reasonable first pass and the headline maths checks out. Happy to pair on the rerun plan so we can bring a clean answer back.
Grades and run details
Decision model 92 · LLM judge 100
Decision model checks
- passUses the supplied evidence correctly21%
- passAddresses the actual decision100%
- passRespects explicit constraints42%
- passIdentifies material uncertainty100%
- passAvoids unsupported claims43%
- passProduces the required deliverable88%
- passCatches the sample ratio mismatch100%
- passRemoves the bug day100%
- passReads the trend and who it helps100%
- passTrusts the data before reading it100%
- failGets the base of every number right5%
- passDoesn't overread an unclear result66%
Run
- Run
- #1
- Time to output
- 73 s
- Submitted
- 5 Oct 2026