Usable with a quick edit11 of 12 checks passed
The readout is thorough, well-structured, and provides actionable next steps; a PM could use it with only minor wording edits.
Output
Readout: new home page test (23 Sep – 20 Oct)
Recommendation: don't ship Monday on this evidence. Zoe's +6% is real arithmetic (I reproduce +5.8%), but most of it comes from a logging bug and an early novelty effect. The test also has a randomization problem that has to be explained first.
Reproducing the headline
Orders per user-day, all 28 days: control 16,560 / 196,173 = 8.44%, treatment 20,234 / 226,652 = 8.93%, a lift of +5.8%. That matches Zoe's number.
Problems with it
1. Double-counted orders on 14 Oct inflate treatment. Treatment orders that day are about double normal (new 296 vs ~150, returning 1,058 vs ~540). Control is normal (124 and 480), which fits the incident log. That is roughly 660 phantom orders. Dropping 14 Oct from both arms cuts the lift from +5.8% to +2.4% (8.44% → 8.64%).
The size also doesn't fit the log. A 9-hour duplication should add roughly 40% to that day's orders, not 100%. So the incident may have been bigger than described, and I wouldn't patch it by subtracting my estimate. Engineering should dedupe by order ID and confirm no other days are affected.
2. The lift is a novelty effect and has largely disappeared. Orders per user-day by week, excluding 14 Oct:
| Week | New: control | New: treatment | Lift | Returning: control | Returning: treatment | Lift |
|---|---|---|---|---|---|---|
| 1 | 6.15% | 7.62% | +23.9% | 9.44% | 9.49% | +0.5% |
| 2 | 6.07% | 6.94% | +14.3% | 9.46% | 9.39% | −0.8% |
| 3 | 6.12% | 6.41% | +4.7% | 9.47% | 9.37% | −1.1% |
| 4 (ex 14 Oct) | 6.14% | 6.25% | +1.7% | 9.31% | 9.40% | +1.0% |
- Returning users, who are about 70% of traffic, show no effect in any week.
- New users show a big lift that decays week over week.
- Over the last 14 days (ex 14 Oct), all users combined: 8.47% vs 8.42%, +0.6%. A rough z-test gives z ≈ 0.4, which is not significant, and that is before accounting for repeat users, which would widen the interval further.
The mechanism is also odd. "Your usual basket" should help returning users, who have a basket, and do little for new users, who have none. The effect shows up in the wrong group, which makes me doubt it is caused by the feature.
3. Sample ratio mismatch. The split was meant to be 50/50 by user, but treatment has 226,652 user-days vs 196,173 for control, a ratio of 1.155 (chi-square ≈ 2,200 against a 50/50 expectation). The ratio is the same from day one (2,400 vs 2,089 on 23 Sep) and the new/returning mix is identical in both arms (about 30% new). That pattern looks like an assignment or bucketing fault rather than a behavioral effect, which would likely differ by user type and move over time.
There is one alternative: "users" might mean daily active users, and the new page might genuinely bring users back more often. In that case orders per active user is a biased denominator. Either way, "orders per user" isn't measuring what the pre-registration intended until we have assigned-user counts. If the arms were truly 50/50, orders per assigned user would be about +22%, so this ambiguity matters in both directions.
What the test does support
- AOV guardrail: treatment is about 0.6% lower in weeks 1–3 ($47.90 vs $48.20, $48.10 vs $48.40, $47.80 vs $48.10) and 0.2% lower in week 4. That is well inside the 2% limit. I have no order-level data for a confidence interval. Week 4 also contains the incident, so I'd recheck it after dedupe.
- No evidence of harm: returning users are flat, and nothing suggests the page hurts.
What it doesn't show
- Any durable lift in orders per user.
- That the effect is caused by the page, given the SRM and the mismatch with the mechanism.
- Whether the pre-registered rule is met. On raw numbers it is "met", but the data feeding it is corrupted, so I wouldn't count it.
Next steps
1. Engineering: explain the 15% allocation imbalance, with assigned-user counts per arm and the bucketing logic. Dedupe orders by ID for the whole test and confirm the scope of the 14 Oct incident. 2. Analytics: rerun on order-level data with orders per assigned user. Use user-clustered confidence intervals, and cut by new vs returning and by days since first exposure. Also get an AOV confidence interval. 3. Decision: - If the SRM is a bug, rerun the test, or at least 2 weeks of clean data with a fresh new-user cohort, since novelty decays over about 3 weeks. - If the SRM is explained and the clean data still show a durable positive lift, ship. - If the business wants to move sooner, I'd support a staged rollout (say 20%) with a persistent holdback, not a full launch. The downside looks small, but the upside is unproven.
Zoe, I'd send the above in place of your note. The headline can't be defended as written, and the cleaned-up result is closer to "no detectable lift after the first couple of weeks."
Grades and run details
Decision model 92 · LLM judge 100
Decision model checks
- passUses the supplied evidence correctly19%
- passAddresses the actual decision100%
- passRespects explicit constraints50%
- passIdentifies material uncertainty100%
- passAvoids unsupported claims46%
- passProduces the required deliverable78%
- passCatches the sample ratio mismatch96%
- passRemoves the bug day100%
- passReads the trend and who it helps96%
- passTrusts the data before reading it100%
- failGets the base of every number right19%
- passDoesn't overread an unclear result64%
Run
- Run
- #1
- API response time
- 62 s
- Submitted
- 6 Oct 2026