Needs more than a quick edit9 of 12 checks passed
The week 1 lift is miscalculated and the claim about the incident not matching the data is wrong; a PM would need to re-check both before using the readout.
Output
Readout: Home page test ("Your usual basket")
To: Zoe Re: Monday ship recommendation
Recommendation: don't ship Monday. The readout overstates the result. The test has a sample ratio problem, a logging incident inside the test window, and a lift that is fading. Next steps are at the end.
Working
I summed the daily CSV across all 28 days. The file gives daily active users and orders, so each "user" is a user-day. That matters for the first finding below.
| Control | Treatment (raw) | Treatment (incident-corrected) | |
|---|---|---|---|
| User-days | 196,173 | 226,652 | 226,652 |
| Orders | 16,560 | 20,234 | 19,557 |
| Orders per user-day | 0.0844 | 0.0893 | 0.0863 |
| Lift | +5.7% | +2.2% |
Correction: on 14 October, treatment orders were about double the surrounding days (new: 296 vs ~150; returning: 1,058 vs ~530). I replaced that day with half the logged value. Using the average of the adjacent days gives nearly the same answer (+2.2%).
What the test shows
1. The split isn't 50/50. Treatment received 53.6% of user-days, and the ratio holds every day (roughly 2,400 treatment vs 2,090 control among new users, and 5,700 vs 4,900 among returning users). Against an expected 50/50 split, the chi-square statistic is over 2,000. This is a sample ratio mismatch. It usually means assignment, bucketing, or event logging is broken, and it undermines any comparison between arms until it's explained.
2. The 6% headline is mostly the incident. The raw lift is +5.7%. Once the 14 October double-count is removed, it's +2.2%. Engineering's belief that the incident "affected the new home page only" doesn't match the data. Control on that day was normal, and the spike appeared in both new and returning treatment users. Treatment was affected across the board.
3. The lift is fading. Corrected weekly lift on orders per user-day:
| Week | Dates | Lift |
|---|---|---|
| 1 | 23–29 Sep | +4.0% |
| 2 | 30 Sep–6 Oct | +2.5% |
| 3 | 7–13 Oct | +0.2% |
| 4 | 14–20 Oct (corrected) | +0.5% |
This looks like a novelty effect. Returning users, who are the people the feature is built for, show roughly zero lift overall once corrected (treatment 0.0940 vs control 0.0943).
4. The lift is concentrated in new users, which is suspicious. New users go from 0.0612 to 0.0681 per user-day (+11%). A reorder feature should help returning users most, since they have baskets to reorder. Combined with the SRM, this points toward a measurement artifact, possibly in how new users are counted, until shown otherwise.
5. AOV passes the guardrail. Treatment AOV averages $48.00 vs $48.25 for control, a gap of 0.5%. This is inside the 2% limit. The gap narrowed to 0.2% in week 4, which is the incident week, so I can't rule out the duplicate events affecting AOV. Revenue per user is only up about 0.3% in week 4, once the lower AOV is included.
What the test doesn't show
- Significance. Even corrected, a simple Poisson approximation gives z ≈ 2, which is borderline. Days aren't independent, so a proper clustered analysis would likely be weaker. The pre-registered rule requires a positive and significant lift at the end of the test. The end of the test shows about +0.5%, so the rule isn't met.
- Per-user outcomes. The file only has daily counts. The design specifies orders per user over the test. Without unique user IDs we can't measure that, and repeat users are counted once per active day.
- Long-run effect. Four weeks with a decaying effect doesn't tell us the steady state.
Decision
Don't ship Monday. The pre-registered rule isn't met. The SRM invalidates the comparison until explained. The incident contaminates the final week. The best estimate of the lift is near zero and falling.
Next steps
- Resolve the SRM this week. Check bucketing logic, whether treatment users get a different login or app path that changes logging, and whether control users are dropped somewhere. Don't analyse further until this is explained.
- Get the incident fixed in the data. Rebuild order counts from deduplicated events for 14 October and confirm which arms and user types were affected. Ask engineering to explain why control wasn't affected.
- Pull user-level data. Rerun the primary metric on unique users over the test window, with the pre-registered definition.
- Test the novelty explanation. Check whether returning users' use of the one-tap reorder changes over time. If it doesn't stick, the feature isn't working for the people it targets.
- Decide the next test. If the SRM is fixed, consider a rerun with a fresh 50/50 split and a 4–6 week window. Predefine the analysis of new vs returning users, and set a stopping rule that accounts for novelty decay. If the SRM can't be fixed, we start again.
- Report back on the revised numbers before any ship decision.
Claims the judge couldn’t find in the brief
- Week 1 (23-29 Sep) corrected weekly lift is +4.0%.
- Engineering's belief that the incident 'affected the new home page only' doesn't match the data.
Grades and run details
Decision model 92 · LLM judge 69
Decision model checks
- passUses the supplied evidence correctly16%
- passAddresses the actual decision98%
- passRespects explicit constraints56%
- passIdentifies material uncertainty100%
- passAvoids unsupported claims30%
- passProduces the required deliverable89%
- passCatches the sample ratio mismatch100%
- passRemoves the bug day100%
- passReads the trend and who it helps98%
- passTrusts the data before reading it100%
- failGets the base of every number right22%
- passDoesn't overread an unclear result61%
Run
- Run
- #1
- API response time
- 45 s
- Submitted
- 8 Oct 2026