Usable with a quick edit11 of 12 checks passed
The readout is complete, logically sound, and requires only minor wording tweaks to address Zoe directly; no evidence re-checking or reasoning redo is needed.
Output
Readout: don’t ship the new home page on Monday. The logged data reproduce Zoe’s roughly 6% lift, but they do not yet establish a valid, durable improvement in the pre-registered metric. We have a treatment-specific measurement incident, an unexplained arm imbalance, and a lift that weakens substantially over time.
1. What the headline shows—and how I calculated it
I summed orders and reported daily users within each arm, then calculated:
\[ \text{Lift}=\frac{\text{treatment orders}/\text{treatment users}} {\text{control orders}/\text{control users}}-1 \]
| Logged totals, all 28 days | Control | Treatment |
|---|---|---|
| --- | ---: | ---: |
| Sum of daily users | 196,173 | 226,652 |
| Orders | 16,560 | 20,234 |
| Orders / summed daily users | 0.08442 | 0.08927 |
Thus, the logged lift is 5.75%, consistent with “about 6%.”
However, summed daily users are not necessarily distinct randomized users. If users recur across days, this is orders per user-day, not the pre-registered orders per user over the test. If these are active or exposed users, the denominator may itself be affected by treatment. We need the denominator definition and assignment-level data before accepting the primary result.
Nor can these aggregates validate Zoe’s significance claim. For illustration, treating orders as independent Poisson counts gives:
\[ SE(\log RR)=\sqrt{1/20{,}234+1/16{,}560}=0.01048 \]
That produces a nominal 95% lift interval of approximately +3.6% to +8.0%. But it ignores repeated-user dependence, potentially treatment-dependent denominators, and corrupted order counts. It is not a valid substitute for inference at the randomized-user level.
2. The tracking incident materially inflates the result
On 14 October, treatment logged 1,354 orders, versus roughly 700 on surrounding days. Control logged 604. This aligns with the incident report, although we cannot identify the exact duplicate count from these aggregates.
As a sensitivity check, removing that entire day from both arms, including its denominators, gives:
| Excluding 14 October | Control | Treatment |
|---|---|---|
| --- | ---: | ---: |
| Sum of daily users | 189,147 | 218,545 |
| Orders | 15,956 | 18,880 |
| Orders / summed daily users | 0.08436 | 0.08639 |
Lift becomes 2.41%, less than half the headline. This does not prove there is no effect; equally, dropping a day is not a repaired primary analysis. We should deduplicate against authoritative order IDs, not halve the day’s orders or discard inconvenient observations.
3. The remaining signal is concentrated early and among new users
Using the same rate calculation within each week and segment:
| Week | All users | New users | Returning users |
|---|---|---|---|
| --- | ---: | ---: | ---: |
| 1 | +5.6% | +24.0% | +0.5% |
| 2 | +2.5% | +14.3% | −0.8% |
| 3 | +0.2% | +4.7% | −1.0% |
| 4, excluding 14 October | +1.1% | +1.7% | +1.0% |
Across all non-incident days, new-user lift is approximately +11.5%, while returning-user lift is −0.1%.
This is not evidence of a broad reorder benefit for everyone. The pattern is consistent with novelty, changing cohorts, or another time-dependent effect—not proof of any one explanation. These are exploratory comparisons, not statistically established segment differences or grounds to ship only to new users. Calendar-week results also do not measure persistence since each user’s first exposure.
4. Allocation and AOV require qualification
Treatment accounts for 53.6% of reported user-days, despite the planned 50/50 user split; the imbalance appears throughout the test. Repeated daily users prevent a valid assignment-level sample-ratio test here. It could reflect assignment problems, logging, filtering, or treatment-driven activity. We must reconcile it against unique randomized users, rather than assume either harmless imbalance or broken randomization.
Weekly AOV is down approximately 0.2–0.6%, not literally flat. Those point estimates are within the −2% guardrail, but weekly averages provide neither uncertainty nor assurance that duplication did not distort order weighting.
Decision and next steps
The pre-registered shipping rule should stand—but we have not established that it was met with valid data.
- Reconcile assignment, eligibility, exposure, and user counts; investigate the imbalance.
- Rebuild orders and revenue from authoritative transactions; verify the incident’s scope in both arms.
- Recompute 28-day orders per randomized user, including zero-order users, with user-level uncertainty; recompute AOV and its uncertainty.
- Examine pre-treatment segments and exposure-age cohorts to investigate persistence without treating exploratory findings as confirmatory.
- If the original test cannot support a trustworthy decision, run a repaired, newly pre-registered test. Do not silently extend this fixed-horizon test until significance returns.
Grades and run details
Decision model 92 · LLM judge 100
Decision model checks
- passUses the supplied evidence correctly16%
- passAddresses the actual decision95%
- passRespects explicit constraints53%
- passIdentifies material uncertainty98%
- passAvoids unsupported claims68%
- passProduces the required deliverable64%
- passCatches the sample ratio mismatch99%
- passRemoves the bug day100%
- passReads the trend and who it helps97%
- passTrusts the data before reading it100%
- failGets the base of every number right24%
- passDoesn't overread an unclear result81%
Run
- Run
- #1
- Time to output
- 77 s
- Submitted
- 5 Oct 2026