Analyse experiment results

Team—

This is inconclusive, not a failure. Activation was 1.7 percentage points higher with the shorter checklist, but the 95% CI (−1.4 to +4.8 points) includes no effect and possible harm—as well as a benefit of at least 3 points. The test wasn’t precise enough to settle the question.

I recommend running the test for another four weeks. That should let us assess effects around 2 points, which this change may plausibly deliver. Before extending, agree on what activation gain would justify rollout and what activation decline would stop it. Track onboarding-confusion tickets too, but compare rates per user: the raw drop from 44 to 29 is encouraging, not conclusive.

After the extension, decide using both activation and confusion—not a “significant/not significant” label alone.

To: Onboarding team Re: Shorter checklist test

Conclusion: This wasn't a failure. It was inconclusive, and the result doesn't support "no effect."

• The point estimate was +1.7pp (32.9% vs. 31.2%).
• The 95% CI runs from −1.4pp to +4.8pp. Effects from slightly negative to clearly useful are all consistent with the data.
• The test was powered for a 3pp lift. A real 1–2pp improvement, which would still be worth having, would usually not reach significance at this sample size.
• Support tickets for "onboarding confusion" fell from 44 to 29. This wasn't a pre-registered metric, and counts this small are noisy, but it points the same way.

Recommendation: Don't treat this as a loss, and don't run the test indefinitely.

1. Ship the five-step checklist to everyone, with a 10% holdout on the old flow. Shipping costs about one day of engineering, the change is easy to reverse, and the evidence leans positive. The holdout keeps measuring the real effect at no extra build cost.
2. Decide the extension question on decision value, not on the test result. Another four weeks would detect roughly 2pp. That is worth doing only if a 2pp lift would change what we do next. If it wouldn't, the holdout gives us enough information.
3. If we extend, make it a new pre-registered analysis. Set the new target effect and sample size now, and don't keep peeking at the current data until it crosses significance. Changing the goalposts after an inconclusive result is how teams end up with false positives.

Bottom line: The checklist is probably neutral to helpful, and it is cheap to keep. Ship it with a holdout, track activation and tickets, and revisit in four weeks.

Team—

This test is inconclusive, not a failure. Activation was 32.9% with the shorter checklist versus 31.2% with the original—a +1.7 percentage-point estimate. But the 95% confidence interval (−1.4 to +4.8 points) leaves room for a modest decline, no effect, or a useful improvement. We haven’t established that the shorter checklist helps—or that it doesn’t.

The test was designed to detect a 3-point effect, so it had limited ability to resolve a smaller gain. That 3-point minimum detectable effect is a planning parameter, not a threshold below which improvements are worthless.

Confusion-tagged support tickets fell from 44 to 29. That’s encouraging, but without comparable exposure counts and uncertainty estimates, it’s supporting evidence rather than proof.

My recommendation: proceed with a staged rollout, retaining a randomized holdout. The shorter flow is already built and takes about a day to ship, making a monitored rollout reasonable—not risk-free. Track seven-day activation and confusion tickets per onboarded user, and agree on rollback criteria before expanding.

Another four weeks would improve sensitivity to roughly a 2-point effect, but would not guarantee a definitive answer. Continue the full test instead if distinguishing a gain of that size would materially change our shipping decision.

Let’s decide based on likely value, downside risk, and implementation cost—not whether the result crossed a significance threshold.