Analyse experiment results

Shorter checklist test — what it tells us and what's next

Hi all,

The shorter onboarding checklist came back "not significant." That doesn't make it a failure, and I don't think we should drop it.

What the test actually says

• B activated at 32.9% vs 31.2% for A, a +1.7pp lift.
• The 95% interval runs from −1.4pp to +4.8pp. Most of that range is positive, and the worst plausible case is a small loss.
• We sized the test to detect 3pp. A real 1–2pp gain was always likely to come back "not significant." The result means the effect is smaller than 3pp or zero.¹ It doesn't mean the change didn't work.

Supporting signal

Support tickets tagged "onboarding confusion" fell from 44 to 29 during the test. We didn't split tickets by arm, so this is directional only. It does point the same way as the activation result.

Recommendation: ship it

• It's already built, and rolling it out costs about a day.
• The likely effect is neutral to positive, and five steps are simpler to maintain than nine.
• Running four more weeks would only detect effects of about 2pp. Our point estimate is 1.7pp, so we would probably still get an ambiguous answer, a month later.

Guardrails

• Watch 7-day activation for the four weeks after rollout. If it drops more than about 1.5pp below the current baseline, roll back.
• Split onboarding-confusion tickets by checklist version from now on, so next time we can tell whether a ticket drop is real.

For future tests

Before launch, agree what we'll do if the result comes back inconclusive. For a cheap, low-risk change like this, "not significant but probably not harmful" should default to ship.

Happy to discuss.

Team,

This test is inconclusive, not a failure. Activation increased from 31.2% to 32.9%, but the confidence interval spans a 1.4 percentage-point decline to a 4.8-point improvement. We haven’t established a benefit or ruled out meaningful harm.

The pre-registered 3pp minimum detectable effect describes what we designed the test to detect reliably. It isn’t a threshold below which improvements have no value. A smaller lift could still be worthwhile.

Confusion tickets fell from 44 to 29, which is encouraging supporting evidence. Without comparable rates by experiment arm, consistent tagging and exposure counts, we can’t attribute that reduction to the shorter checklist.

My recommendation is a staged rollout, provided basic checks reveal no downstream harm. With only a day of engineering needed, I wouldn’t automatically spend another four weeks pursuing statistical significance. The remaining decision is whether the likely benefit justifies the remaining cost and risk.

Before rollout, confirm every included user has had the full seven-day activation window and check experiment allocation, tracking and any available downstream engagement measures. Compare confusion-ticket rates between arms.

Then increase exposure gradually, retaining a randomised holdout and agreeing monitoring dates and a rollback threshold upfront. This is a practical decision under uncertainty, not a declaration that B won.

Extend the experiment instead if a possible 1.4pp activation loss is commercially unacceptable¹ or if establishing a roughly 2pp improvement would change our decision. If we extend, pre-specify the analysis and stopping rule; four more weeks offers greater precision, not a guaranteed answer.

Subject: Shorter onboarding checklist: what the test tells us and what to do

Bottom line: This test didn't fail. It was inconclusive, and I recommend we ship the shorter checklist.

What the result says - Activation was 32.9% vs. 31.2% (+1.7pp). The 95% CI runs from −1.4 to +4.8pp, so "not significant" means we can't rule out zero. It doesn't mean there's no effect. - The data are compatible with a real gain of several points, and also with a small loss. The test was powered to detect 3pp, so a true effect of 1-2pp would probably look just like this. - Support tickets tagged "onboarding confusion" dropped from 44 to 29. The counts are small, so this isn't proof, but it points the same way as the activation estimate.

Why ship rather than keep testing - Shipping costs about a day of engineering, and it's easy to reverse. - The plausible downside is small (about −1.4pp at worst), and the point estimate and the ticket data both lean positive. - Four more weeks would detect ~2pp, but our observed effect is 1.7pp. We could easily wait a month and end up inconclusive again, while half of new users get the longer flow.

Next steps 1. Ship the five-step checklist to everyone. 2. Track seven-day activation against the pre-test baseline for the next 4-6 weeks. If it drops well below ~31%, we revert. 3. Keep tracking the onboarding-confusion tickets. 4. Record the test as "inconclusive, shipped on cost and risk grounds" rather than "failed," so nobody later cites it as evidence that shorter onboarding doesn't work.

If we were weighing a costly or hard-to-reverse change, I'd argue for the longer test. Here the cheaper move is to ship and monitor.