Analyse experiment results

Subject: Onboarding checklist test: Why we should ship Variant B

Team,

Before we write off the 5-step onboarding checklist as a failure, let’s take a closer look at the data. My recommendation is that we ship the shorter checklist immediately rather than rolling it back or extending the test.

What we should conclude "Not statistically significant" doesn't mean the test failed; it just means we didn't hit our ambitious pre-registered 3pp threshold. Looking at the actual numbers, the story is encouraging: * Activation leaned positive: Variant B saw a +1.7pp lift. Looking at the 95% confidence interval (−1.4pp to +4.8pp), the upside potential is much larger than the downside risk. At worst, the shorter checklist performs similarly to the 9-step version; at best, it provides a solid lift. * Less user friction: Support tickets for "onboarding confusion" dropped by a third (from 44 to 29). Even if activation is technically flat, users are clearly having a better experience. * Simpler is better: Cutting four steps reduces UI bloat and future maintenance for our team.

What we should do next We have the option to run the test for another four weeks to detect a 2pp effect, but that isn't a good use of our time. If the true lift is +1.7pp, another month of testing might still yield an "inconclusive" result while tying up our experimentation traffic.

Because Variant B is already built, shipping it to 100% of users will only take a day of engineering time. When a change is low-effort, reduces support volume, simplifies the product, and shows no evidence of harming activation, the default move should be to ship it.

Let’s roll out the 5-step checklist to everyone this week and move on to our next big bet.

Shorter checklist test — what it tells us and what's next

Hi all,

The shorter onboarding checklist came back "not significant." That doesn't make it a failure, and I don't think we should drop it.

What the test actually says

• B activated at 32.9% vs 31.2% for A, a +1.7pp lift.
• The 95% interval runs from −1.4pp to +4.8pp. Most of that range is positive, and the worst plausible case is a small loss.
• We sized the test to detect 3pp. A real 1–2pp gain was always likely to come back "not significant." The result means the effect is smaller than 3pp or zero.¹ It doesn't mean the change didn't work.

Supporting signal

Support tickets tagged "onboarding confusion" fell from 44 to 29 during the test. We didn't split tickets by arm, so this is directional only. It does point the same way as the activation result.

Recommendation: ship it

• It's already built, and rolling it out costs about a day.
• The likely effect is neutral to positive, and five steps are simpler to maintain than nine.
• Running four more weeks would only detect effects of about 2pp. Our point estimate is 1.7pp, so we would probably still get an ambiguous answer, a month later.

Guardrails

• Watch 7-day activation for the four weeks after rollout. If it drops more than about 1.5pp below the current baseline, roll back.
• Split onboarding-confusion tickets by checklist version from now on, so next time we can tell whether a ticket drop is real.

For future tests

Before launch, agree what we'll do if the result comes back inconclusive. For a cheap, low-risk change like this, "not significant but probably not harmful" should default to ship.

Happy to discuss.

To: Onboarding team Re: Shorter checklist test

Conclusion: This wasn't a failure. It was inconclusive, and the result doesn't support "no effect."

• The point estimate was +1.7pp (32.9% vs. 31.2%).
• The 95% CI runs from −1.4pp to +4.8pp. Effects from slightly negative to clearly useful are all consistent with the data.
• The test was powered for a 3pp lift. A real 1–2pp improvement, which would still be worth having, would usually not reach significance at this sample size.
• Support tickets for "onboarding confusion" fell from 44 to 29. This wasn't a pre-registered metric, and counts this small are noisy, but it points the same way.

Recommendation: Don't treat this as a loss, and don't run the test indefinitely.

1. Ship the five-step checklist to everyone, with a 10% holdout on the old flow. Shipping costs about one day of engineering, the change is easy to reverse, and the evidence leans positive. The holdout keeps measuring the real effect at no extra build cost.
2. Decide the extension question on decision value, not on the test result. Another four weeks would detect roughly 2pp. That is worth doing only if a 2pp lift would change what we do next. If it wouldn't, the holdout gives us enough information.
3. If we extend, make it a new pre-registered analysis. Set the new target effect and sample size now, and don't keep peeking at the current data until it crosses significance. Changing the goalposts after an inconclusive result is how teams end up with false positives.

Bottom line: The checklist is probably neutral to helpful, and it is cheap to keep. Ship it with a holdout, track activation and tickets, and revisit in four weeks.