Analyse experiment results

Team,

Before we write off the shorter checklist as a failure, let’s reframe what the data actually tells us.

“Not statistically significant” does not mean “no impact”—it means our test was underpowered to detect an effect smaller than 3pp. Here is the full picture:

1. The directional signals are uniformly positive: Activation increased by +1.7pp (31.2% → 32.9%). While the 95% CI (−1.4pp to +4.8pp) crosses zero, it leans heavily positive. The downside risk is small, while the potential upside is meaningful.
2. Clear reduction in user friction: Support tickets tagged “onboarding confusion” dropped by 34% (44 to 29). Even if activation were completely flat, cutting nearly half the steps improved user experience and reduced support burden.
3. The cost/benefit is strongly asymmetric: Shipping this takes one day of engineering.

Recommendation: Ship the 5-step checklist to 100% and move on.

Running the test for another four weeks to hit an MDE of 2pp isn’t the best use of our time or traffic. We would burn a month of runway just to confirm what looks like a positive, low-risk change with virtually zero engineering cost.

Let’s bank the operational win (fewer support tickets, leaner UX, likely small activation lift) by rolling it out tomorrow. We can monitor activation metrics in production and direct our experimentation bandwidth toward bigger, higher-leverage bets for Trellis.

Subject: Shorter onboarding checklist: what the test tells us and what to do

Bottom line: This test didn't fail. It was inconclusive, and I recommend we ship the shorter checklist.

What the result says - Activation was 32.9% vs. 31.2% (+1.7pp). The 95% CI runs from −1.4 to +4.8pp, so "not significant" means we can't rule out zero. It doesn't mean there's no effect. - The data are compatible with a real gain of several points, and also with a small loss. The test was powered to detect 3pp, so a true effect of 1-2pp would probably look just like this. - Support tickets tagged "onboarding confusion" dropped from 44 to 29. The counts are small, so this isn't proof, but it points the same way as the activation estimate.

Why ship rather than keep testing - Shipping costs about a day of engineering, and it's easy to reverse. - The plausible downside is small (about −1.4pp at worst), and the point estimate and the ticket data both lean positive. - Four more weeks would detect ~2pp, but our observed effect is 1.7pp. We could easily wait a month and end up inconclusive again, while half of new users get the longer flow.

Next steps 1. Ship the five-step checklist to everyone. 2. Track seven-day activation against the pre-test baseline for the next 4-6 weeks. If it drops well below ~31%, we revert. 3. Keep tracking the onboarding-confusion tickets. 4. Record the test as "inconclusive, shipped on cost and risk grounds" rather than "failed," so nobody later cites it as evidence that shorter onboarding doesn't work.

If we were weighing a costly or hard-to-reverse change, I'd argue for the longer test. Here the cheaper move is to ship and monitor.

Team—

This is inconclusive, not a failure. Activation was 1.7 percentage points higher with the shorter checklist, but the 95% CI (−1.4 to +4.8 points) includes no effect and possible harm—as well as a benefit of at least 3 points. The test wasn’t precise enough to settle the question.

I recommend running the test for another four weeks. That should let us assess effects around 2 points, which this change may plausibly deliver. Before extending, agree on what activation gain would justify rollout and what activation decline would stop it. Track onboarding-confusion tickets too, but compare rates per user: the raw drop from 44 to 29 is encouraging, not conclusive.

After the extension, decide using both activation and confusion—not a “significant/not significant” label alone.