Analyse experiment results

Subject: Onboarding checklist test: Why we should ship Variant B

Team,

Before we write off the 5-step onboarding checklist as a failure, let’s take a closer look at the data. My recommendation is that we ship the shorter checklist immediately rather than rolling it back or extending the test.

What we should conclude "Not statistically significant" doesn't mean the test failed; it just means we didn't hit our ambitious pre-registered 3pp threshold. Looking at the actual numbers, the story is encouraging: * Activation leaned positive: Variant B saw a +1.7pp lift. Looking at the 95% confidence interval (−1.4pp to +4.8pp), the upside potential is much larger than the downside risk. At worst, the shorter checklist performs similarly to the 9-step version; at best, it provides a solid lift. * Less user friction: Support tickets for "onboarding confusion" dropped by a third (from 44 to 29). Even if activation is technically flat, users are clearly having a better experience. * Simpler is better: Cutting four steps reduces UI bloat and future maintenance for our team.

What we should do next We have the option to run the test for another four weeks to detect a 2pp effect, but that isn't a good use of our time. If the true lift is +1.7pp, another month of testing might still yield an "inconclusive" result while tying up our experimentation traffic.

Because Variant B is already built, shipping it to 100% of users will only take a day of engineering time. When a change is low-effort, reduces support volume, simplifies the product, and shows no evidence of harming activation, the default move should be to ship it.

Let’s roll out the 5-step checklist to everyone this week and move on to our next big bet.

Team,

Before we write off the shorter checklist as a failure, let’s reframe what the data actually tells us.

“Not statistically significant” does not mean “no impact”—it means our test was underpowered to detect an effect smaller than 3pp. Here is the full picture:

1. The directional signals are uniformly positive: Activation increased by +1.7pp (31.2% → 32.9%). While the 95% CI (−1.4pp to +4.8pp) crosses zero, it leans heavily positive. The downside risk is small, while the potential upside is meaningful.
2. Clear reduction in user friction: Support tickets tagged “onboarding confusion” dropped by 34% (44 to 29). Even if activation were completely flat, cutting nearly half the steps improved user experience and reduced support burden.
3. The cost/benefit is strongly asymmetric: Shipping this takes one day of engineering.

Recommendation: Ship the 5-step checklist to 100% and move on.

Running the test for another four weeks to hit an MDE of 2pp isn’t the best use of our time or traffic. We would burn a month of runway just to confirm what looks like a positive, low-risk change with virtually zero engineering cost.

Let’s bank the operational win (fewer support tickets, leaner UX, likely small activation lift) by rolling it out tomorrow. We can monitor activation metrics in production and direct our experimentation bandwidth toward bigger, higher-leverage bets for Trellis.

Team—

This is inconclusive, not a failure. Activation was 1.7 percentage points higher with the shorter checklist, but the 95% CI (−1.4 to +4.8 points) includes no effect and possible harm—as well as a benefit of at least 3 points. The test wasn’t precise enough to settle the question.

I recommend running the test for another four weeks. That should let us assess effects around 2 points, which this change may plausibly deliver. Before extending, agree on what activation gain would justify rollout and what activation decline would stop it. Track onboarding-confusion tickets too, but compare rates per user: the raw drop from 44 to 29 is encouraging, not conclusive.

After the extension, decide using both activation and confusion—not a “significant/not significant” label alone.