Analyse experiment results

Hi Team,

Before we write off the shorter onboarding checklist as a failure, let's look at the full picture.

While the headline result was statistically non-significant, the numbers and qualitative feedback tell a more nuanced story:

Positive Directional Lift: Group B (5 steps) saw a 32.9% activation rate compared to 31.2% for Group A (9 steps)—a 1.7 percentage point increase.

Underpowered Test: Our pre-registered minimum detectable effect (MDE) was 3pp, meaning the test wasn't powered to reliably catch a 1.7pp or 2pp shift.

Reduced Support Friction: Support tickets tagged "onboarding confusion" dropped significantly from 44 to 29 during the test.

What we should conclude: The test wasn't a failure; it was inconclusive. We likely have a small positive gain accompanied by a meaningful reduction in user confusion.¹

What we should do next: Given that shipping this to everyone only takes about a day of engineering, we should ship it. Continuing to run the test for another four weeks just to chase statistical significance on a small effect isn't a wise use of time² when the engineering cost to launch is so low and support friction has already improved. Let's roll out the 5-step checklist and monitor downstream retention.

Team—

This test is inconclusive, not a failure. Activation was 32.9% with the shorter checklist versus 31.2% with the original—a +1.7 percentage-point estimate. But the 95% confidence interval (−1.4 to +4.8 points) leaves room for a modest decline, no effect, or a useful improvement. We haven’t established that the shorter checklist helps—or that it doesn’t.

The test was designed to detect a 3-point effect, so it had limited ability to resolve a smaller gain. That 3-point minimum detectable effect is a planning parameter, not a threshold below which improvements are worthless.

Confusion-tagged support tickets fell from 44 to 29. That’s encouraging, but without comparable exposure counts and uncertainty estimates, it’s supporting evidence rather than proof.

My recommendation: proceed with a staged rollout, retaining a randomized holdout. The shorter flow is already built and takes about a day to ship, making a monitored rollout reasonable—not risk-free. Track seven-day activation and confusion tickets per onboarded user, and agree on rollback criteria before expanding.

Another four weeks would improve sensitivity to roughly a 2-point effect, but would not guarantee a definitive answer. Continue the full test instead if distinguishing a gain of that size would materially change our shipping decision.

Let’s decide based on likely value, downside risk, and implementation cost—not whether the result crossed a significance threshold.

Team,

This test is inconclusive, not a failure. Activation increased from 31.2% to 32.9%, but the confidence interval spans a 1.4 percentage-point decline to a 4.8-point improvement. We haven’t established a benefit or ruled out meaningful harm.

The pre-registered 3pp minimum detectable effect describes what we designed the test to detect reliably. It isn’t a threshold below which improvements have no value. A smaller lift could still be worthwhile.

Confusion tickets fell from 44 to 29, which is encouraging supporting evidence. Without comparable rates by experiment arm, consistent tagging and exposure counts, we can’t attribute that reduction to the shorter checklist.

My recommendation is a staged rollout, provided basic checks reveal no downstream harm. With only a day of engineering needed, I wouldn’t automatically spend another four weeks pursuing statistical significance. The remaining decision is whether the likely benefit justifies the remaining cost and risk.

Before rollout, confirm every included user has had the full seven-day activation window and check experiment allocation, tracking and any available downstream engagement measures. Compare confusion-ticket rates between arms.

Then increase exposure gradually, retaining a randomised holdout and agreeing monitoring dates and a rollback threshold upfront. This is a practical decision under uncertainty, not a declaration that B won.

Extend the experiment instead if a possible 1.4pp activation loss is commercially unacceptable¹ or if establishing a roughly 2pp improvement would change our decision. If we extend, pre-specify the analysis and stopping rule; four more weeks offers greater precision, not a guaranteed answer.