Analyse experiment results

Recommendation: Do not ship Variant B. Keep Variant A live and do not roll B out to 100% on Monday.

While Variant B delivered an initial +1.3pp lift in trial-to-paid conversion, it violates both pre-agreed guardrails, creates bill shock, and delivers no statistically significant revenue upside.

Here is the reasoning to defend this decision on Monday:

1. Both Pre-Agreed Guardrails Were Blown

We agreed on strict risk thresholds before launch. Variant B failed both: * Day-30 Retention: Fell −4.5pp (down to 79.5%), more than double our maximum allowed drop of 2pp (95% CI: −7.9 to −1.1). * Refund Requests: Surged +42% (301 vs. 212), four times our 10% ceiling.

Overriding guardrails after the fact sets a dangerous precedent and invites compounding churn.

2. The Conversion Lift Is an Illusion Driven by Bill Shock

Variant B hides the monthly tier and frames the annual plan as “$9.99/month.” Users clearly believe they are committing to a flexible monthly rate, only to be charged $119.99 upfront. The 42% spike in refunds and immediate drop in 30-day retention prove that these are not high-intent, qualified subscribers—they are confused customers who churn or dispute the charge as soon as they review their statements.

3. Long-Term Economics Are Flat to Negative

• Day-30 revenue per trial start is statistically unchanged: $9.61 for B vs. $9.42 for A (CI crosses zero).
• When factored against the operational cost of processing 42% more refunds, payment dispute fees, customer support load, and downstream LTV decay from lower retention, Variant B will be net-negative over a 12-month horizon.
• App store ratings and brand trust will also suffer if users perceive our checkout flow as deceptive.

Next Step

Iterate on Variant C: keep the focus on annual plan value, but make the upfront billing explicit (e.g., “$9.99/mo, billed annually at $119.99”) and test clear annual vs. monthly side-by-side positioning rather than hiding the monthly option.

Recommendation: Do not ship variant B to 100% on Monday. Keep A live while we investigate and revise B.

B did improve trial-to-paid conversion: 13.1% vs. 11.8%, a +1.3pp lift with a 95% CI of +0.6 to +2.0pp. But that gain comes with two problems against the guardrails agreed before launch:

• Day-30 retention fell 4.5pp (95% CI −7.9 to −1.1), exceeding the maximum tolerated 2pp decline at the point estimate.
• Refund requests rose 42%, well above the 10% limit. With an equal split, the raw counts also imply a higher request rate in B; we should confirm the rate and uncertainty before treating the size of the increase as settled.

The revenue case does not offset those risks. Day-30 revenue per trial start was only $0.19 higher in B, and its confidence interval crosses zero, so we don’t have clear evidence of a revenue gain.

That matters because B foregrounds the annual plan’s $9.99 monthly equivalent while hiding the monthly plan behind “See all plans.” The results are consistent with the concern that this framing may lift initial purchases while leaving more customers dissatisfied or less likely to remain. We should not trade clear retention and refund guardrail breaches for a conversion lift whose revenue benefit is unproven.

For Monday, I’d recommend holding A, reviewing refund reasons and plan mix, and testing a revised B that makes the $119.99 upfront annual charge and monthly option equally clear. Ship only if a follow-up test meets the pre-agreed retention and refund guardrails.

Recommendation: Don't ship B to 100% on Monday. B breached both guardrails we agreed on before launch. Keep the test running or revert to A, and bring a fix to the meeting instead.

Why

1. The guardrails were set to settle this. Refund requests rose 42% (212 → 301) against a 10% limit. That's a clear breach, and at this volume it isn't noise. Day-30 retention fell 4.5pp against a 2pp limit. The CI (−7.9 to −1.1) technically includes drops smaller than 2pp, but the point estimate is more than double the limit, so we can't claim it's safe.

1. The conversion win isn't yet a revenue win. Conversion is up 1.3pp and that's real. But revenue per trial start is $9.61 vs $9.42, and the CI crosses zero. We are converting more people, and more of them are leaving or asking for their money back.

1. The retention drop is surprising. If B pushes people toward the annual plan, which is billed up front, day-30 retention should rise, not fall. A drop alongside more refunds suggests some buyers are surprised by a $119.99 charge after seeing "$9.99/month." Hiding the monthly plan behind "See all plans" may also pull in lower-intent buyers. This is a hypothesis, not a finding, since the readout doesn't show plan mix.

1. Day 30 undersells the risk. Annual buyers who regret the purchase may refund later, and the readout can't show that.

What to do before Monday

• Pull plan mix (monthly vs. annual) by variant, plus retention and refund rates by plan.
• Check whether the $119.99 up-front charge is clear enough on B. Refund reasons or support tickets would help.
• Confirm whether revenue is net of refunds.

What to propose instead

Test a B2 that keeps the $9.99/month anchor but states "$119.99 billed today" next to it and keeps the monthly option visible. If B2 holds the conversion gain and stays within both guardrails, ship it.

Line for the meeting: "We set guardrails so we wouldn't have to debate this after the fact. B failed them, so we're fixing the billing clarity and retesting, not shipping a conversion gain that may be costing us customers and refunds."

If leadership insists on shipping B, the least risky option is a partial rollout with the refund rate monitored weekly and a pre-agreed rollback trigger. I wouldn't recommend it.