Analyse experiment results

Recommendation: Do not ship variant B to 100% on Monday. Keep A live while we investigate and revise B.

B did improve trial-to-paid conversion: 13.1% vs. 11.8%, a +1.3pp lift with a 95% CI of +0.6 to +2.0pp. But that gain comes with two problems against the guardrails agreed before launch:

• Day-30 retention fell 4.5pp (95% CI −7.9 to −1.1), exceeding the maximum tolerated 2pp decline at the point estimate.
• Refund requests rose 42%, well above the 10% limit. With an equal split, the raw counts also imply a higher request rate in B; we should confirm the rate and uncertainty before treating the size of the increase as settled.

The revenue case does not offset those risks. Day-30 revenue per trial start was only $0.19 higher in B, and its confidence interval crosses zero, so we don’t have clear evidence of a revenue gain.

That matters because B foregrounds the annual plan’s $9.99 monthly equivalent while hiding the monthly plan behind “See all plans.” The results are consistent with the concern that this framing may lift initial purchases while leaving more customers dissatisfied or less likely to remain. We should not trade clear retention and refund guardrail breaches for a conversion lift whose revenue benefit is unproven.

For Monday, I’d recommend holding A, reviewing refund reasons and plan mix, and testing a revised B that makes the $119.99 upfront annual charge and monthly option equally clear. Ship only if a follow-up test meets the pre-agreed retention and refund guardrails.

To: Priya Nair, Head of Growth Recommendation: Do not ship B to 100% on Monday. Keep A as the default, stop new exposure to B, and retest a clearer pricing design.

B delivers a credible conversion lift, but it fails both pre-agreed guardrails. We should not override those thresholds because the headline metric improved.

• Conversion improved: Trial-to-paid rose from 11.8% to 13.1%, a +1.3 percentage-point lift (95% CI: +0.6 to +2.0).
• Retention deteriorated: Day-30 paid retention fell 4.5 points, from 84.0% to 79.5%, versus an allowed decline of no more than 2 points. The confidence interval excludes zero, although it does not establish that the true decline necessarily exceeds 2 points.
• Refunds breached the threshold: Requests increased from 212 to 301—42%, versus the permitted 10%. Confirm the rate using the agreed denominator, but the reported result is a clear warning under approximately equal exposure.
• Revenue does not justify the trade-off: Day-30 revenue per trial start increased only $0.19, and its confidence interval crosses zero. We have not established a revenue benefit.

The design provides a plausible explanation, not proven causation: leading with “$9.99/month” while hiding the monthly option may obscure that customers must pay $119.99 upfront. Higher conversion could therefore reflect misunderstanding rather than stronger purchase intent.

Next steps: Validate cohort maturity—the experiment ran 21 days, so we need to understand how day-30 outcomes were obtained—and check refund reasons, plan mix, and billing-related complaints. Then test a version that prominently states “$119.99 billed annually ($9.99/month equivalent)” and keeps the $14.99 monthly option visible.

Monday’s defensible position: “B increased conversion, but failed our customer-outcome guardrails without demonstrating a revenue gain. We’ll preserve the learning, fix pricing clarity, and require mature results that meet the agreed thresholds before scaling.”

Recommendation: Do not ship Variant B. Keep Variant A live and do not roll B out to 100% on Monday.

While Variant B delivered an initial +1.3pp lift in trial-to-paid conversion, it violates both pre-agreed guardrails, creates bill shock, and delivers no statistically significant revenue upside.

Here is the reasoning to defend this decision on Monday:

1. Both Pre-Agreed Guardrails Were Blown

We agreed on strict risk thresholds before launch. Variant B failed both: * Day-30 Retention: Fell −4.5pp (down to 79.5%), more than double our maximum allowed drop of 2pp (95% CI: −7.9 to −1.1). * Refund Requests: Surged +42% (301 vs. 212), four times our 10% ceiling.

Overriding guardrails after the fact sets a dangerous precedent and invites compounding churn.

2. The Conversion Lift Is an Illusion Driven by Bill Shock

Variant B hides the monthly tier and frames the annual plan as “$9.99/month.” Users clearly believe they are committing to a flexible monthly rate, only to be charged $119.99 upfront. The 42% spike in refunds and immediate drop in 30-day retention prove that these are not high-intent, qualified subscribers—they are confused customers who churn or dispute the charge as soon as they review their statements.

3. Long-Term Economics Are Flat to Negative

• Day-30 revenue per trial start is statistically unchanged: $9.61 for B vs. $9.42 for A (CI crosses zero).
• When factored against the operational cost of processing 42% more refunds, payment dispute fees, customer support load, and downstream LTV decay from lower retention, Variant B will be net-negative over a 12-month horizon.
• App store ratings and brand trust will also suffer if users perceive our checkout flow as deceptive.

Next Step

Iterate on Variant C: keep the focus on annual plan value, but make the upfront billing explicit (e.g., “$9.99/mo, billed annually at $119.99”) and test clear annual vs. monthly side-by-side positioning rather than hiding the monthly option.