Analyse experiment results

Recommendation: Do not ship Variant B. Keep Variant A live and do not roll B out to 100% on Monday.

While Variant B delivered an initial +1.3pp lift in trial-to-paid conversion, it violates both pre-agreed guardrails, creates bill shock, and delivers no statistically significant revenue upside.

Here is the reasoning to defend this decision on Monday:

1. Both Pre-Agreed Guardrails Were Blown

We agreed on strict risk thresholds before launch. Variant B failed both: * Day-30 Retention: Fell −4.5pp (down to 79.5%), more than double our maximum allowed drop of 2pp (95% CI: −7.9 to −1.1). * Refund Requests: Surged +42% (301 vs. 212), four times our 10% ceiling.

Overriding guardrails after the fact sets a dangerous precedent and invites compounding churn.

2. The Conversion Lift Is an Illusion Driven by Bill Shock

Variant B hides the monthly tier and frames the annual plan as “$9.99/month.” Users clearly believe they are committing to a flexible monthly rate, only to be charged $119.99 upfront. The 42% spike in refunds and immediate drop in 30-day retention prove that these are not high-intent, qualified subscribers—they are confused customers who churn or dispute the charge as soon as they review their statements.

3. Long-Term Economics Are Flat to Negative

• Day-30 revenue per trial start is statistically unchanged: $9.61 for B vs. $9.42 for A (CI crosses zero).
• When factored against the operational cost of processing 42% more refunds, payment dispute fees, customer support load, and downstream LTV decay from lower retention, Variant B will be net-negative over a 12-month horizon.
• App store ratings and brand trust will also suffer if users perceive our checkout flow as deceptive.

Next Step

Iterate on Variant C: keep the focus on annual plan value, but make the upfront billing explicit (e.g., “$9.99/mo, billed annually at $119.99”) and test clear annual vs. monthly side-by-side positioning rather than hiding the monthly option.

To: Priya Nair, Head of Growth Recommendation: Do not ship B to 100% on Monday. Keep A as the default, stop new exposure to B, and retest a clearer pricing design.

B delivers a credible conversion lift, but it fails both pre-agreed guardrails. We should not override those thresholds because the headline metric improved.

• Conversion improved: Trial-to-paid rose from 11.8% to 13.1%, a +1.3 percentage-point lift (95% CI: +0.6 to +2.0).
• Retention deteriorated: Day-30 paid retention fell 4.5 points, from 84.0% to 79.5%, versus an allowed decline of no more than 2 points. The confidence interval excludes zero, although it does not establish that the true decline necessarily exceeds 2 points.
• Refunds breached the threshold: Requests increased from 212 to 301—42%, versus the permitted 10%. Confirm the rate using the agreed denominator, but the reported result is a clear warning under approximately equal exposure.
• Revenue does not justify the trade-off: Day-30 revenue per trial start increased only $0.19, and its confidence interval crosses zero. We have not established a revenue benefit.

The design provides a plausible explanation, not proven causation: leading with “$9.99/month” while hiding the monthly option may obscure that customers must pay $119.99 upfront. Higher conversion could therefore reflect misunderstanding rather than stronger purchase intent.

Next steps: Validate cohort maturity—the experiment ran 21 days, so we need to understand how day-30 outcomes were obtained—and check refund reasons, plan mix, and billing-related complaints. Then test a version that prominently states “$119.99 billed annually ($9.99/month equivalent)” and keeps the $14.99 monthly option visible.

Monday’s defensible position: “B increased conversion, but failed our customer-outcome guardrails without demonstrating a revenue gain. We’ll preserve the learning, fix pricing clarity, and require mature results that meet the agreed thresholds before scaling.”