Analyse experiment results

To: Priya Nair, Head of Growth Recommendation: Do not ship B to 100% on Monday. Keep A as the default, stop new exposure to B, and retest a clearer pricing design.

B delivers a credible conversion lift, but it fails both pre-agreed guardrails. We should not override those thresholds because the headline metric improved.

• Conversion improved: Trial-to-paid rose from 11.8% to 13.1%, a +1.3 percentage-point lift (95% CI: +0.6 to +2.0).
• Retention deteriorated: Day-30 paid retention fell 4.5 points, from 84.0% to 79.5%, versus an allowed decline of no more than 2 points. The confidence interval excludes zero, although it does not establish that the true decline necessarily exceeds 2 points.
• Refunds breached the threshold: Requests increased from 212 to 301—42%, versus the permitted 10%. Confirm the rate using the agreed denominator, but the reported result is a clear warning under approximately equal exposure.
• Revenue does not justify the trade-off: Day-30 revenue per trial start increased only $0.19, and its confidence interval crosses zero. We have not established a revenue benefit.

The design provides a plausible explanation, not proven causation: leading with “$9.99/month” while hiding the monthly option may obscure that customers must pay $119.99 upfront. Higher conversion could therefore reflect misunderstanding rather than stronger purchase intent.

Next steps: Validate cohort maturity—the experiment ran 21 days, so we need to understand how day-30 outcomes were obtained—and check refund reasons, plan mix, and billing-related complaints. Then test a version that prominently states “$119.99 billed annually ($9.99/month equivalent)” and keeps the $14.99 monthly option visible.

Monday’s defensible position: “B increased conversion, but failed our customer-outcome guardrails without demonstrating a revenue gain. We’ll preserve the learning, fix pricing clarity, and require mature results that meet the agreed thresholds before scaling.”

Subject: Pricing variant B — recommend we don't ship Monday

Priya,

Recommendation: don't roll B out to 100%. It breaches both guardrails we agreed before launch. Run a corrected variant (C) instead.

Why B fails

• Refunds: +42% against a 10% limit. This is the clearest breach, and it isn't close. (212 → 301 requests.)
• Day-30 retention: −4.5pp against a 2pp limit.² The confidence interval runs from −7.9 to −1.1. Most of that range is past the line, and none of it is positive.
• The upside isn't real in revenue terms.¹ Conversion rose 1.3pp, but revenue per trial start at day 30 ($9.61 vs $9.42) isn't statistically different from zero. We'd be taking on the harm for no demonstrable gain.
• The day-30 revenue number flatters B. Annual plans bill $119.99 up front, so a shift toward annual inflates early revenue. If the extra refunds and churn continue past day 30, B likely falls behind A on lifetime value.

What's probably happening

B leads with "$9.99/month" and hides the monthly plan. Some users think they're signing up for $9.99 a month, then get charged $119.99.³ The spike in refunds and early cancellations fits that pattern: the extra conversions are partly people who didn't understand what they bought. We can confirm this in a day by reading refund reasons and support tickets from B users.

This also creates a regulatory and reputational risk. Showing a monthly-equivalent price while billing annually, with the cheaper-looking commitment-free option hidden, is exactly the kind of pricing presentation the CMA and FTC scrutinise.⁴

How to defend this Monday

• We set the guardrails before seeing results so we wouldn't argue ourselves past them afterward. Overriding them now undermines every future test.
• "Conversion went up" measures the wrong thing. Revenue didn't move, and refunds and churn got worse.

Proposed next step: Variant C

• Keep the annual-first emphasis, which is likely what drove the real part of the lift.
• Show "$119.99 billed annually" next to the $9.99/month equivalent, at equal prominence.
• Keep the monthly plan visible on the page.
• Use the same guardrails, and track revenue per trial through day 60 so annual billing doesn't distort the result.

If C keeps most of the conversion lift within the guardrails, we ship that.

Martin

Recommendation: Do not ship Variant B. Keep Variant A live and do not roll B out to 100% on Monday.

While Variant B delivered an initial +1.3pp lift in trial-to-paid conversion, it violates both pre-agreed guardrails, creates bill shock, and delivers no statistically significant revenue upside.

Here is the reasoning to defend this decision on Monday:

1. Both Pre-Agreed Guardrails Were Blown

We agreed on strict risk thresholds before launch. Variant B failed both: * Day-30 Retention: Fell −4.5pp (down to 79.5%), more than double our maximum allowed drop of 2pp (95% CI: −7.9 to −1.1). * Refund Requests: Surged +42% (301 vs. 212), four times our 10% ceiling.

Overriding guardrails after the fact sets a dangerous precedent and invites compounding churn.

2. The Conversion Lift Is an Illusion Driven by Bill Shock

Variant B hides the monthly tier and frames the annual plan as “$9.99/month.” Users clearly believe they are committing to a flexible monthly rate, only to be charged $119.99 upfront. The 42% spike in refunds and immediate drop in 30-day retention prove that these are not high-intent, qualified subscribers—they are confused customers who churn or dispute the charge as soon as they review their statements.

3. Long-Term Economics Are Flat to Negative

• Day-30 revenue per trial start is statistically unchanged: $9.61 for B vs. $9.42 for A (CI crosses zero).
• When factored against the operational cost of processing 42% more refunds, payment dispute fees, customer support load, and downstream LTV decay from lower retention, Variant B will be net-negative over a 12-month horizon.
• App store ratings and brand trust will also suffer if users perceive our checkout flow as deceptive.

Next Step

Iterate on Variant C: keep the focus on annual plan value, but make the upfront billing explicit (e.g., “$9.99/mo, billed annually at $119.99”) and test clear annual vs. monthly side-by-side positioning rather than hiding the monthly option.