Analyse experiment results

Hi Priya,

Recommendation: Do not ship Variant B to 100% on Monday. While it improves trial-to-paid conversion, it severely breaches our pre-agreed guardrails on both retention and refund requests, threatening long-term revenue health.

Here is the reasoning to defend holding off on Monday:

• Guardrail Breach on Retention: We agreed in advance that Day-30 paid retention must not drop by more than 2 percentage points. Variant B’s retention dropped by 4.5pp¹ (down to 79.5% from A's 84.0%, with a 95% CI of −7.9 to −1.1). This massive drop indicates we are converting users who aren’t genuinely committed to the subscription.²
• Severe Spike in Refunds: Refund requests jumped by 42% (301 requests for B vs. 212 for A), completely shattering our guardrail limit that refunds must not rise by more than 10%. This confirms that Variant B’s presentation³—leading with the $9.99/month equivalent and hiding the monthly plan—creates subscriber confusion or misleading expectations at checkout.
• Neutral Long-Term Revenue: Although Variant B boosts initial trial-to-paid conversion (+1.3pp) and shows a nominal lift in day-30 revenue per trial start ($9.61 vs. $9.42), the confidence interval crosses zero. Combined with the steep drop in retention and surge in refunds, the initial revenue gains are illusory⁴ and will likely erode over time.

Next Steps: Keep Variant A live. Instruct the growth team to iterate on the pricing presentation—addressing the confusion around the annual plan framing—without sacrificing user retention or triggering buyer remorse.

Recommendation: Do not ship Variant B. Keep Variant A live and do not roll B out to 100% on Monday.

While Variant B delivered an initial +1.3pp lift in trial-to-paid conversion, it violates both pre-agreed guardrails, creates bill shock, and delivers no statistically significant revenue upside.

Here is the reasoning to defend this decision on Monday:

1. Both Pre-Agreed Guardrails Were Blown

We agreed on strict risk thresholds before launch. Variant B failed both: * Day-30 Retention: Fell −4.5pp (down to 79.5%), more than double our maximum allowed drop of 2pp (95% CI: −7.9 to −1.1). * Refund Requests: Surged +42% (301 vs. 212), four times our 10% ceiling.

Overriding guardrails after the fact sets a dangerous precedent and invites compounding churn.

2. The Conversion Lift Is an Illusion Driven by Bill Shock

Variant B hides the monthly tier and frames the annual plan as “$9.99/month.” Users clearly believe they are committing to a flexible monthly rate, only to be charged $119.99 upfront. The 42% spike in refunds and immediate drop in 30-day retention prove that these are not high-intent, qualified subscribers—they are confused customers who churn or dispute the charge as soon as they review their statements.

3. Long-Term Economics Are Flat to Negative

• Day-30 revenue per trial start is statistically unchanged: $9.61 for B vs. $9.42 for A (CI crosses zero).
• When factored against the operational cost of processing 42% more refunds, payment dispute fees, customer support load, and downstream LTV decay from lower retention, Variant B will be net-negative over a 12-month horizon.
• App store ratings and brand trust will also suffer if users perceive our checkout flow as deceptive.

Next Step

Iterate on Variant C: keep the focus on annual plan value, but make the upfront billing explicit (e.g., “$9.99/mo, billed annually at $119.99”) and test clear annual vs. monthly side-by-side positioning rather than hiding the monthly option.

Subject: Pricing variant B — recommend we don't ship Monday

Priya,

Recommendation: don't roll B out to 100%. It breaches both guardrails we agreed before launch. Run a corrected variant (C) instead.

Why B fails

• Refunds: +42% against a 10% limit. This is the clearest breach, and it isn't close. (212 → 301 requests.)
• Day-30 retention: −4.5pp against a 2pp limit.² The confidence interval runs from −7.9 to −1.1. Most of that range is past the line, and none of it is positive.
• The upside isn't real in revenue terms.¹ Conversion rose 1.3pp, but revenue per trial start at day 30 ($9.61 vs $9.42) isn't statistically different from zero. We'd be taking on the harm for no demonstrable gain.
• The day-30 revenue number flatters B. Annual plans bill $119.99 up front, so a shift toward annual inflates early revenue. If the extra refunds and churn continue past day 30, B likely falls behind A on lifetime value.

What's probably happening

B leads with "$9.99/month" and hides the monthly plan. Some users think they're signing up for $9.99 a month, then get charged $119.99.³ The spike in refunds and early cancellations fits that pattern: the extra conversions are partly people who didn't understand what they bought. We can confirm this in a day by reading refund reasons and support tickets from B users.

This also creates a regulatory and reputational risk. Showing a monthly-equivalent price while billing annually, with the cheaper-looking commitment-free option hidden, is exactly the kind of pricing presentation the CMA and FTC scrutinise.⁴

How to defend this Monday

• We set the guardrails before seeing results so we wouldn't argue ourselves past them afterward. Overriding them now undermines every future test.
• "Conversion went up" measures the wrong thing. Revenue didn't move, and refunds and churn got worse.

Proposed next step: Variant C

• Keep the annual-first emphasis, which is likely what drove the real part of the lift.
• Show "$119.99 billed annually" next to the $9.99/month equivalent, at equal prominence.
• Keep the monthly plan visible on the page.
• Use the same guardrails, and track revenue per trial through day 60 so annual billing doesn't distort the result.

If C keeps most of the conversion lift within the guardrails, we ship that.

Martin