Analyse experiment results

Recommendation: Don't ship B to 100% on Monday. B breached both guardrails we agreed on before launch. Keep the test running or revert to A, and bring a fix to the meeting instead.

Why

1. The guardrails were set to settle this. Refund requests rose 42% (212 → 301) against a 10% limit. That's a clear breach, and at this volume it isn't noise. Day-30 retention fell 4.5pp against a 2pp limit. The CI (−7.9 to −1.1) technically includes drops smaller than 2pp, but the point estimate is more than double the limit, so we can't claim it's safe.

1. The conversion win isn't yet a revenue win. Conversion is up 1.3pp and that's real. But revenue per trial start is $9.61 vs $9.42, and the CI crosses zero. We are converting more people, and more of them are leaving or asking for their money back.

1. The retention drop is surprising. If B pushes people toward the annual plan, which is billed up front, day-30 retention should rise, not fall. A drop alongside more refunds suggests some buyers are surprised by a $119.99 charge after seeing "$9.99/month." Hiding the monthly plan behind "See all plans" may also pull in lower-intent buyers. This is a hypothesis, not a finding, since the readout doesn't show plan mix.

1. Day 30 undersells the risk. Annual buyers who regret the purchase may refund later, and the readout can't show that.

What to do before Monday

• Pull plan mix (monthly vs. annual) by variant, plus retention and refund rates by plan.
• Check whether the $119.99 up-front charge is clear enough on B. Refund reasons or support tickets would help.
• Confirm whether revenue is net of refunds.

What to propose instead

Test a B2 that keeps the $9.99/month anchor but states "$119.99 billed today" next to it and keeps the monthly option visible. If B2 holds the conversion gain and stays within both guardrails, ship it.

Line for the meeting: "We set guardrails so we wouldn't have to debate this after the fact. B failed them, so we're fixing the billing clarity and retesting, not shipping a conversion gain that may be costing us customers and refunds."

If leadership insists on shipping B, the least risky option is a partial rollout with the refund rate monitored weekly and a pre-agreed rollback trigger. I wouldn't recommend it.

o: Priya Nair, Head of Growth**

Do not ship B to 100% on Monday. Return new traffic to A, continue following existing test cohorts and test a revised page with clearer pricing. B improves conversion, but the readout does not meet the conditions we agreed for rollout.

The conversion lift is credible: +1.3 percentage points, with a 95% confidence interval of +0.6 to +2.0. However, conversion alone cannot establish that B creates better customer or business outcomes.

The guardrails do not support launch. Reported day-30 retention falls 4.5pp against a maximum permitted decline of 2pp. Its confidence interval includes declines within our tolerance, so we cannot say the true loss definitively exceeds 2pp.¹ Equally, we have not established that B is within tolerance.

Refund requests rose from 212 to 301, or 42%, against a 10% limit. Confirm whether the agreed guardrail measures counts or rates, using comparable denominators and observation windows. B generates more paying customers, which could explain some additional requests, but the increase warrants investigation before expansion.

There is also a measurement issue: a 21-day test cannot give every participant 30 days of follow-up. Establish when this readout was produced, which cohorts matured and how retention is defined across monthly and annual plans². If measured immediately after the test, the day-30 figures cannot be complete.

Revenue per trial is only $0.19 higher, with uncertainty spanning zero. That does not establish a revenue benefit, and annual billing up front makes it especially important to distinguish cash collected from earned revenue and account for refunds.

B’s presentation offers a plausible explanation worth testing: “$9.99/month” may attract customers who do not understand the $119.99 upfront commitment. The readout does not prove this; inspect refund reasons and results by plan.

For the next test, display “$119.99 billed annually, equivalent to $9.99/month” prominently and keep the $14.99 monthly option readily visible. Retain the original guardrails and define mature cohort reporting before launch.

For Monday, the defensible decision is simple: the conversion gain earns further testing, but does not justify overriding our agreed customer safeguards.

To: Priya Nair, Head of Growth Recommendation: Do not ship B to 100% on Monday. Keep A as the default, stop new exposure to B, and retest a clearer pricing design.

B delivers a credible conversion lift, but it fails both pre-agreed guardrails. We should not override those thresholds because the headline metric improved.

• Conversion improved: Trial-to-paid rose from 11.8% to 13.1%, a +1.3 percentage-point lift (95% CI: +0.6 to +2.0).
• Retention deteriorated: Day-30 paid retention fell 4.5 points, from 84.0% to 79.5%, versus an allowed decline of no more than 2 points. The confidence interval excludes zero, although it does not establish that the true decline necessarily exceeds 2 points.
• Refunds breached the threshold: Requests increased from 212 to 301—42%, versus the permitted 10%. Confirm the rate using the agreed denominator, but the reported result is a clear warning under approximately equal exposure.
• Revenue does not justify the trade-off: Day-30 revenue per trial start increased only $0.19, and its confidence interval crosses zero. We have not established a revenue benefit.

The design provides a plausible explanation, not proven causation: leading with “$9.99/month” while hiding the monthly option may obscure that customers must pay $119.99 upfront. Higher conversion could therefore reflect misunderstanding rather than stronger purchase intent.

Next steps: Validate cohort maturity—the experiment ran 21 days, so we need to understand how day-30 outcomes were obtained—and check refund reasons, plan mix, and billing-related complaints. Then test a version that prominently states “$119.99 billed annually ($9.99/month equivalent)” and keeps the $14.99 monthly option visible.

Monday’s defensible position: “B increased conversion, but failed our customer-outcome guardrails without demonstrating a revenue gain. We’ll preserve the learning, fix pricing clarity, and require mature results that meet the agreed thresholds before scaling.”