Analyse experiment results

Output

What a PM had to fix

1. 1Decision deferredMake the callTargeted repair

   Source: Guardrails agreed before launchDay-30 retention must not fall more than 2pp. Refund requests must not rise more than 10%.

   What we’d changeMake the argument rather than hedging it: the point estimate breaches the guardrail and most of the interval is past it. The refund question can be settled from the numbers given, so settle it.

2. 2Contradiction missedSurface the contradictionQuick edit

   Source: ScenarioThe annual plan is $119.99, billed up front

   What we’d changeMake the plan-mix point: annual plans are paid up front, so if B moved people onto annual, day-30 retention should have risen on its own. That it fell strengthens the case against B.

Check by check

Got right · 10

• Uses the supplied evidence correctlyThe figures and current-state statements are taken correctly from the supplied context or follow by arithmetic.
• Addresses the actual decisionIt commits early to not shipping B, returning new traffic to A, and testing a revised page, while making clear the guardrails are the gating conditions.
• Respects explicit constraintsIt addresses Priya, stays under the requested length, and its recommendation respects the agreed guardrails rather than overriding them.
• Identifies material uncertaintyIt names the material unknowns—retention follow-up maturity, refund denominator, plan-level results—and says how to resolve them.
• Avoids unsupported claimsThe pricing-misunderstanding explanation is explicitly labelled plausible and worth testing rather than stated as fact.
• Produces the required deliverableIt is a usable decision memo for the named reader with a clear recommendation, reasoning, and next steps.
• Checks guardrails before declaring a winnerIt evaluates both the day-30 retention guardrail and the refund-request guardrail before making a call.
• Separates effect from explanationIt separates the observed conversion and retention effects from the hypothesised pricing-misunderstanding mechanism.
• Trusts the data before reading itIt checks a data-trust issue by flagging that a 21-day test cannot fully support day-30 retention for all users.
• Gets the base of every number rightThe refund increase and revenue difference are computed correctly, and the retention CI interpretation uses the correct 2pp threshold.

Output

What a PM had to fix

1. 1Numbers wrongRedo the arithmeticTargeted repair

   Source: Readout summaryRevenue per trial start at day 30: A $9.42, B $9.61 (CI crosses zero).

   What we’d changeAn interval crossing zero means the revenue lift is uncertain, not absent. Say that instead.

2. 2Contradiction missedSurface the contradictionQuick edit

   Source: Readout summaryTest ran 21 days

   What we’d changeAsk how a 21-day test reports day-30 retention before relying on the figure.

3. 3Hypothesis stated as factReframe it as a hypothesisQuick edit

   What we’d changeLabel this as the likely explanation to confirm from refund reasons, as the next sentence half does.

4. 4Invented evidenceVerify or remove the claimQuick edit

   What we’d changeCut the regulatory claim. It isn't in the evidence, and it gives the room something to argue about.

Check by check

Got wrong · 3

• Uses the supplied evidence correctlyOutput presents unsupported claims about regulatory scrutiny and the revenue number flattering B, which are not in the brief and not derived by arithmetic.
• Avoids unsupported claimsThe claim about the revenue number flattering B is presented as fact without being labeled as a hypothesis, and the regulatory risk claim is stated as a fact with no support.
• Trusts the data before reading itNo trust signal check is performed; the analysis does not verify the sample split ratio or any other data-quality check before interpreting effects.

Mixed · 1

• Separates effect from explanationThe explanation that the revenue number flatters B is not labelled as a hypothesis and is presented as a finding.The two graders disagreed on this one.

Got right · 6

• Addresses the actual decisionRecommends clearly not to ship B and names the decision-maker, with a condition for the next variant (if C keeps lift within guardrails).
• Respects explicit constraintsThe output is well under 400 words and addresses the head of growth as requested.
• Identifies material uncertaintyIdentifies that the revenue lift is not significant, suggests checking refund reasons to confirm hypothesis, and proposes following variant to reduce uncertainty.
• Produces the required deliverableThe deliverable is a complete recommendation with reasoning and next steps, suitable for the named reader.
• Checks guardrails before declaring a winnerExplicitly evaluates both pre-agreed guardrails and confirms they are breached.
• Gets the base of every number rightAll percentages and differences are correctly taken from the supplied data, and the base for the refund increase is clear.

Claims the judge couldn’t find in the brief

• The day-30 revenue number flatters B.
• Showing a monthly-equivalent price while billing annually, with the cheaper-looking commitment-free option hidden, is exactly the kind of pricing presentation the CMA and FTC scrutinise.