Analyse experiment results

Output

Check by check

Got wrong · 2

• Avoids unsupported claimsIt asserts the refund surge 'isn't noise' and that annual up-front billing means retention 'should rise' without supporting evidence or clear labeling.
• Trusts the data before reading itIt does not check any trust signal such as sample ratio, exposure, logging issues, or novelty before interpreting the results.

Got right · 8

• Uses the supplied evidence correctlyAll core metrics, guardrail numbers, and CI values are quoted correctly from the supplied context, with no invented current-situation facts.
• Addresses the actual decisionIt clearly and immediately says don't ship B to 100%, names Priya's decision, and says a B2 that holds conversion and passes guardrails would change the call.
• Respects explicit constraintsIt stays under 400 words, addresses Priya, leads with the recommendation, and proposes enforcement of guardrails via follow-up test and rollback trigger.
• Identifies material uncertaintyIt names missing plan mix, refund reasons, and whether revenue is net of refunds, and says how to resolve them or what would change the call.
• Produces the required deliverableIt provides a usable, complete recommendation with reasoning and next steps for Priya within the length limit.
• Checks guardrails before declaring a winnerIt evaluates both pre-agreed guardrails explicitly, comparing refunds to the 10% limit and retention to the 2pp limit.
• Separates effect from explanationThe buyer-surprise mechanism is labeled as a hypothesis rather than a finding, and alternative explanations are framed with 'may' and 'suggests'.
• Gets the base of every number rightPercentage differences and refund increase are computed from the correct bases and are consistent with the supplied data.

Claims the judge couldn’t find in the brief

• At this volume the refund increase is not noise.
• If B pushes people toward the annual plan, day-30 retention should rise rather than fall because the annual plan is billed up front.

Output

What a PM had to fix

1. 1Numbers wrongRedo the arithmeticTargeted repair

   Source: Readout summaryRevenue per trial start at day 30: A $9.42, B $9.61 (CI crosses zero).

   What we’d changeAn interval crossing zero means the revenue lift is uncertain, not absent. Say that instead.

2. 2Contradiction missedSurface the contradictionQuick edit

   Source: Readout summaryTest ran 21 days

   What we’d changeAsk how a 21-day test reports day-30 retention before relying on the figure.

3. 3Hypothesis stated as factReframe it as a hypothesisQuick edit

   What we’d changeLabel this as the likely explanation to confirm from refund reasons, as the next sentence half does.

4. 4Invented evidenceVerify or remove the claimQuick edit

   What we’d changeCut the regulatory claim. It isn't in the evidence, and it gives the room something to argue about.

Check by check

Got wrong · 3

• Uses the supplied evidence correctlyOutput presents unsupported claims about regulatory scrutiny and the revenue number flattering B, which are not in the brief and not derived by arithmetic.
• Avoids unsupported claimsThe claim about the revenue number flattering B is presented as fact without being labeled as a hypothesis, and the regulatory risk claim is stated as a fact with no support.
• Trusts the data before reading itNo trust signal check is performed; the analysis does not verify the sample split ratio or any other data-quality check before interpreting effects.

Mixed · 1

• Separates effect from explanationThe explanation that the revenue number flatters B is not labelled as a hypothesis and is presented as a finding.The two graders disagreed on this one.

Got right · 6

• Addresses the actual decisionRecommends clearly not to ship B and names the decision-maker, with a condition for the next variant (if C keeps lift within guardrails).
• Respects explicit constraintsThe output is well under 400 words and addresses the head of growth as requested.
• Identifies material uncertaintyIdentifies that the revenue lift is not significant, suggests checking refund reasons to confirm hypothesis, and proposes following variant to reduce uncertainty.
• Produces the required deliverableThe deliverable is a complete recommendation with reasoning and next steps, suitable for the named reader.
• Checks guardrails before declaring a winnerExplicitly evaluates both pre-agreed guardrails and confirms they are breached.
• Gets the base of every number rightAll percentages and differences are correctly taken from the supplied data, and the base for the refund increase is clear.

Claims the judge couldn’t find in the brief

• The day-30 revenue number flatters B.
• Showing a monthly-equivalent price while billing annually, with the cheaper-looking commitment-free option hidden, is exactly the kind of pricing presentation the CMA and FTC scrutinise.