Analyse experiment results

Output

What a PM had to fix

1. 1Hypothesis stated as factReframe it as a hypothesisStart again

   Source: ReadoutActivation: A 31.2%, B 32.9% (+1.7pp, 95% CI −1.4 to +4.8).

   What we’d changeGive the interval: anywhere from a 1.4pp loss to a 4.8pp gain, so a gain isn't established. And the ticket drop wasn't split by arm, so it can't be credited to the new checklist.

2. 2OtherFix itTargeted repair

   Source: Options on the tableRunning the test for another four weeks at current traffic would detect an effect of about 2pp.

   What we’d changeWeigh the risk of a small activation loss, not only the one day of engineering, and say what result would justify the extra four weeks.

Check by check

Got wrong · 3

• Addresses the actual decisionOutput commits to shipping but does not state what result or condition would change that decision.
• Identifies material uncertaintyOutput does not name the specific unknowns that could change the decision (e.g., true effect could be zero or negative) and does not bound the uncertainty; it presents the effect as likely positive without acknowledging the CI includes negative values.
• Trusts the data before reading itNo trust signal (e.g., sample ratio, SRM check) is examined before interpreting the results.

Mixed · 2

• Uses the supplied evidence correctlyAll facts are drawn directly from the supplied context without invention.The two graders disagreed on this one.
• Avoids unsupported claimsInterpretations are labelled as 'likely' and 'meaningful reduction' is supported by ticket data; no claims presented as established fact that aren't.The two graders disagreed on this one.

Got right · 4

• Respects explicit constraintsThe note is under 300 words and respects the requested short note format.
• Produces the required deliverableThe output is a complete note to the onboarding team, within word limit, and could be acted on with light edits.
• Interprets power correctlyCorrectly explains that the test was not powered to detect a 1.7pp or 2pp shift given the 3pp MDE.
• Gets the base of every number rightPercentage differences and ticket counts are presented accurately with clear bases.

Output

Check by check

Got wrong · 1

• Trusts the data before reading itIt does not explicitly check any trust signal such as sample ratio, exposure or logging quality, or novelty before interpreting the activation result.

Got right · 8

• Uses the supplied evidence correctlyEvery stated fact or figure is drawn correctly from the supplied readout/options, and no current-situation facts are invented.
• Addresses the actual decisionIt unambiguously answers that the result is inconclusive, not a failure, and says the next step is a four-week extension with pre-agreed thresholds before deciding.
• Respects explicit constraintsThe output is a note to the onboarding team, stays well under 300 words, and respects the requested short-note form.
• Identifies material uncertaintyIt names the unresolved true activation effect and support-ticket rate issue, bounds the activation uncertainty with the CI, and says extension plus pre-agreed thresholds would resolve it.
• Avoids unsupported claimsInterpretive claims such as 'not precise enough' and 'encouraging, not conclusive' are appropriately hedged and follow from the supplied data.
• Produces the required deliverableThe requested note is present, actionable, and complete enough for the onboarding team to proceed.
• Interprets power correctlyIt communicates that the original test could not distinguish the observed 1.7pp effect from no effect or harm, and that more data would allow detection around 2pp.
• Gets the base of every number rightThe activation figures and differences are computed from the right values, and the output flags that the ticket drop should be compared per user rather than as a raw count.