Tasks / Experiment

Analyse experiment results

Can the model separate evidence from speculation, identify decision-relevant uncertainty and recommend a sensible next action?

Measures the modelTask v1.2 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 14 graded outputs by 7 models. 71% were usable with at most a quick edit.

Reliably right

Gets the base of every number right100% pass
The refund increase and revenue difference are computed correctly, and the retention CI interpretation uses the correct 2pp threshold.
GPT-6 Astra · ChatGPT · Conversion up, retention down
Interprets power correctly100% pass
Explains that the test was designed to detect ~3pp effects, so a smaller but worthwhile lift could still exist and remain undetected.
GPT-6 Astra · ChatGPT · The underpowered onboarding test
Checks guardrails before declaring a winner100% pass
It evaluates both the day-30 retention guardrail and the refund-request guardrail before making a call.
GPT-6 Astra · ChatGPT · Conversion up, retention down

Where it slips

Trusts the data before reading it21% pass
It does not explicitly check a pre-interpretation trust signal such as sample ratio or logging before relying on the lift; the denominator and maturity caveats come after interpreting.
GPT-6.1 Sol · API · Conversion up, retention down
Avoids unsupported claims52% pass
It asserts the refund surge 'isn't noise' and that annual up-front billing means retention 'should rise' without supporting evidence or clear labeling.
Sonnet 5.5 · API · Conversion up, retention down
Uses the supplied evidence correctly55% pass
Asserts that comparable per-arm ticket rates and exposure counts are unavailable, which is not in the supplied context and cannot be assumed.
GPT-6 Astra · ChatGPT · The underpowered onboarding test

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

Our new pricing page variant (B) increased trial-to-paid conversion, and the growth team wants to ship it to 100% on Monday. Review the readout below and write your recommendation to Priya Nair, Head of Growth, who makes the call. Lead with what we should do, then give the reasoning she needs to defend it in the Monday meeting. Keep it under 400 words.

What the model was given5 items: Scenario, Readout summary, Variant description, Guardrails agreed before launch, Refunds

ScenarioLumen is a subscription app for learning languages. The monthly plan is $14.99. The annual plan is $119.99, billed up front, which works out at $9.99 a month.

Readout summaryTest ran 21 days, 50/50 split, 38,412 trial starts. Trial→paid conversion: A 11.8%, B 13.1% (+1.3pp, 95% CI +0.6 to +2.0). Day-30 paid retention: A 84.0%, B 79.5% (−4.5pp, 95% CI −7.9 to −1.1). Revenue per trial start at day 30: A $9.42, B $9.61 (CI crosses zero).

Variant descriptionVariant B leads with the annual plan's monthly-equivalent price ($9.99/month) and moves the monthly plan behind a 'See all plans' link.

Guardrails agreed before launchDay-30 retention must not fall more than 2pp. Refund requests must not rise more than 10%.

RefundsRefund requests during the test: A 212, B 301 (+42%).

What a strong answer doesThe answer key the graders mark against

Do not ship B to 100%. Conversion gain is real but guardrails are breached; the likely mechanism is buyers misreading the monthly-equivalent price. Recommend a follow-up variant that keeps the clarity gain without the misleading anchor, and quantify the revenue picture honestly.

Critical failures (cap the score at 40)

Recommends shipping B to 100%
States the mechanism as established fact

Case

Mid-level · v1.9 · anonymised real · pricing, guardrail breach, B2C SaaS

The outputs

7 setups answered this brief. Pick up to three to read side by side.

GPT-6 AstraChatGPT100Usable with a quick edit GPT-6.1 SolAPI93Usable with a quick edit GPT-6 LunaAPI93Usable with a quick edit Sonnet 5.5API82Usable with a quick edit Opus 5.5Claude61Needs more work Gemini 3.5 Flash-LiteGemini40Critical failure Gemini 3.8 FlashAPI32Critical failure

Gemini 3.8 FlashAPI

Needs more than a quick edit2 of 10 checks passed

A PM would need to remove unsupported causal and forecast claims and add decision-changing conditions. It also needs addressing to Priya before use.

Output

Recommendation: Do not ship Variant B. Keep Variant A live and do not roll B out to 100% on Monday.

While Variant B delivered an initial +1.3pp lift in trial-to-paid conversion, it violates both pre-agreed guardrails, creates bill shock, and delivers no statistically significant revenue upside.

Here is the reasoning to defend this decision on Monday:

1. Both Pre-Agreed Guardrails Were Blown

We agreed on strict risk thresholds before launch. Variant B failed both: * Day-30 Retention: Fell −4.5pp (down to 79.5%), more than double our maximum allowed drop of 2pp (95% CI: −7.9 to −1.1). * Refund Requests: Surged +42% (301 vs. 212), four times our 10% ceiling.

Overriding guardrails after the fact sets a dangerous precedent and invites compounding churn.

2. The Conversion Lift Is an Illusion Driven by Bill Shock

Variant B hides the monthly tier and frames the annual plan as “$9.99/month.” Users clearly believe they are committing to a flexible monthly rate, only to be charged $119.99 upfront. The 42% spike in refunds and immediate drop in 30-day retention prove that these are not high-intent, qualified subscribers—they are confused customers who churn or dispute the charge as soon as they review their statements.

3. Long-Term Economics Are Flat to Negative

Day-30 revenue per trial start is statistically unchanged: $9.61 for B vs. $9.42 for A (CI crosses zero).
When factored against the operational cost of processing 42% more refunds, payment dispute fees, customer support load, and downstream LTV decay from lower retention, Variant B will be net-negative over a 12-month horizon.
App store ratings and brand trust will also suffer if users perceive our checkout flow as deceptive.

Next Step

Iterate on Variant C: keep the focus on annual plan value, but make the upfront billing explicit (e.g., “$9.99/mo, billed annually at $119.99”) and test clear annual vs. monthly side-by-side positioning rather than hiding the monthly option.

Claims the judge couldn’t find in the brief

Users clearly believe they are committing to a flexible monthly rate, only to be charged $119.99 upfront.
The 42% refund spike and immediate drop in 30-day retention prove these are not high-intent, qualified subscribers but confused customers who churn or dispute the charge as soon as they review statements.
Variant B will be net-negative over a 12-month horizon after refund processing, dispute fees, support load, and LTV decay.
App store ratings and brand trust will suffer if users perceive the checkout flow as deceptive.

Grades and run details

Decision model 50 · LLM judge 2 of 11 checks

Decision model checks

failUses the supplied evidence correctly55%
passAddresses the actual decision55%
partialRespects explicit constraints22%
partialIdentifies material uncertainty32%
failAvoids unsupported claims90%
passProduces the required deliverable94%
passChecks guardrails before declaring a winner100%
failSeparates effect from explanation49%
failTrusts the data before reading it99%
passGets the base of every number right77%

Run

Run: #1
API response time: 22 s
Submitted: 1 Oct 2026

Check by check

Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Got wrong 6

Uses the supplied evidence correctlyWrong

The output adds unsupported current-situation claims about user beliefs, proof of confusion, and future net-negative economics.

Respects explicit constraintsWrong

It is under 400 words and leads with the recommendation, but it is not addressed to Priya Nair as requested.

Identifies material uncertaintyWrong

It does not name material unknowns that could change the decision or how they would be resolved.

Avoids unsupported claimsWrong

It presents the misreading/bill-shock mechanism and future harm as established fact rather than hypotheses.

Separates effect from explanationWrong

It treats the conversion lift as an illusion driven by bill shock and user confusion as proven, not as a hypothesis.

Trusts the data before reading itWrong

It does not check trust signals such as the 50/50 split, sample size, exposure, logging, or novelty before interpreting the result.

Mixed 2

Addresses the actual decisionMixed

It clearly recommends not shipping B, but does not state what result or condition would change that recommendation.

Produces the required deliverableMixed

It is a usable recommendation in form, but it is not addressed to the named reader and contains major unsupported claims.

Got right 2

Checks guardrails before declaring a winnerRight

It explicitly checks both pre-agreed guardrails: retention and refunds.

Gets the base of every number rightRight

The stated percentages, differences, and comparisons are computed from the supplied bases and are arithmetically sound.

Results

Every setup we’ve tested on this task, across all cases and repeats, graded on the current checklist. Calibrated: the graders match our PM on 88% of checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	GPT-6 AstrawithChatGPT	100.0	95.0	2	None
2	GPT-6.1 SolwithAPI	94.7	90.5	2	None
3	GPT-6 LunawithAPI	91.9	90.5	2	None
4	Sonnet 5.5withAPI	81.7	75.9	2	None
5	Opus 5.5withClaude	73.6	57.3	2	None
6	Gemini 3.5 Flash-LitewithGemini	60.3	57.7	2	1 capped
7	Gemini 3.8 FlashwithAPI	55.6	24.1	2	1 capped

The PM job

Reading a test readout and deciding what to do next.

Why it matters

Experiment readouts are where false confidence is cheapest to produce and most expensive to act on. A model that declares a winner on conversion while retention quietly falls will ship the wrong thing.

What good looks like

Checks the result can be trusted before interpreting it
Reports effects with their uncertainty
Checks guardrail metrics before declaring a winner
Separates what the data shows from plausible explanations
Recommends a next step proportionate to the evidence

Deliberately not measured

Re-running the statistics from raw data
Chart production

Capability tested

Interpreting results within their limits

The failure we’re looking for

Claims causality, ignores guardrails or recommends arbitrary testing

Grading

Decision model and LLM judge, calibrated against a blind PM review