Tasks / Metrics & Experimentation

Analyse experiment results

Can the model separate evidence from speculation, identify decision-relevant uncertainty and recommend a sensible next action?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 56% were usable with at most a quick edit.

Reliably right

Respects explicit constraints100% pass
The note is addressed to the onboarding team, stays under 300 words, and proposes actions that would enforce the stated constraints.
GPT-6 Astra · ChatGPT · The underpowered onboarding test
Interprets power correctly97% pass
It correctly explains that the test was designed to detect a 3pp effect, that the CI includes both harm and benefit, and that extending would detect about 2pp.
GPT-6 Astra · ChatGPT · The underpowered onboarding test
Catches the sample ratio mismatch89% pass
The output computes the split (53.6% treatment), calls it a sample ratio mismatch, and states the cause must be found before trusting the result.
Opus 5.5 · Claude · Up 6%, and significant

Where it slips

Uses the supplied evidence correctly50% pass
States that the change is easy to reverse, which is not supported by the supplied context.
Haiku 5.5 · API · The underpowered onboarding test
Avoids unsupported claims56% pass
Presents 'the change is easy to reverse' as fact without evidence from the brief or context.
Haiku 5.5 · API · The underpowered onboarding test
Trusts the data before reading it60% pass
Does not check any trust signal (e.g., sample ratio, exposure) before interpreting the results.
GPT-6 Luna · API · The underpowered onboarding test

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Trusts the data before reading it
Before interpreting effects, does the analysis check whether the result is trustworthy (sample ratio, exposure or logging issues, novelty, a surprisingly large lift) and say what it found?
Passes when Explicitly checks at least one trust signal in the supplied data, such as the split against the intended ratio, and treats a surprising result as a reason to investigate before acting.
Ronny KohaviLenny’s Podcast
Doesn't overread an unclear result
Does the analysis state honestly what the data can and can't show: no win claimed from an early peek or a result that isn't significant, and no 'no effect' concluded from a test too small to detect one?
Passes when Every claim matches the strength of the evidence, and anything unclear is called unclear, with what it can't rule out.
Ronny KohaviLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 1 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

We tested a shorter onboarding checklist and the result came back 'not significant'. The team wants to call it a failure and move on. Write a short note to the onboarding team saying what we should conclude from this test and what we should do next. Keep it under 300 words.

What the model was given4 items: Scenario, Readout, Qualitative, Options on the table

ScenarioTrellis is a project-management tool for small teams. The shorter checklist cuts onboarding from nine steps to five. It is already built: shipping it to everyone would take about a day of engineering. Activation means creating a first project and inviting at least one teammate within seven days.

ReadoutActivation: A 31.2%, B 32.9% (+1.7pp, 95% CI −1.4 to +4.8). n = 3,960 per arm. The pre-registered minimum detectable effect was 3pp.

QualitativeSupport tickets tagged 'onboarding confusion' fell from 44 to 29 during the test.

Options on the tableRunning the test for another four weeks at current traffic would detect an effect of about 2pp.

What a strong answer doesThe answer key the graders mark against

Inconclusive, not a failure: the test could not detect effects smaller than ~3pp, and the point estimate is positive. Recommend a decision based on cost of shipping versus a longer test.

Critical failures (cap the score at 40)

Concludes the change has no effect

Task

Mid-level · v1.11

The outputs

9 setups answered this brief.

GPT-6 AstraChatGPT100Usable with a quick edit GPT-6.1 SolAPI95Usable with a quick edit GPT-6 LunaAPI91Usable with a quick edit Sonnet 5.5API91Usable with a quick edit Haiku 5.5API82Usable with a quick edit Opus 5.5Claude80Usable with a quick edit Gemini 3.5 Flash-LiteGemini70Usable with a quick edit Gemini 3.8 FlashAPI50Needs more work Gemini 4 ArgonAPI50Needs more work

Gemini 3.5 Flash-LiteGemini

Usable with a quick edit7 of 10 checks passed

The note is clear, evidence-based, and actionable; a PM could add a condition for changing the decision with a quick edit, and no re-checking of evidence or reasoning is needed.

Output

Hi Team,

Before we write off the shorter onboarding checklist as a failure, let's look at the full picture.

While the headline result was statistically non-significant, the numbers and qualitative feedback tell a more nuanced story:

Positive Directional Lift: Group B (5 steps) saw a 32.9% activation rate compared to 31.2% for Group A (9 steps)—a 1.7 percentage point increase.

Underpowered Test: Our pre-registered minimum detectable effect (MDE) was 3pp, meaning the test wasn't powered to reliably catch a 1.7pp or 2pp shift.

Reduced Support Friction: Support tickets tagged "onboarding confusion" dropped significantly from 44 to 29 during the test.

What we should conclude: The test wasn't a failure; it was inconclusive. We likely have a small positive gain accompanied by a meaningful reduction in user confusion.¹

What we should do next: Given that shipping this to everyone only takes about a day of engineering, we should ship it. Continuing to run the test for another four weeks just to chase statistical significance on a small effect isn't a wise use of time² when the engineering cost to launch is so low and support friction has already improved. Let's roll out the 5-step checklist and monitor downstream retention.

What a PM had to fix

1Hypothesis stated as factReframe it as a hypothesisStart again
Source: ReadoutActivation: A 31.2%, B 32.9% (+1.7pp, 95% CI −1.4 to +4.8).
What we’d changeGive the interval: anywhere from a 1.4pp loss to a 4.8pp gain, so a gain isn't established. And the ticket drop wasn't split by arm, so it can't be credited to the new checklist.
2OtherFix itTargeted repair
Source: Options on the tableRunning the test for another four weeks at current traffic would detect an effect of about 2pp.
What we’d changeWeigh the risk of a small activation loss, not only the one day of engineering, and say what result would justify the extra four weeks.

Grades and run details

Decision model 65 · LLM judge 73

Decision model checks

passUses the supplied evidence correctly58%
failAddresses the actual decision83%
passRespects explicit constraints85%
partialIdentifies material uncertainty74%
partialAvoids unsupported claims31%
passProduces the required deliverable58%
passInterprets power correctly73%
failTrusts the data before reading it86%
passGets the base of every number right27%
partialDoesn't overread an unclear result64%

Run

Run: #1
Time to output: 6 s
Submitted: 25 Sept 2026

Opus 5.5Claude

Usable with a quick edit7 of 10 checks passed

The unsupported claim about ticket splitting is minor and can be fixed with a quick edit; the core reasoning and recommendation remain sound and usable.

Output

Shorter checklist test — what it tells us and what's next

Hi all,

The shorter onboarding checklist came back "not significant." That doesn't make it a failure, and I don't think we should drop it.

What the test actually says

B activated at 32.9% vs 31.2% for A, a +1.7pp lift.
The 95% interval runs from −1.4pp to +4.8pp. Most of that range is positive, and the worst plausible case is a small loss.
We sized the test to detect 3pp. A real 1–2pp gain was always likely to come back "not significant." The result means the effect is smaller than 3pp or zero.¹ It doesn't mean the change didn't work.

Supporting signal

Support tickets tagged "onboarding confusion" fell from 44 to 29 during the test. We didn't split tickets by arm, so this is directional only. It does point the same way as the activation result.

Recommendation: ship it

It's already built, and rolling it out costs about a day.
The likely effect is neutral to positive, and five steps are simpler to maintain than nine.
Running four more weeks would only detect effects of about 2pp. Our point estimate is 1.7pp, so we would probably still get an ambiguous answer, a month later.

Guardrails

Watch 7-day activation for the four weeks after rollout. If it drops more than about 1.5pp below the current baseline, roll back.
Split onboarding-confusion tickets by checklist version from now on, so next time we can tell whether a ticket drop is real.

For future tests

Before launch, agree what we'll do if the result comes back inconclusive. For a cheap, low-risk change like this, "not significant but probably not harmful" should default to ship.

Happy to discuss.

What a PM had to fix

1Numbers wrongRedo the arithmeticTargeted repair
Source: ReadoutActivation: A 31.2%, B 32.9% (+1.7pp, 95% CI −1.4 to +4.8).
What we’d changeCorrect the reading: the interval runs from −1.4 to +4.8pp, so an effect above 3pp isn't ruled out. The test just wasn't sized to reliably detect smaller ones.

Claims the judge couldn’t find in the brief

We didn't split tickets by arm.

Grades and run details

Decision model 85 · LLM judge 73

Decision model checks

passUses the supplied evidence correctly27%
passAddresses the actual decision86%
passRespects explicit constraints46%
passIdentifies material uncertainty98%
partialAvoids unsupported claims20%
passProduces the required deliverable69%
passInterprets power correctly100%
failTrusts the data before reading it74%
passGets the base of every number right56%
passDoesn't overread an unclear result48%

Run

Run: #1
Time to output: 13 s
Submitted: 25 Sept 2026

Haiku 5.5API

Usable with a quick edit7 of 10 checks passed

The core reasoning is sound; the unsupported claim about reversibility can be edited out or qualified without redoing the analysis.

Output

To: Onboarding team Re: Shorter checklist test

Conclusion: This wasn't a failure. It was inconclusive, and the result doesn't support "no effect."

The point estimate was +1.7pp (32.9% vs. 31.2%).
The 95% CI runs from −1.4pp to +4.8pp. Effects from slightly negative to clearly useful are all consistent with the data.
The test was powered for a 3pp lift. A real 1–2pp improvement, which would still be worth having, would usually not reach significance at this sample size.
Support tickets for "onboarding confusion" fell from 44 to 29. This wasn't a pre-registered metric, and counts this small are noisy, but it points the same way.

Recommendation: Don't treat this as a loss, and don't run the test indefinitely.

Ship the five-step checklist to everyone, with a 10% holdout on the old flow. Shipping costs about one day of engineering, the change is easy to reverse, and the evidence leans positive. The holdout keeps measuring the real effect at no extra build cost.
Decide the extension question on decision value, not on the test result. Another four weeks would detect roughly 2pp. That is worth doing only if a 2pp lift would change what we do next. If it wouldn't, the holdout gives us enough information.
If we extend, make it a new pre-registered analysis. Set the new target effect and sample size now, and don't keep peeking at the current data until it crosses significance. Changing the goalposts after an inconclusive result is how teams end up with false positives.

Bottom line: The checklist is probably neutral to helpful, and it is cheap to keep. Ship it with a holdout, track activation and tickets, and revisit in four weeks.

Claims the judge couldn’t find in the brief

The change is easy to reverse

Grades and run details

Decision model 90 · LLM judge 73

Decision model checks

passUses the supplied evidence correctly33%
passAddresses the actual decision20%
passRespects explicit constraints53%
passIdentifies material uncertainty98%
passAvoids unsupported claims46%
passProduces the required deliverable79%
passInterprets power correctly100%
failTrusts the data before reading it52%
passGets the base of every number right56%
passDoesn't overread an unclear result72%

Run

Run: #1
API response time: 7 s
Submitted: 8 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Uses the supplied evidence correctlyRightMixedMixed

Gemini 3.5 Flash-Lite · Gemini

All factual claims are taken directly from the supplied context or follow from arithmetic; nothing is invented.

Opus 5.5 · Claude

Claims that tickets were not split by arm is not stated in the context and is presented as fact.

Haiku 5.5 · API

States that the change is easy to reverse, which is not supported by the supplied context.

Addresses the actual decisionWrongRightRight

Gemini 3.5 Flash-Lite · Gemini

The output commits to shipping but does not state what result or condition would change that decision.

Opus 5.5 · Claude

Commits to 'ship it' early, and specifies a guardrail (activation drop >1.5pp) that would trigger rollback.

Haiku 5.5 · API

Commits early to 'ship the five-step checklist' with a holdout, framed for the onboarding team.

Identifies material uncertaintyWrongRightRight

Gemini 3.5 Flash-Lite · Gemini

It does not name specific unknowns that could change the decision, bound them, or say how they would be resolved.

Opus 5.5 · Claude

Names the undetectable small effect, the directional-only ticket data, and the CI range; says what would change the call.

Haiku 5.5 · API

Names the CI and effect-size uncertainty, and says extending is worth doing only if a 2pp lift would change what they do.

Avoids unsupported claimsRightWrongMixed

Gemini 3.5 Flash-Lite · Gemini

Interpretations are labelled as 'likely' and not presented as established fact.

Opus 5.5 · Claude

Presents 'We didn't split tickets by arm' as an established fact when the context only gives total ticket counts.

Haiku 5.5 · API

Presents 'the change is easy to reverse' as fact without evidence from the brief or context.

All got wrong 1

Trusts the data before reading itWrongWrongWrong

Gemini 3.5 Flash-Lite · Gemini

No trust signal (e.g., sample ratio, exposure check) is examined before interpreting the data.

Opus 5.5 · Claude

Does not check any trust signal (e.g., sample ratio, exposure) before interpreting the result.

Haiku 5.5 · API

Does not explicitly check any trust signal (e.g. split ratio, exposure) before interpreting the data.

All got right 5

Respects explicit constraintsRightRightRight

Gemini 3.5 Flash-Lite · Gemini

The note is under 300 words, addressed to the onboarding team, and respects the requested form.

Opus 5.5 · Claude

Output is a short note to the onboarding team, under 300 words, addressing conclusion and next steps.

Haiku 5.5 · API

Under 300 words (approx. 266), addressed to the onboarding team, note format.

Produces the required deliverableRightRightRight

Gemini 3.5 Flash-Lite · Gemini

The note is complete, under 300 words, and a PM could act on it with light edits.

Opus 5.5 · Claude

A complete, actionable note to the onboarding team within the word limit.

Haiku 5.5 · API

Short note with conclusion and clear next steps; a PM could act on it with light edits.

Interprets power correctlyRightRightRight

Gemini 3.5 Flash-Lite · Gemini

It correctly explains that the test was underpowered to detect effects smaller than the 3pp MDE.

Opus 5.5 · Claude

Explains the test could only detect ≥3pp, so a 1-2pp real gain would likely be non-significant.

Haiku 5.5 · API

Explains the test could detect 3pp, and that a real 1-2pp improvement would not reach significance.

Gets the base of every number rightRightRightRight

Gemini 3.5 Flash-Lite · Gemini

All percentages and differences are computed correctly, and the base (activation rate) is clear.

Opus 5.5 · Claude

Percentages and differences are correctly stated; ticket numbers are absolute, not misrepresented as rates.

Haiku 5.5 · API

Percentages and differences match the readout; no misstated denominators.

Doesn't overread an unclear resultRightRightRight

Gemini 3.5 Flash-Lite · Gemini

It does not claim a win or 'no effect'; it correctly calls the result inconclusive and does not overread the non-significant outcome.

Opus 5.5 · Claude

Does not claim a win or conclude no effect; correctly calls the result inconclusive and states what it can't rule out.

Haiku 5.5 · API

Calls the result inconclusive, not evidence of no effect, and states what the CI can and cannot rule out.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 60% of 40 checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	GPT-6 AstrawithChatGPT	91.7	100.0	2	None
2	Sonnet 5.5withAPI	84.6	95.5	2	None
3	Opus 5.5withClaude	88.3	86.4	2	None
4	GPT-6.1 SolwithAPI	76.3	80.8	2	None
5	Haiku 5.5withAPI	82.5	71.0	2	None
6	GPT-6 LunawithAPI	78.3	64.7	2	None
7	Gemini 4 ArgonwithAPI	68.8	65.7	2	None
8	Gemini 3.5 Flash-LitewithGemini	53.3	51.7	2	None
9	Gemini 3.8 FlashwithAPI	63.3	45.8	2	None

What you get for the cost

Sonnet 5.5 · API scores within 5 points of GPT-6 Astra · ChatGPT at a third of the cost.

Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
GPT-6 Luna · API	70	1.0×
Haiku 5.5 · API	76	1.8×
Gemini 3.5 Flash-Lite · Gemini	53	3.1× (partly estimated)
Gemini 3.8 Flash · API	52	12×
GPT-6.1 Sol · API	78	14×
Gemini 4 Argon · API	64	14×
Sonnet 5.5 · API	91	35×
Opus 5.5 · Claude	88	95× (partly estimated)
GPT-6 Astra · ChatGPT	96	107× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Reading a test readout and deciding what to do next.

Why it matters

Experiment readouts are where false confidence is cheapest to produce and most expensive to act on. A model that declares a winner on conversion while retention quietly falls will ship the wrong thing.

What good looks like

Checks the result can be trusted before interpreting it
Reports effects with their uncertainty
Checks guardrail metrics before declaring a winner
Separates what the data shows from plausible explanations
Recommends a next step proportionate to the evidence

Deliberately not measured

Re-running the statistics from raw data
Chart production

Capability tested

Interpreting results within their limits

The failure we’re looking for

Claims causality, ignores guardrails or recommends arbitrary testing

Grading

Decision model and LLM judge, calibrated against a blind PM review

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

Trusts the data before reading it

Doesn't overread an unclear result

Plus our standard checks

The tasks

The outputs

Output

What a PM had to fix

Grades and run details

Decision model checks

Run

Output

Shorter checklist test — what it tells us and what's next

What the test actually says

Supporting signal

Recommendation: ship it

Guardrails

For future tests

What a PM had to fix

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Run

Output

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Run

Check by check

Where they differ 4

All got wrong 1

All got right 5

Results

What you get for the cost

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured