Tasks / Metrics & Experimentation

Analyse experiment results

Can the model separate evidence from speculation, identify decision-relevant uncertainty and recommend a sensible next action?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 63% were usable with at most a quick edit.

Reliably right

Respects explicit constraints100% pass
The note is addressed to the onboarding team, stays under 300 words, and proposes actions that would enforce the stated constraints.
GPT-6 Astra · ChatGPT · The underpowered onboarding test
Interprets power correctly100% pass
It correctly explains that the test was designed to detect a 3pp effect, that the CI includes both harm and benefit, and that extending would detect about 2pp.
GPT-6 Astra · ChatGPT · The underpowered onboarding test
Addresses the actual decision92% pass
The output commits to a staged rollout as the recommended action, and specifies conditions (commercial unacceptability of a 1.4pp loss or value of a 2pp improvement) that would change the call to extending the test.
GPT-6 Astra · ChatGPT · The underpowered onboarding test

Where it slips

Uses the supplied evidence correctly56% pass
States that the change is easy to reverse, which is not supported by the supplied context.
Haiku 5.5 · API · The underpowered onboarding test
Gets the base of every number right59% pass
The Week 1 lift figure of +4.0% is a miscalculation; the correct rate from the supplied data is about 5.6%.
Haiku 5.5 · API · Up 6%, and significant
Avoids unsupported claims59% pass
Presents 'the change is easy to reverse' as fact without evidence from the brief or context.
Haiku 5.5 · API · The underpowered onboarding test

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Trusts the data before reading it
Before interpreting effects, does the analysis check whether the result is trustworthy (sample ratio, exposure or logging issues, novelty, a surprisingly large lift) and say what it found?
Passes when Explicitly checks at least one trust signal in the supplied data, such as the split against the intended ratio, and treats a surprising result as a reason to investigate before acting.
Ronny KohaviLenny’s Podcast
Doesn't overread an unclear result
Does the analysis state honestly what the data can and can't show: no win claimed from an early peek or a result that isn't significant, and no 'no effect' concluded from a test too small to detect one?
Passes when Every claim matches the strength of the evidence, and anything unclear is called unclear, with what it can't rule out.
Ronny KohaviLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 1 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

We tested a shorter onboarding checklist and the result came back 'not significant'. The team wants to call it a failure and move on. Write a short note to the onboarding team saying what we should conclude from this test and what we should do next. Keep it under 300 words.

What the model was given4 items: Scenario, Readout, Qualitative, Options on the table

ScenarioTrellis is a project-management tool for small teams. The shorter checklist cuts onboarding from nine steps to five. It is already built: shipping it to everyone would take about a day of engineering. Activation means creating a first project and inviting at least one teammate within seven days.

ReadoutActivation: A 31.2%, B 32.9% (+1.7pp, 95% CI −1.4 to +4.8). n = 3,960 per arm. The pre-registered minimum detectable effect was 3pp.

QualitativeSupport tickets tagged 'onboarding confusion' fell from 44 to 29 during the test.

Options on the tableRunning the test for another four weeks at current traffic would detect an effect of about 2pp.

What a strong answer doesThe answer key the graders mark against

Inconclusive, not a failure: the test could not detect effects smaller than ~3pp, and the point estimate is positive. Recommend a decision based on cost of shipping versus a longer test.

Critical failures (cap the score at 40)

Concludes the change has no effect

Task

Mid-level · v1.10

The outputs

8 setups answered this brief.

GPT-6 AstraChatGPT100Usable with a quick edit GPT-6.1 SolAPI98Usable with a quick edit GPT-6 LunaAPI91Usable with a quick edit Sonnet 5.5API89Usable with a quick edit Haiku 5.5API80Usable with a quick edit Opus 5.5Claude73Usable with a quick edit Gemini 3.5 Flash-LiteGemini66Usable with a quick edit Gemini 3.8 FlashAPI50Needs more work

Haiku 5.5API

Usable with a quick edit7 of 10 checks passed

The core reasoning is sound; the unsupported claim about reversibility can be edited out or qualified without redoing the analysis.

Output

To: Onboarding team Re: Shorter checklist test

Conclusion: This wasn't a failure. It was inconclusive, and the result doesn't support "no effect."

The point estimate was +1.7pp (32.9% vs. 31.2%).
The 95% CI runs from −1.4pp to +4.8pp. Effects from slightly negative to clearly useful are all consistent with the data.
The test was powered for a 3pp lift. A real 1–2pp improvement, which would still be worth having, would usually not reach significance at this sample size.
Support tickets for "onboarding confusion" fell from 44 to 29. This wasn't a pre-registered metric, and counts this small are noisy, but it points the same way.

Recommendation: Don't treat this as a loss, and don't run the test indefinitely.

Ship the five-step checklist to everyone, with a 10% holdout on the old flow. Shipping costs about one day of engineering, the change is easy to reverse, and the evidence leans positive. The holdout keeps measuring the real effect at no extra build cost.
Decide the extension question on decision value, not on the test result. Another four weeks would detect roughly 2pp. That is worth doing only if a 2pp lift would change what we do next. If it wouldn't, the holdout gives us enough information.
If we extend, make it a new pre-registered analysis. Set the new target effect and sample size now, and don't keep peeking at the current data until it crosses significance. Changing the goalposts after an inconclusive result is how teams end up with false positives.

Bottom line: The checklist is probably neutral to helpful, and it is cheap to keep. Ship it with a holdout, track activation and tickets, and revisit in four weeks.

Claims the judge couldn’t find in the brief

The change is easy to reverse

Grades and run details

Decision model 85 · LLM judge 73

Decision model checks

passUses the supplied evidence correctly22%
passAddresses the actual decision69%
passRespects explicit constraints53%
passIdentifies material uncertainty100%
partialAvoids unsupported claims59%
passProduces the required deliverable88%
passInterprets power correctly100%
failTrusts the data before reading it52%
passGets the base of every number right73%
passDoesn't overread an unclear result64%

Run

Run: #1
API response time: 7 s
Submitted: 8 Oct 2026

GPT-6 LunaAPI

Usable with a quick edit9 of 10 checks passed

The output is clear and actionable; a PM could use it with minor edits, and the missing trust check is a small omission that does not undermine the core reasoning.

Output

Team—

This is inconclusive, not a failure. Activation was 1.7 percentage points higher with the shorter checklist, but the 95% CI (−1.4 to +4.8 points) includes no effect and possible harm—as well as a benefit of at least 3 points. The test wasn’t precise enough to settle the question.

I recommend running the test for another four weeks. That should let us assess effects around 2 points, which this change may plausibly deliver. Before extending, agree on what activation gain would justify rollout and what activation decline would stop it. Track onboarding-confusion tickets too, but compare rates per user: the raw drop from 44 to 29 is encouraging, not conclusive.

After the extension, decide using both activation and confusion—not a “significant/not significant” label alone.

Grades and run details

Decision model 90 · LLM judge 91

Decision model checks

passUses the supplied evidence correctly82%
passAddresses the actual decision96%
passRespects explicit constraints96%
passIdentifies material uncertainty97%
passAvoids unsupported claims49%
passProduces the required deliverable62%
passInterprets power correctly89%
failTrusts the data before reading it27%
passGets the base of every number right95%
passDoesn't overread an unclear result91%

Run

Run: #1
API response time: 9 s
Submitted: 29 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 2

Uses the supplied evidence correctlyMixedRight

Haiku 5.5 · API

States that the change is easy to reverse, which is not supported by the supplied context.

GPT-6 Luna · API

All factual statements about the current situation are taken directly from the supplied context.

Avoids unsupported claimsWrongRight

Haiku 5.5 · API

Presents 'the change is easy to reverse' as fact without evidence from the brief or context.

GPT-6 Luna · API

Interpretations are clearly labelled as such, and no unsupported claims are presented as fact.

All got wrong 1

Trusts the data before reading itWrongWrong

Haiku 5.5 · API

Does not explicitly check any trust signal (e.g. split ratio, exposure) before interpreting the data.

GPT-6 Luna · API

Does not check any trust signal (e.g., sample ratio, exposure) before interpreting the results.

All got right 7

Addresses the actual decisionRightRight

Haiku 5.5 · API

Commits early to 'ship the five-step checklist' with a holdout, framed for the onboarding team.

GPT-6 Luna · API

Commits to extending the test and specifies that the decision after extension will use both activation and confusion data.

Respects explicit constraintsRightRight

Haiku 5.5 · API

Under 300 words (approx. 266), addressed to the onboarding team, note format.

GPT-6 Luna · API

The note is under 300 words, addressed to the team, and respects the requested form.

Identifies material uncertaintyRightRight

Haiku 5.5 · API

Names the CI and effect-size uncertainty, and says extending is worth doing only if a 2pp lift would change what they do.

GPT-6 Luna · API

Identifies the imprecision of the test and proposes a longer test to resolve the uncertainty.

Produces the required deliverableRightRight

Haiku 5.5 · API

Short note with conclusion and clear next steps; a PM could act on it with light edits.

GPT-6 Luna · API

The note is complete, under 300 words, and the team could act on it without filling major gaps.

Interprets power correctlyRightRight

Haiku 5.5 · API

Explains the test could detect 3pp, and that a real 1-2pp improvement would not reach significance.

GPT-6 Luna · API

Explains that the test was not precise enough and that an extension would detect effects around 2pp.

Gets the base of every number rightRightRight

Haiku 5.5 · API

Percentages and differences match the readout; no misstated denominators.

GPT-6 Luna · API

All derived figures (1.7pp, CI, ticket drop) are correctly stated and no ambiguous bases are used.

Doesn't overread an unclear resultRightRight

Haiku 5.5 · API

Calls the result inconclusive, not evidence of no effect, and states what the CI can and cannot rule out.

GPT-6 Luna · API

Correctly calls the result inconclusive and does not claim a win or no effect from an underpowered test.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 63% of 40 checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	GPT-6 AstrawithChatGPT	95.8	100.0	2	None
2	Sonnet 5.5withAPI	88.3	95.5	2	None
3	Opus 5.5withClaude	80.8	86.4	2	None
4	GPT-6.1 SolwithAPI	89.2	80.8	2	None
5	Haiku 5.5withAPI	88.3	71.0	2	None
6	GPT-6 LunawithAPI	88.8	64.7	2	None
7	Gemini 3.8 FlashwithAPI	69.6	45.8	2	None
8	Gemini 3.5 Flash-LitewithGemini	54.6	51.7	2	None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Nobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
GPT-6 Luna · API	75	1.0×
Haiku 5.5 · API	78	1.8×
Gemini 3.5 Flash-Lite · Gemini	53	3.1× (partly estimated)
Gemini 3.8 Flash · API	55	12×
GPT-6.1 Sol · API	83	14×
Sonnet 5.5 · API	92	35×
Opus 5.5 · Claude	84	95× (partly estimated)
GPT-6 Astra · ChatGPT	98	107× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Reading a test readout and deciding what to do next.

Why it matters

Experiment readouts are where false confidence is cheapest to produce and most expensive to act on. A model that declares a winner on conversion while retention quietly falls will ship the wrong thing.

What good looks like

Checks the result can be trusted before interpreting it
Reports effects with their uncertainty
Checks guardrail metrics before declaring a winner
Separates what the data shows from plausible explanations
Recommends a next step proportionate to the evidence

Deliberately not measured

Re-running the statistics from raw data
Chart production

Capability tested

Interpreting results within their limits

The failure we’re looking for

Claims causality, ignores guardrails or recommends arbitrary testing

Grading

Decision model and LLM judge, calibrated against a blind PM review

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

Trusts the data before reading it

Doesn't overread an unclear result

Plus our standard checks

The tasks

The outputs

Output

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Run

Output

Grades and run details

Decision model checks

Run

Check by check

Where they differ 2

All got wrong 1

All got right 7

Results

What you get for the cost

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured