Tasks / Metrics & Experimentation

Analyse experiment results

Can the model separate evidence from speculation, identify decision-relevant uncertainty and recommend a sensible next action?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 56% were usable with at most a quick edit.

Reliably right

Respects explicit constraints100% pass
The note is addressed to the onboarding team, stays under 300 words, and proposes actions that would enforce the stated constraints.
GPT-6 Astra · ChatGPT · The underpowered onboarding test
Interprets power correctly97% pass
It correctly explains that the test was designed to detect a 3pp effect, that the CI includes both harm and benefit, and that extending would detect about 2pp.
GPT-6 Astra · ChatGPT · The underpowered onboarding test
Catches the sample ratio mismatch89% pass
The output computes the split (53.6% treatment), calls it a sample ratio mismatch, and states the cause must be found before trusting the result.
Opus 5.5 · Claude · Up 6%, and significant

Where it slips

Uses the supplied evidence correctly50% pass
States that the change is easy to reverse, which is not supported by the supplied context.
Haiku 5.5 · API · The underpowered onboarding test
Avoids unsupported claims56% pass
Presents 'the change is easy to reverse' as fact without evidence from the brief or context.
Haiku 5.5 · API · The underpowered onboarding test
Trusts the data before reading it60% pass
Does not check any trust signal (e.g., sample ratio, exposure) before interpreting the results.
GPT-6 Luna · API · The underpowered onboarding test

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Trusts the data before reading it
Before interpreting effects, does the analysis check whether the result is trustworthy (sample ratio, exposure or logging issues, novelty, a surprisingly large lift) and say what it found?
Passes when Explicitly checks at least one trust signal in the supplied data, such as the split against the intended ratio, and treats a surprising result as a reason to investigate before acting.
Ronny KohaviLenny’s Podcast
Doesn't overread an unclear result
Does the analysis state honestly what the data can and can't show: no win claimed from an early peek or a result that isn't significant, and no 'no effect' concluded from a test too small to detect one?
Passes when Every claim matches the strength of the evidence, and anything unclear is called unclear, with what it can't rule out.
Ronny KohaviLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 1 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

We tested a shorter onboarding checklist and the result came back 'not significant'. The team wants to call it a failure and move on. Write a short note to the onboarding team saying what we should conclude from this test and what we should do next. Keep it under 300 words.

What the model was given4 items: Scenario, Readout, Qualitative, Options on the table

ScenarioTrellis is a project-management tool for small teams. The shorter checklist cuts onboarding from nine steps to five. It is already built: shipping it to everyone would take about a day of engineering. Activation means creating a first project and inviting at least one teammate within seven days.

ReadoutActivation: A 31.2%, B 32.9% (+1.7pp, 95% CI −1.4 to +4.8). n = 3,960 per arm. The pre-registered minimum detectable effect was 3pp.

QualitativeSupport tickets tagged 'onboarding confusion' fell from 44 to 29 during the test.

Options on the tableRunning the test for another four weeks at current traffic would detect an effect of about 2pp.

What a strong answer doesThe answer key the graders mark against

Inconclusive, not a failure: the test could not detect effects smaller than ~3pp, and the point estimate is positive. Recommend a decision based on cost of shipping versus a longer test.

Critical failures (cap the score at 40)

Concludes the change has no effect

Task

Mid-level · v1.11

The outputs

9 setups answered this brief.

GPT-6 AstraChatGPT100Usable with a quick edit GPT-6.1 SolAPI95Usable with a quick edit GPT-6 LunaAPI91Usable with a quick edit Sonnet 5.5API91Usable with a quick edit Haiku 5.5API82Usable with a quick edit Opus 5.5Claude80Usable with a quick edit Gemini 3.5 Flash-LiteGemini70Usable with a quick edit Gemini 3.8 FlashAPI50Needs more work Gemini 4 ArgonAPI50Needs more work

Gemini 4 ArgonAPI

Needs more than a quick edit5 of 10 checks passed

Overclaims on user experience and similarity in performance, and lacks an explicit trust check; these would require reworking the reasoning, not just a light edit.

Output

Subject: Onboarding checklist test: Why we should ship Variant B

Team,

Before we write off the 5-step onboarding checklist as a failure, let’s take a closer look at the data. My recommendation is that we ship the shorter checklist immediately rather than rolling it back or extending the test.

What we should conclude "Not statistically significant" doesn't mean the test failed; it just means we didn't hit our ambitious pre-registered 3pp threshold. Looking at the actual numbers, the story is encouraging: * Activation leaned positive: Variant B saw a +1.7pp lift. Looking at the 95% confidence interval (−1.4pp to +4.8pp), the upside potential is much larger than the downside risk. At worst, the shorter checklist performs similarly to the 9-step version; at best, it provides a solid lift. * Less user friction: Support tickets for "onboarding confusion" dropped by a third (from 44 to 29). Even if activation is technically flat, users are clearly having a better experience. * Simpler is better: Cutting four steps reduces UI bloat and future maintenance for our team.

What we should do next We have the option to run the test for another four weeks to detect a 2pp effect, but that isn't a good use of our time. If the true lift is +1.7pp, another month of testing might still yield an "inconclusive" result while tying up our experimentation traffic.

Because Variant B is already built, shipping it to 100% of users will only take a day of engineering time. When a change is low-effort, reduces support volume, simplifies the product, and shows no evidence of harming activation, the default move should be to ship it.

Let’s roll out the 5-step checklist to everyone this week and move on to our next big bet.

Grades and run details

Decision model 50 · LLM judge 55

Decision model checks

passUses the supplied evidence correctly28%
failAddresses the actual decision73%
passRespects explicit constraints58%
partialIdentifies material uncertainty66%
failAvoids unsupported claims33%
partialProduces the required deliverable11%
partialInterprets power correctly33%
failTrusts the data before reading it98%
passGets the base of every number right37%
partialDoesn't overread an unclear result45%

Run

Run: #1
API response time: 6 s
Submitted: 13 Oct 2026

Gemini 3.5 Flash-LiteGemini

Usable with a quick edit7 of 10 checks passed

The note is clear, evidence-based, and actionable; a PM could add a condition for changing the decision with a quick edit, and no re-checking of evidence or reasoning is needed.

Output

Hi Team,

Before we write off the shorter onboarding checklist as a failure, let's look at the full picture.

While the headline result was statistically non-significant, the numbers and qualitative feedback tell a more nuanced story:

Positive Directional Lift: Group B (5 steps) saw a 32.9% activation rate compared to 31.2% for Group A (9 steps)—a 1.7 percentage point increase.

Underpowered Test: Our pre-registered minimum detectable effect (MDE) was 3pp, meaning the test wasn't powered to reliably catch a 1.7pp or 2pp shift.

Reduced Support Friction: Support tickets tagged "onboarding confusion" dropped significantly from 44 to 29 during the test.

What we should conclude: The test wasn't a failure; it was inconclusive. We likely have a small positive gain accompanied by a meaningful reduction in user confusion.¹

What we should do next: Given that shipping this to everyone only takes about a day of engineering, we should ship it. Continuing to run the test for another four weeks just to chase statistical significance on a small effect isn't a wise use of time² when the engineering cost to launch is so low and support friction has already improved. Let's roll out the 5-step checklist and monitor downstream retention.

What a PM had to fix

1Hypothesis stated as factReframe it as a hypothesisStart again
Source: ReadoutActivation: A 31.2%, B 32.9% (+1.7pp, 95% CI −1.4 to +4.8).
What we’d changeGive the interval: anywhere from a 1.4pp loss to a 4.8pp gain, so a gain isn't established. And the ticket drop wasn't split by arm, so it can't be credited to the new checklist.
2OtherFix itTargeted repair
Source: Options on the tableRunning the test for another four weeks at current traffic would detect an effect of about 2pp.
What we’d changeWeigh the risk of a small activation loss, not only the one day of engineering, and say what result would justify the extra four weeks.

Grades and run details

Decision model 65 · LLM judge 73

Decision model checks

passUses the supplied evidence correctly58%
failAddresses the actual decision83%
passRespects explicit constraints85%
partialIdentifies material uncertainty74%
partialAvoids unsupported claims31%
passProduces the required deliverable58%
passInterprets power correctly73%
failTrusts the data before reading it86%
passGets the base of every number right27%
partialDoesn't overread an unclear result64%

Run

Run: #1
Time to output: 6 s
Submitted: 25 Sept 2026

GPT-6 LunaAPI

Usable with a quick edit9 of 10 checks passed

The output is clear and actionable; a PM could use it with minor edits, and the missing trust check is a small omission that does not undermine the core reasoning.

Output

Team—

This is inconclusive, not a failure. Activation was 1.7 percentage points higher with the shorter checklist, but the 95% CI (−1.4 to +4.8 points) includes no effect and possible harm—as well as a benefit of at least 3 points. The test wasn’t precise enough to settle the question.

I recommend running the test for another four weeks. That should let us assess effects around 2 points, which this change may plausibly deliver. Before extending, agree on what activation gain would justify rollout and what activation decline would stop it. Track onboarding-confusion tickets too, but compare rates per user: the raw drop from 44 to 29 is encouraging, not conclusive.

After the extension, decide using both activation and confusion—not a “significant/not significant” label alone.

Grades and run details

Decision model 90 · LLM judge 91

Decision model checks

passUses the supplied evidence correctly94%
passAddresses the actual decision67%
passRespects explicit constraints97%
passIdentifies material uncertainty98%
passAvoids unsupported claims56%
passProduces the required deliverable72%
passInterprets power correctly88%
failTrusts the data before reading it23%
passGets the base of every number right76%
passDoesn't overread an unclear result95%

Run

Run: #1
API response time: 9 s
Submitted: 29 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Addresses the actual decisionMixedWrongRight

Gemini 4 Argon · API

Commits early and unambiguously to shipping the variant, framed for the onboarding team.

Gemini 3.5 Flash-Lite · Gemini

The output commits to shipping but does not state what result or condition would change that decision.

GPT-6 Luna · API

Commits to extending the test and specifies that the decision after extension will use both activation and confusion data.

Identifies material uncertaintyWrongWrongRight

Gemini 4 Argon · API

Does not name specific unknowns that could change the decision nor state what result would change the call; it presents a definitive recommendation without contingency.

Gemini 3.5 Flash-Lite · Gemini

It does not name specific unknowns that could change the decision, bound them, or say how they would be resolved.

GPT-6 Luna · API

Identifies the imprecision of the test and proposes a longer test to resolve the uncertainty.

Avoids unsupported claimsWrongRightRight

Gemini 4 Argon · API

States 'users are clearly having a better experience' as an established fact, when the evidence only shows a drop in support tickets, not a direct measure of experience.

Gemini 3.5 Flash-Lite · Gemini

Interpretations are labelled as 'likely' and not presented as established fact.

GPT-6 Luna · API

Interpretations are clearly labelled as such, and no unsupported claims are presented as fact.

Doesn't overread an unclear resultWrongRightRight

Gemini 4 Argon · API

The statement 'At worst, the shorter checklist performs similarly' overreads the inconclusive result by implying there is no meaningful negative effect, when the test was too small to detect small effects.

Gemini 3.5 Flash-Lite · Gemini

It does not claim a win or 'no effect'; it correctly calls the result inconclusive and does not overread the non-significant outcome.

GPT-6 Luna · API

Correctly calls the result inconclusive and does not claim a win or no effect from an underpowered test.

All got wrong 1

Trusts the data before reading itWrongWrongWrong

Gemini 4 Argon · API

Does not explicitly check any trust signal (e.g., sample ratio, logging issues) before interpreting the effects.

Gemini 3.5 Flash-Lite · Gemini

No trust signal (e.g., sample ratio, exposure check) is examined before interpreting the data.

GPT-6 Luna · API

Does not check any trust signal (e.g., sample ratio, exposure) before interpreting the results.

All got right 5

Uses the supplied evidence correctlyRightRightRight

Gemini 4 Argon · API

Every factual claim about the current situation comes directly from the brief/supplied context or arithmetic.

Gemini 3.5 Flash-Lite · Gemini

All factual claims are taken directly from the supplied context or follow from arithmetic; nothing is invented.

GPT-6 Luna · API

All factual statements about the current situation are taken directly from the supplied context.

Respects explicit constraintsRightRightRight

Gemini 4 Argon · API

Output is a short note addressed to the team, under 300 words, and respects all explicit constraints.

Gemini 3.5 Flash-Lite · Gemini

The note is under 300 words, addressed to the onboarding team, and respects the requested form.

GPT-6 Luna · API

The note is under 300 words, addressed to the team, and respects the requested form.

Produces the required deliverableRightRightRight

Gemini 4 Argon · API

The note is complete, in the right form for the reader, within length, and contains an actionable recommendation.

Gemini 3.5 Flash-Lite · Gemini

The note is complete, under 300 words, and a PM could act on it with light edits.

GPT-6 Luna · API

The note is complete, under 300 words, and the team could act on it without filling major gaps.

Interprets power correctlyRightRightRight

Gemini 4 Argon · API

Correctly explains that the test could not detect effects smaller than ~3pp, and that extending could detect ~2pp.

Gemini 3.5 Flash-Lite · Gemini

It correctly explains that the test was underpowered to detect effects smaller than the 3pp MDE.

GPT-6 Luna · API

Explains that the test was not precise enough and that an extension would detect effects around 2pp.

Gets the base of every number rightRightRightRight

Gemini 4 Argon · API

All percentages and differences are computed correctly, and the drop in tickets is properly presented from 44 to 29.

Gemini 3.5 Flash-Lite · Gemini

All percentages and differences are computed correctly, and the base (activation rate) is clear.

GPT-6 Luna · API

All derived figures (1.7pp, CI, ticket drop) are correctly stated and no ambiguous bases are used.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 60% of 40 checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	GPT-6 AstrawithChatGPT	91.7	100.0	2	None
2	Sonnet 5.5withAPI	84.6	95.5	2	None
3	Opus 5.5withClaude	88.3	86.4	2	None
4	GPT-6.1 SolwithAPI	76.3	80.8	2	None
5	Haiku 5.5withAPI	82.5	71.0	2	None
6	GPT-6 LunawithAPI	78.3	64.7	2	None
7	Gemini 4 ArgonwithAPI	68.8	65.7	2	None
8	Gemini 3.5 Flash-LitewithGemini	53.3	51.7	2	None
9	Gemini 3.8 FlashwithAPI	63.3	45.8	2	None

What you get for the cost

Sonnet 5.5 · API scores within 5 points of GPT-6 Astra · ChatGPT at a third of the cost.

Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
GPT-6 Luna · API	70	1.0×
Haiku 5.5 · API	76	1.8×
Gemini 3.5 Flash-Lite · Gemini	53	3.1× (partly estimated)
Gemini 3.8 Flash · API	52	12×
GPT-6.1 Sol · API	78	14×
Gemini 4 Argon · API	64	14×
Sonnet 5.5 · API	91	35×
Opus 5.5 · Claude	88	95× (partly estimated)
GPT-6 Astra · ChatGPT	96	107× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Reading a test readout and deciding what to do next.

Why it matters

Experiment readouts are where false confidence is cheapest to produce and most expensive to act on. A model that declares a winner on conversion while retention quietly falls will ship the wrong thing.

What good looks like

Checks the result can be trusted before interpreting it
Reports effects with their uncertainty
Checks guardrail metrics before declaring a winner
Separates what the data shows from plausible explanations
Recommends a next step proportionate to the evidence

Deliberately not measured

Re-running the statistics from raw data
Chart production

Capability tested

Interpreting results within their limits

The failure we’re looking for

Claims causality, ignores guardrails or recommends arbitrary testing

Grading

Decision model and LLM judge, calibrated against a blind PM review

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

Trusts the data before reading it

Doesn't overread an unclear result

Plus our standard checks

The tasks

The outputs

Output

Grades and run details

Decision model checks

Run

Output

What a PM had to fix

Grades and run details

Decision model checks

Run

Output

Grades and run details

Decision model checks

Run

Check by check

Where they differ 4

All got wrong 1

All got right 5

Results

What you get for the cost

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured