Tasks / Metrics & Experimentation

Analyse experiment results

Can the model separate evidence from speculation, identify decision-relevant uncertainty and recommend a sensible next action?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 56% were usable with at most a quick edit.

Reliably right

Respects explicit constraints100% pass
The note is addressed to the onboarding team, stays under 300 words, and proposes actions that would enforce the stated constraints.
GPT-6 Astra · ChatGPT · The underpowered onboarding test
Interprets power correctly97% pass
It correctly explains that the test was designed to detect a 3pp effect, that the CI includes both harm and benefit, and that extending would detect about 2pp.
GPT-6 Astra · ChatGPT · The underpowered onboarding test
Catches the sample ratio mismatch89% pass
The output computes the split (53.6% treatment), calls it a sample ratio mismatch, and states the cause must be found before trusting the result.
Opus 5.5 · Claude · Up 6%, and significant

Where it slips

Uses the supplied evidence correctly50% pass
States that the change is easy to reverse, which is not supported by the supplied context.
Haiku 5.5 · API · The underpowered onboarding test
Avoids unsupported claims56% pass
Presents 'the change is easy to reverse' as fact without evidence from the brief or context.
Haiku 5.5 · API · The underpowered onboarding test
Trusts the data before reading it60% pass
Does not check any trust signal (e.g., sample ratio, exposure) before interpreting the results.
GPT-6 Luna · API · The underpowered onboarding test

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Trusts the data before reading it
Before interpreting effects, does the analysis check whether the result is trustworthy (sample ratio, exposure or logging issues, novelty, a surprisingly large lift) and say what it found?
Passes when Explicitly checks at least one trust signal in the supplied data, such as the split against the intended ratio, and treats a surprising result as a reason to investigate before acting.
Ronny KohaviLenny’s Podcast
Doesn't overread an unclear result
Does the analysis state honestly what the data can and can't show: no win claimed from an early peek or a result that isn't significant, and no 'no effect' concluded from a test too small to detect one?
Passes when Every claim matches the strength of the evidence, and anything unclear is called unclear, with what it can't rule out.
Ronny KohaviLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 1 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

We tested a shorter onboarding checklist and the result came back 'not significant'. The team wants to call it a failure and move on. Write a short note to the onboarding team saying what we should conclude from this test and what we should do next. Keep it under 300 words.

What the model was given4 items: Scenario, Readout, Qualitative, Options on the table

ScenarioTrellis is a project-management tool for small teams. The shorter checklist cuts onboarding from nine steps to five. It is already built: shipping it to everyone would take about a day of engineering. Activation means creating a first project and inviting at least one teammate within seven days.

ReadoutActivation: A 31.2%, B 32.9% (+1.7pp, 95% CI −1.4 to +4.8). n = 3,960 per arm. The pre-registered minimum detectable effect was 3pp.

QualitativeSupport tickets tagged 'onboarding confusion' fell from 44 to 29 during the test.

Options on the tableRunning the test for another four weeks at current traffic would detect an effect of about 2pp.

What a strong answer doesThe answer key the graders mark against

Inconclusive, not a failure: the test could not detect effects smaller than ~3pp, and the point estimate is positive. Recommend a decision based on cost of shipping versus a longer test.

Critical failures (cap the score at 40)

Concludes the change has no effect

Task

Mid-level · v1.11

The outputs

9 setups answered this brief.

GPT-6 AstraChatGPT100Usable with a quick edit GPT-6.1 SolAPI95Usable with a quick edit GPT-6 LunaAPI91Usable with a quick edit Sonnet 5.5API91Usable with a quick edit Haiku 5.5API82Usable with a quick edit Opus 5.5Claude80Usable with a quick edit Gemini 3.5 Flash-LiteGemini70Usable with a quick edit Gemini 3.8 FlashAPI50Needs more work Gemini 4 ArgonAPI50Needs more work

Gemini 3.5 Flash-LiteGemini

Usable with a quick edit7 of 10 checks passed

The note is clear, evidence-based, and actionable; a PM could add a condition for changing the decision with a quick edit, and no re-checking of evidence or reasoning is needed.

Output

Hi Team,

Before we write off the shorter onboarding checklist as a failure, let's look at the full picture.

While the headline result was statistically non-significant, the numbers and qualitative feedback tell a more nuanced story:

Positive Directional Lift: Group B (5 steps) saw a 32.9% activation rate compared to 31.2% for Group A (9 steps)—a 1.7 percentage point increase.

Underpowered Test: Our pre-registered minimum detectable effect (MDE) was 3pp, meaning the test wasn't powered to reliably catch a 1.7pp or 2pp shift.

Reduced Support Friction: Support tickets tagged "onboarding confusion" dropped significantly from 44 to 29 during the test.

What we should conclude: The test wasn't a failure; it was inconclusive. We likely have a small positive gain accompanied by a meaningful reduction in user confusion.¹

What we should do next: Given that shipping this to everyone only takes about a day of engineering, we should ship it. Continuing to run the test for another four weeks just to chase statistical significance on a small effect isn't a wise use of time² when the engineering cost to launch is so low and support friction has already improved. Let's roll out the 5-step checklist and monitor downstream retention.

What a PM had to fix

1Hypothesis stated as factReframe it as a hypothesisStart again
Source: ReadoutActivation: A 31.2%, B 32.9% (+1.7pp, 95% CI −1.4 to +4.8).
What we’d changeGive the interval: anywhere from a 1.4pp loss to a 4.8pp gain, so a gain isn't established. And the ticket drop wasn't split by arm, so it can't be credited to the new checklist.
2OtherFix itTargeted repair
Source: Options on the tableRunning the test for another four weeks at current traffic would detect an effect of about 2pp.
What we’d changeWeigh the risk of a small activation loss, not only the one day of engineering, and say what result would justify the extra four weeks.

Grades and run details

Decision model 65 · LLM judge 73

Decision model checks

passUses the supplied evidence correctly58%
failAddresses the actual decision83%
passRespects explicit constraints85%
partialIdentifies material uncertainty74%
partialAvoids unsupported claims31%
passProduces the required deliverable58%
passInterprets power correctly73%
failTrusts the data before reading it86%
passGets the base of every number right27%
partialDoesn't overread an unclear result64%

Run

Run: #1
Time to output: 6 s
Submitted: 25 Sept 2026

Gemini 4 ArgonAPI

Needs more than a quick edit5 of 10 checks passed

Overclaims on user experience and similarity in performance, and lacks an explicit trust check; these would require reworking the reasoning, not just a light edit.

Output

Subject: Onboarding checklist test: Why we should ship Variant B

Team,

Before we write off the 5-step onboarding checklist as a failure, let’s take a closer look at the data. My recommendation is that we ship the shorter checklist immediately rather than rolling it back or extending the test.

What we should conclude "Not statistically significant" doesn't mean the test failed; it just means we didn't hit our ambitious pre-registered 3pp threshold. Looking at the actual numbers, the story is encouraging: * Activation leaned positive: Variant B saw a +1.7pp lift. Looking at the 95% confidence interval (−1.4pp to +4.8pp), the upside potential is much larger than the downside risk. At worst, the shorter checklist performs similarly to the 9-step version; at best, it provides a solid lift. * Less user friction: Support tickets for "onboarding confusion" dropped by a third (from 44 to 29). Even if activation is technically flat, users are clearly having a better experience. * Simpler is better: Cutting four steps reduces UI bloat and future maintenance for our team.

What we should do next We have the option to run the test for another four weeks to detect a 2pp effect, but that isn't a good use of our time. If the true lift is +1.7pp, another month of testing might still yield an "inconclusive" result while tying up our experimentation traffic.

Because Variant B is already built, shipping it to 100% of users will only take a day of engineering time. When a change is low-effort, reduces support volume, simplifies the product, and shows no evidence of harming activation, the default move should be to ship it.

Let’s roll out the 5-step checklist to everyone this week and move on to our next big bet.

Grades and run details

Decision model 50 · LLM judge 55

Decision model checks

passUses the supplied evidence correctly28%
failAddresses the actual decision73%
passRespects explicit constraints58%
partialIdentifies material uncertainty66%
failAvoids unsupported claims33%
partialProduces the required deliverable11%
partialInterprets power correctly33%
failTrusts the data before reading it98%
passGets the base of every number right37%
partialDoesn't overread an unclear result45%

Run

Run: #1
API response time: 6 s
Submitted: 13 Oct 2026

Sonnet 5.5API

Usable with a quick edit9 of 10 checks passed

The note is clear, well-structured, and actionable; a PM could use it with minor wording tweaks and no need to re-verify evidence or logic.

Output

Subject: Shorter onboarding checklist: what the test tells us and what to do

Bottom line: This test didn't fail. It was inconclusive, and I recommend we ship the shorter checklist.

What the result says - Activation was 32.9% vs. 31.2% (+1.7pp). The 95% CI runs from −1.4 to +4.8pp, so "not significant" means we can't rule out zero. It doesn't mean there's no effect. - The data are compatible with a real gain of several points, and also with a small loss. The test was powered to detect 3pp, so a true effect of 1-2pp would probably look just like this. - Support tickets tagged "onboarding confusion" dropped from 44 to 29. The counts are small, so this isn't proof, but it points the same way as the activation estimate.

Why ship rather than keep testing - Shipping costs about a day of engineering, and it's easy to reverse. - The plausible downside is small (about −1.4pp at worst), and the point estimate and the ticket data both lean positive. - Four more weeks would detect ~2pp, but our observed effect is 1.7pp. We could easily wait a month and end up inconclusive again, while half of new users get the longer flow.

Next steps 1. Ship the five-step checklist to everyone. 2. Track seven-day activation against the pre-test baseline for the next 4-6 weeks. If it drops well below ~31%, we revert. 3. Keep tracking the onboarding-confusion tickets. 4. Record the test as "inconclusive, shipped on cost and risk grounds" rather than "failed," so nobody later cites it as evidence that shorter onboarding doesn't work.

If we were weighing a costly or hard-to-reverse change, I'd argue for the longer test. Here the cheaper move is to ship and monitor.

Grades and run details

Decision model 90 · LLM judge 91

Decision model checks

passUses the supplied evidence correctly52%
passAddresses the actual decision95%
passRespects explicit constraints64%
passIdentifies material uncertainty100%
passAvoids unsupported claims49%
passProduces the required deliverable86%
passInterprets power correctly100%
failTrusts the data before reading it89%
passGets the base of every number right49%
passDoesn't overread an unclear result82%

Run

Run: #1
API response time: 11 s
Submitted: 29 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Addresses the actual decisionWrongMixedRight

Gemini 3.5 Flash-Lite · Gemini

The output commits to shipping but does not state what result or condition would change that decision.

Gemini 4 Argon · API

Commits early and unambiguously to shipping the variant, framed for the onboarding team.

Sonnet 5.5 · API

Commits early to shipping the shorter checklist, frames it for the onboarding team, and states that a costly or hard-to-reverse change would change the call.

Identifies material uncertaintyWrongWrongRight

Gemini 3.5 Flash-Lite · Gemini

It does not name specific unknowns that could change the decision, bound them, or say how they would be resolved.

Gemini 4 Argon · API

Does not name specific unknowns that could change the decision nor state what result would change the call; it presents a definitive recommendation without contingency.

Sonnet 5.5 · API

Names the uncertainty (CI includes negative values, true effect could be small), bounds it with the CI, and says monitoring and reverting if activation drops would resolve it.

Avoids unsupported claimsRightWrongRight

Gemini 3.5 Flash-Lite · Gemini

Interpretations are labelled as 'likely' and not presented as established fact.

Gemini 4 Argon · API

States 'users are clearly having a better experience' as an established fact, when the evidence only shows a drop in support tickets, not a direct measure of experience.

Sonnet 5.5 · API

Interpretations like 'this isn't proof' and 'lean positive' are clearly hedged; no cause or forecast is presented as established fact.

Doesn't overread an unclear resultRightWrongRight

Gemini 3.5 Flash-Lite · Gemini

It does not claim a win or 'no effect'; it correctly calls the result inconclusive and does not overread the non-significant outcome.

Gemini 4 Argon · API

The statement 'At worst, the shorter checklist performs similarly' overreads the inconclusive result by implying there is no meaningful negative effect, when the test was too small to detect small effects.

Sonnet 5.5 · API

Correctly calls the result inconclusive, does not claim a win or no effect, and states what the CI can and cannot rule out.

All got wrong 1

Trusts the data before reading itWrongWrongWrong

Gemini 3.5 Flash-Lite · Gemini

No trust signal (e.g., sample ratio, exposure check) is examined before interpreting the data.

Gemini 4 Argon · API

Does not explicitly check any trust signal (e.g., sample ratio, logging issues) before interpreting the effects.

Sonnet 5.5 · API

Does not check any trust signal such as sample ratio, exposure, or logging issues before interpreting the data.

All got right 5

Uses the supplied evidence correctlyRightRightRight

Gemini 3.5 Flash-Lite · Gemini

All factual claims are taken directly from the supplied context or follow from arithmetic; nothing is invented.

Gemini 4 Argon · API

Every factual claim about the current situation comes directly from the brief/supplied context or arithmetic.

Sonnet 5.5 · API

All factual claims are taken directly from the supplied context or derived from it; the 'counts are small' observation is a reasonable characterization of the given numbers.

Respects explicit constraintsRightRightRight

Gemini 3.5 Flash-Lite · Gemini

The note is under 300 words, addressed to the onboarding team, and respects the requested form.

Gemini 4 Argon · API

Output is a short note addressed to the team, under 300 words, and respects all explicit constraints.

Sonnet 5.5 · API

The note is under 300 words, addressed to the onboarding team, and says what to conclude and do next.

Produces the required deliverableRightRightRight

Gemini 3.5 Flash-Lite · Gemini

The note is complete, under 300 words, and a PM could act on it with light edits.

Gemini 4 Argon · API

The note is complete, in the right form for the reader, within length, and contains an actionable recommendation.

Sonnet 5.5 · API

A complete, actionable note under 300 words that the onboarding team could use with light edits.

Interprets power correctlyRightRightRight

Gemini 3.5 Flash-Lite · Gemini

It correctly explains that the test was underpowered to detect effects smaller than the 3pp MDE.

Gemini 4 Argon · API

Correctly explains that the test could not detect effects smaller than ~3pp, and that extending could detect ~2pp.

Sonnet 5.5 · API

Explains the test was powered for 3pp, so a 1-2pp effect would likely look like this, and that four more weeks would detect ~2pp.

Gets the base of every number rightRightRightRight

Gemini 3.5 Flash-Lite · Gemini

All percentages and differences are computed correctly, and the base (activation rate) is clear.

Gemini 4 Argon · API

All percentages and differences are computed correctly, and the drop in tickets is properly presented from 44 to 29.

Sonnet 5.5 · API

All percentages and differences are correctly stated; no base confusion (ticket counts are absolute, activation rates are per arm).

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 60% of 40 checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	GPT-6 AstrawithChatGPT	91.7	100.0	2	None
2	Sonnet 5.5withAPI	84.6	95.5	2	None
3	Opus 5.5withClaude	88.3	86.4	2	None
4	GPT-6.1 SolwithAPI	76.3	80.8	2	None
5	Haiku 5.5withAPI	82.5	71.0	2	None
6	GPT-6 LunawithAPI	78.3	64.7	2	None
7	Gemini 4 ArgonwithAPI	68.8	65.7	2	None
8	Gemini 3.5 Flash-LitewithGemini	53.3	51.7	2	None
9	Gemini 3.8 FlashwithAPI	63.3	45.8	2	None

What you get for the cost

Sonnet 5.5 · API scores within 5 points of GPT-6 Astra · ChatGPT at a third of the cost.

Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
GPT-6 Luna · API	70	1.0×
Haiku 5.5 · API	76	1.8×
Gemini 3.5 Flash-Lite · Gemini	53	3.1× (partly estimated)
Gemini 3.8 Flash · API	52	12×
GPT-6.1 Sol · API	78	14×
Gemini 4 Argon · API	64	14×
Sonnet 5.5 · API	91	35×
Opus 5.5 · Claude	88	95× (partly estimated)
GPT-6 Astra · ChatGPT	96	107× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Reading a test readout and deciding what to do next.

Why it matters

Experiment readouts are where false confidence is cheapest to produce and most expensive to act on. A model that declares a winner on conversion while retention quietly falls will ship the wrong thing.

What good looks like

Checks the result can be trusted before interpreting it
Reports effects with their uncertainty
Checks guardrail metrics before declaring a winner
Separates what the data shows from plausible explanations
Recommends a next step proportionate to the evidence

Deliberately not measured

Re-running the statistics from raw data
Chart production

Capability tested

Interpreting results within their limits

The failure we’re looking for

Claims causality, ignores guardrails or recommends arbitrary testing

Grading

Decision model and LLM judge, calibrated against a blind PM review

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

Trusts the data before reading it

Doesn't overread an unclear result

Plus our standard checks

The tasks

The outputs

Output

What a PM had to fix

Grades and run details

Decision model checks

Run

Output

Grades and run details

Decision model checks

Run

Output

Grades and run details

Decision model checks

Run

Check by check

Where they differ 4

All got wrong 1

All got right 5

Results

What you get for the cost

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured