Tasks / Metrics & Experimentation

Analyse experiment results

Can the model separate evidence from speculation, identify decision-relevant uncertainty and recommend a sensible next action?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 56% were usable with at most a quick edit.

Reliably right

Respects explicit constraints100% pass
The note is addressed to the onboarding team, stays under 300 words, and proposes actions that would enforce the stated constraints.
GPT-6 Astra · ChatGPT · The underpowered onboarding test
Interprets power correctly97% pass
It correctly explains that the test was designed to detect a 3pp effect, that the CI includes both harm and benefit, and that extending would detect about 2pp.
GPT-6 Astra · ChatGPT · The underpowered onboarding test
Catches the sample ratio mismatch89% pass
The output computes the split (53.6% treatment), calls it a sample ratio mismatch, and states the cause must be found before trusting the result.
Opus 5.5 · Claude · Up 6%, and significant

Where it slips

Uses the supplied evidence correctly50% pass
States that the change is easy to reverse, which is not supported by the supplied context.
Haiku 5.5 · API · The underpowered onboarding test
Avoids unsupported claims56% pass
Presents 'the change is easy to reverse' as fact without evidence from the brief or context.
Haiku 5.5 · API · The underpowered onboarding test
Trusts the data before reading it60% pass
Does not check any trust signal (e.g., sample ratio, exposure) before interpreting the results.
GPT-6 Luna · API · The underpowered onboarding test

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Trusts the data before reading it
Before interpreting effects, does the analysis check whether the result is trustworthy (sample ratio, exposure or logging issues, novelty, a surprisingly large lift) and say what it found?
Passes when Explicitly checks at least one trust signal in the supplied data, such as the split against the intended ratio, and treats a surprising result as a reason to investigate before acting.
Ronny KohaviLenny’s Podcast
Doesn't overread an unclear result
Does the analysis state honestly what the data can and can't show: no win claimed from an early peek or a result that isn't significant, and no 'no effect' concluded from a test too small to detect one?
Passes when Every claim matches the strength of the evidence, and anything unclear is called unclear, with what it can't rule out.
Ronny KohaviLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 1 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

We tested a shorter onboarding checklist and the result came back 'not significant'. The team wants to call it a failure and move on. Write a short note to the onboarding team saying what we should conclude from this test and what we should do next. Keep it under 300 words.

What the model was given4 items: Scenario, Readout, Qualitative, Options on the table

ScenarioTrellis is a project-management tool for small teams. The shorter checklist cuts onboarding from nine steps to five. It is already built: shipping it to everyone would take about a day of engineering. Activation means creating a first project and inviting at least one teammate within seven days.

ReadoutActivation: A 31.2%, B 32.9% (+1.7pp, 95% CI −1.4 to +4.8). n = 3,960 per arm. The pre-registered minimum detectable effect was 3pp.

QualitativeSupport tickets tagged 'onboarding confusion' fell from 44 to 29 during the test.

Options on the tableRunning the test for another four weeks at current traffic would detect an effect of about 2pp.

What a strong answer doesThe answer key the graders mark against

Inconclusive, not a failure: the test could not detect effects smaller than ~3pp, and the point estimate is positive. Recommend a decision based on cost of shipping versus a longer test.

Critical failures (cap the score at 40)

Concludes the change has no effect

Task

Mid-level · v1.11

The outputs

9 setups answered this brief.

GPT-6 AstraChatGPT100Usable with a quick edit GPT-6.1 SolAPI95Usable with a quick edit GPT-6 LunaAPI91Usable with a quick edit Sonnet 5.5API91Usable with a quick edit Haiku 5.5API82Usable with a quick edit Opus 5.5Claude80Usable with a quick edit Gemini 3.5 Flash-LiteGemini70Usable with a quick edit Gemini 3.8 FlashAPI50Needs more work Gemini 4 ArgonAPI50Needs more work

Gemini 3.8 FlashAPI

Needs more than a quick edit5 of 10 checks passed

The output makes an unsupported claim about improved user experience and does not state what would change the decision, requiring more than a quick edit.

Output

Team,

Before we write off the shorter checklist as a failure, let’s reframe what the data actually tells us.

“Not statistically significant” does not mean “no impact”—it means our test was underpowered to detect an effect smaller than 3pp. Here is the full picture:

The directional signals are uniformly positive: Activation increased by +1.7pp (31.2% → 32.9%). While the 95% CI (−1.4pp to +4.8pp) crosses zero, it leans heavily positive. The downside risk is small, while the potential upside is meaningful.
Clear reduction in user friction: Support tickets tagged “onboarding confusion” dropped by 34% (44 to 29). Even if activation were completely flat, cutting nearly half the steps improved user experience and reduced support burden.
The cost/benefit is strongly asymmetric: Shipping this takes one day of engineering.

Recommendation: Ship the 5-step checklist to 100% and move on.

Running the test for another four weeks to hit an MDE of 2pp isn’t the best use of our time or traffic. We would burn a month of runway just to confirm what looks like a positive, low-risk change with virtually zero engineering cost.

Let’s bank the operational win (fewer support tickets, leaner UX, likely small activation lift) by rolling it out tomorrow. We can monitor activation metrics in production and direct our experimentation bandwidth toward bigger, higher-leverage bets for Trellis.

Claims the judge couldn’t find in the brief

Cutting nearly half the steps improved user experience

Grades and run details

Decision model 60 · LLM judge 45

Decision model checks

passUses the supplied evidence correctly19%
failAddresses the actual decision85%
passRespects explicit constraints77%
partialIdentifies material uncertainty61%
failAvoids unsupported claims25%
passProduces the required deliverable46%
passInterprets power correctly92%
failTrusts the data before reading it98%
passGets the base of every number right48%
partialDoesn't overread an unclear result46%

Run

Run: #1
API response time: 11 s
Submitted: 1 Oct 2026

Gemini 4 ArgonAPI

Needs more than a quick edit5 of 10 checks passed

Overclaims on user experience and similarity in performance, and lacks an explicit trust check; these would require reworking the reasoning, not just a light edit.

Output

Subject: Onboarding checklist test: Why we should ship Variant B

Team,

Before we write off the 5-step onboarding checklist as a failure, let’s take a closer look at the data. My recommendation is that we ship the shorter checklist immediately rather than rolling it back or extending the test.

What we should conclude "Not statistically significant" doesn't mean the test failed; it just means we didn't hit our ambitious pre-registered 3pp threshold. Looking at the actual numbers, the story is encouraging: * Activation leaned positive: Variant B saw a +1.7pp lift. Looking at the 95% confidence interval (−1.4pp to +4.8pp), the upside potential is much larger than the downside risk. At worst, the shorter checklist performs similarly to the 9-step version; at best, it provides a solid lift. * Less user friction: Support tickets for "onboarding confusion" dropped by a third (from 44 to 29). Even if activation is technically flat, users are clearly having a better experience. * Simpler is better: Cutting four steps reduces UI bloat and future maintenance for our team.

What we should do next We have the option to run the test for another four weeks to detect a 2pp effect, but that isn't a good use of our time. If the true lift is +1.7pp, another month of testing might still yield an "inconclusive" result while tying up our experimentation traffic.

Because Variant B is already built, shipping it to 100% of users will only take a day of engineering time. When a change is low-effort, reduces support volume, simplifies the product, and shows no evidence of harming activation, the default move should be to ship it.

Let’s roll out the 5-step checklist to everyone this week and move on to our next big bet.

Grades and run details

Decision model 50 · LLM judge 55

Decision model checks

passUses the supplied evidence correctly28%
failAddresses the actual decision73%
passRespects explicit constraints58%
partialIdentifies material uncertainty66%
failAvoids unsupported claims33%
partialProduces the required deliverable11%
partialInterprets power correctly33%
failTrusts the data before reading it98%
passGets the base of every number right37%
partialDoesn't overread an unclear result45%

Run

Run: #1
API response time: 6 s
Submitted: 13 Oct 2026

Gemini 3.5 Flash-LiteGemini

Usable with a quick edit7 of 10 checks passed

The note is clear, evidence-based, and actionable; a PM could add a condition for changing the decision with a quick edit, and no re-checking of evidence or reasoning is needed.

Output

Hi Team,

Before we write off the shorter onboarding checklist as a failure, let's look at the full picture.

While the headline result was statistically non-significant, the numbers and qualitative feedback tell a more nuanced story:

Positive Directional Lift: Group B (5 steps) saw a 32.9% activation rate compared to 31.2% for Group A (9 steps)—a 1.7 percentage point increase.

Underpowered Test: Our pre-registered minimum detectable effect (MDE) was 3pp, meaning the test wasn't powered to reliably catch a 1.7pp or 2pp shift.

Reduced Support Friction: Support tickets tagged "onboarding confusion" dropped significantly from 44 to 29 during the test.

What we should conclude: The test wasn't a failure; it was inconclusive. We likely have a small positive gain accompanied by a meaningful reduction in user confusion.¹

What we should do next: Given that shipping this to everyone only takes about a day of engineering, we should ship it. Continuing to run the test for another four weeks just to chase statistical significance on a small effect isn't a wise use of time² when the engineering cost to launch is so low and support friction has already improved. Let's roll out the 5-step checklist and monitor downstream retention.

What a PM had to fix

1Hypothesis stated as factReframe it as a hypothesisStart again
Source: ReadoutActivation: A 31.2%, B 32.9% (+1.7pp, 95% CI −1.4 to +4.8).
What we’d changeGive the interval: anywhere from a 1.4pp loss to a 4.8pp gain, so a gain isn't established. And the ticket drop wasn't split by arm, so it can't be credited to the new checklist.
2OtherFix itTargeted repair
Source: Options on the tableRunning the test for another four weeks at current traffic would detect an effect of about 2pp.
What we’d changeWeigh the risk of a small activation loss, not only the one day of engineering, and say what result would justify the extra four weeks.

Grades and run details

Decision model 65 · LLM judge 73

Decision model checks

passUses the supplied evidence correctly58%
failAddresses the actual decision83%
passRespects explicit constraints85%
partialIdentifies material uncertainty74%
partialAvoids unsupported claims31%
passProduces the required deliverable58%
passInterprets power correctly73%
failTrusts the data before reading it86%
passGets the base of every number right27%
partialDoesn't overread an unclear result64%

Run

Run: #1
Time to output: 6 s
Submitted: 25 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Uses the supplied evidence correctlyMixedRightRight

Gemini 3.8 Flash · API

Claims that the shorter checklist 'improved user experience' are not supported by the supplied context, which only shows a drop in support tickets.

Gemini 4 Argon · API

Every factual claim about the current situation comes directly from the brief/supplied context or arithmetic.

Gemini 3.5 Flash-Lite · Gemini

All factual claims are taken directly from the supplied context or follow from arithmetic; nothing is invented.

Addresses the actual decisionWrongMixedWrong

Gemini 3.8 Flash · API

The output commits to shipping but does not state what result or condition would change that decision.

Gemini 4 Argon · API

Commits early and unambiguously to shipping the variant, framed for the onboarding team.

Gemini 3.5 Flash-Lite · Gemini

The output commits to shipping but does not state what result or condition would change that decision.

Avoids unsupported claimsWrongWrongRight

Gemini 3.8 Flash · API

The output presents 'improved user experience' as an established fact when the evidence only shows a reduction in support tickets.

Gemini 4 Argon · API

States 'users are clearly having a better experience' as an established fact, when the evidence only shows a drop in support tickets, not a direct measure of experience.

Gemini 3.5 Flash-Lite · Gemini

Interpretations are labelled as 'likely' and not presented as established fact.

Doesn't overread an unclear resultRightWrongRight

Gemini 3.8 Flash · API

The output does not claim a significant win or conclude no effect; it acknowledges the CI crosses zero and the test was underpowered.

Gemini 4 Argon · API

The statement 'At worst, the shorter checklist performs similarly' overreads the inconclusive result by implying there is no meaningful negative effect, when the test was too small to detect small effects.

Gemini 3.5 Flash-Lite · Gemini

It does not claim a win or 'no effect'; it correctly calls the result inconclusive and does not overread the non-significant outcome.

All got wrong 2

Identifies material uncertaintyWrongWrongWrong

Gemini 3.8 Flash · API

The output does not name specific unknowns that could change the decision or how they would be resolved.

Gemini 4 Argon · API

Does not name specific unknowns that could change the decision nor state what result would change the call; it presents a definitive recommendation without contingency.

Gemini 3.5 Flash-Lite · Gemini

It does not name specific unknowns that could change the decision, bound them, or say how they would be resolved.

Trusts the data before reading itWrongWrongWrong

Gemini 3.8 Flash · API

The output does not check any trust signal (e.g., sample ratio, exposure, logging) before interpreting the data.

Gemini 4 Argon · API

Does not explicitly check any trust signal (e.g., sample ratio, logging issues) before interpreting the effects.

Gemini 3.5 Flash-Lite · Gemini

No trust signal (e.g., sample ratio, exposure check) is examined before interpreting the data.

All got right 4

Respects explicit constraintsRightRightRight

Gemini 3.8 Flash · API

The note is addressed to the onboarding team, is under 300 words, and recommends a clear next step.

Gemini 4 Argon · API

Output is a short note addressed to the team, under 300 words, and respects all explicit constraints.

Gemini 3.5 Flash-Lite · Gemini

The note is under 300 words, addressed to the onboarding team, and respects the requested form.

Produces the required deliverableRightRightRight

Gemini 3.8 Flash · API

The output is a short note to the onboarding team with a conclusion and recommended action, under 300 words, and is usable as is.

Gemini 4 Argon · API

The note is complete, in the right form for the reader, within length, and contains an actionable recommendation.

Gemini 3.5 Flash-Lite · Gemini

The note is complete, under 300 words, and a PM could act on it with light edits.

Interprets power correctlyRightRightRight

Gemini 3.8 Flash · API

The output correctly explains that the test was underpowered to detect effects smaller than 3pp and what a longer test could detect.

Gemini 4 Argon · API

Correctly explains that the test could not detect effects smaller than ~3pp, and that extending could detect ~2pp.

Gemini 3.5 Flash-Lite · Gemini

It correctly explains that the test was underpowered to detect effects smaller than the 3pp MDE.

Gets the base of every number rightRightRightRight

Gemini 3.8 Flash · API

All percentages and differences are computed correctly from the supplied numbers, and the base for the ticket drop is clear.

Gemini 4 Argon · API

All percentages and differences are computed correctly, and the drop in tickets is properly presented from 44 to 29.

Gemini 3.5 Flash-Lite · Gemini

All percentages and differences are computed correctly, and the base (activation rate) is clear.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 60% of 40 checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	GPT-6 AstrawithChatGPT	91.7	100.0	2	None
2	Sonnet 5.5withAPI	84.6	95.5	2	None
3	Opus 5.5withClaude	88.3	86.4	2	None
4	GPT-6.1 SolwithAPI	76.3	80.8	2	None
5	Haiku 5.5withAPI	82.5	71.0	2	None
6	GPT-6 LunawithAPI	78.3	64.7	2	None
7	Gemini 4 ArgonwithAPI	68.8	65.7	2	None
8	Gemini 3.5 Flash-LitewithGemini	53.3	51.7	2	None
9	Gemini 3.8 FlashwithAPI	63.3	45.8	2	None

What you get for the cost

Sonnet 5.5 · API scores within 5 points of GPT-6 Astra · ChatGPT at a third of the cost.

Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
GPT-6 Luna · API	70	1.0×
Haiku 5.5 · API	76	1.8×
Gemini 3.5 Flash-Lite · Gemini	53	3.1× (partly estimated)
Gemini 3.8 Flash · API	52	12×
GPT-6.1 Sol · API	78	14×
Gemini 4 Argon · API	64	14×
Sonnet 5.5 · API	91	35×
Opus 5.5 · Claude	88	95× (partly estimated)
GPT-6 Astra · ChatGPT	96	107× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Reading a test readout and deciding what to do next.

Why it matters

Experiment readouts are where false confidence is cheapest to produce and most expensive to act on. A model that declares a winner on conversion while retention quietly falls will ship the wrong thing.

What good looks like

Checks the result can be trusted before interpreting it
Reports effects with their uncertainty
Checks guardrail metrics before declaring a winner
Separates what the data shows from plausible explanations
Recommends a next step proportionate to the evidence

Deliberately not measured

Re-running the statistics from raw data
Chart production

Capability tested

Interpreting results within their limits

The failure we’re looking for

Claims causality, ignores guardrails or recommends arbitrary testing

Grading

Decision model and LLM judge, calibrated against a blind PM review

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

Trusts the data before reading it

Doesn't overread an unclear result

Plus our standard checks

The tasks

The outputs

Output

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Run

Output

Grades and run details

Decision model checks

Run

Output

What a PM had to fix

Grades and run details

Decision model checks

Run

Check by check

Where they differ 4

All got wrong 2

All got right 4

Results

What you get for the cost

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured