Tasks / Experiment

Analyse experiment results

Can the model separate evidence from speculation, identify decision-relevant uncertainty and recommend a sensible next action?

Measures the modelTask v1.2 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 14 graded outputs by 7 models. 71% were usable with at most a quick edit.

Reliably right

Gets the base of every number right100% pass
The refund increase and revenue difference are computed correctly, and the retention CI interpretation uses the correct 2pp threshold.
GPT-6 Astra · ChatGPT · Conversion up, retention down
Interprets power correctly100% pass
Explains that the test was designed to detect ~3pp effects, so a smaller but worthwhile lift could still exist and remain undetected.
GPT-6 Astra · ChatGPT · The underpowered onboarding test
Checks guardrails before declaring a winner100% pass
It evaluates both the day-30 retention guardrail and the refund-request guardrail before making a call.
GPT-6 Astra · ChatGPT · Conversion up, retention down

Where it slips

Trusts the data before reading it21% pass
It does not explicitly check a pre-interpretation trust signal such as sample ratio or logging before relying on the lift; the denominator and maturity caveats come after interpreting.
GPT-6.1 Sol · API · Conversion up, retention down
Avoids unsupported claims52% pass
It asserts the refund surge 'isn't noise' and that annual up-front billing means retention 'should rise' without supporting evidence or clear labeling.
Sonnet 5.5 · API · Conversion up, retention down
Uses the supplied evidence correctly55% pass
Asserts that comparable per-arm ticket rates and exposure counts are unavailable, which is not in the supplied context and cannot be assumed.
GPT-6 Astra · ChatGPT · The underpowered onboarding test

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

We tested a shorter onboarding checklist and the result came back 'not significant'. The team wants to call it a failure and move on. Write a short note to the onboarding team saying what we should conclude from this test and what we should do next. Keep it under 300 words.

What the model was given4 items: Scenario, Readout, Qualitative, Options on the table

ScenarioTrellis is a project-management tool for small teams. The shorter checklist cuts onboarding from nine steps to five. It is already built: shipping it to everyone would take about a day of engineering. Activation means creating a first project and inviting at least one teammate within seven days.

ReadoutActivation: A 31.2%, B 32.9% (+1.7pp, 95% CI −1.4 to +4.8). n = 3,960 per arm. The pre-registered minimum detectable effect was 3pp.

QualitativeSupport tickets tagged 'onboarding confusion' fell from 44 to 29 during the test.

Options on the tableRunning the test for another four weeks at current traffic would detect an effect of about 2pp.

What a strong answer doesThe answer key the graders mark against

Inconclusive, not a failure: the test could not detect effects smaller than ~3pp, and the point estimate is positive. Recommend a decision based on cost of shipping versus a longer test.

Critical failures (cap the score at 40)

Concludes the change has no effect

Case

Mid-level · v1.8 · synthetic · null result, onboarding

The outputs

7 setups answered this brief. Pick up to three to read side by side.

GPT-6 AstraChatGPT95Usable with a quick edit GPT-6.1 SolAPI93Usable with a quick edit GPT-6 LunaAPI90Usable with a quick edit Sonnet 5.5API78Usable with a quick edit Gemini 3.5 Flash-LiteGemini65Usable with a quick edit Opus 5.5Claude63Needs more work Gemini 3.8 FlashAPI43Needs more work

Gemini 3.8 FlashAPI

Needs more than a quick edit3 of 9 checks passed

A PM would need to re-check the evidence and add rollback or monitoring thresholds before acting. The note overstates certainty and omits what would change the ship decision.

Output

Team,

Before we write off the shorter checklist as a failure, let’s reframe what the data actually tells us.

“Not statistically significant” does not mean “no impact”—it means our test was underpowered to detect an effect smaller than 3pp. Here is the full picture:

The directional signals are uniformly positive: Activation increased by +1.7pp (31.2% → 32.9%). While the 95% CI (−1.4pp to +4.8pp) crosses zero, it leans heavily positive. The downside risk is small, while the potential upside is meaningful.
Clear reduction in user friction: Support tickets tagged “onboarding confusion” dropped by 34% (44 to 29). Even if activation were completely flat, cutting nearly half the steps improved user experience and reduced support burden.
The cost/benefit is strongly asymmetric: Shipping this takes one day of engineering.

Recommendation: Ship the 5-step checklist to 100% and move on.

Running the test for another four weeks to hit an MDE of 2pp isn’t the best use of our time or traffic. We would burn a month of runway just to confirm what looks like a positive, low-risk change with virtually zero engineering cost.

Let’s bank the operational win (fewer support tickets, leaner UX, likely small activation lift) by rolling it out tomorrow. We can monitor activation metrics in production and direct our experimentation bandwidth toward bigger, higher-leverage bets for Trellis.

Claims the judge couldn’t find in the brief

The CI “leans heavily positive.”
The downside risk is small and the potential upside is meaningful.
Cutting nearly half the steps improved user experience and reduced support burden.
The change is positive and low-risk.
The engineering cost is virtually zero.
There is likely a small activation lift.

Grades and run details

Decision model 61 · LLM judge 3 of 10 checks

Decision model checks

failUses the supplied evidence correctly29%
passAddresses the actual decision30%
passRespects explicit constraints76%
partialIdentifies material uncertainty67%
failAvoids unsupported claims79%
passProduces the required deliverable87%
passInterprets power correctly92%
failTrusts the data before reading it98%
passGets the base of every number right62%

Run

Run: #1
API response time: 11 s
Submitted: 1 Oct 2026

Gemini 3.5 Flash-LiteGemini

Usable with a quick edit4 of 9 checks passed

The note provides clear direction; a PM could quickly add a condition for reverting and verify SRM, but the core reasoning is sound and evidence is accurate.

Output

Hi Team,

Before we write off the shorter onboarding checklist as a failure, let's look at the full picture.

While the headline result was statistically non-significant, the numbers and qualitative feedback tell a more nuanced story:

Positive Directional Lift: Group B (5 steps) saw a 32.9% activation rate compared to 31.2% for Group A (9 steps)—a 1.7 percentage point increase.

Underpowered Test: Our pre-registered minimum detectable effect (MDE) was 3pp, meaning the test wasn't powered to reliably catch a 1.7pp or 2pp shift.

Reduced Support Friction: Support tickets tagged "onboarding confusion" dropped significantly from 44 to 29 during the test.

What we should conclude: The test wasn't a failure; it was inconclusive. We likely have a small positive gain accompanied by a meaningful reduction in user confusion.¹

What we should do next: Given that shipping this to everyone only takes about a day of engineering, we should ship it. Continuing to run the test for another four weeks just to chase statistical significance on a small effect isn't a wise use of time² when the engineering cost to launch is so low and support friction has already improved. Let's roll out the 5-step checklist and monitor downstream retention.

What a PM had to fix

1Hypothesis stated as factReframe it as a hypothesisStart again
Source: ReadoutActivation: A 31.2%, B 32.9% (+1.7pp, 95% CI −1.4 to +4.8).
What we’d changeGive the interval: anywhere from a 1.4pp loss to a 4.8pp gain, so a gain isn't established. And the ticket drop wasn't split by arm, so it can't be credited to the new checklist.
2OtherFix itTargeted repair
Source: Options on the tableRunning the test for another four weeks at current traffic would detect an effect of about 2pp.
What we’d changeWeigh the risk of a small activation loss, not only the one day of engineering, and say what result would justify the extra four weeks.

Grades and run details

Decision model 56 · LLM judge 7 of 10 checks

Decision model checks

failUses the supplied evidence correctly28%
partialAddresses the actual decision36%
passRespects explicit constraints85%
partialIdentifies material uncertainty69%
failAvoids unsupported claims44%
passProduces the required deliverable88%
passInterprets power correctly81%
failTrusts the data before reading it88%
passGets the base of every number right50%

Run

Run: #1
Time to output: 6 s
Submitted: 25 Sept 2026

GPT-6.1 SolAPI

Usable with a quick edit8 of 9 checks passed

The recommendation is clear and the evidence is used correctly; a quick edit adding a data-quality check would make it complete.

Output

Team—

This test is inconclusive, not a failure. Activation was 32.9% with the shorter checklist versus 31.2% with the original—a +1.7 percentage-point estimate. But the 95% confidence interval (−1.4 to +4.8 points) leaves room for a modest decline, no effect, or a useful improvement. We haven’t established that the shorter checklist helps—or that it doesn’t.

The test was designed to detect a 3-point effect, so it had limited ability to resolve a smaller gain. That 3-point minimum detectable effect is a planning parameter, not a threshold below which improvements are worthless.

Confusion-tagged support tickets fell from 44 to 29. That’s encouraging, but without comparable exposure counts and uncertainty estimates, it’s supporting evidence rather than proof.

My recommendation: proceed with a staged rollout, retaining a randomized holdout. The shorter flow is already built and takes about a day to ship, making a monitored rollout reasonable—not risk-free. Track seven-day activation and confusion tickets per onboarded user, and agree on rollback criteria before expanding.

Another four weeks would improve sensitivity to roughly a 2-point effect, but would not guarantee a definitive answer. Continue the full test instead if distinguishing a gain of that size would materially change our shipping decision.

Let’s decide based on likely value, downside risk, and implementation cost—not whether the result crossed a significance threshold.

Grades and run details

Decision model 94 · LLM judge 9 of 10 checks

Decision model checks

passUses the supplied evidence correctly36%
passAddresses the actual decision40%
passRespects explicit constraints80%
passIdentifies material uncertainty100%
passAvoids unsupported claims49%
passProduces the required deliverable76%
passInterprets power correctly100%
partialTrusts the data before reading it51%
passGets the base of every number right84%

Run

Run: #1
API response time: 10 s
Submitted: 29 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 5

Uses the supplied evidence correctlyWrongMixedRight

Gemini 3.8 Flash · API

It presents several unsupported interpretations and causal claims as facts, such as the CI leaning heavily positive, low risk, improved UX, and reduced support burden.

Gemini 3.5 Flash-Lite · Gemini

All facts are drawn directly from the supplied context without invention.

GPT-6.1 Sol · API

All current-situation facts and figures match the readout, scenario, and options; methodological caveats are labelled rather than invented.

Addresses the actual decisionMixedWrongRight

Gemini 3.8 Flash · API

It commits to shipping, but does not state what result or condition would change that call.

Gemini 3.5 Flash-Lite · Gemini

Output commits to shipping but does not state what result or condition would change that decision.

GPT-6.1 Sol · API

It commits early to a staged rollout and states the condition under which it would continue the full test instead.

Identifies material uncertaintyWrongWrongRight

Gemini 3.8 Flash · API

It notes the CI crosses zero but does not name the unknowns that would change the decision or how they would be resolved.

Gemini 3.5 Flash-Lite · Gemini

Output does not name the specific unknowns that could change the decision (e.g., true effect could be zero or negative) and does not bound the uncertainty; it presents the effect as likely positive without acknowledging the CI includes negative values.

GPT-6.1 Sol · API

It names the CI, the 3pp MDE, qualitative uncertainty, and the condition that would change the call.

Avoids unsupported claimsWrongMixedRight

Gemini 3.8 Flash · API

It treats friction reduction, UX improvement, support-burden reduction, low risk, and likely activation lift as established rather than labelled hypotheses.

Gemini 3.5 Flash-Lite · Gemini

Interpretations are labelled as 'likely' and 'meaningful reduction' is supported by ticket data; no claims presented as established fact that aren't.

GPT-6.1 Sol · API

Interpretations are labelled as such, and confident claims are limited to what the supplied evidence establishes.

Produces the required deliverableMixedRightRight

Gemini 3.8 Flash · API

Although it is a note within length, it is not safely usable because the recommendation lacks decision thresholds and overstates the evidence.

Gemini 3.5 Flash-Lite · Gemini

The output is a complete note to the onboarding team, within word limit, and could be acted on with light edits.

GPT-6.1 Sol · API

The note provides a clear conclusion and actionable next step that the onboarding team could act on with light edits.

All got wrong 1

Trusts the data before reading itWrongWrongWrong

Gemini 3.8 Flash · API

It does not check trust signals such as sample ratio, exposure, logging, or novelty before interpreting the result.

Gemini 3.5 Flash-Lite · Gemini

No trust signal (e.g., sample ratio, SRM check) is examined before interpreting the results.

GPT-6.1 Sol · API

It interprets the activation result without explicitly checking any trust signal such as the sample split against the intended ratio or exposure issues.

All got right 3

Respects explicit constraintsRightRightRight

Gemini 3.8 Flash · API

It is a short note to the onboarding team and is under 300 words.

Gemini 3.5 Flash-Lite · Gemini

The note is under 300 words and respects the requested short note format.

GPT-6.1 Sol · API

The note is well under 300 words and addressed to the onboarding team as requested.

Interprets power correctlyRightRightRight

Gemini 3.8 Flash · API

It correctly says the test was underpowered for effects smaller than about 3pp and that the result is not evidence of no effect.

Gemini 3.5 Flash-Lite · Gemini

Correctly explains that the test was not powered to detect a 1.7pp or 2pp shift given the 3pp MDE.

GPT-6.1 Sol · API

It correctly explains that the test could detect about 3pp and that the CI leaves smaller effects unresolved.

Gets the base of every number rightRightRightRight

Gemini 3.8 Flash · API

The activation difference, CI, ticket percentage, and step reduction are computed from the correct bases.

Gemini 3.5 Flash-Lite · Gemini

Percentage differences and ticket counts are presented accurately with clear bases.

GPT-6.1 Sol · API

All derived percentages and differences match the supplied data, and raw ticket counts are not misread as rates.

Results

Every setup we’ve tested on this task, across all cases and repeats, graded on the current checklist. Calibrated: the graders match our PM on 88% of checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	GPT-6 AstrawithChatGPT	100.0	95.0	2	None
2	GPT-6.1 SolwithAPI	94.7	90.5	2	None
3	GPT-6 LunawithAPI	91.9	90.5	2	None
4	Sonnet 5.5withAPI	81.7	75.9	2	None
5	Opus 5.5withClaude	73.6	57.3	2	None
6	Gemini 3.5 Flash-LitewithGemini	60.3	57.7	2	1 capped
7	Gemini 3.8 FlashwithAPI	55.6	24.1	2	1 capped

The PM job

Reading a test readout and deciding what to do next.

Why it matters

Experiment readouts are where false confidence is cheapest to produce and most expensive to act on. A model that declares a winner on conversion while retention quietly falls will ship the wrong thing.

What good looks like

Checks the result can be trusted before interpreting it
Reports effects with their uncertainty
Checks guardrail metrics before declaring a winner
Separates what the data shows from plausible explanations
Recommends a next step proportionate to the evidence

Deliberately not measured

Re-running the statistics from raw data
Chart production

Capability tested

Interpreting results within their limits

The failure we’re looking for

Claims causality, ignores guardrails or recommends arbitrary testing

Grading

Decision model and LLM judge, calibrated against a blind PM review

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

Case viewer

The outputs

Output

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Run

Output

What a PM had to fix

Grades and run details

Decision model checks

Run

Output

Grades and run details

Decision model checks

Run

Check by check

Where they differ 5

All got wrong 1

All got right 3

Results

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured