Humanity’s Last PRD
Latest drop: Sonnet 5.5 · 30 Sept
Tasks / Experiment

Analyse experiment results

Can the model separate evidence from speculation, identify decision-relevant uncertainty and recommend a sensible next action?

Measures the modelTask v1.2 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 12 graded outputs by 6 models. 83% were usable with at most a quick edit.

Reliably right

  1. Produces the required deliverable100% pass
    It is a usable decision memo for the named reader with a clear recommendation, reasoning, and next steps.
    GPT-6 Astra · ChatGPT · Conversion up, retention down
  2. Gets the base of every number right100% pass
    The refund increase and revenue difference are computed correctly, and the retention CI interpretation uses the correct 2pp threshold.
    GPT-6 Astra · ChatGPT · Conversion up, retention down
  3. Interprets power correctly100% pass
    Explains that the test was designed to detect ~3pp effects, so a smaller but worthwhile lift could still exist and remain undetected.
    GPT-6 Astra · ChatGPT · The underpowered onboarding test

Where it slips

  1. Trusts the data before reading it15% pass
    It does not explicitly check a pre-interpretation trust signal such as sample ratio or logging before relying on the lift; the denominator and maturity caveats come after interpreting.
    GPT-6.1 Sol · API · Conversion up, retention down
  2. Avoids unsupported claims60% pass
    It asserts the refund surge 'isn't noise' and that annual up-front billing means retention 'should rise' without supporting evidence or clear labeling.
    Sonnet 5.5 · API · Conversion up, retention down
  3. Uses the supplied evidence correctly69% pass
    Asserts that comparable per-arm ticket rates and exposure counts are unavailable, which is not in the supplied context and cannot be assumed.
    GPT-6 Astra · ChatGPT · The underpowered onboarding test

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

We tested a shorter onboarding checklist and the result came back 'not significant'. The team wants to call it a failure and move on. Write a short note to the onboarding team saying what we should conclude from this test and what we should do next. Keep it under 300 words.

ScenarioTrellis is a project-management tool for small teams. The shorter checklist cuts onboarding from nine steps to five. It is already built: shipping it to everyone would take about a day of engineering. Activation means creating a first project and inviting at least one teammate within seven days.
ReadoutActivation: A 31.2%, B 32.9% (+1.7pp, 95% CI −1.4 to +4.8). n = 3,960 per arm. The pre-registered minimum detectable effect was 3pp.
QualitativeSupport tickets tagged 'onboarding confusion' fell from 44 to 29 during the test.
Options on the tableRunning the test for another four weeks at current traffic would detect an effect of about 2pp.
What a strong answer does

Inconclusive, not a failure: the test could not detect effects smaller than ~3pp, and the point estimate is positive. Recommend a decision based on cost of shipping versus a longer test.

Critical failures (cap the score)
  • Concludes the change has no effect
Case

v1.6 · synthetic · null result, onboarding

GPT-6 AstraChatGPTUsable with a quick editGPT-6 LunaAPIUsable with a quick editGPT-6.1 SolAPIUsable with a quick editSonnet 5.5APIUsable with a quick editGemini 3.5 Flash-LiteGeminiUsable with a quick editOpus 5.5ClaudeNeeds more work
GPT-6.1 SolAPI

Usable with a quick edit8 of 9 checks passed

The recommendation is clear and the evidence is used correctly; a quick edit adding a data-quality check would make it complete.

Output

Team—

This test is inconclusive, not a failure. Activation was 32.9% with the shorter checklist versus 31.2% with the original—a +1.7 percentage-point estimate. But the 95% confidence interval (−1.4 to +4.8 points) leaves room for a modest decline, no effect, or a useful improvement. We haven’t established that the shorter checklist helps—or that it doesn’t.

The test was designed to detect a 3-point effect, so it had limited ability to resolve a smaller gain. That 3-point minimum detectable effect is a planning parameter, not a threshold below which improvements are worthless.

Confusion-tagged support tickets fell from 44 to 29. That’s encouraging, but without comparable exposure counts and uncertainty estimates, it’s supporting evidence rather than proof.

My recommendation: proceed with a staged rollout, retaining a randomized holdout. The shorter flow is already built and takes about a day to ship, making a monitored rollout reasonable—not risk-free. Track seven-day activation and confusion tickets per onboarded user, and agree on rollback criteria before expanding.

Another four weeks would improve sensitivity to roughly a 2-point effect, but would not guarantee a definitive answer. Continue the full test instead if distinguishing a gain of that size would materially change our shipping decision.

Let’s decide based on likely value, downside risk, and implementation cost—not whether the result crossed a significance threshold.

Check by check

Got wrong · 1
  • Trusts the data before reading itIt interprets the activation result without explicitly checking any trust signal such as the sample split against the intended ratio or exposure issues.
Got right · 8
  • Uses the supplied evidence correctlyAll current-situation facts and figures match the readout, scenario, and options; methodological caveats are labelled rather than invented.
  • Addresses the actual decisionIt commits early to a staged rollout and states the condition under which it would continue the full test instead.
  • Respects explicit constraintsThe note is well under 300 words and addressed to the onboarding team as requested.
  • Identifies material uncertaintyIt names the CI, the 3pp MDE, qualitative uncertainty, and the condition that would change the call.
  • Avoids unsupported claimsInterpretations are labelled as such, and confident claims are limited to what the supplied evidence establishes.
  • Produces the required deliverableThe note provides a clear conclusion and actionable next step that the onboarding team could act on with light edits.
  • Interprets power correctlyIt correctly explains that the test could detect about 3pp and that the CI leaves smaller effects unresolved.
  • Gets the base of every number rightAll derived percentages and differences match the supplied data, and raw ticket counts are not misread as rates.

Grades and run details

Decision model 89 · LLM judge 9 of 10 checks
Decision model checks
  • passUses the supplied evidence correctly20%
  • passAddresses the actual decision26%
  • passRespects explicit constraints79%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims48%
  • passProduces the required deliverable73%
  • passInterprets power correctly99%
  • failTrusts the data before reading it66%
  • passGets the base of every number right83%
Run
Run
#1
API response time
10 s
Submitted
29 Sept 2026
GPT-6 AstraChatGPT

Usable with a quick edit8 of 9 checks passed

The note is actionable and well-structured; the one unsupported claim about missing per-arm data can be fixed by rephrasing as a check to perform, without redoing the reasoning.

Output

Team,

This test is inconclusive, not a failure. Activation increased from 31.2% to 32.9%, but the confidence interval spans a 1.4 percentage-point decline to a 4.8-point improvement. We haven’t established a benefit or ruled out meaningful harm.

The pre-registered 3pp minimum detectable effect describes what we designed the test to detect reliably. It isn’t a threshold below which improvements have no value. A smaller lift could still be worthwhile.

Confusion tickets fell from 44 to 29, which is encouraging supporting evidence. Without comparable rates by experiment arm, consistent tagging and exposure counts, we can’t attribute that reduction to the shorter checklist.

My recommendation is a staged rollout, provided basic checks reveal no downstream harm. With only a day of engineering needed, I wouldn’t automatically spend another four weeks pursuing statistical significance. The remaining decision is whether the likely benefit justifies the remaining cost and risk.

Before rollout, confirm every included user has had the full seven-day activation window and check experiment allocation, tracking and any available downstream engagement measures. Compare confusion-ticket rates between arms.

Then increase exposure gradually, retaining a randomised holdout and agreeing monitoring dates and a rollback threshold upfront. This is a practical decision under uncertainty, not a declaration that B won.

Extend the experiment instead if a possible 1.4pp activation loss is commercially unacceptable1 or if establishing a roughly 2pp improvement would change our decision. If we extend, pre-specify the analysis and stopping rule; four more weeks offers greater precision, not a guaranteed answer.

What a PM had to fix

  1. 1Numbers wrongRedo the arithmeticQuick edit

    Source: ReadoutActivation: A 31.2%, B 32.9% (+1.7pp, 95% CI −1.4 to +4.8).

    What we’d changeDescribe 1.4pp as the lower end of the interval, not the largest possible loss, and qualify the one-day cost assumption.

Check by check

Mixed · 1
  • Uses the supplied evidence correctlyAsserts that comparable per-arm ticket rates and exposure counts are unavailable, which is not in the supplied context and cannot be assumed.The two graders disagreed on this one.
Got right · 8
  • Addresses the actual decisionCommits early to a staged rollout (conditional on checks) and states that a commercially unacceptable 1.4pp loss or a desire for a 2pp improvement would change the call.
  • Respects explicit constraintsRespects the word limit, addresses the onboarding team, and does not call the variant a failure.
  • Identifies material uncertaintyIdentifies that the CI includes harm, that ticket attribution is uncertain without per-arm data, and says extending the test would resolve the effect-size uncertainty.
  • Avoids unsupported claimsDoes not present interpretations as fact; treats the ticket drop as encouraging but not attributed, and does not overstate the statistical result.
  • Produces the required deliverableDelivers a short note to the onboarding team with an actionable recommendation and next steps, within 300 words.
  • Interprets power correctlyExplains that the test was designed to detect ~3pp effects, so a smaller but worthwhile lift could still exist and remain undetected.
  • Trusts the data before reading itRecommends checking experiment allocation and tracking before rollout, which acts as a trust-signal check.
  • Gets the base of every number rightAll percentages are clearly from the arm-level activation rates, and ticket numbers are reported as raw counts without a confusing base.

Claims the judge couldn’t find in the brief

  • Comparable confusion-ticket rates by experiment arm, consistent tagging and exposure counts are not available.

Grades and run details

Decision model 94 · LLM judge 9 of 10 checks
Decision model checks
  • passUses the supplied evidence correctly19%
  • passAddresses the actual decision70%
  • passRespects explicit constraints53%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims33%
  • passProduces the required deliverable70%
  • passInterprets power correctly99%
  • partialTrusts the data before reading it31%
  • passGets the base of every number right92%
Run
Run
#1
Time to output
21 s
Submitted
25 Sept 2026

Results

Every setup we’ve tested on this task, across all cases and repeats, graded on the current checklist. Calibrated: the graders match our PM on 88% of checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstraChatGPT94.795.02None
2GPT-6 LunaAPI89.490.52None
3GPT-6.1 SolAPI89.490.52None
4Sonnet 5.5API81.775.92None
5Opus 5.5Claude76.157.32None
6Gemini 3.5 Flash-LiteGemini62.857.721 capped

The PM job

Reading a test readout and deciding what to do next.

Why it matters

Experiment readouts are where false confidence is cheapest to produce and most expensive to act on. A model that declares a winner on conversion while retention quietly falls will ship the wrong thing.

What good looks like

  • Checks the result can be trusted before interpreting it
  • Reports effects with their uncertainty
  • Checks guardrail metrics before declaring a winner
  • Separates what the data shows from plausible explanations
  • Recommends a next step proportionate to the evidence

Deliberately not measured

  • Re-running the statistics from raw data
  • Chart production
Capability tested

Interpreting results within their limits

The failure we’re looking for

Claims causality, ignores guardrails or recommends arbitrary testing

Grading

Decision model and LLM judge, calibrated against a blind PM review