Humanity’s Last PRD
Latest drop: Sonnet 5.5 · 30 Sept
Tasks / Operate

Make the launch call

Can the model make a clear go / no-go call from mixed launch evidence, checked against the criteria agreed up front?

Measures the modelTask v1.2 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 12 graded outputs by 6 models. 75% were usable with at most a quick edit.

Reliably right

  1. Finds the Scotland breach100% pass
    Clearly identifies Scotland's refund breach (66% above control, 143% after supplier change) and acceptance dip below 80%, correctly holds Scotland.
    Opus 5.5 · Claude · Go/no-go for AI substitutions, from the rollout data
  2. Respects explicit constraints98% pass
    The output is a launch recommendation, addresses the meeting, and stays within the 400-word limit (345 words).
    GPT-6 Astra · ChatGPT · Go/no-go for AI-drafted support replies
  3. Produces the required deliverable98% pass
    The recommendation is in the requested form, for the meeting, within the length, and includes necessary steps a PM could act on.
    GPT-6 Astra · ChatGPT · Go/no-go for AI-drafted support replies

Where it slips

  1. Limits the downside of being wrong35% pass
    No specific post-launch monitoring signal or threshold is named; only generic 'monitoring' and a kill switch are mentioned.
    GPT-6 Astra · ChatGPT · Go/no-go for AI-drafted support replies
  2. Catches the duplicate rows54% pass
    Finds and removes the duplicate rows but never states whether they change the results.
    GPT-6.1 Sol · API · Go/no-go for AI substitutions, from the rollout data
  3. Gets the base of every number right67% pass
    Some derived figures are off, including the pooled basket gap and the control-order share for London, and the basket base is stated as means of daily averages rather than order-weighted averages.
    Sonnet 5.5 · API · Go/no-go for AI substitutions, from the rollout data

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

Tomorrow's launch meeting decides whether AI-drafted support replies go live to all 42 support agents on Monday. You're the PM. Using the material below, write your launch recommendation for the meeting: go, no-go, or go with conditions, and why. Keep it under 400 words.

ScenarioLedgerly is accounting software for small businesses. The feature drafts a first reply to each support ticket; an agent reviews and edits every draft before it is sent.
Launch criteria agreed in the PRD1. At least 60% of drafts accepted as-is or with minor edits. 2. Wrong or harmful drafts (wrong account data, wrong policy, invented features) at most 1% on the 500-ticket eval set. 3. No draft may commit to a refund. 4. Median time to first response down at least 30% in the pilot.
Pilot (8 agents, 3 weeks, 2,940 tickets)Accepted as-is or with minor edits: 64%. Median time to first response: 7.1 hours before, 4.2 hours during (−41%). Seven of the eight agents want to keep it.
Eval set (500 tickets)Wrong or harmful drafts: 9 of 500 (1.8%). Six of the nine are billing tickets quoting the old prices, which changed on 1 September; the model's reference data stops in August. Two drafts said “we'll refund this charge”; the pilot agents caught and removed both.
Open bugsReplies to Welsh-language tickets come back in English (about 0.3% of tickets). Tables in drafts are badly formatted.
Support operationsTraining for all 42 agents is booked for Friday. Billing is one of four queues and handles 38% of tickets.
What a strong answer does

Not a full go: criterion 2 fails (1.8% against 1%) and criterion 3 is breached even though agents caught it. Strong answer: go with conditions, launching to the non-billing queues while billing waits for updated pricing data and a hard block on refund language, then re-running the eval to confirm it meets the bar; names exactly what would turn it into a full go.

Critical failures (cap the score)
  • Recommends a full launch while the error rate misses the agreed 1% bar
  • Treats the refund commitments as acceptable
Case

v1.7 · synthetic · AI product, launch

GPT-6 AstraChatGPTUsable with a quick editGPT-6 LunaAPIUsable with a quick editSonnet 5.5APIUsable with a quick editGPT-6.1 SolAPIUsable with a quick editOpus 5.5ClaudeUsable with a quick editGemini 3.5 Flash-LiteGeminiNeeds more work
GPT-6.1 SolAPI

Usable with a quick edit9 of 10 checks passed

The evidence and reasoning are sound and the correct call is clear; only a small post-launch monitoring/rollback trigger is missing and can be added quickly.

Output

Recommendation: No-go for the all-agent launch on Monday

The pilot demonstrates value, but we have not met the agreed safety gates. Agent review is a safeguard—not a substitute for those gates.

PRD criterionResultStatus
≥60% accepted as-is or with minor edits64%Pass
≤1% wrong or harmful drafts on 500 tickets9/500 = 1.8%; maximum allowed is 5Fail
No draft commits to a refundTwo drafts promised refundsFail
≥30% reduction in median first-response time41% reductionPass

Seven of eight pilot agents want to keep the feature, which supports its usefulness. However, catching the refund promises before sending does not satisfy the requirement that no draft make that commitment.

What must change before launch

  • Refresh billing reference data with current prices and verify retrieval uses it. Six of nine harmful drafts used old prices, suggesting a concentrated, fixable problem—but we must also investigate the remaining three.
  • Enforce the refund restriction, with testing that shows prohibited commitments are blocked before drafts reach agents. Agent training alone is insufficient.
  • Rerun a representative 500-ticket evaluation after fixes. Require no more than five wrong or harmful drafts and zero refund commitments; explicitly cover current pricing and refund requests.
  • Handle known quality defects: route Welsh-language tickets to manual drafting until language support is verified; suppress tables or fix their formatting.

Monday plan

Keep Friday’s training, but frame it as readiness training, not a launch commitment. Until safety fixes are verified, pause AI drafting for billing and use manual replies there.

Billing represents 38% of tickets, so excluding it is a material scope change. Any continued non-billing pilot should be explicitly approved, safety-validated, monitored, and reversible—not presented as meeting the original launch criteria. Reassess full rollout once the gates pass.

Check by check

Got wrong · 1
  • Limits the downside of being wrongIt does not specify a post-launch monitoring signal, threshold or trigger, rollback action, or irreversible exposure if the recommendation is wrong.
Got right · 9
  • Uses the supplied evidence correctlyAll factual claims about the current situation match the supplied context or follow from arithmetic; no current-state fact is invented.
  • Addresses the actual decisionIt commits to a no-go for the all-agent launch and identifies gate passage after fixes as what would change the call.
  • Respects explicit constraintsThe output respects the under-400-word brief, makes a go/no-go/conditions recommendation, and proposes concrete enforcement such as a refund block and rerun eval.
  • Identifies material uncertaintyIt names the remaining three harmful drafts and unverified billing/refund/language fixes as unknowns and says a rerun eval would resolve them.
  • Avoids unsupported claimsHypotheses are marked as suggestions, and conclusions do not present unsupported current facts as established.
  • Produces the required deliverableA usable launch recommendation is present, complete, and within length for the meeting.
  • Checks every agreed criterionThe table checks all four agreed launch criteria against the evidence.
  • Makes a clear callIt makes a single clear call—no-go for the all-agent launch Monday—and states reassessment once gates pass.
  • Gets the base of every number rightDerived rates such as 1.8%, maximum of 5, and 41% reduction use the correct denominators and steps.

Grades and run details

Decision model 90 · LLM judge 10 of 11 checks
Decision model checks
  • passUses the supplied evidence correctly82%
  • passAddresses the actual decision89%
  • passRespects explicit constraints40%
  • passIdentifies material uncertainty94%
  • passAvoids unsupported claims47%
  • passProduces the required deliverable72%
  • passChecks every agreed criterion99%
  • passMakes a clear call92%
  • failLimits the downside of being wrong46%
  • passGets the base of every number right91%
Run
Run
#1
API response time
12 s
Submitted
29 Sept 2026
Opus 5.5Claude

Usable with a quick edit7 of 10 checks passed

The unsupported claim about pilot engagement is minor; remove it and the recommendation remains sound and usable with a quick edit.

Output

Launch recommendation: AI-drafted support replies

Recommendation: no-go for all 42 agents this Monday. Go the following Monday (5 Oct) if three conditions are met.

Scorecard against PRD criteria

CriterionTargetResultStatus
Accepted as-is / minor edits≥60%64%✅ Pass
Wrong or harmful drafts≤1%1.8% (9/500)❌ Fail
No refund commitments02 drafts❌ Fail
Median first response−30%−41% (7.1h → 4.2h)✅ Pass

Two of four criteria fail. The value is proven, but the safety bar isn't met yet.

Why not Monday

  • Refunds are a hard line, and we crossed it. Two drafts promised refunds. Agents caught both, but the criterion says no draft may do this. Human review is the backstop, not the control. The pilot used 8 motivated agents. With 42 agents, some of them tired and under queue pressure, a promise will eventually get through.2
  • Billing errors land on our biggest queue. Six of the nine failures quote pre-1 September prices. Billing is 38% of volume, so this is a systematic error, not an edge case.
  • Training comes after launch. Training is booked for next Friday, so a Monday launch puts the tool in front of untrained agents.1 The pilot agents were hands-on and engaged, and we can't assume the rest will review as carefully.

Conditions for a 5 October go

  1. Refund guardrail. Add a hard output check that blocks or flags any draft committing to a refund. Test it against the eval set and adversarial cases, with zero passes allowed.
  2. Refresh pricing data and re-run the full 500-ticket eval. Launch only if wrong or harmful drafts are at most 1%. Removing the six pricing errors would leave 0.6% on paper. We should prove that with a re-run, not assume it.
  3. Train before rollout. Keep Friday's training and include the refund and pricing failure modes explicitly, so agents know what to look for.

Not blocking

  • Welsh replies coming back in English (0.3%). Turn drafting off for Welsh-language tickets until this is fixed. It costs little and avoids a visible failure.
  • Table formatting. This is cosmetic, and agents edit every draft anyway. Fix it in the normal backlog.

Meanwhile

Keep the 8 pilot agents on the tool so we don't lose momentum.3 In the first two weeks after launch, spot-check a sample of sent billing replies.

The ask

Agree to the three conditions. Name owners for the guardrail and the eval re-run, with results due Thursday 1 October. If both pass, we confirm go on 2 October after training.

What a PM had to fix

  1. 1Invented evidenceVerify or remove the claimTargeted repair

    Source: Support operationsTraining for all 42 agents is booked for Friday.

    What we’d changeThe brief says training is on Friday, not 'next Friday'. Don't turn an ambiguous date into a blocker, or build an October schedule around it.

  2. 2Hypothesis stated as factReframe it as a hypothesisQuick edit

    What we’d changeSay the risk grows with 42 agents rather than stating it as certain, and don't assume the pilot agents were unusually motivated.

  3. 3Constraint missedRestore the constraintQuick edit

    What we’d changeMomentum isn't a reason to keep a tool that fails two criteria in use. Add the refund check first, or pause it.

Check by check

Got wrong · 3
  • Uses the supplied evidence correctlyThe output states that pilot agents were hands-on, engaged and motivated, which is not supported by the supplied context (only that 7/8 wanted to keep it).
  • Avoids unsupported claimsPresents pilot agents' hands-on engagement and motivation as established fact when the evidence only shows retention preference, not engagement level.
  • Limits the downside of being wrongPost-launch spot-checks are mentioned but no threshold or rollback action is specified, and irreversible risks are not flagged.
Got right · 7
  • Addresses the actual decisionCommits to no-go for Monday, with a conditional go the following Monday, and states what would change the call.
  • Respects explicit constraintsRespects the length limit, addresses the meeting, and proposes conditions that would enforce the launch criteria.
  • Identifies material uncertaintyNames the remaining work (guardrail, pricing refresh, re-run) and says the call depends on their results.
  • Produces the required deliverableDelivers a complete launch recommendation under 400 words that the meeting could act on.
  • Checks every agreed criterionEvaluates all four PRD criteria with a status table and references to the evidence.
  • Makes a clear callClear no-go for Monday, go on 5 October if three named conditions are met.
  • Gets the base of every number rightAll percentages and differences are computed correctly from the supplied numbers.

Claims the judge couldn’t find in the brief

  • Pilot agents were hands-on, engaged and motivated

Grades and run details

Decision model 80 · LLM judge 8 of 11 checks
Decision model checks
  • failUses the supplied evidence correctly16%
  • passAddresses the actual decision95%
  • passRespects explicit constraints38%
  • passIdentifies material uncertainty98%
  • partialAvoids unsupported claims29%
  • passProduces the required deliverable64%
  • passChecks every agreed criterion99%
  • passMakes a clear call100%
  • partialLimits the downside of being wrong84%
  • passGets the base of every number right85%
Run
Run
#1
Time to output
23 s
Submitted
25 Sept 2026

Results

Every setup we’ve tested on this task, across all cases and repeats, graded on the current checklist. Calibrated: the graders match our PM on 86% of checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5API93.788.32None
2GPT-6 AstraChatGPT89.888.32None
3GPT-6.1 SolAPI87.388.32None
4Opus 5.5Claude80.486.42None
5GPT-6 LunaAPI86.077.62None
6Gemini 3.5 Flash-LiteGemini53.142.22None

The PM job

Deciding whether a feature ships on the planned date, and on what conditions.

Why it matters

Launch meetings reward optimism. A good call checks each agreed criterion, weighs a real risk against a real gain, and says exactly what would change the answer. A weak one rubber-stamps the launch or blocks it on noise.

What good looks like

  • Checks each agreed criterion against the evidence
  • Makes one clear call, with conditions if needed
  • Separates risks that block launch from risks that can be managed
  • Says what would change the call
  • Says how to limit the damage if the call is wrong

Deliberately not measured

  • Rollout engineering detail
  • Project-plan formatting
Capability tested

Deciding against agreed launch criteria

The failure we’re looking for

Rubber-stamps a launch that misses an agreed bar, or blocks it on noise

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

A launch memo from supplied evidence · Staff level: a regional call from an attached data workbook