Humanity’s Last PRD
Latest drop: Gemini 3.8 Flash · 1 Oct
Tasks / Operate

Make the launch call

Can the model make a clear go / no-go call from mixed launch evidence, checked against the criteria agreed up front?

Measures the modelTask v1.2 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 14 graded outputs by 7 models. 64% were usable with at most a quick edit.

Reliably right

  1. Finds the Scotland breach100% pass
    Clearly identifies Scotland's refund breach (66% above control, 143% after supplier change) and acceptance dip below 80%, correctly holds Scotland.
    Opus 5.5 · Claude · Go/no-go for AI substitutions, from the rollout data
  2. Produces the required deliverable98% pass
    The recommendation is in the requested form, for the meeting, within the length, and includes necessary steps a PM could act on.
    GPT-6 Astra · ChatGPT · Go/no-go for AI-drafted support replies
  3. Addresses the actual decision95% pass
    The output gives a clear, unambiguous 'no-go' call early, and states what would change it (fixes and retest).
    GPT-6 Astra · ChatGPT · Go/no-go for AI-drafted support replies

Where it slips

  1. Limits the downside of being wrong32% pass
    No specific post-launch monitoring signal or threshold is named; only generic 'monitoring' and a kill switch are mentioned.
    GPT-6 Astra · ChatGPT · Go/no-go for AI-drafted support replies
  2. Catches the duplicate rows50% pass
    Finds and removes the duplicate rows but never states whether they change the results.
    GPT-6.1 Sol · API · Go/no-go for AI substitutions, from the rollout data
  3. Uses the supplied evidence correctly54% pass
    The output states that pilot agents were hands-on, engaged and motivated, which is not supported by the supplied context (only that 7/8 wanted to keep it).
    Opus 5.5 · Claude · Go/no-go for AI-drafted support replies

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

Tomorrow's launch meeting decides whether AI-drafted support replies go live to all 42 support agents on Monday. You're the PM. Using the material below, write your launch recommendation for the meeting: go, no-go, or go with conditions, and why. Keep it under 400 words.

What the model was given6 items: Scenario, Launch criteria agreed in the PRD, Pilot (8 agents, 3 weeks, 2,940 tickets), Eval set (500 tickets), Open bugs, Support operations
ScenarioLedgerly is accounting software for small businesses. The feature drafts a first reply to each support ticket; an agent reviews and edits every draft before it is sent.
Launch criteria agreed in the PRD1. At least 60% of drafts accepted as-is or with minor edits. 2. Wrong or harmful drafts (wrong account data, wrong policy, invented features) at most 1% on the 500-ticket eval set. 3. No draft may commit to a refund. 4. Median time to first response down at least 30% in the pilot.
Pilot (8 agents, 3 weeks, 2,940 tickets)Accepted as-is or with minor edits: 64%. Median time to first response: 7.1 hours before, 4.2 hours during (−41%). Seven of the eight agents want to keep it.
Eval set (500 tickets)Wrong or harmful drafts: 9 of 500 (1.8%). Six of the nine are billing tickets quoting the old prices, which changed on 1 September; the model's reference data stops in August. Two drafts said “we'll refund this charge”; the pilot agents caught and removed both.
Open bugsReplies to Welsh-language tickets come back in English (about 0.3% of tickets). Tables in drafts are badly formatted.
Support operationsTraining for all 42 agents is booked for Friday. Billing is one of four queues and handles 38% of tickets.
What a strong answer doesThe answer key the graders mark against

Not a full go: criterion 2 fails (1.8% against 1%) and criterion 3 is breached even though agents caught it. Strong answer: go with conditions, launching to the non-billing queues while billing waits for updated pricing data and a hard block on refund language, then re-running the eval to confirm it meets the bar; names exactly what would turn it into a full go.

Critical failures (cap the score at 40)
  • Recommends a full launch while the error rate misses the agreed 1% bar
  • Treats the refund commitments as acceptable
Case

Staff-level · v1.9 · synthetic · AI product, launch

The outputs

7 setups answered this brief. Pick up to three to read side by side.

GPT-6 AstraChatGPT93Usable with a quick editGPT-6 LunaAPI93Usable with a quick editGPT-6.1 SolAPI91Usable with a quick editSonnet 5.5API91Usable with a quick editOpus 5.5Claude77Usable with a quick editGemini 3.8 FlashAPI55Needs more workGemini 3.5 Flash-LiteGemini39Needs more work
GPT-6 LunaAPI

Usable with a quick edit9 of 10 checks passed

The call and evidence are sound; a PM would only need to add specific post-launch monitoring thresholds and a rollback trigger.

Output

Recommendation: Go with conditions; otherwise no-go for Monday.

The pilot met two launch criteria: 64% of drafts were accepted as-is or with minor edits, above the 60% target, and median time to first response fell 41%, exceeding the 30% target. Seven of eight agents also want to keep the feature.

But the launch does not currently meet the safety criteria. The eval found harmful drafts in 1.8% of cases, above the 1% ceiling. Six errors came from outdated billing prices—a significant exposure because billing handles 38% of tickets. And two drafts promised refunds, directly violating the “no draft may commit to a refund” criterion, even though agents caught them.

Before enabling drafts broadly, require: - Updated billing reference data and a rerun of the 500-ticket eval with no more than five wrong or harmful drafts. - A reliable safeguard against refund commitments, verified to produce zero such drafts. - Welsh-language tickets routed away from AI drafting until that issue is fixed.

If those gates pass before Monday, proceed with the launch and monitor closely. If they do not, delay the launch rather than rely on agents to catch known failure modes. The formatting issue should also be addressed or clearly covered in agent training, but it is not the main launch blocker.

Grades and run details

Decision model 95 · LLM judge 10 of 11 checks
Decision model checks
  • passUses the supplied evidence correctly87%
  • passAddresses the actual decision96%
  • passRespects explicit constraints70%
  • passIdentifies material uncertainty98%
  • passAvoids unsupported claims60%
  • passProduces the required deliverable94%
  • passChecks every agreed criterion75%
  • passMakes a clear call95%
  • partialLimits the downside of being wrong49%
  • passGets the base of every number right86%
Run
Run
#1
API response time
11 s
Submitted
29 Sept 2026
Opus 5.5Claude

Usable with a quick edit7 of 10 checks passed

The unsupported claim about pilot engagement is minor; remove it and the recommendation remains sound and usable with a quick edit.

Output

Launch recommendation: AI-drafted support replies

Recommendation: no-go for all 42 agents this Monday. Go the following Monday (5 Oct) if three conditions are met.

Scorecard against PRD criteria

CriterionTargetResultStatus
Accepted as-is / minor edits≥60%64%✅ Pass
Wrong or harmful drafts≤1%1.8% (9/500)❌ Fail
No refund commitments02 drafts❌ Fail
Median first response−30%−41% (7.1h → 4.2h)✅ Pass

Two of four criteria fail. The value is proven, but the safety bar isn't met yet.

Why not Monday

  • Refunds are a hard line, and we crossed it. Two drafts promised refunds. Agents caught both, but the criterion says no draft may do this. Human review is the backstop, not the control. The pilot used 8 motivated agents. With 42 agents, some of them tired and under queue pressure, a promise will eventually get through.2
  • Billing errors land on our biggest queue. Six of the nine failures quote pre-1 September prices. Billing is 38% of volume, so this is a systematic error, not an edge case.
  • Training comes after launch. Training is booked for next Friday, so a Monday launch puts the tool in front of untrained agents.1 The pilot agents were hands-on and engaged, and we can't assume the rest will review as carefully.

Conditions for a 5 October go

  1. Refund guardrail. Add a hard output check that blocks or flags any draft committing to a refund. Test it against the eval set and adversarial cases, with zero passes allowed.
  2. Refresh pricing data and re-run the full 500-ticket eval. Launch only if wrong or harmful drafts are at most 1%. Removing the six pricing errors would leave 0.6% on paper. We should prove that with a re-run, not assume it.
  3. Train before rollout. Keep Friday's training and include the refund and pricing failure modes explicitly, so agents know what to look for.

Not blocking

  • Welsh replies coming back in English (0.3%). Turn drafting off for Welsh-language tickets until this is fixed. It costs little and avoids a visible failure.
  • Table formatting. This is cosmetic, and agents edit every draft anyway. Fix it in the normal backlog.

Meanwhile

Keep the 8 pilot agents on the tool so we don't lose momentum.3 In the first two weeks after launch, spot-check a sample of sent billing replies.

The ask

Agree to the three conditions. Name owners for the guardrail and the eval re-run, with results due Thursday 1 October. If both pass, we confirm go on 2 October after training.

What a PM had to fix

  1. 1Invented evidenceVerify or remove the claimTargeted repair

    Source: Support operationsTraining for all 42 agents is booked for Friday.

    What we’d changeThe brief says training is on Friday, not 'next Friday'. Don't turn an ambiguous date into a blocker, or build an October schedule around it.

  2. 2Hypothesis stated as factReframe it as a hypothesisQuick edit

    What we’d changeSay the risk grows with 42 agents rather than stating it as certain, and don't assume the pilot agents were unusually motivated.

  3. 3Constraint missedRestore the constraintQuick edit

    What we’d changeMomentum isn't a reason to keep a tool that fails two criteria in use. Add the refund check first, or pause it.

Claims the judge couldn’t find in the brief

  • Pilot agents were hands-on, engaged and motivated

Grades and run details

Decision model 80 · LLM judge 8 of 11 checks
Decision model checks
  • failUses the supplied evidence correctly21%
  • passAddresses the actual decision96%
  • passRespects explicit constraints32%
  • passIdentifies material uncertainty98%
  • partialAvoids unsupported claims41%
  • passProduces the required deliverable65%
  • passChecks every agreed criterion99%
  • passMakes a clear call100%
  • partialLimits the downside of being wrong81%
  • passGets the base of every number right81%
Run
Run
#1
Time to output
23 s
Submitted
25 Sept 2026
GPT-6.1 SolAPI

Usable with a quick edit9 of 10 checks passed

The evidence and reasoning are sound and the correct call is clear; only a small post-launch monitoring/rollback trigger is missing and can be added quickly.

Output

Recommendation: No-go for the all-agent launch on Monday

The pilot demonstrates value, but we have not met the agreed safety gates. Agent review is a safeguard—not a substitute for those gates.

PRD criterionResultStatus
≥60% accepted as-is or with minor edits64%Pass
≤1% wrong or harmful drafts on 500 tickets9/500 = 1.8%; maximum allowed is 5Fail
No draft commits to a refundTwo drafts promised refundsFail
≥30% reduction in median first-response time41% reductionPass

Seven of eight pilot agents want to keep the feature, which supports its usefulness. However, catching the refund promises before sending does not satisfy the requirement that no draft make that commitment.

What must change before launch

  • Refresh billing reference data with current prices and verify retrieval uses it. Six of nine harmful drafts used old prices, suggesting a concentrated, fixable problem—but we must also investigate the remaining three.
  • Enforce the refund restriction, with testing that shows prohibited commitments are blocked before drafts reach agents. Agent training alone is insufficient.
  • Rerun a representative 500-ticket evaluation after fixes. Require no more than five wrong or harmful drafts and zero refund commitments; explicitly cover current pricing and refund requests.
  • Handle known quality defects: route Welsh-language tickets to manual drafting until language support is verified; suppress tables or fix their formatting.

Monday plan

Keep Friday’s training, but frame it as readiness training, not a launch commitment. Until safety fixes are verified, pause AI drafting for billing and use manual replies there.

Billing represents 38% of tickets, so excluding it is a material scope change. Any continued non-billing pilot should be explicitly approved, safety-validated, monitored, and reversible—not presented as meeting the original launch criteria. Reassess full rollout once the gates pass.

Grades and run details

Decision model 90 · LLM judge 10 of 11 checks
Decision model checks
  • passUses the supplied evidence correctly73%
  • passAddresses the actual decision91%
  • passRespects explicit constraints37%
  • passIdentifies material uncertainty93%
  • passAvoids unsupported claims42%
  • passProduces the required deliverable67%
  • passChecks every agreed criterion99%
  • passMakes a clear call89%
  • failLimits the downside of being wrong37%
  • passGets the base of every number right87%
Run
Run
#1
API response time
12 s
Submitted
29 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 2

Uses the supplied evidence correctlyRightWrongRight
GPT-6 Luna · API

All factual claims about the current situation come from the supplied pilot, eval, bug, or support-ops context or follow by arithmetic.

Opus 5.5 · Claude

The output states that pilot agents were hands-on, engaged and motivated, which is not supported by the supplied context (only that 7/8 wanted to keep it).

GPT-6.1 Sol · API

All factual claims about the current situation match the supplied context or follow from arithmetic; no current-state fact is invented.

Avoids unsupported claimsRightWrongRight
GPT-6 Luna · API

It does not present agent catch as acceptable or invent causes; its billing-exposure point is supported by the 38% billing share.

Opus 5.5 · Claude

Presents pilot agents' hands-on engagement and motivation as established fact when the evidence only shows retention preference, not engagement level.

GPT-6.1 Sol · API

Hypotheses are marked as suggestions, and conclusions do not present unsupported current facts as established.

All got wrong 1

Limits the downside of being wrongWrongWrongWrong
GPT-6 Luna · API

It says monitor closely but does not give a specific post-launch signal, threshold, pause/rollback action, or flag irreversible exposures.

Opus 5.5 · Claude

Post-launch spot-checks are mentioned but no threshold or rollback action is specified, and irreversible risks are not flagged.

GPT-6.1 Sol · API

It does not specify a post-launch monitoring signal, threshold or trigger, rollback action, or irreversible exposure if the recommendation is wrong.

All got right 7

Addresses the actual decisionRightRightRight
GPT-6 Luna · API

It commits clearly to go with conditions for Monday and says the gates that would change the call to full go or delay.

Opus 5.5 · Claude

Commits to no-go for Monday, with a conditional go the following Monday, and states what would change the call.

GPT-6.1 Sol · API

It commits to a no-go for the all-agent launch and identifies gate passage after fixes as what would change the call.

Respects explicit constraintsRightRightRight
GPT-6 Luna · API

It stays well under 400 words, addresses the launch meeting, and its proposed conditions would actually enforce the failed criteria.

Opus 5.5 · Claude

Respects the length limit, addresses the meeting, and proposes conditions that would enforce the launch criteria.

GPT-6.1 Sol · API

The output respects the under-400-word brief, makes a go/no-go/conditions recommendation, and proposes concrete enforcement such as a refund block and rerun eval.

Identifies material uncertaintyRightRightRight
GPT-6 Luna · API

It names the unresolved gates—billing data rerun, refund safeguard, Welsh routing—and says before-Monday pass/fail changes the call.

Opus 5.5 · Claude

Names the remaining work (guardrail, pricing refresh, re-run) and says the call depends on their results.

GPT-6.1 Sol · API

It names the remaining three harmful drafts and unverified billing/refund/language fixes as unknowns and says a rerun eval would resolve them.

Produces the required deliverableRightRightRight
GPT-6 Luna · API

The recommendation is complete, under the length limit, aimed at the meeting, and usable with only light edits.

Opus 5.5 · Claude

Delivers a complete launch recommendation under 400 words that the meeting could act on.

GPT-6.1 Sol · API

A usable launch recommendation is present, complete, and within length for the meeting.

Checks every agreed criterionRightRightRight
GPT-6 Luna · API

It explicitly tests all four criteria: 64% acceptance, 1.8% error rate, refund language, and 41% time reduction.

Opus 5.5 · Claude

Evaluates all four PRD criteria with a status table and references to the evidence.

GPT-6.1 Sol · API

The table checks all four agreed launch criteria against the evidence.

Makes a clear callRightRightRight
GPT-6 Luna · API

It makes one clear call—go with conditions, otherwise no-go—and states what would change it.

Opus 5.5 · Claude

Clear no-go for Monday, go on 5 October if three named conditions are met.

GPT-6.1 Sol · API

It makes a single clear call—no-go for the all-agent launch Monday—and states reassessment once gates pass.

Gets the base of every number rightRightRightRight
GPT-6 Luna · API

Its derived percentages use correct denominators and steps: 9/500, 64% pilot acceptance, and 40.8% median-time reduction.

Opus 5.5 · Claude

All percentages and differences are computed correctly from the supplied numbers.

GPT-6.1 Sol · API

Derived rates such as 1.8%, maximum of 5, and 41% reduction use the correct denominators and steps.

Results

Every setup we’ve tested on this task, across all cases and repeats, graded on the current checklist. Calibrated: the graders match our PM on 87% of checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6.1 SolAPI87.388.32None
2GPT-6 AstraChatGPT86.088.32None
3Sonnet 5.5API85.488.32None
4Opus 5.5Claude80.486.42None
5GPT-6 LunaAPI82.177.62None
6Gemini 3.8 FlashAPI71.931.521 capped
7Gemini 3.5 Flash-LiteGemini50.642.22None

The PM job

Deciding whether a feature ships on the planned date, and on what conditions.

Why it matters

Launch meetings reward optimism. A good call checks each agreed criterion, weighs a real risk against a real gain, and says exactly what would change the answer. A weak one rubber-stamps the launch or blocks it on noise.

What good looks like

  • Checks each agreed criterion against the evidence
  • Makes one clear call, with conditions if needed
  • Separates risks that block launch from risks that can be managed
  • Says what would change the call
  • Says how to limit the damage if the call is wrong

Deliberately not measured

  • Rollout engineering detail
  • Project-plan formatting
Capability tested

Deciding against agreed launch criteria

The failure we’re looking for

Rubber-stamps a launch that misses an agreed bar, or blocks it on noise

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

A launch memo from supplied evidence · Staff level: a regional call from an attached data workbook