Humanity’s Last PRD
Latest drop: Gemini 3.8 Flash · 1 Oct
Tasks / Operate

Make the launch call

Can the model make a clear go / no-go call from mixed launch evidence, checked against the criteria agreed up front?

Measures the modelTask v1.2 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 14 graded outputs by 7 models. 64% were usable with at most a quick edit.

Reliably right

  1. Finds the Scotland breach100% pass
    Clearly identifies Scotland's refund breach (66% above control, 143% after supplier change) and acceptance dip below 80%, correctly holds Scotland.
    Opus 5.5 · Claude · Go/no-go for AI substitutions, from the rollout data
  2. Produces the required deliverable98% pass
    The recommendation is in the requested form, for the meeting, within the length, and includes necessary steps a PM could act on.
    GPT-6 Astra · ChatGPT · Go/no-go for AI-drafted support replies
  3. Addresses the actual decision95% pass
    The output gives a clear, unambiguous 'no-go' call early, and states what would change it (fixes and retest).
    GPT-6 Astra · ChatGPT · Go/no-go for AI-drafted support replies

Where it slips

  1. Limits the downside of being wrong32% pass
    No specific post-launch monitoring signal or threshold is named; only generic 'monitoring' and a kill switch are mentioned.
    GPT-6 Astra · ChatGPT · Go/no-go for AI-drafted support replies
  2. Catches the duplicate rows50% pass
    Finds and removes the duplicate rows but never states whether they change the results.
    GPT-6.1 Sol · API · Go/no-go for AI substitutions, from the rollout data
  3. Uses the supplied evidence correctly54% pass
    The output states that pilot agents were hands-on, engaged and motivated, which is not supported by the supplied context (only that 7/8 wanted to keep it).
    Opus 5.5 · Claude · Go/no-go for AI-drafted support replies

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

Tomorrow's launch meeting decides whether AI-drafted support replies go live to all 42 support agents on Monday. You're the PM. Using the material below, write your launch recommendation for the meeting: go, no-go, or go with conditions, and why. Keep it under 400 words.

What the model was given6 items: Scenario, Launch criteria agreed in the PRD, Pilot (8 agents, 3 weeks, 2,940 tickets), Eval set (500 tickets), Open bugs, Support operations
ScenarioLedgerly is accounting software for small businesses. The feature drafts a first reply to each support ticket; an agent reviews and edits every draft before it is sent.
Launch criteria agreed in the PRD1. At least 60% of drafts accepted as-is or with minor edits. 2. Wrong or harmful drafts (wrong account data, wrong policy, invented features) at most 1% on the 500-ticket eval set. 3. No draft may commit to a refund. 4. Median time to first response down at least 30% in the pilot.
Pilot (8 agents, 3 weeks, 2,940 tickets)Accepted as-is or with minor edits: 64%. Median time to first response: 7.1 hours before, 4.2 hours during (−41%). Seven of the eight agents want to keep it.
Eval set (500 tickets)Wrong or harmful drafts: 9 of 500 (1.8%). Six of the nine are billing tickets quoting the old prices, which changed on 1 September; the model's reference data stops in August. Two drafts said “we'll refund this charge”; the pilot agents caught and removed both.
Open bugsReplies to Welsh-language tickets come back in English (about 0.3% of tickets). Tables in drafts are badly formatted.
Support operationsTraining for all 42 agents is booked for Friday. Billing is one of four queues and handles 38% of tickets.
What a strong answer doesThe answer key the graders mark against

Not a full go: criterion 2 fails (1.8% against 1%) and criterion 3 is breached even though agents caught it. Strong answer: go with conditions, launching to the non-billing queues while billing waits for updated pricing data and a hard block on refund language, then re-running the eval to confirm it meets the bar; names exactly what would turn it into a full go.

Critical failures (cap the score at 40)
  • Recommends a full launch while the error rate misses the agreed 1% bar
  • Treats the refund commitments as acceptable
Case

Staff-level · v1.9 · synthetic · AI product, launch

The outputs

7 setups answered this brief. Pick up to three to read side by side.

GPT-6 AstraChatGPT93Usable with a quick editGPT-6 LunaAPI93Usable with a quick editGPT-6.1 SolAPI91Usable with a quick editSonnet 5.5API91Usable with a quick editOpus 5.5Claude77Usable with a quick editGemini 3.8 FlashAPI55Needs more workGemini 3.5 Flash-LiteGemini39Needs more work
GPT-6 AstraChatGPT

Usable with a quick edit9 of 10 checks passed

The reasoning is sound, evidence is correctly used, and the recommendation is clear. A PM could add a monitoring threshold in a quick edit without redoing any analysis.

Output

Recommendation: no-go for Monday’s rollout to all 42 agents. Reassess once the failed safety gates have been fixed and retested.

The pilot demonstrates value: 64% of drafts were accepted as-is or with minor edits, exceeding the 60% target. Median first-response time fell 41%, exceeding the 30% target, although the before-and-after comparison does not establish causality. Seven of eight agents want to keep it.

But we failed two explicit launch criteria:

  • Wrong or harmful drafts: 1.8% against a 1% ceiling. Nine failures is above the maximum five permitted on the 500-ticket set.
  • No refund commitments: failed. Two drafts promised refunds. Agents catching them prevented those messages reaching customers; it does not satisfy a criterion about what drafts may say. Human review was already part of the agreed design.

Six errors used outdated billing prices. That gives us a concrete fix, but billing represents 38% of tickets, so this is a substantial exposure. Updating the reference data alone would not establish that the remaining errors or refund promises are resolved.

Required before reconsidering launch:

  1. Engineering and the billing policy owner: update and verify current prices and policies, with a named owner and process for keeping reference data current. Investigate the other three harmful drafts.
  2. Engineering: add and test a control that prevents refund commitments from appearing in agent-facing drafts, including paraphrases.
  3. PM and evaluation owner: rerun the full 500-ticket evaluation after the fixes, meeting the original thresholds, and add targeted pricing and refund tests. Report those additional results separately.

Excluding billing is a possible narrower release, not an automatic pass: we lack queue-level results and do not know whether refund failures are confined to billing. It would require explicit scope approval and validation of both routing and the remaining queues.

Welsh-language tickets should bypass drafting until language handling is fixed. Disable tables or use plain text pending the formatting fix.

Keep Friday’s training as preparation, with launch clearly marked as pending. The booking should not determine readiness. After the gates pass, expand in stages with monitoring and a named owner able to disable drafting.

Grades and run details

Decision model 95 · LLM judge 10 of 11 checks
Decision model checks
  • passUses the supplied evidence correctly69%
  • passAddresses the actual decision94%
  • passRespects explicit constraints44%
  • passIdentifies material uncertainty97%
  • passAvoids unsupported claims74%
  • passProduces the required deliverable86%
  • passChecks every agreed criterion49%
  • passMakes a clear call93%
  • partialLimits the downside of being wrong54%
  • passGets the base of every number right83%
Run
Run
#1
Time to output
34 s
Submitted
25 Sept 2026
Opus 5.5Claude

Usable with a quick edit7 of 10 checks passed

The unsupported claim about pilot engagement is minor; remove it and the recommendation remains sound and usable with a quick edit.

Output

Launch recommendation: AI-drafted support replies

Recommendation: no-go for all 42 agents this Monday. Go the following Monday (5 Oct) if three conditions are met.

Scorecard against PRD criteria

CriterionTargetResultStatus
Accepted as-is / minor edits≥60%64%✅ Pass
Wrong or harmful drafts≤1%1.8% (9/500)❌ Fail
No refund commitments02 drafts❌ Fail
Median first response−30%−41% (7.1h → 4.2h)✅ Pass

Two of four criteria fail. The value is proven, but the safety bar isn't met yet.

Why not Monday

  • Refunds are a hard line, and we crossed it. Two drafts promised refunds. Agents caught both, but the criterion says no draft may do this. Human review is the backstop, not the control. The pilot used 8 motivated agents. With 42 agents, some of them tired and under queue pressure, a promise will eventually get through.2
  • Billing errors land on our biggest queue. Six of the nine failures quote pre-1 September prices. Billing is 38% of volume, so this is a systematic error, not an edge case.
  • Training comes after launch. Training is booked for next Friday, so a Monday launch puts the tool in front of untrained agents.1 The pilot agents were hands-on and engaged, and we can't assume the rest will review as carefully.

Conditions for a 5 October go

  1. Refund guardrail. Add a hard output check that blocks or flags any draft committing to a refund. Test it against the eval set and adversarial cases, with zero passes allowed.
  2. Refresh pricing data and re-run the full 500-ticket eval. Launch only if wrong or harmful drafts are at most 1%. Removing the six pricing errors would leave 0.6% on paper. We should prove that with a re-run, not assume it.
  3. Train before rollout. Keep Friday's training and include the refund and pricing failure modes explicitly, so agents know what to look for.

Not blocking

  • Welsh replies coming back in English (0.3%). Turn drafting off for Welsh-language tickets until this is fixed. It costs little and avoids a visible failure.
  • Table formatting. This is cosmetic, and agents edit every draft anyway. Fix it in the normal backlog.

Meanwhile

Keep the 8 pilot agents on the tool so we don't lose momentum.3 In the first two weeks after launch, spot-check a sample of sent billing replies.

The ask

Agree to the three conditions. Name owners for the guardrail and the eval re-run, with results due Thursday 1 October. If both pass, we confirm go on 2 October after training.

What a PM had to fix

  1. 1Invented evidenceVerify or remove the claimTargeted repair

    Source: Support operationsTraining for all 42 agents is booked for Friday.

    What we’d changeThe brief says training is on Friday, not 'next Friday'. Don't turn an ambiguous date into a blocker, or build an October schedule around it.

  2. 2Hypothesis stated as factReframe it as a hypothesisQuick edit

    What we’d changeSay the risk grows with 42 agents rather than stating it as certain, and don't assume the pilot agents were unusually motivated.

  3. 3Constraint missedRestore the constraintQuick edit

    What we’d changeMomentum isn't a reason to keep a tool that fails two criteria in use. Add the refund check first, or pause it.

Claims the judge couldn’t find in the brief

  • Pilot agents were hands-on, engaged and motivated

Grades and run details

Decision model 80 · LLM judge 8 of 11 checks
Decision model checks
  • failUses the supplied evidence correctly21%
  • passAddresses the actual decision96%
  • passRespects explicit constraints32%
  • passIdentifies material uncertainty98%
  • partialAvoids unsupported claims41%
  • passProduces the required deliverable65%
  • passChecks every agreed criterion99%
  • passMakes a clear call100%
  • partialLimits the downside of being wrong81%
  • passGets the base of every number right81%
Run
Run
#1
Time to output
23 s
Submitted
25 Sept 2026
Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit3 of 10 checks passed

A PM would need to add a plan to re-run the eval set after fixes and explicit post-launch monitoring triggers; the unsupported 'training cutoff' claim also needs correction.

Output

Launch Recommendation: Go with Conditions

As the Product Manager, my recommendation for Monday’s rollout to all 42 agents is Go with Conditions.

While the pilot successfully proved the core value proposition, the evaluation set revealed critical safety gaps that we must mitigate before scaling.3

---

Why We Are Ready (The "Go" Factors)

  • Efficiency Gains Exceeded Target: Median time to first response dropped by 41% (from 7.1 to 4.2 hours), surpassing our 30% goal.
  • High Team Adoption: 7 out of 8 pilot agents want to keep using the feature.
  • Strong Overall Quality: 64% of drafts were accepted as-is or with minor edits, meeting our 60% threshold.

---

Critical Conditions for Monday's Launch

To protect our customers and brand, the following four conditions must be met prior to Monday:

1. Fix the Billing Knowledge Gap (Mandatory): * The Issue: 6 out of the 9 harmful drafts on the eval set quoted old prices because the model's training data cutoff is August4, and prices changed September 1st. * Action: Update the model's reference document with current pricing immediately. 2. Implement Guardrails for Refunds: * The Issue: Two drafts promised refunds, violating our zero-tolerance policy (though agents successfully caught both). * Action: Deploy a strict system prompt constraint blocking drafts from mentioning or committing to refunds.2 3. Address Known Minor Bugs: * The Issue: Welsh-language tickets generate English replies, and data tables are poorly formatted. * Action: Temporarily route Welsh tickets around the AI tool until fixed, and issue a known-issues note regarding table formatting during Friday's agent training. 4. Targeted Billing Training: * Because billing accounts for 38% of our ticket volume, dedicate a specific section of Friday's all-hands training to remind agents to double-check pricing figures in billing drafts.

---

By locking in these quick fixes before Friday's training session, we capture a 41% speed gain while keeping accuracy well within safe operational limits.1

What a PM had to fix

  1. 1Hypothesis stated as factReframe it as a hypothesisSubstantial rework

    Source: Eval set (500 tickets)Wrong or harmful drafts: 9 of 500 (1.8%).

    What we’d changeRequire a re-run of the 500-ticket eval after the fixes, and launch only if it passes. Nothing yet shows accuracy is within the limit.

  2. 2Constraint missedRestore the constraintTargeted repair

    Source: Launch criteria agreed in the PRDNo draft may commit to a refund.

    What we’d changeA prompt instruction isn't a guarantee. Test the control, and add a check that stops refund commitments reaching agents' drafts.

  3. 3Constraint missedRestore the constraintTargeted repair

    Source: Launch criteria agreed in the PRDWrong or harmful drafts (wrong account data, wrong policy, invented features) at most 1% on the 500-ticket eval set. 3. No draft may commit to a refund.

    What we’d changeCheck each agreed criterion and say plainly that two fail: wrong or harmful drafts at 1.8% against 1%, and two refund commitments against none.

  4. 4Invented evidenceVerify or remove the claimQuick edit

    Source: Eval set (500 tickets)the model's reference data stops in August

    What we’d changeThe brief says the model's reference data stops in August, not its training data. That's what makes a pricing refresh a quick fix.

Claims the judge couldn’t find in the brief

  • The model's training data cutoff is August

Grades and run details

Decision model 55 · LLM judge 3 of 11 checks
Decision model checks
  • passUses the supplied evidence correctly55%
  • partialAddresses the actual decision45%
  • partialRespects explicit constraints13%
  • partialIdentifies material uncertainty73%
  • partialAvoids unsupported claims27%
  • partialProduces the required deliverable46%
  • partialChecks every agreed criterion69%
  • partialMakes a clear call76%
  • failLimits the downside of being wrong96%
  • passGets the base of every number right76%
Run
Run
#1
Time to output
6 s
Submitted
25 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 6

Uses the supplied evidence correctlyRightWrongMixed
GPT-6 Astra · ChatGPT

All factual statements about the current situation are directly from the supplied brief or context, or follow from arithmetic; no invented facts.

Opus 5.5 · Claude

The output states that pilot agents were hands-on, engaged and motivated, which is not supported by the supplied context (only that 7/8 wanted to keep it).

Gemini 3.5 Flash-Lite · Gemini

The claim about a 'training data cutoff' is not supported; the context only mentions reference data stopping in August, so an unsupported fact about the current system was introduced.

Addresses the actual decisionRightRightWrong
GPT-6 Astra · ChatGPT

The output gives a clear, unambiguous 'no-go' call early, and states what would change it (fixes and retest).

Opus 5.5 · Claude

Commits to no-go for Monday, with a conditional go the following Monday, and states what would change the call.

Gemini 3.5 Flash-Lite · Gemini

The output does not state what result or condition would change the 'Go with Conditions' call (e.g., failing to meet the conditions would make it a no-go).

Identifies material uncertaintyRightRightWrong
GPT-6 Astra · ChatGPT

It identifies the lack of queue-level results and the unknown scope of refund failures, and explains how those would be resolved.

Opus 5.5 · Claude

Names the remaining work (guardrail, pricing refresh, re-run) and says the call depends on their results.

Gemini 3.5 Flash-Lite · Gemini

It does not name specific unknowns that could change the decision or say how they would be resolved, such as whether the pricing fix alone will bring error rate below 1%.

Avoids unsupported claimsRightWrongWrong
GPT-6 Astra · ChatGPT

The note about causality is framed as a methodological observation, not as a factual claim; no unsupported causes or forecasts are presented as fact.

Opus 5.5 · Claude

Presents pilot agents' hands-on engagement and motivation as established fact when the evidence only shows retention preference, not engagement level.

Gemini 3.5 Flash-Lite · Gemini

The output presents the 'training data cutoff' as a fact when the source only says reference data stops in August, turning an interpretation into an unsupported claim.

Checks every agreed criterionRightRightWrong
GPT-6 Astra · ChatGPT

All four launch criteria are explicitly assessed: acceptance rate, error rate, refund commitment, and time reduction.

Opus 5.5 · Claude

Evaluates all four PRD criteria with a status table and references to the evidence.

Gemini 3.5 Flash-Lite · Gemini

It does not explicitly compare the 1.8% harmful rate against the 1% bar, leaving the second launch criterion unchecked.

Makes a clear callRightRightWrong
GPT-6 Astra · ChatGPT

Clear 'no-go' call up front, with a statement that reassessment follows fixes and retesting.

Opus 5.5 · Claude

Clear no-go for Monday, go on 5 October if three named conditions are met.

Gemini 3.5 Flash-Lite · Gemini

The call is clear but does not state what would change it, e.g., that failing to meet the conditions would lead to a no-go.

All got wrong 1

Limits the downside of being wrongWrongWrongWrong
GPT-6 Astra · ChatGPT

No specific post-launch monitoring signal or threshold is named; only generic 'monitoring' and a kill switch are mentioned.

Opus 5.5 · Claude

Post-launch spot-checks are mentioned but no threshold or rollback action is specified, and irreversible risks are not flagged.

Gemini 3.5 Flash-Lite · Gemini

No post-launch signal, threshold to pause or roll back, or irreversible actions are identified.

All got right 3

Respects explicit constraintsRightRightRight
GPT-6 Astra · ChatGPT

The output is a launch recommendation, addresses the meeting, and stays within the 400-word limit (345 words).

Opus 5.5 · Claude

Respects the length limit, addresses the meeting, and proposes conditions that would enforce the launch criteria.

Gemini 3.5 Flash-Lite · Gemini

The output stays under 400 words and respects the requested form of a launch recommendation for the meeting.

Produces the required deliverableRightRightRight
GPT-6 Astra · ChatGPT

The recommendation is in the requested form, for the meeting, within the length, and includes necessary steps a PM could act on.

Opus 5.5 · Claude

Delivers a complete launch recommendation under 400 words that the meeting could act on.

Gemini 3.5 Flash-Lite · Gemini

The recommendation is under 400 words, addressed to the meeting, and usable with light edits, though it lacks a retest plan.

Gets the base of every number rightRightRightRight
GPT-6 Astra · ChatGPT

All derived figures (41% time reduction, 1.8% error rate, 9/500, 38% of tickets) are computed correctly and bases are clear.

Opus 5.5 · Claude

All percentages and differences are computed correctly from the supplied numbers.

Gemini 3.5 Flash-Lite · Gemini

The 41% time reduction uses the correct base (7.1 hours to 4.2 hours), and other percentages match the supplied data.

Results

Every setup we’ve tested on this task, across all cases and repeats, graded on the current checklist. Calibrated: the graders match our PM on 87% of checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6.1 SolAPI87.388.32None
2GPT-6 AstraChatGPT86.088.32None
3Sonnet 5.5API85.488.32None
4Opus 5.5Claude80.486.42None
5GPT-6 LunaAPI82.177.62None
6Gemini 3.8 FlashAPI71.931.521 capped
7Gemini 3.5 Flash-LiteGemini50.642.22None

The PM job

Deciding whether a feature ships on the planned date, and on what conditions.

Why it matters

Launch meetings reward optimism. A good call checks each agreed criterion, weighs a real risk against a real gain, and says exactly what would change the answer. A weak one rubber-stamps the launch or blocks it on noise.

What good looks like

  • Checks each agreed criterion against the evidence
  • Makes one clear call, with conditions if needed
  • Separates risks that block launch from risks that can be managed
  • Says what would change the call
  • Says how to limit the damage if the call is wrong

Deliberately not measured

  • Rollout engineering detail
  • Project-plan formatting
Capability tested

Deciding against agreed launch criteria

The failure we’re looking for

Rubber-stamps a launch that misses an agreed bar, or blocks it on noise

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

A launch memo from supplied evidence · Staff level: a regional call from an attached data workbook