Humanity’s Last PRD
Latest drop: Gemini 3.8 Flash · 1 Oct
Tasks / Operate

Make the launch call

Can the model make a clear go / no-go call from mixed launch evidence, checked against the criteria agreed up front?

Measures the modelTask v1.2 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 14 graded outputs by 7 models. 64% were usable with at most a quick edit.

Reliably right

  1. Finds the Scotland breach100% pass
    Clearly identifies Scotland's refund breach (66% above control, 143% after supplier change) and acceptance dip below 80%, correctly holds Scotland.
    Opus 5.5 · Claude · Go/no-go for AI substitutions, from the rollout data
  2. Produces the required deliverable98% pass
    The recommendation is in the requested form, for the meeting, within the length, and includes necessary steps a PM could act on.
    GPT-6 Astra · ChatGPT · Go/no-go for AI-drafted support replies
  3. Addresses the actual decision95% pass
    The output gives a clear, unambiguous 'no-go' call early, and states what would change it (fixes and retest).
    GPT-6 Astra · ChatGPT · Go/no-go for AI-drafted support replies

Where it slips

  1. Limits the downside of being wrong32% pass
    No specific post-launch monitoring signal or threshold is named; only generic 'monitoring' and a kill switch are mentioned.
    GPT-6 Astra · ChatGPT · Go/no-go for AI-drafted support replies
  2. Catches the duplicate rows50% pass
    Finds and removes the duplicate rows but never states whether they change the results.
    GPT-6.1 Sol · API · Go/no-go for AI substitutions, from the rollout data
  3. Uses the supplied evidence correctly54% pass
    The output states that pilot agents were hands-on, engaged and motivated, which is not supported by the supplied context (only that 7/8 wanted to keep it).
    Opus 5.5 · Claude · Go/no-go for AI-drafted support replies

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

Tomorrow's launch meeting decides whether AI-drafted support replies go live to all 42 support agents on Monday. You're the PM. Using the material below, write your launch recommendation for the meeting: go, no-go, or go with conditions, and why. Keep it under 400 words.

What the model was given6 items: Scenario, Launch criteria agreed in the PRD, Pilot (8 agents, 3 weeks, 2,940 tickets), Eval set (500 tickets), Open bugs, Support operations
ScenarioLedgerly is accounting software for small businesses. The feature drafts a first reply to each support ticket; an agent reviews and edits every draft before it is sent.
Launch criteria agreed in the PRD1. At least 60% of drafts accepted as-is or with minor edits. 2. Wrong or harmful drafts (wrong account data, wrong policy, invented features) at most 1% on the 500-ticket eval set. 3. No draft may commit to a refund. 4. Median time to first response down at least 30% in the pilot.
Pilot (8 agents, 3 weeks, 2,940 tickets)Accepted as-is or with minor edits: 64%. Median time to first response: 7.1 hours before, 4.2 hours during (−41%). Seven of the eight agents want to keep it.
Eval set (500 tickets)Wrong or harmful drafts: 9 of 500 (1.8%). Six of the nine are billing tickets quoting the old prices, which changed on 1 September; the model's reference data stops in August. Two drafts said “we'll refund this charge”; the pilot agents caught and removed both.
Open bugsReplies to Welsh-language tickets come back in English (about 0.3% of tickets). Tables in drafts are badly formatted.
Support operationsTraining for all 42 agents is booked for Friday. Billing is one of four queues and handles 38% of tickets.
What a strong answer doesThe answer key the graders mark against

Not a full go: criterion 2 fails (1.8% against 1%) and criterion 3 is breached even though agents caught it. Strong answer: go with conditions, launching to the non-billing queues while billing waits for updated pricing data and a hard block on refund language, then re-running the eval to confirm it meets the bar; names exactly what would turn it into a full go.

Critical failures (cap the score at 40)
  • Recommends a full launch while the error rate misses the agreed 1% bar
  • Treats the refund commitments as acceptable
Case

Staff-level · v1.9 · synthetic · AI product, launch

The outputs

7 setups answered this brief. Pick up to three to read side by side.

GPT-6 AstraChatGPT93Usable with a quick editGPT-6 LunaAPI93Usable with a quick editGPT-6.1 SolAPI91Usable with a quick editSonnet 5.5API91Usable with a quick editOpus 5.5Claude77Usable with a quick editGemini 3.8 FlashAPI55Needs more workGemini 3.5 Flash-LiteGemini39Needs more work
Gemini 3.8 FlashAPI

Needs more than a quick edit3 of 10 checks passed

A PM would need to re-check the non-billing error rate, refund-draft queue attribution, and monitoring/rollback plan before using it. The core recommendation is useful but not evidence-safe as written.

Output

Recommendation: Go with Conditions

While the pilot demonstrated strong operational impact—reducing median first-response time by 41% (beating our 30% target) with a 64% acceptance rate—we failed two mandatory launch criteria:

  1. Accuracy (Criterion 2): Harmful/wrong drafts hit 1.8% (9/500) versus the $\le$1.0% threshold.
  2. Refund commitments (Criterion 3): Two drafts explicitly promised refunds, violating our zero-tolerance policy.

However, these failures are isolated and addressable. Six of the nine errors were caused by August reference data lacking the September 1 price changes, and both refund commitments originated in billing workflows.

Rather than cancelling Friday’s 42-agent training or delaying the productivity benefits across the entire organization, we should launch on Monday under the following three conditions:

Launch Conditions

1. Disable AI drafting on the Billing queue for Monday: Billing represents 38% of volume and contains almost all our critical failure points (pricing errors and refund risks). We will launch exclusively to the remaining three queues (62% of ticket volume), where accuracy meets our threshold. 2. Implement deterministic refund guardrails by Friday: Deploy a hard system-prompt restriction and a post-generation regex/keyword filter that blocks or drops drafts containing refund commitments before reaching the agent. 3. Update agent training on Friday: Keep Friday’s training on schedule. Explicitly train agents on: - The pilot's phased rollout (Billing queue excluded initially). - Known minor bugs (Welsh-language tickets defaulting to English; manual table reformatting required). - Strict agent accountability as the final human reviewer for every sent message.

Path to Full Billing Rollout

Engineering will ingest the post-September 1 pricing documentation and run a targeted 200-ticket eval on billing drafts next week. Once billing tickets achieve $<1\%$ error and 0 refund promises, we will enable the feature for the billing queue.

This approach protects Ledgerly’s financial liability and customer trust while immediately capturing significant speed improvements for over 60% of our support volume.

Claims the judge couldn’t find in the brief

  • Both refund commitments originated in billing workflows.
  • Billing contains almost all critical failure points, including pricing errors and refund risks.
  • The remaining three queues meet the accuracy threshold.

Grades and run details

Decision model 90 · LLM judge 3 of 11 checks
Decision model checks
  • passUses the supplied evidence correctly45%
  • passAddresses the actual decision100%
  • passRespects explicit constraints54%
  • passIdentifies material uncertainty96%
  • partialAvoids unsupported claims27%
  • passProduces the required deliverable96%
  • passChecks every agreed criterion24%
  • passMakes a clear call100%
  • partialLimits the downside of being wrong38%
  • passGets the base of every number right86%
Run
Run
#1
API response time
13 s
Submitted
1 Oct 2026
Opus 5.5Claude

Usable with a quick edit7 of 10 checks passed

The unsupported claim about pilot engagement is minor; remove it and the recommendation remains sound and usable with a quick edit.

Output

Launch recommendation: AI-drafted support replies

Recommendation: no-go for all 42 agents this Monday. Go the following Monday (5 Oct) if three conditions are met.

Scorecard against PRD criteria

CriterionTargetResultStatus
Accepted as-is / minor edits≥60%64%✅ Pass
Wrong or harmful drafts≤1%1.8% (9/500)❌ Fail
No refund commitments02 drafts❌ Fail
Median first response−30%−41% (7.1h → 4.2h)✅ Pass

Two of four criteria fail. The value is proven, but the safety bar isn't met yet.

Why not Monday

  • Refunds are a hard line, and we crossed it. Two drafts promised refunds. Agents caught both, but the criterion says no draft may do this. Human review is the backstop, not the control. The pilot used 8 motivated agents. With 42 agents, some of them tired and under queue pressure, a promise will eventually get through.2
  • Billing errors land on our biggest queue. Six of the nine failures quote pre-1 September prices. Billing is 38% of volume, so this is a systematic error, not an edge case.
  • Training comes after launch. Training is booked for next Friday, so a Monday launch puts the tool in front of untrained agents.1 The pilot agents were hands-on and engaged, and we can't assume the rest will review as carefully.

Conditions for a 5 October go

  1. Refund guardrail. Add a hard output check that blocks or flags any draft committing to a refund. Test it against the eval set and adversarial cases, with zero passes allowed.
  2. Refresh pricing data and re-run the full 500-ticket eval. Launch only if wrong or harmful drafts are at most 1%. Removing the six pricing errors would leave 0.6% on paper. We should prove that with a re-run, not assume it.
  3. Train before rollout. Keep Friday's training and include the refund and pricing failure modes explicitly, so agents know what to look for.

Not blocking

  • Welsh replies coming back in English (0.3%). Turn drafting off for Welsh-language tickets until this is fixed. It costs little and avoids a visible failure.
  • Table formatting. This is cosmetic, and agents edit every draft anyway. Fix it in the normal backlog.

Meanwhile

Keep the 8 pilot agents on the tool so we don't lose momentum.3 In the first two weeks after launch, spot-check a sample of sent billing replies.

The ask

Agree to the three conditions. Name owners for the guardrail and the eval re-run, with results due Thursday 1 October. If both pass, we confirm go on 2 October after training.

What a PM had to fix

  1. 1Invented evidenceVerify or remove the claimTargeted repair

    Source: Support operationsTraining for all 42 agents is booked for Friday.

    What we’d changeThe brief says training is on Friday, not 'next Friday'. Don't turn an ambiguous date into a blocker, or build an October schedule around it.

  2. 2Hypothesis stated as factReframe it as a hypothesisQuick edit

    What we’d changeSay the risk grows with 42 agents rather than stating it as certain, and don't assume the pilot agents were unusually motivated.

  3. 3Constraint missedRestore the constraintQuick edit

    What we’d changeMomentum isn't a reason to keep a tool that fails two criteria in use. Add the refund check first, or pause it.

Claims the judge couldn’t find in the brief

  • Pilot agents were hands-on, engaged and motivated

Grades and run details

Decision model 80 · LLM judge 8 of 11 checks
Decision model checks
  • failUses the supplied evidence correctly21%
  • passAddresses the actual decision96%
  • passRespects explicit constraints32%
  • passIdentifies material uncertainty98%
  • partialAvoids unsupported claims41%
  • passProduces the required deliverable65%
  • passChecks every agreed criterion99%
  • passMakes a clear call100%
  • partialLimits the downside of being wrong81%
  • passGets the base of every number right81%
Run
Run
#1
Time to output
23 s
Submitted
25 Sept 2026
Sonnet 5.5API

Usable with a quick edit9 of 10 checks passed

The recommendation is complete, evidence-based, and actionable; the only meaningful gap is a missing post-launch pause threshold, which is a quick edit.

Output

Recommendation: Go with conditions

Not a clean go. Two of the four agreed launch criteria are currently failed, and I don't think we should waive criteria we set ourselves.

CriterionResultStatus
≥60% accepted64%Met
≤1% wrong/harmful1.8% (9/500)Failed
No refund commitments2 drafts promised refundsFailed
≥30% faster first response−41% (7.1h → 4.2h)Met

The value is real: faster responses, good acceptance, and 7 of 8 pilot agents want to keep it. Both failures also have identifiable causes.

  • Wrong drafts: Six of the nine errors come from stale pricing, because the reference data stops in August. Fixing that should bring us to about 3/500 (0.6%), but that is a projection until we re-run the eval.
  • Refunds: Agents caught both drafts, but the criterion says no draft, and with 42 agents we shouldn't rely on catching them by habit.

Conditions for Monday

  1. Refresh the reference data with post-1 September pricing and re-run the full 500-ticket eval. It must come in at 1% or below (5 or fewer wrong).
  2. Add a refund guardrail that blocks or flags commitment language such as "we'll refund." It must show zero refund commitments on the re-run plus a targeted set of refund-request tickets.
  3. Kill switch and flag button. Ops can disable the feature instantly, and agents can flag bad drafts in one click.

Fallback

  • If condition 2 fails, no-go for Monday.
  • If only condition 1 fails, launch on the three non-billing queues (62% of tickets) and keep billing off until it passes.
  • Friday training goes ahead for all 42 agents either way. It should cover the refund rule and checking prices before sending.

Known issues, not blockers

  • Welsh tickets (~0.3%) get English replies. Agents will write these manually, and we'll suppress drafts for them if detection is feasible.
  • Table formatting is cosmetic, since agents edit every draft. We'll fix it after launch.

After launch

  • Sample wrong-draft rates weekly for the first month.
  • Track how often agents edit or discard drafts, as a check on whether review discipline slips.
  • Ask the one pilot agent who didn't want to keep it why.

I'd like the eval re-run results by Thursday so we can make the final call before Friday's training.

Grades and run details

Decision model 90 · LLM judge 10 of 11 checks
Decision model checks
  • passUses the supplied evidence correctly44%
  • passAddresses the actual decision98%
  • passRespects explicit constraints37%
  • passIdentifies material uncertainty100%
  • partialAvoids unsupported claims25%
  • passProduces the required deliverable84%
  • passChecks every agreed criterion100%
  • passMakes a clear call100%
  • partialLimits the downside of being wrong85%
  • passGets the base of every number right79%
Run
Run
#1
API response time
17 s
Submitted
29 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 6

Uses the supplied evidence correctlyMixedWrongRight
Gemini 3.8 Flash · API

It invents or assumes that both refund drafts were billing-related, that non-billing queues meet the accuracy threshold, and that billing contains almost all critical failure points.

Opus 5.5 · Claude

The output states that pilot agents were hands-on, engaged and motivated, which is not supported by the supplied context (only that 7/8 wanted to keep it).

Sonnet 5.5 · API

All current-state facts used are from the supplied context; no invented systems or numbers.

Respects explicit constraintsMixedRightRight
Gemini 3.8 Flash · API

The proposed non-billing launch is not shown to enforce the 1% wrong/harmful criterion, and the refund guardrail proposal is not evidenced as sufficient.

Opus 5.5 · Claude

Respects the length limit, addresses the meeting, and proposes conditions that would enforce the launch criteria.

Sonnet 5.5 · API

Under 400 words, and the proposed eval re-run, 1%/zero refund gates, and non-billing fallback would enforce the agreed criteria.

Identifies material uncertaintyMixedRightRight
Gemini 3.8 Flash · API

It does not name the key unknowns that could change the call, such as the non-billing error rate, whether refund drafts were billing-related, or whether guardrails actually prevent refund commitments.

Opus 5.5 · Claude

Names the remaining work (guardrail, pricing refresh, re-run) and says the call depends on their results.

Sonnet 5.5 · API

It identifies the re-run pricing/refund results as the open unknowns and maps them to specific decision changes.

Avoids unsupported claimsWrongWrongRight
Gemini 3.8 Flash · API

It presents unsupported causal and coverage claims as fact, including that refund commitments originated in billing workflows and that non-billing accuracy meets the threshold.

Opus 5.5 · Claude

Presents pilot agents' hands-on engagement and motivation as established fact when the evidence only shows retention preference, not engagement level.

Sonnet 5.5 · API

The 0.6% estimate is clearly labelled a projection, and causes match the supplied context.

Checks every agreed criterionMixedRightRight
Gemini 3.8 Flash · API

It checks criteria 2, 3, and 4, but does not explicitly test criterion 1 against the 60% acceptance bar.

Opus 5.5 · Claude

Evaluates all four PRD criteria with a status table and references to the evidence.

Sonnet 5.5 · API

The table explicitly checks all four agreed criteria.

Gets the base of every number rightMixedRightRight
Gemini 3.8 Flash · API

It uses the 62% volume base correctly but treats non-billing accuracy as meeting the threshold without a verified denominator or error count.

Opus 5.5 · Claude

All percentages and differences are computed correctly from the supplied numbers.

Sonnet 5.5 · API

All derived rates are correct: 9/500=1.8%, 9−6=3/500=0.6%, 38% billing implies 62% non-billing, and 2.9/7.1=−41%.

All got wrong 1

Limits the downside of being wrongWrongWrongWrong
Gemini 3.8 Flash · API

It does not specify post-launch monitoring, a pause or rollback threshold, or irreversible customer-communication risks.

Opus 5.5 · Claude

Post-launch spot-checks are mentioned but no threshold or rollback action is specified, and irreversible risks are not flagged.

Sonnet 5.5 · API

It describes post-launch monitoring but does not name a threshold or trigger after launch to pause/roll back, nor flag anything irreversible.

All got right 3

Addresses the actual decisionRightRightRight
Gemini 3.8 Flash · API

It clearly recommends go with conditions for Monday and states the billing re-evaluation result that would enable full rollout.

Opus 5.5 · Claude

Commits to no-go for Monday, with a conditional go the following Monday, and states what would change the call.

Sonnet 5.5 · API

It commits to 'Go with conditions' and says what would change the call: no-go if condition 2 fails, non-billing launch if only condition 1 fails.

Produces the required deliverableRightRightRight
Gemini 3.8 Flash · API

It is a concise launch recommendation in the requested form and appears usable for the meeting with light edits.

Opus 5.5 · Claude

Delivers a complete launch recommendation under 400 words that the meeting could act on.

Sonnet 5.5 · API

A complete launch recommendation with criteria table, conditions, fallback and post-launch monitoring, usable as-is.

Makes a clear callRightRightRight
Gemini 3.8 Flash · API

It makes a clear go-with-conditions call and states the billing rollout condition.

Opus 5.5 · Claude

Clear no-go for Monday, go on 5 October if three named conditions are met.

Sonnet 5.5 · API

Clear 'Go with conditions' call with named fallbacks.

Results

Every setup we’ve tested on this task, across all cases and repeats, graded on the current checklist. Calibrated: the graders match our PM on 87% of checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6.1 SolAPI87.388.32None
2GPT-6 AstraChatGPT86.088.32None
3Sonnet 5.5API85.488.32None
4Opus 5.5Claude80.486.42None
5GPT-6 LunaAPI82.177.62None
6Gemini 3.8 FlashAPI71.931.521 capped
7Gemini 3.5 Flash-LiteGemini50.642.22None

The PM job

Deciding whether a feature ships on the planned date, and on what conditions.

Why it matters

Launch meetings reward optimism. A good call checks each agreed criterion, weighs a real risk against a real gain, and says exactly what would change the answer. A weak one rubber-stamps the launch or blocks it on noise.

What good looks like

  • Checks each agreed criterion against the evidence
  • Makes one clear call, with conditions if needed
  • Separates risks that block launch from risks that can be managed
  • Says what would change the call
  • Says how to limit the damage if the call is wrong

Deliberately not measured

  • Rollout engineering detail
  • Project-plan formatting
Capability tested

Deciding against agreed launch criteria

The failure we’re looking for

Rubber-stamps a launch that misses an agreed bar, or blocks it on noise

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

A launch memo from supplied evidence · Staff level: a regional call from an attached data workbook