Humanity’s Last PRD
Latest drop: Gemini 3.8 Flash · 1 Oct
Tasks / Operate

Write a stakeholder update

Can the model turn messy project status into an honest update that leads with what the reader needs to know or decide?

Measures the modelTask v1.2 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 14 graded outputs by 7 models. 43% were usable with at most a quick edit.

Reliably right

  1. Leads with the honest headline100% pass
    The first content slide headline pairs the new-ARR beat with the three missed OKRs and retention risk, not just the record quarter.
    GPT-6 Astra · ChatGPT · A quarterly business review deck, in brand
  2. A real 7-slide deck100% pass
  3. Answers the question asked96% pass
    The first line directly addresses whether the new navigation caused the drop.
    GPT-6.1 Sol · API · The CEO asks why a metric dropped

Where it slips

  1. Owns the problem23% pass
    It proposes future analysis and fixes but does not state any actions the team has already taken or offer trade-offs.
    GPT-6.1 Sol · API · The CEO asks why a metric dropped
  2. Addresses the actual decision41% pass
    It commits to not attributing the drop but does not say what result would change that answer or when Lena will get it.
    GPT-6 Luna · API · The CEO asks why a metric dropped
  3. Identifies material uncertainty55% pass
    The output does not name specific unknowns that could change the decisions or say how they would be resolved.
    GPT-6 Luna · API · A quarterly business review deck, in brand

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

Our CEO, Lena, posted in the leadership channel: “Why did weekly active teams drop 6% last week? Is it the new navigation?” You own the product area. Using the material below, write your reply to Lena, to post in the channel. Keep it under 200 words.

What the model was given4 items: Scenario, Metrics, Incident, Support
ScenarioTrellis is a project-management tool for small teams. The new navigation went to 100% of web users on Tuesday 15 September after two weeks in a 10% beta. The mobile apps did not change.
MetricsWeekly active teams: week of 7 September 18,420; week of 14 September 17,310 (−6.0%). Web only: −6.3%. Mobile only: −5.2%. Teams that were in the navigation beta: −1.1%. The same week last year, weekly active teams fell 4%, which we put down to the start of the school year.
IncidentOn Thursday 17 September, single sign-on failed for two hours. About 30% of teams use single sign-on.
SupportTickets mentioning navigation: 212 last week, up from 40. Most are some version of “where did X go?”
What a strong answer doesThe answer key the graders mark against

Answers directly that the cause is not established and the navigation is unlikely to be the main driver (mobile, which did not change, fell nearly as much, and the beta cohort barely moved), while taking the navigation support tickets seriously. Names the other candidates with their evidence (seasonality, the SSO outage), says what is being checked and when Lena will hear back.

Critical failures (cap the score at 40)
  • States a single cause as established fact
Case

Mid-level · v1.8 · synthetic · metrics, exec communication

The outputs

7 setups answered this brief. Pick up to three to read side by side.

GPT-6.1 SolAPI91Usable with a quick editGPT-6 AstraChatGPT89Usable with a quick editGPT-6 LunaAPI86Usable with a quick editSonnet 5.5API59Needs more workOpus 5.5Claude59Needs more workGemini 3.5 Flash-LiteGemini43Needs more workGemini 3.8 FlashAPI30Needs more work
GPT-6 LunaAPI

Usable with a quick edit8 of 10 checks passed

The evidence and reasoning are correct; only small omissions like a timeline and already-taken actions would need a quick edit.

Output

Lena, we can’t yet attribute the 6% drop to the new navigation. The timing and jump in navigation-related tickets (212, up from 40) make discoverability a real concern, but teams in the two-week beta were down only 1.1%. The decline also appears on mobile (−5.2%), which didn’t change, so navigation alone is unlikely to explain it.

Two other factors may contribute: WAU fell 4% in the same week last year, around the start of the school year, and SSO was unavailable for two hours on Thursday for teams using it (about 30%).

I’d treat the navigation as a usability issue to address, not yet the proven cause of the overall decline. We’ll compare affected and unaffected cohorts, check the SSO impact, and investigate where users are getting stuck; meanwhile, we’ll prioritize fixes for the “where did X go?” feedback.

Grades and run details

Decision model 90 · LLM judge 9 of 11 checks
Decision model checks
  • passUses the supplied evidence correctly69%
  • partialAddresses the actual decision29%
  • passRespects explicit constraints32%
  • passIdentifies material uncertainty62%
  • passAvoids unsupported claims67%
  • passProduces the required deliverable53%
  • passAnswers the question asked90%
  • passSeparates effect from explanation84%
  • partialOwns the problem92%
  • passReady for the reader as delivered91%
Run
Run
#1
API response time
6 s
Submitted
29 Sept 2026
Opus 5.5Claude

Needs more than a quick edit5 of 10 checks passed

A PM would need to verify the weekly-active metric definition and add explicit conditions that would change the conclusion before posting; that is more than a quick edit.

Output

Why I don't think the nav is the main cause

Mobile didn't change and still fell 5.2%. Web fell 6.3%. So about 5 points hit everyone, and the nav can explain at most about 1 point on web.1 Seasonality likely covers most of the shared drop. The same week last year we fell 4% as the school year started. The SSO outage (Thursday, 2 hours, about 30% of teams) is probably minor. A team only needs one session in the week to count as active.2 I'll confirm by comparing SSO and non-SSO teams. Beta teams fell only 1.1%. That looks reassuring, but beta teams may be more engaged than average, so I'm not relying on it.

Where the nav is hurting Navigation tickets rose from 40 to 212, and most are "where did X go?" That's friction now and a churn risk if we leave it.

Next steps, by Wednesday

Split the drop by SSO vs non-SSO teams, and beta vs comparable non-beta teams. Ship quick fixes for the most-asked "where did X go" items. Watch this week's number. If the drop is seasonal, it should flatten.

What a PM had to fix

  1. 1Numbers wrongRedo the arithmeticTargeted repair

    Source: MetricsWeb only: −6.3%. Mobile only: −5.2%.

    What we’d changeDon't cap the navigation's effect at 1 point: web and mobile differ in more than the navigation, so the subtraction doesn't hold. Give seasonality less credit, and hold the SSO reasoning until the split is done.

  2. 2Invented evidenceVerify or remove the claimQuick edit

    What we’d changeThe brief doesn't define an active team. Present it as an assumption to check, since the SSO argument depends on it.

Claims the judge couldn’t find in the brief

  • A team only needs one session in the week to count as active.

Grades and run details

Decision model 80 · LLM judge 5 of 11 checks
Decision model checks
  • failUses the supplied evidence correctly26%
  • passAddresses the actual decision34%
  • passRespects explicit constraints70%
  • passIdentifies material uncertainty91%
  • partialAvoids unsupported claims25%
  • passProduces the required deliverable95%
  • passAnswers the question asked98%
  • passSeparates effect from explanation85%
  • partialOwns the problem80%
  • passReady for the reader as delivered68%
Run
Run
#1
Time to output
30 s
Submitted
27 Sept 2026
GPT-6.1 SolAPI

Usable with a quick edit9 of 10 checks passed

The reasoning and evidence are sound; only a minor addition of already-taken actions or a trade-off is needed.

Output

Lena — we don’t yet have enough evidence to attribute the 6% drop to navigation. My read is that it may be contributing, but it’s unlikely to be the whole story:

  • Mobile-only teams fell 5.2% despite no navigation change, versus 6.3% for web-only teams.
  • Navigation-beta teams fell just 1.1%. That’s reassuring, but they may not be representative.
  • The same week last year fell 4%, suggesting a seasonal component—not proving one.
  • Thursday’s two-hour SSO outage is another plausible contributor; about 30% of teams use SSO.

There is clear navigation friction: tickets rose from 40 to 212, mostly “where did X go?” That warrants action even if it doesn’t explain the activity decline.

I’ll break down daily activity by platform, SSO usage, and beta exposure, and check whether key navigation-dependent workflows deteriorated after Tuesday’s rollout. We’ll also prioritize fixes for the most common findability complaints.

I’ll share an initial read tomorrow, including whether the evidence supports a rollback. I wouldn’t treat seasonality, the outage, and navigation as additive explanations without that analysis.

Grades and run details

Decision model 90 · LLM judge 10 of 11 checks
Decision model checks
  • passUses the supplied evidence correctly66%
  • partialAddresses the actual decision20%
  • passRespects explicit constraints53%
  • passIdentifies material uncertainty90%
  • passAvoids unsupported claims73%
  • passProduces the required deliverable97%
  • passAnswers the question asked75%
  • passSeparates effect from explanation98%
  • partialOwns the problem81%
  • passReady for the reader as delivered94%
Run
Run
#1
API response time
8 s
Submitted
29 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Uses the supplied evidence correctlyRightWrongRight
GPT-6 Luna · API

All named facts, figures, and quotes match the supplied context with no invented current-state information.

Opus 5.5 · Claude

It introduces an unsupported definition that a weekly active team only needs one session, which is not in the supplied context and does not follow by arithmetic.

GPT-6.1 Sol · API

All factual numbers, dates, and quotes are drawn accurately from the supplied context, with interpretations clearly hedged.

Addresses the actual decisionWrongMixedRight
GPT-6 Luna · API

It commits to not attributing the drop but does not say what result would change that answer or when Lena will get it.

Opus 5.5 · Claude

It commits early that the nav is not the main cause but never states what result or condition would overturn that call.

GPT-6.1 Sol · API

The output commits early that navigation is unlikely to be the whole story and says tomorrow's analysis will show whether rollback is supported.

Identifies material uncertaintyRightMixedRight
GPT-6 Luna · API

It names the unknown cause, bounds navigation with beta and mobile data, and states it will compare cohorts and check SSO impact.

Opus 5.5 · Claude

It names unknowns and says how to investigate but does not specify what outcomes would change the decision.

GPT-6.1 Sol · API

It names the non-representative beta cohort, unproven seasonality, and SSO outage as unknowns and outlines the analysis that will resolve them.

Avoids unsupported claimsRightWrongRight
GPT-6 Luna · API

Causal statements are hedged as concerns or possibilities and are tied to supplied evidence.

Opus 5.5 · Claude

It presents the one-session active-definition claim as an established fact with no support.

GPT-6.1 Sol · API

Causes are labelled as possible ('may be contributing', 'plausible', 'suggesting') rather than asserted as fact.

All got wrong 1

Owns the problemWrongWrongWrong
GPT-6 Luna · API

It recommends next steps but does not state what the team has already done about the reported problems or offer trade-offs.

Opus 5.5 · Claude

It reports problems but does not state what the team has already done about them before recommending next steps.

GPT-6.1 Sol · API

It proposes future analysis and fixes but does not state any actions the team has already taken or offer trade-offs.

All got right 5

Respects explicit constraintsRightRightRight
GPT-6 Luna · API

It is a channel reply to Lena, under 200 words, and uses the supplied material.

Opus 5.5 · Claude

The reply is under 200 words and addressed as a channel reply to Lena.

GPT-6.1 Sol · API

The reply is under 200 words, addressed to Lena, and fits a leadership-channel update.

Produces the required deliverableRightRightRight
GPT-6 Luna · API

The required reply is present, complete, under 200 words, and usable as a leadership-channel update.

Opus 5.5 · Claude

The deliverable is a complete, usable channel reply within length.

GPT-6.1 Sol · API

It is a complete channel reply with an answer, evidence, and next steps, within the requested length.

Answers the question askedRightRightRight
GPT-6 Luna · API

The first line directly answers that the drop cannot yet be attributed to the new navigation.

Opus 5.5 · Claude

The first lines answer that the navigation is not the main cause.

GPT-6.1 Sol · API

The first line directly addresses whether the new navigation caused the drop.

Separates effect from explanationRightRightRight
GPT-6 Luna · API

Navigation, seasonality, and SSO are treated as possible explanations, each with supporting or contradicting evidence.

Opus 5.5 · Claude

Causes are labelled with uncertainty and paired with supporting or contradicting evidence.

GPT-6.1 Sol · API

Each candidate explanation is framed as a hypothesis with supporting or contradicting evidence (mobile unchanged, beta low, seasonality, SSO).

Ready for the reader as deliveredRightRightRight
GPT-6 Luna · API

It is written entirely for Lena with no placeholders, unexplained terms, or invented owners or dates.

Opus 5.5 · Claude

Written for Lena with no unclear placeholders or lifted internal wording.

GPT-6.1 Sol · API

It is written entirely for Lena with no placeholders or unexplained internal labels.

Results

Every setup we’ve tested on this task, across all cases and repeats, graded on the current checklist. Calibrated: the graders match our PM on 80% of checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstraChatGPT86.787.12None
2GPT-6.1 SolAPI75.883.02None
3GPT-6 LunaAPI73.882.62None
4Opus 5.5Claude80.451.92None
5Sonnet 5.5API68.864.42None
6Gemini 3.5 Flash-LiteGemini62.139.02None
7Gemini 3.8 FlashAPI44.837.92None

The PM job

Writing the weekly update, or the reply to an executive's question, that the team's credibility rests on.

Why it matters

Updates are where bad news gets softened. A fluent update that buries a slipped date or presents a guess as the cause does more harm than no update: leadership decides on it, and trust goes when the truth comes out.

What good looks like

  • Leads with the news and any decision needed
  • States dates, numbers and causes plainly, including bad news
  • Separates what is known from what is suspected
  • Brings a recommendation and what's been tried, not just the problem
  • Fits the reader and the length asked for

Deliberately not measured

  • Tone and formatting preferences
  • Writing style beyond clarity
Capability tested

Honest, decision-first status communication

The failure we’re looking for

Spin: burying or softening the bad news, or presenting a guess as fact

Grading

Decision model, LLM judge and a browser check (for the deck), calibrated against a blind PM review

Variants

A short written update · Staff level: a QBR deck in brand colours