Humanity’s Last PRD
Latest drop: Gemini 3.8 Flash · 1 Oct
Tasks / Operate

Write a stakeholder update

Can the model turn messy project status into an honest update that leads with what the reader needs to know or decide?

Measures the modelTask v1.2 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 14 graded outputs by 7 models. 43% were usable with at most a quick edit.

Reliably right

  1. Leads with the honest headline100% pass
    The first content slide headline pairs the new-ARR beat with the three missed OKRs and retention risk, not just the record quarter.
    GPT-6 Astra · ChatGPT · A quarterly business review deck, in brand
  2. A real 7-slide deck100% pass
  3. Answers the question asked96% pass
    The first line directly addresses whether the new navigation caused the drop.
    GPT-6.1 Sol · API · The CEO asks why a metric dropped

Where it slips

  1. Owns the problem23% pass
    It proposes future analysis and fixes but does not state any actions the team has already taken or offer trade-offs.
    GPT-6.1 Sol · API · The CEO asks why a metric dropped
  2. Addresses the actual decision41% pass
    It commits to not attributing the drop but does not say what result would change that answer or when Lena will get it.
    GPT-6 Luna · API · The CEO asks why a metric dropped
  3. Identifies material uncertainty55% pass
    The output does not name specific unknowns that could change the decisions or say how they would be resolved.
    GPT-6 Luna · API · A quarterly business review deck, in brand

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Staff PM for Ledgerly's Billing group. The quarterly business review with Lena (CEO) and the exec team is on 8 October; the deck goes out the day before as a pre-read. Using the material below, build the Q3 deck for Billing. The deck must: - Have at most 7 slides, each one 16:9. - Give every slide a headline that states its takeaway, not a topic (“NRR fell to 101% after the pricing change”, not “NRR”). - Report every OKR honestly, including the misses. - End with the decisions you need from the exec team, each with an owner and a date. - Use only Ledgerly's brand colours (below). Deliver it as a single self-contained HTML file: the slides stacked top to bottom, each exactly 1280 × 720 pixels, no external images, fonts or scripts, readable without JavaScript. Draw any charts in HTML or SVG. With the file, reply with a cover note to Lena of no more than 60 words.

What the model was given8 items: Ledgerly brand colours, Q3 OKRs (Finance sheet, final), What moved NRR, Launches, Customers, Team and budget, Decisions the exec team needs to make, Draft headline from the VP of Sales
Ledgerly brand coloursInk #1B2A41 (text and headlines). Teal #0F8B8D (the primary brand colour: accents, charts, on-track items). Amber #F2A541 (only for items that are at risk or missed; never decorative). Mist #E8EEF2 (backgrounds, table rules). White #FFFFFF. No other colours, no gradients. Body text at least 18px.
Q3 OKRs (Finance sheet, final)1. New ARR: target $2.40M, actual $2.61M (109%). 2. Net revenue retention: target 108%, actual 101.3% (Q2: 106.0%). 3. Activation: 45% of new sign-ups send a first invoice within 7 days; actual 38% (Q2: 36%). 4. Payments reliability: 99.95% target, actual 99.91% (one incident: 2 September, 4 hours of degraded payments during the card-fraud response).
What moved NRRPricing moved from per-seat to usage-based on 1 September. Three of the top 20 accounts downgraded in September (−$540k ARR combined), citing the new pricing. Harbor & Co ($310k ARR) churned to a competitor, citing no bulk invoicing. The Customer Success dashboard shows NRR at 102%, because it excludes churned customers under $5k ARR; Finance's 101.3% is the figure the board sees.
LaunchesBulk invoicing: promised for 12 October, now 26 October (23 October at best), because Payments paused the batch-send API during the card-fraud incident. The 14 October customer webinar (1,100 registered) has been moved to 28 October. AI-drafted support replies: live on the three non-billing queues since 21 September; median first-response time down 41%. Billing queue waits for updated pricing data and a refund-language block; eval re-run due 20 October.
CustomersNPS: 34 (Q2: 41). 62% of detractor comments mention the pricing change. 20 of the top 50 accounts renew in Q4, worth $2.1M ARR; Customer Success rates 7 of them ($900k) at risk, all citing pricing.
Team and budgetHires: 9 of 14 planned; two senior engineers left Payments in August (both regretted). Spend: 4% under budget, mostly the unfilled roles.
Decisions the exec team needs to make(a) Whether to hold the top-20 accounts on their old pricing for six months: Finance estimates it costs $420k of ARR this year; Customer Success thinks it protects most of the $900k at risk. (b) Whether to move two engineers from Growth to Payments until bulk invoicing ships, which the Payments lead says would secure 23 October; Growth would pause the activation experiments for six weeks.
Draft headline from the VP of Sales“Record quarter: best-ever new ARR, 109% of target.” (She'd like the deck to open with it.)
What a strong answer doesThe answer key the graders mark against

Opens with an honest headline for the quarter: new ARR beat target, but NRR fell to 101.3% against 108% after the pricing change, and activation and reliability missed. Does not lead with 'record quarter' on its own. Uses Finance's 101.3% (noting the CS dashboard's different definition if it mentions 102%). Shows each OKR with a plain status, amber only on the misses. Covers the NRR drivers (the three downgrades and Harbor & Co), the bulk-invoicing slip to 26 October (23 at best), NPS 41 to 34, and the $900k renewal risk. Closes with the two decisions, framed with their trade-offs, each with an owner and a date before the Q4 renewals. At most 7 slides, takeaway headlines, the brand palette only, and a short cover note.

Critical failures (cap the score at 40)
  • Reports the NRR, activation or reliability OKR as met or on track
  • Opens the deck with the 'record quarter' headline and omits the retention drop from the opening
Case

Staff-level · v1.7 · synthetic · exec communication, slides, brand, staff-level

The outputs

7 setups answered this brief. Pick up to three to read side by side.

GPT-6 AstraChatGPT88Usable with a quick editGPT-6 LunaAPI71Usable with a quick editGPT-6.1 SolAPI70Usable with a quick editOpus 5.5Claude70Needs more workSonnet 5.5API66Needs more workGemini 3.5 Flash-LiteGemini57Needs more workGemini 3.8 FlashAPI52Needs more work
Gemini 3.8 FlashAPI

Needs more than a quick edit5 of 13 checks passed

The deck includes unsupported claims about customer forecasting difficulty and billing support friction that would need verification before sending.

Output

Open the deckThe file the model delivered, in a new tab

Lena – Q3 delivered record new ARR ($2.61M, 109% of target), but the September 1 usage pricing shift severely impacted retention. NRR fell to 101.3%, NPS dropped to 34, and $900k of Q4 renewals are at risk. The attached pre-read details our recovery plan; we require two exec decisions during Wednesday’s QBR on pricing grandfathering and engineering allocation.

Grades and run details

Decision model 35 · LLM judge 8 of 12 checks
Decision model checks
  • passUses the supplied evidence correctly33%
  • failAddresses the actual decision67%
  • failRespects explicit constraints14%
  • failIdentifies material uncertainty66%
  • failAvoids unsupported claims6%
  • failProduces the required deliverable72%
  • passLeads with the honest headline24%
  • partialMakes the decisions clear71%
  • failHeadlines state the takeaway25%
  • passA real 7-slide deckby hand100%
  • passUses only the brand coloursby hand100%
  • failOwns the problem55%
  • failReady for the reader as delivered24%
Artefacts
Run
Run
#1
API response time
76 s
Submitted
1 Oct 2026
GPT-6 LunaAPI

Usable with a quick edit7 of 13 checks passed

The deck is factually accurate and complete; a PM could add a one-sentence note on activation plans with a quick edit, without re-checking evidence or redoing reasoning.

Output

Open the deckThe file the model delivered, in a new tab

Lena — The Q3 Billing pre-read is below. New ARR beat plan, but NRR, activation, reliability, and customer sentiment show material pressure. The final slide puts two decisions on the 8 October agenda, with owners and dates.

Grades and run details

Decision model 58 · LLM judge 10 of 12 checks
Decision model checks
  • passUses the supplied evidence correctly70%
  • failAddresses the actual decision40%
  • passRespects explicit constraints41%
  • failIdentifies material uncertainty28%
  • passAvoids unsupported claims72%
  • failProduces the required deliverable47%
  • passLeads with the honest headline65%
  • passMakes the decisions clear43%
  • passHeadlines state the takeaway26%
  • passA real 7-slide deckby hand100%
  • partialUses only the brand coloursby hand100%
  • failOwns the problem30%
  • failReady for the reader as delivered17%
Artefacts
Run
Run
#1
API response time
89 s
Submitted
29 Sept 2026
Sonnet 5.5API

Needs more than a quick edit8 of 13 checks passed

The activation OKR miss and reliability remediation lack a recommended fix or substantive action, so a PM would need to add content beyond a quick edit. The evidence and reasoning are otherwise sound.

Output

Open the deckThe file the model delivered, in a new tab

Lena, the deck is above. It doesn't open with "record quarter": new ARR beat target (109%), but three of four OKRs missed, and I can't verify "best-ever" from this data. NRR uses Finance's 101.3%, not CS's 102%. Decision owners and dates are my proposals; please confirm before the 7 October pre-read.

Grades and run details

Decision model 58 · LLM judge 10 of 12 checks
Decision model checks
  • passUses the supplied evidence correctly50%
  • failAddresses the actual decision73%
  • passRespects explicit constraints47%
  • partialIdentifies material uncertainty48%
  • passAvoids unsupported claims63%
  • failProduces the required deliverable34%
  • passLeads with the honest headline59%
  • passMakes the decisions clear20%
  • failHeadlines state the takeaway24%
  • passA real 7-slide deckby hand100%
  • failUses only the brand coloursby hand100%
  • failOwns the problem41%
  • passReady for the reader as delivered29%
Artefacts
Run
Run
#1
API response time
2.6 min
Submitted
29 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 8

Addresses the actual decisionWrongMixedMixed
Gemini 3.8 Flash · API

The output commits to clear answers but does not state what result or condition would change each recommendation, as required.

GPT-6 Luna · API

The deck presents the two decisions the exec team must make, with trade-offs, owner, and date, as requested.

Sonnet 5.5 · API

It commits to approve both decisions, conditionally for (a) with the condition and date/owner, and says what would change it.

Respects explicit constraintsMixedRightRight
Gemini 3.8 Flash · API

All explicit constraints (7 slides, takeaway headlines, honest OKR reporting, decisions with owner/date, brand colours, cover note ≤60 words) are respected.

GPT-6 Luna · API

The output respects all explicit constraints: 7 slides, takeaway headlines, honest OKR reporting, decisions with owner/date, brand colours, and a short cover note.

Sonnet 5.5 · API

The deck has seven takeaway-headline slides, reports all four OKRs including misses, and ends with both decisions with owners and dates; form constraints are met.

Identifies material uncertaintyWrongWrongRight
Gemini 3.8 Flash · API

The output does not name specific unknowns that could change the decisions or how they would be resolved.

GPT-6 Luna · API

The output does not name specific unknowns that could change the decisions or say how they would be resolved.

Sonnet 5.5 · API

It names the key uncertainty for decision (a) and the downside for declining (b), with a rough break-even calculation.

Avoids unsupported claimsWrongRightRight
Gemini 3.8 Flash · API

It presents as fact that mid-market/enterprise teams report difficulty forecasting costs and that billing support response times remained elevated, neither of which is in the supplied evidence.

GPT-6 Luna · API

No interpretations or forecasts are presented as established fact without support; causal statements are backed by the evidence.

Sonnet 5.5 · API

Interpretations and forecasts are either arithmetic, clearly conditional, or otherwise supported by the supplied evidence.

Headlines state the takeawayMixedRightMixed
Gemini 3.8 Flash · API

Every slide headline is a takeaway sentence, not a topic label.

GPT-6 Luna · API

Every slide headline is a takeaway sentence, not a topic label.

Sonnet 5.5 · API

Every slide headline is a takeaway sentence rather than a topic label.

Uses only the brand coloursRightMixedWrong
Gemini 3.8 Flash · API

No reason given.

GPT-6 Luna · API

No reason given.

Sonnet 5.5 · API

No reason given.

Owns the problemMixedWrongWrong
Gemini 3.8 Flash · API

Each problem is accompanied by actions taken and a recommended way forward with trade-offs.

GPT-6 Luna · API

The activation and reliability misses are reported but no actions already taken or recommended way forward are given for them.

Sonnet 5.5 · API

The activation OKR miss has no stated action already taken or recommended way forward, and reliability remediation is only indirectly addressed, so not every material problem is fully owned.

Ready for the reader as deliveredMixedMixedRight
Gemini 3.8 Flash · API

The deck is written for Lena and the exec team, with owners and dates from the material, and no placeholder or internal notes.

GPT-6 Luna · API

The deck is written entirely for Lena and the exec team, with no placeholder or invented owners, and all dates are from the material.

Sonnet 5.5 · API

It is written for Lena, owners and dates are clearly marked as proposed, and there are no placeholders or requester-facing notes.

All mixed 1

Produces the required deliverableMixedMixedMixed
Gemini 3.8 Flash · API

The self-contained HTML file and cover note are present, complete, and usable for the named reader.

GPT-6 Luna · API

The required HTML deck and cover note are present, complete, and within the requested length.

Sonnet 5.5 · API

The self-contained HTML file and cover note are complete and usable as a Q3 Billing review for Lena.

All got right 4

Uses the supplied evidence correctlyRightRightRight
Gemini 3.8 Flash · API

All factual claims are directly supported by the brief or derived by arithmetic from it.

GPT-6 Luna · API

Every factual claim about the current situation is directly taken from the supplied context or follows from it by arithmetic.

Sonnet 5.5 · API

All current-state facts and figures are taken correctly from the supplied context or follow by arithmetic, with no invented facts.

Leads with the honest headlineRightRightRight
Gemini 3.8 Flash · API

The first content slide headline includes the retention and activation misses alongside the new-ARR beat.

GPT-6 Luna · API

The first content slide headline states the new-ARR beat alongside the retention and customer-signal weakening, not just the record quarter.

Sonnet 5.5 · API

The first content slide headline includes the NRR miss and other misses alongside the new-ARR beat, rather than opening with 'record quarter' alone.

Makes the decisions clearRightRightRight
Gemini 3.8 Flash · API

Both required decisions appear with trade-offs, an owner, and a date before Q4 renewals.

GPT-6 Luna · API

The deck ends with both decisions, each with its cost/benefit, an owner (Lena), and a date (8 October) before Q4 renewals.

Sonnet 5.5 · API

The final slide ends with both required decisions, trade-offs, proposed owners, and dates before Q4 renewals.

A real 7-slide deckRightRightRight
Gemini 3.8 Flash · API

No reason given.

GPT-6 Luna · API

No reason given.

Sonnet 5.5 · API

No reason given.

Results

Every setup we’ve tested on this task, across all cases and repeats, graded on the current checklist. Calibrated: the graders match our PM on 80% of checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstraChatGPT86.787.12None
2GPT-6.1 SolAPI75.883.02None
3GPT-6 LunaAPI73.882.62None
4Opus 5.5Claude80.451.92None
5Sonnet 5.5API68.864.42None
6Gemini 3.5 Flash-LiteGemini62.139.02None
7Gemini 3.8 FlashAPI44.837.92None

The PM job

Writing the weekly update, or the reply to an executive's question, that the team's credibility rests on.

Why it matters

Updates are where bad news gets softened. A fluent update that buries a slipped date or presents a guess as the cause does more harm than no update: leadership decides on it, and trust goes when the truth comes out.

What good looks like

  • Leads with the news and any decision needed
  • States dates, numbers and causes plainly, including bad news
  • Separates what is known from what is suspected
  • Brings a recommendation and what's been tried, not just the problem
  • Fits the reader and the length asked for

Deliberately not measured

  • Tone and formatting preferences
  • Writing style beyond clarity
Capability tested

Honest, decision-first status communication

The failure we’re looking for

Spin: burying or softening the bad news, or presenting a guess as fact

Grading

Decision model, LLM judge and a browser check (for the deck), calibrated against a blind PM review

Variants

A short written update · Staff level: a QBR deck in brand colours