Humanity’s Last PRD
Latest drop: Gemini 4 Argon · 13 Oct
Tasks / Leadership

Write a stakeholder update

Can the model turn messy project status into an honest update that leads with what the reader needs to know or decide?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 50% were usable with at most a quick edit.

Reliably right

  1. Reports outcomes and learning, not just output100% pass
    It leads with outcome metrics (ARR, support, usage) and reports a key learning (pipeline jump is a definition change, not demand growth), not just activities.
    GPT-6 Astra · ChatGPT · On track, mostly
  2. Leads with the honest headline100% pass
    First slide headline includes the retention drop alongside the new-ARR beat, not just 'record quarter'.
    GPT-6 Astra · ChatGPT · A quarterly business review deck, in brand
  3. A real 7-slide deck100% pass

Where it slips

  1. Uses the supplied evidence correctly51% pass
    The year 2026 is not in the supplied context; all other factual claims are supported.
    GPT-6 Astra · ChatGPT · A quarterly business review deck, in brand
  2. Owns the problem53% pass
    Does not state what the team has already done about the payments reliability miss or the NPS drop; only reports the problems.
    GPT-6 Astra · ChatGPT · A quarterly business review deck, in brand
  3. Flags the redefined pipeline58% pass
    It names the redefinition but does not give the falling like-for-like trend (e.g., pipeline fell from $2.41M to $2.23M on the old definition).
    GPT-6.1 Sol · API · On track, mostly

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Asks leaders only for the big calls

    Does the update separate the decisions it needs from its readers (hard to reverse, high stakes) from the ones the team will make itself, and ask clearly for the former?

    Passes when Names the decisions it needs from the readers, with a recommendation and a date, and says which calls the team is making itself.

    Geoff CharlesLenny’s Podcast

  2. Reports outcomes and learning, not just output

    Does the update report progress as movement on the outcomes the team is accountable for and what it has learned, rather than as features shipped or dates hit?

    Passes when Leads with the outcome metrics and what changed in the team's understanding, with shipping as supporting detail.

    Melissa PerriLenny’s PodcastChristian IdiodiLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Owns the problem · Ready for the reader as delivered · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Head of Product at Cobalt. Write this fortnight's update for the CEO and the exec team, in no more than 450 words, from the weekly metrics and the team notes below. It should say where we stand, what's wrong, what we're doing about it, and what you need from them.

What the model was given3 items: About Cobalt, weekly_metrics_weeks_1_to_8.csv, team_notes.md
About CobaltTeam scheduling software, sold per seat.
weekly_metrics_weeks_1_to_8.csv8 rows · Download
week,new_arr_signed_usd,qualified_pipeline_usd,weekly_active_teams,support_backlog_tickets,support_sla_met_pct
1,182000,2410000,8210,140,96
2,205000,2380000,8260,152,95
3,168000,2350000,8240,171,94
4,214000,2290000,8300,198,91
5,199000,2230000,8280,260,86
6,221000,3140000,8310,344,79
…
team_notes.md7 lines · Download
# Notes from the teams

- Sales ops: from week 6, "qualified pipeline" includes active free trials over 10 seats (they used to be excluded).
- Data: weekly active teams for week 7 is missing because of a pipeline outage; it's being backfilled.
- Support: the backlog grew after the September pricing change (tickets about invoices doubled). SLA target is 90% answered within one business day.
- Quarter target: $2.4M new ARR in 12 weeks ($200,000 a week). Weeks 1 to 8 total: see the file.
- Support hiring: two agents approved, start date not set. Decision needed: whether to pause the planned price change for existing customers in week 10 until the backlog is cleared.
What a strong answer doesThe answer key the graders mark against

Leads with the takeaway: new ARR is ahead of plan ($1.67M in eight weeks against $1.6M), but support is breaking and pipeline is weaker than it looks. Flags that the jump in qualified pipeline in week 6 (from about $2.23M to $3.14M) is the redefinition to include large trials; on the old definition pipeline fell every week from $2.41M to $2.23M, a risk for the rest of the quarter. Reports the support problem plainly: backlog from 140 to 455 tickets and SLA from 96% to 72% against a 90% target since the September pricing change, with what's being done (two agents approved). Notes week 7's active teams are missing and being backfilled rather than hiding it. Asks for the decisions that need the execs: pause the week-10 price change for existing customers until the backlog clears, and set the start date for the support hires. Separates what the team is handling from what needs them.

Critical failures (cap the score at 40)
  • Presents qualified pipeline as having grown about 40%
Task

Staff-level · v1.1

The outputs

9 setups answered this brief.

GPT-6 AstraChatGPT98Usable with a quick editSonnet 5.5API94Usable with a quick editGPT-6.1 SolAPI88Usable with a quick editHaiku 5.5API85Needs more workOpus 5.5Claude77Needs more workGPT-6 LunaAPI75Usable with a quick editGemini 4 ArgonAPI73Needs more workGemini 3.8 FlashAPI65Needs more workGemini 3.5 Flash-LiteGemini56Needs more work
Gemini 4 ArgonAPI

Needs more than a quick edit9 of 12 checks passed

The output contains an unsupported claim about product and support auditing invoice layouts that is not in the source material. A PM would need to verify or remove it before sending.

Output

Subject: Product Fortnightly Update – Weeks 7 & 8

Where We Stand * New ARR: We are tracking ahead of our Q3 target ($2.4M over 12 weeks, or $200k/week). Through Week 8, we have signed $1,673,000 in new ARR (averaging ~$209k/week). Momentum is strong, with Weeks 7 and 8 delivering $236k and $248k, respectively. * Pipeline: Qualified pipeline sits at $3.35M. Note on the data: The ~40% jump starting in Week 6 ($2.23M to $3.14M) reflects a Sales Ops methodology change to include active free trials with >10 seats, rather than a sudden spike in traditional pipeline. * Active Usage: Weekly Active Teams (WAT) dipped to 8,190 in Week 8—our lowest point this quarter (down from 8,310 in Week 6). Week 7 data is currently missing due to a data pipeline outage.

What’s Wrong * Support is underwater: Following the September pricing change, invoice-related tickets doubled. Over eight weeks, our support backlog has more than tripled from 140 to 455 tickets. Consequently, our SLA compliance (target: 90% answered within one business day) has fallen below target since Week 5, hitting 72% in Week 8. * Early churn/engagement risk: While one week isn't a trend, the Week 8 drop in WAT—despite eight weeks of healthy new ARR additions—is concerning. We lacked visibility in Week 7 due to the data outage, masking when this dip actually began.

What We’re Doing About It * Fixing the data blindspot: The Data team is backfilling the Week 7 WAT numbers so we can properly diagnose the Week 8 usage drop and see if it correlates with customers affected by the billing confusion. * Reducing ticket volume at the source: Product is partnering with Support to audit the new invoice layout and billing UX, aiming to ship clearer in-app explanations and self-serve billing FAQs this sprint to deflect incoming tickets. * Expanding capacity: Two new support agents are approved to help clear the backlog, though start dates are not yet finalized.

What I Need From You 1. Decision – Pause the Week 10 legacy price migration: We are scheduled to roll out the price change to existing customers in Week 10. Given that our per-seat billing changes caused invoice tickets to double, rolling this out to our entire install base while SLA sits at 72% will break Support and risk churn. I recommend we pause the Week 10 rollout until the backlog is cleared and the new agents are ramped. Please confirm by Friday if we have approval to hold. 2. Expedite Support Onboarding: I need People/Ops to help us lock in immediate start dates for the two approved support hires.

Claims the judge couldn’t find in the brief

  • Product is partnering with Support to audit the new invoice layout and billing UX, aiming to ship clearer in-app explanations and self-serve billing FAQs this sprint.

Grades and run details

Decision model 83 · LLM judge 69
Decision model checks
  • failUses the supplied evidence correctly59%
  • passAddresses the actual decision27%
  • passRespects explicit constraints49%
  • passIdentifies material uncertainty92%
  • passAvoids unsupported claims23%
  • passProduces the required deliverable94%
  • partialFlags the redefined pipeline92%
  • passReports the support breach plainly100%
  • partialOwns the problem27%
  • passReady for the reader as delivered35%
  • passAsks leaders only for the big calls56%
  • passReports outcomes and learning, not just output97%
Run
Run
#1
API response time
7 s
Submitted
13 Oct 2026
Sonnet 5.5API

Usable with a quick edit11 of 12 checks passed

The output is accurate, complete, and well-structured. A PM could use it with only minor wording tweaks, without re-checking evidence or redoing any reasoning.

Output

Cobalt product update: weeks 7-8

Bottom line: New ARR is on track, but support is deteriorating and our pipeline number is misleading. I need a decision on the week 10 price change.

Where we stand - New ARR: $1.673M signed over 8 weeks, against a pro-rata target of $1.6M. We're $73k ahead. Hitting $2.4M needs about $182k a week over the last four, and we've averaged $209k. - Active teams: 8,190 in week 8, down from a peak of 8,310 in week 6 and roughly flat since week 1. Week 7 is missing because of a data outage and is being backfilled, so I'm not reading a trend into it.

What's wrong 1. Support is failing its SLA. Backlog has risen from 140 to 455 tickets. The SLA hit rate fell from 96% to 72% against a 90% target, and we've been under target since week 5. Support attributes this to the September pricing change, which doubled invoice tickets. Weekly backlog growth slowed in week 8 (+43 versus +68), but it's still growing. 2. Pipeline looks healthier than it is. Reported qualified pipeline jumped from $2.23M to $3.14M in week 6, but that's when Sales Ops started counting active free trials over 10 seats. Before that, pipeline had drifted down for five straight weeks ($2.41M to $2.23M). Until it's restated, I can't tell you whether underlying pipeline is growing, and that matters for next quarter's ARR.

What we're doing - Invoice tickets: We're working with Support to identify the top invoice-confusion causes, then ship clearer invoice explanations and canned responses to cut ticket volume at the source. - Pipeline: I've asked Sales Ops and Data to restate weeks 1-8 on a consistent definition, showing trials separately. I'll bring that next update. - Active teams: Data will backfill week 7, and I'll confirm whether the dip since week 6 is real.

What I need from you 1. Decision: pause the week 10 price change for existing customers until the SLA is back at 90%. The last pricing change drove a ticket surge we haven't absorbed, and a second one on top of a 455-ticket backlog risks churn and more SLA misses. New ARR has a cushion, so I think the delay is affordable. I'd like Finance to confirm the revenue impact of waiting. 2. Set start dates for the two approved support agents. They're approved but have no date, and the backlog is growing now. Two agents may not clear it alone, so I'd also like to discuss temporary cover. 3. Agree that we report pipeline on the restated basis (trials shown separately) from the next update.

Grades and run details

Decision model 88 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly79%
  • passAddresses the actual decision49%
  • passRespects explicit constraints62%
  • passIdentifies material uncertainty98%
  • passAvoids unsupported claims59%
  • passProduces the required deliverable89%
  • passFlags the redefined pipeline100%
  • passReports the support breach plainly100%
  • passOwns the problem76%
  • passReady for the reader as delivered77%
  • partialAsks leaders only for the big calls64%
  • passReports outcomes and learning, not just output100%
Run
Run
#1
API response time
17 s
Submitted
6 Oct 2026
Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit6 of 12 checks passed

The ARR total is wrong ($1.73M instead of $1.673M), a factual error that would require re-checking the evidence before use.

Output

To: CEO & Exec Team From: Head of Product, Cobalt Subject: Fortnightly Update (Weeks 7–8)

Here is where we stand for the last two weeks, the critical issues we face, our action plan, and where I need your input.

Where We Stand

We are currently outperforming our top-line targets. We’ve brought in $1.73M in new ARR across 8 weeks against our $1.6M pro-rata goal ($200k/week), closing Week 8 at a strong $248k. Pipeline looks healthy on paper, jumping to $3.35M. However, caveat emptor: Sales Ops adjusted the pipeline definition in Week 6 to include active free trials over 10 seats, masking underlying organic pipeline velocity. Furthermore, weekly active teams dipped to 8,190 in Week 8 (Week 7 data is currently missing due to a data pipeline outage, currently being backfilled).

What’s Wrong

Customer Support is in a critical state. Our support backlog has exploded to 455 tickets, and SLA compliance has plummeted to 72% (target: 90%).

This was triggered by the September pricing change. Invoicing-related tickets have doubled, and our current support bandwidth is completely overwhelmed. While two new hires have been approved, we do not yet have confirmed start dates.

What We’re Doing About It

  1. Stabilizing Support: We are exploring short-term triage options, including engineering-led documentation updates to deflect self-service billing questions.
  2. Data Recovery: Data engineering is actively backfilling the Week 7 active teams metric to restore full visibility.
  3. Pipeline Audit: Sales Ops is recalibrating the pipeline reporting to separate true enterprise/mid-market opportunities from the newly included free trials so we can accurately forecast Q3 close rates.

What I Need From You

We face an immediate strategic decision regarding Week 10:

  • Decision Needed: Should we pause the planned price change for existing customers scheduled for Week 10 until the current support backlog is cleared and SLAs are back above 90%?

Proceeding with the existing customer rollout while our SLA is at 72% risks a wave of churn and avoidable escalation tickets. I recommend pausing, but I need a final call from the exec team so Product and Support can communicate the adjustment internally and externally.

Claims the judge couldn’t find in the brief

  • We’ve brought in $1.73M in new ARR across 8 weeks

Grades and run details

Decision model 71 · LLM judge 46
Decision model checks
  • failUses the supplied evidence correctly69%
  • partialAddresses the actual decision34%
  • passRespects explicit constraints45%
  • passIdentifies material uncertainty85%
  • partialAvoids unsupported claims40%
  • passProduces the required deliverable58%
  • partialFlags the redefined pipeline95%
  • passReports the support breach plainly100%
  • partialOwns the problem55%
  • passReady for the reader as delivered64%
  • partialAsks leaders only for the big calls35%
  • passReports outcomes and learning, not just output92%
Run
Run
#1
Time to output
3 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 6

Uses the supplied evidence correctlyWrongMixedWrong
Gemini 4 Argon · API

The output claims Product is auditing invoice layouts and shipping FAQs this sprint, an action not in the supplied context.

Sonnet 5.5 · API

All factual claims about the current situation are directly supported by the supplied metrics or team notes, or follow from simple arithmetic.

Gemini 3.5 Flash-Lite · Gemini

The total new ARR is stated as $1.73M, but the sum of weeks 1-8 is $1.673M; this factual error means the output does not use the supplied evidence correctly.

Addresses the actual decisionRightRightWrong
Gemini 4 Argon · API

Clearly recommends pausing the Week 10 price change and expediting support hiring, with a deadline for approval.

Sonnet 5.5 · API

Commits unambiguously to pausing the week 10 price change, frames it for the exec team, and implies the call could change if Finance confirms the revenue impact is unaffordable.

Gemini 3.5 Flash-Lite · Gemini

The output recommends pausing the price change but does not state what result or condition would change that recommendation, as required.

Identifies material uncertaintyRightRightMixed
Gemini 4 Argon · API

Acknowledges the WAT dip isn't a trend, states conditions for unpausing (backlog cleared, agents ramped), and flags the missing data and unconfirmed start dates.

Sonnet 5.5 · API

Identifies the pipeline restatement, active teams backfill, revenue impact of delay, and sufficiency of two agents as unknowns, and says how each will be resolved.

Gemini 3.5 Flash-Lite · Gemini

It does not name the specific unknowns that could change the decision about the price change, nor say how they would be resolved or what would change the call.

Avoids unsupported claimsMixedRightWrong
Gemini 4 Argon · API

Presents the product/support audit and FAQ work as an established action, while the evidence contains no mention of it.

Sonnet 5.5 · API

Causes and forecasts are clearly labelled as such (e.g., 'risks churn', 'I think the delay is affordable'), and confident claims are backed by the evidence.

Gemini 3.5 Flash-Lite · Gemini

The $1.73M ARR figure is presented as fact but is not supported by the data; it is a confident claim that the evidence does not establish.

Produces the required deliverableRightRightMixed
Gemini 4 Argon · API

The memo is complete, covers standing, problems, actions, and asks, and the exec team can act on it with light edits.

Sonnet 5.5 · API

The update is complete, in the right form for the CEO and exec team, within the word limit, and the reader could act on the asks without filling major gaps.

Gemini 3.5 Flash-Lite · Gemini

The update contains a material factual error in the ARR total, so the reader cannot act on it without correcting that number.

Flags the redefined pipelineWrongRightWrong
Gemini 4 Argon · API

Flags the redefinition but does not give the falling like-for-like trend on the old definition (pipeline declined from $2.41M to $2.23M).

Sonnet 5.5 · API

Explicitly names the redefinition and gives the like-for-like trend of five straight weekly declines from $2.41M to $2.23M.

Gemini 3.5 Flash-Lite · Gemini

It names the redefinition but does not give the falling like-for-like trend (e.g., pipeline on the old definition fell from $2.41M to $2.23M).

All got right 6

Respects explicit constraintsRightRightRight
Gemini 4 Argon · API

The update is well within 450 words and uses the requested fortnightly memo form for the exec team.

Sonnet 5.5 · API

The update is under 450 words, written as a fortnightly update for the CEO and exec team, and respects the requested form.

Gemini 3.5 Flash-Lite · Gemini

The update is under 450 words and addressed to the CEO and exec team as requested.

Reports the support breach plainlyRightRightRight
Gemini 4 Argon · API

Reports the backlog and SLA breach with numbers, ties them to the pricing change, and explicitly asks for the pause decision.

Sonnet 5.5 · API

Reports the backlog (140 to 455) and SLA (96% to 72% against 90% target), links them to the pricing change, and asks for the pause decision.

Gemini 3.5 Flash-Lite · Gemini

It reports the backlog (455) and SLA (72% vs 90% target), links them to the pricing change, and asks for the pause decision.

Owns the problemRightRightRight
Gemini 4 Argon · API

For support and WAT drop, the update names actions being taken (data backfill, hiring, product work) and recommends a way forward.

Sonnet 5.5 · API

For each problem, states actions already taken (working with Support, asking Sales Ops and Data to restate) and recommends a way forward with trade-offs.

Gemini 3.5 Flash-Lite · Gemini

For support it notes the approved hires and recommends pausing the price change; for data and pipeline it states the actions being taken.

Ready for the reader as deliveredRightRightRight
Gemini 4 Argon · API

Written directly for the CEO/exec team; no placeholders, internal notes, or unexplained jargon from the brief.

Sonnet 5.5 · API

Written entirely for the exec team, uses only owners and terms from the context, and contains no placeholders or notes to the requester.

Gemini 3.5 Flash-Lite · Gemini

The update is written for the exec team, uses no unexplained jargon from the brief, and all owners/dates are from the material or clearly proposed.

Asks leaders only for the big callsRightRightRight
Gemini 4 Argon · API

Separates the big calls needing exec input (pause price change, expedite hires) from team actions, with a recommendation and timeline.

Sonnet 5.5 · API

Separates the big calls for leaders (pause price change, set start dates, agree reporting) from what the team is handling itself (invoice fixes, pipeline restatement, backfill).

Gemini 3.5 Flash-Lite · Gemini

It asks for the pause decision with a recommendation and a date, and separates it from the operational actions the team is taking.

Reports outcomes and learning, not just outputRightRightRight
Gemini 4 Argon · API

Leads with outcome metrics (ARR, pipeline, WAT, support SLA), explains what changed (methodology, pricing impact), and uses shipping details as supporting actions.

Sonnet 5.5 · API

Leads with outcome metrics (ARR, active teams, SLA, pipeline) and what the team learned (pipeline redefinition, invoice ticket surge), not just activities.

Gemini 3.5 Flash-Lite · Gemini

It leads with outcome metrics (ARR, pipeline, support) and explains what changed (support broke after pricing change), showing learning.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 71% of 118 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstraChatGPT96.392.92None
2GPT-6.1 SolAPI73.396.22None
3Haiku 5.5API79.292.32None
4Sonnet 5.5API68.896.42None
5Opus 5.5Claude84.277.52None
6GPT-6 LunaAPI71.366.82None
7Gemini 4 ArgonAPI70.063.22None
8Gemini 3.5 Flash-LiteGemini73.851.62None
9Gemini 3.8 FlashAPI71.751.92None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.50607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 95, 26× the cheapest (1 of 2 outputs estimated)GPT-6.1 Sol · API: task score 85, 15× the cheapestHaiku 5.5 · API: task score 84, 2.6× the cheapestSonnet 5.5 · API: task score 82, 40× the cheapestOpus 5.5 · Claude: task score 80, 55× the cheapest (1 of 2 outputs estimated)GPT-6 Luna · API: task score 69, the cheapestGemini 4 Argon · API: task score 65, 15× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 61, the cheapest (1 of 2 outputs estimated)Gemini 3.8 Flash · API: task score 61, 13× the cheapestGPT-6 Astra · ChatGPTGPT-6.1 Sol · APIHaiku 5.5 · APISonnet 5.5 · APIOpus 5.5 · ClaudeGPT-6 Luna · APIGemini 4 Argon · APIGemini 3.5 Flash-Lite · GeminiGemini 3.8 Flash · API
Nobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Writing the weekly update, or the reply to an executive's question, that the team's credibility rests on.

Why it matters

Updates are where bad news gets softened. A fluent update that buries a slipped date or presents a guess as the cause does more harm than no update: leadership decides on it, and trust goes when the truth comes out.

What good looks like

  • Leads with the news and any decision needed
  • States dates, numbers and causes plainly, including bad news
  • Separates what is known from what is suspected
  • Brings a recommendation and what's been tried, not just the problem
  • Fits the reader and the length asked for

Deliberately not measured

  • Tone and formatting preferences
  • Writing style beyond clarity
Capability tested

Honest, decision-first status communication

The failure we’re looking for

Spin: burying or softening the bad news, or presenting a guess as fact

Grading

Decision model, LLM judge and a browser check (for the deck), calibrated against a blind PM review

Variants

A short written update · Staff level: a QBR deck in brand colours