Humanity’s Last PRD
Latest drop: Haiku 5.5 · 8 Oct
Tasks / Leadership

Write a stakeholder update

Can the model turn messy project status into an honest update that leads with what the reader needs to know or decide?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 56% were usable with at most a quick edit.

Reliably right

  1. Reports outcomes and learning, not just output100% pass
    Leads with outcome metrics (ARR, active teams, SLA, pipeline) and what the team learned (pipeline redefinition, invoice ticket surge), not just activities.
    Sonnet 5.5 · API · On track, mostly
  2. Leads with the honest headline100% pass
    First slide headline includes the retention drop alongside the new-ARR beat, not just 'record quarter'.
    GPT-6 Astra · ChatGPT · A quarterly business review deck, in brand
  3. A real 7-slide deck100% pass

Where it slips

  1. Owns the problem55% pass
    Does not state what the team has already done about the payments reliability miss or the NPS drop; only reports the problems.
    GPT-6 Astra · ChatGPT · A quarterly business review deck, in brand
  2. Identifies material uncertainty61% pass
    The output identifies the overlap uncertainty for decision A but does not name any material unknown that could change the call for decision B, such as the reliability of the Payments lead's estimate or the impact of pausing activation experiments.
    Opus 5.5 · Claude · A quarterly business review deck, in brand
  3. Flags the redefined pipeline63% pass
    It names the redefinition but does not give the falling like-for-like trend (e.g., pipeline fell from $2.41M to $2.23M on the old definition).
    GPT-6.1 Sol · API · On track, mostly

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Asks leaders only for the big calls

    Does the update separate the decisions it needs from its readers (hard to reverse, high stakes) from the ones the team will make itself, and ask clearly for the former?

    Passes when Names the decisions it needs from the readers, with a recommendation and a date, and says which calls the team is making itself.

    Geoff CharlesLenny’s Podcast

  2. Reports outcomes and learning, not just output

    Does the update report progress as movement on the outcomes the team is accountable for and what it has learned, rather than as features shipped or dates hit?

    Passes when Leads with the outcome metrics and what changed in the team's understanding, with shipping as supporting detail.

    Melissa PerriLenny’s PodcastChristian IdiodiLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Owns the problem · Ready for the reader as delivered · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Head of Product at Cobalt. Write this fortnight's update for the CEO and the exec team, in no more than 450 words, from the weekly metrics and the team notes below. It should say where we stand, what's wrong, what we're doing about it, and what you need from them.

What the model was given3 items: About Cobalt, weekly_metrics_weeks_1_to_8.csv, team_notes.md
About CobaltTeam scheduling software, sold per seat.
weekly_metrics_weeks_1_to_8.csv8 rows · Download
week,new_arr_signed_usd,qualified_pipeline_usd,weekly_active_teams,support_backlog_tickets,support_sla_met_pct
1,182000,2410000,8210,140,96
2,205000,2380000,8260,152,95
3,168000,2350000,8240,171,94
4,214000,2290000,8300,198,91
5,199000,2230000,8280,260,86
6,221000,3140000,8310,344,79
…
team_notes.md7 lines · Download
# Notes from the teams

- Sales ops: from week 6, "qualified pipeline" includes active free trials over 10 seats (they used to be excluded).
- Data: weekly active teams for week 7 is missing because of a pipeline outage; it's being backfilled.
- Support: the backlog grew after the September pricing change (tickets about invoices doubled). SLA target is 90% answered within one business day.
- Quarter target: $2.4M new ARR in 12 weeks ($200,000 a week). Weeks 1 to 8 total: see the file.
- Support hiring: two agents approved, start date not set. Decision needed: whether to pause the planned price change for existing customers in week 10 until the backlog is cleared.
What a strong answer doesThe answer key the graders mark against

Leads with the takeaway: new ARR is ahead of plan ($1.67M in eight weeks against $1.6M), but support is breaking and pipeline is weaker than it looks. Flags that the jump in qualified pipeline in week 6 (from about $2.23M to $3.14M) is the redefinition to include large trials; on the old definition pipeline fell every week from $2.41M to $2.23M, a risk for the rest of the quarter. Reports the support problem plainly: backlog from 140 to 455 tickets and SLA from 96% to 72% against a 90% target since the September pricing change, with what's being done (two agents approved). Notes week 7's active teams are missing and being backfilled rather than hiding it. Asks for the decisions that need the execs: pause the week-10 price change for existing customers until the backlog clears, and set the start date for the support hires. Separates what the team is handling from what needs them.

Critical failures (cap the score at 40)
  • Presents qualified pipeline as having grown about 40%
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Sonnet 5.5API98Usable with a quick editGPT-6 AstraChatGPT96Usable with a quick editGPT-6.1 SolAPI92Usable with a quick editHaiku 5.5API87Needs more workOpus 5.5Claude79Needs more workGPT-6 LunaAPI77Usable with a quick editGemini 3.8 FlashAPI63Needs more workGemini 3.5 Flash-LiteGemini58Needs more work
Opus 5.5Claude

Needs more than a quick edit9 of 12 checks passed

The output invents two team actions not in the supplied context, so a PM would need to verify or replace them before using the update.

Output

Fortnightly Product Update: Weeks 7–8

Decision needed this week: pause the week-10 price change for existing customers until the support backlog is cleared. I recommend we pause.

Where we stand

  • New ARR is ahead of plan. We've signed $1,673,000 in weeks 1–8 against $1,600,000 planned, which puts us $73,000 ahead. To reach $2.4M we need $727,000 over the last four weeks, or about $182,000 a week. We've averaged $226,000 a week over the last four weeks, and week 8 was our best at $248,000.
  • Usage is flat. Weekly active teams have held at 8,190–8,310 all quarter. Week 8 (8,190) is the lowest so far, down 120 from week 6. That's small (about 1.5%), but it's the direction to watch. Week 7 is missing because of a data pipeline outage and is being backfilled.

What's wrong

1. Support is failing its SLA, and the gap is widening. - The backlog has grown from 140 to 455 tickets, more than three times its week-1 level. - SLA has dropped from 96% to 72% against a 90% target. We've been below target since week 5. - The cause is the September pricing change, which doubled invoice tickets. - Backlog growth is slowing (+84, +68, +43 in weeks 6–8), but it is still growing.

2. The pipeline number overstates our position. - The jump from $2.23M to $3.14M in week 6 is mainly a definition change. Active free trials over 10 seats are now counted as qualified. - On the old definition, pipeline fell 7% in weeks 1–5, from $2.41M to $2.23M. - We don't yet know whether it has recovered. Next quarter is the real risk, not this one.

What we're doing

  • Support: two agents are approved, but neither has a start date. I'm pushing to get them in before week 10. Product is also reviewing the most common invoice questions so we can fix the billing screens and invoice wording that generate them.
  • Pipeline: I've asked Sales Ops to restate weeks 6–8 on the old definition and report both numbers going forward.
  • Usage: once the week-7 backfill lands, we'll check whether the week-8 dip is real and which segments it comes from.

What I need from you

  1. Decide whether to pause the week-10 price change. I recommend pausing. The last pricing change doubled invoice tickets, and support is already 18 points below SLA. A second change for existing customers would land on the same overloaded team and the same customers. I suggest we resume once SLA is back above 90% for two consecutive weeks. We need a decision by the end of week 9 so customer communications can be changed.
  2. Set start dates for the two approved support agents this week.
  3. Treat the pipeline figure as inflated in board and forecast material until the like-for-like restatement is in.

Claims the judge couldn’t find in the brief

  • Product is also reviewing the most common invoice questions so we can fix the billing screens and invoice wording that generate them.
  • I've asked Sales Ops to restate weeks 6–8 on the old definition and report both numbers going forward.

Grades and run details

Decision model 96 · LLM judge 69
Decision model checks
  • passUses the supplied evidence correctly42%
  • passAddresses the actual decision98%
  • passRespects explicit constraints27%
  • passIdentifies material uncertainty80%
  • partialAvoids unsupported claims35%
  • passProduces the required deliverable86%
  • passFlags the redefined pipeline100%
  • passReports the support breach plainly100%
  • passOwns the problem68%
  • passReady for the reader as delivered77%
  • passAsks leaders only for the big calls85%
  • passReports outcomes and learning, not just output99%
Run
Run
#1
Time to output
17 s
Submitted
5 Oct 2026
Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit6 of 12 checks passed

The ARR total is wrong ($1.73M instead of $1.673M), a factual error that would require re-checking the evidence before use.

Output

To: CEO & Exec Team From: Head of Product, Cobalt Subject: Fortnightly Update (Weeks 7–8)

Here is where we stand for the last two weeks, the critical issues we face, our action plan, and where I need your input.

Where We Stand

We are currently outperforming our top-line targets. We’ve brought in $1.73M in new ARR across 8 weeks against our $1.6M pro-rata goal ($200k/week), closing Week 8 at a strong $248k. Pipeline looks healthy on paper, jumping to $3.35M. However, caveat emptor: Sales Ops adjusted the pipeline definition in Week 6 to include active free trials over 10 seats, masking underlying organic pipeline velocity. Furthermore, weekly active teams dipped to 8,190 in Week 8 (Week 7 data is currently missing due to a data pipeline outage, currently being backfilled).

What’s Wrong

Customer Support is in a critical state. Our support backlog has exploded to 455 tickets, and SLA compliance has plummeted to 72% (target: 90%).

This was triggered by the September pricing change. Invoicing-related tickets have doubled, and our current support bandwidth is completely overwhelmed. While two new hires have been approved, we do not yet have confirmed start dates.

What We’re Doing About It

  1. Stabilizing Support: We are exploring short-term triage options, including engineering-led documentation updates to deflect self-service billing questions.
  2. Data Recovery: Data engineering is actively backfilling the Week 7 active teams metric to restore full visibility.
  3. Pipeline Audit: Sales Ops is recalibrating the pipeline reporting to separate true enterprise/mid-market opportunities from the newly included free trials so we can accurately forecast Q3 close rates.

What I Need From You

We face an immediate strategic decision regarding Week 10:

  • Decision Needed: Should we pause the planned price change for existing customers scheduled for Week 10 until the current support backlog is cleared and SLAs are back above 90%?

Proceeding with the existing customer rollout while our SLA is at 72% risks a wave of churn and avoidable escalation tickets. I recommend pausing, but I need a final call from the exec team so Product and Support can communicate the adjustment internally and externally.

Claims the judge couldn’t find in the brief

  • We’ve brought in $1.73M in new ARR across 8 weeks

Grades and run details

Decision model 75 · LLM judge 46
Decision model checks
  • partialUses the supplied evidence correctly21%
  • partialAddresses the actual decision49%
  • passRespects explicit constraints24%
  • partialIdentifies material uncertainty68%
  • partialAvoids unsupported claims52%
  • passProduces the required deliverable51%
  • partialFlags the redefined pipeline97%
  • passReports the support breach plainly100%
  • partialOwns the problem52%
  • passReady for the reader as delivered37%
  • passAsks leaders only for the big calls54%
  • passReports outcomes and learning, not just output92%
Run
Run
#1
Time to output
3 s
Submitted
5 Oct 2026
Haiku 5.5API

Needs more than a quick edit11 of 12 checks passed

The output includes an unsupported statement about Product reviewing invoice categories, which would require verification before sending—more than a quick edit.

Output

Subject: Product update, weeks 1 to 8

Where we stand - New ARR: $1.67M signed in weeks 1 to 8, $73K ahead of the $1.6M pace ($200K/week). The last four weeks averaged $226K, and weekly sales have risen since week 5. - Quarter target: To reach $2.4M by week 12, we need $727K over weeks 9 to 12, about $182K a week. At the recent run rate we'd finish near $2.6M, if the trend holds. - Active teams: Flat at 8,190 to 8,310. Week 7 is missing because of a pipeline outage, and Data is backfilling it.

What's wrong - Support is the main problem. The backlog rose from 140 to 455 tickets, and SLA attainment fell from 96% to 72%. We've been below our 90% target since week 5. Invoice tickets doubled after the September pricing change. - The pipeline jump is not growth. Qualified pipeline rose from $2.23M to $3.14M in week 6 because the definition changed: active free trials over 10 seats now count. Weeks 6 to 8 aren't comparable with weeks 1 to 5. On the old definition, pipeline fell from $2.41M to $2.23M over weeks 1 to 5. Until we restate the history, I won't treat the jump as a gain.

What we're doing - Support hiring: Both approved agents still have no start date. I'm working with People to set dates this week. - Invoice tickets: Product is reviewing the invoice ticket categories to find the causes we can fix in the product rather than answer one by one. - Pipeline reporting: I've asked Sales Ops and Finance to restate weeks 1 to 8 on one definition so we can see the real trend.

What I need from you 1. Decision on the week 10 price change for existing customers. I recommend pausing it until SLA is back above 90%. The change would add invoice tickets during our worst support period, and invoices are already the main source of the backlog. I need this decision by the end of next week. 2. Fallback support capacity. If the agents don't start by week 10, we need budget for temporary contract support. I'll size the cost with Support and bring it back to you. 3. Restated pipeline numbers. Please support the restatement so the board sees one consistent series.

Happy to take questions before Thursday's exec meeting.

Head of Product, Cobalt

Claims the judge couldn’t find in the brief

  • Product is reviewing the invoice ticket categories to find causes fixable in the product.
  • I've asked Sales Ops and Finance to restate weeks 1 to 8 on one definition.

Grades and run details

Decision model 96 · LLM judge 85
Decision model checks
  • passUses the supplied evidence correctly41%
  • passAddresses the actual decision76%
  • passRespects explicit constraints49%
  • passIdentifies material uncertainty73%
  • partialAvoids unsupported claims54%
  • passProduces the required deliverable97%
  • passFlags the redefined pipeline100%
  • passReports the support breach plainly100%
  • passOwns the problem62%
  • passReady for the reader as delivered48%
  • passAsks leaders only for the big calls82%
  • passReports outcomes and learning, not just output100%
Run
Run
#1
API response time
13 s
Submitted
8 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 7

Uses the supplied evidence correctlyMixedWrongMixed
Opus 5.5 · Claude

Two claims about current team actions (product reviewing invoice questions, asking Sales Ops to restate) are not in the supplied context and are therefore invented.

Gemini 3.5 Flash-Lite · Gemini

The total new ARR is stated as $1.73M, but the sum of weeks 1-8 is $1.673M; this factual error means the output does not use the supplied evidence correctly.

Haiku 5.5 · API

The output states Product is reviewing invoice categories and that Sales Ops/Finance have been asked to restate pipeline—these current actions are not in the supplied context, violating the evidence rule.

Addresses the actual decisionRightWrongRight
Opus 5.5 · Claude

Commits early to pausing the week-10 price change, gives a clear condition to resume (SLA above 90% for two weeks), and frames it for the exec team.

Gemini 3.5 Flash-Lite · Gemini

The output recommends pausing the price change but does not state what result or condition would change that recommendation, as required.

Haiku 5.5 · API

It recommends pausing the week-10 price change until SLA is above 90%, gives a deadline, and names what would change the call.

Identifies material uncertaintyRightWrongRight
Opus 5.5 · Claude

Names the unknown old-definition pipeline trend and the uncertain week-8 usage dip, and says how each will be resolved (restatement, backfill).

Gemini 3.5 Flash-Lite · Gemini

It does not name the specific unknowns that could change the decision about the price change, nor say how they would be resolved or what would change the call.

Haiku 5.5 · API

It identifies agent start-date uncertainty and pipeline comparability, links them to decisions, and says how they would be resolved.

Avoids unsupported claimsWrongWrongRight
Opus 5.5 · Claude

Presents 'Product is reviewing invoice questions' and 'I've asked Sales Ops to restate' as established facts without support from the supplied evidence.

Gemini 3.5 Flash-Lite · Gemini

The $1.73M ARR figure is presented as fact but is not supported by the data; it is a confident claim that the evidence does not establish.

Haiku 5.5 · API

Interpretations like the pipeline jump being a definition change and invoice doubling are directly supported by the notes; no causes or forecasts are presented as established fact without evidence.

Produces the required deliverableRightMixedRight
Opus 5.5 · Claude

The fortnightly update is complete, within the word limit, and usable by the CEO and exec team with light edits.

Gemini 3.5 Flash-Lite · Gemini

The update contains a material factual error in the ARR total, so the reader cannot act on it without correcting that number.

Haiku 5.5 · API

The memo is complete, properly formatted for the exec team, within the word limit, and provides actionable direction.

Flags the redefined pipelineRightWrongRight
Opus 5.5 · Claude

Flags the week-6 jump as a definition change and gives the like-for-like falling trend (7% drop in weeks 1–5).

Gemini 3.5 Flash-Lite · Gemini

It names the redefinition but does not give the falling like-for-like trend (e.g., pipeline on the old definition fell from $2.41M to $2.23M).

Haiku 5.5 · API

It explicitly names the redefinition, gives the like-for-like falling trend ($2.41M to $2.23M), and explains the jump is not growth.

Owns the problemMixedRightRight
Opus 5.5 · Claude

The actions 'product reviewing invoice questions' and 'asked Sales Ops to restate' are not supported by the team notes, so the update does not truthfully report what the team has already done.

Gemini 3.5 Flash-Lite · Gemini

For support it notes the approved hires and recommends pausing the price change; for data and pipeline it states the actions being taken.

Haiku 5.5 · API

For support and pipeline problems it lists actions taken (agent hiring, People coordination, restatement request) and recommends a way forward.

All got right 5

Respects explicit constraintsRightRightRight
Opus 5.5 · Claude

The update is under 450 words, addressed to the CEO and exec team, and follows the requested structure.

Gemini 3.5 Flash-Lite · Gemini

The update is under 450 words and addressed to the CEO and exec team as requested.

Haiku 5.5 · API

The update is within 450 words, addressed to the CEO/exec team, and covers standing, problems, actions and asks.

Reports the support breach plainlyRightRightRight
Opus 5.5 · Claude

Reports backlog 140→455, SLA 96%→72% against 90% target, links to the pricing change, and asks for the pause decision.

Gemini 3.5 Flash-Lite · Gemini

It reports the backlog (455) and SLA (72% vs 90% target), links them to the pricing change, and asks for the pause decision.

Haiku 5.5 · API

Backlog 140→455, SLA 96%→72% against 90% target, cause (Sept pricing change, invoice tickets doubled) and the pause decision are all reported plainly.

Ready for the reader as deliveredRightRightRight
Opus 5.5 · Claude

Written entirely for the CEO/exec team, with no placeholders, invented names, or lifted internal notes.

Gemini 3.5 Flash-Lite · Gemini

The update is written for the exec team, uses no unexplained jargon from the brief, and all owners/dates are from the material or clearly proposed.

Haiku 5.5 · API

The language is self-contained, no placeholders, and every owner/date is either from the material or clearly proposed.

Asks leaders only for the big callsRightRightRight
Opus 5.5 · Claude

Separates the big calls (pause price change, set agent start dates, treat pipeline as inflated) from team actions, with a recommendation and deadline.

Gemini 3.5 Flash-Lite · Gemini

It asks for the pause decision with a recommendation and a date, and separates it from the operational actions the team is taking.

Haiku 5.5 · API

It separates big calls needing execs (pause price change, fallback budget, restatement support) from team actions, with recommendations and deadlines.

Reports outcomes and learning, not just outputRightRightRight
Opus 5.5 · Claude

Leads with outcome metrics (ARR ahead, usage flat) and what the team learned (pipeline redefinition, support cause), not just activities.

Gemini 3.5 Flash-Lite · Gemini

It leads with outcome metrics (ARR, pipeline, support) and explains what changed (support broke after pricing change), showing learning.

Haiku 5.5 · API

It leads with outcome metrics (ARR, active teams), reports the learning that pipeline jump is a definition change, and links support issues to the pricing change.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 76% of 89 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstraChatGPT94.292.92None
2Sonnet 5.5API84.696.42None
3Haiku 5.5API89.692.32None
4GPT-6.1 SolAPI80.896.22None
5Opus 5.5Claude86.377.52None
6GPT-6 LunaAPI76.766.82None
7Gemini 3.5 Flash-LiteGemini70.851.62None
8Gemini 3.8 FlashAPI66.351.92None

What you get for the cost

Haiku 5.5 · API scores within 5 points of GPT-6 Astra · ChatGPT at a tenth of the cost.

Task score against relative cost for each setup. The table below has the same figures.50607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 94, 26× the cheapest (1 of 2 outputs estimated)Sonnet 5.5 · API: task score 90, 40× the cheapestHaiku 5.5 · API: task score 89, 2.6× the cheapestGPT-6.1 Sol · API: task score 88, 15× the cheapestOpus 5.5 · Claude: task score 81, 55× the cheapest (1 of 2 outputs estimated)GPT-6 Luna · API: task score 71, the cheapestGemini 3.5 Flash-Lite · Gemini: task score 60, the cheapest (1 of 2 outputs estimated)Gemini 3.8 Flash · API: task score 58, 13× the cheapestGPT-6 Astra · ChatGPTSonnet 5.5 · APIHaiku 5.5 · APIGPT-6.1 Sol · APIOpus 5.5 · ClaudeGPT-6 Luna · APIGemini 3.5 Flash-Lite · GeminiGemini 3.8 Flash · API
Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Writing the weekly update, or the reply to an executive's question, that the team's credibility rests on.

Why it matters

Updates are where bad news gets softened. A fluent update that buries a slipped date or presents a guess as the cause does more harm than no update: leadership decides on it, and trust goes when the truth comes out.

What good looks like

  • Leads with the news and any decision needed
  • States dates, numbers and causes plainly, including bad news
  • Separates what is known from what is suspected
  • Brings a recommendation and what's been tried, not just the problem
  • Fits the reader and the length asked for

Deliberately not measured

  • Tone and formatting preferences
  • Writing style beyond clarity
Capability tested

Honest, decision-first status communication

The failure we’re looking for

Spin: burying or softening the bad news, or presenting a guess as fact

Grading

Decision model, LLM judge and a browser check (for the deck), calibrated against a blind PM review

Variants

A short written update · Staff level: a QBR deck in brand colours