Tasks / Leadership

Write a stakeholder update

Can the model turn messy project status into an honest update that leads with what the reader needs to know or decide?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 50% were usable with at most a quick edit.

Reliably right

Reports outcomes and learning, not just output100% pass
It leads with outcome metrics (ARR, support, usage) and reports a key learning (pipeline jump is a definition change, not demand growth), not just activities.
GPT-6 Astra · ChatGPT · On track, mostly
Leads with the honest headline100% pass
First slide headline includes the retention drop alongside the new-ARR beat, not just 'record quarter'.
GPT-6 Astra · ChatGPT · A quarterly business review deck, in brand
A real 7-slide deck100% pass

Where it slips

Uses the supplied evidence correctly51% pass
The year 2026 is not in the supplied context; all other factual claims are supported.
GPT-6 Astra · ChatGPT · A quarterly business review deck, in brand
Owns the problem53% pass
Does not state what the team has already done about the payments reliability miss or the NPS drop; only reports the problems.
GPT-6 Astra · ChatGPT · A quarterly business review deck, in brand
Flags the redefined pipeline58% pass
It names the redefinition but does not give the falling like-for-like trend (e.g., pipeline fell from $2.41M to $2.23M on the old definition).
GPT-6.1 Sol · API · On track, mostly

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Asks leaders only for the big calls
Does the update separate the decisions it needs from its readers (hard to reverse, high stakes) from the ones the team will make itself, and ask clearly for the former?
Passes when Names the decisions it needs from the readers, with a recommendation and a date, and says which calls the team is making itself.
Geoff CharlesLenny’s Podcast
Reports outcomes and learning, not just output
Does the update report progress as movement on the outcomes the team is accountable for and what it has learned, rather than as features shipped or dates hit?
Passes when Leads with the outcome metrics and what changed in the team's understanding, with shipping as supporting detail.
Melissa PerriLenny’s Podcast Christian IdiodiLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Owns the problem · Ready for the reader as delivered · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Head of Product at Cobalt. Write this fortnight's update for the CEO and the exec team, in no more than 450 words, from the weekly metrics and the team notes below. It should say where we stand, what's wrong, what we're doing about it, and what you need from them.

What the model was given3 items: About Cobalt, weekly_metrics_weeks_1_to_8.csv, team_notes.md

About CobaltTeam scheduling software, sold per seat.

weekly_metrics_weeks_1_to_8.csv8 rows · Download

week,new_arr_signed_usd,qualified_pipeline_usd,weekly_active_teams,support_backlog_tickets,support_sla_met_pct
1,182000,2410000,8210,140,96
2,205000,2380000,8260,152,95
3,168000,2350000,8240,171,94
4,214000,2290000,8300,198,91
5,199000,2230000,8280,260,86
6,221000,3140000,8310,344,79
…

team_notes.md7 lines · Download

# Notes from the teams

- Sales ops: from week 6, "qualified pipeline" includes active free trials over 10 seats (they used to be excluded).
- Data: weekly active teams for week 7 is missing because of a pipeline outage; it's being backfilled.
- Support: the backlog grew after the September pricing change (tickets about invoices doubled). SLA target is 90% answered within one business day.
- Quarter target: $2.4M new ARR in 12 weeks ($200,000 a week). Weeks 1 to 8 total: see the file.
- Support hiring: two agents approved, start date not set. Decision needed: whether to pause the planned price change for existing customers in week 10 until the backlog is cleared.

What a strong answer doesThe answer key the graders mark against

Leads with the takeaway: new ARR is ahead of plan ($1.67M in eight weeks against $1.6M), but support is breaking and pipeline is weaker than it looks. Flags that the jump in qualified pipeline in week 6 (from about $2.23M to $3.14M) is the redefinition to include large trials; on the old definition pipeline fell every week from $2.41M to $2.23M, a risk for the rest of the quarter. Reports the support problem plainly: backlog from 140 to 455 tickets and SLA from 96% to 72% against a 90% target since the September pricing change, with what's being done (two agents approved). Notes week 7's active teams are missing and being backfilled rather than hiding it. Asks for the decisions that need the execs: pause the week-10 price change for existing customers until the backlog clears, and set the start date for the support hires. Separates what the team is handling from what needs them.

Critical failures (cap the score at 40)

Presents qualified pipeline as having grown about 40%

Task

Staff-level · v1.1

The outputs

9 setups answered this brief.

GPT-6 AstraChatGPT98Usable with a quick edit Sonnet 5.5API94Usable with a quick edit GPT-6.1 SolAPI88Usable with a quick edit Haiku 5.5API85Needs more work Opus 5.5Claude77Needs more work GPT-6 LunaAPI75Usable with a quick edit Gemini 4 ArgonAPI73Needs more work Gemini 3.8 FlashAPI65Needs more work Gemini 3.5 Flash-LiteGemini56Needs more work

Gemini 4 ArgonAPI

Needs more than a quick edit9 of 12 checks passed

The output contains an unsupported claim about product and support auditing invoice layouts that is not in the source material. A PM would need to verify or remove it before sending.

Output

Subject: Product Fortnightly Update – Weeks 7 & 8

Where We Stand * New ARR: We are tracking ahead of our Q3 target ($2.4M over 12 weeks, or $200k/week). Through Week 8, we have signed $1,673,000 in new ARR (averaging ~$209k/week). Momentum is strong, with Weeks 7 and 8 delivering $236k and $248k, respectively. * Pipeline: Qualified pipeline sits at $3.35M. Note on the data: The ~40% jump starting in Week 6 ($2.23M to $3.14M) reflects a Sales Ops methodology change to include active free trials with >10 seats, rather than a sudden spike in traditional pipeline. * Active Usage: Weekly Active Teams (WAT) dipped to 8,190 in Week 8—our lowest point this quarter (down from 8,310 in Week 6). Week 7 data is currently missing due to a data pipeline outage.

What’s Wrong * Support is underwater: Following the September pricing change, invoice-related tickets doubled. Over eight weeks, our support backlog has more than tripled from 140 to 455 tickets. Consequently, our SLA compliance (target: 90% answered within one business day) has fallen below target since Week 5, hitting 72% in Week 8. * Early churn/engagement risk: While one week isn't a trend, the Week 8 drop in WAT—despite eight weeks of healthy new ARR additions—is concerning. We lacked visibility in Week 7 due to the data outage, masking when this dip actually began.

What We’re Doing About It * Fixing the data blindspot: The Data team is backfilling the Week 7 WAT numbers so we can properly diagnose the Week 8 usage drop and see if it correlates with customers affected by the billing confusion. * Reducing ticket volume at the source: Product is partnering with Support to audit the new invoice layout and billing UX, aiming to ship clearer in-app explanations and self-serve billing FAQs this sprint to deflect incoming tickets. * Expanding capacity: Two new support agents are approved to help clear the backlog, though start dates are not yet finalized.

What I Need From You 1. Decision – Pause the Week 10 legacy price migration: We are scheduled to roll out the price change to existing customers in Week 10. Given that our per-seat billing changes caused invoice tickets to double, rolling this out to our entire install base while SLA sits at 72% will break Support and risk churn. I recommend we pause the Week 10 rollout until the backlog is cleared and the new agents are ramped. Please confirm by Friday if we have approval to hold. 2. Expedite Support Onboarding: I need People/Ops to help us lock in immediate start dates for the two approved support hires.

Claims the judge couldn’t find in the brief

Product is partnering with Support to audit the new invoice layout and billing UX, aiming to ship clearer in-app explanations and self-serve billing FAQs this sprint.

Grades and run details

Decision model 83 · LLM judge 69

Decision model checks

failUses the supplied evidence correctly59%
passAddresses the actual decision27%
passRespects explicit constraints49%
passIdentifies material uncertainty92%
passAvoids unsupported claims23%
passProduces the required deliverable94%
partialFlags the redefined pipeline92%
passReports the support breach plainly100%
partialOwns the problem27%
passReady for the reader as delivered35%
passAsks leaders only for the big calls56%
passReports outcomes and learning, not just output97%

Run

Run: #1
API response time: 7 s
Submitted: 13 Oct 2026

Haiku 5.5API

Needs more than a quick edit11 of 12 checks passed

The output includes an unsupported statement about Product reviewing invoice categories, which would require verification before sending—more than a quick edit.

Output

Subject: Product update, weeks 1 to 8

Where we stand - New ARR: $1.67M signed in weeks 1 to 8, $73K ahead of the $1.6M pace ($200K/week). The last four weeks averaged $226K, and weekly sales have risen since week 5. - Quarter target: To reach $2.4M by week 12, we need $727K over weeks 9 to 12, about $182K a week. At the recent run rate we'd finish near $2.6M, if the trend holds. - Active teams: Flat at 8,190 to 8,310. Week 7 is missing because of a pipeline outage, and Data is backfilling it.

What's wrong - Support is the main problem. The backlog rose from 140 to 455 tickets, and SLA attainment fell from 96% to 72%. We've been below our 90% target since week 5. Invoice tickets doubled after the September pricing change. - The pipeline jump is not growth. Qualified pipeline rose from $2.23M to $3.14M in week 6 because the definition changed: active free trials over 10 seats now count. Weeks 6 to 8 aren't comparable with weeks 1 to 5. On the old definition, pipeline fell from $2.41M to $2.23M over weeks 1 to 5. Until we restate the history, I won't treat the jump as a gain.

What we're doing - Support hiring: Both approved agents still have no start date. I'm working with People to set dates this week. - Invoice tickets: Product is reviewing the invoice ticket categories to find the causes we can fix in the product rather than answer one by one. - Pipeline reporting: I've asked Sales Ops and Finance to restate weeks 1 to 8 on one definition so we can see the real trend.

What I need from you 1. Decision on the week 10 price change for existing customers. I recommend pausing it until SLA is back above 90%. The change would add invoice tickets during our worst support period, and invoices are already the main source of the backlog. I need this decision by the end of next week. 2. Fallback support capacity. If the agents don't start by week 10, we need budget for temporary contract support. I'll size the cost with Support and bring it back to you. 3. Restated pipeline numbers. Please support the restatement so the board sees one consistent series.

Happy to take questions before Thursday's exec meeting.

Head of Product, Cobalt

Claims the judge couldn’t find in the brief

Product is reviewing the invoice ticket categories to find causes fixable in the product.
I've asked Sales Ops and Finance to restate weeks 1 to 8 on one definition.

Grades and run details

Decision model 92 · LLM judge 85

Decision model checks

failUses the supplied evidence correctly66%
passAddresses the actual decision77%
passRespects explicit constraints56%
passIdentifies material uncertainty100%
passAvoids unsupported claims31%
passProduces the required deliverable94%
passFlags the redefined pipeline100%
passReports the support breach plainly100%
passOwns the problem66%
passReady for the reader as delivered53%
passAsks leaders only for the big calls55%
passReports outcomes and learning, not just output98%

Run

Run: #1
API response time: 13 s
Submitted: 8 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 2

Avoids unsupported claimsMixedRight

Gemini 4 Argon · API

Presents the product/support audit and FAQ work as an established action, while the evidence contains no mention of it.

Haiku 5.5 · API

Interpretations like the pipeline jump being a definition change and invoice doubling are directly supported by the notes; no causes or forecasts are presented as established fact without evidence.

Flags the redefined pipelineWrongRight

Gemini 4 Argon · API

Flags the redefinition but does not give the falling like-for-like trend on the old definition (pipeline declined from $2.41M to $2.23M).

Haiku 5.5 · API

It explicitly names the redefinition, gives the like-for-like falling trend ($2.41M to $2.23M), and explains the jump is not growth.

All got wrong 1

Uses the supplied evidence correctlyWrongWrong

Gemini 4 Argon · API

The output claims Product is auditing invoice layouts and shipping FAQs this sprint, an action not in the supplied context.

Haiku 5.5 · API

The output states Product is reviewing invoice categories and that Sales Ops/Finance have been asked to restate pipeline—these current actions are not in the supplied context, violating the evidence rule.

All got right 9

Addresses the actual decisionRightRight

Gemini 4 Argon · API

Clearly recommends pausing the Week 10 price change and expediting support hiring, with a deadline for approval.

Haiku 5.5 · API

It recommends pausing the week-10 price change until SLA is above 90%, gives a deadline, and names what would change the call.

Respects explicit constraintsRightRight

Gemini 4 Argon · API

The update is well within 450 words and uses the requested fortnightly memo form for the exec team.

Haiku 5.5 · API

The update is within 450 words, addressed to the CEO/exec team, and covers standing, problems, actions and asks.

Identifies material uncertaintyRightRight

Gemini 4 Argon · API

Acknowledges the WAT dip isn't a trend, states conditions for unpausing (backlog cleared, agents ramped), and flags the missing data and unconfirmed start dates.

Haiku 5.5 · API

It identifies agent start-date uncertainty and pipeline comparability, links them to decisions, and says how they would be resolved.

Produces the required deliverableRightRight

Gemini 4 Argon · API

The memo is complete, covers standing, problems, actions, and asks, and the exec team can act on it with light edits.

Haiku 5.5 · API

The memo is complete, properly formatted for the exec team, within the word limit, and provides actionable direction.

Reports the support breach plainlyRightRight

Gemini 4 Argon · API

Reports the backlog and SLA breach with numbers, ties them to the pricing change, and explicitly asks for the pause decision.

Haiku 5.5 · API

Backlog 140→455, SLA 96%→72% against 90% target, cause (Sept pricing change, invoice tickets doubled) and the pause decision are all reported plainly.

Owns the problemRightRight

Gemini 4 Argon · API

For support and WAT drop, the update names actions being taken (data backfill, hiring, product work) and recommends a way forward.

Haiku 5.5 · API

For support and pipeline problems it lists actions taken (agent hiring, People coordination, restatement request) and recommends a way forward.

Ready for the reader as deliveredRightRight

Gemini 4 Argon · API

Written directly for the CEO/exec team; no placeholders, internal notes, or unexplained jargon from the brief.

Haiku 5.5 · API

The language is self-contained, no placeholders, and every owner/date is either from the material or clearly proposed.

Asks leaders only for the big callsRightRight

Gemini 4 Argon · API

Separates the big calls needing exec input (pause price change, expedite hires) from team actions, with a recommendation and timeline.

Haiku 5.5 · API

It separates big calls needing execs (pause price change, fallback budget, restatement support) from team actions, with recommendations and deadlines.

Reports outcomes and learning, not just outputRightRight

Gemini 4 Argon · API

Leads with outcome metrics (ARR, pipeline, WAT, support SLA), explains what changed (methodology, pricing impact), and uses shipping details as supporting actions.

Haiku 5.5 · API

It leads with outcome metrics (ARR, active teams), reports the learning that pipeline jump is a definition change, and links support issues to the pricing change.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 71% of 118 checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	GPT-6 AstrawithChatGPT	96.3	92.9	2	None
2	GPT-6.1 SolwithAPI	73.3	96.2	2	None
3	Haiku 5.5withAPI	79.2	92.3	2	None
4	Sonnet 5.5withAPI	68.8	96.4	2	None
5	Opus 5.5withClaude	84.2	77.5	2	None
6	GPT-6 LunawithAPI	71.3	66.8	2	None
7	Gemini 4 ArgonwithAPI	70.0	63.2	2	None
8	Gemini 3.5 Flash-LitewithGemini	73.8	51.6	2	None
9	Gemini 3.8 FlashwithAPI	71.7	51.9	2	None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Nobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
Gemini 3.5 Flash-Lite · Gemini	61	1.0× (partly estimated)
GPT-6 Luna · API	69	1.3×
Haiku 5.5 · API	84	2.6×
Gemini 3.8 Flash · API	61	13×
Gemini 4 Argon · API	65	15×
GPT-6.1 Sol · API	85	15×
GPT-6 Astra · ChatGPT	95	26× (partly estimated)
Sonnet 5.5 · API	82	40×
Opus 5.5 · Claude	80	55× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Writing the weekly update, or the reply to an executive's question, that the team's credibility rests on.

Why it matters

Updates are where bad news gets softened. A fluent update that buries a slipped date or presents a guess as the cause does more harm than no update: leadership decides on it, and trust goes when the truth comes out.

What good looks like

Leads with the news and any decision needed
States dates, numbers and causes plainly, including bad news
Separates what is known from what is suspected
Brings a recommendation and what's been tried, not just the problem
Fits the reader and the length asked for

Deliberately not measured

Tone and formatting preferences
Writing style beyond clarity

Capability tested

Honest, decision-first status communication

The failure we’re looking for

Spin: burying or softening the bad news, or presenting a guess as fact

Grading

Decision model, LLM judge and a browser check (for the deck), calibrated against a blind PM review

Variants

A short written update · Staff level: a QBR deck in brand colours

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

Asks leaders only for the big calls

Reports outcomes and learning, not just output

Plus our standard checks

The tasks

The outputs

Output

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Run

Output

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Run

Check by check

Where they differ 2

All got wrong 1

All got right 9

Results

What you get for the cost

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured