Tasks / Leadership

Write a stakeholder update

Can the model turn messy project status into an honest update that leads with what the reader needs to know or decide?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 56% were usable with at most a quick edit.

Reliably right

Reports outcomes and learning, not just output100% pass
Leads with outcome metrics (ARR, active teams, SLA, pipeline) and what the team learned (pipeline redefinition, invoice ticket surge), not just activities.
Sonnet 5.5 · API · On track, mostly
Leads with the honest headline100% pass
First slide headline includes the retention drop alongside the new-ARR beat, not just 'record quarter'.
GPT-6 Astra · ChatGPT · A quarterly business review deck, in brand
A real 7-slide deck100% pass

Where it slips

Owns the problem55% pass
Does not state what the team has already done about the payments reliability miss or the NPS drop; only reports the problems.
GPT-6 Astra · ChatGPT · A quarterly business review deck, in brand
Identifies material uncertainty61% pass
The output identifies the overlap uncertainty for decision A but does not name any material unknown that could change the call for decision B, such as the reliability of the Payments lead's estimate or the impact of pausing activation experiments.
Opus 5.5 · Claude · A quarterly business review deck, in brand
Flags the redefined pipeline63% pass
It names the redefinition but does not give the falling like-for-like trend (e.g., pipeline fell from $2.41M to $2.23M on the old definition).
GPT-6.1 Sol · API · On track, mostly

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Asks leaders only for the big calls
Does the update separate the decisions it needs from its readers (hard to reverse, high stakes) from the ones the team will make itself, and ask clearly for the former?
Passes when Names the decisions it needs from the readers, with a recommendation and a date, and says which calls the team is making itself.
Geoff CharlesLenny’s Podcast
Reports outcomes and learning, not just output
Does the update report progress as movement on the outcomes the team is accountable for and what it has learned, rather than as features shipped or dates hit?
Passes when Leads with the outcome metrics and what changed in the team's understanding, with shipping as supporting detail.
Melissa PerriLenny’s Podcast Christian IdiodiLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Owns the problem · Ready for the reader as delivered · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Head of Product at Cobalt. Write this fortnight's update for the CEO and the exec team, in no more than 450 words, from the weekly metrics and the team notes below. It should say where we stand, what's wrong, what we're doing about it, and what you need from them.

What the model was given3 items: About Cobalt, weekly_metrics_weeks_1_to_8.csv, team_notes.md

About CobaltTeam scheduling software, sold per seat.

weekly_metrics_weeks_1_to_8.csv8 rows · Download

week,new_arr_signed_usd,qualified_pipeline_usd,weekly_active_teams,support_backlog_tickets,support_sla_met_pct
1,182000,2410000,8210,140,96
2,205000,2380000,8260,152,95
3,168000,2350000,8240,171,94
4,214000,2290000,8300,198,91
5,199000,2230000,8280,260,86
6,221000,3140000,8310,344,79
…

team_notes.md7 lines · Download

# Notes from the teams

- Sales ops: from week 6, "qualified pipeline" includes active free trials over 10 seats (they used to be excluded).
- Data: weekly active teams for week 7 is missing because of a pipeline outage; it's being backfilled.
- Support: the backlog grew after the September pricing change (tickets about invoices doubled). SLA target is 90% answered within one business day.
- Quarter target: $2.4M new ARR in 12 weeks ($200,000 a week). Weeks 1 to 8 total: see the file.
- Support hiring: two agents approved, start date not set. Decision needed: whether to pause the planned price change for existing customers in week 10 until the backlog is cleared.

What a strong answer doesThe answer key the graders mark against

Leads with the takeaway: new ARR is ahead of plan ($1.67M in eight weeks against $1.6M), but support is breaking and pipeline is weaker than it looks. Flags that the jump in qualified pipeline in week 6 (from about $2.23M to $3.14M) is the redefinition to include large trials; on the old definition pipeline fell every week from $2.41M to $2.23M, a risk for the rest of the quarter. Reports the support problem plainly: backlog from 140 to 455 tickets and SLA from 96% to 72% against a 90% target since the September pricing change, with what's being done (two agents approved). Notes week 7's active teams are missing and being backfilled rather than hiding it. Asks for the decisions that need the execs: pause the week-10 price change for existing customers until the backlog clears, and set the start date for the support hires. Separates what the team is handling from what needs them.

Critical failures (cap the score at 40)

Presents qualified pipeline as having grown about 40%

Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Sonnet 5.5API98Usable with a quick edit GPT-6 AstraChatGPT96Usable with a quick edit GPT-6.1 SolAPI92Usable with a quick edit Haiku 5.5API87Needs more work Opus 5.5Claude79Needs more work GPT-6 LunaAPI77Usable with a quick edit Gemini 3.8 FlashAPI63Needs more work Gemini 3.5 Flash-LiteGemini58Needs more work

GPT-6 AstraChatGPT

Usable with a quick edit12 of 12 checks passed

The output is accurate, complete, and well-structured; a product manager could use it with at most light wording edits, without re-checking evidence or redoing reasoning.

Output

Fortnightly product update — end of week 8

Bottom line: New ARR is ahead of pace, but support is deteriorating and usage is flat to slightly down. I recommend pausing the week-10 price change for existing customers until we restore support capacity and service levels.

Where we stand

New ARR: $484k signed in weeks 7–8, up 15% on the prior fortnight. We’ve signed $1.673M against a $1.6M pace target, putting us $73k ahead. We need another $727k over four weeks—about $182k per week—to hit the $2.4M quarter target.
Pipeline: Reported qualified pipeline is $3.35M, but the jump from week 6 includes a definition change: active free trials above 10 seats now count. We should not treat that increase as evidence of stronger demand. Before the change, pipeline had declined for five consecutive weeks.
Usage: Weekly active teams reached 8,190 in week 8, down 1.4% from week 6 and broadly flat versus week 1. Week 7 is missing due to a data outage, so we cannot assess the full fortnight’s trajectory.

What’s wrong

Support backlog increased from 344 to 455 tickets this fortnight, while one-business-day SLA attainment fell from 79% to 72%, against a 90% target. Invoice tickets doubled after September’s pricing change. Another price change risks adding demand to an already overloaded queue.

Bookings are encouraging, but they do not establish healthy adoption or retention; the usage and support signals warrant attention.

Priorities for the next fortnight

Support/Product: Triage invoice issues, identify recurring billing causes, and prioritise fixes and customer guidance that reduce repeat contacts. Track backlog and SLA daily.
Data/Sales Ops: Complete the week-7 usage backfill and provide a like-for-like pipeline view separating newly included trials.
Support hiring: Confirm start dates and ramp plans for the two approved agents.

What I need from the exec team

Approve pausing the week-10 existing-customer price change. Resume only once the backlog is cleared and SLA performance is sustainably back at or above 90%.
Secure start dates for both support hires this week and agree interim coverage if they cannot start before week 10.

Grades and run details

Decision model 92 · LLM judge 100

Decision model checks

passUses the supplied evidence correctly48%
passAddresses the actual decision99%
passRespects explicit constraints43%
passIdentifies material uncertainty37%
partialAvoids unsupported claims31%
passProduces the required deliverable94%
passFlags the redefined pipeline96%
passReports the support breach plainly100%
partialOwns the problem38%
passReady for the reader as delivered78%
passAsks leaders only for the big calls85%
passReports outcomes and learning, not just output98%

Run

Run: #1
Time to output: 17 s
Submitted: 5 Oct 2026

Sonnet 5.5API

Usable with a quick edit12 of 12 checks passed

The output is accurate, complete, and well-structured. A PM could use it with only minor wording tweaks, without re-checking evidence or redoing any reasoning.

Output

Cobalt product update: weeks 7-8

Bottom line: New ARR is on track, but support is deteriorating and our pipeline number is misleading. I need a decision on the week 10 price change.

Where we stand - New ARR: $1.673M signed over 8 weeks, against a pro-rata target of $1.6M. We're $73k ahead. Hitting $2.4M needs about $182k a week over the last four, and we've averaged $209k. - Active teams: 8,190 in week 8, down from a peak of 8,310 in week 6 and roughly flat since week 1. Week 7 is missing because of a data outage and is being backfilled, so I'm not reading a trend into it.

What's wrong 1. Support is failing its SLA. Backlog has risen from 140 to 455 tickets. The SLA hit rate fell from 96% to 72% against a 90% target, and we've been under target since week 5. Support attributes this to the September pricing change, which doubled invoice tickets. Weekly backlog growth slowed in week 8 (+43 versus +68), but it's still growing. 2. Pipeline looks healthier than it is. Reported qualified pipeline jumped from $2.23M to $3.14M in week 6, but that's when Sales Ops started counting active free trials over 10 seats. Before that, pipeline had drifted down for five straight weeks ($2.41M to $2.23M). Until it's restated, I can't tell you whether underlying pipeline is growing, and that matters for next quarter's ARR.

What we're doing - Invoice tickets: We're working with Support to identify the top invoice-confusion causes, then ship clearer invoice explanations and canned responses to cut ticket volume at the source. - Pipeline: I've asked Sales Ops and Data to restate weeks 1-8 on a consistent definition, showing trials separately. I'll bring that next update. - Active teams: Data will backfill week 7, and I'll confirm whether the dip since week 6 is real.

What I need from you 1. Decision: pause the week 10 price change for existing customers until the SLA is back at 90%. The last pricing change drove a ticket surge we haven't absorbed, and a second one on top of a 455-ticket backlog risks churn and more SLA misses. New ARR has a cushion, so I think the delay is affordable. I'd like Finance to confirm the revenue impact of waiting. 2. Set start dates for the two approved support agents. They're approved but have no date, and the backlog is growing now. Two agents may not clear it alone, so I'd also like to discuss temporary cover. 3. Agree that we report pipeline on the restated basis (trials shown separately) from the next update.

Grades and run details

Decision model 96 · LLM judge 100

Decision model checks

passUses the supplied evidence correctly37%
passAddresses the actual decision64%
passRespects explicit constraints57%
passIdentifies material uncertainty76%
partialAvoids unsupported claims50%
passProduces the required deliverable92%
passFlags the redefined pipeline100%
passReports the support breach plainly100%
passOwns the problem68%
passReady for the reader as delivered57%
passAsks leaders only for the big calls76%
passReports outcomes and learning, not just output99%

Run

Run: #1
API response time: 17 s
Submitted: 6 Oct 2026

Haiku 5.5API

Needs more than a quick edit11 of 12 checks passed

The output includes an unsupported statement about Product reviewing invoice categories, which would require verification before sending—more than a quick edit.

Output

Subject: Product update, weeks 1 to 8

Where we stand - New ARR: $1.67M signed in weeks 1 to 8, $73K ahead of the $1.6M pace ($200K/week). The last four weeks averaged $226K, and weekly sales have risen since week 5. - Quarter target: To reach $2.4M by week 12, we need $727K over weeks 9 to 12, about $182K a week. At the recent run rate we'd finish near $2.6M, if the trend holds. - Active teams: Flat at 8,190 to 8,310. Week 7 is missing because of a pipeline outage, and Data is backfilling it.

What's wrong - Support is the main problem. The backlog rose from 140 to 455 tickets, and SLA attainment fell from 96% to 72%. We've been below our 90% target since week 5. Invoice tickets doubled after the September pricing change. - The pipeline jump is not growth. Qualified pipeline rose from $2.23M to $3.14M in week 6 because the definition changed: active free trials over 10 seats now count. Weeks 6 to 8 aren't comparable with weeks 1 to 5. On the old definition, pipeline fell from $2.41M to $2.23M over weeks 1 to 5. Until we restate the history, I won't treat the jump as a gain.

What we're doing - Support hiring: Both approved agents still have no start date. I'm working with People to set dates this week. - Invoice tickets: Product is reviewing the invoice ticket categories to find the causes we can fix in the product rather than answer one by one. - Pipeline reporting: I've asked Sales Ops and Finance to restate weeks 1 to 8 on one definition so we can see the real trend.

What I need from you 1. Decision on the week 10 price change for existing customers. I recommend pausing it until SLA is back above 90%. The change would add invoice tickets during our worst support period, and invoices are already the main source of the backlog. I need this decision by the end of next week. 2. Fallback support capacity. If the agents don't start by week 10, we need budget for temporary contract support. I'll size the cost with Support and bring it back to you. 3. Restated pipeline numbers. Please support the restatement so the board sees one consistent series.

Happy to take questions before Thursday's exec meeting.

Head of Product, Cobalt

Claims the judge couldn’t find in the brief

Product is reviewing the invoice ticket categories to find causes fixable in the product.
I've asked Sales Ops and Finance to restate weeks 1 to 8 on one definition.

Grades and run details

Decision model 96 · LLM judge 85

Decision model checks

passUses the supplied evidence correctly41%
passAddresses the actual decision76%
passRespects explicit constraints49%
passIdentifies material uncertainty73%
partialAvoids unsupported claims54%
passProduces the required deliverable97%
passFlags the redefined pipeline100%
passReports the support breach plainly100%
passOwns the problem62%
passReady for the reader as delivered48%
passAsks leaders only for the big calls82%
passReports outcomes and learning, not just output100%

Run

Run: #1
API response time: 13 s
Submitted: 8 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 1

Uses the supplied evidence correctlyRightRightMixed

GPT-6 Astra · ChatGPT

All factual claims about the current situation are directly supported by the supplied metrics or team notes, with no invented facts.

Sonnet 5.5 · API

All factual claims about the current situation are directly supported by the supplied metrics or team notes, or follow from simple arithmetic.

Haiku 5.5 · API

The output states Product is reviewing invoice categories and that Sales Ops/Finance have been asked to restate pipeline—these current actions are not in the supplied context, violating the evidence rule.

All got right 11

Addresses the actual decisionRightRightRight

GPT-6 Astra · ChatGPT

The output commits early to pausing the week-10 price change, frames it for the CEO/exec team, and states the condition (backlog cleared, SLA ≥90%) that would change the call.

Sonnet 5.5 · API

Commits unambiguously to pausing the week 10 price change, frames it for the exec team, and implies the call could change if Finance confirms the revenue impact is unaffordable.

Haiku 5.5 · API

It recommends pausing the week-10 price change until SLA is above 90%, gives a deadline, and names what would change the call.

Respects explicit constraintsRightRightRight

GPT-6 Astra · ChatGPT

The update is under 450 words, addresses where we stand, what's wrong, what we're doing, and what's needed, and is written as a fortnightly update for the CEO and exec team.

Sonnet 5.5 · API

The update is under 450 words, written as a fortnightly update for the CEO and exec team, and respects the requested form.

Haiku 5.5 · API

The update is within 450 words, addressed to the CEO/exec team, and covers standing, problems, actions and asks.

Identifies material uncertaintyRightRightRight

GPT-6 Astra · ChatGPT

It identifies the missing week 7 data, the pipeline redefinition, and the condition for resuming the price change, naming the unknowns that affect the decision.

Sonnet 5.5 · API

Identifies the pipeline restatement, active teams backfill, revenue impact of delay, and sufficiency of two agents as unknowns, and says how each will be resolved.

Haiku 5.5 · API

It identifies agent start-date uncertainty and pipeline comparability, links them to decisions, and says how they would be resolved.

Avoids unsupported claimsRightRightRight

GPT-6 Astra · ChatGPT

Forecasts and interpretations are labelled as risks or opinions (e.g., 'risks adding demand', 'warrant attention'), not presented as established facts.

Sonnet 5.5 · API

Causes and forecasts are clearly labelled as such (e.g., 'risks churn', 'I think the delay is affordable'), and confident claims are backed by the evidence.

Haiku 5.5 · API

Interpretations like the pipeline jump being a definition change and invoice doubling are directly supported by the notes; no causes or forecasts are presented as established fact without evidence.

Produces the required deliverableRightRightRight

GPT-6 Astra · ChatGPT

The update is complete, within the word limit, structured for the named reader, and contains all required elements to act on.

Sonnet 5.5 · API

The update is complete, in the right form for the CEO and exec team, within the word limit, and the reader could act on the asks without filling major gaps.

Haiku 5.5 · API

The memo is complete, properly formatted for the exec team, within the word limit, and provides actionable direction.

Flags the redefined pipelineRightRightRight

GPT-6 Astra · ChatGPT

It explicitly names the redefinition (free trials over 10 seats now included) and states the like-for-like trend of five consecutive weekly declines.

Sonnet 5.5 · API

Explicitly names the redefinition and gives the like-for-like trend of five straight weekly declines from $2.41M to $2.23M.

Haiku 5.5 · API

It explicitly names the redefinition, gives the like-for-like falling trend ($2.41M to $2.23M), and explains the jump is not growth.

Reports the support breach plainlyRightRightRight

GPT-6 Astra · ChatGPT

It reports the backlog (344 to 455) and SLA (79% to 72%) against the 90% target, links the cause to the pricing change, and asks for the pause decision.

Sonnet 5.5 · API

Reports the backlog (140 to 455) and SLA (96% to 72% against 90% target), links them to the pricing change, and asks for the pause decision.

Haiku 5.5 · API

Backlog 140→455, SLA 96%→72% against 90% target, cause (Sept pricing change, invoice tickets doubled) and the pause decision are all reported plainly.

Owns the problemRightRightRight

GPT-6 Astra · ChatGPT

For the support problem, it lists actions already taken (triage, identify causes, fixes, daily tracking) and recommends pausing the price change with its trade-off.

Sonnet 5.5 · API

For each problem, states actions already taken (working with Support, asking Sales Ops and Data to restate) and recommends a way forward with trade-offs.

Haiku 5.5 · API

For support and pipeline problems it lists actions taken (agent hiring, People coordination, restatement request) and recommends a way forward.

Ready for the reader as deliveredRightRightRight

GPT-6 Astra · ChatGPT

The update is written entirely for the CEO/exec team, with no placeholder names, no lifted internal notes, and all owners and dates are either from the material or clearly proposed.

Sonnet 5.5 · API

Written entirely for the exec team, uses only owners and terms from the context, and contains no placeholders or notes to the requester.

Haiku 5.5 · API

The language is self-contained, no placeholders, and every owner/date is either from the material or clearly proposed.

Asks leaders only for the big callsRightRightRight

GPT-6 Astra · ChatGPT

It separates the exec asks (approve pause, secure start dates) from the team's own priorities, with clear recommendations and timing.

Sonnet 5.5 · API

Separates the big calls for leaders (pause price change, set start dates, agree reporting) from what the team is handling itself (invoice fixes, pipeline restatement, backfill).

Haiku 5.5 · API

It separates big calls needing execs (pause price change, fallback budget, restatement support) from team actions, with recommendations and deadlines.

Reports outcomes and learning, not just outputRightRightRight

GPT-6 Astra · ChatGPT

It leads with outcome metrics (ARR, support, usage) and reports a key learning (pipeline jump is a definition change, not demand growth), not just activities.

Sonnet 5.5 · API

Leads with outcome metrics (ARR, active teams, SLA, pipeline) and what the team learned (pipeline redefinition, invoice ticket surge), not just activities.

Haiku 5.5 · API

It leads with outcome metrics (ARR, active teams), reports the learning that pipeline jump is a definition change, and links support issues to the pricing change.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 76% of 89 checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	GPT-6 AstrawithChatGPT	94.2	92.9	2	None
2	Sonnet 5.5withAPI	84.6	96.4	2	None
3	Haiku 5.5withAPI	89.6	92.3	2	None
4	GPT-6.1 SolwithAPI	80.8	96.2	2	None
5	Opus 5.5withClaude	86.3	77.5	2	None
6	GPT-6 LunawithAPI	76.7	66.8	2	None
7	Gemini 3.5 Flash-LitewithGemini	70.8	51.6	2	None
8	Gemini 3.8 FlashwithAPI	66.3	51.9	2	None

What you get for the cost

Haiku 5.5 · API scores within 5 points of GPT-6 Astra · ChatGPT at a tenth of the cost.

Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
Gemini 3.5 Flash-Lite · Gemini	60	1.0× (partly estimated)
GPT-6 Luna · API	71	1.3×
Haiku 5.5 · API	89	2.6×
Gemini 3.8 Flash · API	58	13×
GPT-6.1 Sol · API	88	15×
GPT-6 Astra · ChatGPT	94	26× (partly estimated)
Sonnet 5.5 · API	90	40×
Opus 5.5 · Claude	81	55× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Writing the weekly update, or the reply to an executive's question, that the team's credibility rests on.

Why it matters

Updates are where bad news gets softened. A fluent update that buries a slipped date or presents a guess as the cause does more harm than no update: leadership decides on it, and trust goes when the truth comes out.

What good looks like

Leads with the news and any decision needed
States dates, numbers and causes plainly, including bad news
Separates what is known from what is suspected
Brings a recommendation and what's been tried, not just the problem
Fits the reader and the length asked for

Deliberately not measured

Tone and formatting preferences
Writing style beyond clarity

Capability tested

Honest, decision-first status communication

The failure we’re looking for

Spin: burying or softening the bad news, or presenting a guess as fact

Grading

Decision model, LLM judge and a browser check (for the deck), calibrated against a blind PM review

Variants

A short written update · Staff level: a QBR deck in brand colours

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

Asks leaders only for the big calls

Reports outcomes and learning, not just output

Plus our standard checks

The tasks

The outputs

Output

Fortnightly product update — end of week 8

Where we stand

What’s wrong

Priorities for the next fortnight

What I need from the exec team

Grades and run details

Decision model checks

Run

Output

Grades and run details

Decision model checks

Run

Output

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Run

Check by check

Where they differ 1

All got right 11

Results

What you get for the cost

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured