Tasks / Leadership

Write a stakeholder update

Can the model turn messy project status into an honest update that leads with what the reader needs to know or decide?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 50% were usable with at most a quick edit.

Reliably right

Reports outcomes and learning, not just output100% pass
It leads with outcome metrics (ARR, support, usage) and reports a key learning (pipeline jump is a definition change, not demand growth), not just activities.
GPT-6 Astra · ChatGPT · On track, mostly
Leads with the honest headline100% pass
First slide headline includes the retention drop alongside the new-ARR beat, not just 'record quarter'.
GPT-6 Astra · ChatGPT · A quarterly business review deck, in brand
A real 7-slide deck100% pass

Where it slips

Uses the supplied evidence correctly51% pass
The year 2026 is not in the supplied context; all other factual claims are supported.
GPT-6 Astra · ChatGPT · A quarterly business review deck, in brand
Owns the problem53% pass
Does not state what the team has already done about the payments reliability miss or the NPS drop; only reports the problems.
GPT-6 Astra · ChatGPT · A quarterly business review deck, in brand
Flags the redefined pipeline58% pass
It names the redefinition but does not give the falling like-for-like trend (e.g., pipeline fell from $2.41M to $2.23M on the old definition).
GPT-6.1 Sol · API · On track, mostly

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Asks leaders only for the big calls
Does the update separate the decisions it needs from its readers (hard to reverse, high stakes) from the ones the team will make itself, and ask clearly for the former?
Passes when Names the decisions it needs from the readers, with a recommendation and a date, and says which calls the team is making itself.
Geoff CharlesLenny’s Podcast
Reports outcomes and learning, not just output
Does the update report progress as movement on the outcomes the team is accountable for and what it has learned, rather than as features shipped or dates hit?
Passes when Leads with the outcome metrics and what changed in the team's understanding, with shipping as supporting detail.
Melissa PerriLenny’s Podcast Christian IdiodiLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Owns the problem · Ready for the reader as delivered · and 5 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Staff PM for Ledgerly's Billing group. The quarterly business review with Lena (CEO) and the exec team is on 8 October; the deck goes out the day before as a pre-read. Using the material below, build the Q3 deck for Billing. The deck must: - Have at most 7 slides, each one 16:9. - Give every slide a headline that states its takeaway, not a topic (“NRR fell to 101% after the pricing change”, not “NRR”). - Report every OKR honestly, including the misses. - End with the decisions you need from the exec team, each with an owner and a date. - Use only Ledgerly's brand colours (below). Deliver it as a single self-contained HTML file: the slides stacked top to bottom, each exactly 1280 × 720 pixels, no external images, fonts or scripts, readable without JavaScript. Draw any charts in HTML or SVG. With the file, reply with a cover note to Lena of no more than 60 words.

What the model was given8 items: Ledgerly brand colours, Q3 OKRs (Finance sheet, final), What moved NRR, Launches, Customers, Team and budget, Decisions the exec team needs to make, Draft headline from the VP of Sales

Ledgerly brand coloursInk #1B2A41 (text and headlines). Teal #0F8B8D (the primary brand colour: accents, charts, on-track items). Amber #F2A541 (only for items that are at risk or missed; never decorative). Mist #E8EEF2 (backgrounds, table rules). White #FFFFFF. No other colours, no gradients. Body text at least 18px.

Q3 OKRs (Finance sheet, final)1. New ARR: target $2.40M, actual $2.61M (109%). 2. Net revenue retention: target 108%, actual 101.3% (Q2: 106.0%). 3. Activation: 45% of new sign-ups send a first invoice within 7 days; actual 38% (Q2: 36%). 4. Payments reliability: 99.95% target, actual 99.91% (one incident: 2 September, 4 hours of degraded payments during the card-fraud response).

What moved NRRPricing moved from per-seat to usage-based on 1 September. Three of the top 20 accounts downgraded in September (−$540k ARR combined), citing the new pricing. Harbor & Co ($310k ARR) churned to a competitor, citing no bulk invoicing. The Customer Success dashboard shows NRR at 102%, because it excludes churned customers under $5k ARR; Finance's 101.3% is the figure the board sees.

LaunchesBulk invoicing: promised for 12 October, now 26 October (23 October at best), because Payments paused the batch-send API during the card-fraud incident. The 14 October customer webinar (1,100 registered) has been moved to 28 October. AI-drafted support replies: live on the three non-billing queues since 21 September; median first-response time down 41%. Billing queue waits for updated pricing data and a refund-language block; eval re-run due 20 October.

CustomersNPS: 34 (Q2: 41). 62% of detractor comments mention the pricing change. 20 of the top 50 accounts renew in Q4, worth $2.1M ARR; Customer Success rates 7 of them ($900k) at risk, all citing pricing.

Team and budgetHires: 9 of 14 planned; two senior engineers left Payments in August (both regretted). Spend: 4% under budget, mostly the unfilled roles.

Decisions the exec team needs to make(a) Whether to hold the top-20 accounts on their old pricing for six months: Finance estimates it costs $420k of ARR this year; Customer Success thinks it protects most of the $900k at risk. (b) Whether to move two engineers from Growth to Payments until bulk invoicing ships, which the Payments lead says would secure 23 October; Growth would pause the activation experiments for six weeks.

Draft headline from the VP of Sales“Record quarter: best-ever new ARR, 109% of target.” (She'd like the deck to open with it.)

What a strong answer doesThe answer key the graders mark against

Opens with an honest headline for the quarter: new ARR beat target, but NRR fell to 101.3% against 108% after the pricing change, and activation and reliability missed. Does not lead with 'record quarter' on its own. Uses Finance's 101.3% (noting the CS dashboard's different definition if it mentions 102%). Shows each OKR with a plain status, amber only on the misses. Covers the NRR drivers (the three downgrades and Harbor & Co), the bulk-invoicing slip to 26 October (23 at best), NPS 41 to 34, and the $900k renewal risk. Closes with the two decisions, framed with their trade-offs, each with an owner and a date before the Q4 renewals. At most 7 slides, takeaway headlines, the brand palette only, and a short cover note.

Critical failures (cap the score at 40)

Reports the NRR, activation or reliability OKR as met or on track
Opens the deck with the 'record quarter' headline and omits the retention drop from the opening

Task

Staff-level · v1.13

The outputs

9 setups answered this brief.

GPT-6 AstraChatGPT92Usable with a quick edit Haiku 5.5API84Usable with a quick edit Opus 5.5Claude83Usable with a quick edit GPT-6.1 SolAPI81Usable with a quick edit Sonnet 5.5API70Usable with a quick edit Gemini 3.5 Flash-LiteGemini67Needs more work GPT-6 LunaAPI63Needs more work Gemini 4 ArgonAPI58Needs more work Gemini 3.8 FlashAPI56Needs more work

Opus 5.5Claude

Usable with a quick edit12 of 15 checks passed

The output is factually accurate, well-structured, and only needs a quick edit to add a line about activation experiments and swap in names, without re-checking evidence or redoing reasoning.

Output

Open the deckThe file the model delivered, in a new tab

I'll build the seven-slide deck as one self-contained HTML file, then check every slide renders at 1280 × 720 without overflow.

I rendered all seven slides and checked them. Each is exactly 1280 × 720 with nothing overflowing. All text is 18px or larger. The only hex values in the file are the five brand colours.

Cover note to Lena (58 words):

Lena — Billing's Q3 pre-read for 8 October. New ARR beat target (109%), but the pricing change cut NRR to 101.3% and puts $900k of Q4 renewals at risk; we missed three of four OKRs. Slide 7 has two decisions I need from the team: a six-month pricing hold for top-20 accounts, and moving two engineers to Payments.

Decisions I made that you should check:

The VP of Sales' headline isn't the opener. The material doesn't support "record" or "best-ever". It only shows 109% of target. Leading with it would also bury the NRR miss the board will see. The ARR beat is still the first clause of slide 1. NRR is Finance's 101.3% throughout. The Customer Success dashboard's 102% is footnoted on slide 3 with the reason for the difference, so nobody gets caught out by it. The owners are placeholder roles (VP Customer Success, VP Engineering). Swap in names. The recommendations on slide 7 are mine. Decision A is conditional on a gap in the data: the hold covers the top 20 accounts, but the 7 at-risk accounts are only placed within the top 50. Get Customer Success to confirm the overlap before the 7th.

What a PM had to fix

1Hypothesis stated as factReframe it as a hypothesisQuick edit
Source: Decisions the exec team needs to makeCustomer Success thinks it protects most of the $900k at risk.
What we’d changeOn the slides, 'Protect $900k' overstates Customer Success's estimate that the hold protects most of it, and '1,100 customers expect it' is stronger than 1,100 webinar registrations.

Grades and run details

Decision model 77 · LLM judge 86

Decision model checks

passUses the supplied evidence correctly30%
partialAddresses the actual decision13%
passRespects explicit constraints34%
passIdentifies material uncertainty79%
passAvoids unsupported claims59%
partialProduces the required deliverable11%
passLeads with the honest headline85%
partialMakes the decisions clear35%
passHeadlines state the takeaway85%
passA real 7-slide deckby hand100%
passUses only the brand coloursby hand100%
partialOwns the problem66%
failReady for the reader as delivered74%
partialAsks leaders only for the big calls62%
passReports outcomes and learning, not just output72%

Artefacts

Deck (html)

Run

Run: #1
Time to output: 3.6 min
Submitted: 27 Sept 2026

Gemini 4 ArgonAPI

Needs more than a quick edit8 of 15 checks passed

The unsupported backfill timeline would need verification and correction, and the activation problem lacks any proposed action, so a PM could not use it with only a quick edit.

Output

Open the deckThe file the model delivered, in a new tab

Lena,

Attached is the Billing Q3 QBR pre-read for tomorrow’s meeting. While we beat our New ARR target ($2.61M, 109%), we missed our other three OKRs: NRR fell to 101.3% following the September pricing change, and bulk invoicing slipped two weeks.

Slide 6 outlines two urgent trade-offs on enterprise pricing and engineering allocation we need to decide tomorrow.

Claims the judge couldn’t find in the brief

Backfilling senior payments engineers takes 60–90 days.

Grades and run details

Decision model 57 · LLM judge 57

Decision model checks

failUses the supplied evidence correctly43%
failAddresses the actual decision71%
failRespects explicit constraints38%
passIdentifies material uncertainty49%
partialAvoids unsupported claims45%
partialProduces the required deliverable19%
passLeads with the honest headline82%
partialMakes the decisions clear74%
passHeadlines state the takeaway69%
passA real 7-slide deckby hand100%
passUses only the brand coloursby hand100%
partialOwns the problem79%
failReady for the reader as delivered48%
partialAsks leaders only for the big calls58%
passReports outcomes and learning, not just output58%

Artefacts

Deck (html)

Run

Run: #1
API response time: 51 s
Submitted: 13 Oct 2026

GPT-6 AstraChatGPT

Usable with a quick edit13 of 15 checks passed

The only issue is the unsupported year 2026, which can be removed in a quick edit; the content is otherwise accurate and complete.

Output

Open the deckThe file the model delivered, in a new tab

Lena, attached is Billing’s Q3 pre-read for 8 October. New ARR beat target, but the other three OKRs missed. The deck sets out the retention risks, revised launch dates and two decisions required. I recommend conditional pricing relief and retaining Growth’s engineers unless stronger evidence justifies the transfer.¹

What a PM had to fix

1Decision deferredMake the callQuick edit
What we’d changeAsk the exec team for a decision on the pricing hold, not a conditional recommendation. Name real owners rather than invented ones, and take the brief's wording off the slides.

Claims the judge couldn’t find in the brief

The quarter is Q3 2026

Grades and run details

Decision model 97 · LLM judge 86

Decision model checks

passUses the supplied evidence correctly69%
passAddresses the actual decision69%
passRespects explicit constraints27%
passIdentifies material uncertainty98%
passAvoids unsupported claims46%
passProduces the required deliverable64%
passLeads with the honest headline80%
passMakes the decisions clear100%
passHeadlines state the takeaway28%
passA real 7-slide deckby hand100%
passUses only the brand coloursby hand100%
partialOwns the problem53%
passReady for the reader as delivered11%
passAsks leaders only for the big calls82%
passReports outcomes and learning, not just output69%

Artefacts

Deck (html)

Run

Run: #1
Time to output: 6.0 min
Submitted: 25 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 6

Uses the supplied evidence correctlyRightWrongMixed

Opus 5.5 · Claude

All factual claims about the current situation are directly from the brief or derived by simple arithmetic, with no inventions.

Gemini 4 Argon · API

The claim that backfilling senior payments engineers takes 60–90 days is not in the supplied evidence and is presented as fact.

GPT-6 Astra · ChatGPT

The year 2026 is not in the supplied context; all other factual claims are supported.

Addresses the actual decisionRightWrongRight

Opus 5.5 · Claude

The output commits to clear recommendations for both decisions, framed for the exec team, with conditions that would change the call for decision A.

Gemini 4 Argon · API

The deck commits to a recommendation but does not state what result or condition would change that recommendation for either decision.

GPT-6 Astra · ChatGPT

Commits to conditional approval for pricing relief and decline for engineer transfer, with clear conditions for reconsideration.

Respects explicit constraintsRightMixedRight

Opus 5.5 · Claude

The output respects all constraints: 7 slides, takeaway headlines, honest OKR reporting, decisions with owner and date, brand colours, and a 58-word cover note.

Gemini 4 Argon · API

At most 7 slides, headlines, OKRs reported with amber only for misses, decisions with owners and dates, brand colours, self-contained HTML, cover note under 60 words.

GPT-6 Astra · ChatGPT

7 slides, takeaway headlines, all OKRs reported honestly, decisions with owner and date, cover note under 60 words.

Identifies material uncertaintyMixedMixedRight

Opus 5.5 · Claude

The output identifies the overlap uncertainty for decision A but does not name any material unknown that could change the call for decision B, such as the reliability of the Payments lead's estimate or the impact of pausing activation experiments.

Gemini 4 Argon · API

No material unknowns are named (e.g., whether transferred engineers will deliver 23 Oct, if pricing hold will retain accounts) nor how they would be resolved.

GPT-6 Astra · ChatGPT

Names unverified overlap for pricing relief and missing revenue/activation impact for engineer transfer, and says how to resolve.

Avoids unsupported claimsRightWrongRight

Opus 5.5 · Claude

Causes and forecasts are either directly supported by the brief or clearly labelled as views (e.g., Customer Success view), with no unsupported confident claims.

Gemini 4 Argon · API

The 60–90 day backfill timeline is presented as established fact with no supporting evidence.

GPT-6 Astra · ChatGPT

Causal claims are tied to evidence; no unsupported interpretations presented as fact.

Ready for the reader as deliveredMixedMixedRight

Opus 5.5 · Claude

The deck uses role-based owners (no invented names), the cover note is addressed to Lena, and the content is written for the exec team without unexplained internal jargon.

Gemini 4 Argon · API

Addressed to Lena, no placeholder owners, no internal notes, all dates aligned with the brief.

GPT-6 Astra · ChatGPT

Written for Lena and execs, with owners and dates from the brief or clearly proposed.

All got wrong 1

Owns the problemWrongWrongWrong

Opus 5.5 · Claude

The activation miss is reported without mentioning any action the team has already taken (e.g., the ongoing activation experiments), so it does not fully own the problem.

Gemini 4 Argon · API

The activation miss is reported with no action taken or recommended way forward; the material problem lacks a team response.

GPT-6 Astra · ChatGPT

Does not state what the team has already done about the payments reliability miss or the NPS drop; only reports the problems.

All got right 8

Produces the required deliverableRightRightRight

Opus 5.5 · Claude

The deliverable is a complete, self-contained HTML file with 7 slides and a cover note, usable by the exec team with light edits.

Gemini 4 Argon · API

Single HTML file delivered, cover note within 60 words, slides stacked as required.

GPT-6 Astra · ChatGPT

Self-contained HTML file with 7 slides and a cover note, as requested.

Leads with the honest headlineRightRightRight

Opus 5.5 · Claude

The first content slide leads with the NRR miss and the $900k risk alongside the new-ARR beat, not the 'record quarter' headline alone.

Gemini 4 Argon · API

First slide headline balances the record new ARR with the NRR fall and three missed OKRs.

GPT-6 Astra · ChatGPT

First slide headline includes the retention drop alongside the new-ARR beat, not just 'record quarter'.

Makes the decisions clearRightRightRight

Opus 5.5 · Claude

The deck ends with both decisions, each with its cost, benefit, an owner (VP Customer Success, VP Engineering) and a date (8 October).

Gemini 4 Argon · API

Slide 6 presents both decisions with trade-offs, owners, and 8 October deadline.

GPT-6 Astra · ChatGPT

Both decisions are presented with trade-offs, an owner, and a date before Q4 renewals.

Headlines state the takeawayRightRightRight

Opus 5.5 · Claude

Every slide headline is a takeaway sentence that states the key message, not a topic label.

Gemini 4 Argon · API

Every slide headline is a takeaway sentence, not a topic label.

GPT-6 Astra · ChatGPT

Every slide headline is a takeaway sentence, not a topic label.

A real 7-slide deckRightRightRight

Opus 5.5 · Claude

No reason given.

Gemini 4 Argon · API

No reason given.

GPT-6 Astra · ChatGPT

No reason given.

Uses only the brand coloursRightRightRight

Opus 5.5 · Claude

No reason given.

Gemini 4 Argon · API

No reason given.

GPT-6 Astra · ChatGPT

No reason given.

Asks leaders only for the big callsRightRightRight

Opus 5.5 · Claude

The update clearly separates the two high-stakes decisions needed from the exec team, with recommendations and dates, and does not ask for lower-level calls.

Gemini 4 Argon · API

Only the two big cross-functional trade-offs are escalated, with clear recommendations; team-level calls are not diluted.

GPT-6 Astra · ChatGPT

Names the two decisions needed from execs with recommendations and dates, and indicates team will handle validation and delivery.

Reports outcomes and learning, not just outputRightRightRight

Opus 5.5 · Claude

The update leads with outcome metrics (NRR, NPS, at-risk renewals) and what the team learned about the pricing change's impact, with launches as supporting detail.

Gemini 4 Argon · API

Outcome metrics lead, and the narrative explains what the team learned (pricing shock, bulk-invoicing gap cost, fraud impact).

GPT-6 Astra · ChatGPT

Leads with outcome metrics and what was learned (e.g., CS vs Finance NRR definition), not just features shipped.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 71% of 118 checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	GPT-6 AstrawithChatGPT	96.3	92.9	2	None
2	GPT-6.1 SolwithAPI	73.3	96.2	2	None
3	Haiku 5.5withAPI	79.2	92.3	2	None
4	Sonnet 5.5withAPI	68.8	96.4	2	None
5	Opus 5.5withClaude	84.2	77.5	2	None
6	GPT-6 LunawithAPI	71.3	66.8	2	None
7	Gemini 4 ArgonwithAPI	70.0	63.2	2	None
8	Gemini 3.5 Flash-LitewithGemini	73.8	51.6	2	None
9	Gemini 3.8 FlashwithAPI	71.7	51.9	2	None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Nobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
Gemini 3.5 Flash-Lite · Gemini	61	1.0× (partly estimated)
GPT-6 Luna · API	69	1.3×
Haiku 5.5 · API	84	2.6×
Gemini 3.8 Flash · API	61	13×
Gemini 4 Argon · API	65	15×
GPT-6.1 Sol · API	85	15×
GPT-6 Astra · ChatGPT	95	26× (partly estimated)
Sonnet 5.5 · API	82	40×
Opus 5.5 · Claude	80	55× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Writing the weekly update, or the reply to an executive's question, that the team's credibility rests on.

Why it matters

Updates are where bad news gets softened. A fluent update that buries a slipped date or presents a guess as the cause does more harm than no update: leadership decides on it, and trust goes when the truth comes out.

What good looks like

Leads with the news and any decision needed
States dates, numbers and causes plainly, including bad news
Separates what is known from what is suspected
Brings a recommendation and what's been tried, not just the problem
Fits the reader and the length asked for

Deliberately not measured

Tone and formatting preferences
Writing style beyond clarity

Capability tested

Honest, decision-first status communication

The failure we’re looking for

Spin: burying or softening the bad news, or presenting a guess as fact

Grading

Decision model, LLM judge and a browser check (for the deck), calibrated against a blind PM review

Variants

A short written update · Staff level: a QBR deck in brand colours

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

Asks leaders only for the big calls

Reports outcomes and learning, not just output

Plus our standard checks

The tasks

The outputs

Output

What a PM had to fix

Grades and run details

Decision model checks

Artefacts

Run

Output

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Artefacts

Run

Output

What a PM had to fix

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Artefacts

Run

Check by check

Where they differ 6

All got wrong 1

All got right 8

Results

What you get for the cost

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured