Tasks / Leadership

Write a stakeholder update

Can the model turn messy project status into an honest update that leads with what the reader needs to know or decide?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 56% were usable with at most a quick edit.

Reliably right

Reports outcomes and learning, not just output100% pass
Leads with outcome metrics (ARR, active teams, SLA, pipeline) and what the team learned (pipeline redefinition, invoice ticket surge), not just activities.
Sonnet 5.5 · API · On track, mostly
Leads with the honest headline100% pass
First slide headline includes the retention drop alongside the new-ARR beat, not just 'record quarter'.
GPT-6 Astra · ChatGPT · A quarterly business review deck, in brand
A real 7-slide deck100% pass

Where it slips

Owns the problem55% pass
Does not state what the team has already done about the payments reliability miss or the NPS drop; only reports the problems.
GPT-6 Astra · ChatGPT · A quarterly business review deck, in brand
Identifies material uncertainty61% pass
The output identifies the overlap uncertainty for decision A but does not name any material unknown that could change the call for decision B, such as the reliability of the Payments lead's estimate or the impact of pausing activation experiments.
Opus 5.5 · Claude · A quarterly business review deck, in brand
Flags the redefined pipeline63% pass
It names the redefinition but does not give the falling like-for-like trend (e.g., pipeline fell from $2.41M to $2.23M on the old definition).
GPT-6.1 Sol · API · On track, mostly

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Asks leaders only for the big calls
Does the update separate the decisions it needs from its readers (hard to reverse, high stakes) from the ones the team will make itself, and ask clearly for the former?
Passes when Names the decisions it needs from the readers, with a recommendation and a date, and says which calls the team is making itself.
Geoff CharlesLenny’s Podcast
Reports outcomes and learning, not just output
Does the update report progress as movement on the outcomes the team is accountable for and what it has learned, rather than as features shipped or dates hit?
Passes when Leads with the outcome metrics and what changed in the team's understanding, with shipping as supporting detail.
Melissa PerriLenny’s Podcast Christian IdiodiLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Owns the problem · Ready for the reader as delivered · and 5 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Staff PM for Ledgerly's Billing group. The quarterly business review with Lena (CEO) and the exec team is on 8 October; the deck goes out the day before as a pre-read. Using the material below, build the Q3 deck for Billing. The deck must: - Have at most 7 slides, each one 16:9. - Give every slide a headline that states its takeaway, not a topic (“NRR fell to 101% after the pricing change”, not “NRR”). - Report every OKR honestly, including the misses. - End with the decisions you need from the exec team, each with an owner and a date. - Use only Ledgerly's brand colours (below). Deliver it as a single self-contained HTML file: the slides stacked top to bottom, each exactly 1280 × 720 pixels, no external images, fonts or scripts, readable without JavaScript. Draw any charts in HTML or SVG. With the file, reply with a cover note to Lena of no more than 60 words.

What the model was given8 items: Ledgerly brand colours, Q3 OKRs (Finance sheet, final), What moved NRR, Launches, Customers, Team and budget, Decisions the exec team needs to make, Draft headline from the VP of Sales

Ledgerly brand coloursInk #1B2A41 (text and headlines). Teal #0F8B8D (the primary brand colour: accents, charts, on-track items). Amber #F2A541 (only for items that are at risk or missed; never decorative). Mist #E8EEF2 (backgrounds, table rules). White #FFFFFF. No other colours, no gradients. Body text at least 18px.

Q3 OKRs (Finance sheet, final)1. New ARR: target $2.40M, actual $2.61M (109%). 2. Net revenue retention: target 108%, actual 101.3% (Q2: 106.0%). 3. Activation: 45% of new sign-ups send a first invoice within 7 days; actual 38% (Q2: 36%). 4. Payments reliability: 99.95% target, actual 99.91% (one incident: 2 September, 4 hours of degraded payments during the card-fraud response).

What moved NRRPricing moved from per-seat to usage-based on 1 September. Three of the top 20 accounts downgraded in September (−$540k ARR combined), citing the new pricing. Harbor & Co ($310k ARR) churned to a competitor, citing no bulk invoicing. The Customer Success dashboard shows NRR at 102%, because it excludes churned customers under $5k ARR; Finance's 101.3% is the figure the board sees.

LaunchesBulk invoicing: promised for 12 October, now 26 October (23 October at best), because Payments paused the batch-send API during the card-fraud incident. The 14 October customer webinar (1,100 registered) has been moved to 28 October. AI-drafted support replies: live on the three non-billing queues since 21 September; median first-response time down 41%. Billing queue waits for updated pricing data and a refund-language block; eval re-run due 20 October.

CustomersNPS: 34 (Q2: 41). 62% of detractor comments mention the pricing change. 20 of the top 50 accounts renew in Q4, worth $2.1M ARR; Customer Success rates 7 of them ($900k) at risk, all citing pricing.

Team and budgetHires: 9 of 14 planned; two senior engineers left Payments in August (both regretted). Spend: 4% under budget, mostly the unfilled roles.

Decisions the exec team needs to make(a) Whether to hold the top-20 accounts on their old pricing for six months: Finance estimates it costs $420k of ARR this year; Customer Success thinks it protects most of the $900k at risk. (b) Whether to move two engineers from Growth to Payments until bulk invoicing ships, which the Payments lead says would secure 23 October; Growth would pause the activation experiments for six weeks.

Draft headline from the VP of Sales“Record quarter: best-ever new ARR, 109% of target.” (She'd like the deck to open with it.)

What a strong answer doesThe answer key the graders mark against

Opens with an honest headline for the quarter: new ARR beat target, but NRR fell to 101.3% against 108% after the pricing change, and activation and reliability missed. Does not lead with 'record quarter' on its own. Uses Finance's 101.3% (noting the CS dashboard's different definition if it mentions 102%). Shows each OKR with a plain status, amber only on the misses. Covers the NRR drivers (the three downgrades and Harbor & Co), the bulk-invoicing slip to 26 October (23 at best), NPS 41 to 34, and the $900k renewal risk. Closes with the two decisions, framed with their trade-offs, each with an owner and a date before the Q4 renewals. At most 7 slides, takeaway headlines, the brand palette only, and a short cover note.

Critical failures (cap the score at 40)

Reports the NRR, activation or reliability OKR as met or on track
Opens the deck with the 'record quarter' headline and omits the retention drop from the opening

Task

Staff-level · v1.12

The outputs

8 setups answered this brief.

GPT-6 AstraChatGPT92Usable with a quick edit Haiku 5.5API92Usable with a quick edit GPT-6.1 SolAPI84Usable with a quick edit Opus 5.5Claude83Usable with a quick edit Sonnet 5.5API81Usable with a quick edit GPT-6 LunaAPI66Needs more work Gemini 3.5 Flash-LiteGemini63Needs more work Gemini 3.8 FlashAPI53Needs more work

Haiku 5.5API

Usable with a quick edit14 of 15 checks passed

All facts are correct, the deck meets every requirement, and the cover note sets context honestly. A PM could use it with only a light proofread.

Output

Open the deckThe file the model delivered, in a new tab

Lena, this deck reports all four Q3 OKRs, including the three misses, and opens with the NRR and bulk-invoicing story. I didn't use the "record quarter" headline because the Finance sheet doesn't support "best-ever." New ARR beat target by 9%, and the deck says so. Owners and dates on slide 7 are my proposals and need confirming on 8 October.

Grades and run details

Decision model 83 · LLM judge 100

Decision model checks

passUses the supplied evidence correctly48%
failAddresses the actual decision25%
passRespects explicit constraints4%
partialIdentifies material uncertainty51%
partialAvoids unsupported claims41%
passProduces the required deliverable43%
passLeads with the honest headline97%
passMakes the decisions clear100%
passHeadlines state the takeaway58%
partialOwns the problem79%
passReady for the reader as delivered56%
passAsks leaders only for the big calls49%
passReports outcomes and learning, not just output85%
passA real 7-slide deckby hand100%
passUses only the brand coloursby hand100%

Artefacts

Deck (html)

Run

Run: #1
API response time: 48 s
Submitted: 8 Oct 2026

Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit9 of 15 checks passed

The cover note contains a factual error ('Q1 pricing transition') and several slides present unsupported causes as fact. A PM would need to re-check evidence and correct these before use.

Output

Open the deckThe file the model delivered, in a new tab

Cover Note to Lena Lena, here is the Q3 Billing pre-read deck for our Oct 8 review. While New ARR hit a record $2.61M (109%), the Q1 pricing transition pulled NRR down to 101.3%¹ and created $900k in Q4 renewal risk. We need two critical decisions on Oct 9³ regarding account pricing holds and engineering reallocation to protect Q4. See you on the 8th.

What a PM had to fix

1Invented evidenceVerify or remove the claimSubstantial rework
Source: What moved NRRPricing moved from per-seat to usage-based on 1 September.
What we’d changeThe pricing change was on 1 September, not in Q1. Correct the dates and launch status, and remove the causes the material doesn't give.
2Invented evidenceVerify or remove the claimTargeted repair
What we’d changeSlide 3 blames the activation miss on delayed bulk invoicing and pricing confusion. The material gives no cause, and the miss needs what's been tried and a way forward.
3Invented evidenceVerify or remove the claimQuick edit
Source: BriefThe quarterly business review with Lena (CEO) and the exec team is on 8 October
What we’d changeThe review is on 8 October, so that's when the decisions are needed.

Claims the judge couldn’t find in the brief

the Q1 pricing transition pulled NRR down to 101.3%
Driven by strong outbound momentum and enterprise expansion before the September pricing transition
Activation missed target due to onboarding friction and incident impact / delayed bulk invoicing and pricing transition confusion
Headcount is 5 roles under budget following two regretted engineering departures
Growth engineering team currently has capacity that could be reallocated

Grades and run details

Decision model 67 · LLM judge 57

Decision model checks

failUses the supplied evidence correctly16%
failAddresses the actual decision18%
partialRespects explicit constraints5%
partialIdentifies material uncertainty32%
partialAvoids unsupported claims49%
passProduces the required deliverable50%
passLeads with the honest headline76%
passMakes the decisions clear100%
passHeadlines state the takeaway79%
passA real 7-slide deckby hand100%
passUses only the brand coloursby hand100%
partialOwns the problem52%
partialReady for the reader as delivered14%
partialAsks leaders only for the big calls24%
passReports outcomes and learning, not just output64%

Artefacts

Deck (html)

Run

Run: #1
Time to output: 20 s
Submitted: 27 Sept 2026

Sonnet 5.5API

Usable with a quick edit12 of 15 checks passed

The deck is complete, honest, and well-structured. A quick edit to add a line about actions taken on reliability and activation would make it fully ready.

Output

Open the deckThe file the model delivered, in a new tab

Lena, the deck is above. It doesn't open with "record quarter": new ARR beat target (109%), but three of four OKRs missed, and I can't verify "best-ever" from this data. NRR uses Finance's 101.3%, not CS's 102%. Decision owners and dates are my proposals; please confirm before the 7 October pre-read.

Grades and run details

Decision model 73 · LLM judge 93

Decision model checks

passUses the supplied evidence correctly45%
failAddresses the actual decision58%
passRespects explicit constraints62%
partialIdentifies material uncertainty75%
passAvoids unsupported claims64%
passProduces the required deliverable8%
passLeads with the honest headline57%
passMakes the decisions clear15%
passHeadlines state the takeaway77%
passA real 7-slide deckby hand100%
failUses only the brand coloursby hand100%
failOwns the problem49%
passReady for the reader as delivered22%
partialAsks leaders only for the big calls9%
passReports outcomes and learning, not just output44%

Artefacts

Deck (html)

Run

Run: #1
API response time: 2.6 min
Submitted: 29 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 6

Uses the supplied evidence correctlyRightWrongRight

Haiku 5.5 · API

Every claim about the current situation is drawn accurately from the supplied context, with no invented facts.

Gemini 3.5 Flash-Lite · Gemini

Multiple unsupported claims: 'Q1 pricing transition' (should be Q3), causes for activation miss, outbound momentum, Growth capacity, and causal link in headcount headline.

Sonnet 5.5 · API

All factual claims about the current situation are directly supported by the brief or supplied context.

Identifies material uncertaintyRightWrongRight

Haiku 5.5 · API

The deck flags that the cost and benefit estimates for decision (a) are unverified, and the 23 October date is described as the best case, acknowledging uncertainty.

Gemini 3.5 Flash-Lite · Gemini

The output does not name specific unknowns that could change the decisions or say how they would be resolved.

Sonnet 5.5 · API

It names the key unknown for decision (a) (whether most of the $900k sits in the top 20) and notes CS judgement is not a churn forecast.

Avoids unsupported claimsRightWrongRight

Haiku 5.5 · API

All interpretations and forecasts are directly supported by the data or clearly labelled as proposals or conditions.

Gemini 3.5 Flash-Lite · Gemini

Interpretations like 'driven by outbound momentum', 'due to onboarding friction', and 'Growth has capacity' are presented as fact without being labelled as hypotheses.

Sonnet 5.5 · API

Interpretations like the pricing change driving the retention miss are supported by the downgrades and detractor comments, and the deck acknowledges both causes.

Uses only the brand coloursRightRightWrong

Haiku 5.5 · API

No reason given.

Gemini 3.5 Flash-Lite · Gemini

No reason given.

Sonnet 5.5 · API

No reason given.

Owns the problemRightWrongWrong

Haiku 5.5 · API

For each problem (NRR, activation, reliability, delays), the deck describes ongoing actions (activation experiments, AI replies, bulk-invoicing plan) and recommends decisions with trade-offs.

Gemini 3.5 Flash-Lite · Gemini

The output reports problems but does not say what the team has already done about them or recommend a specific way forward beyond escalating the two decisions.

Sonnet 5.5 · API

The deck does not state what the team has already done about the reliability miss or the activation miss, only recommending future actions.

Asks leaders only for the big callsRightWrongRight

Haiku 5.5 · API

The deck clearly asks the exec team for the two high-stakes decisions and notes that the team will handle other moves (e.g., AI reply rollout).

Gemini 3.5 Flash-Lite · Gemini

The update does not separate the decisions it needs from leaders from those the team will make itself; it only presents the two big asks.

Sonnet 5.5 · API

It asks for the two high-stakes decisions from the exec team and does not mix in team-level calls.

All mixed 1

Addresses the actual decisionMixedMixedMixed

Haiku 5.5 · API

The deck ends with the two required decisions, each with trade-offs, an owner and a date, addressing the exec team's call as requested.

Gemini 3.5 Flash-Lite · Gemini

The output clearly lists the two decisions the exec team must make, with trade-offs, owners, and dates, as requested.

Sonnet 5.5 · API

The deck commits to clear recommendations for both decisions, framed for Lena, with conditions that would change the call.

All got right 8

Respects explicit constraintsRightRightRight

Haiku 5.5 · API

The visible text respects the 7-slide limit, takeaway headlines, and brand colors; colors cannot be checked in this text review but were not penalized.

Gemini 3.5 Flash-Lite · Gemini

All explicit constraints (7 slides, takeaway headlines, honest OKR reporting, decisions with owner/date, brand colours, cover note ≤60 words) are met.

Sonnet 5.5 · API

The deck has 7 slides, takeaway headlines, honest OKR reporting, decisions with owners and dates, and the cover note is under 60 words.

Produces the required deliverableRightRightRight

Haiku 5.5 · API

The self-contained HTML file and a 60-word cover note are delivered as requested, making the deck and cover usable by the CEO.

Gemini 3.5 Flash-Lite · Gemini

The HTML file and cover note are present, complete, and usable for the QBR pre-read.

Sonnet 5.5 · API

The self-contained HTML file with 7 slides and the cover note are present and complete.

Leads with the honest headlineRightRightRight

Haiku 5.5 · API

The first content slide headline states the New ARR beat and the three missed OKRs, including the NRR drop, rather than only the new-ARR win.

Gemini 3.5 Flash-Lite · Gemini

The first content slide headline includes both the record ARR and the NRR drop, not just the win.

Sonnet 5.5 · API

The first content slide headline includes the NRR miss and the other misses alongside the new-ARR beat.

Makes the decisions clearRightRightRight

Haiku 5.5 · API

Slide 7 presents both decisions with their trade-offs, each with a specific owner and a date before the Q4 renewals.

Gemini 3.5 Flash-Lite · Gemini

The deck ends with both decisions, each with its trade-off, an owner, and a date before Q4 renewals.

Sonnet 5.5 · API

Both decisions are presented with trade-offs, an owner (Lena) and dates (8 Oct and 15 Oct) before the Q4 renewals.

Headlines state the takeawayRightRightRight

Haiku 5.5 · API

Every slide headline is a takeaway sentence (e.g., 'NRR fell to 101.3% after the 1 September pricing change, 6.7 points below target'), not a topic label.

Gemini 3.5 Flash-Lite · Gemini

Every slide headline is a takeaway sentence, not a topic label.

Sonnet 5.5 · API

Every slide headline is a takeaway sentence, not a topic label.

A real 7-slide deckRightRightRight

Haiku 5.5 · API

No reason given.

Gemini 3.5 Flash-Lite · Gemini

No reason given.

Sonnet 5.5 · API

No reason given.

Ready for the reader as deliveredRightRightRight

Haiku 5.5 · API

The deck and cover note are written directly for Lena, using only provided data, with proposed owners and dates clearly marked as needing confirmation.

Gemini 3.5 Flash-Lite · Gemini

The deck is written for Lena, with no placeholders or notes to the requester; proposed owners and dates are clearly marked.

Sonnet 5.5 · API

The deck is written for Lena and the exec team, with owners and dates clearly marked as proposed, and no placeholder or invented names.

Reports outcomes and learning, not just outputRightRightRight

Haiku 5.5 · API

The update leads with outcome metrics (NRR, activation, NPS) and what was learned (pricing impact, fraud incident side effects), with launches as supporting detail.

Gemini 3.5 Flash-Lite · Gemini

The deck leads with outcome metrics (ARR, NRR, activation, reliability, NPS) and shows what changed, with shipping details as supporting information.

Sonnet 5.5 · API

The deck leads with outcome metrics (OKRs, NPS) and what changed in understanding, with shipping details as supporting context.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 76% of 89 checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	GPT-6 AstrawithChatGPT	94.2	92.9	2	None
2	Sonnet 5.5withAPI	84.6	96.4	2	None
3	Haiku 5.5withAPI	89.6	92.3	2	None
4	GPT-6.1 SolwithAPI	80.8	96.2	2	None
5	Opus 5.5withClaude	86.3	77.5	2	None
6	GPT-6 LunawithAPI	76.7	66.8	2	None
7	Gemini 3.5 Flash-LitewithGemini	70.8	51.6	2	None
8	Gemini 3.8 FlashwithAPI	66.3	51.9	2	None

What you get for the cost

Haiku 5.5 · API scores within 5 points of GPT-6 Astra · ChatGPT at a tenth of the cost.

Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
Gemini 3.5 Flash-Lite · Gemini	60	1.0× (partly estimated)
GPT-6 Luna · API	71	1.3×
Haiku 5.5 · API	89	2.6×
Gemini 3.8 Flash · API	58	13×
GPT-6.1 Sol · API	88	15×
GPT-6 Astra · ChatGPT	94	26× (partly estimated)
Sonnet 5.5 · API	90	40×
Opus 5.5 · Claude	81	55× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Writing the weekly update, or the reply to an executive's question, that the team's credibility rests on.

Why it matters

Updates are where bad news gets softened. A fluent update that buries a slipped date or presents a guess as the cause does more harm than no update: leadership decides on it, and trust goes when the truth comes out.

What good looks like

Leads with the news and any decision needed
States dates, numbers and causes plainly, including bad news
Separates what is known from what is suspected
Brings a recommendation and what's been tried, not just the problem
Fits the reader and the length asked for

Deliberately not measured

Tone and formatting preferences
Writing style beyond clarity

Capability tested

Honest, decision-first status communication

The failure we’re looking for

Spin: burying or softening the bad news, or presenting a guess as fact

Grading

Decision model, LLM judge and a browser check (for the deck), calibrated against a blind PM review

Variants

A short written update · Staff level: a QBR deck in brand colours

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

Asks leaders only for the big calls

Reports outcomes and learning, not just output

Plus our standard checks

The tasks

The outputs

Output

Grades and run details

Decision model checks

Artefacts

Run

Output

What a PM had to fix

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Artefacts

Run

Output

Grades and run details

Decision model checks

Artefacts

Run

Check by check

Where they differ 6

All mixed 1

All got right 8

Results

What you get for the cost

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured