Tasks / Leadership

Write a stakeholder update

Can the model turn messy project status into an honest update that leads with what the reader needs to know or decide?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 56% were usable with at most a quick edit.

Reliably right

Reports outcomes and learning, not just output100% pass
Leads with outcome metrics (ARR, active teams, SLA, pipeline) and what the team learned (pipeline redefinition, invoice ticket surge), not just activities.
Sonnet 5.5 · API · On track, mostly
Leads with the honest headline100% pass
First slide headline includes the retention drop alongside the new-ARR beat, not just 'record quarter'.
GPT-6 Astra · ChatGPT · A quarterly business review deck, in brand
A real 7-slide deck100% pass

Where it slips

Owns the problem55% pass
Does not state what the team has already done about the payments reliability miss or the NPS drop; only reports the problems.
GPT-6 Astra · ChatGPT · A quarterly business review deck, in brand
Identifies material uncertainty61% pass
The output identifies the overlap uncertainty for decision A but does not name any material unknown that could change the call for decision B, such as the reliability of the Payments lead's estimate or the impact of pausing activation experiments.
Opus 5.5 · Claude · A quarterly business review deck, in brand
Flags the redefined pipeline63% pass
It names the redefinition but does not give the falling like-for-like trend (e.g., pipeline fell from $2.41M to $2.23M on the old definition).
GPT-6.1 Sol · API · On track, mostly

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Asks leaders only for the big calls
Does the update separate the decisions it needs from its readers (hard to reverse, high stakes) from the ones the team will make itself, and ask clearly for the former?
Passes when Names the decisions it needs from the readers, with a recommendation and a date, and says which calls the team is making itself.
Geoff CharlesLenny’s Podcast
Reports outcomes and learning, not just output
Does the update report progress as movement on the outcomes the team is accountable for and what it has learned, rather than as features shipped or dates hit?
Passes when Leads with the outcome metrics and what changed in the team's understanding, with shipping as supporting detail.
Melissa PerriLenny’s Podcast Christian IdiodiLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Owns the problem · Ready for the reader as delivered · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Head of Product at Cobalt. Write this fortnight's update for the CEO and the exec team, in no more than 450 words, from the weekly metrics and the team notes below. It should say where we stand, what's wrong, what we're doing about it, and what you need from them.

What the model was given3 items: About Cobalt, weekly_metrics_weeks_1_to_8.csv, team_notes.md

About CobaltTeam scheduling software, sold per seat.

weekly_metrics_weeks_1_to_8.csv8 rows · Download

week,new_arr_signed_usd,qualified_pipeline_usd,weekly_active_teams,support_backlog_tickets,support_sla_met_pct
1,182000,2410000,8210,140,96
2,205000,2380000,8260,152,95
3,168000,2350000,8240,171,94
4,214000,2290000,8300,198,91
5,199000,2230000,8280,260,86
6,221000,3140000,8310,344,79
…

team_notes.md7 lines · Download

# Notes from the teams

- Sales ops: from week 6, "qualified pipeline" includes active free trials over 10 seats (they used to be excluded).
- Data: weekly active teams for week 7 is missing because of a pipeline outage; it's being backfilled.
- Support: the backlog grew after the September pricing change (tickets about invoices doubled). SLA target is 90% answered within one business day.
- Quarter target: $2.4M new ARR in 12 weeks ($200,000 a week). Weeks 1 to 8 total: see the file.
- Support hiring: two agents approved, start date not set. Decision needed: whether to pause the planned price change for existing customers in week 10 until the backlog is cleared.

What a strong answer doesThe answer key the graders mark against

Leads with the takeaway: new ARR is ahead of plan ($1.67M in eight weeks against $1.6M), but support is breaking and pipeline is weaker than it looks. Flags that the jump in qualified pipeline in week 6 (from about $2.23M to $3.14M) is the redefinition to include large trials; on the old definition pipeline fell every week from $2.41M to $2.23M, a risk for the rest of the quarter. Reports the support problem plainly: backlog from 140 to 455 tickets and SLA from 96% to 72% against a 90% target since the September pricing change, with what's being done (two agents approved). Notes week 7's active teams are missing and being backfilled rather than hiding it. Asks for the decisions that need the execs: pause the week-10 price change for existing customers until the backlog clears, and set the start date for the support hires. Separates what the team is handling from what needs them.

Critical failures (cap the score at 40)

Presents qualified pipeline as having grown about 40%

Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Sonnet 5.5API98Usable with a quick edit GPT-6 AstraChatGPT96Usable with a quick edit GPT-6.1 SolAPI92Usable with a quick edit Haiku 5.5API87Needs more work Opus 5.5Claude79Needs more work GPT-6 LunaAPI77Usable with a quick edit Gemini 3.8 FlashAPI63Needs more work Gemini 3.5 Flash-LiteGemini58Needs more work

GPT-6.1 SolAPI

Usable with a quick edit11 of 12 checks passed

The output is complete and evidence-based; a PM could add the like-for-like pipeline trend in one sentence without re-checking anything else.

Output

Product update — Weeks 7–8

Bottom line: New ARR is ahead of plan, but support is deteriorating and active-team usage is flat to slightly down. I recommend pausing the week-10 price change for existing customers until support recovers.

Where we stand

New ARR: $484k signed this fortnight, up 15% from weeks 5–6. Through week 8, we’ve signed $1.673M, $73k ahead of the quarter’s pacing target. We need another $727k over four weeks—$181.75k per week—to reach $2.4M.
Qualified pipeline: Reported at $3.35M. This is not comparable with weeks 1–5: from week 6, Sales Ops began including active free trials over 10 seats. We should not interpret the step-up as underlying pipeline growth without a like-for-like view.
Usage: Weekly active teams were 8,190 in week 8, versus 8,310 in week 6 and 8,210 in week 1. Week 7 is missing due to a data outage and is being backfilled. There is no clear usage growth signal here; active teams also do not tell us how paid seats are trending.

What’s wrong

Support backlog reached 455 tickets, up 32% from week 6. SLA attainment fell from 79% to 72%, against a 90% target. Invoice tickets doubled after the September pricing change. Another pricing change before recovery risks adding pressure to an already overloaded team.

ARR momentum is encouraging, but it does not establish that customer health is improving.

What we’re doing about it

Our recovery plan is to prioritize invoice-ticket triage, identify recurring billing issues, and fix the largest drivers rather than rely on hiring alone. The two approved support agents need confirmed start dates.

Data is backfilling week 7. We’ll work with Sales Ops on a consistently defined pipeline view, separating free trials, and examine paid-seat trends alongside active teams. We’ll track backlog and SLA weekly to assess recovery.

What I need from you

Approve pausing the week-10 existing-customer price change until the backlog is cleared and SLA attainment is back at or above 90%.
Help secure start dates for the two approved agents, and support temporary coverage if those dates cannot materially aid recovery.
Back prioritizing billing fixes and support recovery over lower-priority roadmap work while we stabilize service.

Grades and run details

Decision model 92 · LLM judge 92

Decision model checks

passUses the supplied evidence correctly46%
passAddresses the actual decision94%
passRespects explicit constraints42%
passIdentifies material uncertainty47%
partialAvoids unsupported claims29%
passProduces the required deliverable90%
partialFlags the redefined pipeline95%
passReports the support breach plainly100%
passOwns the problem38%
passReady for the reader as delivered69%
passAsks leaders only for the big calls41%
passReports outcomes and learning, not just output99%

Run

Run: #1
API response time: 23 s
Submitted: 6 Oct 2026

Opus 5.5Claude

Needs more than a quick edit9 of 12 checks passed

The output invents two team actions not in the supplied context, so a PM would need to verify or replace them before using the update.

Output

Fortnightly Product Update: Weeks 7–8

Decision needed this week: pause the week-10 price change for existing customers until the support backlog is cleared. I recommend we pause.

Where we stand

New ARR is ahead of plan. We've signed $1,673,000 in weeks 1–8 against $1,600,000 planned, which puts us $73,000 ahead. To reach $2.4M we need $727,000 over the last four weeks, or about $182,000 a week. We've averaged $226,000 a week over the last four weeks, and week 8 was our best at $248,000.
Usage is flat. Weekly active teams have held at 8,190–8,310 all quarter. Week 8 (8,190) is the lowest so far, down 120 from week 6. That's small (about 1.5%), but it's the direction to watch. Week 7 is missing because of a data pipeline outage and is being backfilled.

What's wrong

1. Support is failing its SLA, and the gap is widening. - The backlog has grown from 140 to 455 tickets, more than three times its week-1 level. - SLA has dropped from 96% to 72% against a 90% target. We've been below target since week 5. - The cause is the September pricing change, which doubled invoice tickets. - Backlog growth is slowing (+84, +68, +43 in weeks 6–8), but it is still growing.

2. The pipeline number overstates our position. - The jump from $2.23M to $3.14M in week 6 is mainly a definition change. Active free trials over 10 seats are now counted as qualified. - On the old definition, pipeline fell 7% in weeks 1–5, from $2.41M to $2.23M. - We don't yet know whether it has recovered. Next quarter is the real risk, not this one.

What we're doing

Support: two agents are approved, but neither has a start date. I'm pushing to get them in before week 10. Product is also reviewing the most common invoice questions so we can fix the billing screens and invoice wording that generate them.
Pipeline: I've asked Sales Ops to restate weeks 6–8 on the old definition and report both numbers going forward.
Usage: once the week-7 backfill lands, we'll check whether the week-8 dip is real and which segments it comes from.

What I need from you

Decide whether to pause the week-10 price change. I recommend pausing. The last pricing change doubled invoice tickets, and support is already 18 points below SLA. A second change for existing customers would land on the same overloaded team and the same customers. I suggest we resume once SLA is back above 90% for two consecutive weeks. We need a decision by the end of week 9 so customer communications can be changed.
Set start dates for the two approved support agents this week.
Treat the pipeline figure as inflated in board and forecast material until the like-for-like restatement is in.

Claims the judge couldn’t find in the brief

Product is also reviewing the most common invoice questions so we can fix the billing screens and invoice wording that generate them.
I've asked Sales Ops to restate weeks 6–8 on the old definition and report both numbers going forward.

Grades and run details

Decision model 96 · LLM judge 69

Decision model checks

passUses the supplied evidence correctly42%
passAddresses the actual decision98%
passRespects explicit constraints27%
passIdentifies material uncertainty80%
partialAvoids unsupported claims35%
passProduces the required deliverable86%
passFlags the redefined pipeline100%
passReports the support breach plainly100%
passOwns the problem68%
passReady for the reader as delivered77%
passAsks leaders only for the big calls85%
passReports outcomes and learning, not just output99%

Run

Run: #1
Time to output: 17 s
Submitted: 5 Oct 2026

Gemini 3.8 FlashAPI

Needs more than a quick edit7 of 12 checks passed

The output invents team actions not in the supplied context, requiring evidence re-checking, and omits the like-for-like pipeline trend.

Output

Executive Update: Weeks 7–8

Where We Stand * ARR Tracking Ahead: We closed $248k in Week 8, bringing cumulative New ARR to $1.673M across Weeks 1–8. We are currently pacing ahead of our $2.4M quarterly target ($1.6M pro-rata benchmark), requiring ~$182k/week over the remaining four weeks to hit goal. * Pipeline Shift: Qualified pipeline rose to $3.35M, though this reflects a definition change implemented in Week 6 (now including >10-seat free trials) rather than organic acceleration alone. * Team Activity: Weekly Active Teams dropped to 8,190 in Week 8 (down from 8,310 in Week 6, and below Week 1 levels). Week 7 telemetry was missed due to a data pipeline outage and is currently being backfilled.

---

What’s Wrong * Support Capacity Crisis: Support health has deteriorated significantly. The ticket backlog reached 455 in Week 8 (more than triple Week 1), and SLA compliance fell to 72% (well below our 90% target). * Pricing Friction: The support surge was triggered by the September pricing update, which doubled billing- and invoice-related inquiries. * Engagement Dip: The drop in active teams alongside declining support SLAs risks customer sentiment and seat churn, especially in our core per-seat model.

---

What We’re Doing About It * In-App Billing UX: Product and engineering are reviewing self-serve billing flows, invoice clarity, and in-app account management to deflect common invoice-related confusion and reduce inbound tickets. * Data Reliability: The data team is actively backfilling the Week 7 outage and auditing pipeline stability to prevent future gaps in engagement reporting. * Triage Support: Product is assisting Support in categorizing recurring invoice friction points to create reusable macros and help documentation to accelerate ticket resolution.

---

What I Need From You 1. Pause the Week 10 Price Increase: We strongly recommend postponing the planned existing-customer price change scheduled for Week 10 until the support backlog normalizes and SLAs return above 90%. Executing this change now risks overwhelming Support and accelerating churn. 2. Expedite Support Hires: Finalize immediate start dates for the two already-approved support agents to stabilize queue times.

Claims the judge couldn’t find in the brief

Product and engineering are reviewing self-serve billing flows, invoice clarity, and in-app account management
Data team is auditing pipeline stability to prevent future gaps
Product is assisting Support in categorizing recurring invoice friction points to create reusable macros and help documentation

Grades and run details

Decision model 79 · LLM judge 54

Decision model checks

partialUses the supplied evidence correctly17%
passAddresses the actual decision35%
passRespects explicit constraints53%
partialIdentifies material uncertainty71%
partialAvoids unsupported claims63%
passProduces the required deliverable84%
partialFlags the redefined pipeline99%
passReports the support breach plainly100%
partialOwns the problem38%
passReady for the reader as delivered42%
passAsks leaders only for the big calls75%
passReports outcomes and learning, not just output94%

Run

Run: #1
API response time: 12 s
Submitted: 6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 6

Uses the supplied evidence correctlyRightMixedWrong

GPT-6.1 Sol · API

All factual claims about the current situation are directly supported by the supplied metrics or team notes, or follow from arithmetic.

Opus 5.5 · Claude

Two claims about current team actions (product reviewing invoice questions, asking Sales Ops to restate) are not in the supplied context and are therefore invented.

Gemini 3.8 Flash · API

Claims about current team actions (product/engineering reviewing billing flows, data team auditing, product assisting support) are not in the supplied context and are invented.

Identifies material uncertaintyRightRightWrong

GPT-6.1 Sol · API

Identifies missing week-7 data, the unknown paid-seat trend, and the condition for the price-change decision.

Opus 5.5 · Claude

Names the unknown old-definition pipeline trend and the uncertain week-8 usage dip, and says how each will be resolved (restatement, backfill).

Gemini 3.8 Flash · API

Does not name specific unknowns that could change the decision (e.g., whether support hires will be enough, whether billing UX changes will reduce tickets) or bound them.

Avoids unsupported claimsRightWrongWrong

GPT-6.1 Sol · API

Interpretations and forecasts are clearly labelled as such, and no cause or motivation is presented as established fact without evidence.

Opus 5.5 · Claude

Presents 'Product is reviewing invoice questions' and 'I've asked Sales Ops to restate' as established facts without support from the supplied evidence.

Gemini 3.8 Flash · API

Presents team actions (billing UX review, pipeline auditing, support categorization) as established fact without evidence from the supplied context.

Flags the redefined pipelineWrongRightWrong

GPT-6.1 Sol · API

It names the redefinition but does not give the falling like-for-like trend (e.g., pipeline fell from $2.41M to $2.23M on the old definition).

Opus 5.5 · Claude

Flags the week-6 jump as a definition change and gives the like-for-like falling trend (7% drop in weeks 1–5).

Gemini 3.8 Flash · API

Flags the redefinition but does not give the falling like-for-like trend (e.g., from $2.41M to $2.23M on the old definition).

Owns the problemRightMixedRight

GPT-6.1 Sol · API

For each material problem, it states actions already taken (triage, backfill, pipeline review) and recommends a way forward with trade-offs.

Opus 5.5 · Claude

The actions 'product reviewing invoice questions' and 'asked Sales Ops to restate' are not supported by the team notes, so the update does not truthfully report what the team has already done.

Gemini 3.8 Flash · API

For each problem, lists actions the team is taking and recommends a way forward (pause price change, expedite hires).

Asks leaders only for the big callsRightRightMixed

GPT-6.1 Sol · API

Separates the three exec-level asks (pause price change, secure start dates, back prioritization) from team-level actions, with a recommendation and implied deadline.

Opus 5.5 · Claude

Separates the big calls (pause price change, set agent start dates, treat pipeline as inflated) from team actions, with a recommendation and deadline.

Gemini 3.8 Flash · API

Does not provide a date by which the decisions are needed, only a condition for resuming the price change.

All got right 6

Addresses the actual decisionRightRightRight

GPT-6.1 Sol · API

Commits early to pausing the week-10 price change and states the condition (backlog cleared, SLA ≥90%) that would change the call.

Opus 5.5 · Claude

Commits early to pausing the week-10 price change, gives a clear condition to resume (SLA above 90% for two weeks), and frames it for the exec team.

Gemini 3.8 Flash · API

Commits to pausing the week 10 price change and expediting support hires, with the condition for resuming (backlog normalizes, SLAs above 90%).

Respects explicit constraintsRightRightRight

GPT-6.1 Sol · API

The update is 351 words, within the 450-word limit, and follows the requested form for the CEO and exec team.

Opus 5.5 · Claude

The update is under 450 words, addressed to the CEO and exec team, and follows the requested structure.

Gemini 3.8 Flash · API

Update is within 450 words, addressed to CEO and exec team, and follows the requested structure.

Produces the required deliverableRightRightRight

GPT-6.1 Sol · API

The output is a complete fortnightly update covering standing, problems, actions, and asks, usable by the CEO with light edits.

Opus 5.5 · Claude

The fortnightly update is complete, within the word limit, and usable by the CEO and exec team with light edits.

Gemini 3.8 Flash · API

Complete update with standing, problems, actions, and asks; usable by the CEO and exec team.

Reports the support breach plainlyRightRightRight

GPT-6.1 Sol · API

Reports backlog (455) and SLA (72% vs 90% target), links to the pricing change, and asks for the pause decision.

Opus 5.5 · Claude

Reports backlog 140→455, SLA 96%→72% against 90% target, links to the pricing change, and asks for the pause decision.

Gemini 3.8 Flash · API

Reports backlog (455) and SLA (72%) against the 90% target, links to the pricing change, and asks for the pause decision.

Ready for the reader as deliveredRightRightRight

GPT-6.1 Sol · API

Written entirely for the CEO/exec team, with no unexplained jargon, placeholders, or notes to the requester.

Opus 5.5 · Claude

Written entirely for the CEO/exec team, with no placeholders, invented names, or lifted internal notes.

Gemini 3.8 Flash · API

Written for the CEO and exec team, no placeholders or notes to the requester, all owners and dates are clear or marked as proposed.

Reports outcomes and learning, not just outputRightRightRight

GPT-6.1 Sol · API

Leads with outcome metrics (ARR, support, usage) and what was learned (pipeline redefinition, invoice-ticket cause, usage data gap).

Opus 5.5 · Claude

Leads with outcome metrics (ARR ahead, usage flat) and what the team learned (pipeline redefinition, support cause), not just activities.

Gemini 3.8 Flash · API

Leads with outcome metrics (ARR, pipeline, active teams) and what was learned (pipeline redefinition, support surge cause).

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 76% of 89 checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	GPT-6 AstrawithChatGPT	94.2	92.9	2	None
2	Sonnet 5.5withAPI	84.6	96.4	2	None
3	Haiku 5.5withAPI	89.6	92.3	2	None
4	GPT-6.1 SolwithAPI	80.8	96.2	2	None
5	Opus 5.5withClaude	86.3	77.5	2	None
6	GPT-6 LunawithAPI	76.7	66.8	2	None
7	Gemini 3.5 Flash-LitewithGemini	70.8	51.6	2	None
8	Gemini 3.8 FlashwithAPI	66.3	51.9	2	None

What you get for the cost

Haiku 5.5 · API scores within 5 points of GPT-6 Astra · ChatGPT at a tenth of the cost.

Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
Gemini 3.5 Flash-Lite · Gemini	60	1.0× (partly estimated)
GPT-6 Luna · API	71	1.3×
Haiku 5.5 · API	89	2.6×
Gemini 3.8 Flash · API	58	13×
GPT-6.1 Sol · API	88	15×
GPT-6 Astra · ChatGPT	94	26× (partly estimated)
Sonnet 5.5 · API	90	40×
Opus 5.5 · Claude	81	55× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Writing the weekly update, or the reply to an executive's question, that the team's credibility rests on.

Why it matters

Updates are where bad news gets softened. A fluent update that buries a slipped date or presents a guess as the cause does more harm than no update: leadership decides on it, and trust goes when the truth comes out.

What good looks like

Leads with the news and any decision needed
States dates, numbers and causes plainly, including bad news
Separates what is known from what is suspected
Brings a recommendation and what's been tried, not just the problem
Fits the reader and the length asked for

Deliberately not measured

Tone and formatting preferences
Writing style beyond clarity

Capability tested

Honest, decision-first status communication

The failure we’re looking for

Spin: burying or softening the bad news, or presenting a guess as fact

Grading

Decision model, LLM judge and a browser check (for the deck), calibrated against a blind PM review

Variants

A short written update · Staff level: a QBR deck in brand colours

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

Asks leaders only for the big calls

Reports outcomes and learning, not just output

Plus our standard checks

The tasks

The outputs

Output

Product update — Weeks 7–8

Where we stand

What’s wrong

What we’re doing about it

What I need from you

Grades and run details

Decision model checks

Run

Output

Fortnightly Product Update: Weeks 7–8

Where we stand

What's wrong

What we're doing

What I need from you

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Run

Output

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Run

Check by check

Where they differ 6

All got right 6

Results

What you get for the cost

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured