Tasks / Operate

Write a stakeholder update

Can the model turn messy project status into an honest update that leads with what the reader needs to know or decide?

Measures the modelTask v1.2 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 14 graded outputs by 7 models. 43% were usable with at most a quick edit.

Reliably right

Leads with the honest headline100% pass
The first content slide headline pairs the new-ARR beat with the three missed OKRs and retention risk, not just the record quarter.
GPT-6 Astra · ChatGPT · A quarterly business review deck, in brand
A real 7-slide deck100% pass
Answers the question asked96% pass
The first line directly addresses whether the new navigation caused the drop.
GPT-6.1 Sol · API · The CEO asks why a metric dropped

Where it slips

Owns the problem23% pass
It proposes future analysis and fixes but does not state any actions the team has already taken or offer trade-offs.
GPT-6.1 Sol · API · The CEO asks why a metric dropped
Addresses the actual decision41% pass
It commits to not attributing the drop but does not say what result would change that answer or when Lena will get it.
GPT-6 Luna · API · The CEO asks why a metric dropped
Identifies material uncertainty55% pass
The output does not name specific unknowns that could change the decisions or say how they would be resolved.
GPT-6 Luna · API · A quarterly business review deck, in brand

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Staff PM for Ledgerly's Billing group. The quarterly business review with Lena (CEO) and the exec team is on 8 October; the deck goes out the day before as a pre-read. Using the material below, build the Q3 deck for Billing. The deck must: - Have at most 7 slides, each one 16:9. - Give every slide a headline that states its takeaway, not a topic (“NRR fell to 101% after the pricing change”, not “NRR”). - Report every OKR honestly, including the misses. - End with the decisions you need from the exec team, each with an owner and a date. - Use only Ledgerly's brand colours (below). Deliver it as a single self-contained HTML file: the slides stacked top to bottom, each exactly 1280 × 720 pixels, no external images, fonts or scripts, readable without JavaScript. Draw any charts in HTML or SVG. With the file, reply with a cover note to Lena of no more than 60 words.

What the model was given8 items: Ledgerly brand colours, Q3 OKRs (Finance sheet, final), What moved NRR, Launches, Customers, Team and budget, Decisions the exec team needs to make, Draft headline from the VP of Sales

Ledgerly brand coloursInk #1B2A41 (text and headlines). Teal #0F8B8D (the primary brand colour: accents, charts, on-track items). Amber #F2A541 (only for items that are at risk or missed; never decorative). Mist #E8EEF2 (backgrounds, table rules). White #FFFFFF. No other colours, no gradients. Body text at least 18px.

Q3 OKRs (Finance sheet, final)1. New ARR: target $2.40M, actual $2.61M (109%). 2. Net revenue retention: target 108%, actual 101.3% (Q2: 106.0%). 3. Activation: 45% of new sign-ups send a first invoice within 7 days; actual 38% (Q2: 36%). 4. Payments reliability: 99.95% target, actual 99.91% (one incident: 2 September, 4 hours of degraded payments during the card-fraud response).

What moved NRRPricing moved from per-seat to usage-based on 1 September. Three of the top 20 accounts downgraded in September (−$540k ARR combined), citing the new pricing. Harbor & Co ($310k ARR) churned to a competitor, citing no bulk invoicing. The Customer Success dashboard shows NRR at 102%, because it excludes churned customers under $5k ARR; Finance's 101.3% is the figure the board sees.

LaunchesBulk invoicing: promised for 12 October, now 26 October (23 October at best), because Payments paused the batch-send API during the card-fraud incident. The 14 October customer webinar (1,100 registered) has been moved to 28 October. AI-drafted support replies: live on the three non-billing queues since 21 September; median first-response time down 41%. Billing queue waits for updated pricing data and a refund-language block; eval re-run due 20 October.

CustomersNPS: 34 (Q2: 41). 62% of detractor comments mention the pricing change. 20 of the top 50 accounts renew in Q4, worth $2.1M ARR; Customer Success rates 7 of them ($900k) at risk, all citing pricing.

Team and budgetHires: 9 of 14 planned; two senior engineers left Payments in August (both regretted). Spend: 4% under budget, mostly the unfilled roles.

Decisions the exec team needs to make(a) Whether to hold the top-20 accounts on their old pricing for six months: Finance estimates it costs $420k of ARR this year; Customer Success thinks it protects most of the $900k at risk. (b) Whether to move two engineers from Growth to Payments until bulk invoicing ships, which the Payments lead says would secure 23 October; Growth would pause the activation experiments for six weeks.

Draft headline from the VP of Sales“Record quarter: best-ever new ARR, 109% of target.” (She'd like the deck to open with it.)

What a strong answer doesThe answer key the graders mark against

Opens with an honest headline for the quarter: new ARR beat target, but NRR fell to 101.3% against 108% after the pricing change, and activation and reliability missed. Does not lead with 'record quarter' on its own. Uses Finance's 101.3% (noting the CS dashboard's different definition if it mentions 102%). Shows each OKR with a plain status, amber only on the misses. Covers the NRR drivers (the three downgrades and Harbor & Co), the bulk-invoicing slip to 26 October (23 at best), NPS 41 to 34, and the $900k renewal risk. Closes with the two decisions, framed with their trade-offs, each with an owner and a date before the Q4 renewals. At most 7 slides, takeaway headlines, the brand palette only, and a short cover note.

Critical failures (cap the score at 40)

Reports the NRR, activation or reliability OKR as met or on track
Opens the deck with the 'record quarter' headline and omits the retention drop from the opening

Case

Staff-level · v1.7 · synthetic · exec communication, slides, brand, staff-level

The outputs

7 setups answered this brief. Pick up to three to read side by side.

GPT-6 AstraChatGPT88Usable with a quick edit GPT-6 LunaAPI71Usable with a quick edit GPT-6.1 SolAPI70Usable with a quick edit Opus 5.5Claude70Needs more work Sonnet 5.5API66Needs more work Gemini 3.5 Flash-LiteGemini57Needs more work Gemini 3.8 FlashAPI52Needs more work

Gemini 3.8 FlashAPI

Needs more than a quick edit5 of 13 checks passed

The deck includes unsupported claims about customer forecasting difficulty and billing support friction that would need verification before sending.

Output

Open the deckThe file the model delivered, in a new tab

Lena – Q3 delivered record new ARR ($2.61M, 109% of target), but the September 1 usage pricing shift severely impacted retention. NRR fell to 101.3%, NPS dropped to 34, and $900k of Q4 renewals are at risk. The attached pre-read details our recovery plan; we require two exec decisions during Wednesday’s QBR on pricing grandfathering and engineering allocation.

Grades and run details

Decision model 35 · LLM judge 8 of 12 checks

Decision model checks

passUses the supplied evidence correctly33%
failAddresses the actual decision67%
failRespects explicit constraints14%
failIdentifies material uncertainty66%
failAvoids unsupported claims6%
failProduces the required deliverable72%
passLeads with the honest headline24%
partialMakes the decisions clear71%
failHeadlines state the takeaway25%
passA real 7-slide deckby hand100%
passUses only the brand coloursby hand100%
failOwns the problem55%
failReady for the reader as delivered24%

Artefacts

Deck (html)

Run

Run: #1
API response time: 76 s
Submitted: 1 Oct 2026

Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit6 of 13 checks passed

It needs evidence re-checking to correct the Q1/Q3 error and remove unsupported causes, not just a quick wording edit.

Output

Open the deckThe file the model delivered, in a new tab

Cover Note to Lena Lena, here is the Q3 Billing pre-read deck for our Oct 8 review. While New ARR hit a record $2.61M (109%), the Q1 pricing transition pulled NRR down to 101.3%¹ and created $900k in Q4 renewal risk. We need two critical decisions on Oct 9³ regarding account pricing holds and engineering reallocation to protect Q4. See you on the 8th.

What a PM had to fix

1Invented evidenceVerify or remove the claimSubstantial rework
Source: What moved NRRPricing moved from per-seat to usage-based on 1 September.
What we’d changeThe pricing change was on 1 September, not in Q1. Correct the dates and launch status, and remove the causes the material doesn't give.
2Invented evidenceVerify or remove the claimTargeted repair
What we’d changeSlide 3 blames the activation miss on delayed bulk invoicing and pricing confusion. The material gives no cause, and the miss needs what's been tried and a way forward.
3Invented evidenceVerify or remove the claimQuick edit
Source: BriefThe quarterly business review with Lena (CEO) and the exec team is on 8 October
What we’d changeThe review is on 8 October, so that's when the decisions are needed.

Claims the judge couldn’t find in the brief

The cover note says the Q1 pricing transition pulled NRR down.
New ARR was driven by strong outbound momentum and enterprise expansion before the pricing transition.
Activation missed due to delayed bulk invoicing and pricing transition confusion/onboarding friction.

Grades and run details

Decision model 69 · LLM judge 5 of 12 checks

Decision model checks

failUses the supplied evidence correctly14%
passAddresses the actual decision25%
failRespects explicit constraints17%
partialIdentifies material uncertainty32%
partialAvoids unsupported claims51%
passProduces the required deliverable37%
passLeads with the honest headline78%
passMakes the decisions clear100%
passHeadlines state the takeaway80%
passA real 7-slide deckby hand100%
passUses only the brand coloursby hand100%
partialOwns the problem58%
partialReady for the reader as delivered19%

Artefacts

Deck (html)

Run

Run: #1
Time to output: 20 s
Submitted: 27 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.