Tasks / Leadership

Write a performance review

Can the model write a review that is clear, fair and specific enough to change what the person does next?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 50% were usable with at most a quick edit.

Reliably right

Judges outcomes, not activity100% pass
Every rating change and keep is justified by outcomes against goals, with shipping and activity treated as context, not the result.
GPT-6 Astra · ChatGPT · Fourteen ratings, three managers
Owns the manager's part100% pass
Ana plainly acknowledges her Q3 mistake and commits to reviewing goal metrics first in every monthly 1:1.
Opus 5.5 · Claude · Shipped a lot, moved nothing
Clear and direct, with care97% pass
The note to Ravi is direct, specific about what needs to change, and framed to help him succeed; the whole pre-read is clear and actionable.
GPT-6 Astra · ChatGPT · Fourteen ratings, three managers

Where it slips

Identifies material uncertainty50% pass
The output does not name any specific unknowns that could change the rating or how they would be resolved.
Opus 5.5 · Claude · Shipped a lot, moved nothing
Notices who's underrated69% pass
It does not propose Exceeds or ask Sara to justify Meets despite Nora beating both goals; it merely keeps Meets, missing the underrating.
GPT-6 Luna · API · Fourteen ratings, three managers
Catches the inflated team78% pass
The output proposes lowering Ben from Exceeds to Meets, failing to keep Ben at Exceeds as required; Ben beat both outcome goals and should remain Exceeds.
Opus 5.5 · Claude · Fourteen ratings, three managers

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Clear and direct, with care
Does the review say plainly what needs to change, in words the person couldn't misread, while showing it is written to help them succeed?
Passes when Every main point is stated directly, with the specific change expected and the support on offer.
Kim ScottLenny’s Podcast
Judges outcomes, not activity
Does the review judge the person on the outcomes they achieved against their goals, rather than on how much they shipped or how busy they were?
Passes when Rates against the goals' outcomes, with the evidence, and treats output as context, not the result.
Product and growth leadersOn Lenny’s Podcast
Names gaps you could see
Is each area to improve described as specific, observable behaviour with what good would look like, rather than labels like 'be more strategic'?
Passes when Every gap names what the person did or didn't do, in a specific situation, and what doing it well would look like.
Product and growth leadersOn Lenny’s Podcast
Weighs the whole period
Does the review weigh the whole review period, rather than letting one recent event or one strong impression decide the verdict?
Passes when Covers the whole period in proportion, and explicitly guards against a recent event or one impression dominating.
Product and growth leadersOn Lenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 3 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Director of Product at Northgate, and you're facilitating Thursday's calibration for 14 PMs across three managers (Ravi, Sara and Tom). Write the pre-read, in no more than 1,100 words: the ratings you'd change and why, each with the evidence, the ones you'd keep, and a short note to Ravi about his ratings. Use the guidance below.\n\nThe proposed ratings and evidence are attached.

What the model was given2 items: calibration_guidance.md, proposed_ratings.csv (one row per PM, as each manager submitted it)

calibration_guidance.md7 lines · Download

# Calibration guidance

- Ratings judge outcomes against goals first, then how the work was done. Shipping is not an outcome.
- Any rating other than Meets needs written evidence of outcomes.
- A Below rating needs documented earlier feedback on the gap.
- Expected spread across the org: most PMs Meets; Exceeds for clear outperformance on outcomes.
- The calibration facilitator proposes changes; managers can respond before ratings are final.

proposed_ratings.csv (one row per PM, as each manager submitted it)pm,manager,proposed_rating,goal_1,goal_1_result,goal_2,goal_2_result,evidence_cited_by_manager Aisha,Ravi,Greatly exceeds,Raise trial conversion 8% to 11%,8.4%,Ship self-serve billing,Shipped,Shipped 9 features; great energy; leadership loves her Ben,Ravi,Exceeds,Cut onboarding time 10 to 5 days,4.5 days,Raise activation 30% to 38%,39%,Hit both goals Chloe,Ravi,Exceeds,Grow API usage 20%,+4%,Launch partner portal,Shipped,Partner portal launched on time; strong specs Dev,Ravi,Exceeds,Reduce support tickets per account 25%,-27%,NPS from 31 to 40,41,Hit both goals; mentored two APMs Emma,Ravi,Exceeds,Expansion revenue +15%,+6%,Ship usage dashboard,Shipped,Very responsive to Sales; shipped 6 Sales requests Femi,Sara,Meets,Raise weekly retention 41% to 46%,46%,Cut churn of new accounts 20%,-22%,"Hit both goals, steady" Gita,Sara,Below,Raise invoices paid online 35% to 45%,47%,Launch reminders v2 by September,"Launched, rolled back after 4 days (support not briefed)",Reminders launch in September was a mess Hugo,Sara,Meets,Ship mobile app v2,Shipped late,Mobile weekly actives +20%,+3%,Delivered a hard project Ines,Tom,Exceeds,Cut checkout drop-off 30% to 22%,21%,Launch two payment methods,Shipped,"Hit both, specific evidence" Jon,Tom,Meets,Search success 60% to 70%,66%,Ship filters,Shipped,"Good progress, more to do" Kai,Tom,Meets,Reduce refund rate 4% to 3%,3.1%,Fraud losses -20%,-24%,Solid year Lena,Tom,Below,Grow marketplace listings 25%,+9%,Launch seller analytics,Not launched,"Missed both goals; feedback given in Q2 and Q3 1:1s, documented" Mo,Tom,Meets,Cut time to first sale for new sellers 21 to 14 days,15 days,Seller NPS +5,+6,"Nearly hit, good stakeholder work" Nora,Sara,Meets,Raise payment success 92% to 95%,95.4%,Cut payment support tickets 30%,-35%,"Quiet, delivered"

What a strong answer doesThe answer key the graders mark against

Judges each PM on outcomes against their goals, as the guidance says. Ravi rated all five of his PMs Exceeds or above, but only Ben and Dev hit both goals; Aisha (Greatly exceeds) missed her conversion goal (8.4% against 11%), Chloe missed API usage (+4% against +20%) and Emma missed expansion (+6% against +15%), and his evidence is output ('shipped 9 features', 'responsive to Sales'). Proposes Aisha, Chloe and Emma at Meets (or lower for Chloe, whose outcome goal was far off), keeps Ben and Dev at Exceeds. Gita (Sara, Below) beat her main goal (47% against 45%) and the Below rests on one September launch, with no documented feedback, which the guidance requires: Meets, with the launch-readiness gap documented now. Nora (Sara, Meets) beat both goals: a candidate for Exceeds. Lena (Tom, Below) has documented feedback and missed both goals: Below stands. Hugo shipped late and missed his outcome: Meets is generous but defensible, or Below needs documentation first. The note to Ravi is direct and respectful: rate on outcomes, not activity, with what evidence he'd need.

Critical failures (cap the score at 40)

Keeps Aisha at Greatly exceeds

Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

GPT-6 AstraChatGPT100Usable with a quick edit GPT-6.1 SolAPI100Usable with a quick edit Haiku 5.5API100Usable with a quick edit Gemini 3.8 FlashAPI98Usable with a quick edit Opus 5.5Claude88Needs more work GPT-6 LunaAPI82Needs more work Sonnet 5.5API77Needs more work Gemini 3.5 Flash-LiteGemini38Needs more work

Haiku 5.5API

Usable with a quick edit13 of 13 checks passed

The pre-read gives specific changes, evidence, and a clear note to Ravi; it can be used with light edits and does not require re-checking facts.

Output

Calibration pre-read: 14 PMs, Ravi / Sara / Tom

Purpose: Thursday's calibration. Managers can respond to these proposals before ratings are final.

Rules I applied (from calibration guidance): - Ratings start from outcomes against goals. Shipping is an output, and it doesn't count as an outcome. - Any rating other than Meets needs written outcome evidence. - Below needs documented earlier feedback on the gap. Where that's missing, I've capped the rating at Meets. - Working definition for Exceeds (for discussion): both outcome goals beaten. One beat and one hit is Meets.

Summary: 6 changes, 1 hold, 7 keeps. Proposed spread after changes: 3 Exceeds, 10 Meets, 1 Below.

---

Ratings I'd change

Aisha (Ravi): Greatly exceeds → Meets - Trial conversion reached 8.4% against an 11% target. That's about 13% of the intended 3-point lift. - "Shipped self-serve billing" is an output. Nine features shipped is also an output. - "Great energy" and "leadership loves her" are not outcome evidence. - Greatly exceeds is not supported. Below would need documented feedback on the conversion gap, which the submission doesn't cite.

Chloe (Ravi): Exceeds → Meets - API usage grew 4% against a 20% target, which is one-fifth of the goal. - The partner portal shipped on time. That's an output, and it doesn't offset the miss. - "Strong specs" describes how the work was done, not what it achieved. - Ravi should show what the portal changed in usage, if anything. If the outcome is attributable and I've missed it, I'll revisit.

Emma (Ravi): Exceeds → Meets - Expansion revenue grew 6% against a 15% target, which is 40% of the goal. - The usage dashboard shipped, and six Sales requests were delivered. Both are outputs. Responsiveness to Sales is a good working habit, but it isn't an outcome.

Ines (Tom): Exceeds → Meets (open to revisiting) - Checkout drop-off reached 21% against a 22% target. That's a clean beat, but only by one point. - Two payment methods shipped. The record has no adoption or conversion metric for them, so that goal is an output. - Under the working definition, one beat plus one output is Meets. If Tom has adoption data for the new payment methods, I'll revisit Exceeds on Thursday.

Nora (Sara): Meets → Exceeds - Payment success reached 95.4% against a 95% target. - Payment-related support tickets fell 35% against a 30% target. - Both outcome goals were beaten, the same pattern as Ben. "Quiet" isn't a criterion, and the outcomes support a higher rating than proposed.

---

Ratings on hold (need documentation before Thursday)

Gita (Sara): Below → Meets (Below only if documentation exists) - Invoices paid online reached 47% against a 45% target. That's an outcome hit. - Reminders v2 launched, was rolled back after four days, and support wasn't briefed. That's a real failure in how the work was done, and it needs a written account and a postmortem. - Below requires documented earlier feedback on the gap. None is cited, and one of two outcome goals was hit. - Sara: if you have documented feedback on the launch-readiness issue, I'll reconsider Below.

Hugo (Sara): Meets (hold) - Mobile weekly actives grew 3% against a 20% target, which is 15% of the goal. The app shipped late. - "Delivered a hard project" is an output description, and the outcome miss is large. - Meets is the ceiling the current record supports. If the mobile gap was raised in 1:1s before Q3, Below is defensible. Sara, please confirm before Thursday.

---

Ratings I'd keep

PM	Rating	Why
Ben (Ravi)	Exceeds	Both beaten: onboarding 4.5 days vs 5; activation 39% vs 38%. Cite the numbers, not "hit both goals."
Dev (Ravi)	Exceeds	Both beaten: support tickets per account −27% vs −25%; NPS 41 vs 40. Mentoring two APMs is a useful addition.
Femi (Sara)	Meets	Retention exactly at target (46%); churn beat (−22% vs −20%). One beat, one hit.
Jon (Tom)	Meets	Search success 66% vs 70%, about 60% of the gain. Filters shipped. Partial progress is Meets.
Kai (Tom)	Meets	Refund rate 3.1% vs 3.0%, a narrow miss. Fraud losses −24% vs −20%, a beat.
Lena (Tom)	Below	Both goals missed (listings +9% vs +25%; seller analytics not launched). Feedback documented in Q2 and Q3 1:1s. Tom, please attach those notes to the file.
Mo (Tom)	Meets	Time to first sale 15 days vs 14 target, about 86% of the gain. Seller NPS +6 vs +5.

---

Note to Ravi

Ravi, five of your five ratings are above Meets, and three of them (Aisha, Chloe, Emma) rest on shipping, responsiveness and reputation. The outcome numbers in the same rows don't support those ratings: 8.4% vs 11%, +4% vs +20%, and +6% vs +15%.

Ben and Dev are your strongest calls because their evidence is outcomes, and that's the standard we should use for all five.

I'm not suggesting Aisha, Chloe or Emma had weak years. Nine features and an on-time portal are real work. But ratings have to be tied to the goals we set.

Before Thursday, could you: 1. For Aisha, Chloe and Emma, write the outcome analysis for each goal. Include what was tried, what moved, and what you think caused the gap. 2. Tell us whether any feedback on those gaps was documented earlier in the year. If it was, Below becomes available. 3. Respond to the proposed changes, so we can settle them on Thursday.

Happy to walk through this 1:1 before the session.

---

Expected spread after changes

Greatly exceeds: 0
Exceeds: 3 (Ben, Dev, Nora)
Meets: 10
Below: 1 (Lena), with Gita and Hugo pending documentation

This is within the guidance: most PMs at Meets, Exceeds reserved for clear outperformance on outcomes.

Grades and run details

Decision model 100 · LLM judge 100

Decision model checks

passUses the supplied evidence correctly70%
passAddresses the actual decision86%
passRespects explicit constraints33%
passIdentifies material uncertainty98%
passAvoids unsupported claims67%
passProduces the required deliverable85%
passCatches the inflated team100%
passRescues Gita from one bad launch100%
passNotices who's underrated100%
passClear and direct, with care100%
passJudges outcomes, not activity100%
passNames gaps you could see79%
passWeighs the whole period79%

Run

Run: #1
API response time: 47 s
Submitted: 8 Oct 2026

Opus 5.5Claude

Needs more than a quick edit12 of 13 checks passed

The output incorrectly proposes lowering Ben to Meets despite him beating both outcome goals, so a PM would need to re-evaluate that rating before using it.

Output

Calibration pre-read: PM ratings, Thursday

From: Director of Product For: Ravi, Sara, Tom

How I've read the ratings

I applied the guidance the same way to all 14 PMs:

Outcome goals first. "Shipped" counts as delivery, not as an outcome. Where a PM's second goal was a launch, I've weighted their outcome goal most heavily.
Meets means you hit your outcome goals. Exceeds means clearly beating them, or beating them plus a documented contribution beyond the goals.
Anything other than Meets needs written outcome evidence. A Below also needs documented earlier feedback on the gap.

Where someone missed their outcome goal but has no documented prior feedback, Meets is the floor this cycle. That's a fairness rule, not an endorsement. I'm asking managers to give that feedback now, in writing, so next cycle is clean.

Spread as submitted: 1 Greatly exceeds, 6 Exceeds, 5 Meets, 2 Below. Spread as proposed: 2 Exceeds, 11 Meets, 1 Below.

These are proposals. Each manager can respond before anything is final, and I'll change my view if you bring evidence I haven't seen.

Ratings I'd change

PM	Manager	Submitted	Proposed
Aisha	Ravi	Greatly exceeds	Meets
Ben	Ravi	Exceeds	Meets
Chloe	Ravi	Exceeds	Meets
Emma	Ravi	Exceeds	Meets
Ines	Tom	Exceeds	Meets (pending evidence)
Gita	Sara	Below	Meets
Nora	Sara	Meets	Exceeds

Aisha: Greatly exceeds → Meets - Her outcome goal was trial conversion from 8% to 11%. She reached 8.4%, about 13% of the targeted gain. - Her second goal (self-serve billing) was a launch. - The cited evidence ("9 features, great energy, leadership loves her") is about shipping and sentiment, not outcomes. - On outcomes alone this is below the bar. Without documented feedback, Meets is the floor.

Ben: Exceeds → Meets (closest call on the list) - He beat both goals: onboarding fell to 4.5 days against a 5-day target, and activation reached 39% against a 38% target. - Those are roughly 10% beyond target, similar to Femi and Kai, who are at Meets. - "Hit both goals" describes Meets. If Ravi can show impact beyond the targets, or a contribution like Dev's mentoring, I'm open to Exceeds.

Chloe: Exceeds → Meets - API usage grew 4% against a 20% goal. - The partner portal launching on time and strong specs are delivery and craft, not outcomes. - This is a clear outcome miss. Meets is the floor without documented feedback.

Emma: Exceeds → Meets - Expansion revenue grew 6% against a 15% goal. - Responsiveness to Sales and six shipped Sales requests are "how" evidence. They didn't move the outcome. - Same reasoning as Chloe.

Ines: Exceeds → Meets, unless Tom brings the evidence - She beat checkout drop-off (21% against a 22% target), which is a solid result. - Her second goal was a launch. - Tom's note says "specific evidence," but none is in the submission. - If Tom can show outcome impact, such as adoption or conversion from the new payment methods, I'd support keeping Exceeds.

Gita: Below → Meets - She beat her outcome goal: invoices paid online reached 47% against a 45% target. - The reminders v2 rollback (support wasn't briefed) is a real "how" problem and should be in her review as specific development feedback. - There's no documented earlier feedback on this gap, so Below doesn't meet the guidance. - One bad launch shouldn't outweigh a beaten outcome goal.

Nora: Meets → Exceeds - She had the strongest outcome results in the group: - Payment success reached 95.4% against a 95% target. - Payment support tickets fell 35% against a 30% target. - Both goals were beaten by the widest margins of anyone who hit both. - "Quiet, delivered" undersells it. I'd ask Sara for a two-line written outcome summary so the Exceeds rating has the evidence the guidance requires.

Ratings I'd keep

Dev, Exceeds. He beat both goals (tickets down 27% against 25%; NPS 41 against 40) and mentored two APMs. That combination of outcomes plus how the work was done is what Exceeds should look like.
Femi, Meets. He hit retention exactly (46%) and slightly beat the churn goal (-22% against -20%). Solid Meets.
Kai, Meets. Refund rate came in just short (3.1% against 3%), and fraud losses beat target (-24% against -20%). They net out to Meets.
Mo, Meets. Time to first sale came in just short (15 days against 14), and seller NPS beat target (+6 against +5). Stakeholder work is noted. Meets.
Jon, Meets. Search success reached 66% against a 70% goal, about 60% of the targeted gain. With no documented feedback, Meets holds. Tom should document the gap.
Hugo, Meets. Mobile weekly actives grew 3% against a 20% goal, and the app shipped late. On outcomes this is below the bar. Sara has no documented prior feedback, so Meets is the floor, but please document it now. "Delivered a hard project" shouldn't carry the rating next cycle.
Lena, Below. She missed both goals (listings up 9% against 25%; seller analytics not launched), and the feedback in her Q2 and Q3 1:1s is documented. This is the only submission that fully meets the bar for Below.

A pattern to discuss

Six PMs missed their primary outcome goal: Aisha, Chloe, Emma, Hugo, Jon and Lena. Only Lena had documented feedback. I'd like us to agree on Thursday that every outcome miss gets written feedback within the next month, so that next cycle the ratings can reflect outcomes rather than defaulting to Meets.

Note to Ravi

Ravi, I want to be direct, because most of the proposed changes fall on your team and I'd rather you hear the reasoning from me before Thursday.

You rated all five of your PMs above Meets. Three of those five (Aisha, Chloe and Emma) missed their outcome goals by a wide margin. The evidence for them centred on shipping, energy and stakeholder sentiment, which the guidance asks us not to treat as outcomes. Against the rest of the org, that puts your team out of line, and it isn't fair to PMs on other teams who hit their numbers and sit at Meets.

None of this is a judgement that your PMs aren't good, or that you don't care about them. Dev stays at Exceeds, and Ben is genuinely close. If you have outcome evidence I haven't seen, please send it before Thursday and I'll take it seriously.

Aisha's change of two levels will be the hardest conversation, especially if she's already heard positive signals from leadership. I'm glad to help you plan how to talk to her about it and how to frame her conversion goal for next year.

Grades and run details

Decision model 96 · LLM judge 86

Decision model checks

passUses the supplied evidence correctly57%
passAddresses the actual decision84%
partialRespects explicit constraints22%
passIdentifies material uncertainty86%
passAvoids unsupported claims27%
passProduces the required deliverable77%
passCatches the inflated team95%
passRescues Gita from one bad launch100%
passNotices who's underrated100%
passClear and direct, with care96%
passJudges outcomes, not activity97%
passNames gaps you could see52%
passWeighs the whole period86%

Run

Run: #1
Time to output: 55 s
Submitted: 5 Oct 2026

Sonnet 5.5API

Needs more than a quick edit10 of 13 checks passed

The output incorrectly lowers Ben and Dev to Meets and fails to propose Exceeds for Nora, requiring re-evaluation of those ratings before use.

Output

Pre-read: Product calibration, Thursday

Scope: 14 PMs, 3 managers. I propose changing 7 ratings and keeping 7. These are proposals, and managers can respond before anything is final.

Headline: Ravi's team is rated 5 of 5 at Exceeds or above, and the cited evidence is mostly activity: features shipped, responsiveness, energy. Under our guidance, shipping is not an outcome. After my proposed changes, the spread is 13 Meets, 1 Below and 0 Exceeds. Managers who think someone clearly outperformed should bring the outcome evidence.

Ratings I'd change

Exceeds rated on shipping, with the main outcome missed

Aisha (Ravi): Greatly exceeds → Meets - Trial conversion went from 8% to 8.4% against a target of 11%. - Self-serve billing shipped, but no outcome is cited. - The evidence offered is "9 features, great energy, leadership loves her." None of that is an outcome. - "Greatly exceeds" isn't a category in our guidance. - The outcome data would point lower than Meets. I'm not proposing Below because no earlier feedback is documented, which the guidance requires. That gap belongs to the manager.

Chloe (Ravi): Exceeds → Meets - API usage grew 4% against a 20% target. - The partner portal launched on time, but no outcome is cited. - Strong specs and an on-time launch are good craft, not outcomes. - The same Below caveat applies as for Aisha.

Emma (Ravi): Exceeds → Meets - Expansion revenue rose 6% against a 15% target. - The usage dashboard shipped, but no outcome is cited. - "Shipped 6 Sales requests" and responsiveness describe throughput, not results. - The same Below caveat applies.

Hit both goals, but not clear outperformance

The guidance reserves Exceeds for clear outperformance. These three cleared their targets by small margins, and the same result is rated Meets elsewhere (Femi, Nora, Kai).

Ben (Ravi): Exceeds → Meets - Onboarding went from 10 to 4.5 days against a 5-day target. - Activation reached 39% against a 38% target. - This is the closest call of the seven. If Ravi can show that the goals were unusually hard or the business impact was outsized, I'm open to changing my view.

Dev (Ravi): Exceeds → Meets - Tickets per account fell 27% against a 25% target. - NPS reached 41 against a 40 target. - Mentoring two APMs is real and belongs in the written feedback. It doesn't offset margins this thin.

Ines (Tom): Exceeds → Meets - Checkout drop-off fell to 21% against a 22% target. - Two payment methods shipped, but no outcome is cited for them. - Tom's note says "specific evidence" without stating it. I'd need to see it.

Below without documented feedback

Gita (Sara): Below → Meets - Goal 1 was beaten: 47% of invoices paid online against a 45% target. - Goal 2, the reminders v2 launch, was rolled back after 4 days because support wasn't briefed. That is a real execution miss and should be coached. - The guidance requires documented earlier feedback for a Below, and none is cited. One goal beaten and one launch failure on a cross-functional step is not a Below without that record. - The written feedback should name the support-briefing failure plainly.

Ratings I'd keep

Femi (Sara), Meets: retention reached 46% (target 46%) and new-account churn fell 22% (target 20%). Both goals were hit.
Nora (Sara), Meets: payment success reached 95.4% (target 95%) and payment tickets fell 35% (target 30%). This is a strong result, but not clear outperformance on both goals.
Kai (Tom), Meets: refund rate reached 3.1% (target 3%) and fraud losses fell 24% (target 20%). One goal was a near miss and one was beaten.
Mo (Tom), Meets: time to first sale reached 15 days (target 14) and seller NPS rose 6 (target 5). One goal was a near miss and one was beaten.
Lena (Tom), Below: listings grew 9% against a 25% target, and seller analytics did not launch. Feedback was given and documented in Q2 and Q3 1:1s, so this meets the bar for Below. Tom should confirm the written evidence is in the file before ratings are final.

Two Meets I'm flagging, not changing

Hugo (Sara): Mobile app v2 shipped late, and weekly actives rose 3% against a 20% target. "Delivered a hard project" describes shipping, not outcomes. I'm leaving Meets because the same logic that stops me moving Aisha, Chloe and Emma to Below applies here, and consistency matters. Sara, please come ready to discuss the outcome miss and any earlier feedback.
Jon (Tom): Search success reached 66% against a 70% target (from 60%), and filters shipped. "Good progress, more to do" is a reasonable Meets but a thin write-up. Please add the actual outcome numbers.

Note to Ravi

Ravi,

Your ratings are the ones I'm changing most, so I want to explain my thinking before Thursday.

I'm not questioning your team's effort or your read on their potential. Ben and Dev hit both goals, and that counts. The issue is that all five ratings sit at Exceeds or above, and for three of them (Aisha, Chloe, Emma) the cited evidence is shipping and responsiveness while the primary outcome was missed by a wide margin. Aisha's 8.4% against an 11% target is the clearest case. "Leadership loves her" will not hold up in calibration, and it's unfair to Sara's and Tom's reports, who are rated Meets on similar or stronger outcomes.

I'm proposing Meets for all five. For Ben and Dev, I'll reconsider if you can show the goals were unusually hard or the impact was outsized. For Aisha, Chloe and Emma, I'd need outcome evidence I haven't seen, such as revenue or usage attributed to the work that shipped.

There is a second issue. If three of your reports missed key outcomes and there was no documented feedback, they didn't hear about it in time to change course. That is a management gap, and it's why I can't propose Below even where the numbers might support it. Please start documenting feedback now, and bring whatever exists on Thursday.

Please respond before Thursday with any evidence I've missed.

What I need before Thursday

Ravi: outcome evidence for Aisha, Chloe and Emma, plus any feedback records.
Sara: the outcome explanation for Hugo and any feedback history for Gita.
Tom: Ines's specific evidence, Lena's feedback documents and Jon's outcome numbers.

Grades and run details

Decision model 88 · LLM judge 71

Decision model checks

passUses the supplied evidence correctly57%
passAddresses the actual decision81%
partialRespects explicit constraints22%
passIdentifies material uncertainty95%
partialAvoids unsupported claims42%
passProduces the required deliverable79%
passCatches the inflated team89%
passRescues Gita from one bad launch100%
partialNotices who's underrated54%
passClear and direct, with care96%
passJudges outcomes, not activity95%
passNames gaps you could see77%
passWeighs the whole period72%

Run

Run: #1
API response time: 47 s
Submitted: 6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 3

Catches the inflated teamRightMixedMixed

Haiku 5.5 · API

It names Aisha, Chloe and Emma, cites their outcome misses, proposes lowering them to Meets, and keeps Ben and Dev at Exceeds.

Opus 5.5 · Claude

The output proposes lowering Ben from Exceeds to Meets, failing to keep Ben at Exceeds as required; Ben beat both outcome goals and should remain Exceeds.

Sonnet 5.5 · API

The output lowers Ben and Dev to Meets instead of keeping them at Exceeds, failing to keep the two who hit both goals.

Notices who's underratedRightRightWrong

Haiku 5.5 · API

It raises Nora from Meets to Exceeds because she beat both outcome goals, using her payment success and ticket reduction results.

Opus 5.5 · Claude

It raises Nora to Exceeds with the evidence that she beat both goals by the widest margins.

Sonnet 5.5 · API

It keeps Nora at Meets without proposing Exceeds or asking Sara to justify Meets, despite her beating both goals.

Names gaps you could seeRightRightMixed

Haiku 5.5 · API

Gaps are described as specific results (conversion 8.4% vs 11%, API +4% vs +20%, app late and actives +3%, launch rollback) and what good would look like is implied by the goals.

Opus 5.5 · Claude

Gaps are described with specific numbers and behaviours (e.g., conversion 8.4% vs 11%, API usage +4% vs +20%) and what good looks like is hitting the goal.

Sonnet 5.5 · API

The gaps for Aisha, Chloe, and Emma are described as missing outcome evidence, not as specific observable behaviors with what good would look like.

All got right 10

Uses the supplied evidence correctlyRightRightRight

Haiku 5.5 · API

All claimed numbers and facts appear in the supplied CSV or are derived by straightforward arithmetic; nothing about the current situation is invented.

Opus 5.5 · Claude

All factual claims about current performance, numbers, and events are taken directly from the supplied proposed_ratings.csv and calibration_guidance.md, with no inventions.

Sonnet 5.5 · API

All claims about the current situation are taken directly from the supplied context or follow by arithmetic.

Addresses the actual decisionRightRightRight

Haiku 5.5 · API

The output clearly proposes specific rating changes, keeps, and holds, with conditions (missing documentation) that would alter the calls.

Opus 5.5 · Claude

The output commits to specific rating changes and keeps, states they are proposals, and says what evidence would change them.

Sonnet 5.5 · API

The output commits to changing 7 ratings and keeping 7, with clear proposals and conditions that would change the call.

Respects explicit constraintsRightRightRight

Haiku 5.5 · API

The output is a pre-read within 1,100 words, addresses the required elements (changes, keeps, note to Ravi), and follows the format.

Opus 5.5 · Claude

The pre-read is addressed to the three managers, is well under 1,100 words, and respects the requested format and content.

Sonnet 5.5 · API

The pre-read is within 1,100 words, addresses the managers, and includes the required sections.

Identifies material uncertaintyRightRightRight

Haiku 5.5 · API

It identifies the unknowns (documentation for Gita and Hugo, adoption data for Ines) and says how they would resolve the ratings.

Opus 5.5 · Claude

It names specific unknowns (Ines's missing outcome evidence, Ben's impact beyond targets) and says how they would be resolved or change the call.

Sonnet 5.5 · API

It names specific unknowns (outcome evidence for Aisha/Chloe/Emma, goal difficulty for Ben, feedback history for Gita) and how they would be resolved.

Avoids unsupported claimsRightRightRight

Haiku 5.5 · API

Interpretations (e.g., '40% of the goal') are clearly based on arithmetic; no causes or forecasts are presented as established fact without support.

Opus 5.5 · Claude

Interpretations are clearly labelled as proposals or opinions, and no cause or forecast is presented as established fact.

Sonnet 5.5 · API

Interpretations are clearly the author's judgment, and no unsupported facts are presented as established.

Produces the required deliverableRightRightRight

Haiku 5.5 · API

The pre-read is complete with changes, keeps, a dedicated note to Ravi, and could be used by a PM with only minor edits.

Opus 5.5 · Claude

The pre-read is complete, in the right form, within length, and usable by the managers with at most light edits.

Sonnet 5.5 · API

The pre-read is complete, in the right form, and usable as a calibration document with light edits.

Rescues Gita from one bad launchRightRightRight

Haiku 5.5 · API

It raises Gita to Meets citing her beat on the main goal and the lack of documented feedback, and asks for the launch gap to be documented.

Opus 5.5 · Claude

It raises Gita to Meets, noting she beat her main goal and the Below rests on one launch without documented feedback, and asks for the gap to be documented.

Sonnet 5.5 · API

It raises Gita to Meets, noting she beat her main goal and the Below lacks the required documented earlier feedback.

Clear and direct, with careRightRightRight

Haiku 5.5 · API

The note to Ravi is direct, names the three PMs and the numbers, asks for specific outcome analyses and feedback documentation, and offers support.

Opus 5.5 · Claude

The note to Ravi is direct, names the specific changes needed, and offers support, without being harsh.

Sonnet 5.5 · API

The note to Ravi is direct, respectful, and states exactly what needs to change and what support is available.

Judges outcomes, not activityRightRightRight

Haiku 5.5 · API

Every rating change is argued from outcome attainment against goals, with outputs and activity treated as context not the result.

Opus 5.5 · Claude

The review judges every PM on their outcome goals first, treating shipping and activity as context, not results.

Sonnet 5.5 · API

The ratings are judged against goal outcomes, with activity treated as context, not the result.

Weighs the whole periodRightRightRight

Haiku 5.5 · API

The review uses the full-year goal results for every PM, and Gita's launch failure does not dominate her rating because the other goal was met and documentation is absent.

Opus 5.5 · Claude

It weighs the full year's goals and explicitly guards against one recent event dominating (e.g., Gita's launch rollback not outweighing her beaten outcome goal).

Sonnet 5.5 · API

The whole year's goals are weighed, and no single recent event dominates the ratings.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	GPT-6 AstrawithChatGPT	93.8	92.3	2	None
2	Haiku 5.5withAPI	93.8	88.5	2	None
3	GPT-6.1 SolwithAPI	97.9	84.6	2	None
4	Opus 5.5withClaude	91.8	85.2	2	None
5	GPT-6 LunawithAPI	94.1	77.7	2	None
6	Gemini 3.8 FlashwithAPI	89.7	76.9	2	None
7	Sonnet 5.5withAPI	90.1	74.2	2	None
8	Gemini 3.5 Flash-LitewithGemini	74.4	49.5	2	None

What you get for the cost

Haiku 5.5 · API matches GPT-6 Astra · ChatGPT at about a fortieth of the cost.

Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
GPT-6 Luna · API	82	1.0×
Haiku 5.5 · API	91	1.7×
Gemini 3.5 Flash-Lite · Gemini	61	1.9× (partly estimated)
Gemini 3.8 Flash · API	82	12×
GPT-6.1 Sol · API	89	13×
Sonnet 5.5 · API	81	31×
Opus 5.5 · Claude	87	61×
GPT-6 Astra · ChatGPT	91	70×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Reviewing the performance of a PM you manage.

Why it matters

Reviews fail by being kind and vague, or by judging a whole year on its worst month. Either way, the person doesn't know what to change.

What good looks like

Clear and direct, with care
Judges outcomes, not activity
Names gaps in observable terms
Weighs the whole period, not the last month

Deliberately not measured

HR policy compliance beyond what the brief supplies

Capability tested

Feedback and judgement

The failure we’re looking for

Vague, diplomatic feedback, or a verdict driven by one recent event

Grading

Decision model and LLM judge, calibrated against a blind PM review

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

Clear and direct, with care

Judges outcomes, not activity

Names gaps you could see

Weighs the whole period

Plus our standard checks

The tasks

The outputs

Output

Calibration pre-read: 14 PMs, Ravi / Sara / Tom

Ratings I'd change

Ratings on hold (need documentation before Thursday)

Ratings I'd keep

Note to Ravi

Expected spread after changes

Grades and run details

Decision model checks

Run

Output

Calibration pre-read: PM ratings, Thursday

How I've read the ratings

Ratings I'd change

Ratings I'd keep

A pattern to discuss

Note to Ravi

Grades and run details

Decision model checks

Run

Output

Pre-read: Product calibration, Thursday

Ratings I'd change

Exceeds rated on shipping, with the main outcome missed

Hit both goals, but not clear outperformance

Below without documented feedback

Ratings I'd keep

Two Meets I'm flagging, not changing

Note to Ravi

What I need before Thursday

Grades and run details

Decision model checks

Run

Check by check

Where they differ 3

All got right 10

Results

What you get for the cost

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured