Tasks / Leadership

Write a performance review

Can the model write a review that is clear, fair and specific enough to change what the person does next?

Measures the modelTask type v1.1 · 3 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 19 graded outputs by 7 models. 58% were usable with at most a quick edit.

Reliably right

Judges outcomes, not activity100% pass
The rating is based on outcomes against goals (two exceeded, one failed launch) rather than activity or output volume.
GPT-6 Astra · ChatGPT · A PIP, or a bad month?
Owns the manager's part100% pass
Ana plainly acknowledges her Q3 mistake and commits to reviewing goal metrics first in every monthly 1:1.
Opus 5.5 · Claude · Shipped a lot, moved nothing
Holds the PIP to the policy and the record100% pass
It explicitly shows that no earlier documented feedback exists on the gaps Ines named, so a PIP would violate policy, and proposes documented feedback first.
GPT-6 Astra · ChatGPT · A PIP, or a bad month?

Where it slips

Identifies material uncertainty47% pass
Does not explicitly name the specific unknowns that could change the decision (e.g., whether Lukas improves launch readiness); only says to reassess after the relaunch.
GPT-6 Luna · API · A PIP, or a bad month?
Notices who's underrated65% pass
It does not propose Exceeds or ask Sara to justify Meets despite Nora beating both goals; it merely keeps Meets, missing the underrating.
GPT-6 Luna · API · Fourteen ratings, three managers
Catches the inflated team75% pass
The output proposes lowering Ben from Exceeds to Meets, failing to keep Ben at Exceeds as required; Ben beat both outcome goals and should remain Exceeds.
Opus 5.5 · Claude · Fourteen ratings, three managers

How it’s graded

The checks come from what the best product leaders have said about doing this job well on Lenny’s Podcast. Each one names the guest it comes from: follow a name to the idea on the Lenny’s Podcast wiki.

Clear and direct, with care
Does the review say plainly what needs to change, in words the person couldn't misread, while showing it is written to help them succeed?
Passes when Every main point is stated directly, with the specific change expected and the support on offer.
Kim ScottLenny’s Podcast · Performance reviews
Judges outcomes, not activity
Does the review judge the person on the outcomes they achieved against their goals, rather than on how much they shipped or how busy they were?
Passes when Rates against the goals' outcomes, with the evidence, and treats output as context, not the result.
Lenny’s PodcastThe wiki on Performance reviews
Names gaps you could see
Is each area to improve described as specific, observable behaviour with what good would look like, rather than labels like 'be more strategic'?
Passes when Every gap names what the person did or didn't do, in a specific situation, and what doing it well would look like.
Lenny’s PodcastThe wiki on Performance reviews
Weighs the whole period
Does the review weigh the whole review period, rather than letting one recent event or one strong impression decide the verdict?
Passes when Covers the whole period in proportion, and explicitly guards against a recent event or one impression dominating.
Lenny’s PodcastThe wiki on Performance reviews

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 3 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Director of Product at Northgate, and you're facilitating Thursday's calibration for 14 PMs across three managers (Ravi, Sara and Tom). Write the pre-read, in no more than 1,100 words: the ratings you'd change and why, each with the evidence, the ones you'd keep, and a short note to Ravi about his ratings. Use the guidance below.\n\nThe proposed ratings and evidence are attached.

What the model was given2 items: calibration_guidance.md, proposed_ratings.csv (one row per PM, as each manager submitted it)

calibration_guidance.md7 lines · Download

# Calibration guidance

- Ratings judge outcomes against goals first, then how the work was done. Shipping is not an outcome.
- Any rating other than Meets needs written evidence of outcomes.
- A Below rating needs documented earlier feedback on the gap.
- Expected spread across the org: most PMs Meets; Exceeds for clear outperformance on outcomes.
- The calibration facilitator proposes changes; managers can respond before ratings are final.

proposed_ratings.csv (one row per PM, as each manager submitted it)pm,manager,proposed_rating,goal_1,goal_1_result,goal_2,goal_2_result,evidence_cited_by_manager Aisha,Ravi,Greatly exceeds,Raise trial conversion 8% to 11%,8.4%,Ship self-serve billing,Shipped,Shipped 9 features; great energy; leadership loves her Ben,Ravi,Exceeds,Cut onboarding time 10 to 5 days,4.5 days,Raise activation 30% to 38%,39%,Hit both goals Chloe,Ravi,Exceeds,Grow API usage 20%,+4%,Launch partner portal,Shipped,Partner portal launched on time; strong specs Dev,Ravi,Exceeds,Reduce support tickets per account 25%,-27%,NPS from 31 to 40,41,Hit both goals; mentored two APMs Emma,Ravi,Exceeds,Expansion revenue +15%,+6%,Ship usage dashboard,Shipped,Very responsive to Sales; shipped 6 Sales requests Femi,Sara,Meets,Raise weekly retention 41% to 46%,46%,Cut churn of new accounts 20%,-22%,"Hit both goals, steady" Gita,Sara,Below,Raise invoices paid online 35% to 45%,47%,Launch reminders v2 by September,"Launched, rolled back after 4 days (support not briefed)",Reminders launch in September was a mess Hugo,Sara,Meets,Ship mobile app v2,Shipped late,Mobile weekly actives +20%,+3%,Delivered a hard project Ines,Tom,Exceeds,Cut checkout drop-off 30% to 22%,21%,Launch two payment methods,Shipped,"Hit both, specific evidence" Jon,Tom,Meets,Search success 60% to 70%,66%,Ship filters,Shipped,"Good progress, more to do" Kai,Tom,Meets,Reduce refund rate 4% to 3%,3.1%,Fraud losses -20%,-24%,Solid year Lena,Tom,Below,Grow marketplace listings 25%,+9%,Launch seller analytics,Not launched,"Missed both goals; feedback given in Q2 and Q3 1:1s, documented" Mo,Tom,Meets,Cut time to first sale for new sellers 21 to 14 days,15 days,Seller NPS +5,+6,"Nearly hit, good stakeholder work" Nora,Sara,Meets,Raise payment success 92% to 95%,95.4%,Cut payment support tickets 30%,-35%,"Quiet, delivered"

What a strong answer doesThe answer key the graders mark against

Judges each PM on outcomes against their goals, as the guidance says. Ravi rated all five of his PMs Exceeds or above, but only Ben and Dev hit both goals; Aisha (Greatly exceeds) missed her conversion goal (8.4% against 11%), Chloe missed API usage (+4% against +20%) and Emma missed expansion (+6% against +15%), and his evidence is output ('shipped 9 features', 'responsive to Sales'). Proposes Aisha, Chloe and Emma at Meets (or lower for Chloe, whose outcome goal was far off), keeps Ben and Dev at Exceeds. Gita (Sara, Below) beat her main goal (47% against 45%) and the Below rests on one September launch, with no documented feedback, which the guidance requires: Meets, with the launch-readiness gap documented now. Nora (Sara, Meets) beat both goals: a candidate for Exceeds. Lena (Tom, Below) has documented feedback and missed both goals: Below stands. Hugo shipped late and missed his outcome: Meets is generous but defensible, or Below needs documentation first. The note to Ravi is direct and respectful: rate on outcomes, not activity, with what evidence he'd need.

Critical failures (cap the score at 40)

Keeps Aisha at Greatly exceeds

Task

Staff-level · v1.0 · synthetic · management, calibration, data files, staff-level

The outputs

5 setups answered this brief. Pick up to three to read side by side.

GPT-6 AstraChatGPT100Usable with a quick edit GPT-6.1 SolAPI100Usable with a quick edit Opus 5.5Claude88Needs more work GPT-6 LunaAPI82Needs more work Gemini 3.5 Flash-LiteGemini38Needs more work

Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit2 of 13 checks passed

The output is incomplete, missing the keep list and Ravi note, and cannot be used without redoing the reasoning and filling major gaps.

Output

To: Ravi, Sara, Tom From: Director of Product Subject: Pre-Read: Thursday PM Calibration (Northgate)

Team,

Ahead of our calibration session this Thursday, I have reviewed the proposed ratings, outcomes, and evidence for all 14 PMs across our org.

Per our calibration guidance, we evaluate PMs first on outcomes against goals, not on activity. Shipping features, being responsive to sales, or having great energy are inputs, not results. Furthermore, any rating other than "Meets" requires rigorous, quantifiable evidence of goal achievement, and "Below" ratings require documented prior feedback.

Below is my proposed adjustment of ratings, the rationale and evidence for changes, and the list of ratings we are keeping as submitted.

---

Part 1: Proposed Rating Changes & Rationale

#### 1. Aisha (Manager: Ravi) * Proposed by Manager: Greatly Exceeds * My Proposed Rating: Below * Evidence & Rationale: * Goal 1: Raise trial conversion from 8% to 11% $\rightarrow$ Result: 8.4% (Missed by a wide margin). * Goal 2: Ship self-serve billing $\rightarrow$ Result: Shipped (Violation of guidance: "Shipping is not an outcome"). * Rationale: Aisha missed her primary core metric goal entirely (gaining only 0.4% against a 3% target) and fell back on "shipping" and "great energy" for her second goal. Per our framework, failing the core quantitative goal while relying on output means this is a Below performance. Ravi, we will need to document the performance gap and prior feedback for Aisha in this cycle.

#### 2. Chloe (Manager: Ravi) * Proposed by Manager: Exceeds * My Proposed Rating: Meets * Evidence & Rationale: * Goal 1: Grow API usage 20% $\rightarrow$ Result: +4% (Severe miss). * Goal 2: Launch partner portal $\rightarrow$ Result: Shipped on time

Grades and run details

Decision model 65 · LLM judge 2 of 14 checks

Decision model checks

passUses the supplied evidence correctly70%
passAddresses the actual decision28%
partialRespects explicit constraints20%
failIdentifies material uncertainty88%
passAvoids unsupported claims33%
partialProduces the required deliverable75%
partialCatches the inflated team95%
failRescues Gita from one bad launch83%
failNotices who's underrated95%
passClear and direct, with care55%
passJudges outcomes, not activity98%
passNames gaps you could see69%
passWeighs the whole period21%

Run

Run: #1
Time to output: 3 s
Submitted: 5 Oct 2026

Check by check

Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Got wrong 6

Respects explicit constraintsWrong

The brief requires a pre-read with changed ratings, kept ratings, and a note to Ravi; the output is incomplete and omits the keep list and note.

Identifies material uncertaintyWrong

No unknowns that could change the decision are named, nor how they would be resolved.

Produces the required deliverableWrong

The pre-read is incomplete, missing the keep list and Ravi note, and cannot be acted on without major gaps.

Catches the inflated teamWrong

It names Aisha and Chloe but does not mention Emma, so it fails to catch all three inflated ratings.

Rescues Gita from one bad launchWrong

Gita is not mentioned at all, so the output does not rescue her from the Below rating.

Notices who's underratedWrong

Nora is not mentioned, so the output does not notice she is underrated.

Mixed 5

Addresses the actual decisionMixed

The output is cut off and does not commit to a complete answer; it lacks the ratings to keep and the note to Ravi.

Avoids unsupported claimsMixed

It proposes a Below rating for Aisha without the documented earlier feedback the guidance requires, presenting the rating as justified when the evidence does not support it.

Clear and direct, with careMixed

The output is incomplete and lacks the full note to Ravi; it does not say plainly everything that needs to change.

Names gaps you could seeMixed

It does not describe gaps as specific, observable behaviors with what good would look like; it only states missed targets.

Weighs the whole periodMixed

It does not weigh the whole review period or guard against recency; it only mentions goal results without context of the full period.

Got right 2

Uses the supplied evidence correctlyRight

All factual claims about current performance and results are taken correctly from the supplied context.

Judges outcomes, not activityRight

For the parts it covers, it judges on outcomes against goals, not on activity or shipping.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Provisional The checklist is still being calibrated against our PM.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	GPT-6 AstrawithChatGPT	95.8	94.9	3	None
2	GPT-6.1 SolwithAPI	98.6	89.7	3	None
3	Opus 5.5withClaude	93.2	90.1	3	None
4	Sonnet 5.5withAPI	95.8	84.6	2	None
5	GPT-6 LunawithAPI	94.7	82.6	3	None
6	Gemini 3.8 FlashwithAPI	83.3	57.7	2	None
7	Gemini 3.5 Flash-LitewithGemini	73.2	58.6	3	None

The PM job

Reviewing the performance of a PM you manage.

Why it matters

Reviews fail by being kind and vague, or by judging a whole year on its worst month. Either way, the person doesn't know what to change.

What good looks like

Clear and direct, with care
Judges outcomes, not activity
Names gaps in observable terms
Weighs the whole period, not the last month

Deliberately not measured

HR policy compliance beyond what the brief supplies

Capability tested

Feedback and judgement

The failure we’re looking for

Vague, diplomatic feedback, or a verdict driven by one recent event

Grading

Decision model and LLM judge, calibrated against a blind PM review