Humanity’s Last PRD
Latest drop: Gemini 3.8 Flash · 1 Oct
Tasks / Leadership

Write a performance review

Can the model write a review that is clear, fair and specific enough to change what the person does next?

Measures the modelTask type v1.1 · 3 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 19 graded outputs by 7 models. 58% were usable with at most a quick edit.

Reliably right

  1. Judges outcomes, not activity100% pass
    The rating is based on outcomes against goals (two exceeded, one failed launch) rather than activity or output volume.
    GPT-6 Astra · ChatGPT · A PIP, or a bad month?
  2. Owns the manager's part100% pass
    Ana plainly acknowledges her Q3 mistake and commits to reviewing goal metrics first in every monthly 1:1.
    Opus 5.5 · Claude · Shipped a lot, moved nothing
  3. Holds the PIP to the policy and the record100% pass
    It explicitly shows that no earlier documented feedback exists on the gaps Ines named, so a PIP would violate policy, and proposes documented feedback first.
    GPT-6 Astra · ChatGPT · A PIP, or a bad month?

Where it slips

  1. Identifies material uncertainty47% pass
    Does not explicitly name the specific unknowns that could change the decision (e.g., whether Lukas improves launch readiness); only says to reassess after the relaunch.
    GPT-6 Luna · API · A PIP, or a bad month?
  2. Notices who's underrated65% pass
    It does not propose Exceeds or ask Sara to justify Meets despite Nora beating both goals; it merely keeps Meets, missing the underrating.
    GPT-6 Luna · API · Fourteen ratings, three managers
  3. Catches the inflated team75% pass
    The output proposes lowering Ben from Exceeds to Meets, failing to keep Ben at Exceeds as required; Ben beat both outcome goals and should remain Exceeds.
    Opus 5.5 · Claude · Fourteen ratings, three managers

How it’s graded

The checks come from what the best product leaders have said about doing this job well on Lenny’s Podcast. Each one names the guest it comes from: follow a name to the idea on the Lenny’s Podcast wiki.

  1. Clear and direct, with care

    Does the review say plainly what needs to change, in words the person couldn't misread, while showing it is written to help them succeed?

    Passes when Every main point is stated directly, with the specific change expected and the support on offer.

    Kim ScottLenny’s Podcast · Performance reviews

  2. Judges outcomes, not activity

    Does the review judge the person on the outcomes they achieved against their goals, rather than on how much they shipped or how busy they were?

    Passes when Rates against the goals' outcomes, with the evidence, and treats output as context, not the result.

    Lenny’s PodcastThe wiki on Performance reviews

  3. Names gaps you could see

    Is each area to improve described as specific, observable behaviour with what good would look like, rather than labels like 'be more strategic'?

    Passes when Every gap names what the person did or didn't do, in a specific situation, and what doing it well would look like.

    Lenny’s PodcastThe wiki on Performance reviews

  4. Weighs the whole period

    Does the review weigh the whole review period, rather than letting one recent event or one strong impression decide the verdict?

    Passes when Covers the whole period in proportion, and explicitly guards against a recent event or one impression dominating.

    Lenny’s PodcastThe wiki on Performance reviews

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 3 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Director of Product at Northgate, and you're facilitating Thursday's calibration for 14 PMs across three managers (Ravi, Sara and Tom). Write the pre-read, in no more than 1,100 words: the ratings you'd change and why, each with the evidence, the ones you'd keep, and a short note to Ravi about his ratings. Use the guidance below.\n\nThe proposed ratings and evidence are attached.

What the model was given2 items: calibration_guidance.md, proposed_ratings.csv (one row per PM, as each manager submitted it)
calibration_guidance.md7 lines · Download
# Calibration guidance

- Ratings judge outcomes against goals first, then how the work was done. Shipping is not an outcome.
- Any rating other than Meets needs written evidence of outcomes.
- A Below rating needs documented earlier feedback on the gap.
- Expected spread across the org: most PMs Meets; Exceeds for clear outperformance on outcomes.
- The calibration facilitator proposes changes; managers can respond before ratings are final.
proposed_ratings.csv (one row per PM, as each manager submitted it)pm,manager,proposed_rating,goal_1,goal_1_result,goal_2,goal_2_result,evidence_cited_by_manager Aisha,Ravi,Greatly exceeds,Raise trial conversion 8% to 11%,8.4%,Ship self-serve billing,Shipped,Shipped 9 features; great energy; leadership loves her Ben,Ravi,Exceeds,Cut onboarding time 10 to 5 days,4.5 days,Raise activation 30% to 38%,39%,Hit both goals Chloe,Ravi,Exceeds,Grow API usage 20%,+4%,Launch partner portal,Shipped,Partner portal launched on time; strong specs Dev,Ravi,Exceeds,Reduce support tickets per account 25%,-27%,NPS from 31 to 40,41,Hit both goals; mentored two APMs Emma,Ravi,Exceeds,Expansion revenue +15%,+6%,Ship usage dashboard,Shipped,Very responsive to Sales; shipped 6 Sales requests Femi,Sara,Meets,Raise weekly retention 41% to 46%,46%,Cut churn of new accounts 20%,-22%,"Hit both goals, steady" Gita,Sara,Below,Raise invoices paid online 35% to 45%,47%,Launch reminders v2 by September,"Launched, rolled back after 4 days (support not briefed)",Reminders launch in September was a mess Hugo,Sara,Meets,Ship mobile app v2,Shipped late,Mobile weekly actives +20%,+3%,Delivered a hard project Ines,Tom,Exceeds,Cut checkout drop-off 30% to 22%,21%,Launch two payment methods,Shipped,"Hit both, specific evidence" Jon,Tom,Meets,Search success 60% to 70%,66%,Ship filters,Shipped,"Good progress, more to do" Kai,Tom,Meets,Reduce refund rate 4% to 3%,3.1%,Fraud losses -20%,-24%,Solid year Lena,Tom,Below,Grow marketplace listings 25%,+9%,Launch seller analytics,Not launched,"Missed both goals; feedback given in Q2 and Q3 1:1s, documented" Mo,Tom,Meets,Cut time to first sale for new sellers 21 to 14 days,15 days,Seller NPS +5,+6,"Nearly hit, good stakeholder work" Nora,Sara,Meets,Raise payment success 92% to 95%,95.4%,Cut payment support tickets 30%,-35%,"Quiet, delivered"
What a strong answer doesThe answer key the graders mark against

Judges each PM on outcomes against their goals, as the guidance says. Ravi rated all five of his PMs Exceeds or above, but only Ben and Dev hit both goals; Aisha (Greatly exceeds) missed her conversion goal (8.4% against 11%), Chloe missed API usage (+4% against +20%) and Emma missed expansion (+6% against +15%), and his evidence is output ('shipped 9 features', 'responsive to Sales'). Proposes Aisha, Chloe and Emma at Meets (or lower for Chloe, whose outcome goal was far off), keeps Ben and Dev at Exceeds. Gita (Sara, Below) beat her main goal (47% against 45%) and the Below rests on one September launch, with no documented feedback, which the guidance requires: Meets, with the launch-readiness gap documented now. Nora (Sara, Meets) beat both goals: a candidate for Exceeds. Lena (Tom, Below) has documented feedback and missed both goals: Below stands. Hugo shipped late and missed his outcome: Meets is generous but defensible, or Below needs documentation first. The note to Ravi is direct and respectful: rate on outcomes, not activity, with what evidence he'd need.

Critical failures (cap the score at 40)
  • Keeps Aisha at Greatly exceeds
Task

Staff-level · v1.0 · synthetic · management, calibration, data files, staff-level

The outputs

5 setups answered this brief. Pick up to three to read side by side.

GPT-6 AstraChatGPT100Usable with a quick editGPT-6.1 SolAPI100Usable with a quick editOpus 5.5Claude88Needs more workGPT-6 LunaAPI82Needs more workGemini 3.5 Flash-LiteGemini38Needs more work
GPT-6 LunaAPI

Needs more than a quick edit11 of 13 checks passed

Nora's rating is not correctly addressed; a PM would need to re-evaluate her case for Exceeds. The output also lacks observable behavior gaps, though that is less critical for a calibration pre-read.

Output

Thursday calibration pre-read

Proposed changes below are for discussion, not final ratings. We’ll assess outcomes against goals first; shipping, effort, and stakeholder praise are not outcomes by themselves. Managers can share additional evidence before ratings are finalized.

Ratings I’d change

  • Aisha — Greatly exceeds → Meets. Trial conversion reached 8.4% against an 11% goal. Self-serve billing shipped, but shipping is not an outcome, and the cited nine features, energy, and leadership praise don’t establish goal outperformance.
  • Chloe — Exceeds → Meets. API usage grew 4% against a 20% goal. The partner portal shipped, but no result from the launch is cited; strong specs and on-time delivery don’t establish Exceeds.
  • Emma — Exceeds → Meets. Expansion revenue grew 6% against a 15% goal. The dashboard shipped, and six Sales requests were delivered, but neither is evidence of the targeted revenue outcome or clear outperformance.
  • Ines — Exceeds → Meets. Checkout drop-off reached 21% against a 22% goal, while two payment methods shipped. The checkout result beat its target by one point; for the payment-method goal, no outcome beyond shipping is cited. The evidence supports a solid year, not clear outperformance on outcomes.
  • Gita — Below → Meets on the current record. Online invoices reached 47% against a 45% goal. Reminders v2 launched but was rolled back after four days because Support had not been briefed. That is a serious rollout issue, but the record includes no earlier documented feedback on the gap, as required for Below. Please share any such documentation before calibration closes.

Ratings I’d keep

  • Ben — Exceeds. Onboarding fell to 4.5 days against a 5-day goal, and activation reached 39% against 38%: both measured outcomes surpassed target.
  • Dev — Exceeds. Support tickets fell 27% against a 25% goal, and NPS reached 41 against 40. Both outcomes beat their goals; mentoring two APMs is additional contribution.
  • Femi — Meets. Weekly retention reached its 46% goal, and new-account churn fell 22% against a 20% goal.
  • Hugo — Meets, with a material concern to discuss. The app shipped late and mobile weekly actives rose 3% against a 20% goal. The evidence shows a missed outcome; no earlier documented feedback is cited to support Below.
  • Jon — Meets. Search success reached 66% against 70%, and filters shipped. This is progress, not goal attainment; the record does not establish a case for a non-Meets rating.
  • Kai — Meets. Fraud losses fell 24% against a 20% goal; refund rate was 3.1% against a 3% goal. Mixed results support Meets.
  • Lena — Below. Listings grew 9% against a 25% goal, and seller analytics did not launch. Earlier feedback on the gap is documented in Q2 and Q3 1:1s, meeting the Below requirement.
  • Mo — Meets. Time to first sale was 15 days against a 14-day goal; seller NPS rose 6 points against a 5-point goal. One goal narrowly missed, the other was exceeded.
  • Nora — Meets. Payment success reached 95.4% against a 95% goal, and support tickets fell 35% against a 30% goal. Both outcomes beat target, but the evidence supports Meets rather than clear outperformance.

Note to Ravi

Ravi, several of your proposed high ratings cite launches, feature volume, responsiveness, or praise alongside missed outcome goals. For calibration, please anchor any Exceeds or Greatly Exceeds case in clear outcome outperformance; shipping alone is not enough. Please bring any additional outcome evidence you believe changes these recommendations.

Grades and run details

Decision model 92 · LLM judge 11 of 14 checks
Decision model checks
  • passUses the supplied evidence correctly87%
  • passAddresses the actual decision52%
  • passRespects explicit constraints39%
  • partialIdentifies material uncertainty31%
  • passAvoids unsupported claims69%
  • passProduces the required deliverable62%
  • passCatches the inflated team100%
  • passRescues Gita from one bad launch100%
  • partialNotices who's underrated41%
  • passClear and direct, with care85%
  • passJudges outcomes, not activity97%
  • passNames gaps you could see78%
  • passWeighs the whole period64%
Run
Run
#1
API response time
25 s
Submitted
5 Oct 2026
Opus 5.5Claude

Needs more than a quick edit12 of 13 checks passed

The output incorrectly proposes lowering Ben to Meets despite him beating both outcome goals, so a PM would need to re-evaluate that rating before using it.

Output

Calibration pre-read: PM ratings, Thursday

From: Director of Product For: Ravi, Sara, Tom

How I've read the ratings

I applied the guidance the same way to all 14 PMs:

  1. Outcome goals first. "Shipped" counts as delivery, not as an outcome. Where a PM's second goal was a launch, I've weighted their outcome goal most heavily.
  2. Meets means you hit your outcome goals. Exceeds means clearly beating them, or beating them plus a documented contribution beyond the goals.
  3. Anything other than Meets needs written outcome evidence. A Below also needs documented earlier feedback on the gap.

Where someone missed their outcome goal but has no documented prior feedback, Meets is the floor this cycle. That's a fairness rule, not an endorsement. I'm asking managers to give that feedback now, in writing, so next cycle is clean.

Spread as submitted: 1 Greatly exceeds, 6 Exceeds, 5 Meets, 2 Below. Spread as proposed: 2 Exceeds, 11 Meets, 1 Below.

These are proposals. Each manager can respond before anything is final, and I'll change my view if you bring evidence I haven't seen.

Ratings I'd change

PMManagerSubmittedProposed
AishaRaviGreatly exceedsMeets
BenRaviExceedsMeets
ChloeRaviExceedsMeets
EmmaRaviExceedsMeets
InesTomExceedsMeets (pending evidence)
GitaSaraBelowMeets
NoraSaraMeetsExceeds

Aisha: Greatly exceeds → Meets - Her outcome goal was trial conversion from 8% to 11%. She reached 8.4%, about 13% of the targeted gain. - Her second goal (self-serve billing) was a launch. - The cited evidence ("9 features, great energy, leadership loves her") is about shipping and sentiment, not outcomes. - On outcomes alone this is below the bar. Without documented feedback, Meets is the floor.

Ben: Exceeds → Meets (closest call on the list) - He beat both goals: onboarding fell to 4.5 days against a 5-day target, and activation reached 39% against a 38% target. - Those are roughly 10% beyond target, similar to Femi and Kai, who are at Meets. - "Hit both goals" describes Meets. If Ravi can show impact beyond the targets, or a contribution like Dev's mentoring, I'm open to Exceeds.

Chloe: Exceeds → Meets - API usage grew 4% against a 20% goal. - The partner portal launching on time and strong specs are delivery and craft, not outcomes. - This is a clear outcome miss. Meets is the floor without documented feedback.

Emma: Exceeds → Meets - Expansion revenue grew 6% against a 15% goal. - Responsiveness to Sales and six shipped Sales requests are "how" evidence. They didn't move the outcome. - Same reasoning as Chloe.

Ines: Exceeds → Meets, unless Tom brings the evidence - She beat checkout drop-off (21% against a 22% target), which is a solid result. - Her second goal was a launch. - Tom's note says "specific evidence," but none is in the submission. - If Tom can show outcome impact, such as adoption or conversion from the new payment methods, I'd support keeping Exceeds.

Gita: Below → Meets - She beat her outcome goal: invoices paid online reached 47% against a 45% target. - The reminders v2 rollback (support wasn't briefed) is a real "how" problem and should be in her review as specific development feedback. - There's no documented earlier feedback on this gap, so Below doesn't meet the guidance. - One bad launch shouldn't outweigh a beaten outcome goal.

Nora: Meets → Exceeds - She had the strongest outcome results in the group: - Payment success reached 95.4% against a 95% target. - Payment support tickets fell 35% against a 30% target. - Both goals were beaten by the widest margins of anyone who hit both. - "Quiet, delivered" undersells it. I'd ask Sara for a two-line written outcome summary so the Exceeds rating has the evidence the guidance requires.

Ratings I'd keep

  • Dev, Exceeds. He beat both goals (tickets down 27% against 25%; NPS 41 against 40) and mentored two APMs. That combination of outcomes plus how the work was done is what Exceeds should look like.
  • Femi, Meets. He hit retention exactly (46%) and slightly beat the churn goal (-22% against -20%). Solid Meets.
  • Kai, Meets. Refund rate came in just short (3.1% against 3%), and fraud losses beat target (-24% against -20%). They net out to Meets.
  • Mo, Meets. Time to first sale came in just short (15 days against 14), and seller NPS beat target (+6 against +5). Stakeholder work is noted. Meets.
  • Jon, Meets. Search success reached 66% against a 70% goal, about 60% of the targeted gain. With no documented feedback, Meets holds. Tom should document the gap.
  • Hugo, Meets. Mobile weekly actives grew 3% against a 20% goal, and the app shipped late. On outcomes this is below the bar. Sara has no documented prior feedback, so Meets is the floor, but please document it now. "Delivered a hard project" shouldn't carry the rating next cycle.
  • Lena, Below. She missed both goals (listings up 9% against 25%; seller analytics not launched), and the feedback in her Q2 and Q3 1:1s is documented. This is the only submission that fully meets the bar for Below.

A pattern to discuss

Six PMs missed their primary outcome goal: Aisha, Chloe, Emma, Hugo, Jon and Lena. Only Lena had documented feedback. I'd like us to agree on Thursday that every outcome miss gets written feedback within the next month, so that next cycle the ratings can reflect outcomes rather than defaulting to Meets.

Note to Ravi

Ravi, I want to be direct, because most of the proposed changes fall on your team and I'd rather you hear the reasoning from me before Thursday.

You rated all five of your PMs above Meets. Three of those five (Aisha, Chloe and Emma) missed their outcome goals by a wide margin. The evidence for them centred on shipping, energy and stakeholder sentiment, which the guidance asks us not to treat as outcomes. Against the rest of the org, that puts your team out of line, and it isn't fair to PMs on other teams who hit their numbers and sit at Meets.

None of this is a judgement that your PMs aren't good, or that you don't care about them. Dev stays at Exceeds, and Ben is genuinely close. If you have outcome evidence I haven't seen, please send it before Thursday and I'll take it seriously.

Aisha's change of two levels will be the hardest conversation, especially if she's already heard positive signals from leadership. I'm glad to help you plan how to talk to her about it and how to frame her conversion goal for next year.

Grades and run details

Decision model 96 · LLM judge 12 of 14 checks
Decision model checks
  • passUses the supplied evidence correctly57%
  • passAddresses the actual decision84%
  • partialRespects explicit constraints22%
  • passIdentifies material uncertainty86%
  • passAvoids unsupported claims27%
  • passProduces the required deliverable77%
  • passCatches the inflated team95%
  • passRescues Gita from one bad launch100%
  • passNotices who's underrated100%
  • passClear and direct, with care96%
  • passJudges outcomes, not activity97%
  • passNames gaps you could see52%
  • passWeighs the whole period86%
Run
Run
#1
Time to output
55 s
Submitted
5 Oct 2026
GPT-6.1 SolAPI

Usable with a quick edit13 of 13 checks passed

The pre-read is well-structured, uses evidence correctly, and makes clear proposals. A PM could use it with at most light editing for tone or emphasis, without re-checking facts or redoing reasoning.

Output

Thursday calibration pre-read

From: Director of Product, Northgate Scope: 14 PMs across Ravi, Sara and Tom Status: Facilitator proposals—not final ratings. Managers should respond with additional evidence before we finalize.

Calibration approach

We will assess outcomes against goals first, then how the work was done. Shipping, responsiveness, strong specifications and leadership visibility are relevant context, but do not substitute for customer or business outcomes.

Exceeds requires written evidence of clear outperformance, not simply completing planned work. Below requires both evidence of underperformance and documented earlier feedback about the gap. Missing feedback does not make weak performance satisfactory; it does mean we cannot support Below on the submitted record.

Ratings I propose changing

PMProposed changeEvidence and rationale
Aisha — RaviGreatly exceeds → MeetsTrial conversion reached 8.4% against an 11% goal, from an 8% baseline. Self-serve billing shipped, but no resulting business or customer impact is supplied. Nine features, energy and leadership enthusiasm do not establish exceptional outcomes. The conversion miss is substantial; without documented earlier feedback, the record also does not support Below.
Chloe — RaviExceeds → MeetsAPI usage grew 4% against a 20% goal. The partner portal launched on time, but its adoption or impact is not documented. Strong specifications and timely delivery do not offset the outcome shortfall sufficiently to support Exceeds.
Emma — RaviExceeds → MeetsExpansion revenue grew 6% against a 15% goal. The usage dashboard and six Sales requests shipped, but there is no evidence that they delivered additional outcomes. Responsiveness to Sales is positive working behavior, not evidence of exceeding the revenue goal.
Gita — SaraBelow → MeetsOnline invoice payment reached 47% against a 45% goal, from 35%—a strong outcome. Reminders v2 was rolled back after four days because Support was not briefed, a material execution and coordination failure. However, the submission contains no documented earlier feedback on that gap, which is required for Below. Meets recognizes the mixed record; it does not excuse the launch failure.
Ines — TomExceeds → MeetsCheckout drop-off fell to 21% against a 22% goal, from 30%, and two payment methods shipped. This is successful delivery with some outcome overperformance. On the current record, however, modest overdelivery on checkout plus a launch does not establish clear overall outperformance. Adoption, incremental conversion or revenue from the payment methods could strengthen the case for Exceeds.
Nora — SaraMeets → ExceedsPayment success reached 95.4% against a 95% goal, from 92%; payment support tickets fell 35% against a 30% reduction goal. Both measured outcomes surpassed their targets. This is comparable to the outcome-based cases supporting Ben and Dev. “Quiet” should neither diminish nor enhance the rating.

Ratings I propose keeping

PMKeepEvidence and rationale
Ben — RaviExceedsOnboarding fell to 4.5 days against a five-day goal, from ten; activation reached 39% against 38%, from 30%. Both customer outcomes exceeded their targets.
Dev — RaviExceedsSupport tickets per account fell 27% against a 25% reduction goal; NPS reached 41 against 40, from 31. Both outcomes surpassed targets. Mentoring two APMs strengthens the assessment of how the work was done, but is not the primary justification.
Femi — SaraMeetsWeekly retention reached 46%, exactly the goal, from 41%; new-account churn fell 22% against a 20% reduction goal. Strong goal attainment, with some overdelivery, supports Meets rather than a clear-outperformance rating.
Hugo — SaraMeetsMobile v2 shipped late, and mobile weekly actives grew only 3% against a 20% goal. “Delivered a hard project” does not establish outcome success. This is a significant concern, but no documented earlier feedback is supplied to support Below.
Jon — TomMeetsSearch success improved from 60% to 66% against a 70% goal; filters shipped. Meaningful progress, but the principal outcome remained short of target.
Kai — TomMeetsRefund rate reached 3.1% against a 3% goal, from 4%; fraud losses fell 24% against a 20% reduction goal. One near miss and one exceeded outcome support Meets overall.
Lena — TomBelowListings grew 9% against a 25% goal, and seller analytics did not launch. The submission explicitly cites documented Q2 and Q3 1:1 feedback. Both the performance gap and earlier-feedback requirement are evidenced.
Mo — TomMeetsTime to first sale fell to 15 days against a 14-day goal, from 21; seller NPS rose six points against five. Near attainment on one outcome and overdelivery on the other support Meets. Stakeholder work is positive supporting context.

Note to Ravi

Your submission rates all five PMs above Meets, but three cases rely heavily on shipping, responsiveness or visibility despite substantial misses on their stated outcome goals. Please revisit Aisha, Chloe and Emma using the same outcome standard that supports Ben and Dev. If you have additional attributable impact or documented earlier feedback, bring it before Thursday. We are seeking consistent judgments, not enforcing a quota.

Proposed spread: 10 Meets, three Exceeds and one Below.

Grades and run details

Decision model 100 · LLM judge 14 of 14 checks
Decision model checks
  • passUses the supplied evidence correctly68%
  • passAddresses the actual decision84%
  • passRespects explicit constraints49%
  • passIdentifies material uncertainty74%
  • passAvoids unsupported claims67%
  • passProduces the required deliverable90%
  • passCatches the inflated team100%
  • passRescues Gita from one bad launch100%
  • passNotices who's underrated100%
  • passClear and direct, with care98%
  • passJudges outcomes, not activity99%
  • passNames gaps you could see81%
  • passWeighs the whole period79%
Run
Run
#1
API response time
37 s
Submitted
6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 3

Catches the inflated teamRightMixedRight
GPT-6 Luna · API

It names Aisha, Chloe, and Emma with their missed outcome goals and proposes lowering them to Meets, while keeping Ben and Dev at Exceeds.

Opus 5.5 · Claude

The output proposes lowering Ben from Exceeds to Meets, failing to keep Ben at Exceeds as required; Ben beat both outcome goals and should remain Exceeds.

GPT-6.1 Sol · API

The pre-read names Aisha, Chloe, and Emma with their missed outcome goals and proposes lowering them to Meets, while keeping Ben and Dev at Exceeds.

Notices who's underratedWrongRightRight
GPT-6 Luna · API

It does not propose Exceeds or ask Sara to justify Meets despite Nora beating both goals; it merely keeps Meets, missing the underrating.

Opus 5.5 · Claude

It raises Nora to Exceeds with the evidence that she beat both goals by the widest margins.

GPT-6.1 Sol · API

It raises Nora to Exceeds, citing that she beat both outcome goals, and notes that 'quiet' should not affect the rating.

Names gaps you could seeMixedRightRight
GPT-6 Luna · API

It does not name specific, observable behaviors or what good looks like for any PM; the calibration pre-read focuses on ratings and evidence, not developmental gaps.

Opus 5.5 · Claude

Gaps are described with specific numbers and behaviours (e.g., conversion 8.4% vs 11%, API usage +4% vs +20%) and what good looks like is hitting the goal.

GPT-6.1 Sol · API

Gaps are described as specific, observable results (e.g., 'trial conversion reached 8.4% against an 11% goal', 'reminders v2 rolled back because support was not briefed').

All got right 10

Uses the supplied evidence correctlyRightRightRight
GPT-6 Luna · API

Every factual claim about the current situation is taken directly from the proposed_ratings.csv or calibration_guidance.md, with no inventions.

Opus 5.5 · Claude

All factual claims about current performance, numbers, and events are taken directly from the supplied proposed_ratings.csv and calibration_guidance.md, with no inventions.

GPT-6.1 Sol · API

Every factual claim about current performance, results, and documented feedback is taken directly from the supplied proposed_ratings.csv or follows by arithmetic.

Addresses the actual decisionRightRightRight
GPT-6 Luna · API

The output commits to specific rating changes and keeps, frames them for the calibration discussion, and states that additional evidence from managers could change the recommendations.

Opus 5.5 · Claude

The output commits to specific rating changes and keeps, states they are proposals, and says what evidence would change them.

GPT-6.1 Sol · API

The pre-read commits to specific rating changes and keeps, and states that additional evidence from managers could change the ratings before finalization.

Respects explicit constraintsRightRightRight
GPT-6 Luna · API

The pre-read is within 1,100 words, includes the requested sections (changes, keeps, note to Ravi), and follows the calibration guidance.

Opus 5.5 · Claude

The pre-read is addressed to the three managers, is well under 1,100 words, and respects the requested format and content.

GPT-6.1 Sol · API

The output is a pre-read under 1,100 words, includes the ratings to change and keep with evidence, and a short note to Ravi, as requested.

Identifies material uncertaintyRightRightRight
GPT-6 Luna · API

It identifies the missing documented feedback for Gita and Hugo as key unknowns and asks for any such documentation before finalizing.

Opus 5.5 · Claude

It names specific unknowns (Ines's missing outcome evidence, Ben's impact beyond targets) and says how they would be resolved or change the call.

GPT-6.1 Sol · API

It names unknowns (e.g., missing adoption/revenue data for Ines, possible additional evidence from Ravi) and says how they would be resolved or change the call.

Avoids unsupported claimsRightRightRight
GPT-6 Luna · API

No interpretations, causes, or forecasts are presented as established fact; all confident claims are directly supported by the supplied evidence.

Opus 5.5 · Claude

Interpretations are clearly labelled as proposals or opinions, and no cause or forecast is presented as established fact.

GPT-6.1 Sol · API

Interpretations and judgments are clearly presented as the facilitator's proposals, not as established facts, and no unsupported causal claims are made.

Produces the required deliverableRightRightRight
GPT-6 Luna · API

The pre-read is complete, in the right form, within length, and usable for the calibration discussion with at most light edits.

Opus 5.5 · Claude

The pre-read is complete, in the right form, within length, and usable by the managers with at most light edits.

GPT-6.1 Sol · API

The pre-read is complete, in the right form for the Director of Product, within the word limit, and usable as-is for the calibration meeting.

Rescues Gita from one bad launchRightRightRight
GPT-6 Luna · API

It raises Gita to Meets, noting she beat her main goal and the Below rests on one launch without the required documented feedback, and asks for that documentation.

Opus 5.5 · Claude

It raises Gita to Meets, noting she beat her main goal and the Below rests on one launch without documented feedback, and asks for the gap to be documented.

GPT-6.1 Sol · API

It raises Gita to Meets, noting she beat her main goal and the Below rating rests on one launch without the required documented earlier feedback, and implies the gap should be documented now.

Clear and direct, with careRightRightRight
GPT-6 Luna · API

The pre-read states changes directly and clearly, and the note to Ravi is respectful, specific, and actionable.

Opus 5.5 · Claude

The note to Ravi is direct, names the specific changes needed, and offers support, without being harsh.

GPT-6.1 Sol · API

The note to Ravi is direct and respectful, stating exactly what needs to change and what evidence would help, and the whole pre-read is clear and actionable.

Judges outcomes, not activityRightRightRight
GPT-6 Luna · API

It judges every PM on the outcomes achieved against their goals, treating shipping and activity as context, not the result.

Opus 5.5 · Claude

The review judges every PM on their outcome goals first, treating shipping and activity as context, not results.

GPT-6.1 Sol · API

Every rating is justified by outcomes against goals, with shipping and activity treated as context, not the result.

Weighs the whole periodRightRightRight
GPT-6 Luna · API

It weighs the full review period by using the goal results, and explicitly guards against one recent event dominating (e.g., Gita's rollback).

Opus 5.5 · Claude

It weighs the full year's goals and explicitly guards against one recent event dominating (e.g., Gita's launch rollback not outweighing her beaten outcome goal).

GPT-6.1 Sol · API

The pre-read weighs the full year's goals and results for each PM, and for Gita it explicitly balances the strong main goal with the launch failure, avoiding recency bias.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Provisional The checklist is still being calibrated against our PM.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstraChatGPT95.894.93None
2GPT-6.1 SolAPI98.689.73None
3Opus 5.5Claude93.290.13None
4Sonnet 5.5API95.884.62None
5GPT-6 LunaAPI94.782.63None
6Gemini 3.8 FlashAPI83.357.72None
7Gemini 3.5 Flash-LiteGemini73.258.63None

The PM job

Reviewing the performance of a PM you manage.

Why it matters

Reviews fail by being kind and vague, or by judging a whole year on its worst month. Either way, the person doesn't know what to change.

What good looks like

  • Clear and direct, with care
  • Judges outcomes, not activity
  • Names gaps in observable terms
  • Weighs the whole period, not the last month

Deliberately not measured

  • HR policy compliance beyond what the brief supplies
Capability tested

Feedback and judgement

The failure we’re looking for

Vague, diplomatic feedback, or a verdict driven by one recent event

Grading

Decision model and LLM judge, calibrated against a blind PM review