Humanity’s Last PRD
Latest drop: Haiku 5.5 · 8 Oct
Tasks / Leadership

Write a performance review

Can the model write a review that is clear, fair and specific enough to change what the person does next?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 50% were usable with at most a quick edit.

Reliably right

  1. Judges outcomes, not activity100% pass
    Every rating change and keep is justified by outcomes against goals, with shipping and activity treated as context, not the result.
    GPT-6 Astra · ChatGPT · Fourteen ratings, three managers
  2. Owns the manager's part100% pass
    Ana plainly acknowledges her Q3 mistake and commits to reviewing goal metrics first in every monthly 1:1.
    Opus 5.5 · Claude · Shipped a lot, moved nothing
  3. Clear and direct, with care97% pass
    The note to Ravi is direct, specific about what needs to change, and framed to help him succeed; the whole pre-read is clear and actionable.
    GPT-6 Astra · ChatGPT · Fourteen ratings, three managers

Where it slips

  1. Identifies material uncertainty50% pass
    The output does not name any specific unknowns that could change the rating or how they would be resolved.
    Opus 5.5 · Claude · Shipped a lot, moved nothing
  2. Notices who's underrated69% pass
    It does not propose Exceeds or ask Sara to justify Meets despite Nora beating both goals; it merely keeps Meets, missing the underrating.
    GPT-6 Luna · API · Fourteen ratings, three managers
  3. Catches the inflated team78% pass
    The output proposes lowering Ben from Exceeds to Meets, failing to keep Ben at Exceeds as required; Ben beat both outcome goals and should remain Exceeds.
    Opus 5.5 · Claude · Fourteen ratings, three managers

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Clear and direct, with care

    Does the review say plainly what needs to change, in words the person couldn't misread, while showing it is written to help them succeed?

    Passes when Every main point is stated directly, with the specific change expected and the support on offer.

    Kim ScottLenny’s Podcast

  2. Judges outcomes, not activity

    Does the review judge the person on the outcomes they achieved against their goals, rather than on how much they shipped or how busy they were?

    Passes when Rates against the goals' outcomes, with the evidence, and treats output as context, not the result.

    Product and growth leadersOn Lenny’s Podcast

  3. Names gaps you could see

    Is each area to improve described as specific, observable behaviour with what good would look like, rather than labels like 'be more strategic'?

    Passes when Every gap names what the person did or didn't do, in a specific situation, and what doing it well would look like.

    Product and growth leadersOn Lenny’s Podcast

  4. Weighs the whole period

    Does the review weigh the whole review period, rather than letting one recent event or one strong impression decide the verdict?

    Passes when Covers the whole period in proportion, and explicitly guards against a recent event or one impression dominating.

    Product and growth leadersOn Lenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 3 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Director of Product at Northgate, and you're facilitating Thursday's calibration for 14 PMs across three managers (Ravi, Sara and Tom). Write the pre-read, in no more than 1,100 words: the ratings you'd change and why, each with the evidence, the ones you'd keep, and a short note to Ravi about his ratings. Use the guidance below.\n\nThe proposed ratings and evidence are attached.

What the model was given2 items: calibration_guidance.md, proposed_ratings.csv (one row per PM, as each manager submitted it)
calibration_guidance.md7 lines · Download
# Calibration guidance

- Ratings judge outcomes against goals first, then how the work was done. Shipping is not an outcome.
- Any rating other than Meets needs written evidence of outcomes.
- A Below rating needs documented earlier feedback on the gap.
- Expected spread across the org: most PMs Meets; Exceeds for clear outperformance on outcomes.
- The calibration facilitator proposes changes; managers can respond before ratings are final.
proposed_ratings.csv (one row per PM, as each manager submitted it)pm,manager,proposed_rating,goal_1,goal_1_result,goal_2,goal_2_result,evidence_cited_by_manager Aisha,Ravi,Greatly exceeds,Raise trial conversion 8% to 11%,8.4%,Ship self-serve billing,Shipped,Shipped 9 features; great energy; leadership loves her Ben,Ravi,Exceeds,Cut onboarding time 10 to 5 days,4.5 days,Raise activation 30% to 38%,39%,Hit both goals Chloe,Ravi,Exceeds,Grow API usage 20%,+4%,Launch partner portal,Shipped,Partner portal launched on time; strong specs Dev,Ravi,Exceeds,Reduce support tickets per account 25%,-27%,NPS from 31 to 40,41,Hit both goals; mentored two APMs Emma,Ravi,Exceeds,Expansion revenue +15%,+6%,Ship usage dashboard,Shipped,Very responsive to Sales; shipped 6 Sales requests Femi,Sara,Meets,Raise weekly retention 41% to 46%,46%,Cut churn of new accounts 20%,-22%,"Hit both goals, steady" Gita,Sara,Below,Raise invoices paid online 35% to 45%,47%,Launch reminders v2 by September,"Launched, rolled back after 4 days (support not briefed)",Reminders launch in September was a mess Hugo,Sara,Meets,Ship mobile app v2,Shipped late,Mobile weekly actives +20%,+3%,Delivered a hard project Ines,Tom,Exceeds,Cut checkout drop-off 30% to 22%,21%,Launch two payment methods,Shipped,"Hit both, specific evidence" Jon,Tom,Meets,Search success 60% to 70%,66%,Ship filters,Shipped,"Good progress, more to do" Kai,Tom,Meets,Reduce refund rate 4% to 3%,3.1%,Fraud losses -20%,-24%,Solid year Lena,Tom,Below,Grow marketplace listings 25%,+9%,Launch seller analytics,Not launched,"Missed both goals; feedback given in Q2 and Q3 1:1s, documented" Mo,Tom,Meets,Cut time to first sale for new sellers 21 to 14 days,15 days,Seller NPS +5,+6,"Nearly hit, good stakeholder work" Nora,Sara,Meets,Raise payment success 92% to 95%,95.4%,Cut payment support tickets 30%,-35%,"Quiet, delivered"
What a strong answer doesThe answer key the graders mark against

Judges each PM on outcomes against their goals, as the guidance says. Ravi rated all five of his PMs Exceeds or above, but only Ben and Dev hit both goals; Aisha (Greatly exceeds) missed her conversion goal (8.4% against 11%), Chloe missed API usage (+4% against +20%) and Emma missed expansion (+6% against +15%), and his evidence is output ('shipped 9 features', 'responsive to Sales'). Proposes Aisha, Chloe and Emma at Meets (or lower for Chloe, whose outcome goal was far off), keeps Ben and Dev at Exceeds. Gita (Sara, Below) beat her main goal (47% against 45%) and the Below rests on one September launch, with no documented feedback, which the guidance requires: Meets, with the launch-readiness gap documented now. Nora (Sara, Meets) beat both goals: a candidate for Exceeds. Lena (Tom, Below) has documented feedback and missed both goals: Below stands. Hugo shipped late and missed his outcome: Meets is generous but defensible, or Below needs documentation first. The note to Ravi is direct and respectful: rate on outcomes, not activity, with what evidence he'd need.

Critical failures (cap the score at 40)
  • Keeps Aisha at Greatly exceeds
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

GPT-6 AstraChatGPT100Usable with a quick editGPT-6.1 SolAPI100Usable with a quick editHaiku 5.5API100Usable with a quick editGemini 3.8 FlashAPI98Usable with a quick editOpus 5.5Claude88Needs more workGPT-6 LunaAPI82Needs more workSonnet 5.5API77Needs more workGemini 3.5 Flash-LiteGemini38Needs more work
GPT-6 LunaAPI

Needs more than a quick edit11 of 13 checks passed

Nora's rating is not correctly addressed; a PM would need to re-evaluate her case for Exceeds. The output also lacks observable behavior gaps, though that is less critical for a calibration pre-read.

Output

Thursday calibration pre-read

Proposed changes below are for discussion, not final ratings. We’ll assess outcomes against goals first; shipping, effort, and stakeholder praise are not outcomes by themselves. Managers can share additional evidence before ratings are finalized.

Ratings I’d change

  • Aisha — Greatly exceeds → Meets. Trial conversion reached 8.4% against an 11% goal. Self-serve billing shipped, but shipping is not an outcome, and the cited nine features, energy, and leadership praise don’t establish goal outperformance.
  • Chloe — Exceeds → Meets. API usage grew 4% against a 20% goal. The partner portal shipped, but no result from the launch is cited; strong specs and on-time delivery don’t establish Exceeds.
  • Emma — Exceeds → Meets. Expansion revenue grew 6% against a 15% goal. The dashboard shipped, and six Sales requests were delivered, but neither is evidence of the targeted revenue outcome or clear outperformance.
  • Ines — Exceeds → Meets. Checkout drop-off reached 21% against a 22% goal, while two payment methods shipped. The checkout result beat its target by one point; for the payment-method goal, no outcome beyond shipping is cited. The evidence supports a solid year, not clear outperformance on outcomes.
  • Gita — Below → Meets on the current record. Online invoices reached 47% against a 45% goal. Reminders v2 launched but was rolled back after four days because Support had not been briefed. That is a serious rollout issue, but the record includes no earlier documented feedback on the gap, as required for Below. Please share any such documentation before calibration closes.

Ratings I’d keep

  • Ben — Exceeds. Onboarding fell to 4.5 days against a 5-day goal, and activation reached 39% against 38%: both measured outcomes surpassed target.
  • Dev — Exceeds. Support tickets fell 27% against a 25% goal, and NPS reached 41 against 40. Both outcomes beat their goals; mentoring two APMs is additional contribution.
  • Femi — Meets. Weekly retention reached its 46% goal, and new-account churn fell 22% against a 20% goal.
  • Hugo — Meets, with a material concern to discuss. The app shipped late and mobile weekly actives rose 3% against a 20% goal. The evidence shows a missed outcome; no earlier documented feedback is cited to support Below.
  • Jon — Meets. Search success reached 66% against 70%, and filters shipped. This is progress, not goal attainment; the record does not establish a case for a non-Meets rating.
  • Kai — Meets. Fraud losses fell 24% against a 20% goal; refund rate was 3.1% against a 3% goal. Mixed results support Meets.
  • Lena — Below. Listings grew 9% against a 25% goal, and seller analytics did not launch. Earlier feedback on the gap is documented in Q2 and Q3 1:1s, meeting the Below requirement.
  • Mo — Meets. Time to first sale was 15 days against a 14-day goal; seller NPS rose 6 points against a 5-point goal. One goal narrowly missed, the other was exceeded.
  • Nora — Meets. Payment success reached 95.4% against a 95% goal, and support tickets fell 35% against a 30% goal. Both outcomes beat target, but the evidence supports Meets rather than clear outperformance.

Note to Ravi

Ravi, several of your proposed high ratings cite launches, feature volume, responsiveness, or praise alongside missed outcome goals. For calibration, please anchor any Exceeds or Greatly Exceeds case in clear outcome outperformance; shipping alone is not enough. Please bring any additional outcome evidence you believe changes these recommendations.

Grades and run details

Decision model 92 · LLM judge 79
Decision model checks
  • passUses the supplied evidence correctly87%
  • passAddresses the actual decision52%
  • passRespects explicit constraints39%
  • partialIdentifies material uncertainty31%
  • passAvoids unsupported claims69%
  • passProduces the required deliverable62%
  • passCatches the inflated team100%
  • passRescues Gita from one bad launch100%
  • partialNotices who's underrated41%
  • passClear and direct, with care85%
  • passJudges outcomes, not activity97%
  • passNames gaps you could see78%
  • passWeighs the whole period64%
Run
Run
#1
API response time
25 s
Submitted
5 Oct 2026
Haiku 5.5API

Usable with a quick edit13 of 13 checks passed

The pre-read gives specific changes, evidence, and a clear note to Ravi; it can be used with light edits and does not require re-checking facts.

Output

Calibration pre-read: 14 PMs, Ravi / Sara / Tom

Purpose: Thursday's calibration. Managers can respond to these proposals before ratings are final.

Rules I applied (from calibration guidance): - Ratings start from outcomes against goals. Shipping is an output, and it doesn't count as an outcome. - Any rating other than Meets needs written outcome evidence. - Below needs documented earlier feedback on the gap. Where that's missing, I've capped the rating at Meets. - Working definition for Exceeds (for discussion): both outcome goals beaten. One beat and one hit is Meets.

Summary: 6 changes, 1 hold, 7 keeps. Proposed spread after changes: 3 Exceeds, 10 Meets, 1 Below.

---

Ratings I'd change

Aisha (Ravi): Greatly exceeds → Meets - Trial conversion reached 8.4% against an 11% target. That's about 13% of the intended 3-point lift. - "Shipped self-serve billing" is an output. Nine features shipped is also an output. - "Great energy" and "leadership loves her" are not outcome evidence. - Greatly exceeds is not supported. Below would need documented feedback on the conversion gap, which the submission doesn't cite.

Chloe (Ravi): Exceeds → Meets - API usage grew 4% against a 20% target, which is one-fifth of the goal. - The partner portal shipped on time. That's an output, and it doesn't offset the miss. - "Strong specs" describes how the work was done, not what it achieved. - Ravi should show what the portal changed in usage, if anything. If the outcome is attributable and I've missed it, I'll revisit.

Emma (Ravi): Exceeds → Meets - Expansion revenue grew 6% against a 15% target, which is 40% of the goal. - The usage dashboard shipped, and six Sales requests were delivered. Both are outputs. Responsiveness to Sales is a good working habit, but it isn't an outcome.

Ines (Tom): Exceeds → Meets (open to revisiting) - Checkout drop-off reached 21% against a 22% target. That's a clean beat, but only by one point. - Two payment methods shipped. The record has no adoption or conversion metric for them, so that goal is an output. - Under the working definition, one beat plus one output is Meets. If Tom has adoption data for the new payment methods, I'll revisit Exceeds on Thursday.

Nora (Sara): Meets → Exceeds - Payment success reached 95.4% against a 95% target. - Payment-related support tickets fell 35% against a 30% target. - Both outcome goals were beaten, the same pattern as Ben. "Quiet" isn't a criterion, and the outcomes support a higher rating than proposed.

---

Ratings on hold (need documentation before Thursday)

Gita (Sara): Below → Meets (Below only if documentation exists) - Invoices paid online reached 47% against a 45% target. That's an outcome hit. - Reminders v2 launched, was rolled back after four days, and support wasn't briefed. That's a real failure in how the work was done, and it needs a written account and a postmortem. - Below requires documented earlier feedback on the gap. None is cited, and one of two outcome goals was hit. - Sara: if you have documented feedback on the launch-readiness issue, I'll reconsider Below.

Hugo (Sara): Meets (hold) - Mobile weekly actives grew 3% against a 20% target, which is 15% of the goal. The app shipped late. - "Delivered a hard project" is an output description, and the outcome miss is large. - Meets is the ceiling the current record supports. If the mobile gap was raised in 1:1s before Q3, Below is defensible. Sara, please confirm before Thursday.

---

Ratings I'd keep

PMRatingWhy
Ben (Ravi)ExceedsBoth beaten: onboarding 4.5 days vs 5; activation 39% vs 38%. Cite the numbers, not "hit both goals."
Dev (Ravi)ExceedsBoth beaten: support tickets per account −27% vs −25%; NPS 41 vs 40. Mentoring two APMs is a useful addition.
Femi (Sara)MeetsRetention exactly at target (46%); churn beat (−22% vs −20%). One beat, one hit.
Jon (Tom)MeetsSearch success 66% vs 70%, about 60% of the gain. Filters shipped. Partial progress is Meets.
Kai (Tom)MeetsRefund rate 3.1% vs 3.0%, a narrow miss. Fraud losses −24% vs −20%, a beat.
Lena (Tom)BelowBoth goals missed (listings +9% vs +25%; seller analytics not launched). Feedback documented in Q2 and Q3 1:1s. Tom, please attach those notes to the file.
Mo (Tom)MeetsTime to first sale 15 days vs 14 target, about 86% of the gain. Seller NPS +6 vs +5.

---

Note to Ravi

Ravi, five of your five ratings are above Meets, and three of them (Aisha, Chloe, Emma) rest on shipping, responsiveness and reputation. The outcome numbers in the same rows don't support those ratings: 8.4% vs 11%, +4% vs +20%, and +6% vs +15%.

Ben and Dev are your strongest calls because their evidence is outcomes, and that's the standard we should use for all five.

I'm not suggesting Aisha, Chloe or Emma had weak years. Nine features and an on-time portal are real work. But ratings have to be tied to the goals we set.

Before Thursday, could you: 1. For Aisha, Chloe and Emma, write the outcome analysis for each goal. Include what was tried, what moved, and what you think caused the gap. 2. Tell us whether any feedback on those gaps was documented earlier in the year. If it was, Below becomes available. 3. Respond to the proposed changes, so we can settle them on Thursday.

Happy to walk through this 1:1 before the session.

---

Expected spread after changes

  • Greatly exceeds: 0
  • Exceeds: 3 (Ben, Dev, Nora)
  • Meets: 10
  • Below: 1 (Lena), with Gita and Hugo pending documentation

This is within the guidance: most PMs at Meets, Exceeds reserved for clear outperformance on outcomes.

Grades and run details

Decision model 100 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly70%
  • passAddresses the actual decision86%
  • passRespects explicit constraints33%
  • passIdentifies material uncertainty98%
  • passAvoids unsupported claims67%
  • passProduces the required deliverable85%
  • passCatches the inflated team100%
  • passRescues Gita from one bad launch100%
  • passNotices who's underrated100%
  • passClear and direct, with care100%
  • passJudges outcomes, not activity100%
  • passNames gaps you could see79%
  • passWeighs the whole period79%
Run
Run
#1
API response time
47 s
Submitted
8 Oct 2026
Gemini 3.8 FlashAPI

Usable with a quick edit13 of 13 checks passed

The memo is exhaustive for calibration, uses only supplied evidence, and could be sent with at most minor wording tweaks.

Output

To: Ravi, Sara, Tom From: Director of Product Date: October 24, 2023 Subject: Pre-Read: Product Calibration (Thursday)

Team—ahead of Thursday’s calibration, I have reviewed all 14 proposed ratings against our calibration principles: 1. Outcomes over outputs: We evaluate impact against goals first, then how the work was done. Shipping is an output, not an outcome. 2. Evidence-based: Any rating other than Meets requires documented outcome data. 3. Guardrails on Below: A Below rating requires documented prior feedback on the gap. 4. Distribution: Most PMs should land at Meets, with Exceeds reserved for clear outperformance on business and customer outcomes.

Below are my proposed adjustments, the ratings I recommend keeping, and the discussion points for Thursday.

---

1. Proposed Rating Changes

#### Aisha (Ravi): Greatly Exceeds $\rightarrow$ Meets * Evidence: Trial conversion increased from 8% to 8.4% (well short of the 11% target). Self-serve billing and 9 features shipped. * Rationale: Aisha’s energy and velocity are appreciated, but shipping features and executive visibility do not constitute business outcomes. She significantly missed her primary conversion goal. Delivering features without moving the core metric places her solidly at Meets.

#### Chloe (Ravi): Exceeds $\rightarrow$ Meets * Evidence: API usage grew +4% (target was +20%). Partner portal shipped on time with strong specs. * Rationale: Timely delivery and strong documentation are expected baseline PM capabilities. Chloe missed her outcome metric by 16 percentage points. Meets is the appropriate rating.

#### Emma (Ravi): Exceeds $\rightarrow$ Meets * Evidence: Expansion revenue grew +6% (target was +15%). Shipped usage dashboard and 6 sales requests. * Rationale: Being responsive to Sales and clearing backlog requests reflects high effort, but it produced less than half of the targeted revenue expansion. Meets appropriately reflects delivery without outsized outcome impact.

#### Gita (Sara): Below $\rightarrow$ Meets * Evidence: Invoices paid online reached 47% (exceeded the 45% target, up from 35%). Reminders v2 launched in September and was rolled back after 4 days due to unbriefed support. * Rationale: Gita significantly outperformed on her primary outcome metric (+12 pts). The launch execution for reminders was unacceptable, but: 1. Our policy prioritizes outcomes first, then execution. 2. A Below rating explicitly requires documented prior feedback on the gap, which is absent here. Gita should receive Meets, paired with explicit operational feedback regarding launch readiness.

#### Nora (Sara): Meets $\rightarrow$ Exceeds * Evidence: Payment success increased to 95.4% (beat 95% target, from 92%). Payment support tickets dropped -35% (beat -30% target). * Rationale: Nora exceeded both of her core outcome metrics. Quiet, low-drama delivery should be rewarded just as visibly as vocal execution. This is textbook outperformance.

---

2. Ratings Kept as Proposed

  • Ben (Ravi) — Exceeds: Cut onboarding time to 4.5 days (target: 5) and grew activation to 39% (target: 38%). Clear outperformance on both outcome metrics.
  • Dev (Ravi) — Exceeds: Reduced support tickets by -27% (target: -25%) and lifted NPS to 41 (target: 40), while mentoring two APMs. Strong outcomes combined with positive org impact.
  • Ines (Tom) — Exceeds: Reduced checkout drop-off to 21% (beat target of 22%) and successfully shipped two new payment methods. Met both outcome and delivery bars.
  • Jon (Tom) — Meets: Improved search success to 66% (target: 70%) and shipped filters. Solid progress; Meets is accurate.
  • Kai (Tom) — Meets: Refund rate finished at 3.1% (target: 3.0%) and fraud losses dropped -24% (beat -20% target). Consistent, dependable execution.
  • Mo (Tom) — Meets: Time to first sale dropped to 15 days (target: 14) and seller NPS reached +6 (target: +5). Solid year with strong stakeholder management.
  • Lena (Tom) — Below: Marketplace listings grew only +9% (target: +25%) and seller analytics failed to launch. Missed both goals. Formal feedback was documented in Q2 and Q3 1:1s, meeting our required standard for Below.

---

3. Key Discussion Items for Calibration

  • Hugo (Sara — Proposed: Meets): Hugo shipped mobile v2 late and achieved +3% WAU growth against a +20% target. By outcomes alone, this is Below performance. However, our guidance requires documented earlier feedback to assign a Below. If Sara has prior documented feedback, we should move Hugo to Below. If not, Hugo stays at Meets, but must be placed on a structured performance plan immediately.
  • Femi (Sara — Proposed: Meets): Femi hit weekly retention (46%) and beat account churn reduction (-22% vs. -20%). We will discuss whether Femi’s consistent delivery warrants an Exceeds alongside Nora.

---

4. Org Distribution Impact

* Proposed initial spread: 1 Greatly Exceeds / 5 Exceeds / 6 Meets / 2 Below (Ravi: 100% Exceeds+, Sara: 0% Exceeds+, Tom: 25% Exceeds) * Revised spread: 0 Greatly Exceeds / 4 Exceeds (29%) / 9 Meets (64%) / 1 Below (7%) (pending Hugo/Femi) This aligns with our goal of centering the team at Meets while recognizing demonstrable outperformance.

---

5. Note to Ravi

Ravi,

I want to speak directly to your proposed ratings before we meet Thursday. You put forward 5 out of 5 PMs at Exceeds or Greatly Exceeds. While I appreciate your instinct to back your team, this submission has two significant issues:

  1. Outputs vs. Outcomes: For Aisha, Chloe, and Emma, your rationale centers on features shipped, sales responsiveness, spec quality, and team energy. These are outputs and behaviors. When we look at their business targets, all three fell well short: trial conversion missed by 2.6 pts, API usage missed by 16 pts, and expansion revenue missed by 9 pts. Shipping software that does not move the target metric is not Exceeds performance.
  2. Parity across the org: Tom and Sara evaluated their teams strictly against measurable results. If we reward activity on your team while holding their PMs to hard outcome metrics, we undermine the integrity of the process and demotivate top performers across Northgate (such as Nora, who exceeded all metrics without fanfare).

Ben and Dev demonstrated genuine outperformance on outcomes and have earned their Exceeds ratings. Aisha, Chloe, and Emma are valued contributors, but their results are standard Meets. Let’s align on this calibration standard so we can give your PMs clear, actionable coaching on driving commercial and customer impact, rather than just shipping volume.

Looking forward to a productive session Thursday.

Grades and run details

Decision model 96 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly29%
  • passAddresses the actual decision82%
  • passRespects explicit constraints26%
  • passIdentifies material uncertainty92%
  • partialAvoids unsupported claims50%
  • passProduces the required deliverable83%
  • passCatches the inflated team100%
  • passRescues Gita from one bad launch100%
  • passNotices who's underrated100%
  • passClear and direct, with care99%
  • passJudges outcomes, not activity100%
  • passNames gaps you could see52%
  • passWeighs the whole period78%
Run
Run
#1
API response time
39 s
Submitted
6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 2

Notices who's underratedWrongRightRight
GPT-6 Luna · API

It does not propose Exceeds or ask Sara to justify Meets despite Nora beating both goals; it merely keeps Meets, missing the underrating.

Haiku 5.5 · API

It raises Nora from Meets to Exceeds because she beat both outcome goals, using her payment success and ticket reduction results.

Gemini 3.8 Flash · API

It highlights that Nora beat both goals (95.4%, -35%) and proposes raising her from Meets to Exceeds.

Names gaps you could seeMixedRightRight
GPT-6 Luna · API

It does not name specific, observable behaviors or what good looks like for any PM; the calibration pre-read focuses on ratings and evidence, not developmental gaps.

Haiku 5.5 · API

Gaps are described as specific results (conversion 8.4% vs 11%, API +4% vs +20%, app late and actives +3%, launch rollback) and what good would look like is implied by the goals.

Gemini 3.8 Flash · API

The gaps for Ravi cite concrete evidence (features shipped, sales responsiveness) and define what good looks like (aligning on calibration standard, rewarding metric movement).

All got right 11

Uses the supplied evidence correctlyRightRightRight
GPT-6 Luna · API

Every factual claim about the current situation is taken directly from the proposed_ratings.csv or calibration_guidance.md, with no inventions.

Haiku 5.5 · API

All claimed numbers and facts appear in the supplied CSV or are derived by straightforward arithmetic; nothing about the current situation is invented.

Gemini 3.8 Flash · API

Every performance number and outcome result in the output matches the supplied proposed_ratings.csv.

Addresses the actual decisionRightRightRight
GPT-6 Luna · API

The output commits to specific rating changes and keeps, frames them for the calibration discussion, and states that additional evidence from managers could change the recommendations.

Haiku 5.5 · API

The output clearly proposes specific rating changes, keeps, and holds, with conditions (missing documentation) that would alter the calls.

Gemini 3.8 Flash · API

The memo commits to specific rating changes (Aisha to Meets, Gita to Meets, Nora to Exceeds, etc.) and keeps others, giving a clear answer to the calibration decision.

Respects explicit constraintsRightRightRight
GPT-6 Luna · API

The pre-read is within 1,100 words, includes the requested sections (changes, keeps, note to Ravi), and follows the calibration guidance.

Haiku 5.5 · API

The output is a pre-read within 1,100 words, addresses the required elements (changes, keeps, note to Ravi), and follows the format.

Gemini 3.8 Flash · API

The pre-read is under 1,100 words, respects the calibration guidance, and includes all requested sections (changes, keeps, discussion, note to Ravi).

Identifies material uncertaintyRightRightRight
GPT-6 Luna · API

It identifies the missing documented feedback for Gita and Hugo as key unknowns and asks for any such documentation before finalizing.

Haiku 5.5 · API

It identifies the unknowns (documentation for Gita and Hugo, adoption data for Ines) and says how they would resolve the ratings.

Gemini 3.8 Flash · API

It explicitly flags that Hugo's Below rating depends on whether Sara has documented prior feedback and says what to do in each case.

Avoids unsupported claimsRightRightRight
GPT-6 Luna · API

No interpretations, causes, or forecasts are presented as established fact; all confident claims are directly supported by the supplied evidence.

Haiku 5.5 · API

Interpretations (e.g., '40% of the goal') are clearly based on arithmetic; no causes or forecasts are presented as established fact without support.

Gemini 3.8 Flash · API

Assessments are framed as the director's judgment; no causal claim is asserted as fact beyond the supplied evidence.

Produces the required deliverableRightRightRight
GPT-6 Luna · API

The pre-read is complete, in the right form, within length, and usable for the calibration discussion with at most light edits.

Haiku 5.5 · API

The pre-read is complete with changes, keeps, a dedicated note to Ravi, and could be used by a PM with only minor edits.

Gemini 3.8 Flash · API

The pre-read is complete, in memo format, addressed to the three managers, and actionable as a discussion brief.

Catches the inflated teamRightRightRight
GPT-6 Luna · API

It names Aisha, Chloe, and Emma with their missed outcome goals and proposes lowering them to Meets, while keeping Ben and Dev at Exceeds.

Haiku 5.5 · API

It names Aisha, Chloe and Emma, cites their outcome misses, proposes lowering them to Meets, and keeps Ben and Dev at Exceeds.

Gemini 3.8 Flash · API

It names Aisha (8.4% vs 11%), Chloe (+4% vs +20%), and Emma (+6% vs +15%) and proposes lowering all three to Meets, while keeping Ben and Dev at Exceeds.

Rescues Gita from one bad launchRightRightRight
GPT-6 Luna · API

It raises Gita to Meets, noting she beat her main goal and the Below rests on one launch without the required documented feedback, and asks for that documentation.

Haiku 5.5 · API

It raises Gita to Meets citing her beat on the main goal and the lack of documented feedback, and asks for the launch gap to be documented.

Gemini 3.8 Flash · API

It notes Gita beat her main outcome (47% > 45%) and that Below lacks documented prior feedback, raising her to Meets with a note to document the launch gap.

Clear and direct, with careRightRightRight
GPT-6 Luna · API

The pre-read states changes directly and clearly, and the note to Ravi is respectful, specific, and actionable.

Haiku 5.5 · API

The note to Ravi is direct, names the three PMs and the numbers, asks for specific outcome analyses and feedback documentation, and offers support.

Gemini 3.8 Flash · API

The note to Ravi directly says his ratings are based on outputs, not outcomes, gives specific examples, and offers coaching on driving impact rather than shipping volume.

Judges outcomes, not activityRightRightRight
GPT-6 Luna · API

It judges every PM on the outcomes achieved against their goals, treating shipping and activity as context, not the result.

Haiku 5.5 · API

Every rating change is argued from outcome attainment against goals, with outputs and activity treated as context not the result.

Gemini 3.8 Flash · API

Every rating change is justified by comparing goal results to targets, not by shipped features or stakeholder praise.

Weighs the whole periodRightRightRight
GPT-6 Luna · API

It weighs the full review period by using the goal results, and explicitly guards against one recent event dominating (e.g., Gita's rollback).

Haiku 5.5 · API

The review uses the full-year goal results for every PM, and Gita's launch failure does not dominate her rating because the other goal was met and documentation is absent.

Gemini 3.8 Flash · API

It weighs full-period results (e.g., Gita's strong metric outweighs one bad launch, Hugo's missed WAU is considered against delivery, not just the late shipment).

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstraChatGPT93.892.32None
2Haiku 5.5API93.888.52None
3GPT-6.1 SolAPI97.984.62None
4Opus 5.5Claude91.885.22None
5GPT-6 LunaAPI94.177.72None
6Gemini 3.8 FlashAPI89.776.92None
7Sonnet 5.5API90.174.22None
8Gemini 3.5 Flash-LiteGemini74.449.52None

What you get for the cost

Haiku 5.5 · API matches GPT-6 Astra · ChatGPT at about a fortieth of the cost.

Task score against relative cost for each setup. The table below has the same figures.50607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 91, 70× the cheapestHaiku 5.5 · API: task score 91, 1.7× the cheapestGPT-6.1 Sol · API: task score 89, 13× the cheapestOpus 5.5 · Claude: task score 87, 61× the cheapestGPT-6 Luna · API: task score 82, the cheapestGemini 3.8 Flash · API: task score 82, 12× the cheapestSonnet 5.5 · API: task score 81, 31× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 61, 1.9× the cheapest (1 of 2 outputs estimated)GPT-6 Astra · ChatGPTHaiku 5.5 · APIGPT-6.1 Sol · APIOpus 5.5 · ClaudeGPT-6 Luna · APIGemini 3.8 Flash · APISonnet 5.5 · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Reviewing the performance of a PM you manage.

Why it matters

Reviews fail by being kind and vague, or by judging a whole year on its worst month. Either way, the person doesn't know what to change.

What good looks like

  • Clear and direct, with care
  • Judges outcomes, not activity
  • Names gaps in observable terms
  • Weighs the whole period, not the last month

Deliberately not measured

  • HR policy compliance beyond what the brief supplies
Capability tested

Feedback and judgement

The failure we’re looking for

Vague, diplomatic feedback, or a verdict driven by one recent event

Grading

Decision model and LLM judge, calibrated against a blind PM review