Humanity’s Last PRD
Latest drop: Gemini 4 Argon · 13 Oct
Tasks / Leadership

Write a performance review

Can the model write a review that is clear, fair and specific enough to change what the person does next?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 56% were usable with at most a quick edit.

Reliably right

  1. Judges outcomes, not activity100% pass
    Every rating change and keep is justified by outcomes against goals, with shipping and activity treated as context, not the result.
    GPT-6 Astra · ChatGPT · Fourteen ratings, three managers
  2. Owns the manager's part100% pass
    Ana explicitly owns that she said he was on track in Q3 without discussing the metrics, and commits to reviewing goal metrics in future 1:1s.
    Gemini 4 Argon · API · Shipped a lot, moved nothing
  3. Respects explicit constraints97% pass
    The output is a pre-read under 1,100 words, includes the ratings to change and keep with evidence, and a note to Ravi, respecting all explicit constraints.
    GPT-6 Astra · ChatGPT · Fourteen ratings, three managers

Where it slips

  1. Sets next goals as outcomes, with support47% pass
    Goals are partly process-oriented (implement a mandatory process, create a framework) and lack specific support from Ana beyond a general offer.
    Gemini 3.5 Flash-Lite · Gemini · Shipped a lot, moved nothing
  2. Identifies material uncertainty51% pass
    The output does not name any specific unknowns that could change the rating or how they would be resolved.
    Opus 5.5 · Claude · Shipped a lot, moved nothing
  3. Catches the inflated team69% pass
    The output proposes lowering Ben from Exceeds to Meets, failing to keep Ben at Exceeds as required; Ben beat both outcome goals and should remain Exceeds.
    Opus 5.5 · Claude · Fourteen ratings, three managers

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Clear and direct, with care

    Does the review say plainly what needs to change, in words the person couldn't misread, while showing it is written to help them succeed?

    Passes when Every main point is stated directly, with the specific change expected and the support on offer.

    Kim ScottLenny’s Podcast

  2. Judges outcomes, not activity

    Does the review judge the person on the outcomes they achieved against their goals, rather than on how much they shipped or how busy they were?

    Passes when Rates against the goals' outcomes, with the evidence, and treats output as context, not the result.

    Product and growth leadersOn Lenny’s Podcast

  3. Names gaps you could see

    Is each area to improve described as specific, observable behaviour with what good would look like, rather than labels like 'be more strategic'?

    Passes when Every gap names what the person did or didn't do, in a specific situation, and what doing it well would look like.

    Product and growth leadersOn Lenny’s Podcast

  4. Weighs the whole period

    Does the review weigh the whole review period, rather than letting one recent event or one strong impression decide the verdict?

    Passes when Covers the whole period in proportion, and explicitly guards against a recent event or one impression dominating.

    Product and growth leadersOn Lenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 3 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Director of Product at Northgate, and you're facilitating Thursday's calibration for 14 PMs across three managers (Ravi, Sara and Tom). Write the pre-read, in no more than 1,100 words: the ratings you'd change and why, each with the evidence, the ones you'd keep, and a short note to Ravi about his ratings. Use the guidance below.\n\nThe proposed ratings and evidence are attached.

What the model was given2 items: calibration_guidance.md, proposed_ratings.csv (one row per PM, as each manager submitted it)
calibration_guidance.md7 lines · Download
# Calibration guidance

- Ratings judge outcomes against goals first, then how the work was done. Shipping is not an outcome.
- Any rating other than Meets needs written evidence of outcomes.
- A Below rating needs documented earlier feedback on the gap.
- Expected spread across the org: most PMs Meets; Exceeds for clear outperformance on outcomes.
- The calibration facilitator proposes changes; managers can respond before ratings are final.
proposed_ratings.csv (one row per PM, as each manager submitted it)pm,manager,proposed_rating,goal_1,goal_1_result,goal_2,goal_2_result,evidence_cited_by_manager Aisha,Ravi,Greatly exceeds,Raise trial conversion 8% to 11%,8.4%,Ship self-serve billing,Shipped,Shipped 9 features; great energy; leadership loves her Ben,Ravi,Exceeds,Cut onboarding time 10 to 5 days,4.5 days,Raise activation 30% to 38%,39%,Hit both goals Chloe,Ravi,Exceeds,Grow API usage 20%,+4%,Launch partner portal,Shipped,Partner portal launched on time; strong specs Dev,Ravi,Exceeds,Reduce support tickets per account 25%,-27%,NPS from 31 to 40,41,Hit both goals; mentored two APMs Emma,Ravi,Exceeds,Expansion revenue +15%,+6%,Ship usage dashboard,Shipped,Very responsive to Sales; shipped 6 Sales requests Femi,Sara,Meets,Raise weekly retention 41% to 46%,46%,Cut churn of new accounts 20%,-22%,"Hit both goals, steady" Gita,Sara,Below,Raise invoices paid online 35% to 45%,47%,Launch reminders v2 by September,"Launched, rolled back after 4 days (support not briefed)",Reminders launch in September was a mess Hugo,Sara,Meets,Ship mobile app v2,Shipped late,Mobile weekly actives +20%,+3%,Delivered a hard project Ines,Tom,Exceeds,Cut checkout drop-off 30% to 22%,21%,Launch two payment methods,Shipped,"Hit both, specific evidence" Jon,Tom,Meets,Search success 60% to 70%,66%,Ship filters,Shipped,"Good progress, more to do" Kai,Tom,Meets,Reduce refund rate 4% to 3%,3.1%,Fraud losses -20%,-24%,Solid year Lena,Tom,Below,Grow marketplace listings 25%,+9%,Launch seller analytics,Not launched,"Missed both goals; feedback given in Q2 and Q3 1:1s, documented" Mo,Tom,Meets,Cut time to first sale for new sellers 21 to 14 days,15 days,Seller NPS +5,+6,"Nearly hit, good stakeholder work" Nora,Sara,Meets,Raise payment success 92% to 95%,95.4%,Cut payment support tickets 30%,-35%,"Quiet, delivered"
What a strong answer doesThe answer key the graders mark against

Judges each PM on outcomes against their goals, as the guidance says. Ravi rated all five of his PMs Exceeds or above, but only Ben and Dev hit both goals; Aisha (Greatly exceeds) missed her conversion goal (8.4% against 11%), Chloe missed API usage (+4% against +20%) and Emma missed expansion (+6% against +15%), and his evidence is output ('shipped 9 features', 'responsive to Sales'). Proposes Aisha, Chloe and Emma at Meets (or lower for Chloe, whose outcome goal was far off), keeps Ben and Dev at Exceeds. Gita (Sara, Below) beat her main goal (47% against 45%) and the Below rests on one September launch, with no documented feedback, which the guidance requires: Meets, with the launch-readiness gap documented now. Nora (Sara, Meets) beat both goals: a candidate for Exceeds. Lena (Tom, Below) has documented feedback and missed both goals: Below stands. Hugo shipped late and missed his outcome: Meets is generous but defensible, or Below needs documentation first. The note to Ravi is direct and respectful: rate on outcomes, not activity, with what evidence he'd need.

Critical failures (cap the score at 40)
  • Keeps Aisha at Greatly exceeds
Task

Staff-level · v1.1

The outputs

9 setups answered this brief.

GPT-6 AstraChatGPT98Usable with a quick editGPT-6.1 SolAPI98Usable with a quick editHaiku 5.5API98Usable with a quick editGemini 3.8 FlashAPI96Usable with a quick editGemini 4 ArgonAPI89Usable with a quick editOpus 5.5Claude82Needs more workGPT-6 LunaAPI79Needs more workSonnet 5.5API73Needs more workGemini 3.5 Flash-LiteGemini29Needs more work
Gemini 4 ArgonAPI

Usable with a quick edit11 of 13 checks passed

The output contains a minor factual error (claiming four PMs beat two pure outcome goals instead of three); that can be fixed with a quick edit, and the reasoning and decisions otherwise stand.

Output

To: Ravi, Sara, Tom From: Director of Product Subject: Pre-read: Thursday PM Calibration

Team,

Ahead of Thursday’s calibration for our 14 PMs, I’ve reviewed your submissions against our calibration guidance.

As a reminder, our standard is: 1. Ratings judge outcomes against goals first, then how the work was done. Shipping is not an outcome. 2. Any rating other than Meets requires written evidence of outcomes. 3. A Below rating requires documented earlier feedback on the gap. 4. Most PMs should land at Meets, with Exceeds reserved for clear outperformance on outcomes.

Right now, our proposed spread is top-heavy (1 Greatly Exceeds, 5 Exceeds, 6 Meets, 2 Below) and applies uneven bars across managers. Below are my proposed adjustments to ensure we are grading on a consistent, outcome-driven standard across Northgate. These are proposals—come ready on Thursday to respond with additional context or push back with evidence before we finalize.

---

Proposed Rating Changes (6 PMs)

1. Aisha (Ravi): Greatly Exceeds $\rightarrow$ Meets (or Below, pending discussion) * Evidence: Goal 1 (raise trial conversion from 8% to 11%) landed at 8.4%. Goal 2 (ship self-serve billing) was shipped. Cited evidence: "Shipped 9 features; great energy; leadership loves her." * Why change: Shipping 9 features and self-serve billing are outputs, not outcomes. On her sole outcome metric, she achieved less than 15% of the targeted lift (8.4% vs. 11%). While positive energy and leadership visibility are great, ratings judge outcomes first. Because she missed her primary outcome goal, she cannot be rated above Meets. (If documented feedback was given on the conversion miss, Below would be on the table, though I assume none exists given the proposed rating.)

2. Chloe (Ravi): Exceeds $\rightarrow$ Meets * Evidence: Goal 1 (grow API usage 20%) landed at +4%. Goal 2 (launch partner portal) shipped on time with strong specs. * Why change: Launching the portal and writing strong specs are outputs and craft inputs, not outcomes. On her primary business outcome, Chloe missed the 20% growth target by a wide margin (+4%). Missing the main outcome goal rules out Exceeds.

3. Emma (Ravi): Exceeds $\rightarrow$ Meets * Evidence: Goal 1 (expansion revenue +15%) landed at +6%. Goal 2 (ship usage dashboard) shipped. Cited evidence: "Very responsive to Sales; shipped 6 Sales requests." * Why change: Responsiveness to Sales and shipping requests/dashboards are outputs. Those outputs did not translate into the target outcome: expansion revenue grew 6% against a 15% goal. Without outperformance on outcomes, Meets is the ceiling.

4. Gita (Sara): Below $\rightarrow$ Meets * Evidence: Goal 1 (raise invoices paid online 35% to 45%) landed at 47%. Goal 2 (launch reminders v2 by Sept) launched, then rolled back after 4 days because support wasn't briefed. Cited evidence: "Reminders launch in September was a mess." * Why change: Two reasons against our guidance: 1. Outcomes first: Gita outperformed her primary outcome goal by 2 percentage points (47% vs. 45%). The botched launch of reminders v2 is a legitimate execution ("how") failure that prevented an Exceeds, but outcomes come first. 2. Process requirement: A Below rating requires documented earlier feedback on the gap. A single botched launch in September with no cited record of prior performance feedback does not meet the bar for Below.

5. Nora (Sara): Meets $\rightarrow$ Exceeds * Evidence: Goal 1 (raise payment success 92% to 95%) landed at 95.4%. Goal 2 (cut payment support tickets 30%) landed at -35%. Cited evidence: "Quiet, delivered." * Why change: Nora is one of only four PMs in the org who beat two pure outcome goals—and she beat her support ticket reduction goal substantially (-35% vs. -30%). Being "quiet" is a personality trait, not a performance deficit. We must reward actual business impact over volume/visibility. Sara, please bring a brief written summary of Nora’s outcome evidence to formalize this Exceeds.

6. Hugo (Sara): Meets $\rightarrow$ Hold at Meets (Flagged for discussion / Q1 plan) * Evidence: Goal 1 (ship mobile app v2) shipped late. Goal 2 (mobile weekly actives +20%) landed at +3%. Cited evidence: "Delivered a hard project." * Why flag: On outcomes alone (+3% vs +20% target, plus a late ship), Hugo’s performance looks like Below. However, because Sara has not cited documented earlier feedback on the gap, our guidance prevents moving him to Below in this cycle. He stays at Meets by rule, but Sara and I need to align on Thursday about setting clear, documented expectations with him immediately.

---

Ratings to Keep (8 PMs)

  • Ben (Ravi) – Keep at Exceeds: Beat both pure outcome goals (onboarding time cut to 4.5 days vs. 5-day goal; activation raised to 39% vs. 38% goal). Clear outperformance. (Note: Ravi, please expand the 3-word written evidence slightly for the final record.)
  • Dev (Ravi) – Keep at Exceeds: Beat both pure outcome goals (support tickets -27% vs. -25% goal; NPS 41 vs. 40 goal) and demonstrated strong "how" by mentoring two APMs.
  • Ines (Tom) – Keep at Exceeds: Outperformed her primary outcome goal (checkout drop-off cut to 21% vs. 22% target) and delivered her two payment methods with specific evidence.
  • Femi (Sara) – Keep at Meets: Hit Goal 1 on the nose (46% retention) and beat Goal 2 (-22% churn vs. -20%). Strong, steady year. Borderline for Exceeds, so worth a 2-minute check on Thursday compared to Ben/Dev/Nora, but Meets is defensible.
  • Jon (Tom) – Keep at Meets: Solid progress on search success (66% vs. 70% goal) and shipped filters. Textbook Meets.
  • Kai (Tom) – Keep at Meets: Nearly hit refund rate (3.1% vs. 3.0%) and beat fraud loss reduction (-24% vs. -20%). Balanced out to a solid Meets.
  • Mo (Tom) – Keep at Meets: Nearly hit time to first sale (15 days vs. 14) and beat Seller NPS (+6 vs. +5) with good stakeholder execution. Solid Meets.
  • Lena (Tom) – Keep at Below: Missed both outcome and output goals significantly (+9% listings vs. 25% goal; analytics not launched), and Tom has documented feedback from Q2 and Q3 1:1s. Meets all criteria for Below.

Revised Org Spread: 4 Exceeds (29%), 9 Meets (64%), 1 Below (7%). This matches our expected org distribution and applies a single standard to everyone.

---

A Quick Note to Ravi

Ravi—looking at your submissions as a group, you proposed every one of your 5 PMs for Exceeds or Greatly Exceeds. While Ben and Dev genuinely crushed their outcome targets and earned Exceeds, the cases for Aisha, Chloe, and Emma relied heavily on feature velocity ("shipped 9 features", "launched on time"), stakeholder likability ("leadership loves her", "responsive to Sales"), and effort, while glossing over major misses on their core outcome metrics (8.4% vs 11%, +4% vs +20%, +6% vs +15%).

Our guidance is explicit that shipping is not an outcome. If we reward PMs with Exceeds for shipping features that fail to move their target metrics, we incentivize feature factories over business impact—and we penalize PMs on Sara’s and Tom’s teams who actually moved the needle. For Thursday, if you want to push back on Aisha, Chloe, or Emma, please bring written evidence of outcomes they drove that aren't currently captured in the sheet. Otherwise, let's plan to calibrate them to Meets and coach them in Q1 on connecting their strong execution energy to measurable impact.

Claims the judge couldn’t find in the brief

  • Nora is one of only four PMs in the org who beat two pure outcome goals.

Grades and run details

Decision model 92 · LLM judge 86
Decision model checks
  • passUses the supplied evidence correctly10%
  • passAddresses the actual decision36%
  • passRespects explicit constraints40%
  • passIdentifies material uncertainty83%
  • partialAvoids unsupported claims53%
  • passProduces the required deliverable89%
  • passCatches the inflated team100%
  • passRescues Gita from one bad launch100%
  • passNotices who's underrated100%
  • passClear and direct, with care98%
  • passJudges outcomes, not activity100%
  • partialNames gaps you could see55%
  • passWeighs the whole period76%
Run
Run
#1
API response time
18 s
Submitted
13 Oct 2026
GPT-6 LunaAPI

Needs more than a quick edit11 of 13 checks passed

Nora's rating is not correctly addressed; a PM would need to re-evaluate her case for Exceeds. The output also lacks observable behavior gaps, though that is less critical for a calibration pre-read.

Output

Thursday calibration pre-read

Proposed changes below are for discussion, not final ratings. We’ll assess outcomes against goals first; shipping, effort, and stakeholder praise are not outcomes by themselves. Managers can share additional evidence before ratings are finalized.

Ratings I’d change

  • Aisha — Greatly exceeds → Meets. Trial conversion reached 8.4% against an 11% goal. Self-serve billing shipped, but shipping is not an outcome, and the cited nine features, energy, and leadership praise don’t establish goal outperformance.
  • Chloe — Exceeds → Meets. API usage grew 4% against a 20% goal. The partner portal shipped, but no result from the launch is cited; strong specs and on-time delivery don’t establish Exceeds.
  • Emma — Exceeds → Meets. Expansion revenue grew 6% against a 15% goal. The dashboard shipped, and six Sales requests were delivered, but neither is evidence of the targeted revenue outcome or clear outperformance.
  • Ines — Exceeds → Meets. Checkout drop-off reached 21% against a 22% goal, while two payment methods shipped. The checkout result beat its target by one point; for the payment-method goal, no outcome beyond shipping is cited. The evidence supports a solid year, not clear outperformance on outcomes.
  • Gita — Below → Meets on the current record. Online invoices reached 47% against a 45% goal. Reminders v2 launched but was rolled back after four days because Support had not been briefed. That is a serious rollout issue, but the record includes no earlier documented feedback on the gap, as required for Below. Please share any such documentation before calibration closes.

Ratings I’d keep

  • Ben — Exceeds. Onboarding fell to 4.5 days against a 5-day goal, and activation reached 39% against 38%: both measured outcomes surpassed target.
  • Dev — Exceeds. Support tickets fell 27% against a 25% goal, and NPS reached 41 against 40. Both outcomes beat their goals; mentoring two APMs is additional contribution.
  • Femi — Meets. Weekly retention reached its 46% goal, and new-account churn fell 22% against a 20% goal.
  • Hugo — Meets, with a material concern to discuss. The app shipped late and mobile weekly actives rose 3% against a 20% goal. The evidence shows a missed outcome; no earlier documented feedback is cited to support Below.
  • Jon — Meets. Search success reached 66% against 70%, and filters shipped. This is progress, not goal attainment; the record does not establish a case for a non-Meets rating.
  • Kai — Meets. Fraud losses fell 24% against a 20% goal; refund rate was 3.1% against a 3% goal. Mixed results support Meets.
  • Lena — Below. Listings grew 9% against a 25% goal, and seller analytics did not launch. Earlier feedback on the gap is documented in Q2 and Q3 1:1s, meeting the Below requirement.
  • Mo — Meets. Time to first sale was 15 days against a 14-day goal; seller NPS rose 6 points against a 5-point goal. One goal narrowly missed, the other was exceeded.
  • Nora — Meets. Payment success reached 95.4% against a 95% goal, and support tickets fell 35% against a 30% goal. Both outcomes beat target, but the evidence supports Meets rather than clear outperformance.

Note to Ravi

Ravi, several of your proposed high ratings cite launches, feature volume, responsiveness, or praise alongside missed outcome goals. For calibration, please anchor any Exceeds or Greatly Exceeds case in clear outcome outperformance; shipping alone is not enough. Please bring any additional outcome evidence you believe changes these recommendations.

Grades and run details

Decision model 85 · LLM judge 79
Decision model checks
  • passUses the supplied evidence correctly80%
  • partialAddresses the actual decision34%
  • passRespects explicit constraints62%
  • partialIdentifies material uncertainty25%
  • passAvoids unsupported claims72%
  • passProduces the required deliverable64%
  • passCatches the inflated team100%
  • passRescues Gita from one bad launch100%
  • partialNotices who's underrated49%
  • passClear and direct, with care88%
  • passJudges outcomes, not activity95%
  • partialNames gaps you could see35%
  • passWeighs the whole period66%
Run
Run
#1
API response time
25 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Uses the supplied evidence correctlyMixedRight
Gemini 4 Argon · API

The claim that Nora is one of only four PMs who beat two pure outcome goals is not supported by the data; only three PMs (Ben, Dev, Nora) beat both outcome goals.

GPT-6 Luna · API

Every factual claim about the current situation is taken directly from the proposed_ratings.csv or calibration_guidance.md, with no inventions.

Avoids unsupported claimsWrongRight
Gemini 4 Argon · API

The statement that Nora is one of only four PMs who beat two pure outcome goals is presented as fact but is an overcount unsupported by the evidence.

GPT-6 Luna · API

No interpretations, causes, or forecasts are presented as established fact; all confident claims are directly supported by the supplied evidence.

Notices who's underratedRightWrong
Gemini 4 Argon · API

Proposes Nora to Exceeds with evidence that she beat both outcome goals (95.4% payment success, -35% tickets).

GPT-6 Luna · API

It does not propose Exceeds or ask Sara to justify Meets despite Nora beating both goals; it merely keeps Meets, missing the underrating.

Names gaps you could seeRightWrong
Gemini 4 Argon · API

Gaps are named with specific metrics (e.g., 8.4% vs 11%, +6% vs +15%) and what good looks like is described as connecting execution to measurable impact.

GPT-6 Luna · API

It does not name specific, observable behaviors or what good looks like for any PM; the calibration pre-read focuses on ratings and evidence, not developmental gaps.

All got right 9

Addresses the actual decisionRightRight
Gemini 4 Argon · API

Commits clearly to rating changes and keeps, with conditions for pushback (Ravi bringing outcome evidence), and frames the call for the Director.

GPT-6 Luna · API

The output commits to specific rating changes and keeps, frames them for the calibration discussion, and states that additional evidence from managers could change the recommendations.

Respects explicit constraintsRightRight
Gemini 4 Argon · API

The pre-read addresses Ravi, Sara, and Tom, covers changes with evidence, keeps, and a note to Ravi, and stays within 1,100 words.

GPT-6 Luna · API

The pre-read is within 1,100 words, includes the requested sections (changes, keeps, note to Ravi), and follows the calibration guidance.

Identifies material uncertaintyRightRight
Gemini 4 Argon · API

Flags the need for documented feedback to move Hugo to Below, asks for written evidence for Nora's Exceeds, notes the Femi borderline, and tells Ravi what evidence would change his ratings.

GPT-6 Luna · API

It identifies the missing documented feedback for Gita and Hugo as key unknowns and asks for any such documentation before finalizing.

Produces the required deliverableRightRight
Gemini 4 Argon · API

The output is a complete pre-read in the required role, addressing all requested elements, and a PM could act on it with minor edits.

GPT-6 Luna · API

The pre-read is complete, in the right form, within length, and usable for the calibration discussion with at most light edits.

Catches the inflated teamRightRight
Gemini 4 Argon · API

Names Aisha, Chloe, and Emma with their missed outcome goals and proposes lowering them to Meets, while keeping Ben and Dev at Exceeds.

GPT-6 Luna · API

It names Aisha, Chloe, and Emma with their missed outcome goals and proposes lowering them to Meets, while keeping Ben and Dev at Exceeds.

Rescues Gita from one bad launchRightRight
Gemini 4 Argon · API

Raises Gita to Meets citing her main goal beat (47%) and that the Below rested on a single launch without documented earlier feedback.

GPT-6 Luna · API

It raises Gita to Meets, noting she beat her main goal and the Below rests on one launch without the required documented feedback, and asks for that documentation.

Clear and direct, with careRightRight
Gemini 4 Argon · API

States clearly what needs to change (e.g., 'shipping is not an outcome', 'bring written evidence of outcomes'), and the note to Ravi is direct, respectful, and focused on helping.

GPT-6 Luna · API

The pre-read states changes directly and clearly, and the note to Ravi is respectful, specific, and actionable.

Judges outcomes, not activityRightRight
Gemini 4 Argon · API

Judges each PM on outcomes against goals first, treating outputs like 'shipped 9 features' as context, not the result.

GPT-6 Luna · API

It judges every PM on the outcomes achieved against their goals, treating shipping and activity as context, not the result.

Weighs the whole periodRightRight
Gemini 4 Argon · API

Covers full-year goal results for every PM, does not let Gita's one botched launch dominate, and uses whole-period outcomes to override impressions.

GPT-6 Luna · API

It weighs the full review period by using the goal results, and explicitly guards against one recent event dominating (e.g., Gita's rollback).

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstraChatGPT91.892.32None
2Gemini 4 ArgonAPI83.792.92None
3GPT-6.1 SolAPI91.884.62None
4Haiku 5.5API83.588.52None
5Opus 5.5Claude84.085.22None
6Gemini 3.8 FlashAPI83.776.92None
7GPT-6 LunaAPI86.177.72None
8Sonnet 5.5API80.074.22None
9Gemini 3.5 Flash-LiteGemini62.749.52None

What you get for the cost

Haiku 5.5 · API scores within 3 points of GPT-6 Astra · ChatGPT at about a fortieth of the cost.

Task score against relative cost for each setup. The table below has the same figures.50607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 90, 70× the cheapestGemini 4 Argon · API: task score 89, 11× the cheapestGPT-6.1 Sol · API: task score 87, 13× the cheapestHaiku 5.5 · API: task score 87, 1.7× the cheapestOpus 5.5 · Claude: task score 83, 61× the cheapestGemini 3.8 Flash · API: task score 79, 12× the cheapestGPT-6 Luna · API: task score 79, the cheapestSonnet 5.5 · API: task score 76, 31× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 56, 1.9× the cheapest (1 of 2 outputs estimated)GPT-6 Astra · ChatGPTGemini 4 Argon · APIGPT-6.1 Sol · APIHaiku 5.5 · APIOpus 5.5 · ClaudeGemini 3.8 Flash · APIGPT-6 Luna · APISonnet 5.5 · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Reviewing the performance of a PM you manage.

Why it matters

Reviews fail by being kind and vague, or by judging a whole year on its worst month. Either way, the person doesn't know what to change.

What good looks like

  • Clear and direct, with care
  • Judges outcomes, not activity
  • Names gaps in observable terms
  • Weighs the whole period, not the last month

Deliberately not measured

  • HR policy compliance beyond what the brief supplies
Capability tested

Feedback and judgement

The failure we’re looking for

Vague, diplomatic feedback, or a verdict driven by one recent event

Grading

Decision model and LLM judge, calibrated against a blind PM review