Humanity’s Last PRD
Latest drop: Gemini 3.8 Flash · 1 Oct
Tasks / Leadership

Write a performance review

Can the model write a review that is clear, fair and specific enough to change what the person does next?

Measures the modelTask type v1.1 · 3 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 19 graded outputs by 7 models. 58% were usable with at most a quick edit.

Reliably right

  1. Judges outcomes, not activity100% pass
    The rating is based on outcomes against goals (two exceeded, one failed launch) rather than activity or output volume.
    GPT-6 Astra · ChatGPT · A PIP, or a bad month?
  2. Owns the manager's part100% pass
    Ana plainly acknowledges her Q3 mistake and commits to reviewing goal metrics first in every monthly 1:1.
    Opus 5.5 · Claude · Shipped a lot, moved nothing
  3. Holds the PIP to the policy and the record100% pass
    It explicitly shows that no earlier documented feedback exists on the gaps Ines named, so a PIP would violate policy, and proposes documented feedback first.
    GPT-6 Astra · ChatGPT · A PIP, or a bad month?

Where it slips

  1. Identifies material uncertainty47% pass
    Does not explicitly name the specific unknowns that could change the decision (e.g., whether Lukas improves launch readiness); only says to reassess after the relaunch.
    GPT-6 Luna · API · A PIP, or a bad month?
  2. Notices who's underrated65% pass
    It does not propose Exceeds or ask Sara to justify Meets despite Nora beating both goals; it merely keeps Meets, missing the underrating.
    GPT-6 Luna · API · Fourteen ratings, three managers
  3. Catches the inflated team75% pass
    The output proposes lowering Ben from Exceeds to Meets, failing to keep Ben at Exceeds as required; Ben beat both outcome goals and should remain Exceeds.
    Opus 5.5 · Claude · Fourteen ratings, three managers

How it’s graded

The checks come from what the best product leaders have said about doing this job well on Lenny’s Podcast. Each one names the guest it comes from: follow a name to the idea on the Lenny’s Podcast wiki.

  1. Clear and direct, with care

    Does the review say plainly what needs to change, in words the person couldn't misread, while showing it is written to help them succeed?

    Passes when Every main point is stated directly, with the specific change expected and the support on offer.

    Kim ScottLenny’s Podcast · Performance reviews

  2. Judges outcomes, not activity

    Does the review judge the person on the outcomes they achieved against their goals, rather than on how much they shipped or how busy they were?

    Passes when Rates against the goals' outcomes, with the evidence, and treats output as context, not the result.

    Lenny’s PodcastThe wiki on Performance reviews

  3. Names gaps you could see

    Is each area to improve described as specific, observable behaviour with what good would look like, rather than labels like 'be more strategic'?

    Passes when Every gap names what the person did or didn't do, in a specific situation, and what doing it well would look like.

    Lenny’s PodcastThe wiki on Performance reviews

  4. Weighs the whole period

    Does the review weigh the whole review period, rather than letting one recent event or one strong impression decide the verdict?

    Passes when Covers the whole period in proportion, and explicitly guards against a recent event or one impression dominating.

    Lenny’s PodcastThe wiki on Performance reviews

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 3 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Director of Product at Northgate, and you're facilitating Thursday's calibration for 14 PMs across three managers (Ravi, Sara and Tom). Write the pre-read, in no more than 1,100 words: the ratings you'd change and why, each with the evidence, the ones you'd keep, and a short note to Ravi about his ratings. Use the guidance below.\n\nThe proposed ratings and evidence are attached.

What the model was given2 items: calibration_guidance.md, proposed_ratings.csv (one row per PM, as each manager submitted it)
calibration_guidance.md7 lines · Download
# Calibration guidance

- Ratings judge outcomes against goals first, then how the work was done. Shipping is not an outcome.
- Any rating other than Meets needs written evidence of outcomes.
- A Below rating needs documented earlier feedback on the gap.
- Expected spread across the org: most PMs Meets; Exceeds for clear outperformance on outcomes.
- The calibration facilitator proposes changes; managers can respond before ratings are final.
proposed_ratings.csv (one row per PM, as each manager submitted it)pm,manager,proposed_rating,goal_1,goal_1_result,goal_2,goal_2_result,evidence_cited_by_manager Aisha,Ravi,Greatly exceeds,Raise trial conversion 8% to 11%,8.4%,Ship self-serve billing,Shipped,Shipped 9 features; great energy; leadership loves her Ben,Ravi,Exceeds,Cut onboarding time 10 to 5 days,4.5 days,Raise activation 30% to 38%,39%,Hit both goals Chloe,Ravi,Exceeds,Grow API usage 20%,+4%,Launch partner portal,Shipped,Partner portal launched on time; strong specs Dev,Ravi,Exceeds,Reduce support tickets per account 25%,-27%,NPS from 31 to 40,41,Hit both goals; mentored two APMs Emma,Ravi,Exceeds,Expansion revenue +15%,+6%,Ship usage dashboard,Shipped,Very responsive to Sales; shipped 6 Sales requests Femi,Sara,Meets,Raise weekly retention 41% to 46%,46%,Cut churn of new accounts 20%,-22%,"Hit both goals, steady" Gita,Sara,Below,Raise invoices paid online 35% to 45%,47%,Launch reminders v2 by September,"Launched, rolled back after 4 days (support not briefed)",Reminders launch in September was a mess Hugo,Sara,Meets,Ship mobile app v2,Shipped late,Mobile weekly actives +20%,+3%,Delivered a hard project Ines,Tom,Exceeds,Cut checkout drop-off 30% to 22%,21%,Launch two payment methods,Shipped,"Hit both, specific evidence" Jon,Tom,Meets,Search success 60% to 70%,66%,Ship filters,Shipped,"Good progress, more to do" Kai,Tom,Meets,Reduce refund rate 4% to 3%,3.1%,Fraud losses -20%,-24%,Solid year Lena,Tom,Below,Grow marketplace listings 25%,+9%,Launch seller analytics,Not launched,"Missed both goals; feedback given in Q2 and Q3 1:1s, documented" Mo,Tom,Meets,Cut time to first sale for new sellers 21 to 14 days,15 days,Seller NPS +5,+6,"Nearly hit, good stakeholder work" Nora,Sara,Meets,Raise payment success 92% to 95%,95.4%,Cut payment support tickets 30%,-35%,"Quiet, delivered"
What a strong answer doesThe answer key the graders mark against

Judges each PM on outcomes against their goals, as the guidance says. Ravi rated all five of his PMs Exceeds or above, but only Ben and Dev hit both goals; Aisha (Greatly exceeds) missed her conversion goal (8.4% against 11%), Chloe missed API usage (+4% against +20%) and Emma missed expansion (+6% against +15%), and his evidence is output ('shipped 9 features', 'responsive to Sales'). Proposes Aisha, Chloe and Emma at Meets (or lower for Chloe, whose outcome goal was far off), keeps Ben and Dev at Exceeds. Gita (Sara, Below) beat her main goal (47% against 45%) and the Below rests on one September launch, with no documented feedback, which the guidance requires: Meets, with the launch-readiness gap documented now. Nora (Sara, Meets) beat both goals: a candidate for Exceeds. Lena (Tom, Below) has documented feedback and missed both goals: Below stands. Hugo shipped late and missed his outcome: Meets is generous but defensible, or Below needs documentation first. The note to Ravi is direct and respectful: rate on outcomes, not activity, with what evidence he'd need.

Critical failures (cap the score at 40)
  • Keeps Aisha at Greatly exceeds
Task

Staff-level · v1.0 · synthetic · management, calibration, data files, staff-level

The outputs

5 setups answered this brief. Pick up to three to read side by side.

GPT-6 AstraChatGPT100Usable with a quick editGPT-6.1 SolAPI100Usable with a quick editOpus 5.5Claude88Needs more workGPT-6 LunaAPI82Needs more workGemini 3.5 Flash-LiteGemini38Needs more work
Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit2 of 13 checks passed

The output is incomplete, missing the keep list and Ravi note, and cannot be used without redoing the reasoning and filling major gaps.

Output

To: Ravi, Sara, Tom From: Director of Product Subject: Pre-Read: Thursday PM Calibration (Northgate)

Team,

Ahead of our calibration session this Thursday, I have reviewed the proposed ratings, outcomes, and evidence for all 14 PMs across our org.

Per our calibration guidance, we evaluate PMs first on outcomes against goals, not on activity. Shipping features, being responsive to sales, or having great energy are inputs, not results. Furthermore, any rating other than "Meets" requires rigorous, quantifiable evidence of goal achievement, and "Below" ratings require documented prior feedback.

Below is my proposed adjustment of ratings, the rationale and evidence for changes, and the list of ratings we are keeping as submitted.

---

Part 1: Proposed Rating Changes & Rationale

#### 1. Aisha (Manager: Ravi) * Proposed by Manager: Greatly Exceeds * My Proposed Rating: Below * Evidence & Rationale: * Goal 1: Raise trial conversion from 8% to 11% $\rightarrow$ Result: 8.4% (Missed by a wide margin). * Goal 2: Ship self-serve billing $\rightarrow$ Result: Shipped (Violation of guidance: "Shipping is not an outcome"). * Rationale: Aisha missed her primary core metric goal entirely (gaining only 0.4% against a 3% target) and fell back on "shipping" and "great energy" for her second goal. Per our framework, failing the core quantitative goal while relying on output means this is a Below performance. Ravi, we will need to document the performance gap and prior feedback for Aisha in this cycle.

#### 2. Chloe (Manager: Ravi) * Proposed by Manager: Exceeds * My Proposed Rating: Meets * Evidence & Rationale: * Goal 1: Grow API usage 20% $\rightarrow$ Result: +4% (Severe miss). * Goal 2: Launch partner portal $\rightarrow$ Result: Shipped on time

Grades and run details

Decision model 65 · LLM judge 2 of 14 checks
Decision model checks
  • passUses the supplied evidence correctly70%
  • passAddresses the actual decision28%
  • partialRespects explicit constraints20%
  • failIdentifies material uncertainty88%
  • passAvoids unsupported claims33%
  • partialProduces the required deliverable75%
  • partialCatches the inflated team95%
  • failRescues Gita from one bad launch83%
  • failNotices who's underrated95%
  • passClear and direct, with care55%
  • passJudges outcomes, not activity98%
  • passNames gaps you could see69%
  • passWeighs the whole period21%
Run
Run
#1
Time to output
3 s
Submitted
5 Oct 2026
GPT-6 AstraChatGPT

Usable with a quick edit13 of 13 checks passed

The pre-read is evidence-based, complete, and directly actionable; a PM could use it with minor wording tweaks without rechecking facts or redoing reasoning.

Output

Thursday calibration pre-read

From: Director of Product, Northgate Scope: 14 PMs across Ravi, Sara and Tom

These are my proposed changes, not final decisions. Please respond before Thursday with missing outcome evidence, relevant goal context, or documented earlier feedback that could change the assessment.

Standard I’m applying

We rate outcomes against goals first, then how the work was done. Shipping, responsiveness, visibility and strong execution matter, but do not substitute for impact.

For Exceeds, I’m looking for clear outperformance—not simply delivery or attainment. On this evidence, exceeding both measurable outcome goals supports Exceeds; meeting one and modestly beating another generally supports Meets. A modest beat on one outcome plus a shipped deliverable is not enough by itself.

A Below requires both evidence of underperformance and documented earlier feedback. Where that feedback is absent from the packet, I am not proposing Below. That does not mean the missed goals are acceptable or should go unaddressed.

Ratings I propose changing

PMProposed changeEvidence and rationale
Aisha — RaviGreatly exceeds → MeetsTrial conversion rose from 8% to 8.4%, against 11%. Self-serve billing shipped, but no resulting customer or business impact is supplied. Nine features, energy and leadership enthusiasm do not establish exceptional outcomes. The conversion miss is substantial; however, the packet contains no documented earlier feedback supporting Below.
Chloe — RaviExceeds → MeetsAPI usage grew 4%, against 20%. The partner portal launched on time, and strong specifications support execution quality, not outcome outperformance. There is no written evidence supporting Exceeds or documented earlier feedback supporting Below.
Emma — RaviExceeds → MeetsExpansion revenue grew 6%, against 15%. Shipping the dashboard and six Sales requests, plus responsiveness to Sales, do not offset the revenue shortfall without evidence of impact. No documented earlier feedback is supplied to support Below.
Gita — SaraBelow → MeetsOnline invoice payment reached 47%, above the 45% target from a 35% baseline. Reminders v2 was rolled back after four days because Support was not briefed—a meaningful execution failure. We should account for that failure, but not let it erase the payment outcome. The packet also lacks the documented earlier feedback required for Below. Please bring any evidence of rollback impact and prior feedback.
Ines — TomExceeds → MeetsCheckout drop-off fell from 30% to 21%, against 22%: a strong result, modestly above target. Two payment methods shipped, but shipment is not a second outcome. On the supplied evidence, this is solid attainment rather than clear overall outperformance. Incremental adoption, conversion or revenue evidence could change the assessment.
Nora — SaraMeets → ExceedsPayment success reached 95.4%, against 95% from 92%; payment-support tickets fell 35%, against 30%. She exceeded both measurable outcome goals. That warrants the same recognition as comparable performance elsewhere. “Quiet” is not a reason to discount demonstrated impact.

Ratings I propose keeping

PMRatingEvidence and rationale
Ben — RaviExceedsOnboarding fell from 10 to 4.5 days, beating 5 days; activation reached 39%, beating 38% from 30%. Both outcome goals were exceeded.
Dev — RaviExceedsSupport tickets per account fell 27%, against 25%; NPS rose from 31 to 41, against 40. Both outcome goals were exceeded. Mentoring two APMs reinforces the assessment of how he worked; it is not the basis for the rating.
Femi — SaraMeetsWeekly retention reached 46%, exactly on target from 41%; new-account churn fell 22%, against 20%. Strong delivery against goals, with a modest beat on one—not enough evidence of clear overall outperformance.
Hugo — SaraMeets, pending feedback reviewMobile weekly actives grew 3%, against 20%, and app v2 shipped late. “A hard project” does not establish successful outcomes. This is a material performance concern, but the packet lacks documented earlier feedback needed for Below.
Jon — TomMeetsSearch success improved from 60% to 66%, short of 70%. Filters shipped, but no incremental impact is given. There is neither evidence for Exceeds nor documented earlier feedback supporting Below. The remaining outcome gap needs explicit follow-up.
Kai — TomMeetsRefund rate reached 3.1%, narrowly missing 3% from 4%; fraud losses fell 24%, beating 20%. Mixed attainment supports Meets.
Lena — TomBelowListings grew 9%, against 25%, and seller analytics did not launch. The packet documents feedback in Q2 and Q3 1:1s. Both the performance evidence and earlier-feedback requirement are present.
Mo — TomMeetsTime to first sale fell from 21 to 15 days, narrowly missing 14 days; seller NPS increased 6 points, against 5. Mixed attainment and good stakeholder work support Meets.

Note to Ravi

Ravi, all five of your submissions are above Meets, but the evidence supports that distinction for Ben and Dev—not Aisha, Chloe or Emma. In those three cases, the write-ups emphasize shipping, responsiveness or visibility despite substantial outcome misses.

Please revise those rationales around goals, actual results and attributable impact, and bring any missing evidence or earlier feedback. This is not a quota correction: strong results should earn high ratings regardless of manager. We need the same evidentiary bar across teams.

Proposed distribution: 10 Meets, 3 Exceeds, 1 Below; no Greatly exceeds. This is a consequence of the individual assessments, not a forced curve.

Grades and run details

Decision model 100 · LLM judge 14 of 14 checks
Decision model checks
  • passUses the supplied evidence correctly83%
  • passAddresses the actual decision87%
  • passRespects explicit constraints46%
  • passIdentifies material uncertainty88%
  • passAvoids unsupported claims64%
  • passProduces the required deliverable85%
  • passCatches the inflated team100%
  • passRescues Gita from one bad launch100%
  • passNotices who's underrated100%
  • passClear and direct, with care98%
  • passJudges outcomes, not activity100%
  • passNames gaps you could see73%
  • passWeighs the whole period79%
Run
Run
#1
Time to output
32 s
Submitted
5 Oct 2026
GPT-6 LunaAPI

Needs more than a quick edit11 of 13 checks passed

Nora's rating is not correctly addressed; a PM would need to re-evaluate her case for Exceeds. The output also lacks observable behavior gaps, though that is less critical for a calibration pre-read.

Output

Thursday calibration pre-read

Proposed changes below are for discussion, not final ratings. We’ll assess outcomes against goals first; shipping, effort, and stakeholder praise are not outcomes by themselves. Managers can share additional evidence before ratings are finalized.

Ratings I’d change

  • Aisha — Greatly exceeds → Meets. Trial conversion reached 8.4% against an 11% goal. Self-serve billing shipped, but shipping is not an outcome, and the cited nine features, energy, and leadership praise don’t establish goal outperformance.
  • Chloe — Exceeds → Meets. API usage grew 4% against a 20% goal. The partner portal shipped, but no result from the launch is cited; strong specs and on-time delivery don’t establish Exceeds.
  • Emma — Exceeds → Meets. Expansion revenue grew 6% against a 15% goal. The dashboard shipped, and six Sales requests were delivered, but neither is evidence of the targeted revenue outcome or clear outperformance.
  • Ines — Exceeds → Meets. Checkout drop-off reached 21% against a 22% goal, while two payment methods shipped. The checkout result beat its target by one point; for the payment-method goal, no outcome beyond shipping is cited. The evidence supports a solid year, not clear outperformance on outcomes.
  • Gita — Below → Meets on the current record. Online invoices reached 47% against a 45% goal. Reminders v2 launched but was rolled back after four days because Support had not been briefed. That is a serious rollout issue, but the record includes no earlier documented feedback on the gap, as required for Below. Please share any such documentation before calibration closes.

Ratings I’d keep

  • Ben — Exceeds. Onboarding fell to 4.5 days against a 5-day goal, and activation reached 39% against 38%: both measured outcomes surpassed target.
  • Dev — Exceeds. Support tickets fell 27% against a 25% goal, and NPS reached 41 against 40. Both outcomes beat their goals; mentoring two APMs is additional contribution.
  • Femi — Meets. Weekly retention reached its 46% goal, and new-account churn fell 22% against a 20% goal.
  • Hugo — Meets, with a material concern to discuss. The app shipped late and mobile weekly actives rose 3% against a 20% goal. The evidence shows a missed outcome; no earlier documented feedback is cited to support Below.
  • Jon — Meets. Search success reached 66% against 70%, and filters shipped. This is progress, not goal attainment; the record does not establish a case for a non-Meets rating.
  • Kai — Meets. Fraud losses fell 24% against a 20% goal; refund rate was 3.1% against a 3% goal. Mixed results support Meets.
  • Lena — Below. Listings grew 9% against a 25% goal, and seller analytics did not launch. Earlier feedback on the gap is documented in Q2 and Q3 1:1s, meeting the Below requirement.
  • Mo — Meets. Time to first sale was 15 days against a 14-day goal; seller NPS rose 6 points against a 5-point goal. One goal narrowly missed, the other was exceeded.
  • Nora — Meets. Payment success reached 95.4% against a 95% goal, and support tickets fell 35% against a 30% goal. Both outcomes beat target, but the evidence supports Meets rather than clear outperformance.

Note to Ravi

Ravi, several of your proposed high ratings cite launches, feature volume, responsiveness, or praise alongside missed outcome goals. For calibration, please anchor any Exceeds or Greatly Exceeds case in clear outcome outperformance; shipping alone is not enough. Please bring any additional outcome evidence you believe changes these recommendations.

Grades and run details

Decision model 92 · LLM judge 11 of 14 checks
Decision model checks
  • passUses the supplied evidence correctly87%
  • passAddresses the actual decision52%
  • passRespects explicit constraints39%
  • partialIdentifies material uncertainty31%
  • passAvoids unsupported claims69%
  • passProduces the required deliverable62%
  • passCatches the inflated team100%
  • passRescues Gita from one bad launch100%
  • partialNotices who's underrated41%
  • passClear and direct, with care85%
  • passJudges outcomes, not activity97%
  • passNames gaps you could see78%
  • passWeighs the whole period64%
Run
Run
#1
API response time
25 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 11

Addresses the actual decisionMixedRightRight
Gemini 3.5 Flash-Lite · Gemini

The output is cut off and does not commit to a complete answer; it lacks the ratings to keep and the note to Ravi.

GPT-6 Astra · ChatGPT

The output commits to specific rating changes and keeps, with evidence, and states that missing outcome evidence or documented feedback could change the assessment.

GPT-6 Luna · API

The output commits to specific rating changes and keeps, frames them for the calibration discussion, and states that additional evidence from managers could change the recommendations.

Respects explicit constraintsWrongRightRight
Gemini 3.5 Flash-Lite · Gemini

The brief requires a pre-read with changed ratings, kept ratings, and a note to Ravi; the output is incomplete and omits the keep list and note.

GPT-6 Astra · ChatGPT

The output is a pre-read under 1,100 words, includes the ratings to change and keep with evidence, and a note to Ravi, respecting all explicit constraints.

GPT-6 Luna · API

The pre-read is within 1,100 words, includes the requested sections (changes, keeps, note to Ravi), and follows the calibration guidance.

Identifies material uncertaintyWrongRightRight
Gemini 3.5 Flash-Lite · Gemini

No unknowns that could change the decision are named, nor how they would be resolved.

GPT-6 Astra · ChatGPT

It identifies missing documented feedback for several PMs, asks for rollback impact and prior feedback for Gita, and notes that incremental evidence could change Ines's rating.

GPT-6 Luna · API

It identifies the missing documented feedback for Gita and Hugo as key unknowns and asks for any such documentation before finalizing.

Avoids unsupported claimsMixedRightRight
Gemini 3.5 Flash-Lite · Gemini

It proposes a Below rating for Aisha without the documented earlier feedback the guidance requires, presenting the rating as justified when the evidence does not support it.

GPT-6 Astra · ChatGPT

Interpretations and standards are clearly presented as such, and no cause, forecast, or motivation is stated as fact without support.

GPT-6 Luna · API

No interpretations, causes, or forecasts are presented as established fact; all confident claims are directly supported by the supplied evidence.

Produces the required deliverableWrongRightRight
Gemini 3.5 Flash-Lite · Gemini

The pre-read is incomplete, missing the keep list and Ravi note, and cannot be acted on without major gaps.

GPT-6 Astra · ChatGPT

The pre-read is complete, in the requested format, within the word limit, and directly usable by the managers with at most light edits.

GPT-6 Luna · API

The pre-read is complete, in the right form, within length, and usable for the calibration discussion with at most light edits.

Catches the inflated teamWrongRightRight
Gemini 3.5 Flash-Lite · Gemini

It names Aisha and Chloe but does not mention Emma, so it fails to catch all three inflated ratings.

GPT-6 Astra · ChatGPT

It identifies that Aisha, Chloe, and Emma missed outcome goals and proposes lowering them to Meets, while keeping Ben and Dev at Exceeds, and calls out Ravi's emphasis on output.

GPT-6 Luna · API

It names Aisha, Chloe, and Emma with their missed outcome goals and proposes lowering them to Meets, while keeping Ben and Dev at Exceeds.

Rescues Gita from one bad launchWrongRightRight
Gemini 3.5 Flash-Lite · Gemini

Gita is not mentioned at all, so the output does not rescue her from the Below rating.

GPT-6 Astra · ChatGPT

It raises Gita to Meets because she beat her main goal, notes the Below rested on one launch without required documented feedback, and asks for the launch gap to be documented.

GPT-6 Luna · API

It raises Gita to Meets, noting she beat her main goal and the Below rests on one launch without the required documented feedback, and asks for that documentation.

Notices who's underratedWrongRightWrong
Gemini 3.5 Flash-Lite · Gemini

Nora is not mentioned, so the output does not notice she is underrated.

GPT-6 Astra · ChatGPT

It notices Nora beat both outcome goals and proposes raising her from Meets to Exceeds with the specific evidence.

GPT-6 Luna · API

It does not propose Exceeds or ask Sara to justify Meets despite Nora beating both goals; it merely keeps Meets, missing the underrating.

Clear and direct, with careMixedRightRight
Gemini 3.5 Flash-Lite · Gemini

The output is incomplete and lacks the full note to Ravi; it does not say plainly everything that needs to change.

GPT-6 Astra · ChatGPT

The note to Ravi is direct, specific about what needs to change, and framed to help him succeed; the whole pre-read is clear and actionable.

GPT-6 Luna · API

The pre-read states changes directly and clearly, and the note to Ravi is respectful, specific, and actionable.

Names gaps you could seeMixedRightMixed
Gemini 3.5 Flash-Lite · Gemini

It does not describe gaps as specific, observable behaviors with what good would look like; it only states missed targets.

GPT-6 Astra · ChatGPT

Gaps are described as specific, observable results (e.g., rollback due to unbriefed support, late shipment, missed targets) rather than vague labels.

GPT-6 Luna · API

It does not name specific, observable behaviors or what good looks like for any PM; the calibration pre-read focuses on ratings and evidence, not developmental gaps.

Weighs the whole periodMixedRightRight
Gemini 3.5 Flash-Lite · Gemini

It does not weigh the whole review period or guard against recency; it only mentions goal results without context of the full period.

GPT-6 Astra · ChatGPT

The assessment weighs the full year's goal results and explicitly guards against letting the recent rollback erase Gita's overall outcome.

GPT-6 Luna · API

It weighs the full review period by using the goal results, and explicitly guards against one recent event dominating (e.g., Gita's rollback).

All got right 2

Uses the supplied evidence correctlyRightRightRight
Gemini 3.5 Flash-Lite · Gemini

All factual claims about current performance and results are taken correctly from the supplied context.

GPT-6 Astra · ChatGPT

All factual claims about current performance, numbers, and events are taken directly from the supplied proposed_ratings.csv or calibration_guidance.md, with no invented facts.

GPT-6 Luna · API

Every factual claim about the current situation is taken directly from the proposed_ratings.csv or calibration_guidance.md, with no inventions.

Judges outcomes, not activityRightRightRight
Gemini 3.5 Flash-Lite · Gemini

For the parts it covers, it judges on outcomes against goals, not on activity or shipping.

GPT-6 Astra · ChatGPT

Every rating change and keep is justified by outcomes against goals, with shipping and activity treated as context, not the result.

GPT-6 Luna · API

It judges every PM on the outcomes achieved against their goals, treating shipping and activity as context, not the result.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Provisional The checklist is still being calibrated against our PM.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstraChatGPT95.894.93None
2GPT-6.1 SolAPI98.689.73None
3Opus 5.5Claude93.290.13None
4Sonnet 5.5API95.884.62None
5GPT-6 LunaAPI94.782.63None
6Gemini 3.8 FlashAPI83.357.72None
7Gemini 3.5 Flash-LiteGemini73.258.63None

The PM job

Reviewing the performance of a PM you manage.

Why it matters

Reviews fail by being kind and vague, or by judging a whole year on its worst month. Either way, the person doesn't know what to change.

What good looks like

  • Clear and direct, with care
  • Judges outcomes, not activity
  • Names gaps in observable terms
  • Weighs the whole period, not the last month

Deliberately not measured

  • HR policy compliance beyond what the brief supplies
Capability tested

Feedback and judgement

The failure we’re looking for

Vague, diplomatic feedback, or a verdict driven by one recent event

Grading

Decision model and LLM judge, calibrated against a blind PM review