Humanity’s Last PRD
Latest drop: Haiku 5.5 · 8 Oct
Tasks / Leadership

Write a performance review

Can the model write a review that is clear, fair and specific enough to change what the person does next?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 50% were usable with at most a quick edit.

Reliably right

  1. Judges outcomes, not activity100% pass
    Every rating change and keep is justified by outcomes against goals, with shipping and activity treated as context, not the result.
    GPT-6 Astra · ChatGPT · Fourteen ratings, three managers
  2. Owns the manager's part100% pass
    Ana plainly acknowledges her Q3 mistake and commits to reviewing goal metrics first in every monthly 1:1.
    Opus 5.5 · Claude · Shipped a lot, moved nothing
  3. Clear and direct, with care97% pass
    The note to Ravi is direct, specific about what needs to change, and framed to help him succeed; the whole pre-read is clear and actionable.
    GPT-6 Astra · ChatGPT · Fourteen ratings, three managers

Where it slips

  1. Identifies material uncertainty50% pass
    The output does not name any specific unknowns that could change the rating or how they would be resolved.
    Opus 5.5 · Claude · Shipped a lot, moved nothing
  2. Notices who's underrated69% pass
    It does not propose Exceeds or ask Sara to justify Meets despite Nora beating both goals; it merely keeps Meets, missing the underrating.
    GPT-6 Luna · API · Fourteen ratings, three managers
  3. Catches the inflated team78% pass
    The output proposes lowering Ben from Exceeds to Meets, failing to keep Ben at Exceeds as required; Ben beat both outcome goals and should remain Exceeds.
    Opus 5.5 · Claude · Fourteen ratings, three managers

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Clear and direct, with care

    Does the review say plainly what needs to change, in words the person couldn't misread, while showing it is written to help them succeed?

    Passes when Every main point is stated directly, with the specific change expected and the support on offer.

    Kim ScottLenny’s Podcast

  2. Judges outcomes, not activity

    Does the review judge the person on the outcomes they achieved against their goals, rather than on how much they shipped or how busy they were?

    Passes when Rates against the goals' outcomes, with the evidence, and treats output as context, not the result.

    Product and growth leadersOn Lenny’s Podcast

  3. Names gaps you could see

    Is each area to improve described as specific, observable behaviour with what good would look like, rather than labels like 'be more strategic'?

    Passes when Every gap names what the person did or didn't do, in a specific situation, and what doing it well would look like.

    Product and growth leadersOn Lenny’s Podcast

  4. Weighs the whole period

    Does the review weigh the whole review period, rather than letting one recent event or one strong impression decide the verdict?

    Passes when Covers the whole period in proportion, and explicitly guards against a recent event or one impression dominating.

    Product and growth leadersOn Lenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 3 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Director of Product at Northgate, and you're facilitating Thursday's calibration for 14 PMs across three managers (Ravi, Sara and Tom). Write the pre-read, in no more than 1,100 words: the ratings you'd change and why, each with the evidence, the ones you'd keep, and a short note to Ravi about his ratings. Use the guidance below.\n\nThe proposed ratings and evidence are attached.

What the model was given2 items: calibration_guidance.md, proposed_ratings.csv (one row per PM, as each manager submitted it)
calibration_guidance.md7 lines · Download
# Calibration guidance

- Ratings judge outcomes against goals first, then how the work was done. Shipping is not an outcome.
- Any rating other than Meets needs written evidence of outcomes.
- A Below rating needs documented earlier feedback on the gap.
- Expected spread across the org: most PMs Meets; Exceeds for clear outperformance on outcomes.
- The calibration facilitator proposes changes; managers can respond before ratings are final.
proposed_ratings.csv (one row per PM, as each manager submitted it)pm,manager,proposed_rating,goal_1,goal_1_result,goal_2,goal_2_result,evidence_cited_by_manager Aisha,Ravi,Greatly exceeds,Raise trial conversion 8% to 11%,8.4%,Ship self-serve billing,Shipped,Shipped 9 features; great energy; leadership loves her Ben,Ravi,Exceeds,Cut onboarding time 10 to 5 days,4.5 days,Raise activation 30% to 38%,39%,Hit both goals Chloe,Ravi,Exceeds,Grow API usage 20%,+4%,Launch partner portal,Shipped,Partner portal launched on time; strong specs Dev,Ravi,Exceeds,Reduce support tickets per account 25%,-27%,NPS from 31 to 40,41,Hit both goals; mentored two APMs Emma,Ravi,Exceeds,Expansion revenue +15%,+6%,Ship usage dashboard,Shipped,Very responsive to Sales; shipped 6 Sales requests Femi,Sara,Meets,Raise weekly retention 41% to 46%,46%,Cut churn of new accounts 20%,-22%,"Hit both goals, steady" Gita,Sara,Below,Raise invoices paid online 35% to 45%,47%,Launch reminders v2 by September,"Launched, rolled back after 4 days (support not briefed)",Reminders launch in September was a mess Hugo,Sara,Meets,Ship mobile app v2,Shipped late,Mobile weekly actives +20%,+3%,Delivered a hard project Ines,Tom,Exceeds,Cut checkout drop-off 30% to 22%,21%,Launch two payment methods,Shipped,"Hit both, specific evidence" Jon,Tom,Meets,Search success 60% to 70%,66%,Ship filters,Shipped,"Good progress, more to do" Kai,Tom,Meets,Reduce refund rate 4% to 3%,3.1%,Fraud losses -20%,-24%,Solid year Lena,Tom,Below,Grow marketplace listings 25%,+9%,Launch seller analytics,Not launched,"Missed both goals; feedback given in Q2 and Q3 1:1s, documented" Mo,Tom,Meets,Cut time to first sale for new sellers 21 to 14 days,15 days,Seller NPS +5,+6,"Nearly hit, good stakeholder work" Nora,Sara,Meets,Raise payment success 92% to 95%,95.4%,Cut payment support tickets 30%,-35%,"Quiet, delivered"
What a strong answer doesThe answer key the graders mark against

Judges each PM on outcomes against their goals, as the guidance says. Ravi rated all five of his PMs Exceeds or above, but only Ben and Dev hit both goals; Aisha (Greatly exceeds) missed her conversion goal (8.4% against 11%), Chloe missed API usage (+4% against +20%) and Emma missed expansion (+6% against +15%), and his evidence is output ('shipped 9 features', 'responsive to Sales'). Proposes Aisha, Chloe and Emma at Meets (or lower for Chloe, whose outcome goal was far off), keeps Ben and Dev at Exceeds. Gita (Sara, Below) beat her main goal (47% against 45%) and the Below rests on one September launch, with no documented feedback, which the guidance requires: Meets, with the launch-readiness gap documented now. Nora (Sara, Meets) beat both goals: a candidate for Exceeds. Lena (Tom, Below) has documented feedback and missed both goals: Below stands. Hugo shipped late and missed his outcome: Meets is generous but defensible, or Below needs documentation first. The note to Ravi is direct and respectful: rate on outcomes, not activity, with what evidence he'd need.

Critical failures (cap the score at 40)
  • Keeps Aisha at Greatly exceeds
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

GPT-6 AstraChatGPT100Usable with a quick editGPT-6.1 SolAPI100Usable with a quick editHaiku 5.5API100Usable with a quick editGemini 3.8 FlashAPI98Usable with a quick editOpus 5.5Claude88Needs more workGPT-6 LunaAPI82Needs more workSonnet 5.5API77Needs more workGemini 3.5 Flash-LiteGemini38Needs more work
GPT-6.1 SolAPI

Usable with a quick edit13 of 13 checks passed

The pre-read is well-structured, uses evidence correctly, and makes clear proposals. A PM could use it with at most light editing for tone or emphasis, without re-checking facts or redoing reasoning.

Output

Thursday calibration pre-read

From: Director of Product, Northgate Scope: 14 PMs across Ravi, Sara and Tom Status: Facilitator proposals—not final ratings. Managers should respond with additional evidence before we finalize.

Calibration approach

We will assess outcomes against goals first, then how the work was done. Shipping, responsiveness, strong specifications and leadership visibility are relevant context, but do not substitute for customer or business outcomes.

Exceeds requires written evidence of clear outperformance, not simply completing planned work. Below requires both evidence of underperformance and documented earlier feedback about the gap. Missing feedback does not make weak performance satisfactory; it does mean we cannot support Below on the submitted record.

Ratings I propose changing

PMProposed changeEvidence and rationale
Aisha — RaviGreatly exceeds → MeetsTrial conversion reached 8.4% against an 11% goal, from an 8% baseline. Self-serve billing shipped, but no resulting business or customer impact is supplied. Nine features, energy and leadership enthusiasm do not establish exceptional outcomes. The conversion miss is substantial; without documented earlier feedback, the record also does not support Below.
Chloe — RaviExceeds → MeetsAPI usage grew 4% against a 20% goal. The partner portal launched on time, but its adoption or impact is not documented. Strong specifications and timely delivery do not offset the outcome shortfall sufficiently to support Exceeds.
Emma — RaviExceeds → MeetsExpansion revenue grew 6% against a 15% goal. The usage dashboard and six Sales requests shipped, but there is no evidence that they delivered additional outcomes. Responsiveness to Sales is positive working behavior, not evidence of exceeding the revenue goal.
Gita — SaraBelow → MeetsOnline invoice payment reached 47% against a 45% goal, from 35%—a strong outcome. Reminders v2 was rolled back after four days because Support was not briefed, a material execution and coordination failure. However, the submission contains no documented earlier feedback on that gap, which is required for Below. Meets recognizes the mixed record; it does not excuse the launch failure.
Ines — TomExceeds → MeetsCheckout drop-off fell to 21% against a 22% goal, from 30%, and two payment methods shipped. This is successful delivery with some outcome overperformance. On the current record, however, modest overdelivery on checkout plus a launch does not establish clear overall outperformance. Adoption, incremental conversion or revenue from the payment methods could strengthen the case for Exceeds.
Nora — SaraMeets → ExceedsPayment success reached 95.4% against a 95% goal, from 92%; payment support tickets fell 35% against a 30% reduction goal. Both measured outcomes surpassed their targets. This is comparable to the outcome-based cases supporting Ben and Dev. “Quiet” should neither diminish nor enhance the rating.

Ratings I propose keeping

PMKeepEvidence and rationale
Ben — RaviExceedsOnboarding fell to 4.5 days against a five-day goal, from ten; activation reached 39% against 38%, from 30%. Both customer outcomes exceeded their targets.
Dev — RaviExceedsSupport tickets per account fell 27% against a 25% reduction goal; NPS reached 41 against 40, from 31. Both outcomes surpassed targets. Mentoring two APMs strengthens the assessment of how the work was done, but is not the primary justification.
Femi — SaraMeetsWeekly retention reached 46%, exactly the goal, from 41%; new-account churn fell 22% against a 20% reduction goal. Strong goal attainment, with some overdelivery, supports Meets rather than a clear-outperformance rating.
Hugo — SaraMeetsMobile v2 shipped late, and mobile weekly actives grew only 3% against a 20% goal. “Delivered a hard project” does not establish outcome success. This is a significant concern, but no documented earlier feedback is supplied to support Below.
Jon — TomMeetsSearch success improved from 60% to 66% against a 70% goal; filters shipped. Meaningful progress, but the principal outcome remained short of target.
Kai — TomMeetsRefund rate reached 3.1% against a 3% goal, from 4%; fraud losses fell 24% against a 20% reduction goal. One near miss and one exceeded outcome support Meets overall.
Lena — TomBelowListings grew 9% against a 25% goal, and seller analytics did not launch. The submission explicitly cites documented Q2 and Q3 1:1 feedback. Both the performance gap and earlier-feedback requirement are evidenced.
Mo — TomMeetsTime to first sale fell to 15 days against a 14-day goal, from 21; seller NPS rose six points against five. Near attainment on one outcome and overdelivery on the other support Meets. Stakeholder work is positive supporting context.

Note to Ravi

Your submission rates all five PMs above Meets, but three cases rely heavily on shipping, responsiveness or visibility despite substantial misses on their stated outcome goals. Please revisit Aisha, Chloe and Emma using the same outcome standard that supports Ben and Dev. If you have additional attributable impact or documented earlier feedback, bring it before Thursday. We are seeking consistent judgments, not enforcing a quota.

Proposed spread: 10 Meets, three Exceeds and one Below.

Grades and run details

Decision model 100 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly68%
  • passAddresses the actual decision84%
  • passRespects explicit constraints49%
  • passIdentifies material uncertainty74%
  • passAvoids unsupported claims67%
  • passProduces the required deliverable90%
  • passCatches the inflated team100%
  • passRescues Gita from one bad launch100%
  • passNotices who's underrated100%
  • passClear and direct, with care98%
  • passJudges outcomes, not activity99%
  • passNames gaps you could see81%
  • passWeighs the whole period79%
Run
Run
#1
API response time
37 s
Submitted
6 Oct 2026
GPT-6 AstraChatGPT

Usable with a quick edit13 of 13 checks passed

The pre-read is evidence-based, complete, and directly actionable; a PM could use it with minor wording tweaks without rechecking facts or redoing reasoning.

Output

Thursday calibration pre-read

From: Director of Product, Northgate Scope: 14 PMs across Ravi, Sara and Tom

These are my proposed changes, not final decisions. Please respond before Thursday with missing outcome evidence, relevant goal context, or documented earlier feedback that could change the assessment.

Standard I’m applying

We rate outcomes against goals first, then how the work was done. Shipping, responsiveness, visibility and strong execution matter, but do not substitute for impact.

For Exceeds, I’m looking for clear outperformance—not simply delivery or attainment. On this evidence, exceeding both measurable outcome goals supports Exceeds; meeting one and modestly beating another generally supports Meets. A modest beat on one outcome plus a shipped deliverable is not enough by itself.

A Below requires both evidence of underperformance and documented earlier feedback. Where that feedback is absent from the packet, I am not proposing Below. That does not mean the missed goals are acceptable or should go unaddressed.

Ratings I propose changing

PMProposed changeEvidence and rationale
Aisha — RaviGreatly exceeds → MeetsTrial conversion rose from 8% to 8.4%, against 11%. Self-serve billing shipped, but no resulting customer or business impact is supplied. Nine features, energy and leadership enthusiasm do not establish exceptional outcomes. The conversion miss is substantial; however, the packet contains no documented earlier feedback supporting Below.
Chloe — RaviExceeds → MeetsAPI usage grew 4%, against 20%. The partner portal launched on time, and strong specifications support execution quality, not outcome outperformance. There is no written evidence supporting Exceeds or documented earlier feedback supporting Below.
Emma — RaviExceeds → MeetsExpansion revenue grew 6%, against 15%. Shipping the dashboard and six Sales requests, plus responsiveness to Sales, do not offset the revenue shortfall without evidence of impact. No documented earlier feedback is supplied to support Below.
Gita — SaraBelow → MeetsOnline invoice payment reached 47%, above the 45% target from a 35% baseline. Reminders v2 was rolled back after four days because Support was not briefed—a meaningful execution failure. We should account for that failure, but not let it erase the payment outcome. The packet also lacks the documented earlier feedback required for Below. Please bring any evidence of rollback impact and prior feedback.
Ines — TomExceeds → MeetsCheckout drop-off fell from 30% to 21%, against 22%: a strong result, modestly above target. Two payment methods shipped, but shipment is not a second outcome. On the supplied evidence, this is solid attainment rather than clear overall outperformance. Incremental adoption, conversion or revenue evidence could change the assessment.
Nora — SaraMeets → ExceedsPayment success reached 95.4%, against 95% from 92%; payment-support tickets fell 35%, against 30%. She exceeded both measurable outcome goals. That warrants the same recognition as comparable performance elsewhere. “Quiet” is not a reason to discount demonstrated impact.

Ratings I propose keeping

PMRatingEvidence and rationale
Ben — RaviExceedsOnboarding fell from 10 to 4.5 days, beating 5 days; activation reached 39%, beating 38% from 30%. Both outcome goals were exceeded.
Dev — RaviExceedsSupport tickets per account fell 27%, against 25%; NPS rose from 31 to 41, against 40. Both outcome goals were exceeded. Mentoring two APMs reinforces the assessment of how he worked; it is not the basis for the rating.
Femi — SaraMeetsWeekly retention reached 46%, exactly on target from 41%; new-account churn fell 22%, against 20%. Strong delivery against goals, with a modest beat on one—not enough evidence of clear overall outperformance.
Hugo — SaraMeets, pending feedback reviewMobile weekly actives grew 3%, against 20%, and app v2 shipped late. “A hard project” does not establish successful outcomes. This is a material performance concern, but the packet lacks documented earlier feedback needed for Below.
Jon — TomMeetsSearch success improved from 60% to 66%, short of 70%. Filters shipped, but no incremental impact is given. There is neither evidence for Exceeds nor documented earlier feedback supporting Below. The remaining outcome gap needs explicit follow-up.
Kai — TomMeetsRefund rate reached 3.1%, narrowly missing 3% from 4%; fraud losses fell 24%, beating 20%. Mixed attainment supports Meets.
Lena — TomBelowListings grew 9%, against 25%, and seller analytics did not launch. The packet documents feedback in Q2 and Q3 1:1s. Both the performance evidence and earlier-feedback requirement are present.
Mo — TomMeetsTime to first sale fell from 21 to 15 days, narrowly missing 14 days; seller NPS increased 6 points, against 5. Mixed attainment and good stakeholder work support Meets.

Note to Ravi

Ravi, all five of your submissions are above Meets, but the evidence supports that distinction for Ben and Dev—not Aisha, Chloe or Emma. In those three cases, the write-ups emphasize shipping, responsiveness or visibility despite substantial outcome misses.

Please revise those rationales around goals, actual results and attributable impact, and bring any missing evidence or earlier feedback. This is not a quota correction: strong results should earn high ratings regardless of manager. We need the same evidentiary bar across teams.

Proposed distribution: 10 Meets, 3 Exceeds, 1 Below; no Greatly exceeds. This is a consequence of the individual assessments, not a forced curve.

Grades and run details

Decision model 100 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly83%
  • passAddresses the actual decision87%
  • passRespects explicit constraints46%
  • passIdentifies material uncertainty88%
  • passAvoids unsupported claims64%
  • passProduces the required deliverable85%
  • passCatches the inflated team100%
  • passRescues Gita from one bad launch100%
  • passNotices who's underrated100%
  • passClear and direct, with care98%
  • passJudges outcomes, not activity100%
  • passNames gaps you could see73%
  • passWeighs the whole period79%
Run
Run
#1
Time to output
32 s
Submitted
5 Oct 2026
Sonnet 5.5API

Needs more than a quick edit10 of 13 checks passed

The output incorrectly lowers Ben and Dev to Meets and fails to propose Exceeds for Nora, requiring re-evaluation of those ratings before use.

Output

Pre-read: Product calibration, Thursday

Scope: 14 PMs, 3 managers. I propose changing 7 ratings and keeping 7. These are proposals, and managers can respond before anything is final.

Headline: Ravi's team is rated 5 of 5 at Exceeds or above, and the cited evidence is mostly activity: features shipped, responsiveness, energy. Under our guidance, shipping is not an outcome. After my proposed changes, the spread is 13 Meets, 1 Below and 0 Exceeds. Managers who think someone clearly outperformed should bring the outcome evidence.

Ratings I'd change

Exceeds rated on shipping, with the main outcome missed

Aisha (Ravi): Greatly exceeds → Meets - Trial conversion went from 8% to 8.4% against a target of 11%. - Self-serve billing shipped, but no outcome is cited. - The evidence offered is "9 features, great energy, leadership loves her." None of that is an outcome. - "Greatly exceeds" isn't a category in our guidance. - The outcome data would point lower than Meets. I'm not proposing Below because no earlier feedback is documented, which the guidance requires. That gap belongs to the manager.

Chloe (Ravi): Exceeds → Meets - API usage grew 4% against a 20% target. - The partner portal launched on time, but no outcome is cited. - Strong specs and an on-time launch are good craft, not outcomes. - The same Below caveat applies as for Aisha.

Emma (Ravi): Exceeds → Meets - Expansion revenue rose 6% against a 15% target. - The usage dashboard shipped, but no outcome is cited. - "Shipped 6 Sales requests" and responsiveness describe throughput, not results. - The same Below caveat applies.

Hit both goals, but not clear outperformance

The guidance reserves Exceeds for clear outperformance. These three cleared their targets by small margins, and the same result is rated Meets elsewhere (Femi, Nora, Kai).

Ben (Ravi): Exceeds → Meets - Onboarding went from 10 to 4.5 days against a 5-day target. - Activation reached 39% against a 38% target. - This is the closest call of the seven. If Ravi can show that the goals were unusually hard or the business impact was outsized, I'm open to changing my view.

Dev (Ravi): Exceeds → Meets - Tickets per account fell 27% against a 25% target. - NPS reached 41 against a 40 target. - Mentoring two APMs is real and belongs in the written feedback. It doesn't offset margins this thin.

Ines (Tom): Exceeds → Meets - Checkout drop-off fell to 21% against a 22% target. - Two payment methods shipped, but no outcome is cited for them. - Tom's note says "specific evidence" without stating it. I'd need to see it.

Below without documented feedback

Gita (Sara): Below → Meets - Goal 1 was beaten: 47% of invoices paid online against a 45% target. - Goal 2, the reminders v2 launch, was rolled back after 4 days because support wasn't briefed. That is a real execution miss and should be coached. - The guidance requires documented earlier feedback for a Below, and none is cited. One goal beaten and one launch failure on a cross-functional step is not a Below without that record. - The written feedback should name the support-briefing failure plainly.

Ratings I'd keep

  • Femi (Sara), Meets: retention reached 46% (target 46%) and new-account churn fell 22% (target 20%). Both goals were hit.
  • Nora (Sara), Meets: payment success reached 95.4% (target 95%) and payment tickets fell 35% (target 30%). This is a strong result, but not clear outperformance on both goals.
  • Kai (Tom), Meets: refund rate reached 3.1% (target 3%) and fraud losses fell 24% (target 20%). One goal was a near miss and one was beaten.
  • Mo (Tom), Meets: time to first sale reached 15 days (target 14) and seller NPS rose 6 (target 5). One goal was a near miss and one was beaten.
  • Lena (Tom), Below: listings grew 9% against a 25% target, and seller analytics did not launch. Feedback was given and documented in Q2 and Q3 1:1s, so this meets the bar for Below. Tom should confirm the written evidence is in the file before ratings are final.

Two Meets I'm flagging, not changing

  • Hugo (Sara): Mobile app v2 shipped late, and weekly actives rose 3% against a 20% target. "Delivered a hard project" describes shipping, not outcomes. I'm leaving Meets because the same logic that stops me moving Aisha, Chloe and Emma to Below applies here, and consistency matters. Sara, please come ready to discuss the outcome miss and any earlier feedback.
  • Jon (Tom): Search success reached 66% against a 70% target (from 60%), and filters shipped. "Good progress, more to do" is a reasonable Meets but a thin write-up. Please add the actual outcome numbers.

Note to Ravi

Ravi,

Your ratings are the ones I'm changing most, so I want to explain my thinking before Thursday.

I'm not questioning your team's effort or your read on their potential. Ben and Dev hit both goals, and that counts. The issue is that all five ratings sit at Exceeds or above, and for three of them (Aisha, Chloe, Emma) the cited evidence is shipping and responsiveness while the primary outcome was missed by a wide margin. Aisha's 8.4% against an 11% target is the clearest case. "Leadership loves her" will not hold up in calibration, and it's unfair to Sara's and Tom's reports, who are rated Meets on similar or stronger outcomes.

I'm proposing Meets for all five. For Ben and Dev, I'll reconsider if you can show the goals were unusually hard or the impact was outsized. For Aisha, Chloe and Emma, I'd need outcome evidence I haven't seen, such as revenue or usage attributed to the work that shipped.

There is a second issue. If three of your reports missed key outcomes and there was no documented feedback, they didn't hear about it in time to change course. That is a management gap, and it's why I can't propose Below even where the numbers might support it. Please start documenting feedback now, and bring whatever exists on Thursday.

Please respond before Thursday with any evidence I've missed.

What I need before Thursday

  • Ravi: outcome evidence for Aisha, Chloe and Emma, plus any feedback records.
  • Sara: the outcome explanation for Hugo and any feedback history for Gita.
  • Tom: Ines's specific evidence, Lena's feedback documents and Jon's outcome numbers.

Grades and run details

Decision model 88 · LLM judge 71
Decision model checks
  • passUses the supplied evidence correctly57%
  • passAddresses the actual decision81%
  • partialRespects explicit constraints22%
  • passIdentifies material uncertainty95%
  • partialAvoids unsupported claims42%
  • passProduces the required deliverable79%
  • passCatches the inflated team89%
  • passRescues Gita from one bad launch100%
  • partialNotices who's underrated54%
  • passClear and direct, with care96%
  • passJudges outcomes, not activity95%
  • passNames gaps you could see77%
  • passWeighs the whole period72%
Run
Run
#1
API response time
47 s
Submitted
6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 3

Catches the inflated teamRightRightMixed
GPT-6.1 Sol · API

The pre-read names Aisha, Chloe, and Emma with their missed outcome goals and proposes lowering them to Meets, while keeping Ben and Dev at Exceeds.

GPT-6 Astra · ChatGPT

It identifies that Aisha, Chloe, and Emma missed outcome goals and proposes lowering them to Meets, while keeping Ben and Dev at Exceeds, and calls out Ravi's emphasis on output.

Sonnet 5.5 · API

The output lowers Ben and Dev to Meets instead of keeping them at Exceeds, failing to keep the two who hit both goals.

Notices who's underratedRightRightWrong
GPT-6.1 Sol · API

It raises Nora to Exceeds, citing that she beat both outcome goals, and notes that 'quiet' should not affect the rating.

GPT-6 Astra · ChatGPT

It notices Nora beat both outcome goals and proposes raising her from Meets to Exceeds with the specific evidence.

Sonnet 5.5 · API

It keeps Nora at Meets without proposing Exceeds or asking Sara to justify Meets, despite her beating both goals.

Names gaps you could seeRightRightMixed
GPT-6.1 Sol · API

Gaps are described as specific, observable results (e.g., 'trial conversion reached 8.4% against an 11% goal', 'reminders v2 rolled back because support was not briefed').

GPT-6 Astra · ChatGPT

Gaps are described as specific, observable results (e.g., rollback due to unbriefed support, late shipment, missed targets) rather than vague labels.

Sonnet 5.5 · API

The gaps for Aisha, Chloe, and Emma are described as missing outcome evidence, not as specific observable behaviors with what good would look like.

All got right 10

Uses the supplied evidence correctlyRightRightRight
GPT-6.1 Sol · API

Every factual claim about current performance, results, and documented feedback is taken directly from the supplied proposed_ratings.csv or follows by arithmetic.

GPT-6 Astra · ChatGPT

All factual claims about current performance, numbers, and events are taken directly from the supplied proposed_ratings.csv or calibration_guidance.md, with no invented facts.

Sonnet 5.5 · API

All claims about the current situation are taken directly from the supplied context or follow by arithmetic.

Addresses the actual decisionRightRightRight
GPT-6.1 Sol · API

The pre-read commits to specific rating changes and keeps, and states that additional evidence from managers could change the ratings before finalization.

GPT-6 Astra · ChatGPT

The output commits to specific rating changes and keeps, with evidence, and states that missing outcome evidence or documented feedback could change the assessment.

Sonnet 5.5 · API

The output commits to changing 7 ratings and keeping 7, with clear proposals and conditions that would change the call.

Respects explicit constraintsRightRightRight
GPT-6.1 Sol · API

The output is a pre-read under 1,100 words, includes the ratings to change and keep with evidence, and a short note to Ravi, as requested.

GPT-6 Astra · ChatGPT

The output is a pre-read under 1,100 words, includes the ratings to change and keep with evidence, and a note to Ravi, respecting all explicit constraints.

Sonnet 5.5 · API

The pre-read is within 1,100 words, addresses the managers, and includes the required sections.

Identifies material uncertaintyRightRightRight
GPT-6.1 Sol · API

It names unknowns (e.g., missing adoption/revenue data for Ines, possible additional evidence from Ravi) and says how they would be resolved or change the call.

GPT-6 Astra · ChatGPT

It identifies missing documented feedback for several PMs, asks for rollback impact and prior feedback for Gita, and notes that incremental evidence could change Ines's rating.

Sonnet 5.5 · API

It names specific unknowns (outcome evidence for Aisha/Chloe/Emma, goal difficulty for Ben, feedback history for Gita) and how they would be resolved.

Avoids unsupported claimsRightRightRight
GPT-6.1 Sol · API

Interpretations and judgments are clearly presented as the facilitator's proposals, not as established facts, and no unsupported causal claims are made.

GPT-6 Astra · ChatGPT

Interpretations and standards are clearly presented as such, and no cause, forecast, or motivation is stated as fact without support.

Sonnet 5.5 · API

Interpretations are clearly the author's judgment, and no unsupported facts are presented as established.

Produces the required deliverableRightRightRight
GPT-6.1 Sol · API

The pre-read is complete, in the right form for the Director of Product, within the word limit, and usable as-is for the calibration meeting.

GPT-6 Astra · ChatGPT

The pre-read is complete, in the requested format, within the word limit, and directly usable by the managers with at most light edits.

Sonnet 5.5 · API

The pre-read is complete, in the right form, and usable as a calibration document with light edits.

Rescues Gita from one bad launchRightRightRight
GPT-6.1 Sol · API

It raises Gita to Meets, noting she beat her main goal and the Below rating rests on one launch without the required documented earlier feedback, and implies the gap should be documented now.

GPT-6 Astra · ChatGPT

It raises Gita to Meets because she beat her main goal, notes the Below rested on one launch without required documented feedback, and asks for the launch gap to be documented.

Sonnet 5.5 · API

It raises Gita to Meets, noting she beat her main goal and the Below lacks the required documented earlier feedback.

Clear and direct, with careRightRightRight
GPT-6.1 Sol · API

The note to Ravi is direct and respectful, stating exactly what needs to change and what evidence would help, and the whole pre-read is clear and actionable.

GPT-6 Astra · ChatGPT

The note to Ravi is direct, specific about what needs to change, and framed to help him succeed; the whole pre-read is clear and actionable.

Sonnet 5.5 · API

The note to Ravi is direct, respectful, and states exactly what needs to change and what support is available.

Judges outcomes, not activityRightRightRight
GPT-6.1 Sol · API

Every rating is justified by outcomes against goals, with shipping and activity treated as context, not the result.

GPT-6 Astra · ChatGPT

Every rating change and keep is justified by outcomes against goals, with shipping and activity treated as context, not the result.

Sonnet 5.5 · API

The ratings are judged against goal outcomes, with activity treated as context, not the result.

Weighs the whole periodRightRightRight
GPT-6.1 Sol · API

The pre-read weighs the full year's goals and results for each PM, and for Gita it explicitly balances the strong main goal with the launch failure, avoiding recency bias.

GPT-6 Astra · ChatGPT

The assessment weighs the full year's goal results and explicitly guards against letting the recent rollback erase Gita's overall outcome.

Sonnet 5.5 · API

The whole year's goals are weighed, and no single recent event dominates the ratings.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstraChatGPT93.892.32None
2Haiku 5.5API93.888.52None
3GPT-6.1 SolAPI97.984.62None
4Opus 5.5Claude91.885.22None
5GPT-6 LunaAPI94.177.72None
6Gemini 3.8 FlashAPI89.776.92None
7Sonnet 5.5API90.174.22None
8Gemini 3.5 Flash-LiteGemini74.449.52None

What you get for the cost

Haiku 5.5 · API matches GPT-6 Astra · ChatGPT at about a fortieth of the cost.

Task score against relative cost for each setup. The table below has the same figures.50607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 91, 70× the cheapestHaiku 5.5 · API: task score 91, 1.7× the cheapestGPT-6.1 Sol · API: task score 89, 13× the cheapestOpus 5.5 · Claude: task score 87, 61× the cheapestGPT-6 Luna · API: task score 82, the cheapestGemini 3.8 Flash · API: task score 82, 12× the cheapestSonnet 5.5 · API: task score 81, 31× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 61, 1.9× the cheapest (1 of 2 outputs estimated)GPT-6 Astra · ChatGPTHaiku 5.5 · APIGPT-6.1 Sol · APIOpus 5.5 · ClaudeGPT-6 Luna · APIGemini 3.8 Flash · APISonnet 5.5 · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Reviewing the performance of a PM you manage.

Why it matters

Reviews fail by being kind and vague, or by judging a whole year on its worst month. Either way, the person doesn't know what to change.

What good looks like

  • Clear and direct, with care
  • Judges outcomes, not activity
  • Names gaps in observable terms
  • Weighs the whole period, not the last month

Deliberately not measured

  • HR policy compliance beyond what the brief supplies
Capability tested

Feedback and judgement

The failure we’re looking for

Vague, diplomatic feedback, or a verdict driven by one recent event

Grading

Decision model and LLM judge, calibrated against a blind PM review