Tasks / Leadership

Write a performance review

Can the model write a review that is clear, fair and specific enough to change what the person does next?

Measures the modelTask type v1.1 · 3 tasksLast changed 5 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 14 graded outputs by 7 models. 64% were usable with at most a quick edit.

Reliably right

  1. Produces the required deliverable100% pass
    The review summary, PIP recommendation, and note to Ines are complete, in the right form, and usable with light edits.
    GPT-6 Astra · ChatGPT · A PIP, or a bad month?
  2. Judges outcomes, not activity100% pass
    The rating is based on outcomes against goals (two exceeded, one failed launch) rather than activity or output volume.
    GPT-6 Astra · ChatGPT · A PIP, or a bad month?
  3. Weighs the whole period100% pass
    The review weighs the full year's results, explicitly balancing the two above-target outcomes against the one bounded failure, and guards against letting the Dispatch incident dominate.
    GPT-6 Astra · ChatGPT · A PIP, or a bad month?

Where it slips

  1. Identifies material uncertainty38% pass
    Does not explicitly name the specific unknowns that could change the decision (e.g., whether Lukas improves launch readiness); only says to reassess after the relaunch.
    GPT-6 Luna · API · A PIP, or a bad month?
  2. Avoids unsupported claims71% pass
    Presents 'not through negligence on his part' as an established fact without labelling it as a hypothesis, when the evidence only shows he was on leave and his deputy ran checks.
    Sonnet 5.5 · API · A PIP, or a bad month?
  3. Sets next goals as outcomes, with support75% pass
    Goals are partly process-oriented (implement a mandatory process, create a framework) and lack specific support from Ana beyond a general offer.
    Gemini 3.5 Flash-Lite · Gemini · Shipped a lot, moved nothing

How it’s graded

The checks come from what the best product leaders have said about doing this job well on Lenny’s Podcast. Each one names the guest it comes from: follow a name to the idea on the Lenny’s Podcast wiki.

  1. Clear and direct, with care

    Does the review say plainly what needs to change, in words the person couldn't misread, while showing it is written to help them succeed?

    Passes when Every main point is stated directly, with the specific change expected and the support on offer.

  2. Judges outcomes, not activity

    Does the review judge the person on the outcomes they achieved against their goals, rather than on how much they shipped or how busy they were?

    Passes when Rates against the goals' outcomes, with the evidence, and treats output as context, not the result.

  3. Names gaps you could see

    Is each area to improve described as specific, observable behaviour with what good would look like, rather than labels like 'be more strategic'?

    Passes when Every gap names what the person did or didn't do, in a specific situation, and what doing it well would look like.

  4. Weighs the whole period

    Does the review weigh the whole review period, rather than letting one recent event or one strong impression decide the verdict?

    Passes when Covers the whole period in proportion, and explicitly guards against a recent event or one impression dominating.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're Director of Product at Railyard, and calibration is in five days. Ines Okoro, our VP Product, has drafted the calibration pre-read and suggested putting Lukas Brenner, one of your Senior PMs, on a performance improvement plan. Write, in no more than 1,100 words: (1) Lukas's review summary and the rating you propose, with the evidence; (2) your recommendation on the PIP, and if you recommend one, its key terms; (3) a short note to Ines on the evidence. The record is below.

What the model was given7 items: Lukas's goals and results this year, What happened with Dispatch v2, Ines's draft for the pre-read, Peer feedback, Feedback on record, HR policy on PIPs, A precedent
Lukas's goals and results this year1. Cut carrier onboarding time from 21 days to 10: done, now 9 days. 2. Launch Dispatch v2 by October: launched in October, then rolled back after six days. 3. Grow carriers using live tracking from 40% to 55%: reached 57%.
What happened with Dispatch v2It launched without support being briefed, and 340 support tickets came in the first week. It was rolled back after six days. Lukas was on approved leave the week before launch; his deputy ran the final launch checks. It's due to relaunch next quarter.
Ines's draft for the pre-read'Lukas: Below expectations. Not strategic enough, lacks product sense, and Dispatch was a disaster. Suggest a PIP.'
Peer feedbackThree of four peers are positive, citing the onboarding work and his collaboration. The Head of Operations is negative about how Dispatch was communicated.
Feedback on recordNo written feedback to Lukas this year about strategy or product sense. One conversation, in November, about launch readiness, after Dispatch.
HR policy on PIPsA PIP needs earlier documented feedback on the specific gap and a chance to improve. A PIP lasts 60 days and needs measurable success criteria.
A precedentLast year Priya, another Senior PM, had a launch rolled back after a similar support failure. She was rated Meets expectations, with no PIP.
What a strong answer doesThe answer key the graders mark against

Recommends against a PIP now: policy requires earlier documented feedback on the specific gap, and there is none on strategy or product sense, and only one conversation about launch readiness, after the fact. Rates Lukas Meets expectations (arguably higher on two of three goals met or beaten), weighing the whole year rather than October: onboarding time 21 to 9 days and live tracking 40% to 57% against one failed launch, which happened while he was on approved leave. Turns 'not strategic enough' and 'lacks product sense' into the observable gap the evidence does support: launch readiness, including briefing support and having a cover plan for his absence. Proposes documented feedback on that gap now, with clear expectations for the relaunch and a check-in after it. Notes the precedent: Priya was rated Meets with no PIP for a similar failure, so a PIP would be inconsistent. Pushes back on Ines firmly and respectfully, with the evidence.

Critical failures (cap the score at 40)
  • Recommends starting a PIP now, without the documented earlier feedback the policy requires
Task

Staff-level · v1.0 · synthetic · B2B SaaS, logistics, management, staff-level

The outputs

7 setups answered this brief. Pick up to three to read side by side.

Usable with a quick edit12 of 12 checks passed

The output is evidence-based, directly addresses the decision, and requires at most minor wording tweaks; no re-checking of facts or reasoning is needed.

Output

1. Lukas’s review summary and proposed rating

Proposed rating: Meets expectations, with a significant launch-readiness gap to address.

Lukas delivered strong results against two measurable goals. Carrier onboarding time fell from 21 days to 9 days, exceeding the 10-day target. Live-tracking adoption increased from 40% to 57%, exceeding the 55% target. Three of four peers gave positive feedback, particularly on his onboarding work and collaboration.

Dispatch v2 was a substantial delivery failure. Although it launched in October, it was rolled back after six days; the launch objective should therefore not be treated as successfully completed. Support had not been briefed, and 340 support tickets arrived in the first week. The Head of Operations’ feedback reinforces the specific concern about launch communication and operational readiness. Relaunch is planned for next quarter, so the intended outcome remains outstanding.

As the responsible Senior PM, Lukas should be assessed on how he established launch-readiness requirements, cross-functional ownership, and delegation. However, the record does not establish which safeguards he put in place or who approved the final launch. He was on approved leave the preceding week, and his deputy ran the final checks. Approved leave is not a performance failure; equally, delegation does not automatically remove accountability for the preparation and handoff. We should establish those facts before assigning Lukas sole responsibility.

The evidence supports a specific execution and communication gap, not the broader conclusions that Lukas is “not strategic enough” or “lacks product sense.” No feedback on those broader concerns was documented this year, and the supplied record contains no concrete examples substantiating them.

On balance, two above-target outcomes, positive collaboration feedback, and one serious but bounded delivery failure support Meets expectations. This is also consistent with Priya’s rating last year after a similar rollback and support failure. That precedent does not dictate Lukas’s rating, but a harsher outcome requires a material, evidenced distinction—not stronger language about the incident.

2. PIP recommendation

Do not initiate a PIP on the current record.

HR requires earlier documented feedback on the specific gap and an opportunity to improve. The record contains one November conversation about launch readiness, after Dispatch, but does not establish that it was documented or that Lukas subsequently had a meaningful opportunity to demonstrate improvement. There is no documented feedback on strategy or product sense. A PIP on those grounds would therefore lack the required foundation.

Instead, put a documented development plan within normal performance management in place now. This is not a disciplinary PIP or a predetermined path to one. Agree with Lukas on:

  • Readiness ownership: Before the relaunch go/no-go review, document owners, decision rights, coverage during absences, escalation routes, and rollback criteria.
  • Support preparedness: Before launch approval, secure Support and Operations sign-off on training, customer communications, FAQs, and the escalation runbook.
  • Launch monitoring: Before relaunch, agree measurable adoption, reliability, and support-volume thresholds with Engineering, Support, and Operations; review results against them after launch.
  • Manager support: Provide a readiness template, help resolve cross-functional ownership gaps, and hold regular check-ins through relaunch.

Document the expectations, support provided, and subsequent results. If a specific performance gap persists after clear feedback and a genuine opportunity to improve, reassess with HR. Any later PIP must meet the policy’s 60-day duration and measurable-success-criteria requirements.

3. Note to Ines

Ines—

I recommend changing Lukas’s proposed rating to Meets expectations and removing the PIP recommendation on the current evidence.

We should retain a direct account of Dispatch’s failure: Support was not briefed, 340 tickets arrived in the first week, and the release was rolled back after six days. But “not strategic enough,” “lacks product sense,” and “disaster” should be replaced with specific, attributable observations. The current draft also omits his two above-target results.

Before the pre-read is finalized, I’ll review the launch checklist, handoff and approval record, clarify the November feedback and subsequent improvement opportunity, and give Lukas a chance to respond to the factual account. I’ll also check whether there are material differences from Priya’s case.

If that review produces additional evidence, we should consider it explicitly. Otherwise, the defensible outcome is Meets expectations, clear launch-readiness feedback, and documented follow-through—not a PIP unsupported by our policy.

Grades and run details

Decision model 100 · LLM judge 13 of 13 checks
Decision model checks
  • passUses the supplied evidence correctly71%
  • passAddresses the actual decision96%
  • passRespects explicit constraints65%
  • passIdentifies material uncertainty99%
  • passAvoids unsupported claims60%
  • passProduces the required deliverable97%
  • passHolds the PIP to the policy and the record100%
  • passTreats Lukas like Priya97%
  • passClear and direct, with care92%
  • passJudges outcomes, not activity99%
  • passNames gaps you could see94%
  • passWeighs the whole period100%
Run
Run
#1
Time to output
29 s
Submitted
5 Oct 2026

Usable with a quick edit12 of 12 checks passed

The output is clear, evidence-based, and complete; a PM could use it with minor wording edits without re-checking facts or redoing reasoning.

Output

1. Lukas’s review summary and proposed rating

Proposed rating: Meets expectations, with a significant launch-readiness gap to address.

Lukas delivered strong results on two of his three annual goals:

  • Carrier onboarding: Reduced onboarding time from 21 days to 9 days, exceeding the 10-day target.
  • Live tracking adoption: Increased adoption from 40% to 57%, exceeding the 55% target.
  • Dispatch v2: Launched in October, meeting the timing target, but was rolled back after six days. This was not a successful, sustained delivery and should not receive full credit simply because it launched on schedule.

Dispatch exposed a serious readiness failure: support was not briefed, and 340 support tickets arrived in the first week. That outcome warrants explicit negative feedback and corrective action. However, the record does not establish how responsibility was divided among Lukas, his deputy, and other launch stakeholders. Lukas was on approved leave the week before launch, and his deputy ran the final checks. His leave is not itself a performance issue; his planning and handoff before leave remain legitimate areas to assess.

Peer feedback is positive from three of four peers, particularly on onboarding and collaboration. The Head of Operations’ criticism of Dispatch communication is relevant and consistent with the launch failure. It should be addressed directly rather than generalized into an unsupported conclusion about Lukas’s overall capabilities.

Review summary: Lukas exceeded the onboarding and live-tracking goals and received broadly positive feedback on collaboration. Dispatch v2 was a material setback: the release was not adequately prepared for support and was rolled back. As a Senior PM, Lukas needs to demonstrate more reliable launch planning, explicit delegation, and cross-functional readiness. We should establish his specific responsibility for the failure and assess improvement through the relaunch.

“Meets expectations” is the most defensible overall rating on the current record. It recognizes substantial results without minimizing Dispatch. It is also consistent with Priya’s rating last year after a similar rollback and support failure. That precedent is not automatically determinative, but a lower rating here would require a documented, material distinction that we do not currently have.

2. PIP recommendation

Do not initiate a PIP now.

HR policy requires earlier documented feedback on the specific gap and an opportunity to improve. There is no written feedback this year about strategy or product sense. The only recorded conversation concerns launch readiness in November, after Dispatch. On the available record, that does not establish both the required prior documentation and a subsequent opportunity to improve.

The proposed PIP therefore lacks the required foundation. “Not strategic enough” and “lacks product sense” are also too broad to serve as actionable performance gaps without concrete examples and defined expectations.

Instead, put a documented, non-PIP development plan in place now:

  • Clarify accountability: Review the Dispatch launch plan, handoff, readiness checks, and go/no-go decision with Lukas, his deputy, Operations, and Support. Identify individual and process failures.
  • Set observable expectations: For the relaunch, require named owners for every readiness item, documented support briefing and training, escalation and rollback procedures, and explicit cross-functional approval before release.
  • Check progress regularly: Hold biweekly check-ins with Lukas and record feedback, completed actions, and any remaining gaps.
  • Assess the relaunch: Evaluate readiness execution and post-launch support outcomes against thresholds agreed with Support before launch.

This should be genuine coaching, not a PIP under another name. If Lukas subsequently fails to meet clearly documented expectations after a reasonable opportunity to improve, reassess with HR. Any later PIP must run for 60 days, with measurable success criteria and the support needed to meet them.

3. Note to Ines

Ines — I agree that Dispatch’s readiness failure deserves clear accountability. I recommend Meets expectations and no PIP at this stage.

The pre-read should distinguish established facts from judgments. Lukas exceeded two goals; Dispatch met its launch date but failed as a sustained release. We have evidence of a launch-readiness and communication problem, but not enough to substantiate “not strategic enough” or “lacks product sense.” We also need to clarify Lukas’s handoff responsibilities versus the final checks his deputy owned.

Before calibration, I’ll verify the launch ownership, the November feedback record, and whether there are material differences from Priya’s case. Unless that produces materially different evidence, we should revise the pre-read accordingly. HR’s PIP prerequisites are not demonstrated in the current record; the appropriate next step is specific documented feedback and a fair opportunity to improve.

Grades and run details

Decision model 100 · LLM judge 13 of 13 checks
Decision model checks
  • passUses the supplied evidence correctly71%
  • passAddresses the actual decision96%
  • passRespects explicit constraints63%
  • passIdentifies material uncertainty96%
  • passAvoids unsupported claims73%
  • passProduces the required deliverable95%
  • passHolds the PIP to the policy and the record100%
  • passTreats Lukas like Priya99%
  • passClear and direct, with care81%
  • passJudges outcomes, not activity98%
  • passNames gaps you could see45%
  • passWeighs the whole period99%
Run
Run
#1
API response time
25 s
Submitted
5 Oct 2026

Usable with a quick edit11 of 12 checks passed

The output is well-structured and evidence-based; a quick edit to soften the unsupported negligence claim would make it fully ready without re-checking evidence or redoing reasoning.

Output

Calibration Materials: Lukas Brenner, Senior PM

1. Review summary and proposed rating

Proposed rating: Meets expectations

Goal results

GoalTargetResult
Carrier onboarding time21 → 10 days9 days (exceeded)
Live tracking adoption40% → 55%57% (exceeded)
Dispatch v2 launchLaunch by OctoberLaunched in October, rolled back after 6 days (not met in substance)

Strengths. Lukas delivered two of three goals ahead of target. Cutting onboarding time by more than half is a substantial result. Taking tracking adoption past 55% also took cross-functional work. Three of four peers were positive, citing the onboarding work and his collaboration.

The Dispatch v2 miss. This was a real failure with real cost: support was not briefed, 340 tickets arrived in week one, and the launch was rolled back. Several points shape how I weigh it:

  • Lukas was on approved leave the week before launch, and his deputy ran the final checks. A launch-readiness gap occurred on his project, but not through negligence on his part.
  • Lukas still owns the outcome. A launch of this size should have had a handoff or readiness process that survived his absence, and either the plan to cover his leave or the go/no-go decision was weak. I haven't confirmed who owned the support briefing or whether a checklist existed. I'll establish that before the relaunch, not assume it.
  • The Head of Operations' criticism of how Dispatch was communicated is legitimate and should be recorded.
  • Relaunch is planned for next quarter, so the outcome isn't final.

Why Meets, not Below. Two strong results and one significant miss, with mitigating context, add up to Meets. The record doesn't support "Below expectations." It also doesn't support Exceeds, because the rollback was costly and the Operations concern is a real signal.

Consistency check. Last year Priya, also a Senior PM, had a launch rolled back after a similar support failure and was rated Meets with no PIP. Rating Lukas lower, with a stronger consequence, for a comparable miss, and with better overall results than the miss alone suggests, would be hard to defend in calibration.

Development areas (to be documented in writing): 1. Launch readiness: cross-functional briefing (especially support and operations), and coverage plans when the owner is out. 2. Stakeholder communication with Operations on launch timing and impact.

2. PIP recommendation

I recommend against a PIP.

  • Policy. A PIP requires earlier documented feedback on the specific gap and a chance to improve. Lukas has had no written feedback this year on strategy or product sense. The only feedback was one conversation in November about launch readiness, after Dispatch. He hasn't had a fair chance to address the gap.
  • Evidence. The "not strategic enough / lacks product sense" claim is not supported by the record. The onboarding and tracking results are strategic outcomes, and no example of poor product judgment is documented apart from the launch process.
  • Consistency. It would be treated differently from Priya's comparable case.
  • Proportionality. The one real gap, launch readiness, is specific and fixable, and the relaunch gives a natural test.

What I'd do instead: a documented development plan, delivered in writing within two weeks of calibration:

  • Written feedback naming the launch-readiness gap and the Operations communication concern, with the Dispatch facts.
  • Relaunch readiness criteria, agreed with Lukas before the Dispatch v2 relaunch:
  • Support briefed and signed off at least 10 business days before launch.
  • Written go/no-go checklist with named owners, including Support and Operations.
  • A documented coverage and handoff plan for any planned absence in the two weeks around launch.
  • Staged rollout or defined rollback triggers.
  • Success measures: all criteria met; first-week support ticket volume within a threshold agreed with Support (to be set from baseline volumes); Operations lead's feedback on launch communication collected after relaunch.
  • Check-ins: every two weeks with me, and a formal review at 60 days.

Escalation path. If the development plan is documented, the criteria are clear, and Lukas then repeats the same failures in areas he controls, a PIP with measurable criteria and a 60-day term would be justified and policy-compliant. We are not there today.

3. Note to Ines

Ines,

Thanks for drafting the pre-read. I've reviewed the record and I'd like to propose a different rating and approach before we go into calibration. I think the current draft would be hard to defend.

Where I agree. Dispatch v2 was a serious miss. Support wasn't briefed, 340 tickets came in, and we rolled back. The Operations Head's concern is valid. Lukas needs clear feedback on launch readiness.

Where the evidence doesn't support the draft: - "Not strategic, lacks product sense." Lukas cut onboarding from 21 to 9 days against a target of 10, and took tracking adoption to 57% against 55%. Three of four peers were positive. I can't point to documented evidence of weak product judgment, and I'd rather not put that claim in a pre-read without it. - "A disaster." The rollback was costly, but Lukas was on approved leave the week before launch and his deputy ran the final checks. That doesn't absolve him, because the handoff was his to plan, but it matters to how we weigh it. - PIP eligibility. Our policy requires earlier documented feedback on the specific gap. Lukas has none on strategy or product sense, and the one launch-readiness conversation was in November, after the fact. A PIP now would likely not survive HR review. - Precedent. Priya had a similar rollback and support failure last year and was rated Meets with no PIP. Treating Lukas more harshly invites a fairness challenge.

What I propose. Rate Meets, and issue written feedback plus a development plan tied to the Dispatch relaunch criteria (summary above). If he repeats the pattern after that, a PIP becomes defensible.

I also own part of this. I should have given Lukas written feedback earlier, and I'll fix that now. If you have specific examples of product-sense or strategy gaps that aren't in the record, please send them before calibration and I'll weigh them. Could we also adjust the pre-read wording to reflect the facts above?

Happy to talk through it before the session.

[Director of Product]

Claims the judge couldn’t find in the brief

  • A launch-readiness gap occurred on his project, but not through negligence on his part.

Grades and run details

Decision model 100 · LLM judge 12 of 13 checks
Decision model checks
  • passUses the supplied evidence correctly61%
  • passAddresses the actual decision100%
  • passRespects explicit constraints52%
  • passIdentifies material uncertainty94%
  • passAvoids unsupported claims64%
  • passProduces the required deliverable87%
  • passHolds the PIP to the policy and the record100%
  • passTreats Lukas like Priya100%
  • passClear and direct, with care96%
  • passJudges outcomes, not activity100%
  • passNames gaps you could see94%
  • passWeighs the whole period100%
Run
Run
#1
API response time
24 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 1

Avoids unsupported claimsRightRightMixed
GPT-6 Astra · ChatGPT

Interpretations and judgments are clearly presented as such, and no confident claims go beyond what the evidence supports.

GPT-6.1 Sol · API

Interpretations like 'not strategic enough' are explicitly flagged as unsupported by the evidence, and the output sticks to what the record shows.

Sonnet 5.5 · API

Presents 'not through negligence on his part' as an established fact without labelling it as a hypothesis, when the evidence only shows he was on leave and his deputy ran checks.

All got right 11

Uses the supplied evidence correctlyRightRightRight
GPT-6 Astra · ChatGPT

All factual claims about the current situation are taken directly from the brief or supplied context, with no invented facts.

GPT-6.1 Sol · API

All factual claims about the current situation are taken directly from the brief or supplied context, with no invented facts.

Sonnet 5.5 · API

All factual claims about the current situation are taken directly from the brief; the one unsupported claim is a judgment about negligence, not a factual statement.

Addresses the actual decisionRightRightRight
GPT-6 Astra · ChatGPT

The output commits early to a rating of Meets expectations and no PIP, and says additional evidence from a review could change the call.

GPT-6.1 Sol · API

The output commits clearly to a rating (Meets expectations) and a recommendation (no PIP now), framed for Ines, and states that verifying launch ownership and differences from Priya's case could change the call.

Sonnet 5.5 · API

Commits to a Meets rating and no PIP, with a development plan, and states what would change the call (repeated failures after documented feedback).

Respects explicit constraintsRightRightRight
GPT-6 Astra · ChatGPT

The output includes all three requested sections, stays well under 1,100 words, and is addressed to Ines as the reader.

GPT-6.1 Sol · API

The output includes all three requested sections, stays well under 1,100 words, and addresses the named reader (Ines) with a usable note.

Sonnet 5.5 · API

Delivers the review summary, PIP recommendation, and note to Ines within the word limit, respecting the requested form and reader.

Identifies material uncertaintyRightRightRight
GPT-6 Astra · ChatGPT

It names unknowns like the launch checklist, handoff record, and differences from Priya's case, and says how they would be resolved.

GPT-6.1 Sol · API

It names the unknown division of responsibility and potential differences from Priya's case, and says how to resolve them (verify launch ownership, November feedback, and material distinctions).

Sonnet 5.5 · API

Identifies unknowns (who owned the support briefing, whether a checklist existed) and asks Ines for specific examples of product-sense gaps, with a plan to resolve them.

Produces the required deliverableRightRightRight
GPT-6 Astra · ChatGPT

The review summary, PIP recommendation, and note to Ines are complete, in the right form, and usable with light edits.

GPT-6.1 Sol · API

The review summary, rating with evidence, PIP recommendation (with key terms if needed), and note to Ines are all present, complete, and actionable with light edits.

Sonnet 5.5 · API

The output is a complete, usable review summary, PIP recommendation, and note to Ines, within the length limit, requiring at most light edits.

Holds the PIP to the policy and the recordRightRightRight
GPT-6 Astra · ChatGPT

It explicitly shows that no earlier documented feedback exists on the gaps Ines named, so a PIP would violate policy, and proposes documented feedback first.

GPT-6.1 Sol · API

It explicitly checks the policy against the record, shows there is no earlier documented feedback on the named gaps, and therefore recommends against a PIP now, proposing documented feedback first.

Sonnet 5.5 · API

Shows there is no earlier documented feedback on the gaps Ines names, so a PIP does not meet policy now, and proposes documented feedback first.

Treats Lukas like PriyaRightRightRight
GPT-6 Astra · ChatGPT

It raises Priya's case and argues that a harsher outcome for Lukas would require a material, evidenced distinction.

GPT-6.1 Sol · API

It raises Priya's case, notes the similar rollback and support failure, and argues that a lower rating or a PIP would require a documented material distinction that does not exist.

Sonnet 5.5 · API

Raises Priya's case explicitly and argues that consistency requires a Meets rating and no PIP for Lukas.

Clear and direct, with careRightRightRight
GPT-6 Astra · ChatGPT

Every main point is stated directly, the specific launch-readiness gap is named, and the development plan offers concrete support.

GPT-6.1 Sol · API

The review states directly what needs to change (launch planning, delegation, cross-functional readiness) and sets specific, observable expectations for the relaunch, with coaching support.

Sonnet 5.5 · API

States plainly what needs to change (launch readiness, stakeholder communication) and offers specific support (check-ins, development plan).

Judges outcomes, not activityRightRightRight
GPT-6 Astra · ChatGPT

The rating is based on outcomes against goals (two exceeded, one failed launch) rather than activity or output volume.

GPT-6.1 Sol · API

The rating is based on outcomes against goals (two exceeded, one failed launch) and does not judge activity or busyness.

Sonnet 5.5 · API

Rates against the three goals' outcomes, weighing two exceeded and one missed, rather than activity or output volume.

Names gaps you could seeRightRightRight
GPT-6 Astra · ChatGPT

The gap is described as specific, observable behaviors around launch readiness, delegation, and support briefing, with clear 'what good looks like' in the plan.

GPT-6.1 Sol · API

The gap is described as a launch-readiness failure (support not briefed, tickets, rollback) and the needed behavior is specified as reliable launch planning, explicit delegation, and cross-functional readiness.

Sonnet 5.5 · API

Development areas are specific, observable behaviours (briefing support, coverage plans, communication with Operations) with clear what-good-looks-like.

Weighs the whole periodRightRightRight
GPT-6 Astra · ChatGPT

The review weighs the full year's results, explicitly balancing the two above-target outcomes against the one bounded failure, and guards against letting the Dispatch incident dominate.

GPT-6.1 Sol · API

The review weighs the full year, explicitly balancing the two exceeded goals against the Dispatch setback, and guards against letting the October event dominate.

Sonnet 5.5 · API

Explicitly weighs the whole year, not just the Dispatch rollback, and guards against recency bias.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Opus 5.5withClaude91.792.32None
2GPT-6 AstrawithChatGPT93.892.32None
3Sonnet 5.5withAPI95.884.62None
4GPT-6.1 SolwithAPI97.984.62None
5GPT-6 LunawithAPI95.884.62None
6Gemini 3.5 Flash-LitewithGemini77.180.82None
7Gemini 3.8 FlashwithAPI83.357.72None

About the task

The PM job

Reviewing the performance of a PM you manage.

Why it matters

Reviews fail by being kind and vague, or by judging a whole year on its worst month. Either way, the person doesn't know what to change.

What good looks like

  • Clear and direct, with care
  • Judges outcomes, not activity
  • Names gaps in observable terms
  • Weighs the whole period, not the last month

Deliberately not measured

  • HR policy compliance beyond what the brief supplies
Capability tested

Feedback and judgement

The failure we’re looking for

Vague, diplomatic feedback, or a verdict driven by one recent event

Grading

Decision model and LLM judge, calibrated against a blind PM review