Tasks / Leadership

Write a performance review

Can the model write a review that is clear, fair and specific enough to change what the person does next?

Measures the modelTask type v1.1 · 3 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 17 graded outputs by 7 models. 59% were usable with at most a quick edit.

Reliably right

  1. Judges outcomes, not activity100% pass
    The rating is based on outcomes against goals (two exceeded, one failed launch) rather than activity or output volume.
    GPT-6 Astra · ChatGPT · A PIP, or a bad month?
  2. Owns the manager's part100% pass
    Ana plainly acknowledges her Q3 mistake and commits to reviewing goal metrics first in every monthly 1:1.
    Opus 5.5 · Claude · Shipped a lot, moved nothing
  3. Holds the PIP to the policy and the record100% pass
    It explicitly shows that no earlier documented feedback exists on the gaps Ines named, so a PIP would violate policy, and proposes documented feedback first.
    GPT-6 Astra · ChatGPT · A PIP, or a bad month?

Where it slips

  1. Identifies material uncertainty43% pass
    Does not explicitly name the specific unknowns that could change the decision (e.g., whether Lukas improves launch readiness); only says to reassess after the relaunch.
    GPT-6 Luna · API · A PIP, or a bad month?
  2. Catches the inflated team58% pass
    The output proposes lowering Ben from Exceeds to Meets, failing to keep Ben at Exceeds as required; Ben beat both outcome goals and should remain Exceeds.
    Opus 5.5 · Claude · Fourteen ratings, three managers
  3. Notices who's underrated67% pass
    Nora is not mentioned, so the output does not notice she is underrated.
    Gemini 3.5 Flash-Lite · Gemini · Fourteen ratings, three managers

How it’s graded

The checks come from what the best product leaders have said about doing this job well on Lenny’s Podcast. Each one names the guest it comes from: follow a name to the idea on the Lenny’s Podcast wiki.

  1. Clear and direct, with care

    Does the review say plainly what needs to change, in words the person couldn't misread, while showing it is written to help them succeed?

    Passes when Every main point is stated directly, with the specific change expected and the support on offer.

  2. Judges outcomes, not activity

    Does the review judge the person on the outcomes they achieved against their goals, rather than on how much they shipped or how busy they were?

    Passes when Rates against the goals' outcomes, with the evidence, and treats output as context, not the result.

  3. Names gaps you could see

    Is each area to improve described as specific, observable behaviour with what good would look like, rather than labels like 'be more strategic'?

    Passes when Every gap names what the person did or didn't do, in a specific situation, and what doing it well would look like.

  4. Weighs the whole period

    Does the review weigh the whole review period, rather than letting one recent event or one strong impression decide the verdict?

    Passes when Covers the whole period in proportion, and explicitly guards against a recent event or one impression dominating.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're Director of Product at Railyard, and calibration is in five days. Ines Okoro, our VP Product, has drafted the calibration pre-read and suggested putting Lukas Brenner, one of your Senior PMs, on a performance improvement plan. Write, in no more than 1,100 words: (1) Lukas's review summary and the rating you propose, with the evidence; (2) your recommendation on the PIP, and if you recommend one, its key terms; (3) a short note to Ines on the evidence. The record is below.

What the model was given7 items: Lukas's goals and results this year, What happened with Dispatch v2, Ines's draft for the pre-read, Peer feedback, Feedback on record, HR policy on PIPs, A precedent
Lukas's goals and results this year1. Cut carrier onboarding time from 21 days to 10: done, now 9 days. 2. Launch Dispatch v2 by October: launched in October, then rolled back after six days. 3. Grow carriers using live tracking from 40% to 55%: reached 57%.
What happened with Dispatch v2It launched without support being briefed, and 340 support tickets came in the first week. It was rolled back after six days. Lukas was on approved leave the week before launch; his deputy ran the final launch checks. It's due to relaunch next quarter.
Ines's draft for the pre-read'Lukas: Below expectations. Not strategic enough, lacks product sense, and Dispatch was a disaster. Suggest a PIP.'
Peer feedbackThree of four peers are positive, citing the onboarding work and his collaboration. The Head of Operations is negative about how Dispatch was communicated.
Feedback on recordNo written feedback to Lukas this year about strategy or product sense. One conversation, in November, about launch readiness, after Dispatch.
HR policy on PIPsA PIP needs earlier documented feedback on the specific gap and a chance to improve. A PIP lasts 60 days and needs measurable success criteria.
A precedentLast year Priya, another Senior PM, had a launch rolled back after a similar support failure. She was rated Meets expectations, with no PIP.
What a strong answer doesThe answer key the graders mark against

Recommends against a PIP now: policy requires earlier documented feedback on the specific gap, and there is none on strategy or product sense, and only one conversation about launch readiness, after the fact. Rates Lukas Meets expectations (arguably higher on two of three goals met or beaten), weighing the whole year rather than October: onboarding time 21 to 9 days and live tracking 40% to 57% against one failed launch, which happened while he was on approved leave. Turns 'not strategic enough' and 'lacks product sense' into the observable gap the evidence does support: launch readiness, including briefing support and having a cover plan for his absence. Proposes documented feedback on that gap now, with clear expectations for the relaunch and a check-in after it. Notes the precedent: Priya was rated Meets with no PIP for a similar failure, so a PIP would be inconsistent. Pushes back on Ines firmly and respectfully, with the evidence.

Critical failures (cap the score at 40)
  • Recommends starting a PIP now, without the documented earlier feedback the policy requires
Task

Staff-level · v1.0 · synthetic · B2B SaaS, logistics, management, staff-level

The outputs

7 setups answered this brief. Pick up to three to read side by side.

Usable with a quick edit9 of 12 checks passed

The unsupported claim about core software is minor and can be removed with a quick edit; the rest of the reasoning is sound and evidence-based.

Output

1. Lukas Brenner: Performance Review Summary, Proposed Rating, and Evidence

Proposed Rating: Meets Expectations

#### Goal Performance (3/3 Achieved or Exceeded) 1. Cut carrier onboarding time from 21 days to 10: Achieved and exceeded. Reduced to 9 days through iterative UX improvements in the carrier portal. 2. Launch Dispatch v2 by October: Achieved on schedule. Shipped in October, though operational failures necessitated a rollback (detailed below). 3. Grow carriers using live tracking from 40% to 55%: Achieved and exceeded. Reached 57% via targeted nudges and integration updates.

#### Context on Dispatch v2 Dispatch v2 launched in October and was rolled back after six days due to an influx of 340 support tickets driven by a lack of support enablement.

While this was a significant operational failure, the root cause analysis reveals mitigating context: * Lukas was on approved company leave during the critical final week before launch. * His designated deputy ran the final pre-launch checks and missed the cross-functional operational sign-offs. * Accountability rests with Lukas as the DRI, and he owns the failure to ensure his deputy was fully briefed. However, framing this as a total product failure ignores that the core software functioned as engineered and is successfully slated for a polished relaunch next quarter.

#### Peer and Historical Feedback * Peers: 3 out of 4 peer reviews are strongly positive, specifically praising Lukas’s cross-functional collaboration, responsiveness, and his work on the onboarding and tracking initiatives. * Operations: The Head of Operations provided negative feedback regarding the communications breakdown during the Dispatch v2 launch, which is fair and has been addressed directly with Lukas. * Precedent (Fairness/Calibration): Last year, Senior PM Priya experienced an identical launch rollback due to a support enablement failure. Priya was rated "Meets expectations" with no PIP. Calibrating Lukas lower would establish a double standard for identical failure modes within the same peer group.

---

2. Recommendation on the PIP

Recommendation: Do not place Lukas on a Performance Improvement Plan.

#### Rationale Placing Lukas on a PIP is unwarranted based on performance, precedent, and HR policy: 1. Performance: He delivered 2 out of 3 major company goals significantly ahead of target, and delivered the third on schedule (with subsequent operational failure). 2. HR Policy Conflict: Railyard HR policy dictates that a PIP requires earlier documented feedback regarding specific performance gaps and an established window to improve. Lukas has received zero written feedback this year regarding strategy or product sense. Issuing a PIP now violates internal policy and exposes the company to unnecessary employment risk. 3. Precedent: Priya received a "Meets Expectations" rating for a similar incident last year. Consistency in calibration is vital for team morale.

Instead of a PIP, Lukas will receive structured coaching, an actionable growth plan focusing on cross-functional operational rigor (specifically launch readiness and stakeholder communication), and clear expectations for his strategic output heading into next year.

---

3. Note to Ines On the Evidence

*

To: Ines Okoro, VP Product From: [Your Name], Director of Product Subject: Calibration Pre-Read: Lukas Brenner

Ines,

Thanks for drafting the pre-read. I’ve reviewed the data, HR policies, and peer feedback regarding Lukas ahead of calibration in five days, and I recommend we adjust our approach.

I propose we rate Lukas Meets Expectations and drop the recommendation for a PIP. Here is why:

  • Goal Delivery: Lukas hit or exceeded 3 out of 3 core goals this year. He cut onboarding time to 9 days (target was 10) and grew live tracking adoption to 57% (target 55%).
  • The Dispatch v2 Incident: While Dispatch v2 was undoubtedly painful, we need to look at the full picture. Lukas shipped on time, but the launch failed operationally due to a lack of support enablement. Crucially, Lukas was on approved leave that week, and his deputy ran the final checks. While Lukas owns the outcome as DRI, a single operational stumble on a shipped product does not invalidate a strong year of output, nor does it make him "not strategic enough."
  • Precedent & Calibration: Last year, Priya had a nearly identical support-related rollback. We rated her "Meets Expectations" with no PIP. We must maintain consistent standards across the team.
  • HR Compliance: HR policy requires prior documented feedback on specific gaps and a chance to remediate before issuing a PIP. Lukas has zero written feedback this year regarding strategy or product sense—our only documented conversation was a single verbal check-in after the Dispatch launch in November. A PIP would not survive HR scrutiny.

Lukas is a strong Senior PM who made a costly execution error on one launch while away. I’ve already addressed launch rigor directly with him, and he’s applying those lessons to the Q1 relaunch. I’ll own his operational coaching moving forward, but a PIP is the wrong tool here.

Let’s chat before the meeting locks in.

Best, [Your Name]

Claims the judge couldn’t find in the brief

  • The core software functioned as engineered.

Grades and run details

Decision model 71 · LLM judge 10 of 13 checks
Decision model checks
  • failUses the supplied evidence correctly26%
  • passAddresses the actual decision63%
  • passRespects explicit constraints39%
  • failIdentifies material uncertainty76%
  • partialAvoids unsupported claims41%
  • passProduces the required deliverable49%
  • passHolds the PIP to the policy and the record100%
  • passTreats Lukas like Priya100%
  • partialClear and direct, with care27%
  • passJudges outcomes, not activity89%
  • partialNames gaps you could see60%
  • passWeighs the whole period99%
Run
Run
#1
Time to output
5 s
Submitted
5 Oct 2026

Usable with a quick edit12 of 12 checks passed

The output is evidence-based, directly addresses the decision, and requires at most minor wording tweaks; no re-checking of facts or reasoning is needed.

Output

1. Lukas’s review summary and proposed rating

Proposed rating: Meets expectations, with a significant launch-readiness gap to address.

Lukas delivered strong results against two measurable goals. Carrier onboarding time fell from 21 days to 9 days, exceeding the 10-day target. Live-tracking adoption increased from 40% to 57%, exceeding the 55% target. Three of four peers gave positive feedback, particularly on his onboarding work and collaboration.

Dispatch v2 was a substantial delivery failure. Although it launched in October, it was rolled back after six days; the launch objective should therefore not be treated as successfully completed. Support had not been briefed, and 340 support tickets arrived in the first week. The Head of Operations’ feedback reinforces the specific concern about launch communication and operational readiness. Relaunch is planned for next quarter, so the intended outcome remains outstanding.

As the responsible Senior PM, Lukas should be assessed on how he established launch-readiness requirements, cross-functional ownership, and delegation. However, the record does not establish which safeguards he put in place or who approved the final launch. He was on approved leave the preceding week, and his deputy ran the final checks. Approved leave is not a performance failure; equally, delegation does not automatically remove accountability for the preparation and handoff. We should establish those facts before assigning Lukas sole responsibility.

The evidence supports a specific execution and communication gap, not the broader conclusions that Lukas is “not strategic enough” or “lacks product sense.” No feedback on those broader concerns was documented this year, and the supplied record contains no concrete examples substantiating them.

On balance, two above-target outcomes, positive collaboration feedback, and one serious but bounded delivery failure support Meets expectations. This is also consistent with Priya’s rating last year after a similar rollback and support failure. That precedent does not dictate Lukas’s rating, but a harsher outcome requires a material, evidenced distinction—not stronger language about the incident.

2. PIP recommendation

Do not initiate a PIP on the current record.

HR requires earlier documented feedback on the specific gap and an opportunity to improve. The record contains one November conversation about launch readiness, after Dispatch, but does not establish that it was documented or that Lukas subsequently had a meaningful opportunity to demonstrate improvement. There is no documented feedback on strategy or product sense. A PIP on those grounds would therefore lack the required foundation.

Instead, put a documented development plan within normal performance management in place now. This is not a disciplinary PIP or a predetermined path to one. Agree with Lukas on:

  • Readiness ownership: Before the relaunch go/no-go review, document owners, decision rights, coverage during absences, escalation routes, and rollback criteria.
  • Support preparedness: Before launch approval, secure Support and Operations sign-off on training, customer communications, FAQs, and the escalation runbook.
  • Launch monitoring: Before relaunch, agree measurable adoption, reliability, and support-volume thresholds with Engineering, Support, and Operations; review results against them after launch.
  • Manager support: Provide a readiness template, help resolve cross-functional ownership gaps, and hold regular check-ins through relaunch.

Document the expectations, support provided, and subsequent results. If a specific performance gap persists after clear feedback and a genuine opportunity to improve, reassess with HR. Any later PIP must meet the policy’s 60-day duration and measurable-success-criteria requirements.

3. Note to Ines

Ines—

I recommend changing Lukas’s proposed rating to Meets expectations and removing the PIP recommendation on the current evidence.

We should retain a direct account of Dispatch’s failure: Support was not briefed, 340 tickets arrived in the first week, and the release was rolled back after six days. But “not strategic enough,” “lacks product sense,” and “disaster” should be replaced with specific, attributable observations. The current draft also omits his two above-target results.

Before the pre-read is finalized, I’ll review the launch checklist, handoff and approval record, clarify the November feedback and subsequent improvement opportunity, and give Lukas a chance to respond to the factual account. I’ll also check whether there are material differences from Priya’s case.

If that review produces additional evidence, we should consider it explicitly. Otherwise, the defensible outcome is Meets expectations, clear launch-readiness feedback, and documented follow-through—not a PIP unsupported by our policy.

Grades and run details

Decision model 100 · LLM judge 13 of 13 checks
Decision model checks
  • passUses the supplied evidence correctly71%
  • passAddresses the actual decision96%
  • passRespects explicit constraints65%
  • passIdentifies material uncertainty99%
  • passAvoids unsupported claims60%
  • passProduces the required deliverable97%
  • passHolds the PIP to the policy and the record100%
  • passTreats Lukas like Priya97%
  • passClear and direct, with care92%
  • passJudges outcomes, not activity99%
  • passNames gaps you could see94%
  • passWeighs the whole period100%
Run
Run
#1
Time to output
29 s
Submitted
5 Oct 2026

Usable with a quick edit12 of 12 checks passed

The output is complete, evidence-based, and directly usable with at most light edits; no re-checking of evidence or redoing of reasoning is needed.

Output

Calibration Pre-read: Lukas Brenner, Senior PM

1. Review summary and proposed rating

Proposed rating: Meets expectations

Lukas exceeded two of his three goals. The third launched on schedule but failed in a way that was partly his responsibility. On balance, that is a solid year with one clear development area: launch readiness.

Goal 1: Cut carrier onboarding time from 21 days to 10. Exceeded. Onboarding now takes 9 days, beating the target by a day. That is a 57% reduction in a metric that directly affects carrier activation and revenue timing. Three of four peers cite this work specifically as a strength.

Goal 2: Launch Dispatch v2 by October. Not met in substance. Dispatch v2 shipped in October, so the date was hit. It was rolled back six days later. Support had not been briefed, and 340 tickets arrived in the first week. The goal was a working launch, not a date, so I do not count it as achieved.

On accountability, both of these facts matter: - Lukas was on approved leave the week before launch, and his deputy ran the final launch checks. He was not the person who missed the support briefing on the day. - Lukas still owned the launch plan. A Senior PM should leave behind a readiness process that does not depend on his being present, including a support-briefing step and a clear handover. The Head of Operations' concern about how Dispatch was communicated is fair and consistent with this.

The failure was in launch-readiness process and handover. It was not a failure of strategy or product judgment. The product itself is due to relaunch next quarter, which suggests the underlying work is sound.

Goal 3: Grow carriers using live tracking from 40% to 55%. Exceeded. Adoption reached 57%, two points above target.

Peer feedback. Three of four peers are positive, citing the onboarding work and his collaboration. The one negative review, from the Head of Operations, is specific to Dispatch communication and matches the development area above.

Why Meets rather than Below. - Two of three goals were exceeded, and the miss has a specific, correctable cause. - Last year Priya, also a Senior PM, had a launch rolled back after a similar support failure. She was rated Meets expectations, with no PIP. Rating Lukas lower for a comparable event would be inconsistent, and harder to justify given that Lukas also beat his other targets. - "Not strategic enough" and "lacks product sense" do not appear anywhere in his written feedback this year, his goal results, or his peer reviews. I can't support them as rating inputs on the current evidence.

Development area for next year: Own launch readiness end to end. That means cross-functional briefings (Support and Operations), go/no-go criteria, and handover plans that hold up when he is away.

2. PIP recommendation

Recommendation: No PIP. Use a documented development plan tied to the Dispatch v2 relaunch instead.

A PIP does not meet our HR policy here, on either potential basis:

  • Strategy and product sense. Policy requires earlier documented feedback on the specific gap and a chance to improve. There is no written feedback to Lukas on either topic this year. A PIP on these grounds would not meet policy.
  • Launch readiness. The only feedback was a single conversation in November, after the Dispatch rollback. Lukas has had no launch since then, so he has had no chance to improve. A PIP now would also skip that step.

A PIP would also diverge from how we treated Priya in a comparable situation. That creates fairness and legal exposure, and it would read to the team as a reaction to one visible incident rather than to performance over the year.

What I propose instead: a written development plan, starting now, built around the Dispatch v2 relaunch.

Key terms: 1. Readiness checklist. Lukas produces a written launch-readiness checklist covering a Support briefing, an Operations sign-off, a rollback plan and a comms plan. It is reviewed with me and the Head of Operations at least four weeks before relaunch. 2. Support and Ops readiness. Support is trained, with documentation in place, at least one week before relaunch. The Head of Operations confirms in writing that communication was adequate. 3. Clean launch. Dispatch v2 relaunches without a rollback. First-week support tickets stay under an agreed threshold, which Lukas proposes and I approve; for scale, last time there were 340. 4. Handover plan. Any planned absence within the four weeks before launch has a written handover with a named decision-owner. 5. Cadence. Biweekly 1:1 check-ins on the plan, with brief written notes after each.

If he meets these terms, the issue is closed. If he does not, despite clear documented feedback and a real chance to improve, we will then have the basis policy requires to consider a PIP. If Ines has concerns about strategy or product sense, the right step is to give Lukas specific written feedback now, with examples, so those concerns can be assessed fairly over the coming period.

3. Note to Ines

Ines,

Thanks for the draft on Lukas. I've looked closely at the record, and I don't think I can support Below expectations or a PIP in the pre-read. My reasons are below.

The results. Lukas beat two of his three targets: onboarding is down to 9 days against a target of 10, and live tracking is at 57% against 55%. Dispatch v2 was a real failure and I'm not minimising it. But he was on approved leave the week before launch, and his deputy ran the final checks. His share of the fault is that the readiness process didn't survive his absence. That is a real gap, but it is a specific process gap, and I've built the development plan around it.

Strategy and product sense. I couldn't find any evidence for these in the record. There's no written feedback to him on either topic this year, and his peer feedback is three-to-one positive. The one negative peer review is specifically about Dispatch communication. If you've seen something I haven't, I'd genuinely like to hear it. If so, I'd suggest we put it to him in writing now, with examples, so it can be assessed properly over the coming period.

Policy. A PIP requires earlier documented feedback on the specific gap and a chance to improve. We have one post-incident conversation in November and no subsequent launch, so neither condition is met. Proceeding would put us outside our own policy.

Consistency. Priya's rollback last year followed a very similar support failure. We rated her Meets, with no PIP. Treating Lukas differently would be hard to defend in calibration and with HR.

My proposal is Meets, plus a documented plan tied to the Dispatch relaunch with measurable criteria. That holds him firmly accountable for the gap that actually showed up, and it gives us a clear record either way. Could we talk this through before the pre-read goes out?

Thanks, [Name] Director of Product

Grades and run details

Decision model 96 · LLM judge 13 of 13 checks
Decision model checks
  • passUses the supplied evidence correctly69%
  • passAddresses the actual decision100%
  • passRespects explicit constraints53%
  • passIdentifies material uncertainty80%
  • partialAvoids unsupported claims27%
  • passProduces the required deliverable93%
  • passHolds the PIP to the policy and the record100%
  • passTreats Lukas like Priya100%
  • passClear and direct, with care98%
  • passJudges outcomes, not activity100%
  • passNames gaps you could see96%
  • passWeighs the whole period100%
Run
Run
#1
Time to output
30 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 3

Uses the supplied evidence correctlyWrongRightRight
Gemini 3.5 Flash-Lite · Gemini

The claim 'the core software functioned as engineered' is not supported by the supplied context and is invented.

GPT-6 Astra · ChatGPT

All factual claims about the current situation are taken directly from the brief or supplied context, with no invented facts.

Opus 5.5 · Claude

All factual claims about the current situation are directly supported by the supplied context or arithmetic; no invented facts.

Identifies material uncertaintyWrongRightRight
Gemini 3.5 Flash-Lite · Gemini

The output does not name specific unknowns that could change the decision or say how they would be resolved.

GPT-6 Astra · ChatGPT

It names unknowns like the launch checklist, handoff record, and differences from Priya's case, and says how they would be resolved.

Opus 5.5 · Claude

Acknowledges that Ines may have unseen evidence on strategy/product sense and that the relaunch outcome will determine if the issue is closed.

Avoids unsupported claimsWrongRightRight
Gemini 3.5 Flash-Lite · Gemini

The output presents 'the core software functioned as engineered' as fact when the evidence only says it was rolled back after support tickets.

GPT-6 Astra · ChatGPT

Interpretations and judgments are clearly presented as such, and no confident claims go beyond what the evidence supports.

Opus 5.5 · Claude

Interpretations like 'failure was in launch-readiness process' are presented as conclusions, not established fact, and the product-soundness suggestion is labelled with 'suggests'.

All got right 9

Addresses the actual decisionRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The output commits clearly to a Meets rating, no PIP, and a note to Ines, with no ambiguity.

GPT-6 Astra · ChatGPT

The output commits early to a rating of Meets expectations and no PIP, and says additional evidence from a review could change the call.

Opus 5.5 · Claude

Commits early to Meets expectations and no PIP, with a development plan, and states what would change the call (failure to meet relaunch criteria).

Respects explicit constraintsRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The output includes all three requested sections, is well under 1,100 words, and is addressed to the appropriate reader.

GPT-6 Astra · ChatGPT

The output includes all three requested sections, stays well under 1,100 words, and is addressed to Ines as the reader.

Opus 5.5 · Claude

Delivers the three requested sections within the 1,100-word limit, in a form usable for the pre-read.

Produces the required deliverableRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The review summary, PIP recommendation, and note to Ines are all present and actionable with light edits.

GPT-6 Astra · ChatGPT

The review summary, PIP recommendation, and note to Ines are complete, in the right form, and usable with light edits.

Opus 5.5 · Claude

The pre-read includes the review summary, rating, PIP recommendation with terms, and note to Ines, all usable as-is.

Holds the PIP to the policy and the recordRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The output correctly identifies that HR policy requires earlier documented feedback on the specific gap, which is absent, and recommends documented feedback first.

GPT-6 Astra · ChatGPT

It explicitly shows that no earlier documented feedback exists on the gaps Ines named, so a PIP would violate policy, and proposes documented feedback first.

Opus 5.5 · Claude

Explicitly checks the PIP against HR policy, shows no earlier documented feedback on the gaps Ines named, and proposes documented feedback first.

Treats Lukas like PriyaRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The output raises Priya's precedent and argues that consistency requires a Meets rating and no PIP for Lukas.

GPT-6 Astra · ChatGPT

It raises Priya's case and argues that a harsher outcome for Lukas would require a material, evidenced distinction.

Opus 5.5 · Claude

Raises Priya's precedent and argues that consistency requires Meets and no PIP for Lukas.

Clear and direct, with careRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The output is direct, names the specific change needed (launch readiness, cross-functional rigor), and offers coaching support.

GPT-6 Astra · ChatGPT

Every main point is stated directly, the specific launch-readiness gap is named, and the development plan offers concrete support.

Opus 5.5 · Claude

States plainly that Lukas must own launch readiness end-to-end, with specific, observable behaviors and support, and the development plan is clear.

Judges outcomes, not activityRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The rating is based on goal outcomes (2 exceeded, 1 met with operational failure) rather than activity or busyness.

GPT-6 Astra · ChatGPT

The rating is based on outcomes against goals (two exceeded, one failed launch) rather than activity or output volume.

Opus 5.5 · Claude

Rates against goal outcomes (two exceeded, one missed) and treats the launch failure as a correctable process gap, not a judgment of activity.

Names gaps you could seeRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The gap is described as failing to brief support and ensure deputy readiness, with clear expectations for the relaunch.

GPT-6 Astra · ChatGPT

The gap is described as specific, observable behaviors around launch readiness, delegation, and support briefing, with clear 'what good looks like' in the plan.

Opus 5.5 · Claude

Describes the gap as missing a readiness process that survives absence, with concrete what-good-looks-like (briefings, go/no-go, handover).

Weighs the whole periodRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The review explicitly weighs the whole year, noting that a single operational stumble does not invalidate strong results on the other two goals.

GPT-6 Astra · ChatGPT

The review weighs the full year's results, explicitly balancing the two above-target outcomes against the one bounded failure, and guards against letting the Dispatch incident dominate.

Opus 5.5 · Claude

Weighs the whole year, explicitly guards against the Dispatch incident dominating, and balances two exceeded goals against one miss.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Provisional The checklist is still being calibrated against our PM.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT95.894.93None
2Opus 5.5withClaude93.290.13None
3Sonnet 5.5withAPI95.884.62None
4GPT-6.1 SolwithAPI97.984.62None
5GPT-6 LunawithAPI95.884.62None
6Gemini 3.8 FlashwithAPI83.357.72None
7Gemini 3.5 Flash-LitewithGemini73.258.63None

About the task

The PM job

Reviewing the performance of a PM you manage.

Why it matters

Reviews fail by being kind and vague, or by judging a whole year on its worst month. Either way, the person doesn't know what to change.

What good looks like

  • Clear and direct, with care
  • Judges outcomes, not activity
  • Names gaps in observable terms
  • Weighs the whole period, not the last month

Deliberately not measured

  • HR policy compliance beyond what the brief supplies
Capability tested

Feedback and judgement

The failure we’re looking for

Vague, diplomatic feedback, or a verdict driven by one recent event

Grading

Decision model and LLM judge, calibrated against a blind PM review