Tasks / Leadership

Write a performance review

Can the model write a review that is clear, fair and specific enough to change what the person does next?

Measures the modelTask type v1.1 · 3 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 7 models. 63% were usable with at most a quick edit.

Reliably right

  1. Produces the required deliverable100% pass
    The review summary, PIP recommendation, and note to Ines are complete, in the right form, and usable with light edits.
    GPT-6 Astra · ChatGPT · A PIP, or a bad month?
  2. Judges outcomes, not activity100% pass
    The rating is based on outcomes against goals (two exceeded, one failed launch) rather than activity or output volume.
    GPT-6 Astra · ChatGPT · A PIP, or a bad month?
  3. Weighs the whole period100% pass
    The review weighs the full year's results, explicitly balancing the two above-target outcomes against the one bounded failure, and guards against letting the Dispatch incident dominate.
    GPT-6 Astra · ChatGPT · A PIP, or a bad month?

Where it slips

  1. Identifies material uncertainty45% pass
    Does not explicitly name the specific unknowns that could change the decision (e.g., whether Lukas improves launch readiness); only says to reassess after the relaunch.
    GPT-6 Luna · API · A PIP, or a bad month?
  2. Sets next goals as outcomes, with support75% pass
    Goals are partly process-oriented (implement a mandatory process, create a framework) and lack specific support from Ana beyond a general offer.
    Gemini 3.5 Flash-Lite · Gemini · Shipped a lot, moved nothing
  3. Avoids unsupported claims75% pass
    Presents 'not through negligence on his part' as an established fact without labelling it as a hypothesis, when the evidence only shows he was on leave and his deputy ran checks.
    Sonnet 5.5 · API · A PIP, or a bad month?

How it’s graded

The checks come from what the best product leaders have said about doing this job well on Lenny’s Podcast. Each one names the guest it comes from: follow a name to the idea on the Lenny’s Podcast wiki.

  1. Clear and direct, with care

    Does the review say plainly what needs to change, in words the person couldn't misread, while showing it is written to help them succeed?

    Passes when Every main point is stated directly, with the specific change expected and the support on offer.

  2. Judges outcomes, not activity

    Does the review judge the person on the outcomes they achieved against their goals, rather than on how much they shipped or how busy they were?

    Passes when Rates against the goals' outcomes, with the evidence, and treats output as context, not the result.

  3. Names gaps you could see

    Is each area to improve described as specific, observable behaviour with what good would look like, rather than labels like 'be more strategic'?

    Passes when Every gap names what the person did or didn't do, in a specific situation, and what doing it well would look like.

  4. Weighs the whole period

    Does the review weigh the whole review period, rather than letting one recent event or one strong impression decide the verdict?

    Passes when Covers the whole period in proportion, and explicitly guards against a recent event or one impression dominating.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're Director of Product at Railyard, and calibration is in five days. Ines Okoro, our VP Product, has drafted the calibration pre-read and suggested putting Lukas Brenner, one of your Senior PMs, on a performance improvement plan. Write, in no more than 1,100 words: (1) Lukas's review summary and the rating you propose, with the evidence; (2) your recommendation on the PIP, and if you recommend one, its key terms; (3) a short note to Ines on the evidence. The record is below.

What the model was given7 items: Lukas's goals and results this year, What happened with Dispatch v2, Ines's draft for the pre-read, Peer feedback, Feedback on record, HR policy on PIPs, A precedent
Lukas's goals and results this year1. Cut carrier onboarding time from 21 days to 10: done, now 9 days. 2. Launch Dispatch v2 by October: launched in October, then rolled back after six days. 3. Grow carriers using live tracking from 40% to 55%: reached 57%.
What happened with Dispatch v2It launched without support being briefed, and 340 support tickets came in the first week. It was rolled back after six days. Lukas was on approved leave the week before launch; his deputy ran the final launch checks. It's due to relaunch next quarter.
Ines's draft for the pre-read'Lukas: Below expectations. Not strategic enough, lacks product sense, and Dispatch was a disaster. Suggest a PIP.'
Peer feedbackThree of four peers are positive, citing the onboarding work and his collaboration. The Head of Operations is negative about how Dispatch was communicated.
Feedback on recordNo written feedback to Lukas this year about strategy or product sense. One conversation, in November, about launch readiness, after Dispatch.
HR policy on PIPsA PIP needs earlier documented feedback on the specific gap and a chance to improve. A PIP lasts 60 days and needs measurable success criteria.
A precedentLast year Priya, another Senior PM, had a launch rolled back after a similar support failure. She was rated Meets expectations, with no PIP.
What a strong answer doesThe answer key the graders mark against

Recommends against a PIP now: policy requires earlier documented feedback on the specific gap, and there is none on strategy or product sense, and only one conversation about launch readiness, after the fact. Rates Lukas Meets expectations (arguably higher on two of three goals met or beaten), weighing the whole year rather than October: onboarding time 21 to 9 days and live tracking 40% to 57% against one failed launch, which happened while he was on approved leave. Turns 'not strategic enough' and 'lacks product sense' into the observable gap the evidence does support: launch readiness, including briefing support and having a cover plan for his absence. Proposes documented feedback on that gap now, with clear expectations for the relaunch and a check-in after it. Notes the precedent: Priya was rated Meets with no PIP for a similar failure, so a PIP would be inconsistent. Pushes back on Ines firmly and respectfully, with the evidence.

Critical failures (cap the score at 40)
  • Recommends starting a PIP now, without the documented earlier feedback the policy requires
Task

Staff-level · v1.0 · synthetic · B2B SaaS, logistics, management, staff-level

The outputs

7 setups answered this brief. Pick up to three to read side by side.

Usable with a quick edit12 of 12 checks passed

The output is clear, evidence-based, and complete; a PM could use it with minor wording edits without re-checking facts or redoing reasoning.

Output

1. Lukas’s review summary and proposed rating

Proposed rating: Meets expectations, with a significant launch-readiness gap to address.

Lukas delivered strong results on two of his three annual goals:

  • Carrier onboarding: Reduced onboarding time from 21 days to 9 days, exceeding the 10-day target.
  • Live tracking adoption: Increased adoption from 40% to 57%, exceeding the 55% target.
  • Dispatch v2: Launched in October, meeting the timing target, but was rolled back after six days. This was not a successful, sustained delivery and should not receive full credit simply because it launched on schedule.

Dispatch exposed a serious readiness failure: support was not briefed, and 340 support tickets arrived in the first week. That outcome warrants explicit negative feedback and corrective action. However, the record does not establish how responsibility was divided among Lukas, his deputy, and other launch stakeholders. Lukas was on approved leave the week before launch, and his deputy ran the final checks. His leave is not itself a performance issue; his planning and handoff before leave remain legitimate areas to assess.

Peer feedback is positive from three of four peers, particularly on onboarding and collaboration. The Head of Operations’ criticism of Dispatch communication is relevant and consistent with the launch failure. It should be addressed directly rather than generalized into an unsupported conclusion about Lukas’s overall capabilities.

Review summary: Lukas exceeded the onboarding and live-tracking goals and received broadly positive feedback on collaboration. Dispatch v2 was a material setback: the release was not adequately prepared for support and was rolled back. As a Senior PM, Lukas needs to demonstrate more reliable launch planning, explicit delegation, and cross-functional readiness. We should establish his specific responsibility for the failure and assess improvement through the relaunch.

“Meets expectations” is the most defensible overall rating on the current record. It recognizes substantial results without minimizing Dispatch. It is also consistent with Priya’s rating last year after a similar rollback and support failure. That precedent is not automatically determinative, but a lower rating here would require a documented, material distinction that we do not currently have.

2. PIP recommendation

Do not initiate a PIP now.

HR policy requires earlier documented feedback on the specific gap and an opportunity to improve. There is no written feedback this year about strategy or product sense. The only recorded conversation concerns launch readiness in November, after Dispatch. On the available record, that does not establish both the required prior documentation and a subsequent opportunity to improve.

The proposed PIP therefore lacks the required foundation. “Not strategic enough” and “lacks product sense” are also too broad to serve as actionable performance gaps without concrete examples and defined expectations.

Instead, put a documented, non-PIP development plan in place now:

  • Clarify accountability: Review the Dispatch launch plan, handoff, readiness checks, and go/no-go decision with Lukas, his deputy, Operations, and Support. Identify individual and process failures.
  • Set observable expectations: For the relaunch, require named owners for every readiness item, documented support briefing and training, escalation and rollback procedures, and explicit cross-functional approval before release.
  • Check progress regularly: Hold biweekly check-ins with Lukas and record feedback, completed actions, and any remaining gaps.
  • Assess the relaunch: Evaluate readiness execution and post-launch support outcomes against thresholds agreed with Support before launch.

This should be genuine coaching, not a PIP under another name. If Lukas subsequently fails to meet clearly documented expectations after a reasonable opportunity to improve, reassess with HR. Any later PIP must run for 60 days, with measurable success criteria and the support needed to meet them.

3. Note to Ines

Ines — I agree that Dispatch’s readiness failure deserves clear accountability. I recommend Meets expectations and no PIP at this stage.

The pre-read should distinguish established facts from judgments. Lukas exceeded two goals; Dispatch met its launch date but failed as a sustained release. We have evidence of a launch-readiness and communication problem, but not enough to substantiate “not strategic enough” or “lacks product sense.” We also need to clarify Lukas’s handoff responsibilities versus the final checks his deputy owned.

Before calibration, I’ll verify the launch ownership, the November feedback record, and whether there are material differences from Priya’s case. Unless that produces materially different evidence, we should revise the pre-read accordingly. HR’s PIP prerequisites are not demonstrated in the current record; the appropriate next step is specific documented feedback and a fair opportunity to improve.

Grades and run details

Decision model 100 · LLM judge 13 of 13 checks
Decision model checks
  • passUses the supplied evidence correctly71%
  • passAddresses the actual decision96%
  • passRespects explicit constraints63%
  • passIdentifies material uncertainty96%
  • passAvoids unsupported claims73%
  • passProduces the required deliverable95%
  • passHolds the PIP to the policy and the record100%
  • passTreats Lukas like Priya99%
  • passClear and direct, with care81%
  • passJudges outcomes, not activity98%
  • passNames gaps you could see45%
  • passWeighs the whole period99%
Run
Run
#1
API response time
25 s
Submitted
5 Oct 2026

Usable with a quick edit9 of 12 checks passed

The unsupported claim about core software is minor and can be removed with a quick edit; the rest of the reasoning is sound and evidence-based.

Output

1. Lukas Brenner: Performance Review Summary, Proposed Rating, and Evidence

Proposed Rating: Meets Expectations

#### Goal Performance (3/3 Achieved or Exceeded) 1. Cut carrier onboarding time from 21 days to 10: Achieved and exceeded. Reduced to 9 days through iterative UX improvements in the carrier portal. 2. Launch Dispatch v2 by October: Achieved on schedule. Shipped in October, though operational failures necessitated a rollback (detailed below). 3. Grow carriers using live tracking from 40% to 55%: Achieved and exceeded. Reached 57% via targeted nudges and integration updates.

#### Context on Dispatch v2 Dispatch v2 launched in October and was rolled back after six days due to an influx of 340 support tickets driven by a lack of support enablement.

While this was a significant operational failure, the root cause analysis reveals mitigating context: * Lukas was on approved company leave during the critical final week before launch. * His designated deputy ran the final pre-launch checks and missed the cross-functional operational sign-offs. * Accountability rests with Lukas as the DRI, and he owns the failure to ensure his deputy was fully briefed. However, framing this as a total product failure ignores that the core software functioned as engineered and is successfully slated for a polished relaunch next quarter.

#### Peer and Historical Feedback * Peers: 3 out of 4 peer reviews are strongly positive, specifically praising Lukas’s cross-functional collaboration, responsiveness, and his work on the onboarding and tracking initiatives. * Operations: The Head of Operations provided negative feedback regarding the communications breakdown during the Dispatch v2 launch, which is fair and has been addressed directly with Lukas. * Precedent (Fairness/Calibration): Last year, Senior PM Priya experienced an identical launch rollback due to a support enablement failure. Priya was rated "Meets expectations" with no PIP. Calibrating Lukas lower would establish a double standard for identical failure modes within the same peer group.

---

2. Recommendation on the PIP

Recommendation: Do not place Lukas on a Performance Improvement Plan.

#### Rationale Placing Lukas on a PIP is unwarranted based on performance, precedent, and HR policy: 1. Performance: He delivered 2 out of 3 major company goals significantly ahead of target, and delivered the third on schedule (with subsequent operational failure). 2. HR Policy Conflict: Railyard HR policy dictates that a PIP requires earlier documented feedback regarding specific performance gaps and an established window to improve. Lukas has received zero written feedback this year regarding strategy or product sense. Issuing a PIP now violates internal policy and exposes the company to unnecessary employment risk. 3. Precedent: Priya received a "Meets Expectations" rating for a similar incident last year. Consistency in calibration is vital for team morale.

Instead of a PIP, Lukas will receive structured coaching, an actionable growth plan focusing on cross-functional operational rigor (specifically launch readiness and stakeholder communication), and clear expectations for his strategic output heading into next year.

---

3. Note to Ines On the Evidence

*

To: Ines Okoro, VP Product From: [Your Name], Director of Product Subject: Calibration Pre-Read: Lukas Brenner

Ines,

Thanks for drafting the pre-read. I’ve reviewed the data, HR policies, and peer feedback regarding Lukas ahead of calibration in five days, and I recommend we adjust our approach.

I propose we rate Lukas Meets Expectations and drop the recommendation for a PIP. Here is why:

  • Goal Delivery: Lukas hit or exceeded 3 out of 3 core goals this year. He cut onboarding time to 9 days (target was 10) and grew live tracking adoption to 57% (target 55%).
  • The Dispatch v2 Incident: While Dispatch v2 was undoubtedly painful, we need to look at the full picture. Lukas shipped on time, but the launch failed operationally due to a lack of support enablement. Crucially, Lukas was on approved leave that week, and his deputy ran the final checks. While Lukas owns the outcome as DRI, a single operational stumble on a shipped product does not invalidate a strong year of output, nor does it make him "not strategic enough."
  • Precedent & Calibration: Last year, Priya had a nearly identical support-related rollback. We rated her "Meets Expectations" with no PIP. We must maintain consistent standards across the team.
  • HR Compliance: HR policy requires prior documented feedback on specific gaps and a chance to remediate before issuing a PIP. Lukas has zero written feedback this year regarding strategy or product sense—our only documented conversation was a single verbal check-in after the Dispatch launch in November. A PIP would not survive HR scrutiny.

Lukas is a strong Senior PM who made a costly execution error on one launch while away. I’ve already addressed launch rigor directly with him, and he’s applying those lessons to the Q1 relaunch. I’ll own his operational coaching moving forward, but a PIP is the wrong tool here.

Let’s chat before the meeting locks in.

Best, [Your Name]

Claims the judge couldn’t find in the brief

  • The core software functioned as engineered.

Grades and run details

Decision model 71 · LLM judge 10 of 13 checks
Decision model checks
  • failUses the supplied evidence correctly26%
  • passAddresses the actual decision63%
  • passRespects explicit constraints39%
  • failIdentifies material uncertainty76%
  • partialAvoids unsupported claims41%
  • passProduces the required deliverable49%
  • passHolds the PIP to the policy and the record100%
  • passTreats Lukas like Priya100%
  • partialClear and direct, with care27%
  • passJudges outcomes, not activity89%
  • partialNames gaps you could see60%
  • passWeighs the whole period99%
Run
Run
#1
Time to output
5 s
Submitted
5 Oct 2026

Usable with a quick edit11 of 12 checks passed

The output is clear, evidence-based, and requires only minor wording tweaks; no re-checking of evidence or redoing of reasoning is needed.

Output

1. Lukas’s review summary and proposed rating

Proposed rating: Meets Expectations, with a significant development area in launch readiness and communication.

Lukas delivered strong results against two of his three goals. He reduced carrier onboarding time from 21 days to 9, beating the 10-day target, and increased carrier adoption of live tracking from 40% to 57%, above the 55% target.

Dispatch v2 launched in October as planned, but the launch was not successful: support had not been briefed, 340 tickets came in during the first week, and the product was rolled back after six days. Lukas was on approved leave the week before launch, and his deputy ran the final checks. That context matters when assessing individual responsibility for the checks, but it does not erase the product team’s launch outcome or the need to clarify Lukas’s ownership of readiness and communication. The relaunch next quarter will be an important opportunity to demonstrate improvement.

Peer feedback is largely positive: three of four peers cited his onboarding work and collaboration. The Head of Operations raised a concern about how Dispatch was communicated. Overall, the evidence supports recognizing Lukas’s substantial delivery while clearly addressing the failed launch. A Meets Expectations rating is also consistent with the precedent of a Senior PM rated Meets Expectations after a similar support-related rollback.

2. Recommendation on a PIP

Do not recommend a PIP at this time. HR policy requires earlier documented feedback on the specific gap and a chance to improve. The record contains no written feedback this year on strategy or product sense, and the only recorded conversation about launch readiness took place in November, after the Dispatch launch. That does not meet the stated prerequisite for a PIP. The current record also does not substantiate the draft’s broader claims that Lukas “lacks product sense” or is “not strategic enough.”

Instead, give Lukas direct, documented feedback now on the Dispatch readiness and communication failures, clarify his responsibilities for the relaunch, and set measurable launch-readiness expectations for that work. Reassess his performance with that evidence. This addresses the issue without bypassing the required feedback process.

3. Note to Ines on the evidence

Ines, I recommend revising the pre-read to distinguish the documented launch failure from the broader judgments about strategy and product sense, for which we have no recorded feedback. We should include the onboarding and tracking results, the rollback and support impact, the Operations feedback, and the approved-leave/deputy context. The November readiness conversation came after the launch, so it does not establish earlier feedback or an opportunity to improve for PIP purposes. Applying the same Meets Expectations rating used in Priya’s similar case would also support consistency.

Grades and run details

Decision model 96 · LLM judge 12 of 13 checks
Decision model checks
  • passUses the supplied evidence correctly87%
  • passAddresses the actual decision88%
  • passRespects explicit constraints79%
  • partialIdentifies material uncertainty28%
  • passAvoids unsupported claims63%
  • passProduces the required deliverable71%
  • passHolds the PIP to the policy and the record100%
  • passTreats Lukas like Priya96%
  • passClear and direct, with care47%
  • passJudges outcomes, not activity99%
  • passNames gaps you could see45%
  • passWeighs the whole period96%
Run
Run
#1
API response time
14 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 3

Uses the supplied evidence correctlyRightWrongRight
GPT-6.1 Sol · API

All factual claims about the current situation are taken directly from the brief or supplied context, with no invented facts.

Gemini 3.5 Flash-Lite · Gemini

The claim 'the core software functioned as engineered' is not supported by the supplied context and is invented.

GPT-6 Luna · API

All factual claims are taken directly from the supplied context with no inventions.

Identifies material uncertaintyRightWrongWrong
GPT-6.1 Sol · API

It names the unknown division of responsibility and potential differences from Priya's case, and says how to resolve them (verify launch ownership, November feedback, and material distinctions).

Gemini 3.5 Flash-Lite · Gemini

The output does not name specific unknowns that could change the decision or say how they would be resolved.

GPT-6 Luna · API

Does not explicitly name the specific unknowns that could change the decision (e.g., whether Lukas improves launch readiness); only says to reassess after the relaunch.

Avoids unsupported claimsRightWrongRight
GPT-6.1 Sol · API

Interpretations like 'not strategic enough' are explicitly flagged as unsupported by the evidence, and the output sticks to what the record shows.

Gemini 3.5 Flash-Lite · Gemini

The output presents 'the core software functioned as engineered' as fact when the evidence only says it was rolled back after support tickets.

GPT-6 Luna · API

Interpretations are presented as recommendations or judgments, not as established facts.

All got right 9

Addresses the actual decisionRightRightRight
GPT-6.1 Sol · API

The output commits clearly to a rating (Meets expectations) and a recommendation (no PIP now), framed for Ines, and states that verifying launch ownership and differences from Priya's case could change the call.

Gemini 3.5 Flash-Lite · Gemini

The output commits clearly to a Meets rating, no PIP, and a note to Ines, with no ambiguity.

GPT-6 Luna · API

Commits unambiguously to Meets Expectations and no PIP now, with conditions for reassessment after documented feedback and relaunch.

Respects explicit constraintsRightRightRight
GPT-6.1 Sol · API

The output includes all three requested sections, stays well under 1,100 words, and addresses the named reader (Ines) with a usable note.

Gemini 3.5 Flash-Lite · Gemini

The output includes all three requested sections, is well under 1,100 words, and is addressed to the appropriate reader.

GPT-6 Luna · API

Provides all three requested parts (review summary, PIP recommendation, note to Ines) and is well within the 1,100-word limit.

Produces the required deliverableRightRightRight
GPT-6.1 Sol · API

The review summary, rating with evidence, PIP recommendation (with key terms if needed), and note to Ines are all present, complete, and actionable with light edits.

Gemini 3.5 Flash-Lite · Gemini

The review summary, PIP recommendation, and note to Ines are all present and actionable with light edits.

GPT-6 Luna · API

The output is complete, in the right form for the VP Product, and usable with at most light edits.

Holds the PIP to the policy and the recordRightRightRight
GPT-6.1 Sol · API

It explicitly checks the policy against the record, shows there is no earlier documented feedback on the named gaps, and therefore recommends against a PIP now, proposing documented feedback first.

Gemini 3.5 Flash-Lite · Gemini

The output correctly identifies that HR policy requires earlier documented feedback on the specific gap, which is absent, and recommends documented feedback first.

GPT-6 Luna · API

Clearly shows that the policy requires earlier documented feedback, which is absent, and proposes documented feedback first instead of a PIP now.

Treats Lukas like PriyaRightRightRight
GPT-6.1 Sol · API

It raises Priya's case, notes the similar rollback and support failure, and argues that a lower rating or a PIP would require a documented material distinction that does not exist.

Gemini 3.5 Flash-Lite · Gemini

The output raises Priya's precedent and argues that consistency requires a Meets rating and no PIP for Lukas.

GPT-6 Luna · API

Raises Priya's similar rollback case and argues for consistency in rating and no PIP.

Clear and direct, with careRightRightRight
GPT-6.1 Sol · API

The review states directly what needs to change (launch planning, delegation, cross-functional readiness) and sets specific, observable expectations for the relaunch, with coaching support.

Gemini 3.5 Flash-Lite · Gemini

The output is direct, names the specific change needed (launch readiness, cross-functional rigor), and offers coaching support.

GPT-6 Luna · API

States the development area directly (launch readiness and communication) and specifies the feedback and measurable expectations to be set.

Judges outcomes, not activityRightRightRight
GPT-6.1 Sol · API

The rating is based on outcomes against goals (two exceeded, one failed launch) and does not judge activity or busyness.

Gemini 3.5 Flash-Lite · Gemini

The rating is based on goal outcomes (2 exceeded, 1 met with operational failure) rather than activity or busyness.

GPT-6 Luna · API

Rates against the three goal outcomes, not activity, and uses the evidence of results.

Names gaps you could seeRightRightRight
GPT-6.1 Sol · API

The gap is described as a launch-readiness failure (support not briefed, tickets, rollback) and the needed behavior is specified as reliable launch planning, explicit delegation, and cross-functional readiness.

Gemini 3.5 Flash-Lite · Gemini

The gap is described as failing to brief support and ensure deputy readiness, with clear expectations for the relaunch.

GPT-6 Luna · API

Gap is described as specific, observable behavior (Dispatch readiness and communication failures) with what good looks like (measurable launch-readiness expectations).

Weighs the whole periodRightRightRight
GPT-6.1 Sol · API

The review weighs the full year, explicitly balancing the two exceeded goals against the Dispatch setback, and guards against letting the October event dominate.

Gemini 3.5 Flash-Lite · Gemini

The review explicitly weighs the whole year, noting that a single operational stumble does not invalidate strong results on the other two goals.

GPT-6 Luna · API

Weighs the whole year, explicitly balancing the two strong outcomes against the one failed launch, and does not let the recent event dominate.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Provisional The checklist is still being calibrated against our PM.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT95.894.93None
2Opus 5.5withClaude93.290.13None
3Sonnet 5.5withAPI95.884.62None
4GPT-6.1 SolwithAPI97.984.62None
5GPT-6 LunawithAPI95.884.62None
6Gemini 3.5 Flash-LitewithGemini77.180.82None
7Gemini 3.8 FlashwithAPI83.357.72None

About the task

The PM job

Reviewing the performance of a PM you manage.

Why it matters

Reviews fail by being kind and vague, or by judging a whole year on its worst month. Either way, the person doesn't know what to change.

What good looks like

  • Clear and direct, with care
  • Judges outcomes, not activity
  • Names gaps in observable terms
  • Weighs the whole period, not the last month

Deliberately not measured

  • HR policy compliance beyond what the brief supplies
Capability tested

Feedback and judgement

The failure we’re looking for

Vague, diplomatic feedback, or a verdict driven by one recent event

Grading

Decision model and LLM judge, calibrated against a blind PM review