Tasks / Leadership

Write a performance review

Can the model write a review that is clear, fair and specific enough to change what the person does next?

Measures the modelTask type v1.1 · 3 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 17 graded outputs by 7 models. 59% were usable with at most a quick edit.

Reliably right

  1. Judges outcomes, not activity100% pass
    The rating is based on outcomes against goals (two exceeded, one failed launch) rather than activity or output volume.
    GPT-6 Astra · ChatGPT · A PIP, or a bad month?
  2. Owns the manager's part100% pass
    Ana plainly acknowledges her Q3 mistake and commits to reviewing goal metrics first in every monthly 1:1.
    Opus 5.5 · Claude · Shipped a lot, moved nothing
  3. Holds the PIP to the policy and the record100% pass
    It explicitly shows that no earlier documented feedback exists on the gaps Ines named, so a PIP would violate policy, and proposes documented feedback first.
    GPT-6 Astra · ChatGPT · A PIP, or a bad month?

Where it slips

  1. Identifies material uncertainty43% pass
    Does not explicitly name the specific unknowns that could change the decision (e.g., whether Lukas improves launch readiness); only says to reassess after the relaunch.
    GPT-6 Luna · API · A PIP, or a bad month?
  2. Catches the inflated team58% pass
    The output proposes lowering Ben from Exceeds to Meets, failing to keep Ben at Exceeds as required; Ben beat both outcome goals and should remain Exceeds.
    Opus 5.5 · Claude · Fourteen ratings, three managers
  3. Notices who's underrated67% pass
    Nora is not mentioned, so the output does not notice she is underrated.
    Gemini 3.5 Flash-Lite · Gemini · Fourteen ratings, three managers

How it’s graded

The checks come from what the best product leaders have said about doing this job well on Lenny’s Podcast. Each one names the guest it comes from: follow a name to the idea on the Lenny’s Podcast wiki.

  1. Clear and direct, with care

    Does the review say plainly what needs to change, in words the person couldn't misread, while showing it is written to help them succeed?

    Passes when Every main point is stated directly, with the specific change expected and the support on offer.

  2. Judges outcomes, not activity

    Does the review judge the person on the outcomes they achieved against their goals, rather than on how much they shipped or how busy they were?

    Passes when Rates against the goals' outcomes, with the evidence, and treats output as context, not the result.

  3. Names gaps you could see

    Is each area to improve described as specific, observable behaviour with what good would look like, rather than labels like 'be more strategic'?

    Passes when Every gap names what the person did or didn't do, in a specific situation, and what doing it well would look like.

  4. Weighs the whole period

    Does the review weigh the whole review period, rather than letting one recent event or one strong impression decide the verdict?

    Passes when Covers the whole period in proportion, and explicitly guards against a recent event or one impression dominating.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're Ana Ruiz, Group PM at Plotwise. Write Theo Brandt's annual review: the review Theo will read, with his strengths, what to work on, his overall rating on our scale, and his goals for the next half. No more than 800 words. What we know is below.

What the model was given6 items: Our rating scale, Theo's goals for the year, Results, Usage of what shipped, Peer feedback, Your 1:1 notes
Our rating scaleBelow expectations, Meets expectations, Exceeds expectations, Greatly exceeds expectations. Calibration asks for written evidence for any rating other than Meets.
Theo's goals for the year1. Raise the share of teams using Timeline view each week from 22% to 30%. 2. Cut the time new teams take to build their first plan from 3 days to 1.
ResultsTimeline weekly use: 22% to 23%. Time to first plan: 3.0 days to 2.6. Theo's squad shipped 14 features, 12 of them on time.
Usage of what shippedSix of the 14 features are used by fewer than 2% of teams. Five were requests from Sales; two of those were for a single customer each.
Peer feedbackEngineering lead: 'Theo keeps the team unblocked, his specs are clear, and he never misses a sprint review.' Designer: 'Great to work with. I wish we'd tested more before building; we shipped three things I'm not sure anyone uses.' Sales: 'Very responsive: shipped five of our asks.'
Your 1:1 notesMost 1:1s covered delivery and stakeholder updates. In the Q3 1:1, Theo asked whether he was on track for his goals. Your note says: 'Told him yes, great velocity.' You never discussed that the goal metrics weren't moving.
What a strong answer doesThe answer key the graders mark against

A review that rates Theo Meets expectations (not Exceeds): his execution and collaboration are genuinely strong, but neither goal moved meaningfully (23% against 30%; 2.6 days against 1), and six of 14 features are barely used. Says so clearly and kindly. Names specific, observable things to change: test before building (the designer's point), say no to single-customer requests, and judge work by the goal metrics. Owns the manager's part: Ana told him in Q3 he was on track, so this shouldn't land as a surprise without acknowledging that, and she commits to discussing the goal metrics in 1:1s. Sets next-half goals as outcomes, with the support on offer.

Critical failures (cap the score at 40)
  • Rates Theo Exceeds expectations or higher on the strength of what he shipped
Task

Mid-level · v1.0 · synthetic · B2B SaaS, feedback, management

The outputs

7 setups answered this brief. Pick up to three to read side by side.

Needs more than a quick edit7 of 12 checks passed

Unsupported claims about squad state would require re-checking evidence; process goals need rework to be outcome-focused.

Output

Annual Performance Review Employee: Theo Brandt, Product Manager Manager: Ana Ruiz, Group Product Manager Review Period: FY Annual Review Overall Rating: Meets Expectations

---

Performance Summary & Overall Rating

Overall Rating: Meets Expectations

Theo has had a productive year characterized by high operational output, exceptional execution rigor, and strong squad leadership. He shipped 14 features—12 of them on time—and built great trust across engineering and sales.

However, our primary product goals were tied to outcome metrics rather than feature velocity: 1. Weekly Timeline View Adoption: Target 30% | Actual: 23% (baseline 22%) 2. Time to First Plan: Target 1.0 day | Actual: 2.6 days (baseline 3.0 days)

While the delivery machine Theo built is top-tier, the strategic impact of what was delivered fell short of our targets. Six of the 14 features shipped are used by fewer than 2% of teams, reflecting an emphasis on shipping volume and ad-hoc requests over high-leverage product discovery.

I also want to acknowledge my own role here: when you asked in Q3 whether you were on track, I pointed to your velocity and said yes, rather than re-anchoring our conversations on our metric shortfalls. Your execution on what was put in front of you was excellent; our focus now must pivot from output to measurable impact.

---

Strengths

  • Execution Rigor and Delivery Reliability: Theo is an exceptional delivery PM. Shipping 14 features with an 85% on-time delivery rate is a testament to his clarity and organizational discipline. As his Engineering Lead noted, “Theo keeps the team unblocked, his specs are clear, and he never misses a sprint review.” His squad runs with rhythm and confidence.
  • Stakeholder Responsiveness and Partnership: Theo is highly collaborative and service-oriented. Sales commended him as “very responsive,” appreciating that he delivered five key requests to support revenue efforts.
  • Squad Morale and Clarity: The squad feels supported and clear on day-to-day priorities. Theo shields the team from ambiguity during delivery sprints.

---

Areas for Growth

  • Shift from Output to Outcomes: High velocity without metric movement does not move Plotwise forward. Five of the shipped features were sales requests (two serving only a single customer), and nearly half the year’s releases see under 2% usage. Theo must practice saying "not now" to low-leverage requests to protect squad capacity for the core metrics we need to move.
  • Pre-Build Discovery and Validation: As our product designer noted, “I wish we’d tested more before building; we shipped three things I’m not sure anyone uses.” Theo should partner earlier with design and analytics to de-risk value and usability before committing engineering time. Building prototypes and running customer problem interviews will prevent spending sprint cycles on features that fail to achieve adoption.
  • Metric Ownership in Day-to-Day Decisions: Moving forward, every feature spec should define a clear hypothesis, adoption target, and post-launch metric check. Our 1:1s will also reflect this shift—prioritizing metric movement over sprint status.

---

Goals for the Next Half (H1)

1. Drive Timeline Weekly Adoption from 23% to 28% * Success Measure: Rather than building net-new surfaces, focus on instrumenting workflows, user onboarding, and targeted iterations on the current Timeline experience to increase recurring weekly team usage. 2. Reduce New Team Time-to-First-Plan from 2.6 Days to 1.5 Days * Success Measure: Conduct discovery sessions with recently onboarded teams, identify drop-off bottlenecks, and ship validated onboarding improvements with design. 3. Establish a Customer Discovery & Validation Gate * Success Measure: Implement a lightweight validation process for all incoming feature ideas (especially commercial requests). No feature will be queued for full development without documented user validation and an estimated TAM/adoption threshold (>5% of active teams or explicit strategic tier-1 value). 4. Post-Launch Adoption Audits * Success Measure: Conduct 30- and 60-day post-launch usage reviews for all new releases, tracking engagement against targets and sunsetting or iterating on low-performing functionality.

Claims the judge couldn’t find in the brief

  • His squad runs with rhythm and confidence.
  • Theo shields the team from ambiguity during delivery sprints.
  • The squad feels supported and clear on day-to-day priorities.

Grades and run details

Decision model 83 · LLM judge 7 of 13 checks
Decision model checks
  • passUses the supplied evidence correctly56%
  • passAddresses the actual decision73%
  • partialRespects explicit constraints17%
  • failIdentifies material uncertainty90%
  • partialAvoids unsupported claims59%
  • passProduces the required deliverable95%
  • passOwns the manager's part100%
  • passSets next goals as outcomes, with support71%
  • passClear and direct, with care96%
  • passJudges outcomes, not activity65%
  • passNames gaps you could see89%
  • passWeighs the whole period82%
Run
Run
#1
API response time
23 s
Submitted
5 Oct 2026

Needs more than a quick edit9 of 12 checks passed

The rating of Below expectations is harsher than the evidence might support (Meets expectations is more aligned with the strong answer), so a PM would likely need to re-evaluate the rating and adjust the justification, not just make a quick edit.

Output

Theo Brandt — Annual Review

Ana Ruiz, Group PM, Plotwise

Overall rating: Below expectations

Theo, you brought strong execution discipline, clear communication, and dependable partnership to the team this year. However, our two primary goals were to increase weekly Timeline use and shorten the time new teams take to build their first plan. We made only modest progress on both:

  • Weekly Timeline use: increased from 22% to 23%, against a goal of 30%.
  • Time to first plan: decreased from 3.0 days to 2.6 days, against a goal of 1 day.

Those results are the basis for this rating. Your squad shipped 14 features, 12 on time, but delivery volume and timeliness did not translate into the customer outcomes we committed to achieve. Six of those features are used by fewer than 2% of teams. Low adoption does not automatically make a feature unsuccessful, but it does require us to establish what value it created and whether it justified the investment.

I also owe you an acknowledgment of my part in this. In Q3, you explicitly asked whether you were on track, and I told you yes, citing great velocity. That was misleading: I should have distinguished delivery performance from progress against your goals. Our 1:1s focused too heavily on delivery and stakeholder updates, and I did not address the stalled metrics. You should not be hearing this assessment for the first time in an annual review. That management failure is mine; it does not erase the outcome gap, but it matters to how we understand the year and move forward.

Strengths

Reliable execution and engineering partnership. You keep the team unblocked and provide clear specs. The engineering lead specifically highlighted both, along with your consistent participation in sprint reviews. Shipping 12 of 14 features on time reflects real delivery discipline.

Strong collaboration and responsiveness. Design describes you as great to work with, and Sales values your responsiveness. You create productive working relationships and follow through on commitments. Those strengths are important foundations for effective product leadership.

What to work on

Shift from managing delivery to managing outcomes. The roadmap needs a clearer connection to Timeline adoption and first-plan speed. For each meaningful investment, define the customer problem, expected metric movement, and how we will assess success after launch. When the metrics are not moving, bring that into our discussions early and recommend a change in approach.

Validate demand before committing engineering capacity. Design’s concern that we should have tested more, together with the low usage of several shipped features, points to a need for stronger discovery. Use prototypes, customer conversations, and lightweight experiments to test the riskiest assumptions before building.

Make prioritization trade-offs explicit. Five shipped features were Sales requests, including two for individual customers. Those requests may have legitimate business value, but responsiveness alone is not a sufficient reason to prioritize them. Assess their customer reach, strategic or commercial value, cost, and opportunity cost—and make clear when an exception is worth taking.

Goals for the next half

  1. Increase weekly Timeline use from 23% to 27%. Identify the main adoption barriers, test proposed solutions, and prioritize the interventions with the strongest evidence.
  2. Reduce time to first plan from 2.6 days to 1.8 days. Diagnose where new teams stall and measure whether each onboarding improvement reduces that friction.
  3. Strengthen investment decisions. For every substantial roadmap item, document the problem, supporting evidence, intended outcome, and success measure before committing to build. Review results within six weeks of launch and decide whether to iterate, expand, or stop.

We will confirm metric definitions and tracking at the start of the half, then review progress monthly. I will give you explicit feedback on whether you are on track, help resolve prioritization conflicts, and avoid treating velocity as a substitute for impact.

You have a strong execution foundation. The next step is to apply that strength to fewer, better-supported bets that measurably improve customers’ experience.

Grades and run details

Decision model 96 · LLM judge 9 of 13 checks
Decision model checks
  • passUses the supplied evidence correctly78%
  • passAddresses the actual decision49%
  • passRespects explicit constraints3%
  • partialIdentifies material uncertainty56%
  • passAvoids unsupported claims47%
  • passProduces the required deliverable80%
  • passOwns the manager's part100%
  • passSets next goals as outcomes, with support99%
  • passClear and direct, with care93%
  • passJudges outcomes, not activity99%
  • passNames gaps you could see92%
  • passWeighs the whole period80%
Run
Run
#1
API response time
31 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 2

Uses the supplied evidence correctlyMixedRight
Gemini 3.8 Flash · API

Claims about squad rhythm, shielding from ambiguity, and squad feelings are not in the brief and are presented as facts.

GPT-6.1 Sol · API

All statements about the current situation are directly from the brief or supplied context.

Avoids unsupported claimsWrongRight
Gemini 3.8 Flash · API

Presents interpretations (squad rhythm, feeling supported) as established facts without labelling them as hypotheses.

GPT-6.1 Sol · API

Interpretations are presented as suggestions or conclusions from the evidence, not as unsupported facts.

All got wrong 1

Identifies material uncertaintyWrongWrong
Gemini 3.8 Flash · API

No unknowns that could change the rating are identified.

GPT-6.1 Sol · API

The output does not name any unknowns that could change the decision or how they would be resolved.

All mixed 2

Addresses the actual decisionMixedMixed
Gemini 3.8 Flash · API

Does not state what result or condition would change the rating.

GPT-6.1 Sol · API

The output does not explicitly state what result or condition would change the rating.

Sets next goals as outcomes, with supportMixedMixed
Gemini 3.8 Flash · API

Goals 3 and 4 are process changes, not outcome goals with baselines and targets; specific support from Ana is not tied to each goal.

GPT-6.1 Sol · API

The third goal ('Strengthen investment decisions') is a process goal without a baseline or target, not an outcome goal.

All got right 7

Respects explicit constraintsRightRight
Gemini 3.8 Flash · API

Within 800 words, correct form, addresses Theo with required sections.

GPT-6.1 Sol · API

The review is under 800 words, addresses Theo, and includes all required sections.

Produces the required deliverableRightRight
Gemini 3.8 Flash · API

Complete review with rating, strengths, areas to improve, and goals; usable as is.

GPT-6.1 Sol · API

The review includes strengths, areas to work on, an overall rating, and next-half goals, and is usable as is.

Owns the manager's partRightRight
Gemini 3.8 Flash · API

Acknowledges Ana's Q3 response and commits to shifting 1:1s to metric focus.

GPT-6.1 Sol · API

Ana acknowledges telling Theo he was on track in Q3, admits the management failure, and commits to giving explicit feedback and avoiding velocity as a substitute for impact.

Clear and direct, with careRightRight
Gemini 3.8 Flash · API

Directly states what needs to change (say no, test before building, metric ownership) with care.

GPT-6.1 Sol · API

The review states plainly what needs to change, with specific behaviors and the support on offer.

Judges outcomes, not activityRightRight
Gemini 3.8 Flash · API

Rates against goal outcomes (23% vs 30%, 2.6 vs 1.0 days), not feature count.

GPT-6.1 Sol · API

The rating is based on the outcome metrics against the goals, not on the number of features shipped.

Names gaps you could seeRightRight
Gemini 3.8 Flash · API

Gaps are described as specific behaviors (saying 'not now', partnering earlier, defining hypotheses).

GPT-6.1 Sol · API

Each area to improve describes specific, observable behaviors (e.g., bring stalled metrics into discussions, use prototypes to test assumptions, assess customer reach and opportunity cost).

Weighs the whole periodRightRight
Gemini 3.8 Flash · API

Covers full year, references Q3 conversation without letting it dominate.

GPT-6.1 Sol · API

The review references the full year, including Q3 1:1 and the entire set of shipped features, without letting one event dominate.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Provisional The checklist is still being calibrated against our PM.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT95.894.93None
2Opus 5.5withClaude93.290.13None
3Sonnet 5.5withAPI95.884.62None
4GPT-6.1 SolwithAPI97.984.62None
5GPT-6 LunawithAPI95.884.62None
6Gemini 3.8 FlashwithAPI83.357.72None
7Gemini 3.5 Flash-LitewithGemini73.258.63None

About the task

The PM job

Reviewing the performance of a PM you manage.

Why it matters

Reviews fail by being kind and vague, or by judging a whole year on its worst month. Either way, the person doesn't know what to change.

What good looks like

  • Clear and direct, with care
  • Judges outcomes, not activity
  • Names gaps in observable terms
  • Weighs the whole period, not the last month

Deliberately not measured

  • HR policy compliance beyond what the brief supplies
Capability tested

Feedback and judgement

The failure we’re looking for

Vague, diplomatic feedback, or a verdict driven by one recent event

Grading

Decision model and LLM judge, calibrated against a blind PM review