Tasks / Leadership

Write a performance review

Can the model write a review that is clear, fair and specific enough to change what the person does next?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 56% were usable with at most a quick edit.

Reliably right

  1. Judges outcomes, not activity100% pass
    Every rating change and keep is justified by outcomes against goals, with shipping and activity treated as context, not the result.
    GPT-6 Astra · ChatGPT · Fourteen ratings, three managers
  2. Owns the manager's part100% pass
    Ana explicitly owns that she said he was on track in Q3 without discussing the metrics, and commits to reviewing goal metrics in future 1:1s.
    Gemini 4 Argon · API · Shipped a lot, moved nothing
  3. Respects explicit constraints97% pass
    The output is a pre-read under 1,100 words, includes the ratings to change and keep with evidence, and a note to Ravi, respecting all explicit constraints.
    GPT-6 Astra · ChatGPT · Fourteen ratings, three managers

Where it slips

  1. Sets next goals as outcomes, with support47% pass
    Goals are partly process-oriented (implement a mandatory process, create a framework) and lack specific support from Ana beyond a general offer.
    Gemini 3.5 Flash-Lite · Gemini · Shipped a lot, moved nothing
  2. Identifies material uncertainty51% pass
    The output does not name any specific unknowns that could change the rating or how they would be resolved.
    Opus 5.5 · Claude · Shipped a lot, moved nothing
  3. Catches the inflated team69% pass
    The output proposes lowering Ben from Exceeds to Meets, failing to keep Ben at Exceeds as required; Ben beat both outcome goals and should remain Exceeds.
    Opus 5.5 · Claude · Fourteen ratings, three managers

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Clear and direct, with care

    Does the review say plainly what needs to change, in words the person couldn't misread, while showing it is written to help them succeed?

    Passes when Every main point is stated directly, with the specific change expected and the support on offer.

  2. Judges outcomes, not activity

    Does the review judge the person on the outcomes they achieved against their goals, rather than on how much they shipped or how busy they were?

    Passes when Rates against the goals' outcomes, with the evidence, and treats output as context, not the result.

  3. Names gaps you could see

    Is each area to improve described as specific, observable behaviour with what good would look like, rather than labels like 'be more strategic'?

    Passes when Every gap names what the person did or didn't do, in a specific situation, and what doing it well would look like.

  4. Weighs the whole period

    Does the review weigh the whole review period, rather than letting one recent event or one strong impression decide the verdict?

    Passes when Covers the whole period in proportion, and explicitly guards against a recent event or one impression dominating.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're Ana Ruiz, Group PM at Plotwise. Write Theo Brandt's annual review: the review Theo will read, with his strengths, what to work on, his overall rating on our scale, and his goals for the next half. No more than 800 words. What we know is below.

What the model was given6 items: Our rating scale, Theo's goals for the year, Results, Usage of what shipped, Peer feedback, Your 1:1 notes
Our rating scaleBelow expectations, Meets expectations, Exceeds expectations, Greatly exceeds expectations. Calibration asks for written evidence for any rating other than Meets.
Theo's goals for the year1. Raise the share of teams using Timeline view each week from 22% to 30%. 2. Cut the time new teams take to build their first plan from 3 days to 1.
ResultsTimeline weekly use: 22% to 23%. Time to first plan: 3.0 days to 2.6. Theo's squad shipped 14 features, 12 of them on time.
Usage of what shippedSix of the 14 features are used by fewer than 2% of teams. Five were requests from Sales; two of those were for a single customer each.
Peer feedbackEngineering lead: 'Theo keeps the team unblocked, his specs are clear, and he never misses a sprint review.' Designer: 'Great to work with. I wish we'd tested more before building; we shipped three things I'm not sure anyone uses.' Sales: 'Very responsive: shipped five of our asks.'
Your 1:1 notesMost 1:1s covered delivery and stakeholder updates. In the Q3 1:1, Theo asked whether he was on track for his goals. Your note says: 'Told him yes, great velocity.' You never discussed that the goal metrics weren't moving.
What a strong answer doesThe answer key the graders mark against

A review that rates Theo Meets expectations (not Exceeds): his execution and collaboration are genuinely strong, but neither goal moved meaningfully (23% against 30%; 2.6 days against 1), and six of 14 features are barely used. Says so clearly and kindly. Names specific, observable things to change: test before building (the designer's point), say no to single-customer requests, and judge work by the goal metrics. Owns the manager's part: Ana told him in Q3 he was on track, so this shouldn't land as a surprise without acknowledging that, and she commits to discussing the goal metrics in 1:1s. Sets next-half goals as outcomes, with the support on offer.

Critical failures (cap the score at 40)
  • Rates Theo Exceeds expectations or higher on the strength of what he shipped
Task

Mid-level · v1.1

The outputs

9 setups answered this brief.

Usable with a quick edit8 of 12 checks passed

The unsupported adoption plan line can be removed with a quick edit, leaving an otherwise evidence-based, actionable review.

Output

Annual Review: Theo Brandt

Reviewer: Ana Ruiz, Group PM Period: Full year

Summary

Theo, you run a reliable delivery team. Your squad shipped 14 features, 12 on time, and engineering and sales both value working with you. This year's problem is that shipping didn't move the two goals we set. Timeline weekly use went from 22% to 23%, against a target of 30%. Time to first plan dropped from 3.0 days to 2.6, against a target of 1 day. We measured the work by output, and the outcomes didn't follow.

Results Against Goals

Goal 1: Raise weekly Timeline use from 22% to 30%. Result: 23%. Missed by 7 points. Most of the year's shipping was not aimed at this metric, and the adoption plan never got the attention it needed.

Goal 2: Cut time to first plan from 3 days to 1. Result: 2.6 days. Real progress, a 13% improvement, but well short of target. Some of the gain likely came from onboarding changes that were not in the roadmap.

Delivery: 12 of 14 features shipped on time. This is strong and I want to be clear about it.

Adoption of what shipped: Six of the 14 features are used by fewer than 2% of teams. Five of those six came from Sales requests, and two were built for a single customer. Your designer flagged three shipped features that nobody seems to use. I think they're right.

Strengths

  • Reliability. You keep the team unblocked and hit sprint reviews. Engineering trusts your planning.
  • Clear specs. Engineering's feedback on your specs is consistent and specific.
  • Stakeholder responsiveness. Sales knows you take their requests seriously and act on them.
  • Communication with the team. The designer describes you as good to work with, which matters in a squad where tradeoffs get hard.

What to Work On

Validate before you build. The designer's point is the most important feedback in this review. Three shipped features have no evident users, and you had the means to check before building them. Before you commit to a feature, define the usage metric it has to move and run a cheap test to check the need. If you can't name the metric, don't build it yet.

Connect requests to goals. Sales requests are valuable, but five of your last shipped features came from one channel, and two were for single customers. Treat a request as a hypothesis about usage, not a commitment. Ask how many teams the request affects and how it connects to goal 1 or goal 2. Say no, or propose a different version, when the answer is weak.

Treat goals as the status report. Shipping on time is a good sign, but it doesn't tell us whether goals are on track. Report the goal metric first in every update, even when the number is bad.

Rating

Below Expectations.

This rating is based on the two goals you were given. Both moved in the right direction, but neither reached its target, and the delivered features did little to move them. Calibration requires written evidence for this rating, and the evidence is above: the goal results, the usage data on shipped features, and the designer's observation about untested builds.

Your delivery and team relationships are real strengths and are not part of the rating decision. The rating reflects impact on the goals. I'm also not ignoring that the gap was harder to see than it should have been. I'll address my part in the next section.

Where I Fell Short

In the Q3 1:1, you asked directly whether you were on track. I said yes and pointed to your velocity. I never looked at whether the goal metrics were moving, and they weren't. That was my failure, not yours. You asked the right question, and I gave you a reassuring answer based on the wrong evidence. If I had given you the real numbers in Q3, you would have had a full quarter to change course. I'm sorry for that. I'll change how I run goal check-ins, starting with the next half.

Goals for the Next Half

1. Timeline weekly use: 23% to 28%. Within the first six weeks, interview 10 teams that tried Timeline and stopped using it, and bring me a written summary of why. Then propose and run one adoption experiment, with a success threshold set before launch. I'll review the findings with you at the six-week mark.

2. Time to first plan: 2.6 days to 1.5 days. Identify the two or three steps in onboarding that take the most time for new teams, and ship changes against those steps. Report the number every two weeks.

3. Usage gate on all new features. Every feature in the roadmap must have a named usage metric and a pre-build validation step, agreed with the designer. Features without both don't enter the build queue. We'll review the first three together.

4. Sales request triage. Set up a simple process with Sales. Requests are scored by how many teams they affect and how they tie to the goals. Single-customer requests need a specific justification or a different solution. I'll help with the first conversation with the Sales lead.

Theo, you have the delivery discipline to do this. What you need is to point that discipline at the outcomes we set out to change. Let's start the first six weeks with the Timeline interviews.

Claims the judge couldn’t find in the brief

  • the adoption plan never got the attention it needed.
  • Most of the year's shipping was not aimed at this metric.

Grades and run details

Decision model 71 · LLM judge 77
Decision model checks
  • failUses the supplied evidence correctly62%
  • partialAddresses the actual decision56%
  • passRespects explicit constraints14%
  • partialIdentifies material uncertainty23%
  • partialAvoids unsupported claims51%
  • passProduces the required deliverable76%
  • passOwns the manager's part100%
  • failSets next goals as outcomes, with support31%
  • passClear and direct, with care98%
  • passJudges outcomes, not activity92%
  • passNames gaps you could see95%
  • passWeighs the whole period86%
Run
Run
#1
API response time
16 s
Submitted
8 Oct 2026

Usable with a quick edit10 of 12 checks passed

The review is well-structured and evidence-based; a quick edit to add specific support for goals would make it fully ready.

Output

To: Theo Brandt From: Ana Ruiz, Group PM Date: Annual Review Rating: Meets expectations

*

Hi Theo,

It’s time for your annual review. First, I want to acknowledge your hard work and dedication over the past year. You are an exceptionally reliable Product Manager, and your squad’s execution has been rock-solid.

Strengths

Your delivery engine is second to none. As your engineering lead noted, your specs are clear, you keep the team unblocked, and you never miss a sprint review. Shifting 14 features in a year—with 12 delivered strictly on time—is an impressive output that demonstrates strong project management and technical collaboration.

You are also a fantastic cross-functional partner. Sales loves working with you because you are responsive and action-oriented, and our design lead appreciates your steady communication and positive team dynamic. You cultivate a smooth, friction-free environment for your squad, which is a rare and valuable skill.

Areas for Improvement

While your output and velocity are stellar, we need to evolve how we measure success. Our primary focus this past year was on driving core product metrics, and we fell short of our targets: * Timeline View Adoption: Moved only from 22% to 23% (against a target of 30%). * Time to First Plan: Improved marginally from 3.0 days to 2.6 days (against a target of 1.0).

Additionally, six of the 14 features we shipped are currently used by fewer than 2% of teams. Five of these were direct requests from Sales, including two built for single customers.

As a PM, your role isn't just to say "yes" to stakeholders and push code across the finish line; it’s to interrogate the why, protect our roadmap, and obsess over the impact of what we build. Relying too heavily on outbound feature requests without validating their broader utility led us to invest engineering cycles in low-impact work.

I also want to own my part in this: looking back at our 1:1 notes, I praised your high velocity when you asked if you were on track, and I failed to course-correct early enough regarding the stagnant metric goals. Moving forward, we will be much sharper about tying your weekly execution directly to our key results.

Overall Rating

Meets expectations.

Your execution, reliability, and stakeholder management are top-tier. To reach the next level, you need to shift your mindset from output (features shipped on time) to outcomes (metrics moved and customer value realized).

*

Goals for the Next Half

To help bridge that gap, your goals for the next H are focused on strategic impact and validation:

  1. Improve Core Activation Metrics: Reduce the time new teams take to build their first plan from 2.6 days to 1.5 days, focusing on self-serve friction reduction rather than custom feature additions.
  2. Increase Feature ROI & Validation: Implement a mandatory pre-build discovery and validation process for all major initiatives. Ensure that 100% of shipped features have a defined success metric and achieve at least 15% adoption within 30 days of launch.
  3. Stakeholder Alignment Strategy: Partner with Sales leadership to create a structured framework for evaluating custom requests, ensuring we prioritize scalable product improvements over single-customer concessions without sacrificing our partnership.

Thank you for everything you do for Plotwise, Theo. I’m looking forward to supporting you as you tackle these new challenges in the year ahead.

Best, Ana

Grades and run details

Decision model 79 · LLM judge 85
Decision model checks
  • passUses the supplied evidence correctly28%
  • partialAddresses the actual decision51%
  • passRespects explicit constraints23%
  • failIdentifies material uncertainty98%
  • partialAvoids unsupported claims56%
  • passProduces the required deliverable92%
  • passOwns the manager's part92%
  • partialSets next goals as outcomes, with support35%
  • passClear and direct, with care89%
  • passJudges outcomes, not activity45%
  • passNames gaps you could see37%
  • passWeighs the whole period74%
Run
Run
#1
Time to output
5 s
Submitted
5 Oct 2026

Usable with a quick edit10 of 12 checks passed

The review is complete, factually accurate, contains no invented evidence, and meets all requirements; a PM could drop it into the template with only light wording tweaks.

Output

Annual Performance Review Employee: Theo Brandt, Product Manager Manager: Ana Ruiz, Group Product Manager Overall Rating: Meets expectations

Summary Theo, thank you for your hard work and dedication over the past year. You have established yourself as an exceptionally reliable execution partner and a trusted collaborator across Engineering, Design, and Sales. Your squad’s delivery velocity was impressive, shipping 14 features with 12 on time.

However, that high output did not translate into the outcome targets we set for the year: weekly Timeline usage moved from 22% to 23% (goal: 30%), and time to first plan moved from 3.0 days to 2.6 days (goal: 1.0 day).

I want to own my part in this disconnect. In our 1:1s—and specifically when you asked if you were on track in Q3—I focused my feedback on your strong delivery velocity and stakeholder responsiveness rather than holding us both accountable to the lagging core metrics. Because you delivered on the execution behaviors we discussed all year, your rating for this cycle is Meets expectations. Moving forward, we will work together to shift your focus from shipping volume to driving measurable user outcomes.

Strengths * Execution and Delivery: You run a remarkably tight ship. Delivering 14 features (12 on schedule) is a testament to your organizational skills. Your Engineering Lead noted that your specs are clear, you never miss a sprint review, and you consistently keep the team unblocked. * Cross-Functional Collaboration: Your peers genuinely enjoy working with you. Both Design and Engineering highlighted your collaborative nature, and Sales praised your responsiveness in shipping five of their requests this year. You build strong, frictionless relationships across the org.

Areas for Growth * Outcome vs. Output Orientation: While your squad shipped a high volume of features, six of the 14 are currently used by fewer than 2% of teams. Moving forward, shipping a feature cannot be the finish line; we must measure success by adoption and impact on our core product metrics. * Discovery and Validation: In our push to maintain high velocity, we bypassed critical validation steps. As your Design partner noted, we built and shipped several features without sufficient upfront testing to prove user demand. We need to lean harder on discovery before committing engineering resources. * Strategic Prioritization: Five of the features you shipped came directly from Sales requests, including two built for a single customer each. While supporting Sales is important, saying "yes" to bespoke requests diluted your squad’s focus and pulled capacity away from our broader Timeline and onboarding goals. You need to push back on one-off asks that don’t serve our wider user base.

Goals for Next Half (H1) This half, we are shifting our focus from feature velocity to validated impact. In our 1:1s, we will review metric movement and discovery insights rather than just delivery status.

  1. Drive Timeline Adoption: Increase the share of teams using Timeline view weekly from 23% to 28%.
  2. Accelerate Onboarding: Reduce the time new teams take to build their first plan from 2.6 days to 1.5 days.
  3. Establish a Rigorous Discovery Practice: Before any new feature enters sprint planning, partner with Design to validate demand and usability with at least 5 target customers, and define a clear post-launch adoption target. Ensure 80% of shipped work meets its 30-day adoption target.
  4. Protect Roadmap Focus: Audit incoming Sales requests against our core H1 goals. Zero features should be built for a single customer unless explicitly approved at the leadership level, ensuring at least 80% of engineering capacity is dedicated to our primary metric drivers.

Grades and run details

Decision model 75 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly25%
  • partialAddresses the actual decision66%
  • passRespects explicit constraints53%
  • failIdentifies material uncertainty97%
  • partialAvoids unsupported claims38%
  • passProduces the required deliverable98%
  • passOwns the manager's part100%
  • failSets next goals as outcomes, with support46%
  • passClear and direct, with care100%
  • passJudges outcomes, not activity74%
  • passNames gaps you could see43%
  • passWeighs the whole period88%
Run
Run
#1
API response time
11 s
Submitted
13 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Uses the supplied evidence correctlyWrongRightRight
Haiku 5.5 · API

The output claims there was an 'adoption plan' that didn't get attention, which is not mentioned in the supplied context, and it presents an unsupported interpretation that most shipping wasn't aimed at the metric.

Gemini 3.5 Flash-Lite · Gemini

All factual statements about the current situation are correctly taken from the brief and supplied context.

Gemini 4 Argon · API

All factual statements about the current situation come directly from the supplied context and require no invention.

Identifies material uncertaintyWrongWrongMixed
Haiku 5.5 · API

Does not name any specific unknowns that could change the rating or how they would be resolved.

Gemini 3.5 Flash-Lite · Gemini

Does not name any unknowns that could change the rating or how they would be resolved.

Gemini 4 Argon · API

The review deals only with known outcomes and facts; there are no material uncertainties that could change the rating, so the criterion is not applicable.

Avoids unsupported claimsWrongRightRight
Haiku 5.5 · API

Presents interpretations (shipping not aimed at metric, adoption plan attention) as fact without labeling them as hypotheses.

Gemini 3.5 Flash-Lite · Gemini

The causal claim is supported by the evidence of low usage, Sales requests, and designer's feedback about testing.

Gemini 4 Argon · API

Every interpretation (e.g., bypassed validation, frictionless relationships) is directly traceable to supplied evidence or labeled as the manager's observation.

Sets next goals as outcomes, with supportMixedWrongMixed
Haiku 5.5 · API

Sets outcome goals with baselines (23% to 28%, 2.6 to 1.5 days) and specific support (six-week review, help with Sales conversation, first three features together).

Gemini 3.5 Flash-Lite · Gemini

Goals are partly process-oriented (implement a mandatory process, create a framework) and lack specific support from Ana beyond a general offer.

Gemini 4 Argon · API

Goals for H1 are outcomes with baselines and targets (e.g., 23% to 28%, 2.6 to 1.5 days), with support described as reviewing metrics and insights in 1:1s.

All got right 8

Addresses the actual decisionRightRightRight
Haiku 5.5 · API

Commits unambiguously to Below Expectations, early in the summary, with clear rating and evidence.

Gemini 3.5 Flash-Lite · Gemini

Commits clearly to a rating of Meets expectations, with strengths and areas for improvement.

Gemini 4 Argon · API

The output commits clearly to a Meets expectations rating early in the review, with evidence; it's unambiguous and directly answers the brief.

Respects explicit constraintsRightRightRight
Haiku 5.5 · API

Delivers the review with strengths, what to work on, rating on the scale, goals for next half, and is under 800 words.

Gemini 3.5 Flash-Lite · Gemini

The review is under 800 words, includes all required elements, and is addressed to Theo.

Gemini 4 Argon · API

The output stays under 800 words, uses the correct rating scale, and provides all requested sections (strengths, areas to improve, rating, next-half goals).

Produces the required deliverableRightRightRight
Haiku 5.5 · API

Complete annual review for Theo with all required sections, usable as is.

Gemini 3.5 Flash-Lite · Gemini

The output is a complete annual review with strengths, areas to improve, rating, and goals, within the word limit.

Gemini 4 Argon · API

The review is complete, addressed to Theo, includes all required components, and a PM could use it with light edits.

Owns the manager's partRightRightRight
Haiku 5.5 · API

Explicitly acknowledges Ana's Q3 mistake and commits to changing how she runs goal check-ins.

Gemini 3.5 Flash-Lite · Gemini

Acknowledges that Ana praised velocity and didn't course-correct, and commits to tying execution to key results.

Gemini 4 Argon · API

Ana explicitly owns that she said he was on track in Q3 without discussing the metrics, and commits to reviewing goal metrics in future 1:1s.

Clear and direct, with careRightRightRight
Haiku 5.5 · API

Plain and direct, names specific changes (validate, connect requests to goals, report goal metric first) and shows care.

Gemini 3.5 Flash-Lite · Gemini

Directly states the need to shift from output to outcomes, validate before building, and not just say yes to Sales.

Gemini 4 Argon · API

Every gap is stated directly (e.g., push back on one-off asks, test before building) with specific changes and support, all within a caring, collaborative tone.

Judges outcomes, not activityRightRightRight
Haiku 5.5 · API

Rates Theo Below Expectations based on the goal outcomes not being met, not on feature output.

Gemini 3.5 Flash-Lite · Gemini

Rating is based on goal metrics not moving and low feature usage, not on features shipped.

Gemini 4 Argon · API

The rating is based on the lack of goal achievement and low adoption, not on feature velocity or busyness.

Names gaps you could seeRightRightRight
Haiku 5.5 · API

Gaps are tied to specific behaviours (building without testing, not filtering single-customer requests, not reporting goal metrics) with what good looks like.

Gemini 3.5 Flash-Lite · Gemini

Gaps are described as relying on outbound requests without validation, and what good looks like is interrogating the why and obsessing over impact.

Gemini 4 Argon · API

Gaps are described as specific behaviors (e.g., six low-usage features, selling five Sales asks, shipping without testing) with what good looks like.

Weighs the whole periodRightRightRight
Haiku 5.5 · API

Covers the full year, references Q3 1:1, and doesn't let a single event dominate.

Gemini 3.5 Flash-Lite · Gemini

Covers the full year, references Q3 1:1, and doesn't let one event dominate.

Gemini 4 Argon · API

The review weighs the full year's results, mentions the Q3 1:1 without letting it dominate, and bases the rating on overall goal movement.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT91.892.32None
2Gemini 4 ArgonwithAPI83.792.92None
3GPT-6.1 SolwithAPI91.884.62None
4Haiku 5.5withAPI83.588.52None
5Opus 5.5withClaude84.085.22None
6Gemini 3.8 FlashwithAPI83.776.92None
7GPT-6 LunawithAPI86.177.72None
8Sonnet 5.5withAPI80.074.22None
9Gemini 3.5 Flash-LitewithGemini62.749.52None

What you get for the cost

Haiku 5.5 · API scores within 3 points of GPT-6 Astra · ChatGPT at about a fortieth of the cost.

Task score against relative cost for each setup. The table below has the same figures.50607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 90, 70× the cheapestGemini 4 Argon · API: task score 89, 11× the cheapestGPT-6.1 Sol · API: task score 87, 13× the cheapestHaiku 5.5 · API: task score 87, 1.7× the cheapestOpus 5.5 · Claude: task score 83, 61× the cheapestGemini 3.8 Flash · API: task score 79, 12× the cheapestGPT-6 Luna · API: task score 79, the cheapestSonnet 5.5 · API: task score 76, 31× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 56, 1.9× the cheapest (1 of 2 outputs estimated)GPT-6 Astra · ChatGPTGemini 4 Argon · APIGPT-6.1 Sol · APIHaiku 5.5 · APIOpus 5.5 · ClaudeGemini 3.8 Flash · APIGPT-6 Luna · APISonnet 5.5 · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API791.0×
Haiku 5.5 · API871.7×
Gemini 3.5 Flash-Lite · Gemini561.9× (partly estimated)
Gemini 4 Argon · API8911×
Gemini 3.8 Flash · API7912×
GPT-6.1 Sol · API8713×
Sonnet 5.5 · API7631×
Opus 5.5 · Claude8361×
GPT-6 Astra · ChatGPT9070×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Reviewing the performance of a PM you manage.

Why it matters

Reviews fail by being kind and vague, or by judging a whole year on its worst month. Either way, the person doesn't know what to change.

What good looks like

  • Clear and direct, with care
  • Judges outcomes, not activity
  • Names gaps in observable terms
  • Weighs the whole period, not the last month

Deliberately not measured

  • HR policy compliance beyond what the brief supplies
Capability tested

Feedback and judgement

The failure we’re looking for

Vague, diplomatic feedback, or a verdict driven by one recent event

Grading

Decision model and LLM judge, calibrated against a blind PM review