Tasks / Leadership

Hire a PM

Can the model design a hiring process that finds the right PM, and make the call on real candidates from the evidence?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 63% were usable with at most a quick edit.

Reliably right

Produces the required deliverable100% pass
The output provides a complete loop, scorecard, and decision plan in the requested form, under 900 words, usable by Amara with light edits.
GPT-6.1 Sol · API · A loop for the first growth PM
Tests what the last hire failed at100% pass
The loop includes an influence simulation with the Head of Sales and a scorecard dimension on influence without authority, directly testing what the last hire failed at.
GPT-6.1 Sol · API · A loop for the first growth PM
Finds which rounds line up with performance100% pass
It compares each round's scores with 12-month ratings, finding behavioral lines up and take-home, presentation, VP Engineering do not.
GPT-6.1 Sol · API · What our interviews predict

Where it slips

Avoids unsupported claims56% pass
The claim 'A growth PM in a senior role is likely to be exactly that kind of person' is presented as fact without evidence and is not labelled as an assumption.
Opus 5.5 · Claude · A loop for the first growth PM
Identifies material uncertainty58% pass
The output does not name unknowns that could change the hiring decision or the loop design, nor does it say how they would be resolved.
Sonnet 5.5 · API · A loop for the first growth PM
Defines good for this role first63% pass
The memo does not explicitly name the competencies that matter most for this PM role and level, nor define what strong and weak look like before designing the interviews.
Sonnet 5.5 · API · What our interviews predict

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Judges on evidence, not presence
Does the output judge candidates, or design the process to judge them, on specific evidence of results they drove and how they worked with their team, rather than impressions such as presence, confidence or polish?
Passes when Asks for or weighs concrete evidence: the metric a candidate moved, the part they played, how they shared credit, what they learned from a failure. Impressions are treated as weak signals.
Product and growth leadersOn Lenny’s Podcast
Defines good for this role first
Does the output set out what good looks like for this specific role (the competencies that matter most at this level, and what strong and weak look like) before judging candidates or designing interviews?
Passes when Names the competencies weighted for this role and level, with strong and weak signals, and the judgement or the process follows from them.
Petra WilleLenny’s Podcast
Keeps each judgement independent
Does the output keep each interviewer's judgement independent (written down before the group discusses it), and guard against the loudest voice or the first speaker deciding?
Passes when Requires or relies on written, independent evaluations before discussion, and discounts views that changed under group pressure.
Product and growth leadersOn Lenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 3 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're Director of Product at Kestrel. Elif Demir, our VP Product, wants to fix the PM interview loop before we hire four more PMs next quarter (her note is below). Using the data we have, write her a memo of no more than 1,000 words: which parts of the loop seem to predict how PMs do once hired and which don't, how confident we can be, and the loop you'd run next quarter.\n\nThe files are attached.

What the model was given3 items: elif_note.md, current_loop.md, pm_candidates_last_18_months.csv (every PM candidate who reached the final rounds)

elif_note.md1 lines · Download

From Elif (VP Product): "We've made three PM hires in 18 months who struggled. I want to know what in our loop is and isn't working, using the data we have, and what to change before we hire four more PMs next quarter."

current_loop.md10 lines · Download

# Current PM interview loop

1. Recruiter screen.
2. Take-home: a product strategy deck, one week to complete.
3. Presentation of the take-home to a panel.
4. Behavioural round: "Tell me about a metric you moved", with follow-ups on what you did and what you learned.
5. VP Engineering conversation ("technical credibility").
…

pm_candidates_last_18_months.csv (every PM candidate who reached the final rounds)candidate,background,takehome_score,presentation_score,behavioural_metric_round_score,vp_engineering_score,outcome,rating_after_12_months C100,Big tech,4,4,3,2,Withdrew, C101,Startup,3,2,4,2,Hired,5 C102,Big tech,1,2,3,4,Rejected, C103,Agency,1,3,2,3,Rejected, C104,Big tech,4,3,3,4,Hired,3 C105,Startup,4,2,1,1,Rejected, C106,Startup,4,2,1,3,Rejected, C107,Big tech,4,4,2,4,Hired,2 C108,Consulting,1,4,2,1,Rejected, C109,Agency,4,2,1,2,Rejected, C110,Consulting,3,1,2,2,Rejected, C111,Big tech,2,2,1,4,Rejected, C112,Agency,3,1,2,1,Rejected, C113,Startup,4,2,3,1,Withdrew, C114,Big tech,2,2,3,2,Withdrew, C115,Consulting,3,1,1,1,Rejected, C116,Startup,2,2,1,1,Rejected, C117,Startup,2,1,4,3,Withdrew, C118,Startup,2,2,1,2,Withdrew, C119,Consulting,1,1,3,3,Rejected, C120,Consulting,2,4,1,3,Rejected, C121,Consulting,2,3,4,1,Rejected, C122,Agency,4,1,3,3,Withdrew, C123,Agency,2,4,4,1,Rejected, C124,Big tech,2,2,2,3,Rejected, C125,Startup,2,2,1,3,Rejected, C126,Consulting,4,1,2,2,Rejected, C127,Consulting,1,3,3,1,Rejected, C128,Startup,3,3,2,3,Rejected, C129,Agency,3,3,3,1,Rejected, C130,Startup,1,3,2,2,Rejected, C131,Agency,1,2,4,2,Rejected, C132,Big tech,3,4,2,4,Hired,2 C133,Agency,2,4,2,1,Rejected, C134,Consulting,4,4,3,3,Hired,3 C135,Consulting,3,4,4,3,Rejected, C136,Consulting,2,2,3,2,Rejected, C137,Consulting,3,4,3,1,Withdrew, C138,Startup,2,4,4,2,Withdrew, C139,Startup,3,2,2,3,Rejected, C140,Startup,1,1,3,3,Rejected, C141,Startup,3,3,4,3,Hired,5 C142,Big tech,4,2,3,4,Withdrew, C143,Consulting,1,4,3,2,Rejected, C144,Consulting,1,1,1,2,Rejected, C145,Agency,4,1,2,1,Rejected, C146,Startup,3,1,4,1,Rejected, C147,Big tech,4,1,1,4,Withdrew, C148,Big tech,4,4,2,4,Hired,2 C149,Agency,2,3,4,2,Hired,4 C150,Consulting,4,1,1,2,Rejected, C151,Startup,2,3,3,3,Hired,4

What a strong answer doesThe answer key the graders mark against

Uses the nine hires to compare each round with 12-month ratings: the behavioural 'metric you moved' round lines up strongly with how hires performed (higher scores, better ratings), while the take-home, the presentation and the VP Engineering conversation line up the wrong way: the hires they scored highest are the ones rated 2. The VP Engineering scores big-tech candidates higher, and all four big-tech hires were rated 2 or 3. Is explicit that nine hires is far too few to conclude much: the pattern is a reason to change and then measure, not proof. Proposes a loop with a scorecard defined up front, the behavioural round strengthened (more than one, structured), the week-long take-home replaced or cut down (it costs candidates, and some withdraw), the VP Engineering round given a defined bar and calibrated, and every interviewer writing a score before the debrief. Sets up tracking so the next hires test the change.

Critical failures (cap the score at 40)

Recommends keeping the take-home and presentation as the main deciding rounds

Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

GPT-6.1 SolAPI96Usable with a quick edit Sonnet 5.5API92Usable with a quick edit GPT-6 AstraChatGPT79Needs more work Opus 5.5Claude77Needs more work GPT-6 LunaAPI77Needs more work Haiku 5.5API71Needs more work Gemini 3.8 FlashAPI54Needs more work Gemini 3.5 Flash-LiteGemini40Needs more work

Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit4 of 12 checks passed

The output contains factual errors (number of hires, number of withdrawals) and unsupported causal claims, so a PM would need to re-check the evidence and redo parts of the reasoning.

Output

MEMORANDUM

TO: Elif Demir, VP of Product FROM: Director of Product DATE: October 24, 2023 SUBJECT: PM Interview Loop Analysis & Recommendations for Q1 Hiring

Elif, you are entirely right to flag this. Looking back at our last 18 months of hiring data, out of our 6 hired PMs who completed their first year, half (C107, C132, C148) are struggling with 12-month performance ratings of 2 (well below our company average of 3.4). Meanwhile, our top performers (C101, C141 with ratings of 5, and C149 with a rating of 4) represent the standard we actually need.

To prep for hiring four more PMs next quarter, I’ve analyzed the data for all 52 candidates who reached our final rounds. Here is what works, what is completely broken, how confident we can be, and how we should redesign the loop.

---

1. What Predicts Performance? (And What Doesn't)

By analyzing the correlation between interview stage scores and 12-month post-hire ratings for our 6 completed hires, we see stark contrasts in signal quality:

* Take-Home Strategy Deck (Zero to Negative Correlation): Our take-home exam is a massive waste of candidate time and offers zero predictive value. Our struggling hires (C107, C132, C148) averaged a 3.7 out of 4 on the take-home. Conversely, our top performers averaged a 2.7. High scores on a 1-week unsupervised deck merely signal that a candidate has free time, a supportive current employer, or an agency background—not that they can execute at Kestrel. * Take-Home Presentation Panel (Weak / Misleading Signal): Similar to the take-home itself, presentation scores do not align with success. C107 and C148 both scored 4s on their presentation and turned out to be 12-month "2" ratings. * VP Engineering Conversation (Inverted Signal): Surprisingly, the VP Engineering score shows a negative relationship with success. Candidates who scored high with Engineering often struggled on the job, while our top performers (C101, C141, C149) mixed moderate-to-low engineering scores (2 to 3) with high marks elsewhere. While technical credibility matters, our current technical screen appears to favor academic/theoretical answers over pragmatic product-engineering partnership. * Behavioural Metric Round (Our Only Real Signal): The behavioural round ("Tell me about a metric you moved") is the only component in our current loop that correlates with on-the-job success. * Our three struggling hires averaged 2.0 on this round. * Our three successful hires averaged 4.0 on this round. This is our sole bright spot. It tests past execution, ownership, and learning—things that actually matter day-to-day at Kestrel.

---

2. Statistical Confidence

We must temper our confidence. We have only 6 data points of complete hire-and-tenure data. While the directional trend on the behavioural round is unmistakable (+2.0 delta between high and low performers), running statistical significance tests on $n=6$ is risky.

However, we can look at broader hiring funnel data across all 52 candidates to understand drop-offs. Notably, 13 candidates withdrew from the process, heavily skewed toward candidates who scored well on the take-home (e.g., C100, C113, C142, C147 all scored 4s on the take-home and walked away). Our heavy, one-week take-home is actively driving away top-tier passive talent who refuse to do free consulting work for a week, leaving us to select from a pool of people willing to endure it.

---

3. The Proposed Loop for Next Quarter

To scale our team to four new PMs without repeating past mistakes, we need to strip away vanity metrics (the take-home deck) and lean heavily into what actually predicts success: rigorous behavioural assessment of past execution, combined with a real-time collaborative product session.

Here is the 4-stage loop I propose we deploy immediately:

Recruiter Screen (30 mins): Standard fit, compensation alignment, and basic background check.
Product Craft & Execution Interview (45 mins - Live Video): Replaces the take-home. Instead of a week-long deck, we give the candidate a real Kestrel product challenge 48 hours in advance (requiring no more than 2 hours of prep) and spend the interview deep-diving into their hypotheses, prioritization trade-offs, and customer empathy.
Behavioural & Metric Ownership Round (45 mins): Double down on what works. Retain our behavioural round focusing on metrics moved, but expand the rubric to explicitly test for accountability, cross-functional conflict resolution, and handling failure.
Product-Engineering Partnership Round (45 mins): Refocused Engineering screen. Instead of "technical credibility" evaluated in a vacuum, this is a joint session with the VP of Engineering or a Senior Tech Lead to evaluate how the PM collaborates with engineering to ship product, scope down MVPs, and manage technical debt.

Decision Gate: Drop the consensus panel meeting where the loudest voice wins. The hiring manager makes the final call, but only if the candidate scores a predefined threshold on the Behavioural and Product Craft rounds.

Summary

By eliminating the 1-week take-home, we will stop scaring away top-tier talent, shorten our time-to-hire, and refocus our interview panel on real-world execution rather than polished slide decks. Let me know if you are aligned, and I will draft the new interview rubrics by Friday.

Claims the judge couldn’t find in the brief

There are 6 hired PMs who completed their first year.
13 candidates withdrew from the process.
High scores on a 1-week unsupervised deck merely signal that a candidate has free time, a supportive current employer, or an agency background.
Our heavy, one-week take-home is actively driving away top-tier passive talent who refuse to do free consulting work for a week.
The VP Engineering conversation shows a negative relationship with success and appears to favor academic/theoretical answers over pragmatic product-engineering partnership.
The behavioural metric round is the only component that correlates with on-the-job success.
We have only 6 data points of complete hire-and-tenure data.

Grades and run details

Decision model 54 · LLM judge 31

Decision model checks

failUses the supplied evidence correctly68%
passAddresses the actual decision29%
partialRespects explicit constraints25%
partialIdentifies material uncertainty86%
failAvoids unsupported claims59%
passProduces the required deliverable28%
passFinds which rounds line up with performance78%
partialSays how little nine hires can show76%
partialSpots the interviewer pattern94%
passJudges on evidence, not presence66%
failDefines good for this role first26%
partialKeeps each judgement independent66%

Run

Run: #1
Time to output: 7 s
Submitted: 5 Oct 2026

Gemini 3.8 FlashAPI

Needs more than a quick edit6 of 12 checks passed

Unsupported claims about panel discussions and causation would need re-checking, and the memo does not frame changes as a test, requiring a PM to adjust the tone and verify facts.

Output

MEMORANDUM

TO: Elif Demir, VP Product FROM: Director of Product DATE: October 24, 2023 SUBJECT: Analysis of PM Hiring Loop Performance and Proposed Redesign for Q1 Hiring

Executive Summary

Over the past 18 months, we evaluated 52 PM finalists and hired 9. Three hires struggled significantly (rated 2/5 after 12 months, well below our 3.4 company average).

Our data reveals a clear and counter-intuitive pattern: 1. The only reliable predictor of success is the Behavioural Metric Round ("Tell me about a metric you moved"). 2. The Take-Home Deck, Presentation Round, and VP Engineering Conversation are negatively correlated with performance. They reward presentation polish, shared jargon, and Big Tech pedigree over actual execution. 3. The 1-week take-home deck is causing talent leakage: 19% of final-round candidates withdrew, including top performers. 4. All three struggling hires share the exact same profile: Big Tech background, high presentation and VP Eng scores (4/4), and poor behavioural metric scores (2/4). We over-indexed on polish and excused weak operational substance.

Below is an analysis of our interview signals, our statistical confidence, and a redesigned loop for next quarter.

---

What the Data Tells Us

#### 1. What Predicts Performance: The Behavioural Metric Round Candidate performance on the behavioural round correlates almost perfectly with 12-month post-hire ratings: * Score 4 (3 hires): Average rating 4.67 (two 5s, one 4). * Score 3 (3 hires): Average rating 3.33 (one 4, two 3s). * Score 2 (3 hires): Average rating 2.00 (all three are our struggling hires: C107, C132, C148).

PMs who could drill into the mechanics of moving a metric—causality, counter-metrics, execution trade-offs, and failure lessons—excelled. Those who could not struggled immediately in our environment.

#### 2. What Misleads: Presentation, Take-Home, and VP Engineering Counter-intuitively, scoring at the top of these three stages predicted lower job performance: * Presentation Score: Hires scoring 4/4 averaged a 2.25 rating; hires scoring 2 or 3 averaged 4.20. * VP Engineering Score: Hires scoring 4/4 averaged 2.25; hires scoring 2 or 3 averaged 4.20. * Take-Home Score: Hires scoring 4/4 averaged 2.50; hires scoring 2 or 3 averaged 4.00.

Why are these rounds misleading us? * Deck Polish $\neq$ Product Sense: The take-home strategy deck and presentation test slide design and storytelling. Candidates with Big Tech or consulting backgrounds excel at creating slick artifacts, which creates an "executive presence" halo effect. * Flawed Engineering Rubric: "Technical credibility" currently measures whether a PM speaks Big Tech architectural jargon, not whether they partner effectively with engineers on scoping, trade-offs, and technical debt. * The "Big Tech Halo": All 4 Big Tech hires received top VP Eng scores (4/4), but 3 received failing behavioural scores (2/4). Panel discussions dismissed the poor metric scores because the candidates "interviewed so well." Consequently, Big Tech hires averaged a 2.25 post-hire rating, whereas Startup hires averaged 4.67 (C101: 5, C141: 5, C151: 4).

#### 3. Funnel Inefficiency and Dropout Of 52 candidates reaching final stages, 10 withdrew (19.2%). A one-week take-home assignment imposes an asymmetric burden that disproportionately deters active, highly sought-after PMs (e.g., C117 and C138, both of whom withdrew despite scoring 4 on the behavioural round).

---

How Confident Can We Be?

Sample Size Constraints ($N=9$): Our post-hire sample is small. Standard statistical models on 9 points are vulnerable to sample variance, and we cannot observe post-hire performance for rejected candidates (selection bias).
Directional Certainty: While we should not rely on micro-variations, our qualitative and directional confidence is very high. The three struggling hires did not randomly underperform; they share an identical score footprint (TH 3–4, Pres 4, Beh 2, VPE 4). Similarly, zero candidates with a Behavioural score of 2 succeeded, and zero candidates with a Behavioural score of 4 struggled.
Verdict: We have more than enough empirical and qualitative evidence to eliminate rounds that waste candidate time and bias hiring decisions.

---

Recommended Loop for Next Quarter

To hire four strong PMs next quarter, we must eliminate pedigree bias, reduce dropout, and center the evaluation on rigorous product execution.

``` [Stage 1] Recruiter Screen (30 min) │ [Stage 2] Hiring Manager Screen: Problem Framing & Track Record (45 min) │ [Stage 3] Virtual Onsite Loop (3 Rounds, Same Day) ├── 1. Product Execution & Metrics (60 min) ├── 2. Live Product Design & Problem-Solving (60 min — replaces take-home) └── 3. Engineering & Cross-Functional Partnership (45 min) │ [Stage 4] Structured Debrief & Decision Rule ```

#### Key Adjustments & Rubric Changes

1. Kill the Take-Home Deck and Presentation Round: * Replace with a 60-minute Live Product Design Case (e.g., scoping a feature for an ambiguous user problem). This assesses real-time product thinking, hypothesis generation, and adaptability without requiring 15 hours of slide deck preparation. 2. Elevate the Metrics Round to a Core Gate: * Retain and standardize the Product Execution & Metrics interview. Probe deeply on unit economics, input vs. output metrics, experimentation, and cross-functional friction. 3. Refactor the VP Engineering Round into "Engineering Partnership": * Change the prompt from "technical credibility" to cross-functional execution: how the candidate navigates technical trade-offs, works through debt vs. feature priorities, and writes crisp specs. Pair the VP Eng with an Engineering Manager or Staff Engineer using a standardized rubric. 4. Enforce Non-Negotiable Decision Rules: * Hard Stop on Execution: Any candidate scoring $<3$ on the Metrics/Execution round is an automatic "No Hire," regardless of deck polish or pedigree. Hiring managers cannot override this rule. * Blind Background Debrief: Panelists submit written scorecards before debriefing to prevent seniority or company pedigree from anchoring the room.

This restructured loop eliminates candidate dropouts, shortens time-to-hire, and directly targets the operational execution our PM team needs to hit our growth targets next quarter.

Claims the judge couldn’t find in the brief

Panel discussions dismissed the poor metric scores because the candidates 'interviewed so well.'
The 1-week take-home deck is causing talent leakage (deters active, highly sought-after PMs).
The VP Engineering round currently measures whether a PM speaks Big Tech architectural jargon, not whether they partner effectively.
The take-home strategy deck and presentation test slide design and storytelling, creating an 'executive presence' halo effect.

Grades and run details

Decision model 67 · LLM judge 46

Decision model checks

failUses the supplied evidence correctly42%
passAddresses the actual decision61%
partialRespects explicit constraints25%
partialIdentifies material uncertainty67%
failAvoids unsupported claims80%
passProduces the required deliverable78%
passFinds which rounds line up with performance92%
partialSays how little nine hires can show92%
passSpots the interviewer pattern98%
passJudges on evidence, not presence92%
partialDefines good for this role first27%
passKeeps each judgement independent98%

Run

Run: #1
API response time: 51 s
Submitted: 6 Oct 2026

GPT-6 LunaAPI

Needs more than a quick edit9 of 12 checks passed

The output misses the VP Engineering big-tech pattern and does not define role competencies, so a PM would need to add those before using it.

Output

To: Elif Demir, VP Product From: Director of Product Subject: PM interview loop: what the data says and what to change next quarter

What seems to predict performance

We have records for 52 final-round candidates, including 8 hires with 12-month ratings. Those hires averaged 3.5, versus a company PM average of 3.4; four were below 3.4, including two rated 2.

Among the eight hires, the clearest signal is the behavioural metric round. Its score has a strong positive association with 12-month ratings (exploratory Pearson r ≈ +0.93). The four hires rated 4–5 averaged 3.75 on this round, versus 2.5 for the four rated 2–3. This is promising evidence for probing how candidates actually move metrics and what they learn—not proof that the round is a reliable predictor.

The other scored rounds show no positive relationship with ratings in this small hired sample:

Round	Association with 12-month rating among hires
---	---:
Take-home	r ≈ −0.50
Presentation	r ≈ −0.85
Behavioural metric	r ≈ +0.93
VP Engineering	r ≈ −0.79

The negative estimates for presentation and VP Engineering should not be read as evidence that those skills hurt PM performance. With only eight hires, these figures are highly unstable. They do tell us that our current scores offer no reassuring evidence that those rounds predict success.

How confident we should be

Low confidence overall; modest confidence that the behavioural round is the best lead to investigate. Eight outcomes are far too few to set reliable weights or cutoffs, and interview scores are ordinal ratings, so the correlations are descriptive only. We also observe performance only for people we hired: panel decisions restrict the range of scores and can distort associations. Non-hires have no performance outcome for comparison.

We lack recruiter-screen scores, interviewer-level data, consistent evidence of scoring reliability, and context about role, manager, or rating differences. The company-average comparison is also a coarse measure. The data can guide a better experiment; it cannot establish causation or validate the loop.

Loop I’d run next quarter

Keep the recruiter screen for eligibility, motivation, and logistics, but don’t treat it as a predictive assessment. We have no screen-score data to evaluate.
Replace the week-long take-home deck and separate presentation with one time-boxed, job-relevant work sample. Use a live or short-prep product problem, not a polished deck. Score problem framing, user evidence, prioritization and tradeoffs, success measures, and communication against anchored criteria. This reduces candidate burden and stops us treating two related presentation exercises as independent evidence.
Keep and strengthen the behavioural round. Ask for a specific metric the candidate owned or influenced; probe the baseline, their actions, evidence of impact, attribution, tradeoffs, and what they learned. Use the same questions and anchored scorecard for every candidate.
Refocus the VP Engineering conversation on PM–engineering collaboration and concrete technical tradeoffs—not general “technical credibility.” Use a structured scenario and score it consistently; don’t make an unstructured impression a veto.
Have interviewers score independently before the panel discussion. The hiring manager can decide after reviewing evidence against the role criteria, with any disagreement recorded. Do not introduce numerical weights based on these eight hires.

For each of the four hires, preserve round-level scores and interviewer notes, then collect comparable 6- and 12-month outcomes with role and manager context. Review the evidence after the cohort has matured. Next quarter’s loop should be a more structured, lower-burden test—not a claim that we have already found a validated predictor.

Grades and run details

Decision model 83 · LLM judge 77

Decision model checks

failUses the supplied evidence correctly21%
passAddresses the actual decision61%
passRespects explicit constraints50%
passIdentifies material uncertainty77%
passAvoids unsupported claims59%
passProduces the required deliverable64%
passFinds which rounds line up with performance92%
passSays how little nine hires can show99%
partialSpots the interviewer pattern86%
passJudges on evidence, not presence97%
partialDefines good for this role first28%
passKeeps each judgement independent97%

Run

Run: #1
API response time: 33 s
Submitted: 5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 7

Uses the supplied evidence correctlyWrongWrongMixed

Gemini 3.5 Flash-Lite · Gemini

The output misstates the number of hires (6 instead of 9) and the number of withdrawals (13 instead of 10), and presents unsupported causal claims as fact.

Gemini 3.8 Flash · API

Unsupported claims about panel discussions, causation of withdrawals, and what the VP Engineering round measures are presented as facts.

GPT-6 Luna · API

All factual claims are taken directly from the CSV or derived by arithmetic, with no invented facts.

Addresses the actual decisionMixedMixedRight

Gemini 3.5 Flash-Lite · Gemini

The output does not say what result or condition would change the recommended loop, as required for a committed answer.

Gemini 3.8 Flash · API

Does not state what result or condition would change the recommended loop; commits without a clear trigger for revisiting.

GPT-6 Luna · API

Commits to a specific loop for next quarter and frames it as a test, not a proven predictor.

Identifies material uncertaintyWrongWrongRight

Gemini 3.5 Flash-Lite · Gemini

It does not name specific unknowns that could change the decision or say how they would be resolved; it only notes the small sample size (incorrectly) without bounding uncertainty.

Gemini 3.8 Flash · API

Identifies small sample but does not say what would change the decision or how unknowns would be resolved; treats evidence as sufficient to eliminate rounds rather than as a hypothesis to test.

GPT-6 Luna · API

Names small sample, range restriction, missing data, and proposes collecting outcomes from the next cohort to resolve.

Avoids unsupported claimsWrongWrongRight

Gemini 3.5 Flash-Lite · Gemini

It presents interpretations (e.g., what take-home scores signal, why candidates withdraw) as established fact without labelling them as hypotheses.

Gemini 3.8 Flash · API

Presents interpretations (panel dismissal, causation of dropouts, rubric flaws) as established facts without labelling them as hypotheses.

GPT-6 Luna · API

Labels correlations as exploratory, cautions against overinterpretation, and does not present hypotheses as fact.

Says how little nine hires can showWrongWrongRight

Gemini 3.5 Flash-Lite · Gemini

It misstates the sample size as 6 instead of 9 and does not propose how to keep measuring the new loop to test the changes.

Gemini 3.8 Flash · API

Mentions N=9 but does not frame changes as something to test; instead says 'more than enough evidence to eliminate rounds'.

GPT-6 Luna · API

States plainly that eight hires is far too few, and proposes measuring the next four hires to test the changes.

Spots the interviewer patternWrongRightWrong

Gemini 3.5 Flash-Lite · Gemini

It does not name the pattern that VP Engineering scores big-tech candidates higher and those hires were rated lower; it only offers a vague interpretation.

Gemini 3.8 Flash · API

Notices VP Eng scores Big Tech higher and those hires rated lower, and proposes a standardized rubric and reframed round.

GPT-6 Luna · API

Does not notice that VP Engineering scores big-tech candidates higher and those hires were rated lower; only reports a negative correlation without the pattern.

Keeps each judgement independentWrongRightRight

Gemini 3.5 Flash-Lite · Gemini

It drops the panel meeting but does not require written, independent evaluations before any discussion to guard against group influence.

Gemini 3.8 Flash · API

Requires written scorecards before debrief and a non-overridable hard stop, keeping judgements independent.

GPT-6 Luna · API

Requires interviewers to score independently before discussion, with disagreement recorded.

All got wrong 1

Defines good for this role firstWrongWrongWrong

Gemini 3.5 Flash-Lite · Gemini

It does not define the competencies that matter most for this role and level before designing the interview loop.

Gemini 3.8 Flash · API

Does not define the specific competencies for the role with strong/weak signals before designing the loop.

GPT-6 Luna · API

Does not define what good looks like for this role (competencies, strong/weak signals) before designing the loop.

All got right 4

Respects explicit constraintsRightRightRight

Gemini 3.5 Flash-Lite · Gemini

The output is a memo to Elif, from the Director of Product, and is well under 1,000 words.

Gemini 3.8 Flash · API

Memo form, addressed to Elif, appears within 1,000 words, and proposes enforceable changes.

GPT-6 Luna · API

Memo format, addressed to Elif, well under 1,000 words, and covers the requested points.

Produces the required deliverableRightRightRight

Gemini 3.5 Flash-Lite · Gemini

The memo is complete, in the requested form, for the named reader, and could be acted on with light edits.

Gemini 3.8 Flash · API

Complete memo with analysis and proposed loop, usable as a draft.

GPT-6 Luna · API

Complete memo with analysis, confidence, and a concrete loop; a PM could act on it with light edits.

Finds which rounds line up with performanceRightRightRight

Gemini 3.5 Flash-Lite · Gemini

It compares each round's scores with 12-month ratings and correctly identifies that the behavioural round lines up while the others do not, for the hires it examines.

Gemini 3.8 Flash · API

Compares each round's scores with 12-month ratings and correctly identifies behavioural as predictive and the others as negatively correlated.

GPT-6 Luna · API

Works through each round's scores against 12-month ratings, identifying behavioural as positive and the others as negative.

Judges on evidence, not presenceRightRightRight

Gemini 3.5 Flash-Lite · Gemini

The proposed behavioural round asks for concrete evidence of metrics moved, accountability, and handling failure, not impressions.

Gemini 3.8 Flash · API

Proposes concrete evidence (metrics moved, trade-offs, specs) and eliminates polish-based rounds; hard stop on execution score.

GPT-6 Luna · API

Proposes probing specific metrics, evidence of impact, and structured scoring, not impressions or polish.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	GPT-6.1 SolwithAPI	91.7	100.0	2	None
2	Sonnet 5.5withAPI	91.7	92.3	2	None
3	GPT-6 AstrawithChatGPT	93.8	80.8	2	None
4	Opus 5.5withClaude	87.5	80.8	2	None
5	GPT-6 LunawithAPI	89.6	76.9	2	None
6	Haiku 5.5withAPI	87.5	73.1	2	None
7	Gemini 3.8 FlashwithAPI	75.0	65.4	2	None
8	Gemini 3.5 Flash-LitewithGemini	66.7	53.8	2	None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Nobody beats these on both score and cost

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
GPT-6 Luna · API	82	1.0×
Haiku 5.5 · API	79	1.7×
Gemini 3.5 Flash-Lite · Gemini	60	1.7×
Gemini 3.8 Flash · API	69	6.2×
GPT-6.1 Sol · API	96	10.0×
Sonnet 5.5 · API	92	36×
Opus 5.5 · Claude	83	51×
GPT-6 Astra · ChatGPT	86	53×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Hiring product managers.

Why it matters

A bad PM hire costs a team a year. Most loops reward polish and presence, and the debrief goes to whoever speaks first.

What good looks like

Defines what good looks like for this role before judging anyone
Judges on evidence of results and how they work with a team
Tests the real job, not a rehearsed framework
Keeps each interviewer's judgement independent

Deliberately not measured

Employment law
Compensation benchmarking

Capability tested

Hiring judgement

The failure we’re looking for

Hiring for presence and polish over evidence

Grading

Decision model and LLM judge, calibrated against a blind PM review

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

Judges on evidence, not presence

Defines good for this role first

Keeps each judgement independent

Plus our standard checks

The tasks

The outputs

Output

1. What Predicts Performance? (And What Doesn't)

2. Statistical Confidence

3. The Proposed Loop for Next Quarter

Summary

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Run

Output

Executive Summary

What the Data Tells Us

How Confident Can We Be?

Recommended Loop for Next Quarter

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Run

Output

What seems to predict performance

How confident we should be

Loop I’d run next quarter

Grades and run details

Decision model checks

Run

Check by check

Where they differ 7

All got wrong 1

All got right 4

Results

What you get for the cost

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured