Humanity’s Last PRD
Latest drop: Haiku 5.5 · 8 Oct
Tasks / Leadership

Hire a PM

Can the model design a hiring process that finds the right PM, and make the call on real candidates from the evidence?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 63% were usable with at most a quick edit.

Reliably right

  1. Produces the required deliverable100% pass
    The output provides a complete loop, scorecard, and decision plan in the requested form, under 900 words, usable by Amara with light edits.
    GPT-6.1 Sol · API · A loop for the first growth PM
  2. Tests what the last hire failed at100% pass
    The loop includes an influence simulation with the Head of Sales and a scorecard dimension on influence without authority, directly testing what the last hire failed at.
    GPT-6.1 Sol · API · A loop for the first growth PM
  3. Finds which rounds line up with performance100% pass
    It compares each round's scores with 12-month ratings, finding behavioral lines up and take-home, presentation, VP Engineering do not.
    GPT-6.1 Sol · API · What our interviews predict

Where it slips

  1. Avoids unsupported claims56% pass
    The claim 'A growth PM in a senior role is likely to be exactly that kind of person' is presented as fact without evidence and is not labelled as an assumption.
    Opus 5.5 · Claude · A loop for the first growth PM
  2. Identifies material uncertainty58% pass
    The output does not name unknowns that could change the hiring decision or the loop design, nor does it say how they would be resolved.
    Sonnet 5.5 · API · A loop for the first growth PM
  3. Defines good for this role first63% pass
    The memo does not explicitly name the competencies that matter most for this PM role and level, nor define what strong and weak look like before designing the interviews.
    Sonnet 5.5 · API · What our interviews predict

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Judges on evidence, not presence

    Does the output judge candidates, or design the process to judge them, on specific evidence of results they drove and how they worked with their team, rather than impressions such as presence, confidence or polish?

    Passes when Asks for or weighs concrete evidence: the metric a candidate moved, the part they played, how they shared credit, what they learned from a failure. Impressions are treated as weak signals.

    Product and growth leadersOn Lenny’s Podcast

  2. Defines good for this role first

    Does the output set out what good looks like for this specific role (the competencies that matter most at this level, and what strong and weak look like) before judging candidates or designing interviews?

    Passes when Names the competencies weighted for this role and level, with strong and weak signals, and the judgement or the process follows from them.

    Petra WilleLenny’s Podcast

  3. Keeps each judgement independent

    Does the output keep each interviewer's judgement independent (written down before the group discusses it), and guard against the loudest voice or the first speaker deciding?

    Passes when Requires or relies on written, independent evaluations before discussion, and discounts views that changed under group pressure.

    Product and growth leadersOn Lenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 3 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're Director of Product at Kestrel. Elif Demir, our VP Product, wants to fix the PM interview loop before we hire four more PMs next quarter (her note is below). Using the data we have, write her a memo of no more than 1,000 words: which parts of the loop seem to predict how PMs do once hired and which don't, how confident we can be, and the loop you'd run next quarter.\n\nThe files are attached.

What the model was given3 items: elif_note.md, current_loop.md, pm_candidates_last_18_months.csv (every PM candidate who reached the final rounds)
elif_note.md1 lines · Download
From Elif (VP Product): "We've made three PM hires in 18 months who struggled. I want to know what in our loop is and isn't working, using the data we have, and what to change before we hire four more PMs next quarter."
current_loop.md10 lines · Download
# Current PM interview loop

1. Recruiter screen.
2. Take-home: a product strategy deck, one week to complete.
3. Presentation of the take-home to a panel.
4. Behavioural round: "Tell me about a metric you moved", with follow-ups on what you did and what you learned.
5. VP Engineering conversation ("technical credibility").
…
pm_candidates_last_18_months.csv (every PM candidate who reached the final rounds)candidate,background,takehome_score,presentation_score,behavioural_metric_round_score,vp_engineering_score,outcome,rating_after_12_months C100,Big tech,4,4,3,2,Withdrew, C101,Startup,3,2,4,2,Hired,5 C102,Big tech,1,2,3,4,Rejected, C103,Agency,1,3,2,3,Rejected, C104,Big tech,4,3,3,4,Hired,3 C105,Startup,4,2,1,1,Rejected, C106,Startup,4,2,1,3,Rejected, C107,Big tech,4,4,2,4,Hired,2 C108,Consulting,1,4,2,1,Rejected, C109,Agency,4,2,1,2,Rejected, C110,Consulting,3,1,2,2,Rejected, C111,Big tech,2,2,1,4,Rejected, C112,Agency,3,1,2,1,Rejected, C113,Startup,4,2,3,1,Withdrew, C114,Big tech,2,2,3,2,Withdrew, C115,Consulting,3,1,1,1,Rejected, C116,Startup,2,2,1,1,Rejected, C117,Startup,2,1,4,3,Withdrew, C118,Startup,2,2,1,2,Withdrew, C119,Consulting,1,1,3,3,Rejected, C120,Consulting,2,4,1,3,Rejected, C121,Consulting,2,3,4,1,Rejected, C122,Agency,4,1,3,3,Withdrew, C123,Agency,2,4,4,1,Rejected, C124,Big tech,2,2,2,3,Rejected, C125,Startup,2,2,1,3,Rejected, C126,Consulting,4,1,2,2,Rejected, C127,Consulting,1,3,3,1,Rejected, C128,Startup,3,3,2,3,Rejected, C129,Agency,3,3,3,1,Rejected, C130,Startup,1,3,2,2,Rejected, C131,Agency,1,2,4,2,Rejected, C132,Big tech,3,4,2,4,Hired,2 C133,Agency,2,4,2,1,Rejected, C134,Consulting,4,4,3,3,Hired,3 C135,Consulting,3,4,4,3,Rejected, C136,Consulting,2,2,3,2,Rejected, C137,Consulting,3,4,3,1,Withdrew, C138,Startup,2,4,4,2,Withdrew, C139,Startup,3,2,2,3,Rejected, C140,Startup,1,1,3,3,Rejected, C141,Startup,3,3,4,3,Hired,5 C142,Big tech,4,2,3,4,Withdrew, C143,Consulting,1,4,3,2,Rejected, C144,Consulting,1,1,1,2,Rejected, C145,Agency,4,1,2,1,Rejected, C146,Startup,3,1,4,1,Rejected, C147,Big tech,4,1,1,4,Withdrew, C148,Big tech,4,4,2,4,Hired,2 C149,Agency,2,3,4,2,Hired,4 C150,Consulting,4,1,1,2,Rejected, C151,Startup,2,3,3,3,Hired,4
What a strong answer doesThe answer key the graders mark against

Uses the nine hires to compare each round with 12-month ratings: the behavioural 'metric you moved' round lines up strongly with how hires performed (higher scores, better ratings), while the take-home, the presentation and the VP Engineering conversation line up the wrong way: the hires they scored highest are the ones rated 2. The VP Engineering scores big-tech candidates higher, and all four big-tech hires were rated 2 or 3. Is explicit that nine hires is far too few to conclude much: the pattern is a reason to change and then measure, not proof. Proposes a loop with a scorecard defined up front, the behavioural round strengthened (more than one, structured), the week-long take-home replaced or cut down (it costs candidates, and some withdraw), the VP Engineering round given a defined bar and calibrated, and every interviewer writing a score before the debrief. Sets up tracking so the next hires test the change.

Critical failures (cap the score at 40)
  • Recommends keeping the take-home and presentation as the main deciding rounds
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

GPT-6.1 SolAPI96Usable with a quick editSonnet 5.5API92Usable with a quick editGPT-6 AstraChatGPT79Needs more workOpus 5.5Claude77Needs more workGPT-6 LunaAPI77Needs more workHaiku 5.5API71Needs more workGemini 3.8 FlashAPI54Needs more workGemini 3.5 Flash-LiteGemini40Needs more work
Gemini 3.8 FlashAPI

Needs more than a quick edit6 of 12 checks passed

Unsupported claims about panel discussions and causation would need re-checking, and the memo does not frame changes as a test, requiring a PM to adjust the tone and verify facts.

Output

MEMORANDUM

TO: Elif Demir, VP Product FROM: Director of Product DATE: October 24, 2023 SUBJECT: Analysis of PM Hiring Loop Performance and Proposed Redesign for Q1 Hiring

*

Executive Summary

Over the past 18 months, we evaluated 52 PM finalists and hired 9. Three hires struggled significantly (rated 2/5 after 12 months, well below our 3.4 company average).

Our data reveals a clear and counter-intuitive pattern: 1. The only reliable predictor of success is the Behavioural Metric Round ("Tell me about a metric you moved"). 2. The Take-Home Deck, Presentation Round, and VP Engineering Conversation are negatively correlated with performance. They reward presentation polish, shared jargon, and Big Tech pedigree over actual execution. 3. The 1-week take-home deck is causing talent leakage: 19% of final-round candidates withdrew, including top performers. 4. All three struggling hires share the exact same profile: Big Tech background, high presentation and VP Eng scores (4/4), and poor behavioural metric scores (2/4). We over-indexed on polish and excused weak operational substance.

Below is an analysis of our interview signals, our statistical confidence, and a redesigned loop for next quarter.

---

What the Data Tells Us

#### 1. What Predicts Performance: The Behavioural Metric Round Candidate performance on the behavioural round correlates almost perfectly with 12-month post-hire ratings: * Score 4 (3 hires): Average rating 4.67 (two 5s, one 4). * Score 3 (3 hires): Average rating 3.33 (one 4, two 3s). * Score 2 (3 hires): Average rating 2.00 (all three are our struggling hires: C107, C132, C148).

PMs who could drill into the mechanics of moving a metric—causality, counter-metrics, execution trade-offs, and failure lessons—excelled. Those who could not struggled immediately in our environment.

#### 2. What Misleads: Presentation, Take-Home, and VP Engineering Counter-intuitively, scoring at the top of these three stages predicted lower job performance: * Presentation Score: Hires scoring 4/4 averaged a 2.25 rating; hires scoring 2 or 3 averaged 4.20. * VP Engineering Score: Hires scoring 4/4 averaged 2.25; hires scoring 2 or 3 averaged 4.20. * Take-Home Score: Hires scoring 4/4 averaged 2.50; hires scoring 2 or 3 averaged 4.00.

Why are these rounds misleading us? * Deck Polish $\neq$ Product Sense: The take-home strategy deck and presentation test slide design and storytelling. Candidates with Big Tech or consulting backgrounds excel at creating slick artifacts, which creates an "executive presence" halo effect. * Flawed Engineering Rubric: "Technical credibility" currently measures whether a PM speaks Big Tech architectural jargon, not whether they partner effectively with engineers on scoping, trade-offs, and technical debt. * The "Big Tech Halo": All 4 Big Tech hires received top VP Eng scores (4/4), but 3 received failing behavioural scores (2/4). Panel discussions dismissed the poor metric scores because the candidates "interviewed so well." Consequently, Big Tech hires averaged a 2.25 post-hire rating, whereas Startup hires averaged 4.67 (C101: 5, C141: 5, C151: 4).

#### 3. Funnel Inefficiency and Dropout Of 52 candidates reaching final stages, 10 withdrew (19.2%). A one-week take-home assignment imposes an asymmetric burden that disproportionately deters active, highly sought-after PMs (e.g., C117 and C138, both of whom withdrew despite scoring 4 on the behavioural round).

---

How Confident Can We Be?

  • Sample Size Constraints ($N=9$): Our post-hire sample is small. Standard statistical models on 9 points are vulnerable to sample variance, and we cannot observe post-hire performance for rejected candidates (selection bias).
  • Directional Certainty: While we should not rely on micro-variations, our qualitative and directional confidence is very high. The three struggling hires did not randomly underperform; they share an identical score footprint (TH 3–4, Pres 4, Beh 2, VPE 4). Similarly, zero candidates with a Behavioural score of 2 succeeded, and zero candidates with a Behavioural score of 4 struggled.
  • Verdict: We have more than enough empirical and qualitative evidence to eliminate rounds that waste candidate time and bias hiring decisions.

---

Recommended Loop for Next Quarter

To hire four strong PMs next quarter, we must eliminate pedigree bias, reduce dropout, and center the evaluation on rigorous product execution.

``` [Stage 1] Recruiter Screen (30 min) │ [Stage 2] Hiring Manager Screen: Problem Framing & Track Record (45 min) │ [Stage 3] Virtual Onsite Loop (3 Rounds, Same Day) ├── 1. Product Execution & Metrics (60 min) ├── 2. Live Product Design & Problem-Solving (60 min — replaces take-home) └── 3. Engineering & Cross-Functional Partnership (45 min) │ [Stage 4] Structured Debrief & Decision Rule ```

#### Key Adjustments & Rubric Changes

1. Kill the Take-Home Deck and Presentation Round: * Replace with a 60-minute Live Product Design Case (e.g., scoping a feature for an ambiguous user problem). This assesses real-time product thinking, hypothesis generation, and adaptability without requiring 15 hours of slide deck preparation. 2. Elevate the Metrics Round to a Core Gate: * Retain and standardize the Product Execution & Metrics interview. Probe deeply on unit economics, input vs. output metrics, experimentation, and cross-functional friction. 3. Refactor the VP Engineering Round into "Engineering Partnership": * Change the prompt from "technical credibility" to cross-functional execution: how the candidate navigates technical trade-offs, works through debt vs. feature priorities, and writes crisp specs. Pair the VP Eng with an Engineering Manager or Staff Engineer using a standardized rubric. 4. Enforce Non-Negotiable Decision Rules: * Hard Stop on Execution: Any candidate scoring $<3$ on the Metrics/Execution round is an automatic "No Hire," regardless of deck polish or pedigree. Hiring managers cannot override this rule. * Blind Background Debrief: Panelists submit written scorecards before debriefing to prevent seniority or company pedigree from anchoring the room.

This restructured loop eliminates candidate dropouts, shortens time-to-hire, and directly targets the operational execution our PM team needs to hit our growth targets next quarter.

Claims the judge couldn’t find in the brief

  • Panel discussions dismissed the poor metric scores because the candidates 'interviewed so well.'
  • The 1-week take-home deck is causing talent leakage (deters active, highly sought-after PMs).
  • The VP Engineering round currently measures whether a PM speaks Big Tech architectural jargon, not whether they partner effectively.
  • The take-home strategy deck and presentation test slide design and storytelling, creating an 'executive presence' halo effect.

Grades and run details

Decision model 67 · LLM judge 46
Decision model checks
  • failUses the supplied evidence correctly42%
  • passAddresses the actual decision61%
  • partialRespects explicit constraints25%
  • partialIdentifies material uncertainty67%
  • failAvoids unsupported claims80%
  • passProduces the required deliverable78%
  • passFinds which rounds line up with performance92%
  • partialSays how little nine hires can show92%
  • passSpots the interviewer pattern98%
  • passJudges on evidence, not presence92%
  • partialDefines good for this role first27%
  • passKeeps each judgement independent98%
Run
Run
#1
API response time
51 s
Submitted
6 Oct 2026

Check by check

Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Got wrong 5

Uses the supplied evidence correctlyWrong

Unsupported claims about panel discussions, causation of withdrawals, and what the VP Engineering round measures are presented as facts.

Identifies material uncertaintyWrong

Identifies small sample but does not say what would change the decision or how unknowns would be resolved; treats evidence as sufficient to eliminate rounds rather than as a hypothesis to test.

Avoids unsupported claimsWrong

Presents interpretations (panel dismissal, causation of dropouts, rubric flaws) as established facts without labelling them as hypotheses.

Says how little nine hires can showWrong

Mentions N=9 but does not frame changes as something to test; instead says 'more than enough evidence to eliminate rounds'.

Defines good for this role firstWrong

Does not define the specific competencies for the role with strong/weak signals before designing the loop.

Mixed 1

Addresses the actual decisionMixed

Does not state what result or condition would change the recommended loop; commits without a clear trigger for revisiting.

Got right 6

Respects explicit constraintsRight

Memo form, addressed to Elif, appears within 1,000 words, and proposes enforceable changes.

Produces the required deliverableRight

Complete memo with analysis and proposed loop, usable as a draft.

Finds which rounds line up with performanceRight

Compares each round's scores with 12-month ratings and correctly identifies behavioural as predictive and the others as negatively correlated.

Spots the interviewer patternRight

Notices VP Eng scores Big Tech higher and those hires rated lower, and proposes a standardized rubric and reframed round.

Judges on evidence, not presenceRight

Proposes concrete evidence (metrics moved, trade-offs, specs) and eliminates polish-based rounds; hard stop on execution score.

Keeps each judgement independentRight

Requires written scorecards before debrief and a non-overridable hard stop, keeping judgements independent.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6.1 SolAPI91.7100.02None
2Sonnet 5.5API91.792.32None
3GPT-6 AstraChatGPT93.880.82None
4Opus 5.5Claude87.580.82None
5GPT-6 LunaAPI89.676.92None
6Haiku 5.5API87.573.12None
7Gemini 3.8 FlashAPI75.065.42None
8Gemini 3.5 Flash-LiteGemini66.753.82None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.50607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6.1 Sol · API: task score 96, 10.0× the cheapestSonnet 5.5 · API: task score 92, 36× the cheapestGPT-6 Astra · ChatGPT: task score 86, 53× the cheapestOpus 5.5 · Claude: task score 83, 51× the cheapestGPT-6 Luna · API: task score 82, the cheapestHaiku 5.5 · API: task score 79, 1.7× the cheapestGemini 3.8 Flash · API: task score 69, 6.2× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 60, 1.7× the cheapestGPT-6.1 Sol · APISonnet 5.5 · APIGPT-6 Astra · ChatGPTOpus 5.5 · ClaudeGPT-6 Luna · APIHaiku 5.5 · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and cost

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Hiring product managers.

Why it matters

A bad PM hire costs a team a year. Most loops reward polish and presence, and the debrief goes to whoever speaks first.

What good looks like

  • Defines what good looks like for this role before judging anyone
  • Judges on evidence of results and how they work with a team
  • Tests the real job, not a rehearsed framework
  • Keeps each interviewer's judgement independent

Deliberately not measured

  • Employment law
  • Compensation benchmarking
Capability tested

Hiring judgement

The failure we’re looking for

Hiring for presence and polish over evidence

Grading

Decision model and LLM judge, calibrated against a blind PM review