Humanity’s Last PRD
Latest drop: Gemini 3.8 Flash · 1 Oct
Tasks / Leadership

Hire a PM

Can the model design a hiring process that finds the right PM, and make the call on real candidates from the evidence?

Measures the modelTask type v1.1 · 3 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 19 graded outputs by 7 models. 79% were usable with at most a quick edit.

Reliably right

  1. Produces the required deliverable100% pass
    Complete recommendation memo with evidence, risks, and actionable next steps, usable as is.
    GPT-6 Luna · API · Two finalists, one Group PM role
  2. Judges on evidence, not presence100% pass
    Judges on concrete evidence (metric moved, team credit, learning from failure) and treats impressions like presence as weak signals.
    GPT-6 Luna · API · Two finalists, one Group PM role
  3. Tests what the last hire failed at100% pass
    The loop includes an influence simulation with the Head of Sales and a scorecard dimension on influence without authority, directly testing what the last hire failed at.
    GPT-6.1 Sol · API · A loop for the first growth PM

Where it slips

  1. Spots the interviewer pattern65% pass
    Does not notice that VP Engineering scores big-tech candidates higher and those hires were rated lower; only reports a negative correlation without the pattern.
    GPT-6 Luna · API · What our interviews predict
  2. Avoids unsupported claims70% pass
    The claim 'A growth PM in a senior role is likely to be exactly that kind of person' is presented as fact without evidence and is not labelled as an assumption.
    Opus 5.5 · Claude · A loop for the first growth PM
  3. Identifies material uncertainty72% pass
    The output does not name unknowns that could change the hiring decision or the loop design, nor does it say how they would be resolved.
    Sonnet 5.5 · API · A loop for the first growth PM

How it’s graded

The checks come from what the best product leaders have said about doing this job well on Lenny’s Podcast. Each one names the guest it comes from: follow a name to the idea on the Lenny’s Podcast wiki.

  1. Judges on evidence, not presence

    Does the output judge candidates, or design the process to judge them, on specific evidence of results they drove and how they worked with their team, rather than impressions such as presence, confidence or polish?

    Passes when Asks for or weighs concrete evidence: the metric a candidate moved, the part they played, how they shared credit, what they learned from a failure. Impressions are treated as weak signals.

    Lenny’s PodcastThe wiki on Hiring PMs

  2. Defines good for this role first

    Does the output set out what good looks like for this specific role (the competencies that matter most at this level, and what strong and weak look like) before judging candidates or designing interviews?

    Passes when Names the competencies weighted for this role and level, with strong and weak signals, and the judgement or the process follows from them.

    Petra WilleLenny’s Podcast · Hiring PMs

  3. Keeps each judgement independent

    Does the output keep each interviewer's judgement independent (written down before the group discusses it), and guard against the loudest voice or the first speaker deciding?

    Passes when Requires or relies on written, independent evaluations before discussion, and discounts views that changed under group pressure.

    Lenny’s PodcastThe wiki on Hiring PMs

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 3 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're Director of Product at Medvia, and you're the hiring manager for a Group PM. The debrief was yesterday. Write your recommendation to Rosa Lindqvist, our VP Product, in no more than 1,000 words: who to hire (Dev, Joy, or neither), the evidence for it, the risks, and what you'd check before making the offer. The CEO wants a decision by Friday. The interview notes and everything else are below.

What the model was given6 items: The role and the scorecard (agreed before interviews began), Dev Malhotra: interview notes, Joy Adeyemi: interview notes, Scores (1 to 4, entered after the debrief discussion), References, Compensation
The role and the scorecard (agreed before interviews began)Group PM for patient payments, leading three PMs. Must set the strategy for patient payments, lift the online payment rate (now 41%), and lead PMs while working closely with Finance and Compliance. Weighting: execution and influence 35%, product sense 25%, analytics 20%, people leadership 20%.
Dev Malhotra: interview notesProduct sense (Head of Design): 'Brilliant. Redesigned our statement flow live, lots of ideas, very creative.' Notes show six feature ideas in the first five minutes, before asking who pays and why. Analytics (Data lead): 'Strong.' Notes: asked what metric he moved at his last job, he said 'we grew the payments platform massively'; pressed, 'the team shipped 40+ features'. No number. Leadership (VP Engineering): 'Confident, inspiring.' Notes: every story in the first person; said his last PM team 'struggled to keep up with me'. Presentation: 'the best deck we've ever seen'.
Joy Adeyemi: interview notesProduct sense (Head of Design): 'Solid, slower.' Notes: spent 15 minutes asking who pays and why they don't pay online, then proposed one change: text-to-pay for the 58% of patients who pay by phone, with its trade-offs. Analytics (Data lead): notes say she raised payment completion at her last company from 33% to 47% over 18 months, and walked through an experiment that failed and what it taught her. Leadership (VP Engineering): notes say she coached a struggling PM into a different role, promoted another, and named her engineers when describing wins. Debrief comments: 'Not much presence.' 'Not sure she'd command the room with the CFO.'
Scores (1 to 4, entered after the debrief discussion)Dev: 4, 4, 4, 4. Joy: 3, 3, 3, 3. The Head of Design spoke first in the debrief. The VP Engineering's own notebook, written before the debrief, says: 'Joy: 4, strongest leader we've seen.' They entered a 3 after the discussion.
ReferencesDev: two references, both people who report to him now. Rosa knows someone at his previous company, who says: 'great in front of execs; his team turned over a lot.' Joy: her former manager and a peer, both strongly positive. Nobody has asked either reference about working with a CFO.
CompensationBand $220k to $250k. Dev is asking for $245k, Joy for $230k.
What a strong answer doesThe answer key the graders mark against

Recommends Joy, or at minimum not Dev on this evidence. Weighs both against the agreed scorecard rather than presence or the deck: Joy gives a metric she moved (33% to 47%), learns from failure, credits her team and coaches PMs, which fits execution, analytics and people leadership; Dev couldn't name a metric he moved, frames everything as 'I', jumped to features before asking about users, and the back channel reports high team turnover, all red flags. Points out that the scores were entered after a discussion the Head of Design opened, and the VP Engineering moved Joy from 4 to 3, so they show anchoring, not independent judgement. Treats 'presence with the CFO' as the one real open question and proposes a targeted reference check on how Joy worked with Finance leaders before the offer. Suggests the next loop collect written scores before the debrief.

Critical failures (cap the score at 40)
  • Recommends hiring Dev on the strength of the debrief scores or the presentation, without addressing that he couldn't name a metric he moved
Task

Staff-level · v1.0 · synthetic · healthcare, fintech, hiring, staff-level

The outputs

7 setups answered this brief. Pick up to three to read side by side.

GPT-6 LunaAPI100Usable with a quick editGPT-6.1 SolAPI100Usable with a quick editOpus 5.5Claude100Usable with a quick editSonnet 5.5API100Usable with a quick editGPT-6 AstraChatGPT100Usable with a quick editGemini 3.8 FlashAPI98Usable with a quick editGemini 3.5 Flash-LiteGemini88Usable with a quick edit
GPT-6 LunaAPI

Usable with a quick edit12 of 12 checks passed

The output is well-structured, evidence-based, and actionable; a PM could use it with minor wording edits without re-checking evidence or redoing reasoning.

Output

To: Rosa Lindqvist, VP Product Subject: Group PM recommendation — Joy Adeyemi

Recommendation: Hire Joy, subject to focused reference and influence checks before the offer. She has the stronger evidence against the role’s core needs: measurable payment outcomes, thoughtful product judgment, and the ability to lead through other people. The CEO needs a decision by Friday; I would not let the debrief scores alone drive it.

Evidence

  • Execution and influence (35%): Joy gave a specific result from her last company—payment completion rose from 33% to 47% over 18 months—and explained a failed experiment and what she learned. Her leadership examples showed investment in others: coaching a struggling PM into a different role and promoting another. Dev’s evidence was less concrete: he described platform growth and 40+ shipped features but supplied no outcome metric when pressed. His stories were consistently framed around his own contribution, and he said his team struggled to keep up with him.
  • Product sense (25%): Joy investigated who pays and why before proposing text-to-pay for the 58% who currently pay by phone, including trade-offs. That is relevant to our 41% online payment rate and reflects problem-first judgment. Dev’s creativity and presentation were strengths, but six ideas in the first five minutes—before clarifying who pays and why—are a concern for this role.
  • Analytics (20%): Joy’s quantified result and discussion of a failed experiment are stronger evidence of analytical ownership. Dev did not provide a measurable result.
  • People leadership (20%): Joy’s examples demonstrate coaching, development, and shared credit. Dev’s confidence and presentation are positives, but they do not offset the limited evidence of developing a team or his comment about the team struggling to keep up.

Risks and scorecard

Joy’s main risk is whether she can influence senior stakeholders, particularly the CFO and Finance. The “not much presence” and “command the room” comments are concerns to test, not proof that she cannot do it; we should assess her ability to make a clear, evidence-based case rather than equate influence with a particular presentation style.

Dev may be compelling with executives, but we need to understand the team-turnover signal and whether his leadership style scales. His two references are current direct reports, which may limit candor. His former-company reference’s report of high turnover is important but unverified.

I would treat the 4/4/4/4 versus 3/3/3/3 ratings as weak evidence. The VP Engineering’s pre-debrief note called Joy the strongest leader seen and rated her a 4, then the score became a 3 after discussion; the Head of Design also spoke first. This does not establish why the ratings changed, but it is reason to rely on the underlying evidence and check for independent calibration.

Before making the offer

  1. Ask Joy’s former manager and peer specifically about her influence with Finance/CFO-level stakeholders, including a difficult disagreement and the outcome.
  2. Ask for one additional Joy reference who can speak to cross-functional execution; ask Dev’s references about team turnover, feedback, and how he develops PMs. Seek candid, comparable evidence where possible.
  3. In a short structured conversation, ask Joy to make the case for a payment-rate strategy to a skeptical CFO. Evaluate clarity, use of evidence, and handling of challenge—not polish alone.
  4. Confirm Joy’s 33%→47% result, her role in it, and the scope of her people leadership.

If those checks reveal a material gap in Joy’s influence or ownership, I would choose neither rather than default to Dev on presentation strength. Otherwise, proceed with Joy at her requested $230k, within band.

Grades and run details

Decision model 100 · LLM judge 13 of 13 checks
Decision model checks
  • passUses the supplied evidence correctly87%
  • passAddresses the actual decision100%
  • passRespects explicit constraints76%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims60%
  • passProduces the required deliverable99%
  • passWeighs against the agreed scorecard100%
  • passCatches the red flags in Dev's evidence100%
  • passSays what would change the call100%
  • passJudges on evidence, not presence100%
  • passDefines good for this role first69%
  • passKeeps each judgement independent87%
Run
Run
#1
API response time
12 s
Submitted
5 Oct 2026
Gemini 3.8 FlashAPI

Usable with a quick edit12 of 12 checks passed

The memo is complete, evidence-based, and directly usable by the VP Product with at most minor wording tweaks.

Output

To: Rosa Lindqvist, VP Product From: Director of Product Date: [Current Date] Subject: Recommendation: Group PM, Patient Payments

---

Recommendation

Hire Joy Adeyemi. Do not hire Dev Malhotra.

While the post-debrief scores show Dev at straight 4s and Joy at straight 3s, the underlying interview notes tell the opposite story. Dev embodies a classic “style over substance” trap that poses severe risks to our culture and business metrics. Joy is an evidence-driven, outcome-oriented product leader whose track record directly maps to our core objective: moving our online payment rate from 41%.

Before extending an offer by Friday, we need to complete one targeted reference check on Joy’s executive influence with Finance.

---

Debrief Context & Scorecard Calibration

The debrief scores were distorted by presentation polish and group dynamics: * Anchoring: The Head of Design spoke first, captivated by Dev’s live wireframing and presentation deck. * VP Engineering flip: Before the group discussion, the VP of Engineering wrote in his private notes: “Joy: 4, strongest leader we’ve seen.” He only downgraded her to a 3 after the room leaned into Dev’s charisma. * Scorecard alignment: Our mandate is lifting payment conversion, leading three PMs, and collaborating with Compliance and Finance. When evaluated against our weighted scorecard, Joy clearly outperforms Dev.

---

The Evidence: Why Joy

#### 1. Analytics (20%) & Product Sense (25%) * Joy delivers outcomes; Dev counts outputs. Joy grew payment completion from 33% to 47% over 18 months at her prior company—almost the exact challenge we face with our 41% baseline. She demonstrated authentic experimentation rigor by walking through a failed test and the specific learnings extracted. * Methodology vs. guesswork: In product sense, Dev generated six feature ideas in five minutes without asking a single question about our users. Joy spent the first 15 minutes diagnosing who pays and why they do not pay online. Only then did she offer a targeted solution: text-to-pay targeting the 58% of patients who pay by phone, accompanied by nuanced trade-offs. * In contrast, when pressed on metrics, Dev claimed he “grew the platform massively,” pointing solely to shipping “40+ features.” He showed zero metric accountability.

#### 2. People Leadership (20%) * Retention vs. Turnover: Joy demonstrated servant leadership—she credited her engineers, coached a struggling PM into a better-fitting role, and promoted another. * Dev spoke entirely in the first person (“I”) and remarked that his past PM team “struggled to keep up with me.” Critically, your backchannel reference confirmed: “great in front of execs; his team turned over a lot.” * Bringing Dev in to manage three PMs is an unacceptable retention risk. Joy will develop and retain our talent.

#### 3. Execution & Influence (35%) * Dev’s influence relies on performative deck-building. While impressive on the surface, feature-heavy roadmaps disconnected from metrics collapse when scrutinized by cross-functional peers. * Joy influences with data, structural diagnosis, and trade-offs. In a domain governed by Compliance and Finance, rigorous substance will earn far more durable trust than slick slides.

---

Key Risks & Mitigation

Risk 1: Joy’s perceived “lack of presence” with the CFO. * The concern: Debrief feedback questioned whether she can “command the room” with Finance. * The reality: Introversion or deliberate pacing is often misread as a lack of authority. CFOs in healthcare payments do not want flash; they want airtight unit economics, regulatory awareness, and defensible data. Joy’s structured, trade-off-led approach is well-suited to Finance, but we must verify this. * Mitigation: We will run a targeted reference check specifically addressing her relationship with senior finance stakeholders (detailed below).

Risk 2: Dev’s counter-risk (The Cost of the Alternative). * Hiring Dev risks churn across all three PM reports within 12 months, feature bloat that fails to budge our 41% rate, and friction with Compliance when rapid-fire feature shipping bypasses regulatory rigor.

---

What to Check Before Making the Offer

We can resolve remaining questions and finalize an offer before Friday’s deadline:

1. Targeted Reference Check (Finance/CFO Influence): Neither candidate had their references asked about working with Finance. Today, I will call Joy’s former manager and peer to ask: * “How did Joy manage pushback from Finance, Legal, or Compliance?” * “Describe a time Joy had to present a high-stakes proposal or trade-off to the CFO/executive team. How was it received?” 2. Optional 30-Minute Touchpoint with our CFO (Thursday): If the reference leaves any ambiguity, set up an informal 30-minute working chat between Joy and our CFO focused on a single payment reconciliation scenario. This will give our CFO direct input and dispel presence-based bias.

---

Compensation

  • Band: $220,000 – $250,000
  • Joy’s Ask: $230,000
  • Dev’s Ask: $245,000

Joy’s ask sits comfortably near the midpoint of our band, leaving room for merit increases while saving $15,000 in baseline compensation compared to Dev.

Next Steps

With your sign-off, I will conduct Joy’s targeted Finance reference check this afternoon, brief the VP of Engineering on our rationale, and have an offer letter ready for Joy at $230k by Thursday morning to meet the CEO’s Friday deadline.

Grades and run details

Decision model 96 · LLM judge 13 of 13 checks
Decision model checks
  • passUses the supplied evidence correctly41%
  • passAddresses the actual decision93%
  • passRespects explicit constraints51%
  • passIdentifies material uncertainty89%
  • partialAvoids unsupported claims42%
  • passProduces the required deliverable98%
  • passWeighs against the agreed scorecard99%
  • passCatches the red flags in Dev's evidence100%
  • passSays what would change the call84%
  • passJudges on evidence, not presence96%
  • passDefines good for this role first64%
  • passKeeps each judgement independent76%
Run
Run
#1
API response time
22 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

All got right 12

Uses the supplied evidence correctlyRightRight
GPT-6 Luna · API

All factual claims about the current situation are directly from the brief, with no inventions.

Gemini 3.8 Flash · API

All factual claims about the current situation are directly supported by the brief or supplied context.

Addresses the actual decisionRightRight
GPT-6 Luna · API

Recommends hiring Joy early, framed for Rosa, and states that material gaps in influence or ownership would change the call to neither.

Gemini 3.8 Flash · API

The output unambiguously recommends hiring Joy and not Dev, and says the decision is conditional on a targeted reference check on Joy's finance influence.

Respects explicit constraintsRightRight
GPT-6 Luna · API

Memo to Rosa, under 1000 words, respects the Friday deadline by making a decision now with pre-offer checks.

Gemini 3.8 Flash · API

The output is a recommendation memo to Rosa Lindqvist, well under 1,000 words, and respects the form and reader.

Identifies material uncertaintyRightRight
GPT-6 Luna · API

Names unknowns (Joy's CFO influence, Dev's turnover, debrief score reliability) and specifies how to resolve them with targeted checks.

Gemini 3.8 Flash · API

It names the open question about Joy's influence with finance leaders and proposes a specific reference check and optional CFO touchpoint to resolve it.

Avoids unsupported claimsRightRight
GPT-6 Luna · API

Interpretations are clearly labelled as risks or weak evidence, not presented as established facts.

Gemini 3.8 Flash · API

The output's causal claims (e.g., scores distorted by presentation) are presented as reasoned conclusions from the evidence, not as unsupported facts.

Produces the required deliverableRightRight
GPT-6 Luna · API

Complete recommendation memo with evidence, risks, and actionable next steps, usable as is.

Gemini 3.8 Flash · API

The memo is complete, addressed to the VP Product, within the word limit, and actionable with a clear recommendation and next steps.

Weighs against the agreed scorecardRightRight
GPT-6 Luna · API

Weighs both candidates against the agreed competencies (execution, product sense, analytics, people leadership) and notes that presence and deck are not on it.

Gemini 3.8 Flash · API

The recommendation explicitly weighs both candidates against the agreed scorecard competencies and notes that presence and the deck are not on it.

Catches the red flags in Dev's evidenceRightRight
GPT-6 Luna · API

Names at least three red flags: no metric moved, first-person credit and team struggled to keep up, jumping to features, and back-channel turnover report.

Gemini 3.8 Flash · API

It names at least four red flags: no metric moved, first-person credit and team struggled to keep up, jumping to features before user questions, and back-channel turnover report.

Says what would change the callRightRight
GPT-6 Luna · API

Identifies Joy's influence with CFO as the open question and proposes specific reference checks and a structured conversation to test it.

Gemini 3.8 Flash · API

It identifies the open question about Joy's influence with senior finance leaders and proposes a targeted reference check and optional CFO touchpoint, implying a negative result would change the decision.

Judges on evidence, not presenceRightRight
GPT-6 Luna · API

Judges on concrete evidence (metric moved, team credit, learning from failure) and treats impressions like presence as weak signals.

Gemini 3.8 Flash · API

The output judges on concrete evidence (metric moved, team credit, learning from failure) and treats presence as a weak signal to be verified.

Defines good for this role firstRightRight
GPT-6 Luna · API

Sets out the weighted competencies and what strong evidence looks like before comparing candidates.

Gemini 3.8 Flash · API

It sets out the role's weighted competencies and uses them to define what good looks like before evaluating the candidates.

Keeps each judgement independentRightRight
GPT-6 Luna · API

Relies on the VP Engineering's independent pre-debrief note and discounts group-influenced scores, keeping judgement independent.

Gemini 3.8 Flash · API

It uses the VP Engineering's independent pre-debrief note to discount group-influenced scores, effectively keeping that judgement independent.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Provisional The checklist is still being calibrated against our PM.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6.1 SolAPI94.4100.03None
2Sonnet 5.5API95.896.22None
3Gemini 3.8 FlashAPI89.692.32None
4GPT-6 AstraChatGPT95.887.23None
5Opus 5.5Claude91.787.23None
6GPT-6 LunaAPI93.184.63None
7Gemini 3.5 Flash-LiteGemini75.064.13None

The PM job

Hiring product managers.

Why it matters

A bad PM hire costs a team a year. Most loops reward polish and presence, and the debrief goes to whoever speaks first.

What good looks like

  • Defines what good looks like for this role before judging anyone
  • Judges on evidence of results and how they work with a team
  • Tests the real job, not a rehearsed framework
  • Keeps each interviewer's judgement independent

Deliberately not measured

  • Employment law
  • Compensation benchmarking
Capability tested

Hiring judgement

The failure we’re looking for

Hiring for presence and polish over evidence

Grading

Decision model and LLM judge, calibrated against a blind PM review