Tasks / Leadership

Hire a PM

Can the model design a hiring process that finds the right PM, and make the call on real candidates from the evidence?

Measures the modelTask type v1.1 · 3 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 19 graded outputs by 7 models. 79% were usable with at most a quick edit.

Reliably right

Produces the required deliverable100% pass
Complete recommendation memo with evidence, risks, and actionable next steps, usable as is.
GPT-6 Luna · API · Two finalists, one Group PM role
Judges on evidence, not presence100% pass
Judges on concrete evidence (metric moved, team credit, learning from failure) and treats impressions like presence as weak signals.
GPT-6 Luna · API · Two finalists, one Group PM role
Tests what the last hire failed at100% pass
The loop includes an influence simulation with the Head of Sales and a scorecard dimension on influence without authority, directly testing what the last hire failed at.
GPT-6.1 Sol · API · A loop for the first growth PM

Where it slips

Spots the interviewer pattern65% pass
Does not notice that VP Engineering scores big-tech candidates higher and those hires were rated lower; only reports a negative correlation without the pattern.
GPT-6 Luna · API · What our interviews predict
Avoids unsupported claims70% pass
The claim 'A growth PM in a senior role is likely to be exactly that kind of person' is presented as fact without evidence and is not labelled as an assumption.
Opus 5.5 · Claude · A loop for the first growth PM
Identifies material uncertainty72% pass
The output does not name unknowns that could change the hiring decision or the loop design, nor does it say how they would be resolved.
Sonnet 5.5 · API · A loop for the first growth PM

How it’s graded

The checks come from what the best product leaders have said about doing this job well on Lenny’s Podcast. Each one names the guest it comes from: follow a name to the idea on the Lenny’s Podcast wiki.

Judges on evidence, not presence
Does the output judge candidates, or design the process to judge them, on specific evidence of results they drove and how they worked with their team, rather than impressions such as presence, confidence or polish?
Passes when Asks for or weighs concrete evidence: the metric a candidate moved, the part they played, how they shared credit, what they learned from a failure. Impressions are treated as weak signals.
Lenny’s PodcastThe wiki on Hiring PMs
Defines good for this role first
Does the output set out what good looks like for this specific role (the competencies that matter most at this level, and what strong and weak look like) before judging candidates or designing interviews?
Passes when Names the competencies weighted for this role and level, with strong and weak signals, and the judgement or the process follows from them.
Petra WilleLenny’s Podcast · Hiring PMs
Keeps each judgement independent
Does the output keep each interviewer's judgement independent (written down before the group discusses it), and guard against the loudest voice or the first speaker deciding?
Passes when Requires or relies on written, independent evaluations before discussion, and discounts views that changed under group pressure.
Lenny’s PodcastThe wiki on Hiring PMs

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 3 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're helping Amara Osei, VP Product at Copperline, hire our first Growth PM (a senior role). She's shared her draft interview loop and asked you to redesign it. Write the loop you'd run (each round: who runs it, what it tests, how long it takes), the scorecard (what strong and weak look like for each thing we're testing), and how we'll make the decision at the end. No more than 900 words. What we know is below.

What the model was given6 items: About Copperline, The role, Amara's draft loop, The last two PM hires, From the recruiter, Who can interview

About CopperlineInvoicing and payments software for small accountancy firms. 60 people. Trial to paid conversion is 9%.

The roleOwns trial conversion and expansion revenue. Works with three engineers and a designer, reports to Amara, and has to ship experiments every week. Needs Sales and Marketing to change how they work, without managing them.

Amara's draft loop1. Recruiter screen. 2. Take-home: 'design a growth strategy for Copperline', a 10-page deck, five days to complete. 3. Presentation to a panel of six. 4. Culture fit with the CEO: 'would I grab a beer with them?' 5. Debrief: everyone discusses, then votes by show of hands. There's no scorecard.

The last two PM hiresOne left after five months: 'couldn't get Sales to change anything'. The other is doing well. Both sets of interview notes say mainly 'great presence' and 'very smart'.

From the recruiterOf the last 40 candidates given the take-home, 14 withdrew, saying they didn't have time. Nine of the 14 had young children or demanding current roles.

Who can interviewAmara, the Head of Engineering, a senior designer, the Head of Sales, a data analyst and the CEO. Each can give at most two hours to this hire. Amara wants an offer out within three weeks of the first interview.

What a strong answer doesThe answer key the graders mark against

A loop built on a scorecard written first: for a senior growth PM, experimentation and analytics, execution, and influence without authority (weighted up, since the last hire failed on it), with strong and weak signals for each. Rounds test the real job: a live diagnosis of Copperline's own trial funnel instead of a five-day generic deck (which loses candidates, mostly parents and people in demanding roles), a behavioural round on past experiments and metrics actually moved, and an influence round run by the Head of Sales. Replaces the 'grab a beer' culture fit with a structured values or collaboration conversation. Each interviewer writes their evaluation before the debrief, and the decision is made against the scorecard, not a show of hands. Fits each interviewer's two hours and the three-week target, and adds reference checks.

Critical failures (cap the score at 40)

Keeps a culture-fit test based on whether the interviewer would like a beer with the candidate

Task

Mid-level · v1.0 · synthetic · B2B SaaS, hiring, growth

The outputs

7 setups answered this brief. Pick up to three to read side by side.

GPT-6.1 SolAPI96Usable with a quick edit Sonnet 5.5API92Usable with a quick edit GPT-6 AstraChatGPT92Usable with a quick edit Opus 5.5Claude88Usable with a quick edit GPT-6 LunaAPI87Usable with a quick edit Gemini 3.8 FlashAPI85Usable with a quick edit Gemini 3.5 Flash-LiteGemini79Usable with a quick edit

Gemini 3.5 Flash-LiteGemini

Usable with a quick edit9 of 12 checks passed

The output is clear, actionable, and covers all required parts; a quick edit to add conditions or label interpretations would make it fully ready.

Output

Amara—here is the redesigned interview loop. Your draft had a fatal flaw: the 10-page take-home is a diversity filter (driving away parents and senior operators) and tests academic strategy over execution. Furthermore, our last hiring failure showed we hired for "presence" rather than the ability to influence Sales.

Here is a 3-week, high-signal loop designed to test gritty experimentation, cross-functional persuasion, and data fluency.

---

Part 1: The Interview Loop

Round 1: Recruiter Screen * Who: Recruiter (30 mins) * Tests: Baseline communication, compensation alignment, current trial-to-paid context, and general interest in B2B SaaS.

Round 2: The Operating Screen (Live Case) * Who: Amara, VP Product (45 mins, video) * Tests: Problem decomposition, metric intuition, and ability to move fast. * Format: No slides. We present our 9% trial-to-paid conversion rate and a specific funnel drop-off. We spend 30 minutes live-whiteboarding how they would diagnose and attack it within their first 30 days.

Round 3: The Cross-Functional Influence & Craft Deep-Dive * Who: Head of Engineering & Senior Designer (45 mins, video) * Tests: Can this person ship weekly? Do they respect engineering constraints? Do they partner well with design on user psychology? * Format: Deep-dive into a past project where they shipped rapid iterations. How do they handle tech debt vs. speed tradeoffs?

Round 4: The Sales & Data Crucible * Who: Head of Sales & Data Analyst (45 mins, video) * Tests: Crucial. Can they influence Sales without authority? Can they audit data to find buried conversion leaks? * Format: A roleplay. The Head of Sales plays a skeptical account executive who thinks the PM's new trial-onboarding flow is stealing their pipeline. The candidate must handle the pushback, find common ground, and align incentives.

Round 5: Founder Alignment & Values * Who: CEO (30 mins, in-person or video) * Tests: Velocity mindset, ownership mentality, and cultural add (dropping the "beer test" bias, which screens for homogeneity). * Format: Discussion on Copperline’s mission, how they handle failure, and what autonomy looks like here.

---

Part 2: The Scorecard

We will grade candidates on four core competencies using a 1–4 scale (1 = Unacceptable, 4 = Exceptional).

#### 1. Rigorous Experimentation & Velocity * Strong (3–4): Relentlessly focused on the funnel. Talks easily about cadence, MVP scoping, and shipping weekly. Bases ideas on user behavior, not just "best practices." * Weak (1–2): Prefers massive, multi-month redesign projects. Hand-waves about metrics or treats data as an afterthought.

#### 2. Cross-Functional Influence (Specifically with Sales) * Strong (3–4): Listens to Sales objections, understands quota pressure, and designs incentives that make Sales want to use the new product motions. Uses data to persuade. * Weak (1–2): Views Sales as "resistant" or "the enemy." Relies on positional authority ("Amara said we're doing this") rather than building trust.

#### 3. Data Fluency * Strong (3–4): Can spot a vanity metric instantly. Identifies cohort retention issues and funnel bottlenecks unprompted. Comfortable querying or partnering deeply with analysts. * Weak (1–2): Confuses activity metrics (clicks) with outcome metrics (expansion revenue). Gets lost when looking at raw conversion funnels.

#### 4. Product Craft & Empathy (Accountants) * Strong (3–4): Quickly grasps the nuanced workflow of a small accountancy firm. Designs intuitive onboarding that respects the user's time. * Weak (1–2): Generic "growth hacking" mentality that applies consumer tactics (spammy pop-ups, dark patterns) to professional B2B software.

---

Part 3: Decision-Making Process

To avoid the vague "they seemed smart" trap, we will use a structured debrief.

Independent Scoring: Within 24 hours of their final round, every interviewer must submit their independent scores and notes into a shared doc before reading anyone else's.
The Debrief Meeting (30 mins): Led by Amara. We do not "vote by show of hands." Instead, each interviewer speaks to their specific competency (e.g., Sales speaks to Round 4, Eng speaks to Round 3).
The Bar: A hire requires a minimum score of 3 across all four competencies, with no score of 1 or 2 in Cross-Functional Influence. If Sales or Engineering flags a collaboration red flag, it's a hard veto.

By cutting the take-home, we keep top talent in the funnel. By replacing the "beer test" with a Sales roleplay, we ensure our next PM doesn't repeat the exact failure mode of our last one.

Claims the judge couldn’t find in the brief

The beer test screens for homogeneity.

Grades and run details

Decision model 79 · LLM judge 10 of 13 checks

Decision model checks

passUses the supplied evidence correctly39%
passAddresses the actual decision81%
partialRespects explicit constraints38%
failIdentifies material uncertainty70%
failAvoids unsupported claims33%
passProduces the required deliverable53%
passTests what the last hire failed at100%
passFixes the take-home's cost to candidates100%
passFits the people and the time24%
passJudges on evidence, not presence67%
passDefines good for this role first84%
passKeeps each judgement independent99%

Run

Run: #1
Time to output: 6 s
Submitted: 5 Oct 2026

GPT-6 LunaAPI

Usable with a quick edit9 of 12 checks passed

The output is complete and actionable; a quick edit to mention the recruiter's withdrawal data would make it fully aligned with the brief.

Output

Interview loop

Run this over 2–3 weeks. Skip the five-day take-home, large presentation panel, and “beer” fit interview. Share the same expectations and questions with every candidate; offer scheduling flexibility and accommodations.

1. Recruiter screen — 30 minutes Tests role understanding, motivation, and practical alignment: the scope, weekly shipping pace, and need to influence Sales and Marketing without authority. Not a “polish” screen.

2. Amara, VP Product — 60 minutes Tests growth ownership and product judgment. Ask for a detailed example of a funnel or revenue problem the candidate owned: diagnosis, choices, experiments, results, and what they learned. Probe how they balance conversion, customer trust, and expansion.

3. Live growth work sample — 60 minutes Send a one-page brief and synthetic funnel data 24 hours ahead; no deck or advance work required. Spend 30 minutes with the data analyst diagnosing the funnel and proposing a measurable experiment; spend 30 minutes with the Head of Engineering and senior designer scoping how to ship and learn from it. Use the same prompt and time limits for all candidates.

4. Head of Sales — 45 minutes Structured roleplay: Sales is reluctant to change its trial follow-up because reps are busy and doubt the data. Ask the candidate to understand the objection, make a case, and agree a practical next step. Then probe a real example of influencing a team they did not manage.

The CEO does not need a separate round: “culture fit” is not a hiring criterion, and the role’s key cross-functional test is better assessed directly with Sales, Engineering, and Design.

Scorecard

Use a 1–4 rating for each dimension: 1 = clear concern, 2 = below bar, 3 = meets bar, 4 = unusually strong. Record evidence, not impressions.

Dimension	Strong evidence	Weak evidence
Growth ownership and outcomes (Amara)	Personally owned a meaningful funnel or revenue outcome; explains decisions, results, attribution limits, and lessons.	Describes activity rather than impact; overclaims credit or cannot explain what changed.
Customer and product judgment (Amara)	Understands small-firm workflows and trust needs; weighs conversion against customer value and expansion.	Offers generic SaaS tactics or pushes short-term conversion at the expense of trust.
Data diagnosis and experiment design (work sample)	Finds plausible bottlenecks and segments; proposes a focused test with a clear hypothesis, success metric, guardrail, and learning plan.	Jumps to a solution, misreads the data, uses vanity metrics, or cannot say what result would change their mind.
Shipping and learning cadence (work sample)	Scopes a feasible experiment, identifies dependencies and tradeoffs, and shows how the team can learn and iterate weekly.	Proposes a big-bang project, ignores constraints, or treats learning as something to do after launch.
Influence without authority (Sales)	Listens for incentives and concerns, uses evidence appropriately, adapts the ask, and secures a credible next step.	Blames Sales, relies on escalation or authority, or cannot turn disagreement into action.
Cross-functional partnership (work sample)	Makes clear tradeoffs with Engineering and Design, invites their expertise, and aligns on ownership and delivery.	Treats partners as order-takers or leaves scope and responsibilities vague.

Decision

Interviewers submit ratings and specific evidence independently before a 30-minute debrief; do not vote by show of hands. Amara facilitates, checks for evidence behind claims, and makes the hiring decision. Do not use “presence,” “smart,” or general “fit” as evidence.

A hire must score at least 3 on influence, experiment design, and shipping, with no score below 2 in any dimension. Do not average away a weakness in a core requirement. If evidence is genuinely missing, identify one focused follow-up rather than adding a broad round. Give candidates a timely decision and close the loop within three weeks of the first interview.

Grades and run details

Decision model 96 · LLM judge 10 of 13 checks

Decision model checks

passUses the supplied evidence correctly51%
passAddresses the actual decision82%
passRespects explicit constraints26%
partialIdentifies material uncertainty64%
passAvoids unsupported claims22%
passProduces the required deliverable78%
passTests what the last hire failed at100%
passFixes the take-home's cost to candidates68%
passFits the people and the time64%
passJudges on evidence, not presence100%
passDefines good for this role first86%
passKeeps each judgement independent98%

Run

Run: #1
API response time: 39 s
Submitted: 5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.