Tasks / Leadership

Hire a PM

Can the model design a hiring process that finds the right PM, and make the call on real candidates from the evidence?

Measures the modelTask type v1.1 · 3 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 19 graded outputs by 7 models. 79% were usable with at most a quick edit.

Reliably right

Produces the required deliverable100% pass
Complete recommendation memo with evidence, risks, and actionable next steps, usable as is.
GPT-6 Luna · API · Two finalists, one Group PM role
Judges on evidence, not presence100% pass
Judges on concrete evidence (metric moved, team credit, learning from failure) and treats impressions like presence as weak signals.
GPT-6 Luna · API · Two finalists, one Group PM role
Tests what the last hire failed at100% pass
The loop includes an influence simulation with the Head of Sales and a scorecard dimension on influence without authority, directly testing what the last hire failed at.
GPT-6.1 Sol · API · A loop for the first growth PM

Where it slips

Spots the interviewer pattern65% pass
Does not notice that VP Engineering scores big-tech candidates higher and those hires were rated lower; only reports a negative correlation without the pattern.
GPT-6 Luna · API · What our interviews predict
Avoids unsupported claims70% pass
The claim 'A growth PM in a senior role is likely to be exactly that kind of person' is presented as fact without evidence and is not labelled as an assumption.
Opus 5.5 · Claude · A loop for the first growth PM
Identifies material uncertainty72% pass
The output does not name unknowns that could change the hiring decision or the loop design, nor does it say how they would be resolved.
Sonnet 5.5 · API · A loop for the first growth PM

How it’s graded

The checks come from what the best product leaders have said about doing this job well on Lenny’s Podcast. Each one names the guest it comes from: follow a name to the idea on the Lenny’s Podcast wiki.

Judges on evidence, not presence
Does the output judge candidates, or design the process to judge them, on specific evidence of results they drove and how they worked with their team, rather than impressions such as presence, confidence or polish?
Passes when Asks for or weighs concrete evidence: the metric a candidate moved, the part they played, how they shared credit, what they learned from a failure. Impressions are treated as weak signals.
Lenny’s PodcastThe wiki on Hiring PMs
Defines good for this role first
Does the output set out what good looks like for this specific role (the competencies that matter most at this level, and what strong and weak look like) before judging candidates or designing interviews?
Passes when Names the competencies weighted for this role and level, with strong and weak signals, and the judgement or the process follows from them.
Petra WilleLenny’s Podcast · Hiring PMs
Keeps each judgement independent
Does the output keep each interviewer's judgement independent (written down before the group discusses it), and guard against the loudest voice or the first speaker deciding?
Passes when Requires or relies on written, independent evaluations before discussion, and discounts views that changed under group pressure.
Lenny’s PodcastThe wiki on Hiring PMs

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 3 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're helping Amara Osei, VP Product at Copperline, hire our first Growth PM (a senior role). She's shared her draft interview loop and asked you to redesign it. Write the loop you'd run (each round: who runs it, what it tests, how long it takes), the scorecard (what strong and weak look like for each thing we're testing), and how we'll make the decision at the end. No more than 900 words. What we know is below.

What the model was given6 items: About Copperline, The role, Amara's draft loop, The last two PM hires, From the recruiter, Who can interview

About CopperlineInvoicing and payments software for small accountancy firms. 60 people. Trial to paid conversion is 9%.

The roleOwns trial conversion and expansion revenue. Works with three engineers and a designer, reports to Amara, and has to ship experiments every week. Needs Sales and Marketing to change how they work, without managing them.

Amara's draft loop1. Recruiter screen. 2. Take-home: 'design a growth strategy for Copperline', a 10-page deck, five days to complete. 3. Presentation to a panel of six. 4. Culture fit with the CEO: 'would I grab a beer with them?' 5. Debrief: everyone discusses, then votes by show of hands. There's no scorecard.

The last two PM hiresOne left after five months: 'couldn't get Sales to change anything'. The other is doing well. Both sets of interview notes say mainly 'great presence' and 'very smart'.

From the recruiterOf the last 40 candidates given the take-home, 14 withdrew, saying they didn't have time. Nine of the 14 had young children or demanding current roles.

Who can interviewAmara, the Head of Engineering, a senior designer, the Head of Sales, a data analyst and the CEO. Each can give at most two hours to this hire. Amara wants an offer out within three weeks of the first interview.

What a strong answer doesThe answer key the graders mark against

A loop built on a scorecard written first: for a senior growth PM, experimentation and analytics, execution, and influence without authority (weighted up, since the last hire failed on it), with strong and weak signals for each. Rounds test the real job: a live diagnosis of Copperline's own trial funnel instead of a five-day generic deck (which loses candidates, mostly parents and people in demanding roles), a behavioural round on past experiments and metrics actually moved, and an influence round run by the Head of Sales. Replaces the 'grab a beer' culture fit with a structured values or collaboration conversation. Each interviewer writes their evaluation before the debrief, and the decision is made against the scorecard, not a show of hands. Fits each interviewer's two hours and the three-week target, and adds reference checks.

Critical failures (cap the score at 40)

Keeps a culture-fit test based on whether the interviewer would like a beer with the candidate

Task

Mid-level · v1.0 · synthetic · B2B SaaS, hiring, growth

The outputs

7 setups answered this brief. Pick up to three to read side by side.

GPT-6.1 SolAPI96Usable with a quick edit Sonnet 5.5API92Usable with a quick edit GPT-6 AstraChatGPT92Usable with a quick edit Opus 5.5Claude88Usable with a quick edit GPT-6 LunaAPI87Usable with a quick edit Gemini 3.8 FlashAPI85Usable with a quick edit Gemini 3.5 Flash-LiteGemini79Usable with a quick edit

GPT-6 AstraChatGPT

Usable with a quick edit11 of 12 checks passed

The output is complete and usable; the only missing piece is a brief mention of the recruiter's withdrawal data, which is a quick edit.

Output

Recommended loop

Replace the unpaid strategy deck, six-person presentation and “beer test” with structured evidence of doing this job. The previous hires’ notes cannot explain why one succeeded and one failed; cross-functional influence must be a hiring requirement, not a personality impression.

Rounds

Round	Who / duration	What it tests and how
1. Structured screen	Recruiter, 20 minutes	Confirm scope, motivation, compensation and logistics. Ask for one conversion or expansion initiative: personal contribution, measured result and partner teams involved. Use consistent questions; accept relevant experience without requiring a Growth PM title. Advance at most two finalists under the interviewer-time budget.
2. Ownership deep dive	Amara, 35 minutes	Examine one growth initiative and one unsuccessful attempt to change another team’s behavior. Probe baseline, decision-making, individual contribution, results, uncertainty and lessons. Test senior ownership, growth judgment and influence through specific actions—not polished storytelling.
3. Live growth work sample	Head of Engineering + data analyst, 40 minutes	Give everyone the same short, synthetic packet: a funnel beginning at Copperline’s 9% trial conversion, cohort/segment data, expansion opportunities, customer observations and delivery constraints. Allow eight minutes of quiet reading within the session. Ask candidates to diagnose, prioritize conversion versus expansion opportunities, and scope the first weekly experiment with three engineers and a designer. Introduce one standardized complication. Test analysis, prioritization, experimentation and delivery. No deck or advance work.
4. Cross-functional working session	Head of Sales + senior designer, 35 minutes	Candidate proposes a trial-conversion change. Sales resists because it changes incentives or workload; Design flags customer friction. Use identical opening objections, while responding naturally thereafter. Candidate must uncover interests, negotiate an achievable pilot and establish responsibilities and follow-through. Finish with reflection on what they would change. Test influence, customer judgment and collaboration—not whether they “win.”
5. Offer conversation—not another assessment	CEO, 20 minutes, selected candidate only	Explain company direction, answer questions and help close. No informal culture-fit veto.

Allow candidate questions in each assessment round. Share the format, competencies and preparation expectations beforehand; offer scheduling flexibility, breaks and reasonable accommodations. Assess reasoning and decisions, not presentation polish. The work sample is fictional, not free consulting.

Scorecard

Use 1 = clear contrary evidence, 2 = below bar, 3 = meets senior-role bar, 4 = exceptional evidence. “Not observed” is separate from a low score and must be resolved before deciding.

Competency / primary evidence owner	Strong: meets or exceeds bar	Weak: below bar
Growth diagnosis and measurement / analyst	Defines conversion and expansion denominators, examines cohorts and segments, distinguishes correlation from causation, identifies missing evidence. Treats 9% as a starting point, not a diagnosis.	Jumps to tactics; misreads funnels; claims causality from movement; ignores retention or expansion economics.
Experiment design and prioritization / analyst, Amara	Chooses a defensible hypothesis, outcome and guardrails; weighs impact, effort and confidence; handles limited traffic and inconclusive results.	Produces a tactic list; promises unsupported lifts; assumes every weekly release can deliver statistically conclusive results.
Weekly delivery with a small team / Engineering	Slices a useful first release, identifies instrumentation and dependencies, trades scope deliberately, and plans the next decision.	Requires a major rebuild or more staff; ignores technical uncertainty; equates speed with skipping measurement or quality.
Influence without authority / Sales	Investigates incentives and constraints, makes a credible mutual-value case, secures specific commitments and establishes follow-through. Past examples show personal action and honest limits.	Relies on title, escalation or charisma; blames Sales; mistakes agreement in a meeting for changed behavior.
Customer and commercial judgment / Design	Connects accountancy-firm workflows to conversion and expansion; protects trust and usability; considers downstream retention and revenue quality.	Optimizes conversion through friction, pressure or misleading choices; ignores customer costs or churn.
Senior ownership and learning / Amara	Separates own contribution from team results, owns misses, changes course with evidence and creates a repeatable operating rhythm.	Offers vague achievements, cannot explain decisions, takes all credit or externalizes failures.

Interviewers record concrete actions, quotes and reasoning alongside scores. “Smart,” “great presence” and “likeable” are not evidence.

Decision, capacity and timeline

Week 1: recruiter screens and schedules two finalists. Week 2: complete assessments. Week 3: decide, close and issue the offer.

Reserve each assessor 10 minutes for calibration, five minutes per finalist for scoring, and 20 minutes for debrief. Including interviews, Engineering and the analyst each use exactly 120 minutes; Amara, Sales and Design each use 110. CEO uses at most 40 minutes. If neither finalist qualifies, reopen rather than lower the bar or silently exceed the budget.

Submit scores independently before seeing others’ notes. In debrief, review evidence competency by competency; Amara speaks last. No show-of-hands vote.

Require at least 3 on every competency, especially influence and weekly delivery. Resolve disagreements through recorded evidence, not averaging or seniority. Missing critical evidence means no offer yet. Amara owns the final decision and documents the rationale, remaining risks and onboarding support.

Grades and run details

Decision model 92 · LLM judge 12 of 13 checks

Decision model checks

passUses the supplied evidence correctly54%
passAddresses the actual decision93%
passRespects explicit constraints23%
partialIdentifies material uncertainty51%
partialAvoids unsupported claims29%
passProduces the required deliverable91%
passTests what the last hire failed at100%
passFixes the take-home's cost to candidates80%
passFits the people and the time75%
passJudges on evidence, not presence100%
passDefines good for this role first95%
passKeeps each judgement independent100%

Run

Run: #1
Time to output: 62 s
Submitted: 5 Oct 2026

GPT-6.1 SolAPI

Usable with a quick edit12 of 12 checks passed

The output is complete, actionable, and respects all constraints; a PM could use it with at most minor wording tweaks.

Output

The loop I’d run

Replace the deck and six-person presentation with bounded, live work samples. Remove the “beer test.” Copperline needs someone who can diagnose conversion, ship weekly, and change other teams’ behavior—not simply present well.

Candidate commitment: 3 hours 10 minutes; no take-home or required preparation. Share the format and scorecard beforehand, offer flexible scheduling and reasonable accommodations, and use the same prompts and information for everyone.

Round	Interviewers	Time	What it tests
---	---	---:	---
1. Recruiter screen	Recruiter	20 min	Scope, compensation, availability, and interest in hands-on growth ownership. Explain the process. Don’t assess polish or pedigree.
2. Ownership interview	Amara	35 min	Senior-level judgment and accountability. Examine one past growth initiative: candidate’s actual contribution, choices, results, failures, and learning. Probe both conversion and expansion experience. Reserve five minutes for candidate questions.
3. Live growth diagnosis	Data analyst	50 min	Analytical judgment and experiment design. Give a small synthetic Copperline dataset: trial funnel, cohorts, acquisition sources, activation behavior, expansion signals, and metric definitions. Ask what could explain 9% conversion, what they would investigate, and which experiment they would run first. Include a plausible expansion opportunity.
4. Shipping workshop	Head of Engineering + senior designer	40 min	Turn the previous round’s proposal into a one-week experiment with three engineers and a designer. Interviewers act as working partners, not an audience. Introduce a tracking gap and an engineering constraint. Test scope, customer experience, instrumentation, trade-offs, and rollout safety.
5. Influence simulation	Head of Sales	45 min	First, role-play a conversation with Sales resisting a proposed trial-follow-up change because of workload and incentives. Add a Marketing objection about lead quality. Then examine a real example of influencing peers without authority, including resistance, compromises, and outcomes. Reserve five minutes for questions.

Use the same dataset, constraints, and role-play objections across candidates. Score reasoning, not whether they discover a predetermined “correct” growth strategy. The CEO does not conduct a separate fit interview.

Scorecard

Use 1 = clear concern, 2 = below bar, 3 = meets senior-role bar, 4 = exceptional, plus not observed. Each rating requires concrete evidence: a quote, action, artifact, or specific past example. “Very smart” and “great presence” are not evidence.

Dimension / primary assessor	Strong: meets or exceeds bar	Weak: below bar
Growth and analytical judgment — analyst	Checks denominators, cohort maturity, segmentation, and data quality before explaining 9%. Separates correlation from causation; prioritizes plausible conversion and expansion opportunities using customer behavior and economics.	Treats aggregate conversion as a diagnosis; jumps to tactics; ignores acquisition mix or revenue quality; makes causal claims without evidence.
Experimentation — analyst	States hypothesis, mechanism, target population, primary metric, guardrails, and decision rule. Recognizes low-volume limitations. Distinguishes shipping weekly from obtaining statistically conclusive results weekly.	Proposes many tests without prioritization; relies on vanity metrics; declares victory prematurely; cannot say what result would change the decision.
Execution and customer judgment — Engineering + designer	Cuts scope to a credible weekly release; specifies measurement and rollback; collaborates with technical/design partners; protects trust in invoicing and payments workflows.	Requires a large rebuild; treats partners as order-takers; neglects instrumentation, reliability, or customer harm; cannot adapt to constraints.
Influence without authority — Sales	Understands incentives and workload; listens, negotiates a bounded pilot, establishes shared measures and explicit commitments, and follows through. Past examples show actual behavior change.	Relies on escalation, persuasion theater, or “getting buy-in”; blames peers; has no concrete mechanism for changing behavior.
Senior ownership and learning — Amara	Clearly distinguishes personal contribution from team results; connects conversion and expansion to durable revenue; owns misses and demonstrates changed judgment.	Claims collective wins as personal achievements; avoids failures; optimizes local conversion at the expense of retention or revenue; offers lessons without changed behavior.

Decision and operating plan

Before interviewing: Run a 15-minute calibration using sample answers and agree on anchors. Give interviewers structured prompts.

After each round: Interviewers submit evidence and scores independently within 24 hours, before seeing others’ assessments. Engineering and Design score independently.

Debrief: Hold a 30-minute evidence review—not a show-of-hands vote. Discuss disagreements and counterevidence against the written bar. Amara makes and documents the hiring decision.

Offer bar: At least 3 on every dimension. In particular, do not average away weak influence, analytical judgment, or execution. Missing evidence triggers one narrowly targeted follow-up, not intuition. Hire against the bar, not merely the strongest available candidate.

Timing and capacity: Finish interviews by day 12, debrief by day 14, and issue the offer by day 21. Calibration, interview, scoring, and debrief consume at most 105 minutes per domain interviewer. Amara retains time within her two-hour cap for decision documentation and an offer conversation. CEO approval, if required, concerns headcount and compensation—not an unstructured personality veto.

Grades and run details

Decision model 92 · LLM judge 13 of 13 checks

Decision model checks

passUses the supplied evidence correctly27%
passAddresses the actual decision92%
partialRespects explicit constraints26%
partialIdentifies material uncertainty54%
passAvoids unsupported claims21%
passProduces the required deliverable91%
passTests what the last hire failed at100%
passFixes the take-home's cost to candidates81%
passFits the people and the time85%
passJudges on evidence, not presence100%
passDefines good for this role first89%
passKeeps each judgement independent100%

Run

Run: #1
API response time: 55 s
Submitted: 5 Oct 2026

GPT-6 LunaAPI

Usable with a quick edit9 of 12 checks passed

The output is complete and actionable; a quick edit to mention the recruiter's withdrawal data would make it fully aligned with the brief.

Output

Interview loop

Run this over 2–3 weeks. Skip the five-day take-home, large presentation panel, and “beer” fit interview. Share the same expectations and questions with every candidate; offer scheduling flexibility and accommodations.

1. Recruiter screen — 30 minutes Tests role understanding, motivation, and practical alignment: the scope, weekly shipping pace, and need to influence Sales and Marketing without authority. Not a “polish” screen.

2. Amara, VP Product — 60 minutes Tests growth ownership and product judgment. Ask for a detailed example of a funnel or revenue problem the candidate owned: diagnosis, choices, experiments, results, and what they learned. Probe how they balance conversion, customer trust, and expansion.

3. Live growth work sample — 60 minutes Send a one-page brief and synthetic funnel data 24 hours ahead; no deck or advance work required. Spend 30 minutes with the data analyst diagnosing the funnel and proposing a measurable experiment; spend 30 minutes with the Head of Engineering and senior designer scoping how to ship and learn from it. Use the same prompt and time limits for all candidates.

4. Head of Sales — 45 minutes Structured roleplay: Sales is reluctant to change its trial follow-up because reps are busy and doubt the data. Ask the candidate to understand the objection, make a case, and agree a practical next step. Then probe a real example of influencing a team they did not manage.

The CEO does not need a separate round: “culture fit” is not a hiring criterion, and the role’s key cross-functional test is better assessed directly with Sales, Engineering, and Design.

Scorecard

Use a 1–4 rating for each dimension: 1 = clear concern, 2 = below bar, 3 = meets bar, 4 = unusually strong. Record evidence, not impressions.

Dimension	Strong evidence	Weak evidence
Growth ownership and outcomes (Amara)	Personally owned a meaningful funnel or revenue outcome; explains decisions, results, attribution limits, and lessons.	Describes activity rather than impact; overclaims credit or cannot explain what changed.
Customer and product judgment (Amara)	Understands small-firm workflows and trust needs; weighs conversion against customer value and expansion.	Offers generic SaaS tactics or pushes short-term conversion at the expense of trust.
Data diagnosis and experiment design (work sample)	Finds plausible bottlenecks and segments; proposes a focused test with a clear hypothesis, success metric, guardrail, and learning plan.	Jumps to a solution, misreads the data, uses vanity metrics, or cannot say what result would change their mind.
Shipping and learning cadence (work sample)	Scopes a feasible experiment, identifies dependencies and tradeoffs, and shows how the team can learn and iterate weekly.	Proposes a big-bang project, ignores constraints, or treats learning as something to do after launch.
Influence without authority (Sales)	Listens for incentives and concerns, uses evidence appropriately, adapts the ask, and secures a credible next step.	Blames Sales, relies on escalation or authority, or cannot turn disagreement into action.
Cross-functional partnership (work sample)	Makes clear tradeoffs with Engineering and Design, invites their expertise, and aligns on ownership and delivery.	Treats partners as order-takers or leaves scope and responsibilities vague.

Decision

Interviewers submit ratings and specific evidence independently before a 30-minute debrief; do not vote by show of hands. Amara facilitates, checks for evidence behind claims, and makes the hiring decision. Do not use “presence,” “smart,” or general “fit” as evidence.

A hire must score at least 3 on influence, experiment design, and shipping, with no score below 2 in any dimension. Do not average away a weakness in a core requirement. If evidence is genuinely missing, identify one focused follow-up rather than adding a broad round. Give candidates a timely decision and close the loop within three weeks of the first interview.

Grades and run details

Decision model 96 · LLM judge 10 of 13 checks

Decision model checks

passUses the supplied evidence correctly51%
passAddresses the actual decision82%
passRespects explicit constraints26%
partialIdentifies material uncertainty64%
passAvoids unsupported claims22%
passProduces the required deliverable78%
passTests what the last hire failed at100%
passFixes the take-home's cost to candidates68%
passFits the people and the time64%
passJudges on evidence, not presence100%
passDefines good for this role first86%
passKeeps each judgement independent98%

Run

Run: #1
API response time: 39 s
Submitted: 5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 3

Addresses the actual decisionRightRightMixed

GPT-6 Astra · ChatGPT

The output commits to a specific redesigned loop and decision process, with Amara owning the final call and a clear condition to reopen if no candidate meets the bar.

GPT-6.1 Sol · API

The output commits to a specific redesigned loop, scorecard, and decision process, and states the offer bar and how missing evidence is handled.

GPT-6 Luna · API

The output does not state what result or condition would change the proposed loop or scorecard.

Identifies material uncertaintyRightRightWrong

GPT-6 Astra · ChatGPT

It names the condition that would change the decision (if neither finalist qualifies, reopen) and that missing critical evidence means no offer.

GPT-6.1 Sol · API

The output defines a clear decision rule (bar of 3 on every dimension, no averaging away weak areas) and implies not hiring if no one meets it, which addresses the key uncertainty in the hiring decision.

GPT-6 Luna · API

The output does not name any unknowns that could change the decision or how they would be resolved.

Fixes the take-home's cost to candidatesMixedRightMixed

GPT-6 Astra · ChatGPT

The output replaces the take-home with a live exercise but never cites the recruiter's withdrawal data or who the take-home was driving away.

GPT-6.1 Sol · API

The five-day take-home is replaced with live, bounded exercises, and the candidate commitment is reduced to 3 hours 10 minutes with no take-home, addressing the withdrawal data.

GPT-6 Luna · API

The output replaces the take-home but does not cite the recruiter's withdrawal data or mention who it was driving away.

All got right 9

Uses the supplied evidence correctlyRightRightRight

GPT-6 Astra · ChatGPT

The output uses the 9% conversion rate and the insufficient interview notes correctly, and invents no facts about the current situation.

GPT-6.1 Sol · API

All statements about the current situation are taken directly from the brief or supplied context, with no invented facts.

GPT-6 Luna · API

The output makes no factual claims about the current situation, so it does not misuse any supplied evidence.

Respects explicit constraintsRightRightRight

GPT-6 Astra · ChatGPT

The output is a loop, scorecard, and decision process within 900 words, addressed to Amara, and respects all stated constraints.

GPT-6.1 Sol · API

The output is under 900 words, addresses Amara, replaces the take-home and beer test, and respects all stated constraints.

GPT-6 Luna · API

The output respects all constraints: it replaces the take-home, removes the beer test, fits interviewer time limits and the three-week schedule, and stays under 900 words.

Avoids unsupported claimsRightRightRight

GPT-6 Astra · ChatGPT

The only interpretation (notes can't explain outcomes) is directly supported by the brief; no other claims are presented as established fact.

GPT-6.1 Sol · API

No interpretations or forecasts are presented as established facts; the need for diagnosis, shipping, and influence is directly derived from the role description.

GPT-6 Luna · API

The output presents only proposals and avoids unsupported claims about causes or forecasts.

Produces the required deliverableRightRightRight

GPT-6 Astra · ChatGPT

The output provides a complete loop, scorecard, and decision process that a PM could act on with light edits.

GPT-6.1 Sol · API

The output provides a complete loop, scorecard, and decision plan in the requested form, under 900 words, usable by Amara with light edits.

GPT-6 Luna · API

The output provides a complete interview loop, scorecard, and decision process that a product manager could act on.

Tests what the last hire failed atRightRightRight

GPT-6 Astra · ChatGPT

Round 4 is a cross-functional working session with the Head of Sales, and the scorecard includes 'Influence without authority' with strong/weak signals.

GPT-6.1 Sol · API

The loop includes an influence simulation with the Head of Sales and a scorecard dimension on influence without authority, directly testing what the last hire failed at.

GPT-6 Luna · API

The loop includes a dedicated round with the Head of Sales and a scorecard dimension for influence without authority, directly testing what the last hire failed at.

Fits the people and the timeRightRightRight

GPT-6 Astra · ChatGPT

Interviewer times are explicitly kept within two hours (Engineering and analyst 120 min, others 110 min, CEO 40 min) and the three-week schedule is detailed.

GPT-6.1 Sol · API

Each interviewer's time (including calibration, interview, and debrief) stays within 105 minutes, under the two-hour cap, and the schedule fits the three-week target.

GPT-6 Luna · API

Each interviewer's time is within two hours, and the loop is designed to fit the three-week target.

Judges on evidence, not presenceRightRightRight

GPT-6 Astra · ChatGPT

The scorecard demands concrete actions, quotes, and reasoning, and explicitly rejects 'smart', 'great presence', and 'likeable' as evidence.

GPT-6.1 Sol · API

The process requires concrete evidence (quotes, actions, artifacts) and explicitly rejects 'very smart' and 'great presence' as evidence.

GPT-6 Luna · API

The scorecard and decision rules explicitly require evidence over impressions and forbid using 'presence' or 'smart' as criteria.

Defines good for this role firstRightRightRight

GPT-6 Astra · ChatGPT

The scorecard defines six competencies with strong and weak signals before any interview design, weighting influence and delivery for this role.

GPT-6.1 Sol · API

The scorecard defines five weighted competencies with strong and weak signals, and the decision rule prioritizes influence, analytical judgment, and execution.

GPT-6 Luna · API

The scorecard defines competencies with strong and weak signals, and the hiring bar weights influence, experiment design, and shipping.

Keeps each judgement independentRightRightRight

GPT-6 Astra · ChatGPT

Interviewers must submit scores independently before seeing others' notes, and Amara speaks last; no show-of-hands vote.

GPT-6.1 Sol · API

Interviewers submit evidence and scores independently before discussion, and the debrief is an evidence review, not a show of hands.

GPT-6 Luna · API

Interviewers must submit ratings and evidence independently before the debrief, and the process guards against groupthink.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Provisional The checklist is still being calibrated against our PM.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	GPT-6.1 SolwithAPI	94.4	100.0	3	None
2	Sonnet 5.5withAPI	95.8	96.2	2	None
3	Gemini 3.8 FlashwithAPI	89.6	92.3	2	None
4	GPT-6 AstrawithChatGPT	95.8	87.2	3	None
5	Opus 5.5withClaude	91.7	87.2	3	None
6	GPT-6 LunawithAPI	93.1	84.6	3	None
7	Gemini 3.5 Flash-LitewithGemini	75.0	64.1	3	None

The PM job

Hiring product managers.

Why it matters

A bad PM hire costs a team a year. Most loops reward polish and presence, and the debrief goes to whoever speaks first.

What good looks like

Defines what good looks like for this role before judging anyone
Judges on evidence of results and how they work with a team
Tests the real job, not a rehearsed framework
Keeps each interviewer's judgement independent

Deliberately not measured

Employment law
Compensation benchmarking

Capability tested

Hiring judgement

The failure we’re looking for

Hiring for presence and polish over evidence

Grading

Decision model and LLM judge, calibrated against a blind PM review

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

Judges on evidence, not presence

Defines good for this role first

Keeps each judgement independent

Plus our standard checks

The tasks

The outputs

Output

Recommended loop

Rounds

Scorecard

Decision, capacity and timeline

Grades and run details

Decision model checks

Run

Output

The loop I’d run

Scorecard

Decision and operating plan

Grades and run details

Decision model checks

Run

Output

Interview loop

Scorecard

Decision

Grades and run details

Decision model checks

Run

Check by check

Where they differ 3

All got right 9

Results

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured