Tasks / Leadership

Hire a PM

Can the model design a hiring process that finds the right PM, and make the call on real candidates from the evidence?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 63% were usable with at most a quick edit.

Reliably right

  1. Produces the required deliverable100% pass
    The output provides a complete loop, scorecard, and decision plan in the requested form, under 900 words, usable by Amara with light edits.
    GPT-6.1 Sol · API · A loop for the first growth PM
  2. Tests what the last hire failed at100% pass
    The loop includes an influence simulation with the Head of Sales and a scorecard dimension on influence without authority, directly testing what the last hire failed at.
    GPT-6.1 Sol · API · A loop for the first growth PM
  3. Finds which rounds line up with performance100% pass
    It compares each round's scores with 12-month ratings, finding behavioral lines up and take-home, presentation, VP Engineering do not.
    GPT-6.1 Sol · API · What our interviews predict

Where it slips

  1. Avoids unsupported claims56% pass
    The claim 'A growth PM in a senior role is likely to be exactly that kind of person' is presented as fact without evidence and is not labelled as an assumption.
    Opus 5.5 · Claude · A loop for the first growth PM
  2. Identifies material uncertainty58% pass
    The output does not name unknowns that could change the hiring decision or the loop design, nor does it say how they would be resolved.
    Sonnet 5.5 · API · A loop for the first growth PM
  3. Defines good for this role first63% pass
    The memo does not explicitly name the competencies that matter most for this PM role and level, nor define what strong and weak look like before designing the interviews.
    Sonnet 5.5 · API · What our interviews predict

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Judges on evidence, not presence

    Does the output judge candidates, or design the process to judge them, on specific evidence of results they drove and how they worked with their team, rather than impressions such as presence, confidence or polish?

    Passes when Asks for or weighs concrete evidence: the metric a candidate moved, the part they played, how they shared credit, what they learned from a failure. Impressions are treated as weak signals.

  2. Defines good for this role first

    Does the output set out what good looks like for this specific role (the competencies that matter most at this level, and what strong and weak look like) before judging candidates or designing interviews?

    Passes when Names the competencies weighted for this role and level, with strong and weak signals, and the judgement or the process follows from them.

  3. Keeps each judgement independent

    Does the output keep each interviewer's judgement independent (written down before the group discusses it), and guard against the loudest voice or the first speaker deciding?

    Passes when Requires or relies on written, independent evaluations before discussion, and discounts views that changed under group pressure.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 3 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're helping Amara Osei, VP Product at Copperline, hire our first Growth PM (a senior role). She's shared her draft interview loop and asked you to redesign it. Write the loop you'd run (each round: who runs it, what it tests, how long it takes), the scorecard (what strong and weak look like for each thing we're testing), and how we'll make the decision at the end. No more than 900 words. What we know is below.

What the model was given6 items: About Copperline, The role, Amara's draft loop, The last two PM hires, From the recruiter, Who can interview
About CopperlineInvoicing and payments software for small accountancy firms. 60 people. Trial to paid conversion is 9%.
The roleOwns trial conversion and expansion revenue. Works with three engineers and a designer, reports to Amara, and has to ship experiments every week. Needs Sales and Marketing to change how they work, without managing them.
Amara's draft loop1. Recruiter screen. 2. Take-home: 'design a growth strategy for Copperline', a 10-page deck, five days to complete. 3. Presentation to a panel of six. 4. Culture fit with the CEO: 'would I grab a beer with them?' 5. Debrief: everyone discusses, then votes by show of hands. There's no scorecard.
The last two PM hiresOne left after five months: 'couldn't get Sales to change anything'. The other is doing well. Both sets of interview notes say mainly 'great presence' and 'very smart'.
From the recruiterOf the last 40 candidates given the take-home, 14 withdrew, saying they didn't have time. Nine of the 14 had young children or demanding current roles.
Who can interviewAmara, the Head of Engineering, a senior designer, the Head of Sales, a data analyst and the CEO. Each can give at most two hours to this hire. Amara wants an offer out within three weeks of the first interview.
What a strong answer doesThe answer key the graders mark against

A loop built on a scorecard written first: for a senior growth PM, experimentation and analytics, execution, and influence without authority (weighted up, since the last hire failed on it), with strong and weak signals for each. Rounds test the real job: a live diagnosis of Copperline's own trial funnel instead of a five-day generic deck (which loses candidates, mostly parents and people in demanding roles), a behavioural round on past experiments and metrics actually moved, and an influence round run by the Head of Sales. Replaces the 'grab a beer' culture fit with a structured values or collaboration conversation. Each interviewer writes their evaluation before the debrief, and the decision is made against the scorecard, not a show of hands. Fits each interviewer's two hours and the three-week target, and adds reference checks.

Critical failures (cap the score at 40)
  • Keeps a culture-fit test based on whether the interviewer would like a beer with the candidate
Task

Mid-level · v1.0

The outputs

8 setups answered this brief.

Usable with a quick edit10 of 12 checks passed

The output is fully structured and actionable, needing only a one-sentence addition citing the recruiter's withdrawal data to complete the take-home justification.

Output

Redesigned Growth PM Interview Loop

Principles: Every round tests a named criterion. Scoring happens in writing before anyone discusses. Candidate time is capped. Interviewer time is capped at two hours each.

The loop (about three weeks)

Round 1: Recruiter screen (30 min) Runs: Recruiter Tests: Evidence of shipping experiments on a weekly cadence, compensation, logistics. Explain the full loop and its time cap up front so candidates can decide early. Knockout: No recent experience running experiments with a regular cadence.

Round 2: Analytical work sample (candidate time capped at 2 hours; due within 5 days, candidate picks when) Runs: Data Analyst scores; Amara reviews. Tests: Diagnosis and experiment rigour. Brief: An anonymised trial dataset showing the 9% conversion rate. Where is the funnel leaking, what would you test first, and what would you need from Sales and Engineering? Output: A two-page memo, not a deck. One diagnosis, three ranked experiments, and one fully specified (hypothesis, primary metric, guardrail metric, sample size, engineering effort). Alternative: A 60-minute live version on a shorter dataset, for candidates who can't spare two hours. Both versions are scored on the same rubric. Interviewer time: 45 min review.

Round 3: Build session (60 min, joint) Runs: Head of Engineering and senior designer. Tests: Engineering partnership and design judgment. Can the candidate cut their top experiment to something shippable this week? Do they know what needs instrumentation and feature flags, and what the design needs to look like? Interviewer time: 60 min each.

Round 4: Influence role-play (45 min) Runs: Head of Sales. Tests: Influence without authority, and understanding of sales incentives. Scenario: Sales wants to keep manual follow-ups for trial users. The candidate needs reps to run a three-week test that changes their routine. The Head of Sales plays a realistic, sceptical rep. Interviewer time: 45 min.

Round 5: Ownership and judgment (60 min) Runs: Amara. Tests: Ownership, learning from failure, decisions on incomplete data. Format: Structured behavioural questions. Ask for three past experiments: one that failed, one where they changed a team's behaviour without authority, one decision made on incomplete data. Probe what the candidate personally did, not what the team did. Interviewer time: 60 min.

Round 6: Working norms (30 min) Runs: CEO. Tests: Candour, comfort with pace and ambiguity. This replaces "would I grab a beer with them" with defined questions: - Tell me about disagreeing with a senior leader. What happened next? - A test you championed lost. What did you do? - What pace and level of ambiguity do you need to do your best work? Interviewer time: 30 min.

Final debrief (45 min): Amara and all interviewers, using written scores.

Timeline: Days 1–2 screen and brief issued. Days 2–6 work sample. Week 2: Rounds 3–5. Week 3: Round 6, debrief within 48 hours, offer.

Interviewer load: Data Analyst about 1h; Head of Engineering and senior designer 1h each; Head of Sales 45 min; Amara about 2h 15m (Round 5 plus review and debrief; she can reduce this by handing Round 2 review to the analyst, as above); CEO 30 min plus debrief.

Scorecard

Rate each criterion 1–4. Scores of 3 or 4 are "meets bar." Criteria marked ★ are must-haves.

CriterionStrong (3–4)Weak (1–2)
★ Experiment rigourNames a clear hypothesis, primary metric, guardrail, and sample size. Has run experiments weekly. Kills losing tests cleanly and says what they learned.Describes strategy but not tests. Can't say how many experiments they ran or what changed as a result.
Funnel diagnosisReads the data before proposing ideas. Questions the metric and names where the leak is and how they'd confirm it.Jumps to features or ideas without evidence. Takes the 9% at face value.
★ Influence without authorityGives a specific example of changing a team that didn't report to them. Uses the other team's incentives. Names trade-offs and what they gave up."I presented it and they agreed." Blames Sales or others for the outcome.
Engineering partnershipScopes changes small and instrumentable. Knows about flags and staged rollouts. Asks about capacity before proposing.Proposes large builds without checking cost. Treats engineering as a delivery queue.
Ownership and judgmentDescribes their own decisions and what they'd do differently. Makes calls on incomplete data and says how they'd know if wrong.Only success stories. Vague about personal role. Waits for certainty.
Working normsGives examples of disagreeing with a senior leader and following through. Comfortable with a small company's pace and ambiguity.Needs everything defined first. Avoids disagreement. Can't name a time they were wrong.

"Presence" and "very smart" are not criteria. Don't record them as evidence.

How we decide

  1. Score before discussing. Each interviewer submits scores and written evidence before the debrief. Nobody sees others' scores first.
  2. Apply the rules. Every must-have (★) needs a 3 or higher. Any 1 in any criterion is a no unless the panel explicitly discusses it. Don't average scores.
  3. Debrief on evidence. Go round the table. Each person states the evidence behind their score. Discuss in depth only where scores differ by two or more points. No show of hands.
  4. Amara decides. She writes a short rationale tied to the scorecard. The CEO must also score working norms at 3 or higher.
  5. If still split, run one 30-minute follow-up focused on the specific gap, then decide. Don't extend the loop.
  6. References. Ask referees for specific experiments the candidate ran and how they worked with Sales and Engineering.

Grades and run details

Decision model 88 · LLM judge 85
Decision model checks
  • passUses the supplied evidence correctly41%
  • passAddresses the actual decision85%
  • partialRespects explicit constraints46%
  • partialIdentifies material uncertainty51%
  • passAvoids unsupported claims37%
  • passProduces the required deliverable91%
  • passTests what the last hire failed at100%
  • passFixes the take-home's cost to candidates43%
  • partialFits the people and the time58%
  • passJudges on evidence, not presence100%
  • passDefines good for this role first85%
  • passKeeps each judgement independent100%
Run
Run
#1
API response time
32 s
Submitted
8 Oct 2026

Usable with a quick edit12 of 12 checks passed

The output is complete, actionable, and respects all constraints; a PM could use it with at most minor wording tweaks.

Output

The loop I’d run

Replace the deck and six-person presentation with bounded, live work samples. Remove the “beer test.” Copperline needs someone who can diagnose conversion, ship weekly, and change other teams’ behavior—not simply present well.

Candidate commitment: 3 hours 10 minutes; no take-home or required preparation. Share the format and scorecard beforehand, offer flexible scheduling and reasonable accommodations, and use the same prompts and information for everyone.

RoundInterviewersTimeWhat it tests
---------:---
1. Recruiter screenRecruiter20 minScope, compensation, availability, and interest in hands-on growth ownership. Explain the process. Don’t assess polish or pedigree.
2. Ownership interviewAmara35 minSenior-level judgment and accountability. Examine one past growth initiative: candidate’s actual contribution, choices, results, failures, and learning. Probe both conversion and expansion experience. Reserve five minutes for candidate questions.
3. Live growth diagnosisData analyst50 minAnalytical judgment and experiment design. Give a small synthetic Copperline dataset: trial funnel, cohorts, acquisition sources, activation behavior, expansion signals, and metric definitions. Ask what could explain 9% conversion, what they would investigate, and which experiment they would run first. Include a plausible expansion opportunity.
4. Shipping workshopHead of Engineering + senior designer40 minTurn the previous round’s proposal into a one-week experiment with three engineers and a designer. Interviewers act as working partners, not an audience. Introduce a tracking gap and an engineering constraint. Test scope, customer experience, instrumentation, trade-offs, and rollout safety.
5. Influence simulationHead of Sales45 minFirst, role-play a conversation with Sales resisting a proposed trial-follow-up change because of workload and incentives. Add a Marketing objection about lead quality. Then examine a real example of influencing peers without authority, including resistance, compromises, and outcomes. Reserve five minutes for questions.

Use the same dataset, constraints, and role-play objections across candidates. Score reasoning, not whether they discover a predetermined “correct” growth strategy. The CEO does not conduct a separate fit interview.

Scorecard

Use 1 = clear concern, 2 = below bar, 3 = meets senior-role bar, 4 = exceptional, plus not observed. Each rating requires concrete evidence: a quote, action, artifact, or specific past example. “Very smart” and “great presence” are not evidence.

Dimension / primary assessorStrong: meets or exceeds barWeak: below bar
Growth and analytical judgment — analystChecks denominators, cohort maturity, segmentation, and data quality before explaining 9%. Separates correlation from causation; prioritizes plausible conversion and expansion opportunities using customer behavior and economics.Treats aggregate conversion as a diagnosis; jumps to tactics; ignores acquisition mix or revenue quality; makes causal claims without evidence.
Experimentation — analystStates hypothesis, mechanism, target population, primary metric, guardrails, and decision rule. Recognizes low-volume limitations. Distinguishes shipping weekly from obtaining statistically conclusive results weekly.Proposes many tests without prioritization; relies on vanity metrics; declares victory prematurely; cannot say what result would change the decision.
Execution and customer judgment — Engineering + designerCuts scope to a credible weekly release; specifies measurement and rollback; collaborates with technical/design partners; protects trust in invoicing and payments workflows.Requires a large rebuild; treats partners as order-takers; neglects instrumentation, reliability, or customer harm; cannot adapt to constraints.
Influence without authority — SalesUnderstands incentives and workload; listens, negotiates a bounded pilot, establishes shared measures and explicit commitments, and follows through. Past examples show actual behavior change.Relies on escalation, persuasion theater, or “getting buy-in”; blames peers; has no concrete mechanism for changing behavior.
Senior ownership and learning — AmaraClearly distinguishes personal contribution from team results; connects conversion and expansion to durable revenue; owns misses and demonstrates changed judgment.Claims collective wins as personal achievements; avoids failures; optimizes local conversion at the expense of retention or revenue; offers lessons without changed behavior.

Decision and operating plan

Before interviewing: Run a 15-minute calibration using sample answers and agree on anchors. Give interviewers structured prompts.

After each round: Interviewers submit evidence and scores independently within 24 hours, before seeing others’ assessments. Engineering and Design score independently.

Debrief: Hold a 30-minute evidence review—not a show-of-hands vote. Discuss disagreements and counterevidence against the written bar. Amara makes and documents the hiring decision.

Offer bar: At least 3 on every dimension. In particular, do not average away weak influence, analytical judgment, or execution. Missing evidence triggers one narrowly targeted follow-up, not intuition. Hire against the bar, not merely the strongest available candidate.

Timing and capacity: Finish interviews by day 12, debrief by day 14, and issue the offer by day 21. Calibration, interview, scoring, and debrief consume at most 105 minutes per domain interviewer. Amara retains time within her two-hour cap for decision documentation and an offer conversation. CEO approval, if required, concerns headcount and compensation—not an unstructured personality veto.

Grades and run details

Decision model 92 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly27%
  • passAddresses the actual decision92%
  • partialRespects explicit constraints26%
  • partialIdentifies material uncertainty54%
  • passAvoids unsupported claims21%
  • passProduces the required deliverable91%
  • passTests what the last hire failed at100%
  • passFixes the take-home's cost to candidates81%
  • passFits the people and the time85%
  • passJudges on evidence, not presence100%
  • passDefines good for this role first89%
  • passKeeps each judgement independent100%
Run
Run
#1
API response time
55 s
Submitted
5 Oct 2026

Usable with a quick edit10 of 12 checks passed

The output is well-structured and complete; the one unsupported generalization can be removed with a quick edit without affecting the loop design.

Output

Copperline Growth PM: Interview Loop

Why the draft needs changing

  • The take-home is filtering out the wrong people. 14 of 40 candidates withdrew, and most of them were parents or in demanding jobs. A growth PM in a senior role is likely to be exactly that kind of person.
  • The panel of six and the "beer" test reward presence. Presence is what your interview notes captured last time, and it didn't predict success.
  • Nothing tested influence over Sales. That is why the last hire failed.

The new loop tests five things, each owned by one interviewer.

Time budget

Each interviewer has two hours in total for this hire. That limits the loop to three finalists. Per interviewer, the time breaks down as:

  • 3 × 30-minute interviews
  • 3 × 5-minute scorecards
  • one 15-minute debrief

That totals 2 hours, or 1.5 hours for the CEO. The recruiter screens with no time limit, so screening carries the shortlisting load.

The loop

RoundWhoTestsLength
0. ScreenRecruiterHas owned a conversion or expansion metric; has shipped experiments at pace. Structured questions, scored pass/fail.30 min
1. Growth deep-diveAmaraExperiment velocity. Walk through one experiment programme they ran: cadence, hit rate, what they killed.30 min
2. Funnel caseData analystQuantitative judgement. Read an anonymised Copperline trial funnel in the session (10 min) and say where the 9% is leaking, what they'd test first and how they'd know. No prep.30 min
3. Influence role-playHead of SalesInfluence without authority. The data shows Sales' handling of trial accounts hurts conversion. The Head of Sales plays himself and pushes back realistically. Ends with "tell me about a time this went badly."30 min
4. Build sessionHead of Engineering + senior designerWorking with the team. Take one idea from the funnel case and scope it to ship in a week: what gets cut, what gets instrumented, what the designer needs.30 min
5. Commercial judgementCEOCommercial sense. Where expansion revenue comes from in small accountancy firms, pricing levers, and what they'd not do. Structured questions. No "culture fit."20 min

The candidate spends about 2.5 hours in total, ideally in one afternoon (remote is fine). Candidates get the round list and competencies in advance.

Scorecard

Each interviewer scores only their own competency, on a scale of 1 to 4. Every score must cite specific evidence: what the candidate said or did. The words "presence," "smart" and "great energy" don't count as evidence.

CompetencyStrong (4)Weak (1)
Experiment velocity (must-have)Names a real cadence (e.g. weekly), a real hit rate, and experiments they killed fast. Talks about learning per week, not big launches.Describes one large project. Can't say how many tests they ran or what failed. "Strategy" without shipping.
Quantitative judgementSpots the biggest leak quickly. Questions data quality. Sizes the opportunity. Defines a success metric and minimum sample before testing.Jumps to solutions. Treats all drop-offs as equal. No view on how to measure the result.
Influence without authority (must-have)Asks what Sales is paid on and afraid of. Proposes a small reversible pilot with Sales sharing the credit. Stays calm under pushback. Has a real story of winning over a resistant team.Leans on data or seniority ("the numbers say so"). Escalates to Amara or the CEO early. Gives up or bulldozes. Past examples are all teams they managed.
Working with Engineering and DesignCuts scope hard to hit a week. Asks engineers for options rather than dictating. Gives the designer a problem, not a mock-up. Plans instrumentation up front.Specs a month of work. Dictates solutions. Forgets tracking.
Commercial senseUnderstands how small accountancy firms buy and grow. Links trial and expansion to revenue and margin. Names trade-offs and things they'd refuse to do.Generic SaaS playbook. Can't connect activity to revenue. Agrees with everything.

Overall recommendation for each interviewer: Strong hire, Hire, No hire or Strong no hire.

How we decide

1. Score independently. Scorecards are submitted within two hours of the interview and before the debrief. Nobody sees others' scores until theirs is in. 2. Short debrief. One 15-minute debrief covers all three finalists, run by Amara after the final loop. Everyone reads the scorecards beforehand. The meeting is for resolving disagreements, not for retelling the interviews. 3. Speaking order. The most junior interviewer speaks first (data analyst, then designer, Engineering, Sales, CEO). The CEO and Amara speak last, so seniority doesn't anchor the room. 4. No show of hands. The bar is explicit: - A 3 or higher on both must-haves: experiment velocity and influence. - No 1s anywhere. - An average of at least 3. 5. Who decides. Amara makes the final call among candidates who clear the bar. A 1 from the Head of Sales on influence blocks the hire. This is the exact failure mode from last time. 6. If nobody clears the bar, we don't hire. We reopen the search rather than lowering the bar to hit the deadline. 7. References. We take one reference from someone in Sales or Marketing who worked alongside the candidate, not someone who reported to them. We ask specifically how the candidate got them to change something.

Timeline (offer within three weeks)

DaysActivity
1–5Recruiter screens. Shortlist three finalists.
6–10Finalist loops, all in the same week so comparisons stay fresh.
11Debrief and decision.
12–14References.
15Offer out. This leaves about a week of buffer for scheduling slips or a second-choice candidate.

Grades and run details

Decision model 83 · LLM judge 92
Decision model checks
  • passUses the supplied evidence correctly19%
  • passAddresses the actual decision90%
  • partialRespects explicit constraints17%
  • failIdentifies material uncertainty16%
  • partialAvoids unsupported claims38%
  • passProduces the required deliverable88%
  • passTests what the last hire failed at100%
  • passFixes the take-home's cost to candidates100%
  • passFits the people and the time53%
  • passJudges on evidence, not presence100%
  • passDefines good for this role first93%
  • passKeeps each judgement independent100%
Run
Run
#1
Time to output
48 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 3

Identifies material uncertaintyWrongRightMixed
Haiku 5.5 · API

It does not explicitly name specific unknowns that could change the hiring decision (e.g., true past influence on Sales) and how they would be resolved, even though reference checks and a split-decision follow-up are included.

GPT-6.1 Sol · API

The output defines a clear decision rule (bar of 3 on every dimension, no averaging away weak areas) and implies not hiring if no one meets it, which addresses the key uncertainty in the hiring decision.

Opus 5.5 · Claude

It identifies that if no candidate clears the bar, the search reopens, and provides buffer for scheduling slips.

Avoids unsupported claimsRightRightWrong
Haiku 5.5 · API

No interpretations, causes, or forecasts are presented as established facts; the role-play scenario is clearly labelled as a hypothetical.

GPT-6.1 Sol · API

No interpretations or forecasts are presented as established facts; the need for diagnosis, shipping, and influence is directly derived from the role description.

Opus 5.5 · Claude

The claim 'A growth PM in a senior role is likely to be exactly that kind of person' is presented as fact without evidence and is not labelled as an assumption.

Fixes the take-home's cost to candidatesMixedRightRight
Haiku 5.5 · API

The output replaces the 5-day deck with a short exercise but never cites the recruiter's withdrawal data (14 withdrew, 9 had young children or demanding roles) to explain who it was driving away.

GPT-6.1 Sol · API

The five-day take-home is replaced with live, bounded exercises, and the candidate commitment is reduced to 3 hours 10 minutes with no take-home, addressing the withdrawal data.

Opus 5.5 · Claude

It replaces the five-day take-home with a live funnel case and cites the withdrawal data and who it drove away.

All got right 9

Uses the supplied evidence correctlyRightRightRight
Haiku 5.5 · API

The only factual claim about the current situation is the 9% conversion rate, which is directly supported by the brief; no other present-state facts are invented.

GPT-6.1 Sol · API

All statements about the current situation are taken directly from the brief or supplied context, with no invented facts.

Opus 5.5 · Claude

All factual claims about the current situation are taken directly from the brief or supplied context, with no inventions.

Addresses the actual decisionRightRightRight
Haiku 5.5 · API

Output commits unambiguously to a fully redesigned interview loop, scorecard, and decision process, framed for Amara.

GPT-6.1 Sol · API

The output commits to a specific redesigned loop, scorecard, and decision process, and states the offer bar and how missing evidence is handled.

Opus 5.5 · Claude

The output commits to a specific interview loop, scorecard, and decision process, framed for Amara, with clear conditions for not hiring.

Respects explicit constraintsRightRightRight
Haiku 5.5 · API

Output is under 900 words, removes the 5-day take-home and beer test, fits each interviewer within 2 hours (with a minor hand-off to stay under), and meets the 3-week target.

GPT-6.1 Sol · API

The output is under 900 words, addresses Amara, replaces the take-home and beer test, and respects all stated constraints.

Opus 5.5 · Claude

The output respects the 900-word limit, replaces the take-home and beer test, and fits interviewer time and three-week deadline.

Produces the required deliverableRightRightRight
Haiku 5.5 · API

The loop, scorecard with strong/weak signals, and decision process are present and complete, within the word limit, and usable by Amara.

GPT-6.1 Sol · API

The output provides a complete loop, scorecard, and decision plan in the requested form, under 900 words, usable by Amara with light edits.

Opus 5.5 · Claude

The output is a complete interview loop, scorecard, and decision process, within the word limit, usable by Amara.

Tests what the last hire failed atRightRightRight
Haiku 5.5 · API

Round 4 (influence role-play with Head of Sales) and Round 5 (behavioral questions on changing team behavior without authority) directly test influence, with strong/weak signals on the scorecard.

GPT-6.1 Sol · API

The loop includes an influence simulation with the Head of Sales and a scorecard dimension on influence without authority, directly testing what the last hire failed at.

Opus 5.5 · Claude

The loop includes an influence role-play with the Head of Sales and probes past examples of changing another team's work.

Fits the people and the timeRightRightRight
Haiku 5.5 · API

Each interviewer stays at or under 2 hours (Head of Sales 45 min + debrief 45 min, etc.), and the timeline fits within three weeks.

GPT-6.1 Sol · API

Each interviewer's time (including calibration, interview, and debrief) stays within 105 minutes, under the two-hour cap, and the schedule fits the three-week target.

Opus 5.5 · Claude

Each interviewer's time sums to at most 2 hours, and the timeline fits within three weeks.

Judges on evidence, not presenceRightRightRight
Haiku 5.5 · API

Scorecard explicitly excludes 'presence' and 'smart' as criteria, demanding concrete evidence of experiments run, metrics moved, and influence gained.

GPT-6.1 Sol · API

The process requires concrete evidence (quotes, actions, artifacts) and explicitly rejects 'very smart' and 'great presence' as evidence.

Opus 5.5 · Claude

The scorecard requires specific evidence of results and bans impression words like 'presence' and 'smart'.

Defines good for this role firstRightRightRight
Haiku 5.5 · API

The scorecard sets out the competencies weighted for this role (experiment rigour, influence, ownership) with distinct strong and weak signals before any round descriptions.

GPT-6.1 Sol · API

The scorecard defines five weighted competencies with strong and weak signals, and the decision rule prioritizes influence, analytical judgment, and execution.

Opus 5.5 · Claude

It defines competencies with strong/weak signals, weighted for the role, before designing interviews.

Keeps each judgement independentRightRightRight
Haiku 5.5 · API

Interviewers submit written scores before the debrief, nobody sees others' scores first, and the group discusses evidence only, preventing groupthink.

GPT-6.1 Sol · API

Interviewers submit evidence and scores independently before discussion, and the debrief is an evidence review, not a show of hands.

Opus 5.5 · Claude

It requires written independent evaluations before debrief and orders speaking to avoid anchoring.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6.1 SolwithAPI91.7100.02None
2Sonnet 5.5withAPI91.792.32None
3GPT-6 AstrawithChatGPT93.880.82None
4Opus 5.5withClaude87.580.82None
5GPT-6 LunawithAPI89.676.92None
6Haiku 5.5withAPI87.573.12None
7Gemini 3.8 FlashwithAPI75.065.42None
8Gemini 3.5 Flash-LitewithGemini66.753.82None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.50607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6.1 Sol · API: task score 96, 10.0× the cheapestSonnet 5.5 · API: task score 92, 36× the cheapestGPT-6 Astra · ChatGPT: task score 86, 53× the cheapestOpus 5.5 · Claude: task score 83, 51× the cheapestGPT-6 Luna · API: task score 82, the cheapestHaiku 5.5 · API: task score 79, 1.7× the cheapestGemini 3.8 Flash · API: task score 69, 6.2× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 60, 1.7× the cheapestGPT-6.1 Sol · APISonnet 5.5 · APIGPT-6 Astra · ChatGPTOpus 5.5 · ClaudeGPT-6 Luna · APIHaiku 5.5 · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API821.0×
Haiku 5.5 · API791.7×
Gemini 3.5 Flash-Lite · Gemini601.7×
Gemini 3.8 Flash · API696.2×
GPT-6.1 Sol · API9610.0×
Sonnet 5.5 · API9236×
Opus 5.5 · Claude8351×
GPT-6 Astra · ChatGPT8653×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Hiring product managers.

Why it matters

A bad PM hire costs a team a year. Most loops reward polish and presence, and the debrief goes to whoever speaks first.

What good looks like

  • Defines what good looks like for this role before judging anyone
  • Judges on evidence of results and how they work with a team
  • Tests the real job, not a rehearsed framework
  • Keeps each interviewer's judgement independent

Deliberately not measured

  • Employment law
  • Compensation benchmarking
Capability tested

Hiring judgement

The failure we’re looking for

Hiring for presence and polish over evidence

Grading

Decision model and LLM judge, calibrated against a blind PM review