Tasks / Leadership

Hire a PM

Can the model design a hiring process that finds the right PM, and make the call on real candidates from the evidence?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 61% were usable with at most a quick edit.

Reliably right

  1. Produces the required deliverable100% pass
    The output provides a complete loop, scorecard, and decision plan in the requested form, under 900 words, usable by Amara with light edits.
    GPT-6.1 Sol · API · A loop for the first growth PM
  2. Tests what the last hire failed at100% pass
    The loop includes an influence simulation with the Head of Sales and a scorecard dimension on influence without authority, directly testing what the last hire failed at.
    GPT-6.1 Sol · API · A loop for the first growth PM
  3. Finds which rounds line up with performance100% pass
    It compares each round's scores with 12-month ratings, finding behavioral lines up and take-home, presentation, VP Engineering do not.
    GPT-6.1 Sol · API · What our interviews predict

Where it slips

  1. Uses the supplied evidence correctly56% pass
    The take-home correlation is stated as -0.57, but the data yields approximately -0.33; this fact about the current situation is incorrect.
    GPT-6 Astra · ChatGPT · What our interviews predict
  2. Avoids unsupported claims60% pass
    The claim 'A growth PM in a senior role is likely to be exactly that kind of person' is presented as fact without evidence and is not labelled as an assumption.
    Opus 5.5 · Claude · A loop for the first growth PM
  3. Defines good for this role first63% pass
    The memo does not explicitly name the competencies that matter most for this PM role and level, nor define what strong and weak look like before designing the interviews.
    Sonnet 5.5 · API · What our interviews predict

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Judges on evidence, not presence

    Does the output judge candidates, or design the process to judge them, on specific evidence of results they drove and how they worked with their team, rather than impressions such as presence, confidence or polish?

    Passes when Asks for or weighs concrete evidence: the metric a candidate moved, the part they played, how they shared credit, what they learned from a failure. Impressions are treated as weak signals.

  2. Defines good for this role first

    Does the output set out what good looks like for this specific role (the competencies that matter most at this level, and what strong and weak look like) before judging candidates or designing interviews?

    Passes when Names the competencies weighted for this role and level, with strong and weak signals, and the judgement or the process follows from them.

  3. Keeps each judgement independent

    Does the output keep each interviewer's judgement independent (written down before the group discusses it), and guard against the loudest voice or the first speaker deciding?

    Passes when Requires or relies on written, independent evaluations before discussion, and discounts views that changed under group pressure.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 3 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're helping Amara Osei, VP Product at Copperline, hire our first Growth PM (a senior role). She's shared her draft interview loop and asked you to redesign it. Write the loop you'd run (each round: who runs it, what it tests, how long it takes), the scorecard (what strong and weak look like for each thing we're testing), and how we'll make the decision at the end. No more than 900 words. What we know is below.

What the model was given6 items: About Copperline, The role, Amara's draft loop, The last two PM hires, From the recruiter, Who can interview
About CopperlineInvoicing and payments software for small accountancy firms. 60 people. Trial to paid conversion is 9%.
The roleOwns trial conversion and expansion revenue. Works with three engineers and a designer, reports to Amara, and has to ship experiments every week. Needs Sales and Marketing to change how they work, without managing them.
Amara's draft loop1. Recruiter screen. 2. Take-home: 'design a growth strategy for Copperline', a 10-page deck, five days to complete. 3. Presentation to a panel of six. 4. Culture fit with the CEO: 'would I grab a beer with them?' 5. Debrief: everyone discusses, then votes by show of hands. There's no scorecard.
The last two PM hiresOne left after five months: 'couldn't get Sales to change anything'. The other is doing well. Both sets of interview notes say mainly 'great presence' and 'very smart'.
From the recruiterOf the last 40 candidates given the take-home, 14 withdrew, saying they didn't have time. Nine of the 14 had young children or demanding current roles.
Who can interviewAmara, the Head of Engineering, a senior designer, the Head of Sales, a data analyst and the CEO. Each can give at most two hours to this hire. Amara wants an offer out within three weeks of the first interview.
What a strong answer doesThe answer key the graders mark against

A loop built on a scorecard written first: for a senior growth PM, experimentation and analytics, execution, and influence without authority (weighted up, since the last hire failed on it), with strong and weak signals for each. Rounds test the real job: a live diagnosis of Copperline's own trial funnel instead of a five-day generic deck (which loses candidates, mostly parents and people in demanding roles), a behavioural round on past experiments and metrics actually moved, and an influence round run by the Head of Sales. Replaces the 'grab a beer' culture fit with a structured values or collaboration conversation. Each interviewer writes their evaluation before the debrief, and the decision is made against the scorecard, not a show of hands. Fits each interviewer's two hours and the three-week target, and adds reference checks.

Critical failures (cap the score at 40)
  • Keeps a culture-fit test based on whether the interviewer would like a beer with the candidate
Task

Mid-level · v1.1

The outputs

9 setups answered this brief.

Usable with a quick edit11 of 12 checks passed

The output is complete and usable; the only missing piece is a brief mention of the recruiter's withdrawal data, which is a quick edit.

Output

Recommended loop

Replace the unpaid strategy deck, six-person presentation and “beer test” with structured evidence of doing this job. The previous hires’ notes cannot explain why one succeeded and one failed; cross-functional influence must be a hiring requirement, not a personality impression.

Rounds

RoundWho / durationWhat it tests and how
1. Structured screenRecruiter, 20 minutesConfirm scope, motivation, compensation and logistics. Ask for one conversion or expansion initiative: personal contribution, measured result and partner teams involved. Use consistent questions; accept relevant experience without requiring a Growth PM title. Advance at most two finalists under the interviewer-time budget.
2. Ownership deep diveAmara, 35 minutesExamine one growth initiative and one unsuccessful attempt to change another team’s behavior. Probe baseline, decision-making, individual contribution, results, uncertainty and lessons. Test senior ownership, growth judgment and influence through specific actions—not polished storytelling.
3. Live growth work sampleHead of Engineering + data analyst, 40 minutesGive everyone the same short, synthetic packet: a funnel beginning at Copperline’s 9% trial conversion, cohort/segment data, expansion opportunities, customer observations and delivery constraints. Allow eight minutes of quiet reading within the session. Ask candidates to diagnose, prioritize conversion versus expansion opportunities, and scope the first weekly experiment with three engineers and a designer. Introduce one standardized complication. Test analysis, prioritization, experimentation and delivery. No deck or advance work.
4. Cross-functional working sessionHead of Sales + senior designer, 35 minutesCandidate proposes a trial-conversion change. Sales resists because it changes incentives or workload; Design flags customer friction. Use identical opening objections, while responding naturally thereafter. Candidate must uncover interests, negotiate an achievable pilot and establish responsibilities and follow-through. Finish with reflection on what they would change. Test influence, customer judgment and collaboration—not whether they “win.”
5. Offer conversation—not another assessmentCEO, 20 minutes, selected candidate onlyExplain company direction, answer questions and help close. No informal culture-fit veto.

Allow candidate questions in each assessment round. Share the format, competencies and preparation expectations beforehand; offer scheduling flexibility, breaks and reasonable accommodations. Assess reasoning and decisions, not presentation polish. The work sample is fictional, not free consulting.

Scorecard

Use 1 = clear contrary evidence, 2 = below bar, 3 = meets senior-role bar, 4 = exceptional evidence. “Not observed” is separate from a low score and must be resolved before deciding.

Competency / primary evidence ownerStrong: meets or exceeds barWeak: below bar
Growth diagnosis and measurement / analystDefines conversion and expansion denominators, examines cohorts and segments, distinguishes correlation from causation, identifies missing evidence. Treats 9% as a starting point, not a diagnosis.Jumps to tactics; misreads funnels; claims causality from movement; ignores retention or expansion economics.
Experiment design and prioritization / analyst, AmaraChooses a defensible hypothesis, outcome and guardrails; weighs impact, effort and confidence; handles limited traffic and inconclusive results.Produces a tactic list; promises unsupported lifts; assumes every weekly release can deliver statistically conclusive results.
Weekly delivery with a small team / EngineeringSlices a useful first release, identifies instrumentation and dependencies, trades scope deliberately, and plans the next decision.Requires a major rebuild or more staff; ignores technical uncertainty; equates speed with skipping measurement or quality.
Influence without authority / SalesInvestigates incentives and constraints, makes a credible mutual-value case, secures specific commitments and establishes follow-through. Past examples show personal action and honest limits.Relies on title, escalation or charisma; blames Sales; mistakes agreement in a meeting for changed behavior.
Customer and commercial judgment / DesignConnects accountancy-firm workflows to conversion and expansion; protects trust and usability; considers downstream retention and revenue quality.Optimizes conversion through friction, pressure or misleading choices; ignores customer costs or churn.
Senior ownership and learning / AmaraSeparates own contribution from team results, owns misses, changes course with evidence and creates a repeatable operating rhythm.Offers vague achievements, cannot explain decisions, takes all credit or externalizes failures.

Interviewers record concrete actions, quotes and reasoning alongside scores. “Smart,” “great presence” and “likeable” are not evidence.

Decision, capacity and timeline

Week 1: recruiter screens and schedules two finalists. Week 2: complete assessments. Week 3: decide, close and issue the offer.

Reserve each assessor 10 minutes for calibration, five minutes per finalist for scoring, and 20 minutes for debrief. Including interviews, Engineering and the analyst each use exactly 120 minutes; Amara, Sales and Design each use 110. CEO uses at most 40 minutes. If neither finalist qualifies, reopen rather than lower the bar or silently exceed the budget.

Submit scores independently before seeing others’ notes. In debrief, review evidence competency by competency; Amara speaks last. No show-of-hands vote.

Require at least 3 on every competency, especially influence and weekly delivery. Resolve disagreements through recorded evidence, not averaging or seniority. Missing critical evidence means no offer yet. Amara owns the final decision and documents the rationale, remaining risks and onboarding support.

Grades and run details

Decision model 92 · LLM judge 92
Decision model checks
  • passUses the supplied evidence correctly56%
  • passAddresses the actual decision51%
  • partialRespects explicit constraints14%
  • passIdentifies material uncertainty95%
  • passAvoids unsupported claims48%
  • passProduces the required deliverable92%
  • passTests what the last hire failed at100%
  • partialFixes the take-home's cost to candidates85%
  • passFits the people and the time73%
  • passJudges on evidence, not presence100%
  • passDefines good for this role first95%
  • passKeeps each judgement independent100%
Run
Run
#1
Time to output
62 s
Submitted
5 Oct 2026

Usable with a quick edit9 of 12 checks passed

The output is a complete, actionable loop with clear evidence-based scorecard and decision process. A PM could adopt it with minimal edits, even though it omits explicit uncertainty acknowledgment.

Output

Here is the redesigned interview loop for Copperline’s first Growth PM.

We’ve removed the 10-page take-home—which was costing us 35% of our pipeline, disproportionately filtering out parents and busy top performers—and replaced the six-person panel, which burns interviewer hours and selects for "presence" over substance. We’ve also replaced the CEO’s "beer test" and the hand-raise vote to eliminate groupthink and bias.

This new loop fits your 3-week timeline, requires zero unpaid homework, and keeps every interviewer well under their 2-hour cap per candidate.

1. The Interview Loop

Stage 1: Recruiter Screen (30 mins) * Interviewer: Recruiter * What it tests: Role alignment, compensation, B2B SaaS growth experience, and baseline appetite for high-velocity experimentation.

Stage 2: Hiring Manager Screen (45 mins) * Interviewer: Amara (VP Product) * What it tests: Past Impact & Strategic Thinking. Deep dive into a past funnel or expansion initiative they owned end-to-end. Focuses on how they diagnose bottlenecks and drive commercial outcomes.

Stage 3: Live Growth Jam (60 mins) * Interviewers: Data Analyst & Senior Designer (plus 15 mins prep) * What it tests: Experimentation Velocity & Squad Collaboration. No prep required. We share a mock snapshot of Copperline’s 9% trial-to-paid funnel during the call. The candidate works live with the Analyst and Designer to diagnose drop-offs, form hypotheses, and scope two scrappy experiments to ship next week.

Stage 4: Go-To-Market Alignment (45 mins) * Interviewer: Head of Sales * What it tests: Cross-Functional Influence. Directly addresses why our last PM left. Behavioral questions and a live roleplay on getting Sales and Marketing to change workflows without direct authority.

Stage 5: Delivery & Ambition (60 mins — two 30-min back-to-backs) * Part A: Head of Engineering (30 mins). Tests Pragmatic Execution: balancing tech debt with weekly experiment shipping, and how they work with engineers. * Part B: CEO (30 mins). Tests Commercial Empathy & Ownership: passion for small accountancy firms, comfort with startup ambiguity, and executive alignment.

Time commitment per finalist: Amara (1.25 hrs), Analyst/Designer (1.5 hrs), Sales (1.25 hrs), Eng/CEO (1 hr)—including the 30-minute debrief.

---

2. The Scorecard

Interviewers will grade only their assigned competencies on a 1–4 scale (1 = Strong No, 2 = No, 3 = Yes, 4 = Strong Yes), citing specific evidence—never "great presence" or "very smart."

1. Experimentation Velocity & Rigor (Tested in: Live Jam, Eng) * Strong: Comfortably forms hypotheses from imperfect data. ruthlessly scopes MVPs down to ship within a week. Focuses on learning velocity over perfection. * Weak: Wants months of research before shipping. Designs massive features instead of tests. Confuses shipping features with moving metrics.

2. Cross-Functional Influence (Tested in: GTM Alignment, HM Screen) * Strong: Speaks the language of Sales (quota, pipeline, commission) and Marketing. Co-creates solutions, anticipates resistance, and uses data and quick wins to earn trust. Has clear examples of changing GTM behavior. * Weak: Expects Sales/Marketing to comply "because the data says so." Complains about stubborn stakeholders or escalates to the VP immediately when blocked.

3. Funnel & Expansion Strategy (Tested in: HM Screen, Live Jam) * Strong: Connects micro-experiments to macro revenue goals. Clearly articulates levers for moving a 9% conversion rate and driving expansion in a B2B workflow tool. * Weak: Throws random growth hacks at the wall. Focuses purely on top-of-funnel acquisition or B2C viral loops that don’t apply to accountancy firms.

4. Squad Collaboration & Execution (Tested in: Live Jam, Eng) * Strong: Treats Design, Data, and Eng as problem-solving partners, not ticket-takers. Welcomes pushback during the Live Jam and adapts their ideas. * Weak: Dictates exact UI solutions to the designer; ignores technical constraints raised by engineering; talks over teammates.

5. Commercial & Customer Empathy (Tested in: CEO, HM Screen) * Strong: Genuinely curious about how small accountancy firms work, make money, and experience friction with invoicing/payments. High ownership mentality. * Weak: Views users solely as rows in a database. Uninterested in the unglamorous realities of accounting software.

---

3. Making the Decision

  1. Blind Submission: Interviewers must submit written scorecards in our ATS within 24 hours of their interview. No one can see others’ feedback until their own is locked in.
  2. Pre-Read (5 mins): At the start of the 30-minute debrief, everyone silently reads all submitted scorecards.
  3. Evidence-Based Discussion (20 mins): No show of hands. We discuss only areas of calibration discrepancy (e.g., Sales scored Influence a 2; Amara scored it a 4) or missing evidence. If someone says a candidate is "smart," the moderator (Amara or Recruiter) will ask: "What specific behavior showed that, and how does it map to the scorecard?"
  4. The Call (5 mins): If a candidate scores below a 3 on Cross-Functional Influence or Experimentation Velocity, they are an automatic pass. Otherwise, Amara—as the hiring manager—weighs the evidence and makes the final hiring decision. Single-threaded ownership, no voting.

Grades and run details

Decision model 75 · LLM judge 92
Decision model checks
  • failUses the supplied evidence correctly56%
  • failAddresses the actual decision31%
  • partialRespects explicit constraints19%
  • passIdentifies material uncertainty35%
  • partialAvoids unsupported claims62%
  • passProduces the required deliverable72%
  • passTests what the last hire failed at100%
  • passFixes the take-home's cost to candidates98%
  • passFits the people and the time52%
  • passJudges on evidence, not presence100%
  • passDefines good for this role first85%
  • passKeeps each judgement independent100%
Run
Run
#1
API response time
15 s
Submitted
13 Oct 2026

Usable with a quick edit11 of 12 checks passed

The output is well-structured and complete; the one unsupported generalization can be removed with a quick edit without affecting the loop design.

Output

Copperline Growth PM: Interview Loop

Why the draft needs changing

  • The take-home is filtering out the wrong people. 14 of 40 candidates withdrew, and most of them were parents or in demanding jobs. A growth PM in a senior role is likely to be exactly that kind of person.
  • The panel of six and the "beer" test reward presence. Presence is what your interview notes captured last time, and it didn't predict success.
  • Nothing tested influence over Sales. That is why the last hire failed.

The new loop tests five things, each owned by one interviewer.

Time budget

Each interviewer has two hours in total for this hire. That limits the loop to three finalists. Per interviewer, the time breaks down as:

  • 3 × 30-minute interviews
  • 3 × 5-minute scorecards
  • one 15-minute debrief

That totals 2 hours, or 1.5 hours for the CEO. The recruiter screens with no time limit, so screening carries the shortlisting load.

The loop

RoundWhoTestsLength
0. ScreenRecruiterHas owned a conversion or expansion metric; has shipped experiments at pace. Structured questions, scored pass/fail.30 min
1. Growth deep-diveAmaraExperiment velocity. Walk through one experiment programme they ran: cadence, hit rate, what they killed.30 min
2. Funnel caseData analystQuantitative judgement. Read an anonymised Copperline trial funnel in the session (10 min) and say where the 9% is leaking, what they'd test first and how they'd know. No prep.30 min
3. Influence role-playHead of SalesInfluence without authority. The data shows Sales' handling of trial accounts hurts conversion. The Head of Sales plays himself and pushes back realistically. Ends with "tell me about a time this went badly."30 min
4. Build sessionHead of Engineering + senior designerWorking with the team. Take one idea from the funnel case and scope it to ship in a week: what gets cut, what gets instrumented, what the designer needs.30 min
5. Commercial judgementCEOCommercial sense. Where expansion revenue comes from in small accountancy firms, pricing levers, and what they'd not do. Structured questions. No "culture fit."20 min

The candidate spends about 2.5 hours in total, ideally in one afternoon (remote is fine). Candidates get the round list and competencies in advance.

Scorecard

Each interviewer scores only their own competency, on a scale of 1 to 4. Every score must cite specific evidence: what the candidate said or did. The words "presence," "smart" and "great energy" don't count as evidence.

CompetencyStrong (4)Weak (1)
Experiment velocity (must-have)Names a real cadence (e.g. weekly), a real hit rate, and experiments they killed fast. Talks about learning per week, not big launches.Describes one large project. Can't say how many tests they ran or what failed. "Strategy" without shipping.
Quantitative judgementSpots the biggest leak quickly. Questions data quality. Sizes the opportunity. Defines a success metric and minimum sample before testing.Jumps to solutions. Treats all drop-offs as equal. No view on how to measure the result.
Influence without authority (must-have)Asks what Sales is paid on and afraid of. Proposes a small reversible pilot with Sales sharing the credit. Stays calm under pushback. Has a real story of winning over a resistant team.Leans on data or seniority ("the numbers say so"). Escalates to Amara or the CEO early. Gives up or bulldozes. Past examples are all teams they managed.
Working with Engineering and DesignCuts scope hard to hit a week. Asks engineers for options rather than dictating. Gives the designer a problem, not a mock-up. Plans instrumentation up front.Specs a month of work. Dictates solutions. Forgets tracking.
Commercial senseUnderstands how small accountancy firms buy and grow. Links trial and expansion to revenue and margin. Names trade-offs and things they'd refuse to do.Generic SaaS playbook. Can't connect activity to revenue. Agrees with everything.

Overall recommendation for each interviewer: Strong hire, Hire, No hire or Strong no hire.

How we decide

1. Score independently. Scorecards are submitted within two hours of the interview and before the debrief. Nobody sees others' scores until theirs is in. 2. Short debrief. One 15-minute debrief covers all three finalists, run by Amara after the final loop. Everyone reads the scorecards beforehand. The meeting is for resolving disagreements, not for retelling the interviews. 3. Speaking order. The most junior interviewer speaks first (data analyst, then designer, Engineering, Sales, CEO). The CEO and Amara speak last, so seniority doesn't anchor the room. 4. No show of hands. The bar is explicit: - A 3 or higher on both must-haves: experiment velocity and influence. - No 1s anywhere. - An average of at least 3. 5. Who decides. Amara makes the final call among candidates who clear the bar. A 1 from the Head of Sales on influence blocks the hire. This is the exact failure mode from last time. 6. If nobody clears the bar, we don't hire. We reopen the search rather than lowering the bar to hit the deadline. 7. References. We take one reference from someone in Sales or Marketing who worked alongside the candidate, not someone who reported to them. We ask specifically how the candidate got them to change something.

Timeline (offer within three weeks)

DaysActivity
1–5Recruiter screens. Shortlist three finalists.
6–10Finalist loops, all in the same week so comparisons stay fresh.
11Debrief and decision.
12–14References.
15Offer out. This leaves about a week of buffer for scheduling slips or a second-choice candidate.

Grades and run details

Decision model 96 · LLM judge 92
Decision model checks
  • passUses the supplied evidence correctly33%
  • passAddresses the actual decision46%
  • passRespects explicit constraints19%
  • passIdentifies material uncertainty88%
  • partialAvoids unsupported claims34%
  • passProduces the required deliverable86%
  • passTests what the last hire failed at100%
  • passFixes the take-home's cost to candidates100%
  • passFits the people and the time59%
  • passJudges on evidence, not presence100%
  • passDefines good for this role first94%
  • passKeeps each judgement independent100%
Run
Run
#1
Time to output
48 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 5

Uses the supplied evidence correctlyRightMixedRight
GPT-6 Astra · ChatGPT

The output uses the 9% conversion rate and the insufficient interview notes correctly, and invents no facts about the current situation.

Gemini 4 Argon · API

All factual claims about the current situation are directly supported by the brief or supplied context.

Opus 5.5 · Claude

All factual claims about the current situation are taken directly from the brief or supplied context, with no inventions.

Addresses the actual decisionRightMixedRight
GPT-6 Astra · ChatGPT

The output commits to a specific redesigned loop and decision process, with Amara owning the final call and a clear condition to reopen if no candidate meets the bar.

Gemini 4 Argon · API

The output commits to a specific redesigned loop, scorecard, and decision process, framed for Amara, and states clear conditions that would lead to a pass (score below 3 on key competencies) and the final hiring call.

Opus 5.5 · Claude

The output commits to a specific interview loop, scorecard, and decision process, framed for Amara, with clear conditions for not hiring.

Identifies material uncertaintyRightMixedRight
GPT-6 Astra · ChatGPT

It names the condition that would change the decision (if neither finalist qualifies, reopen) and that missing critical evidence means no offer.

Gemini 4 Argon · API

The output does not name any unknowns that could alter the decision or describe how they would be resolved; it presents the process as definitive without acknowledging external uncertainties.

Opus 5.5 · Claude

It identifies that if no candidate clears the bar, the search reopens, and provides buffer for scheduling slips.

Avoids unsupported claimsRightRightWrong
GPT-6 Astra · ChatGPT

The only interpretation (notes can't explain outcomes) is directly supported by the brief; no other claims are presented as established fact.

Gemini 4 Argon · API

Interpretations like 'disproportionately filtering out parents and busy top performers' are directly anchored in the recruiter's withdrawal data and not presented as speculation.

Opus 5.5 · Claude

The claim 'A growth PM in a senior role is likely to be exactly that kind of person' is presented as fact without evidence and is not labelled as an assumption.

Fixes the take-home's cost to candidatesWrongRightRight
GPT-6 Astra · ChatGPT

The output replaces the take-home with a live exercise but never cites the recruiter's withdrawal data or who the take-home was driving away.

Gemini 4 Argon · API

The five-day take-home is replaced with a live 60-minute exercise on Copperline's real funnel, and the withdrawal data (35%, parents/busy candidates) is explicitly cited.

Opus 5.5 · Claude

It replaces the five-day take-home with a live funnel case and cites the withdrawal data and who it drove away.

All got right 7

Respects explicit constraintsRightRightRight
GPT-6 Astra · ChatGPT

The output is a loop, scorecard, and decision process within 900 words, addressed to Amara, and respects all stated constraints.

Gemini 4 Argon · API

The loop respects the 900-word limit, keeps each interviewer under 2 hours, fits the 3-week timeline, and provides the required loop, scorecard, and decision method.

Opus 5.5 · Claude

The output respects the 900-word limit, replaces the take-home and beer test, and fits interviewer time and three-week deadline.

Produces the required deliverableRightRightRight
GPT-6 Astra · ChatGPT

The output provides a complete loop, scorecard, and decision process that a PM could act on with light edits.

Gemini 4 Argon · API

The output includes a complete redesigned loop, a scorecard with strong/weak signals, and a clear decision-making procedure, all within the word limit and usable by a product manager.

Opus 5.5 · Claude

The output is a complete interview loop, scorecard, and decision process, within the word limit, usable by Amara.

Tests what the last hire failed atRightRightRight
GPT-6 Astra · ChatGPT

Round 4 is a cross-functional working session with the Head of Sales, and the scorecard includes 'Influence without authority' with strong/weak signals.

Gemini 4 Argon · API

A dedicated round with the Head of Sales tests cross-functional influence, directly addressing the reason the last PM left, with strong and weak signals defined.

Opus 5.5 · Claude

The loop includes an influence role-play with the Head of Sales and probes past examples of changing another team's work.

Fits the people and the timeRightRightRight
GPT-6 Astra · ChatGPT

Interviewer times are explicitly kept within two hours (Engineering and analyst 120 min, others 110 min, CEO 40 min) and the three-week schedule is detailed.

Gemini 4 Argon · API

Stated interviewer times (Amara 1.25h, Analyst/Designer 1.5h, Sales 1.25h, Eng/CEO 1h) all fall under the 2-hour cap, and the structure supports a three-week offer timeline.

Opus 5.5 · Claude

Each interviewer's time sums to at most 2 hours, and the timeline fits within three weeks.

Judges on evidence, not presenceRightRightRight
GPT-6 Astra · ChatGPT

The scorecard demands concrete actions, quotes, and reasoning, and explicitly rejects 'smart', 'great presence', and 'likeable' as evidence.

Gemini 4 Argon · API

The process explicitly forbids vague impressions, requires specific evidence in scorecards, and defines behaviors (moved metrics, handled pushback) as the basis for ratings.

Opus 5.5 · Claude

The scorecard requires specific evidence of results and bans impression words like 'presence' and 'smart'.

Defines good for this role firstRightRightRight
GPT-6 Astra · ChatGPT

The scorecard defines six competencies with strong and weak signals before any interview design, weighting influence and delivery for this role.

Gemini 4 Argon · API

The scorecard defines the key competencies for this senior growth PM role with strong/weak signals, and the process automatically fails candidates below 3 on the weighted competencies.

Opus 5.5 · Claude

It defines competencies with strong/weak signals, weighted for the role, before designing interviews.

Keeps each judgement independentRightRightRight
GPT-6 Astra · ChatGPT

Interviewers must submit scores independently before seeing others' notes, and Amara speaks last; no show-of-hands vote.

Gemini 4 Argon · API

Interviewers must submit written scorecards independently before seeing others' feedback, and discussion is limited to calibration discrepancies, not group opinion.

Opus 5.5 · Claude

It requires written independent evaluations before debrief and orders speaking to avoid anchoring.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6.1 SolwithAPI91.7100.02None
2Sonnet 5.5withAPI85.492.32None
3GPT-6 AstrawithChatGPT89.680.82None
4Opus 5.5withClaude89.680.82None
5GPT-6 LunawithAPI87.576.92None
6Haiku 5.5withAPI79.273.12None
7Gemini 4 ArgonwithAPI70.880.82None
8Gemini 3.8 FlashwithAPI75.065.42None
9Gemini 3.5 Flash-LitewithGemini66.753.82None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.50607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6.1 Sol · API: task score 96, 10.0× the cheapestSonnet 5.5 · API: task score 89, 36× the cheapestGPT-6 Astra · ChatGPT: task score 84, 53× the cheapestOpus 5.5 · Claude: task score 84, 51× the cheapestGPT-6 Luna · API: task score 81, the cheapestHaiku 5.5 · API: task score 75, 1.7× the cheapestGemini 4 Argon · API: task score 75, 8.1× the cheapestGemini 3.8 Flash · API: task score 69, 6.2× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 60, 1.7× the cheapestGPT-6.1 Sol · APISonnet 5.5 · APIGPT-6 Astra · ChatGPTOpus 5.5 · ClaudeGPT-6 Luna · APIHaiku 5.5 · APIGemini 4 Argon · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API811.0×
Haiku 5.5 · API751.7×
Gemini 3.5 Flash-Lite · Gemini601.7×
Gemini 3.8 Flash · API696.2×
Gemini 4 Argon · API758.1×
GPT-6.1 Sol · API9610.0×
Sonnet 5.5 · API8936×
Opus 5.5 · Claude8451×
GPT-6 Astra · ChatGPT8453×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Hiring product managers.

Why it matters

A bad PM hire costs a team a year. Most loops reward polish and presence, and the debrief goes to whoever speaks first.

What good looks like

  • Defines what good looks like for this role before judging anyone
  • Judges on evidence of results and how they work with a team
  • Tests the real job, not a rehearsed framework
  • Keeps each interviewer's judgement independent

Deliberately not measured

  • Employment law
  • Compensation benchmarking
Capability tested

Hiring judgement

The failure we’re looking for

Hiring for presence and polish over evidence

Grading

Decision model and LLM judge, calibrated against a blind PM review