Tasks / Leadership

Hire a PM

Can the model design a hiring process that finds the right PM, and make the call on real candidates from the evidence?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 61% were usable with at most a quick edit.

Reliably right

  1. Produces the required deliverable100% pass
    The output provides a complete loop, scorecard, and decision plan in the requested form, under 900 words, usable by Amara with light edits.
    GPT-6.1 Sol · API · A loop for the first growth PM
  2. Tests what the last hire failed at100% pass
    The loop includes an influence simulation with the Head of Sales and a scorecard dimension on influence without authority, directly testing what the last hire failed at.
    GPT-6.1 Sol · API · A loop for the first growth PM
  3. Finds which rounds line up with performance100% pass
    It compares each round's scores with 12-month ratings, finding behavioral lines up and take-home, presentation, VP Engineering do not.
    GPT-6.1 Sol · API · What our interviews predict

Where it slips

  1. Uses the supplied evidence correctly56% pass
    The take-home correlation is stated as -0.57, but the data yields approximately -0.33; this fact about the current situation is incorrect.
    GPT-6 Astra · ChatGPT · What our interviews predict
  2. Avoids unsupported claims60% pass
    The claim 'A growth PM in a senior role is likely to be exactly that kind of person' is presented as fact without evidence and is not labelled as an assumption.
    Opus 5.5 · Claude · A loop for the first growth PM
  3. Defines good for this role first63% pass
    The memo does not explicitly name the competencies that matter most for this PM role and level, nor define what strong and weak look like before designing the interviews.
    Sonnet 5.5 · API · What our interviews predict

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Judges on evidence, not presence

    Does the output judge candidates, or design the process to judge them, on specific evidence of results they drove and how they worked with their team, rather than impressions such as presence, confidence or polish?

    Passes when Asks for or weighs concrete evidence: the metric a candidate moved, the part they played, how they shared credit, what they learned from a failure. Impressions are treated as weak signals.

  2. Defines good for this role first

    Does the output set out what good looks like for this specific role (the competencies that matter most at this level, and what strong and weak look like) before judging candidates or designing interviews?

    Passes when Names the competencies weighted for this role and level, with strong and weak signals, and the judgement or the process follows from them.

  3. Keeps each judgement independent

    Does the output keep each interviewer's judgement independent (written down before the group discusses it), and guard against the loudest voice or the first speaker deciding?

    Passes when Requires or relies on written, independent evaluations before discussion, and discounts views that changed under group pressure.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 3 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're helping Amara Osei, VP Product at Copperline, hire our first Growth PM (a senior role). She's shared her draft interview loop and asked you to redesign it. Write the loop you'd run (each round: who runs it, what it tests, how long it takes), the scorecard (what strong and weak look like for each thing we're testing), and how we'll make the decision at the end. No more than 900 words. What we know is below.

What the model was given6 items: About Copperline, The role, Amara's draft loop, The last two PM hires, From the recruiter, Who can interview
About CopperlineInvoicing and payments software for small accountancy firms. 60 people. Trial to paid conversion is 9%.
The roleOwns trial conversion and expansion revenue. Works with three engineers and a designer, reports to Amara, and has to ship experiments every week. Needs Sales and Marketing to change how they work, without managing them.
Amara's draft loop1. Recruiter screen. 2. Take-home: 'design a growth strategy for Copperline', a 10-page deck, five days to complete. 3. Presentation to a panel of six. 4. Culture fit with the CEO: 'would I grab a beer with them?' 5. Debrief: everyone discusses, then votes by show of hands. There's no scorecard.
The last two PM hiresOne left after five months: 'couldn't get Sales to change anything'. The other is doing well. Both sets of interview notes say mainly 'great presence' and 'very smart'.
From the recruiterOf the last 40 candidates given the take-home, 14 withdrew, saying they didn't have time. Nine of the 14 had young children or demanding current roles.
Who can interviewAmara, the Head of Engineering, a senior designer, the Head of Sales, a data analyst and the CEO. Each can give at most two hours to this hire. Amara wants an offer out within three weeks of the first interview.
What a strong answer doesThe answer key the graders mark against

A loop built on a scorecard written first: for a senior growth PM, experimentation and analytics, execution, and influence without authority (weighted up, since the last hire failed on it), with strong and weak signals for each. Rounds test the real job: a live diagnosis of Copperline's own trial funnel instead of a five-day generic deck (which loses candidates, mostly parents and people in demanding roles), a behavioural round on past experiments and metrics actually moved, and an influence round run by the Head of Sales. Replaces the 'grab a beer' culture fit with a structured values or collaboration conversation. Each interviewer writes their evaluation before the debrief, and the decision is made against the scorecard, not a show of hands. Fits each interviewer's two hours and the three-week target, and adds reference checks.

Critical failures (cap the score at 40)
  • Keeps a culture-fit test based on whether the interviewer would like a beer with the candidate
Task

Mid-level · v1.1

The outputs

9 setups answered this brief.

Usable with a quick edit10 of 12 checks passed

The output is a complete, actionable plan that a PM could implement with minor wording tweaks; no evidence re-checking or reasoning redo is needed.

Output

Growth PM interview loop

Assumptions: The two-hour cap is each person's total across all candidates, so I've designed for three finalists. Early rounds are cheap and don't use the panel's time. Only Amara's time is spent before the finalist stage.

Why the change

  • The 5-day, 10-page deck cost us 14 of 40 candidates, mostly people with caregiving duties or demanding jobs. It also tests deck-making, not the job.
  • The last hire failed on influencing Sales, and nothing in the old loop tested that.
  • "Great presence" and "very smart" in the notes are not evidence, and a show-of-hands vote rewards the loudest voice.

The loop

#RoundWhoTestsTime per candidate
1Screen + written questionRecruiter, scored by AmaraBasics, comp, and a 300-word written answer: "A conversion experiment you ran: baseline, result, what you'd change."30 min call, 20 min writing
2Ownership deep-diveAmaraTrack record on revenue outcomes20 min
3Funnel caseData analystConversion judgment30 min
4Ship-it scopingHead of Eng + senior designerWeekly shipping with a small team30 min
5Influence role-playHead of SalesChanging others' behavior without authority30 min
6Business and working styleCEOJudgment, first-hire fit20 min
7DebriefAll interviewersDecision20 min total

Rounds 3–6 can be done in one sitting, or split across two days on video. Total candidate time is about 3 hours.

Round details - Round 1: Amara spends 5 minutes on each of about six written answers (30 min total) and picks three finalists. - Round 3: We send a one-page anonymized trial funnel 24 hours ahead, with a note to spend no more than 30 minutes on it. In the room the candidate says where they'd look first, picks two experiments, and designs one, including metric, guardrail, and what a realistic sample size allows at our volume. - Round 4: We give a half-formed experiment idea. The candidate must cut it to something shippable within a week by three engineers and a designer. The interviewers push back on scope. - Round 5: The Head of Sales plays a skeptical rep or manager who thinks trial follow-up is a distraction. The candidate has to get agreement on a small pilot. The last 5 minutes cover a real past example. - Round 6: A structured conversation covering how the candidate would think about expansion revenue for small accountancy firms, what being the first growth hire means, and a time they disagreed and then committed. This replaces "would I grab a beer."

Scorecard

Each criterion is rated 1–4, with no midpoint. Every interviewer writes specific observed behaviors, not impressions. "Presence" and "smart" don't count as evidence.

1. Influence without authority (weight 30%; Sales, plus Amara and references) - Strong: Starts from what Sales cares about (quota, time, commissions). Proposes a small, reversible pilot. Offers something in return. Has a past example where a team they didn't manage changed behavior, with details. - Weak: Says "align stakeholders." Escalates to the boss. Presents the change as a mandate. Blames other teams in past stories.

2. Conversion judgment (25%; Analyst) - Strong: Segments before theorizing. Notes that our volume limits what tests can detect. Ties each experiment to a hypothesis and a guardrail. Asks about accountancy seasonality and who actually decides to pay. - Weak: Lists generic tactics (checklists, gamification). Has no view on sample size. Doesn't ask about the data.

3. Shipping cadence (20%; Eng + Design) - Strong: Cuts scope readily to the smallest test. Gives real numbers on past experiments per week or month. Treats engineers and designers as partners and takes their objections seriously. - Weak: Proposes large projects with distant payoffs. Writes specs without hypotheses. Defends scope instead of trading it off.

4. Ownership of revenue outcomes (15%; Amara) - Strong: Gives baselines, deltas, and their specific contribution. Talks candidly about experiments that failed. - Weak: Vague credit for team results. Describes outputs (launches, decks) instead of outcomes.

5. Judgment and fit for a 60-person company (10%; CEO) - Strong: Is energized by ambiguity. Builds their own tools and doesn't wait for resources. Disagrees directly and then commits. - Weak: Needs a playbook or a team. Can't prioritize without data. Becomes defensive when challenged.

Note for the Head of Sales: Score what the candidate did in the role-play, not whether you enjoyed being challenged.

Decision process

  1. Independent scoring. Each interviewer submits ratings and evidence before the debrief and can't see others' scores.
  2. Debrief (20 min). Amara reads out the scores, lowest first, on each criterion. Disagreements get discussed against the evidence, not general impressions.
  3. Must-haves. A score of 2 or below on influence or shipping means no offer. The last miss was an influence failure, and weak shipping defeats the weekly-experiment job.
  4. Ranking. Among candidates who pass, the highest weighted score wins. Close calls go to Amara.
  5. Decision owner. Amara decides, and there is no vote. The CEO can block only by citing specific observed evidence tied to the scorecard.
  6. References. The recruiter checks references for the top candidate, using Amara's script. At least one reference must be a cross-functional partner such as someone from sales or marketing, asking, "Did they change how your team worked? How?"

Timeline and time budget

  • Week 1: Recruiter screens and written questions. Amara picks finalists by the end of the week.
  • Week 2: Amara's deep-dives, then rounds 3–6 for all three finalists.
  • Week 3: Debrief on Monday, references on Tuesday and Wednesday, offer by Friday.
PersonTime
Amara30 min (written answers) + 60 min (3 deep-dives) + 20 min (debrief) = 110 min
Analyst, Sales, Eng, Design90 min (3 interviews) + 20 min (debrief) = 110 min each
CEO60 min + 20 min = 80 min

If more than three candidates reach round 2, Amara's time is the first limit. In that case, tighten the written screen before asking for more of her time.

Grades and run details

Decision model 88 · LLM judge 92
Decision model checks
  • passUses the supplied evidence correctly28%
  • failAddresses the actual decision25%
  • passRespects explicit constraints38%
  • passIdentifies material uncertainty70%
  • partialAvoids unsupported claims30%
  • passProduces the required deliverable91%
  • passTests what the last hire failed at100%
  • passFixes the take-home's cost to candidates100%
  • passFits the people and the time82%
  • passJudges on evidence, not presence100%
  • passDefines good for this role first91%
  • passKeeps each judgement independent98%
Run
Run
#1
API response time
53 s
Submitted
5 Oct 2026

Usable with a quick edit9 of 12 checks passed

The output is fully structured and actionable, needing only a one-sentence addition citing the recruiter's withdrawal data to complete the take-home justification.

Output

Redesigned Growth PM Interview Loop

Principles: Every round tests a named criterion. Scoring happens in writing before anyone discusses. Candidate time is capped. Interviewer time is capped at two hours each.

The loop (about three weeks)

Round 1: Recruiter screen (30 min) Runs: Recruiter Tests: Evidence of shipping experiments on a weekly cadence, compensation, logistics. Explain the full loop and its time cap up front so candidates can decide early. Knockout: No recent experience running experiments with a regular cadence.

Round 2: Analytical work sample (candidate time capped at 2 hours; due within 5 days, candidate picks when) Runs: Data Analyst scores; Amara reviews. Tests: Diagnosis and experiment rigour. Brief: An anonymised trial dataset showing the 9% conversion rate. Where is the funnel leaking, what would you test first, and what would you need from Sales and Engineering? Output: A two-page memo, not a deck. One diagnosis, three ranked experiments, and one fully specified (hypothesis, primary metric, guardrail metric, sample size, engineering effort). Alternative: A 60-minute live version on a shorter dataset, for candidates who can't spare two hours. Both versions are scored on the same rubric. Interviewer time: 45 min review.

Round 3: Build session (60 min, joint) Runs: Head of Engineering and senior designer. Tests: Engineering partnership and design judgment. Can the candidate cut their top experiment to something shippable this week? Do they know what needs instrumentation and feature flags, and what the design needs to look like? Interviewer time: 60 min each.

Round 4: Influence role-play (45 min) Runs: Head of Sales. Tests: Influence without authority, and understanding of sales incentives. Scenario: Sales wants to keep manual follow-ups for trial users. The candidate needs reps to run a three-week test that changes their routine. The Head of Sales plays a realistic, sceptical rep. Interviewer time: 45 min.

Round 5: Ownership and judgment (60 min) Runs: Amara. Tests: Ownership, learning from failure, decisions on incomplete data. Format: Structured behavioural questions. Ask for three past experiments: one that failed, one where they changed a team's behaviour without authority, one decision made on incomplete data. Probe what the candidate personally did, not what the team did. Interviewer time: 60 min.

Round 6: Working norms (30 min) Runs: CEO. Tests: Candour, comfort with pace and ambiguity. This replaces "would I grab a beer with them" with defined questions: - Tell me about disagreeing with a senior leader. What happened next? - A test you championed lost. What did you do? - What pace and level of ambiguity do you need to do your best work? Interviewer time: 30 min.

Final debrief (45 min): Amara and all interviewers, using written scores.

Timeline: Days 1–2 screen and brief issued. Days 2–6 work sample. Week 2: Rounds 3–5. Week 3: Round 6, debrief within 48 hours, offer.

Interviewer load: Data Analyst about 1h; Head of Engineering and senior designer 1h each; Head of Sales 45 min; Amara about 2h 15m (Round 5 plus review and debrief; she can reduce this by handing Round 2 review to the analyst, as above); CEO 30 min plus debrief.

Scorecard

Rate each criterion 1–4. Scores of 3 or 4 are "meets bar." Criteria marked ★ are must-haves.

CriterionStrong (3–4)Weak (1–2)
★ Experiment rigourNames a clear hypothesis, primary metric, guardrail, and sample size. Has run experiments weekly. Kills losing tests cleanly and says what they learned.Describes strategy but not tests. Can't say how many experiments they ran or what changed as a result.
Funnel diagnosisReads the data before proposing ideas. Questions the metric and names where the leak is and how they'd confirm it.Jumps to features or ideas without evidence. Takes the 9% at face value.
★ Influence without authorityGives a specific example of changing a team that didn't report to them. Uses the other team's incentives. Names trade-offs and what they gave up."I presented it and they agreed." Blames Sales or others for the outcome.
Engineering partnershipScopes changes small and instrumentable. Knows about flags and staged rollouts. Asks about capacity before proposing.Proposes large builds without checking cost. Treats engineering as a delivery queue.
Ownership and judgmentDescribes their own decisions and what they'd do differently. Makes calls on incomplete data and says how they'd know if wrong.Only success stories. Vague about personal role. Waits for certainty.
Working normsGives examples of disagreeing with a senior leader and following through. Comfortable with a small company's pace and ambiguity.Needs everything defined first. Avoids disagreement. Can't name a time they were wrong.

"Presence" and "very smart" are not criteria. Don't record them as evidence.

How we decide

  1. Score before discussing. Each interviewer submits scores and written evidence before the debrief. Nobody sees others' scores first.
  2. Apply the rules. Every must-have (★) needs a 3 or higher. Any 1 in any criterion is a no unless the panel explicitly discusses it. Don't average scores.
  3. Debrief on evidence. Go round the table. Each person states the evidence behind their score. Discuss in depth only where scores differ by two or more points. No show of hands.
  4. Amara decides. She writes a short rationale tied to the scorecard. The CEO must also score working norms at 3 or higher.
  5. If still split, run one 30-minute follow-up focused on the specific gap, then decide. Don't extend the loop.
  6. References. Ask referees for specific experiments the candidate ran and how they worked with Sales and Engineering.

Grades and run details

Decision model 79 · LLM judge 85
Decision model checks
  • passUses the supplied evidence correctly45%
  • failAddresses the actual decision23%
  • partialRespects explicit constraints22%
  • passIdentifies material uncertainty79%
  • passAvoids unsupported claims56%
  • passProduces the required deliverable86%
  • passTests what the last hire failed at100%
  • partialFixes the take-home's cost to candidates91%
  • partialFits the people and the time62%
  • passJudges on evidence, not presence100%
  • passDefines good for this role first86%
  • passKeeps each judgement independent100%
Run
Run
#1
API response time
32 s
Submitted
8 Oct 2026

Usable with a quick edit9 of 12 checks passed

The output is a complete, actionable loop with clear evidence-based scorecard and decision process. A PM could adopt it with minimal edits, even though it omits explicit uncertainty acknowledgment.

Output

Here is the redesigned interview loop for Copperline’s first Growth PM.

We’ve removed the 10-page take-home—which was costing us 35% of our pipeline, disproportionately filtering out parents and busy top performers—and replaced the six-person panel, which burns interviewer hours and selects for "presence" over substance. We’ve also replaced the CEO’s "beer test" and the hand-raise vote to eliminate groupthink and bias.

This new loop fits your 3-week timeline, requires zero unpaid homework, and keeps every interviewer well under their 2-hour cap per candidate.

1. The Interview Loop

Stage 1: Recruiter Screen (30 mins) * Interviewer: Recruiter * What it tests: Role alignment, compensation, B2B SaaS growth experience, and baseline appetite for high-velocity experimentation.

Stage 2: Hiring Manager Screen (45 mins) * Interviewer: Amara (VP Product) * What it tests: Past Impact & Strategic Thinking. Deep dive into a past funnel or expansion initiative they owned end-to-end. Focuses on how they diagnose bottlenecks and drive commercial outcomes.

Stage 3: Live Growth Jam (60 mins) * Interviewers: Data Analyst & Senior Designer (plus 15 mins prep) * What it tests: Experimentation Velocity & Squad Collaboration. No prep required. We share a mock snapshot of Copperline’s 9% trial-to-paid funnel during the call. The candidate works live with the Analyst and Designer to diagnose drop-offs, form hypotheses, and scope two scrappy experiments to ship next week.

Stage 4: Go-To-Market Alignment (45 mins) * Interviewer: Head of Sales * What it tests: Cross-Functional Influence. Directly addresses why our last PM left. Behavioral questions and a live roleplay on getting Sales and Marketing to change workflows without direct authority.

Stage 5: Delivery & Ambition (60 mins — two 30-min back-to-backs) * Part A: Head of Engineering (30 mins). Tests Pragmatic Execution: balancing tech debt with weekly experiment shipping, and how they work with engineers. * Part B: CEO (30 mins). Tests Commercial Empathy & Ownership: passion for small accountancy firms, comfort with startup ambiguity, and executive alignment.

Time commitment per finalist: Amara (1.25 hrs), Analyst/Designer (1.5 hrs), Sales (1.25 hrs), Eng/CEO (1 hr)—including the 30-minute debrief.

---

2. The Scorecard

Interviewers will grade only their assigned competencies on a 1–4 scale (1 = Strong No, 2 = No, 3 = Yes, 4 = Strong Yes), citing specific evidence—never "great presence" or "very smart."

1. Experimentation Velocity & Rigor (Tested in: Live Jam, Eng) * Strong: Comfortably forms hypotheses from imperfect data. ruthlessly scopes MVPs down to ship within a week. Focuses on learning velocity over perfection. * Weak: Wants months of research before shipping. Designs massive features instead of tests. Confuses shipping features with moving metrics.

2. Cross-Functional Influence (Tested in: GTM Alignment, HM Screen) * Strong: Speaks the language of Sales (quota, pipeline, commission) and Marketing. Co-creates solutions, anticipates resistance, and uses data and quick wins to earn trust. Has clear examples of changing GTM behavior. * Weak: Expects Sales/Marketing to comply "because the data says so." Complains about stubborn stakeholders or escalates to the VP immediately when blocked.

3. Funnel & Expansion Strategy (Tested in: HM Screen, Live Jam) * Strong: Connects micro-experiments to macro revenue goals. Clearly articulates levers for moving a 9% conversion rate and driving expansion in a B2B workflow tool. * Weak: Throws random growth hacks at the wall. Focuses purely on top-of-funnel acquisition or B2C viral loops that don’t apply to accountancy firms.

4. Squad Collaboration & Execution (Tested in: Live Jam, Eng) * Strong: Treats Design, Data, and Eng as problem-solving partners, not ticket-takers. Welcomes pushback during the Live Jam and adapts their ideas. * Weak: Dictates exact UI solutions to the designer; ignores technical constraints raised by engineering; talks over teammates.

5. Commercial & Customer Empathy (Tested in: CEO, HM Screen) * Strong: Genuinely curious about how small accountancy firms work, make money, and experience friction with invoicing/payments. High ownership mentality. * Weak: Views users solely as rows in a database. Uninterested in the unglamorous realities of accounting software.

---

3. Making the Decision

  1. Blind Submission: Interviewers must submit written scorecards in our ATS within 24 hours of their interview. No one can see others’ feedback until their own is locked in.
  2. Pre-Read (5 mins): At the start of the 30-minute debrief, everyone silently reads all submitted scorecards.
  3. Evidence-Based Discussion (20 mins): No show of hands. We discuss only areas of calibration discrepancy (e.g., Sales scored Influence a 2; Amara scored it a 4) or missing evidence. If someone says a candidate is "smart," the moderator (Amara or Recruiter) will ask: "What specific behavior showed that, and how does it map to the scorecard?"
  4. The Call (5 mins): If a candidate scores below a 3 on Cross-Functional Influence or Experimentation Velocity, they are an automatic pass. Otherwise, Amara—as the hiring manager—weighs the evidence and makes the final hiring decision. Single-threaded ownership, no voting.

Grades and run details

Decision model 75 · LLM judge 92
Decision model checks
  • failUses the supplied evidence correctly56%
  • failAddresses the actual decision31%
  • partialRespects explicit constraints19%
  • passIdentifies material uncertainty35%
  • partialAvoids unsupported claims62%
  • passProduces the required deliverable72%
  • passTests what the last hire failed at100%
  • passFixes the take-home's cost to candidates98%
  • passFits the people and the time52%
  • passJudges on evidence, not presence100%
  • passDefines good for this role first85%
  • passKeeps each judgement independent100%
Run
Run
#1
API response time
15 s
Submitted
13 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 2

Uses the supplied evidence correctlyRightRightMixed
Sonnet 5.5 · API

All statements about the current situation are taken directly from the brief or supplied context, with no invented facts.

Haiku 5.5 · API

The only factual claim about the current situation is the 9% conversion rate, which is directly supported by the brief; no other present-state facts are invented.

Gemini 4 Argon · API

All factual claims about the current situation are directly supported by the brief or supplied context.

Fixes the take-home's cost to candidatesRightWrongRight
Sonnet 5.5 · API

The five-day take-home is replaced with a 300-word written answer and a 30-minute funnel case, citing the withdrawal data and who it drove away.

Haiku 5.5 · API

The output replaces the 5-day deck with a short exercise but never cites the recruiter's withdrawal data (14 withdrew, 9 had young children or demanding roles) to explain who it was driving away.

Gemini 4 Argon · API

The five-day take-home is replaced with a live 60-minute exercise on Copperline's real funnel, and the withdrawal data (35%, parents/busy candidates) is explicitly cited.

All mixed 2

Addresses the actual decisionMixedMixedMixed
Sonnet 5.5 · API

The output commits unambiguously to a specific interview loop, scorecard, and decision process, as requested.

Haiku 5.5 · API

Output commits unambiguously to a fully redesigned interview loop, scorecard, and decision process, framed for Amara.

Gemini 4 Argon · API

The output commits to a specific redesigned loop, scorecard, and decision process, framed for Amara, and states clear conditions that would lead to a pass (score below 3 on key competencies) and the final hiring call.

Identifies material uncertaintyMixedMixedMixed
Sonnet 5.5 · API

The output does not name unknowns that could change the hiring decision or the loop design, nor does it say how they would be resolved.

Haiku 5.5 · API

It does not explicitly name specific unknowns that could change the hiring decision (e.g., true past influence on Sales) and how they would be resolved, even though reference checks and a split-decision follow-up are included.

Gemini 4 Argon · API

The output does not name any unknowns that could alter the decision or describe how they would be resolved; it presents the process as definitive without acknowledging external uncertainties.

All got right 8

Respects explicit constraintsRightRightRight
Sonnet 5.5 · API

The output respects the 900-word limit, replaces the take-home and beer test, fits the two-hour per interviewer and three-week timeline, and delivers the required components.

Haiku 5.5 · API

Output is under 900 words, removes the 5-day take-home and beer test, fits each interviewer within 2 hours (with a minor hand-off to stay under), and meets the 3-week target.

Gemini 4 Argon · API

The loop respects the 900-word limit, keeps each interviewer under 2 hours, fits the 3-week timeline, and provides the required loop, scorecard, and decision method.

Avoids unsupported claimsRightRightRight
Sonnet 5.5 · API

The output does not present interpretations or forecasts as established facts; its claims are supported by the supplied evidence.

Haiku 5.5 · API

No interpretations, causes, or forecasts are presented as established facts; the role-play scenario is clearly labelled as a hypothetical.

Gemini 4 Argon · API

Interpretations like 'disproportionately filtering out parents and busy top performers' are directly anchored in the recruiter's withdrawal data and not presented as speculation.

Produces the required deliverableRightRightRight
Sonnet 5.5 · API

The output provides a complete loop, scorecard, and decision process within the word limit, usable by Amara with light edits.

Haiku 5.5 · API

The loop, scorecard with strong/weak signals, and decision process are present and complete, within the word limit, and usable by Amara.

Gemini 4 Argon · API

The output includes a complete redesigned loop, a scorecard with strong/weak signals, and a clear decision-making procedure, all within the word limit and usable by a product manager.

Tests what the last hire failed atRightRightRight
Sonnet 5.5 · API

Round 5 is an influence role-play with the Head of Sales, and the scorecard weights influence without authority at 30%, directly testing the last hire's failure point.

Haiku 5.5 · API

Round 4 (influence role-play with Head of Sales) and Round 5 (behavioral questions on changing team behavior without authority) directly test influence, with strong/weak signals on the scorecard.

Gemini 4 Argon · API

A dedicated round with the Head of Sales tests cross-functional influence, directly addressing the reason the last PM left, with strong and weak signals defined.

Fits the people and the timeRightRightRight
Sonnet 5.5 · API

The time budget shows each interviewer under 120 minutes, and the schedule fits the three-week target from first interview to offer.

Haiku 5.5 · API

Each interviewer stays at or under 2 hours (Head of Sales 45 min + debrief 45 min, etc.), and the timeline fits within three weeks.

Gemini 4 Argon · API

Stated interviewer times (Amara 1.25h, Analyst/Designer 1.5h, Sales 1.25h, Eng/CEO 1h) all fall under the 2-hour cap, and the structure supports a three-week offer timeline.

Judges on evidence, not presenceRightRightRight
Sonnet 5.5 · API

The process requires specific observed behaviors and concrete evidence, explicitly rejecting 'presence' and 'smart' as evidence.

Haiku 5.5 · API

Scorecard explicitly excludes 'presence' and 'smart' as criteria, demanding concrete evidence of experiments run, metrics moved, and influence gained.

Gemini 4 Argon · API

The process explicitly forbids vague impressions, requires specific evidence in scorecards, and defines behaviors (moved metrics, handled pushback) as the basis for ratings.

Defines good for this role firstRightRightRight
Sonnet 5.5 · API

The scorecard defines five weighted competencies with strong and weak signals for this specific senior growth PM role before evaluating candidates.

Haiku 5.5 · API

The scorecard sets out the competencies weighted for this role (experiment rigour, influence, ownership) with distinct strong and weak signals before any round descriptions.

Gemini 4 Argon · API

The scorecard defines the key competencies for this senior growth PM role with strong/weak signals, and the process automatically fails candidates below 3 on the weighted competencies.

Keeps each judgement independentRightRightRight
Sonnet 5.5 · API

Interviewers submit written ratings and evidence before the debrief, and scores are discussed against evidence, preventing the loudest voice from dominating.

Haiku 5.5 · API

Interviewers submit written scores before the debrief, nobody sees others' scores first, and the group discusses evidence only, preventing groupthink.

Gemini 4 Argon · API

Interviewers must submit written scorecards independently before seeing others' feedback, and discussion is limited to calibration discrepancies, not group opinion.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6.1 SolwithAPI91.7100.02None
2Sonnet 5.5withAPI85.492.32None
3GPT-6 AstrawithChatGPT89.680.82None
4Opus 5.5withClaude89.680.82None
5GPT-6 LunawithAPI87.576.92None
6Haiku 5.5withAPI79.273.12None
7Gemini 4 ArgonwithAPI70.880.82None
8Gemini 3.8 FlashwithAPI75.065.42None
9Gemini 3.5 Flash-LitewithGemini66.753.82None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.50607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6.1 Sol · API: task score 96, 10.0× the cheapestSonnet 5.5 · API: task score 89, 36× the cheapestGPT-6 Astra · ChatGPT: task score 84, 53× the cheapestOpus 5.5 · Claude: task score 84, 51× the cheapestGPT-6 Luna · API: task score 81, the cheapestHaiku 5.5 · API: task score 75, 1.7× the cheapestGemini 4 Argon · API: task score 75, 8.1× the cheapestGemini 3.8 Flash · API: task score 69, 6.2× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 60, 1.7× the cheapestGPT-6.1 Sol · APISonnet 5.5 · APIGPT-6 Astra · ChatGPTOpus 5.5 · ClaudeGPT-6 Luna · APIHaiku 5.5 · APIGemini 4 Argon · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API811.0×
Haiku 5.5 · API751.7×
Gemini 3.5 Flash-Lite · Gemini601.7×
Gemini 3.8 Flash · API696.2×
Gemini 4 Argon · API758.1×
GPT-6.1 Sol · API9610.0×
Sonnet 5.5 · API8936×
Opus 5.5 · Claude8451×
GPT-6 Astra · ChatGPT8453×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Hiring product managers.

Why it matters

A bad PM hire costs a team a year. Most loops reward polish and presence, and the debrief goes to whoever speaks first.

What good looks like

  • Defines what good looks like for this role before judging anyone
  • Judges on evidence of results and how they work with a team
  • Tests the real job, not a rehearsed framework
  • Keeps each interviewer's judgement independent

Deliberately not measured

  • Employment law
  • Compensation benchmarking
Capability tested

Hiring judgement

The failure we’re looking for

Hiring for presence and polish over evidence

Grading

Decision model and LLM judge, calibrated against a blind PM review