Humanity’s Last PRD
Latest drop: Gemini 4 Argon · 13 Oct
Tasks / Leadership

Hire a PM

Can the model design a hiring process that finds the right PM, and make the call on real candidates from the evidence?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 61% were usable with at most a quick edit.

Reliably right

  1. Produces the required deliverable100% pass
    The output provides a complete loop, scorecard, and decision plan in the requested form, under 900 words, usable by Amara with light edits.
    GPT-6.1 Sol · API · A loop for the first growth PM
  2. Tests what the last hire failed at100% pass
    The loop includes an influence simulation with the Head of Sales and a scorecard dimension on influence without authority, directly testing what the last hire failed at.
    GPT-6.1 Sol · API · A loop for the first growth PM
  3. Finds which rounds line up with performance100% pass
    It compares each round's scores with 12-month ratings, finding behavioral lines up and take-home, presentation, VP Engineering do not.
    GPT-6.1 Sol · API · What our interviews predict

Where it slips

  1. Uses the supplied evidence correctly56% pass
    The take-home correlation is stated as -0.57, but the data yields approximately -0.33; this fact about the current situation is incorrect.
    GPT-6 Astra · ChatGPT · What our interviews predict
  2. Avoids unsupported claims60% pass
    The claim 'A growth PM in a senior role is likely to be exactly that kind of person' is presented as fact without evidence and is not labelled as an assumption.
    Opus 5.5 · Claude · A loop for the first growth PM
  3. Defines good for this role first63% pass
    The memo does not explicitly name the competencies that matter most for this PM role and level, nor define what strong and weak look like before designing the interviews.
    Sonnet 5.5 · API · What our interviews predict

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Judges on evidence, not presence

    Does the output judge candidates, or design the process to judge them, on specific evidence of results they drove and how they worked with their team, rather than impressions such as presence, confidence or polish?

    Passes when Asks for or weighs concrete evidence: the metric a candidate moved, the part they played, how they shared credit, what they learned from a failure. Impressions are treated as weak signals.

    Product and growth leadersOn Lenny’s Podcast

  2. Defines good for this role first

    Does the output set out what good looks like for this specific role (the competencies that matter most at this level, and what strong and weak look like) before judging candidates or designing interviews?

    Passes when Names the competencies weighted for this role and level, with strong and weak signals, and the judgement or the process follows from them.

    Petra WilleLenny’s Podcast

  3. Keeps each judgement independent

    Does the output keep each interviewer's judgement independent (written down before the group discusses it), and guard against the loudest voice or the first speaker deciding?

    Passes when Requires or relies on written, independent evaluations before discussion, and discounts views that changed under group pressure.

    Product and growth leadersOn Lenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 3 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're helping Amara Osei, VP Product at Copperline, hire our first Growth PM (a senior role). She's shared her draft interview loop and asked you to redesign it. Write the loop you'd run (each round: who runs it, what it tests, how long it takes), the scorecard (what strong and weak look like for each thing we're testing), and how we'll make the decision at the end. No more than 900 words. What we know is below.

What the model was given6 items: About Copperline, The role, Amara's draft loop, The last two PM hires, From the recruiter, Who can interview
About CopperlineInvoicing and payments software for small accountancy firms. 60 people. Trial to paid conversion is 9%.
The roleOwns trial conversion and expansion revenue. Works with three engineers and a designer, reports to Amara, and has to ship experiments every week. Needs Sales and Marketing to change how they work, without managing them.
Amara's draft loop1. Recruiter screen. 2. Take-home: 'design a growth strategy for Copperline', a 10-page deck, five days to complete. 3. Presentation to a panel of six. 4. Culture fit with the CEO: 'would I grab a beer with them?' 5. Debrief: everyone discusses, then votes by show of hands. There's no scorecard.
The last two PM hiresOne left after five months: 'couldn't get Sales to change anything'. The other is doing well. Both sets of interview notes say mainly 'great presence' and 'very smart'.
From the recruiterOf the last 40 candidates given the take-home, 14 withdrew, saying they didn't have time. Nine of the 14 had young children or demanding current roles.
Who can interviewAmara, the Head of Engineering, a senior designer, the Head of Sales, a data analyst and the CEO. Each can give at most two hours to this hire. Amara wants an offer out within three weeks of the first interview.
What a strong answer doesThe answer key the graders mark against

A loop built on a scorecard written first: for a senior growth PM, experimentation and analytics, execution, and influence without authority (weighted up, since the last hire failed on it), with strong and weak signals for each. Rounds test the real job: a live diagnosis of Copperline's own trial funnel instead of a five-day generic deck (which loses candidates, mostly parents and people in demanding roles), a behavioural round on past experiments and metrics actually moved, and an influence round run by the Head of Sales. Replaces the 'grab a beer' culture fit with a structured values or collaboration conversation. Each interviewer writes their evaluation before the debrief, and the decision is made against the scorecard, not a show of hands. Fits each interviewer's two hours and the three-week target, and adds reference checks.

Critical failures (cap the score at 40)
  • Keeps a culture-fit test based on whether the interviewer would like a beer with the candidate
Task

Mid-level · v1.1

The outputs

9 setups answered this brief.

GPT-6.1 SolAPI98Usable with a quick editOpus 5.5Claude94Usable with a quick editGPT-6 AstraChatGPT92Usable with a quick editSonnet 5.5API90Usable with a quick editGPT-6 LunaAPI87Usable with a quick editGemini 3.8 FlashAPI87Usable with a quick editGemini 4 ArgonAPI85Usable with a quick editHaiku 5.5API83Usable with a quick editGemini 3.5 Flash-LiteGemini81Usable with a quick edit
Gemini 4 ArgonAPI

Usable with a quick edit9 of 12 checks passed

The output is a complete, actionable loop with clear evidence-based scorecard and decision process. A PM could adopt it with minimal edits, even though it omits explicit uncertainty acknowledgment.

Output

Here is the redesigned interview loop for Copperline’s first Growth PM.

We’ve removed the 10-page take-home—which was costing us 35% of our pipeline, disproportionately filtering out parents and busy top performers—and replaced the six-person panel, which burns interviewer hours and selects for "presence" over substance. We’ve also replaced the CEO’s "beer test" and the hand-raise vote to eliminate groupthink and bias.

This new loop fits your 3-week timeline, requires zero unpaid homework, and keeps every interviewer well under their 2-hour cap per candidate.

1. The Interview Loop

Stage 1: Recruiter Screen (30 mins) * Interviewer: Recruiter * What it tests: Role alignment, compensation, B2B SaaS growth experience, and baseline appetite for high-velocity experimentation.

Stage 2: Hiring Manager Screen (45 mins) * Interviewer: Amara (VP Product) * What it tests: Past Impact & Strategic Thinking. Deep dive into a past funnel or expansion initiative they owned end-to-end. Focuses on how they diagnose bottlenecks and drive commercial outcomes.

Stage 3: Live Growth Jam (60 mins) * Interviewers: Data Analyst & Senior Designer (plus 15 mins prep) * What it tests: Experimentation Velocity & Squad Collaboration. No prep required. We share a mock snapshot of Copperline’s 9% trial-to-paid funnel during the call. The candidate works live with the Analyst and Designer to diagnose drop-offs, form hypotheses, and scope two scrappy experiments to ship next week.

Stage 4: Go-To-Market Alignment (45 mins) * Interviewer: Head of Sales * What it tests: Cross-Functional Influence. Directly addresses why our last PM left. Behavioral questions and a live roleplay on getting Sales and Marketing to change workflows without direct authority.

Stage 5: Delivery & Ambition (60 mins — two 30-min back-to-backs) * Part A: Head of Engineering (30 mins). Tests Pragmatic Execution: balancing tech debt with weekly experiment shipping, and how they work with engineers. * Part B: CEO (30 mins). Tests Commercial Empathy & Ownership: passion for small accountancy firms, comfort with startup ambiguity, and executive alignment.

Time commitment per finalist: Amara (1.25 hrs), Analyst/Designer (1.5 hrs), Sales (1.25 hrs), Eng/CEO (1 hr)—including the 30-minute debrief.

---

2. The Scorecard

Interviewers will grade only their assigned competencies on a 1–4 scale (1 = Strong No, 2 = No, 3 = Yes, 4 = Strong Yes), citing specific evidence—never "great presence" or "very smart."

1. Experimentation Velocity & Rigor (Tested in: Live Jam, Eng) * Strong: Comfortably forms hypotheses from imperfect data. ruthlessly scopes MVPs down to ship within a week. Focuses on learning velocity over perfection. * Weak: Wants months of research before shipping. Designs massive features instead of tests. Confuses shipping features with moving metrics.

2. Cross-Functional Influence (Tested in: GTM Alignment, HM Screen) * Strong: Speaks the language of Sales (quota, pipeline, commission) and Marketing. Co-creates solutions, anticipates resistance, and uses data and quick wins to earn trust. Has clear examples of changing GTM behavior. * Weak: Expects Sales/Marketing to comply "because the data says so." Complains about stubborn stakeholders or escalates to the VP immediately when blocked.

3. Funnel & Expansion Strategy (Tested in: HM Screen, Live Jam) * Strong: Connects micro-experiments to macro revenue goals. Clearly articulates levers for moving a 9% conversion rate and driving expansion in a B2B workflow tool. * Weak: Throws random growth hacks at the wall. Focuses purely on top-of-funnel acquisition or B2C viral loops that don’t apply to accountancy firms.

4. Squad Collaboration & Execution (Tested in: Live Jam, Eng) * Strong: Treats Design, Data, and Eng as problem-solving partners, not ticket-takers. Welcomes pushback during the Live Jam and adapts their ideas. * Weak: Dictates exact UI solutions to the designer; ignores technical constraints raised by engineering; talks over teammates.

5. Commercial & Customer Empathy (Tested in: CEO, HM Screen) * Strong: Genuinely curious about how small accountancy firms work, make money, and experience friction with invoicing/payments. High ownership mentality. * Weak: Views users solely as rows in a database. Uninterested in the unglamorous realities of accounting software.

---

3. Making the Decision

  1. Blind Submission: Interviewers must submit written scorecards in our ATS within 24 hours of their interview. No one can see others’ feedback until their own is locked in.
  2. Pre-Read (5 mins): At the start of the 30-minute debrief, everyone silently reads all submitted scorecards.
  3. Evidence-Based Discussion (20 mins): No show of hands. We discuss only areas of calibration discrepancy (e.g., Sales scored Influence a 2; Amara scored it a 4) or missing evidence. If someone says a candidate is "smart," the moderator (Amara or Recruiter) will ask: "What specific behavior showed that, and how does it map to the scorecard?"
  4. The Call (5 mins): If a candidate scores below a 3 on Cross-Functional Influence or Experimentation Velocity, they are an automatic pass. Otherwise, Amara—as the hiring manager—weighs the evidence and makes the final hiring decision. Single-threaded ownership, no voting.

Grades and run details

Decision model 75 · LLM judge 92
Decision model checks
  • failUses the supplied evidence correctly56%
  • failAddresses the actual decision31%
  • partialRespects explicit constraints19%
  • passIdentifies material uncertainty35%
  • partialAvoids unsupported claims62%
  • passProduces the required deliverable72%
  • passTests what the last hire failed at100%
  • passFixes the take-home's cost to candidates98%
  • passFits the people and the time52%
  • passJudges on evidence, not presence100%
  • passDefines good for this role first85%
  • passKeeps each judgement independent100%
Run
Run
#1
API response time
15 s
Submitted
13 Oct 2026
GPT-6 LunaAPI

Usable with a quick edit9 of 12 checks passed

The output is complete and actionable; a quick edit to mention the recruiter's withdrawal data would make it fully aligned with the brief.

Output

Interview loop

Run this over 2–3 weeks. Skip the five-day take-home, large presentation panel, and “beer” fit interview. Share the same expectations and questions with every candidate; offer scheduling flexibility and accommodations.

1. Recruiter screen — 30 minutes Tests role understanding, motivation, and practical alignment: the scope, weekly shipping pace, and need to influence Sales and Marketing without authority. Not a “polish” screen.

2. Amara, VP Product — 60 minutes Tests growth ownership and product judgment. Ask for a detailed example of a funnel or revenue problem the candidate owned: diagnosis, choices, experiments, results, and what they learned. Probe how they balance conversion, customer trust, and expansion.

3. Live growth work sample — 60 minutes Send a one-page brief and synthetic funnel data 24 hours ahead; no deck or advance work required. Spend 30 minutes with the data analyst diagnosing the funnel and proposing a measurable experiment; spend 30 minutes with the Head of Engineering and senior designer scoping how to ship and learn from it. Use the same prompt and time limits for all candidates.

4. Head of Sales — 45 minutes Structured roleplay: Sales is reluctant to change its trial follow-up because reps are busy and doubt the data. Ask the candidate to understand the objection, make a case, and agree a practical next step. Then probe a real example of influencing a team they did not manage.

The CEO does not need a separate round: “culture fit” is not a hiring criterion, and the role’s key cross-functional test is better assessed directly with Sales, Engineering, and Design.

Scorecard

Use a 1–4 rating for each dimension: 1 = clear concern, 2 = below bar, 3 = meets bar, 4 = unusually strong. Record evidence, not impressions.

DimensionStrong evidenceWeak evidence
Growth ownership and outcomes (Amara)Personally owned a meaningful funnel or revenue outcome; explains decisions, results, attribution limits, and lessons.Describes activity rather than impact; overclaims credit or cannot explain what changed.
Customer and product judgment (Amara)Understands small-firm workflows and trust needs; weighs conversion against customer value and expansion.Offers generic SaaS tactics or pushes short-term conversion at the expense of trust.
Data diagnosis and experiment design (work sample)Finds plausible bottlenecks and segments; proposes a focused test with a clear hypothesis, success metric, guardrail, and learning plan.Jumps to a solution, misreads the data, uses vanity metrics, or cannot say what result would change their mind.
Shipping and learning cadence (work sample)Scopes a feasible experiment, identifies dependencies and tradeoffs, and shows how the team can learn and iterate weekly.Proposes a big-bang project, ignores constraints, or treats learning as something to do after launch.
Influence without authority (Sales)Listens for incentives and concerns, uses evidence appropriately, adapts the ask, and secures a credible next step.Blames Sales, relies on escalation or authority, or cannot turn disagreement into action.
Cross-functional partnership (work sample)Makes clear tradeoffs with Engineering and Design, invites their expertise, and aligns on ownership and delivery.Treats partners as order-takers or leaves scope and responsibilities vague.

Decision

Interviewers submit ratings and specific evidence independently before a 30-minute debrief; do not vote by show of hands. Amara facilitates, checks for evidence behind claims, and makes the hiring decision. Do not use “presence,” “smart,” or general “fit” as evidence.

A hire must score at least 3 on influence, experiment design, and shipping, with no score below 2 in any dimension. Do not average away a weakness in a core requirement. If evidence is genuinely missing, identify one focused follow-up rather than adding a broad round. Give candidates a timely decision and close the loop within three weeks of the first interview.

Grades and run details

Decision model 96 · LLM judge 77
Decision model checks
  • passUses the supplied evidence correctly61%
  • passAddresses the actual decision28%
  • passRespects explicit constraints40%
  • passIdentifies material uncertainty73%
  • passAvoids unsupported claims41%
  • passProduces the required deliverable88%
  • passTests what the last hire failed at100%
  • partialFixes the take-home's cost to candidates94%
  • passFits the people and the time64%
  • passJudges on evidence, not presence100%
  • passDefines good for this role first86%
  • passKeeps each judgement independent97%
Run
Run
#1
API response time
39 s
Submitted
5 Oct 2026
GPT-6.1 SolAPI

Usable with a quick edit12 of 12 checks passed

The output is complete, actionable, and respects all constraints; a PM could use it with at most minor wording tweaks.

Output

The loop I’d run

Replace the deck and six-person presentation with bounded, live work samples. Remove the “beer test.” Copperline needs someone who can diagnose conversion, ship weekly, and change other teams’ behavior—not simply present well.

Candidate commitment: 3 hours 10 minutes; no take-home or required preparation. Share the format and scorecard beforehand, offer flexible scheduling and reasonable accommodations, and use the same prompts and information for everyone.

RoundInterviewersTimeWhat it tests
---------:---
1. Recruiter screenRecruiter20 minScope, compensation, availability, and interest in hands-on growth ownership. Explain the process. Don’t assess polish or pedigree.
2. Ownership interviewAmara35 minSenior-level judgment and accountability. Examine one past growth initiative: candidate’s actual contribution, choices, results, failures, and learning. Probe both conversion and expansion experience. Reserve five minutes for candidate questions.
3. Live growth diagnosisData analyst50 minAnalytical judgment and experiment design. Give a small synthetic Copperline dataset: trial funnel, cohorts, acquisition sources, activation behavior, expansion signals, and metric definitions. Ask what could explain 9% conversion, what they would investigate, and which experiment they would run first. Include a plausible expansion opportunity.
4. Shipping workshopHead of Engineering + senior designer40 minTurn the previous round’s proposal into a one-week experiment with three engineers and a designer. Interviewers act as working partners, not an audience. Introduce a tracking gap and an engineering constraint. Test scope, customer experience, instrumentation, trade-offs, and rollout safety.
5. Influence simulationHead of Sales45 minFirst, role-play a conversation with Sales resisting a proposed trial-follow-up change because of workload and incentives. Add a Marketing objection about lead quality. Then examine a real example of influencing peers without authority, including resistance, compromises, and outcomes. Reserve five minutes for questions.

Use the same dataset, constraints, and role-play objections across candidates. Score reasoning, not whether they discover a predetermined “correct” growth strategy. The CEO does not conduct a separate fit interview.

Scorecard

Use 1 = clear concern, 2 = below bar, 3 = meets senior-role bar, 4 = exceptional, plus not observed. Each rating requires concrete evidence: a quote, action, artifact, or specific past example. “Very smart” and “great presence” are not evidence.

Dimension / primary assessorStrong: meets or exceeds barWeak: below bar
Growth and analytical judgment — analystChecks denominators, cohort maturity, segmentation, and data quality before explaining 9%. Separates correlation from causation; prioritizes plausible conversion and expansion opportunities using customer behavior and economics.Treats aggregate conversion as a diagnosis; jumps to tactics; ignores acquisition mix or revenue quality; makes causal claims without evidence.
Experimentation — analystStates hypothesis, mechanism, target population, primary metric, guardrails, and decision rule. Recognizes low-volume limitations. Distinguishes shipping weekly from obtaining statistically conclusive results weekly.Proposes many tests without prioritization; relies on vanity metrics; declares victory prematurely; cannot say what result would change the decision.
Execution and customer judgment — Engineering + designerCuts scope to a credible weekly release; specifies measurement and rollback; collaborates with technical/design partners; protects trust in invoicing and payments workflows.Requires a large rebuild; treats partners as order-takers; neglects instrumentation, reliability, or customer harm; cannot adapt to constraints.
Influence without authority — SalesUnderstands incentives and workload; listens, negotiates a bounded pilot, establishes shared measures and explicit commitments, and follows through. Past examples show actual behavior change.Relies on escalation, persuasion theater, or “getting buy-in”; blames peers; has no concrete mechanism for changing behavior.
Senior ownership and learning — AmaraClearly distinguishes personal contribution from team results; connects conversion and expansion to durable revenue; owns misses and demonstrates changed judgment.Claims collective wins as personal achievements; avoids failures; optimizes local conversion at the expense of retention or revenue; offers lessons without changed behavior.

Decision and operating plan

Before interviewing: Run a 15-minute calibration using sample answers and agree on anchors. Give interviewers structured prompts.

After each round: Interviewers submit evidence and scores independently within 24 hours, before seeing others’ assessments. Engineering and Design score independently.

Debrief: Hold a 30-minute evidence review—not a show-of-hands vote. Discuss disagreements and counterevidence against the written bar. Amara makes and documents the hiring decision.

Offer bar: At least 3 on every dimension. In particular, do not average away weak influence, analytical judgment, or execution. Missing evidence triggers one narrowly targeted follow-up, not intuition. Hire against the bar, not merely the strongest available candidate.

Timing and capacity: Finish interviews by day 12, debrief by day 14, and issue the offer by day 21. Calibration, interview, scoring, and debrief consume at most 105 minutes per domain interviewer. Amara retains time within her two-hour cap for decision documentation and an offer conversation. CEO approval, if required, concerns headcount and compensation—not an unstructured personality veto.

Grades and run details

Decision model 96 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly41%
  • passAddresses the actual decision41%
  • passRespects explicit constraints27%
  • passIdentifies material uncertainty77%
  • passAvoids unsupported claims48%
  • passProduces the required deliverable94%
  • passTests what the last hire failed at100%
  • partialFixes the take-home's cost to candidates83%
  • passFits the people and the time83%
  • passJudges on evidence, not presence100%
  • passDefines good for this role first91%
  • passKeeps each judgement independent100%
Run
Run
#1
API response time
55 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Uses the supplied evidence correctlyMixedRightRight
Gemini 4 Argon · API

All factual claims about the current situation are directly supported by the brief or supplied context.

GPT-6 Luna · API

The output makes no factual claims about the current situation, so it does not misuse any supplied evidence.

GPT-6.1 Sol · API

All statements about the current situation are taken directly from the brief or supplied context, with no invented facts.

Addresses the actual decisionMixedMixedRight
Gemini 4 Argon · API

The output commits to a specific redesigned loop, scorecard, and decision process, framed for Amara, and states clear conditions that would lead to a pass (score below 3 on key competencies) and the final hiring call.

GPT-6 Luna · API

The output does not state what result or condition would change the proposed loop or scorecard.

GPT-6.1 Sol · API

The output commits to a specific redesigned loop, scorecard, and decision process, and states the offer bar and how missing evidence is handled.

Identifies material uncertaintyMixedMixedRight
Gemini 4 Argon · API

The output does not name any unknowns that could alter the decision or describe how they would be resolved; it presents the process as definitive without acknowledging external uncertainties.

GPT-6 Luna · API

The output does not name any unknowns that could change the decision or how they would be resolved.

GPT-6.1 Sol · API

The output defines a clear decision rule (bar of 3 on every dimension, no averaging away weak areas) and implies not hiring if no one meets it, which addresses the key uncertainty in the hiring decision.

Fixes the take-home's cost to candidatesRightWrongRight
Gemini 4 Argon · API

The five-day take-home is replaced with a live 60-minute exercise on Copperline's real funnel, and the withdrawal data (35%, parents/busy candidates) is explicitly cited.

GPT-6 Luna · API

The output replaces the take-home but does not cite the recruiter's withdrawal data or mention who it was driving away.

GPT-6.1 Sol · API

The five-day take-home is replaced with live, bounded exercises, and the candidate commitment is reduced to 3 hours 10 minutes with no take-home, addressing the withdrawal data.

All got right 8

Respects explicit constraintsRightRightRight
Gemini 4 Argon · API

The loop respects the 900-word limit, keeps each interviewer under 2 hours, fits the 3-week timeline, and provides the required loop, scorecard, and decision method.

GPT-6 Luna · API

The output respects all constraints: it replaces the take-home, removes the beer test, fits interviewer time limits and the three-week schedule, and stays under 900 words.

GPT-6.1 Sol · API

The output is under 900 words, addresses Amara, replaces the take-home and beer test, and respects all stated constraints.

Avoids unsupported claimsRightRightRight
Gemini 4 Argon · API

Interpretations like 'disproportionately filtering out parents and busy top performers' are directly anchored in the recruiter's withdrawal data and not presented as speculation.

GPT-6 Luna · API

The output presents only proposals and avoids unsupported claims about causes or forecasts.

GPT-6.1 Sol · API

No interpretations or forecasts are presented as established facts; the need for diagnosis, shipping, and influence is directly derived from the role description.

Produces the required deliverableRightRightRight
Gemini 4 Argon · API

The output includes a complete redesigned loop, a scorecard with strong/weak signals, and a clear decision-making procedure, all within the word limit and usable by a product manager.

GPT-6 Luna · API

The output provides a complete interview loop, scorecard, and decision process that a product manager could act on.

GPT-6.1 Sol · API

The output provides a complete loop, scorecard, and decision plan in the requested form, under 900 words, usable by Amara with light edits.

Tests what the last hire failed atRightRightRight
Gemini 4 Argon · API

A dedicated round with the Head of Sales tests cross-functional influence, directly addressing the reason the last PM left, with strong and weak signals defined.

GPT-6 Luna · API

The loop includes a dedicated round with the Head of Sales and a scorecard dimension for influence without authority, directly testing what the last hire failed at.

GPT-6.1 Sol · API

The loop includes an influence simulation with the Head of Sales and a scorecard dimension on influence without authority, directly testing what the last hire failed at.

Fits the people and the timeRightRightRight
Gemini 4 Argon · API

Stated interviewer times (Amara 1.25h, Analyst/Designer 1.5h, Sales 1.25h, Eng/CEO 1h) all fall under the 2-hour cap, and the structure supports a three-week offer timeline.

GPT-6 Luna · API

Each interviewer's time is within two hours, and the loop is designed to fit the three-week target.

GPT-6.1 Sol · API

Each interviewer's time (including calibration, interview, and debrief) stays within 105 minutes, under the two-hour cap, and the schedule fits the three-week target.

Judges on evidence, not presenceRightRightRight
Gemini 4 Argon · API

The process explicitly forbids vague impressions, requires specific evidence in scorecards, and defines behaviors (moved metrics, handled pushback) as the basis for ratings.

GPT-6 Luna · API

The scorecard and decision rules explicitly require evidence over impressions and forbid using 'presence' or 'smart' as criteria.

GPT-6.1 Sol · API

The process requires concrete evidence (quotes, actions, artifacts) and explicitly rejects 'very smart' and 'great presence' as evidence.

Defines good for this role firstRightRightRight
Gemini 4 Argon · API

The scorecard defines the key competencies for this senior growth PM role with strong/weak signals, and the process automatically fails candidates below 3 on the weighted competencies.

GPT-6 Luna · API

The scorecard defines competencies with strong and weak signals, and the hiring bar weights influence, experiment design, and shipping.

GPT-6.1 Sol · API

The scorecard defines five weighted competencies with strong and weak signals, and the decision rule prioritizes influence, analytical judgment, and execution.

Keeps each judgement independentRightRightRight
Gemini 4 Argon · API

Interviewers must submit written scorecards independently before seeing others' feedback, and discussion is limited to calibration discrepancies, not group opinion.

GPT-6 Luna · API

Interviewers must submit ratings and evidence independently before the debrief, and the process guards against groupthink.

GPT-6.1 Sol · API

Interviewers submit evidence and scores independently before discussion, and the debrief is an evidence review, not a show of hands.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6.1 SolAPI91.7100.02None
2Sonnet 5.5API85.492.32None
3GPT-6 AstraChatGPT89.680.82None
4Opus 5.5Claude89.680.82None
5GPT-6 LunaAPI87.576.92None
6Haiku 5.5API79.273.12None
7Gemini 4 ArgonAPI70.880.82None
8Gemini 3.8 FlashAPI75.065.42None
9Gemini 3.5 Flash-LiteGemini66.753.82None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.50607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6.1 Sol · API: task score 96, 10.0× the cheapestSonnet 5.5 · API: task score 89, 36× the cheapestGPT-6 Astra · ChatGPT: task score 84, 53× the cheapestOpus 5.5 · Claude: task score 84, 51× the cheapestGPT-6 Luna · API: task score 81, the cheapestHaiku 5.5 · API: task score 75, 1.7× the cheapestGemini 4 Argon · API: task score 75, 8.1× the cheapestGemini 3.8 Flash · API: task score 69, 6.2× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 60, 1.7× the cheapestGPT-6.1 Sol · APISonnet 5.5 · APIGPT-6 Astra · ChatGPTOpus 5.5 · ClaudeGPT-6 Luna · APIHaiku 5.5 · APIGemini 4 Argon · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and cost

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Hiring product managers.

Why it matters

A bad PM hire costs a team a year. Most loops reward polish and presence, and the debrief goes to whoever speaks first.

What good looks like

  • Defines what good looks like for this role before judging anyone
  • Judges on evidence of results and how they work with a team
  • Tests the real job, not a rehearsed framework
  • Keeps each interviewer's judgement independent

Deliberately not measured

  • Employment law
  • Compensation benchmarking
Capability tested

Hiring judgement

The failure we’re looking for

Hiring for presence and polish over evidence

Grading

Decision model and LLM judge, calibrated against a blind PM review