Hire a PM

To: Elif Demir, VP Product From: Director of Product Subject: PM interview loop: what the data says and what to change next quarter

What seems to predict performance

We have records for 52 final-round candidates, including 8 hires with 12-month ratings. Those hires averaged 3.5, versus a company PM average of 3.4; four were below 3.4, including two rated 2.

Among the eight hires, the clearest signal is the behavioural metric round. Its score has a strong positive association with 12-month ratings (exploratory Pearson r ≈ +0.93). The four hires rated 4–5 averaged 3.75 on this round, versus 2.5 for the four rated 2–3. This is promising evidence for probing how candidates actually move metrics and what they learn—not proof that the round is a reliable predictor.

The other scored rounds show no positive relationship with ratings in this small hired sample:

The negative estimates for presentation and VP Engineering should not be read as evidence that those skills hurt PM performance. With only eight hires, these figures are highly unstable. They do tell us that our current scores offer no reassuring evidence that those rounds predict success.

How confident we should be

Low confidence overall; modest confidence that the behavioural round is the best lead to investigate. Eight outcomes are far too few to set reliable weights or cutoffs, and interview scores are ordinal ratings, so the correlations are descriptive only. We also observe performance only for people we hired: panel decisions restrict the range of scores and can distort associations. Non-hires have no performance outcome for comparison.

We lack recruiter-screen scores, interviewer-level data, consistent evidence of scoring reliability, and context about role, manager, or rating differences. The company-average comparison is also a coarse measure. The data can guide a better experiment; it cannot establish causation or validate the loop.

Loop I’d run next quarter

1. Keep the recruiter screen for eligibility, motivation, and logistics, but don’t treat it as a predictive assessment. We have no screen-score data to evaluate.
2. Replace the week-long take-home deck and separate presentation with one time-boxed, job-relevant work sample. Use a live or short-prep product problem, not a polished deck. Score problem framing, user evidence, prioritization and tradeoffs, success measures, and communication against anchored criteria. This reduces candidate burden and stops us treating two related presentation exercises as independent evidence.
3. Keep and strengthen the behavioural round. Ask for a specific metric the candidate owned or influenced; probe the baseline, their actions, evidence of impact, attribution, tradeoffs, and what they learned. Use the same questions and anchored scorecard for every candidate.
4. Refocus the VP Engineering conversation on PM–engineering collaboration and concrete technical tradeoffs—not general “technical credibility.” Use a structured scenario and score it consistently; don’t make an unstructured impression a veto.
5. Have interviewers score independently before the panel discussion. The hiring manager can decide after reviewing evidence against the role criteria, with any disagreement recorded. Do not introduce numerical weights based on these eight hires.

For each of the four hires, preserve round-level scores and interviewer notes, then collect comparable 6- and 12-month outcomes with role and manager context. Review the evidence after the cohort has matured. Next quarter’s loop should be a more structured, lower-burden test—not a claim that we have already found a validated predictor.

To: Elif Demir, VP Product From: Director of Product Re: What our PM loop predicts, and the loop for next quarter

Summary

Of the five stages in our loop, only the behavioural "metric you moved" round tracks how hires perform after 12 months. The take-home, the presentation and the VP Engineering conversation all point the wrong way: hires who scored higher on them performed worse. The evidence is thin (nine hires), so I'd treat this as a strong hint rather than proof. Even so, it is consistent enough to change the loop now and to measure the new loop properly.

What the data shows

Over 18 months, 52 candidates reached the final rounds. We hired 9, rejected 33, and 10 withdrew. Our hires averaged 3.3 at 12 months, slightly below the PM average of 3.4. The three who struggled are C107, C132 and C148, all rated 2.

Average 12-month rating of hires, by interview score

The three strugglers share an identical profile:

• 4 on the presentation
• 4 on VP Engineering
• 3 or 4 on the take-home
• 2 on the behavioural round
• Big-tech background

Our two strongest hires, C101 and C141 (both rated 5), had middling presentation and VP Engineering scores but scored 4 on the behavioural round.

In other words, we have been hiring for polish and for technical fluency, and overriding the one signal that worked.

Background is tangled up in this. All four big-tech hires were rated 2 or 3, averaging 2.25. The five others averaged 4.2. The VP Engineering round looks especially exposed to this:

• Every VP Engineering score of 4 in the dataset went to a big-tech candidate (9 of 11 big-tech candidates).
• No other candidate scored 4.

The round may be measuring familiarity with big-tech engineering culture rather than how well someone will work with our engineers.

The behavioural result holds within each group. This is the most reassuring part of the analysis.

• Among non-big-tech hires, the behavioural score still tracks performance. Those who scored 4 were rated 5, 5 and 4. Those who scored 3 were rated 3 and 4.
• Among big-tech hires, the one who scored 3 was rated 3 and the three who scored 2 were rated 2.

So the behavioural signal isn't simply a stand-in for background. Within non-big-tech hires, the presentation and take-home scores still lean negative.

How confident we can be

I'm only moderately confident, for five reasons:

• Nine data points. One or two different outcomes could change the picture, and none of these correlations would survive a strict significance test.
• We only see outcomes for people we hired. Hires were selected partly on high presentation and VP Engineering scores, so we can't see how low scorers on those stages would have done. We also don't know about strong behavioural candidates we rejected (C121, C123, C131, C135, C146), or about withdrawals. C135, for example, was rejected despite scoring 3, 4, 4 and 3, among the strongest sets of scores in the pool.
• Background is confounded. With four big-tech hires, I can't fully separate "big-tech PMs struggle here" from "our loop rewards the wrong things in big-tech PMs." Either way, the fix is to the instruments, not to filter on background.
• 12-month ratings are one manager's judgement, and they also reflect team and scope.
• Interviewer effects. We don't know who scored which candidates, so some of this may come from individual raters rather than from the stages themselves.

What I am confident about is narrower. The take-home, presentation and VP Engineering scores give us no evidence that they help. Given what they cost candidates and the panel, that is reason enough to stop relying on them.

The loop I'd run next quarter

1. Recruiter screen. Unchanged.

2. Structured behavioural round, now the core of the loop (two interviewers, about 60 minutes). - Keep "a metric you moved," and add a second prompt: "a decision you got wrong and what you changed." - Use a written rubric that scores: - specificity about the candidate's own actions; - how they diagnosed the problem; - the trade-offs they made; - what they learned. - Score the substance, not the delivery. Each interviewer scores independently.

3. Replace the week-long take-home and presentation with a 90-minute live working session. - Give the candidate a real, anonymised Kestrel problem with messy data, and work through it with a PM and an engineer. - Score how they reason, prioritise and respond to pushback, not how finished the output looks. - This also cuts the candidate time we ask for. Ten of 52 finalists withdrew; I can't say the week-long take-home caused that, but it won't have helped.

4. Redesign the VP Engineering round as a structured "working with engineering" round. - Base it on a scenario such as a scope cut, a technical-debt trade-off or an estimate dispute, with a rubric agreed with Engineering. - Score it pass/concern only until we have evidence that it predicts anything, so it can flag real problems without rewarding big-tech vocabulary.

5. Change how we decide. - All scores and written evidence go in before the debrief, so nobody anchors on the loudest voice. - No hire with a behavioural score below 3. - The hiring manager still decides. If they hire against the behavioural score, they write down why.

6. Measure it this time. - Log every interviewer's scores, not just the panel's. - Collect 6-month and 12-month ratings against a consistent rubric. - With four more hires we'll have 13 to analyse. We should review the data in two quarters, not two years. - I'd also like a quick look back at the strong behavioural candidates we rejected, to see whether the debrief was systematically discounting that round.

The ask

I'd like your sign-off to:

• pilot this loop for the four upcoming hires;
• get two hours of interviewer calibration on the new rubrics;
• secure the VP Engineering's agreement to redesign their round.

I can have the rubrics and the working-session problem ready within three weeks.

To: Elif Demir, VP Product From: Director of Product Subject: PM interview loop: what the evidence supports and what to change

Recommendation

For next quarter’s four hires, retain and standardise the behavioural metric round; replace the week-long strategy deck and presentation with a bounded, realistic work sample; and replace “technical credibility” with a structured engineering-partnership interview.

The evidence supports changing what we reward—not claiming we have discovered a validated hiring formula.

What the data says

We have 52 finalists: nine hired, 33 rejected and ten withdrawn. Only the nine hires have performance outcomes. Their average 12-month rating is 3.33, close to the company PM average of 3.4. Three received a 2; none received a 1. That average masks the individual misses Elif is concerned about, but does not establish that the loop underperforms an alternative.

Among those nine hires:

Descriptive Pearson correlations on ordinal scores; not validated predictive estimates.

The behavioural pattern is particularly clear: the three hires scoring 2 all received performance ratings of 2; those scoring 3 averaged 3.33; those scoring 4 averaged 4.67. Probing actual ownership, measurement and learning appears promising.

Conversely, all three hires rated 2 had scored 4 on both presentation and VP Engineering. Two also scored 4 on the take-home. These rounds clearly failed to screen out our weakest performers. They may reward polish, familiarity or pedigree rather than effective PM work—but the data does not establish that mechanism.

We cannot evaluate the recruiter screen: it has neither scores nor data on candidates eliminated before finals. Nor can we isolate the value of panel discussion or hiring-manager judgment without decision rationales.

How confident should we be?

Enough to change the process; not enough to treat these correlations as durable prediction.

There are four important limitations:

• Nine outcomes are very few. One or two different hires could materially change the estimates. We cannot credibly estimate each round’s independent contribution.
• We observe performance only for selected candidates. Hiring already depended on these interviews. Conditioning on selection can distort associations; we do not know how rejected or withdrawn candidates would have performed.
• Background and scores are entangled. All four big-tech hires received VP Engineering scores of 4 and averaged 2.25 in performance; the three startup hires averaged 4.67. That does not establish a background effect—or justify preferring startup candidates. It prevents us from separating interview signal, background and role fit.
• Performance ratings are imperfect outcomes. We lack information on manager calibration, assignment difficulty, onboarding and opportunities to deliver results.

Accordingly, I would prioritise the behavioural round, but would not reverse-score presentations or engineering interviews, adopt background preferences, or derive numerical weights from nine people. The negative correlations show that those scores were not useful positive signals here; they do not prove the underlying skills are unimportant.

The loop I would run next quarter

1. Structured recruiter screen — 25 minutes. Check role requirements, relevant scope, motivation and practical constraints against a common checklist. Do not use employer prestige as a proxy for capability. Explain the process and time commitment.

2. Realistic work sample — 75 minutes, replacing both deck and presentation. Give every candidate the same bounded, fictional product problem, customer evidence, metric definitions and engineering constraints. Ask them to diagnose the problem, prioritise a response, define success and revise their recommendation when new evidence arrives. No slide deck or advance homework.

Score problem framing, evidence use, prioritisation, measurement and adaptation. Assess communication through clarity of reasoning, not presentation polish. Offer reasonable accommodations and equivalent administration conditions.

3. Structured behavioural metric interview — 50 minutes. Retain the current core question, with consistent probes: baseline, personal contribution, alternatives considered, causal evidence, trade-offs, setbacks and learning. Use a second example to distinguish repeatable judgment from one well-rehearsed story. Anchor scores to demonstrated ownership and reasoning, not the absolute size of a metric increase.

4. Engineering partnership interview — 40 minutes. An engineering leader uses a common scenario involving feasibility, scope, reliability and disagreement. Score how candidates uncover constraints and negotiate trade-offs—not jargon, coding knowledge or perceived “credibility.”

Decision process. Before interviewing, agree on competency definitions, anchored 1–4 ratings and minimum acceptable evidence for critical competencies. Interviewers submit scores and supporting observations independently before debriefing. Discuss discrepancies against the rubric; do not mechanically average scores. The hiring manager records the decision, unresolved risks and any exception to the agreed bar.

How we will learn

Record scores, assessor identity, decision reasons, candidate time and withdrawal stage for every finalist. Audit scoring consistency and progression by background and, where appropriately collected, demographic group.

For hires, collect consistently defined 3-, 6- and 12-month outcomes, with role context and performance assessment kept independent of interview scores where feasible. Review process quality after the next four hires, but accumulate more cohorts before claiming predictive validity. Four additional hires will test whether we can run a better loop—not prove that it works.