Hire a PM

To: Elif Demir, VP Product From: Director of Product Re: What our PM loop predicts, and the loop I'd run next quarter

Bottom line

The behavioural metric round is the only part of the loop that tracks how hires perform. The presentation and VP Engineering rounds seem to drive hiring decisions without predicting performance. All three hires who struggled scored 4 on VP Engineering and 2 on the metric round. I'd make the metric round the centre of the loop and stop weighting the others as heavily. The sample is small, so this is a strong hypothesis to test, not a settled answer.

What the data covers

• 52 candidates reached the final rounds over 18 months: 9 hired, 33 rejected, 10 withdrew.
• Only the 9 hires have 12-month ratings. We can't tell whether our rejections were right, so we can't measure false negatives.
• We don't record withdrawal stage or reason, or which interviewer gave each score, so we can't check consistency across panels.

What the data shows

Specifics:

• Metric round: Hires who scored 2 all rated 2 at 12 months (3 of 3). Hires who scored 3 or 4 averaged 4.0. Its average gap between hired and rejected candidates is the smallest of the four, so the panel leaned on the other rounds more.
• The three struggling hires (rated 2 after 12 months) all scored 4 on VP Engineering and 4 on presentation, and scored 3–4 on the take-home. They looked strong everywhere except the metric round, which was the signal that predicted their problems.
• VP Engineering: All four hires who scored 4 here were rated 2 or 3 at 12 months.
• Withdrawals: 8 of the 10 candidates who withdrew scored 3 or higher on the metric round. We are losing strong candidates, and we don't know where or why.
• Background (hypothesis only): Big-tech hires (4) averaged 2.25 at 12 months; startup hires (3) averaged 4.7. The numbers are far too small to act on, but we should track it.

How confident we should be

• Metric round as the best predictor: moderate. A correlation of about 0.9 across nine hires is striking, and it fits what PM work involves: diagnosing a metric and deciding what to do about it. But with nine people, one or two different ratings would move the result a lot.
• Negative readings for presentation and VP Engineering: low. We only see hires, so the range is restricted, and the 1–4 scale gives few distinct values. The defensible claim is that these rounds did not predict success. I wouldn't claim they predict failure.
• Take-home as a screen-out: low to moderate. Zero hires among ten candidates who scored 1 is consistent with the take-home filtering out weak work, but that's a small group.
• Anything about rejected candidates: no confidence. We have no data on them.

The loop I'd run next quarter

1. Recruiter screen: No change.
2. Take-home: Keep it as a gate. Candidates who score 1 don't advance without a written override. Cap the time we ask for, since we're losing strong candidates and may be losing some to the length of the process.
3. Metric round (anchor): Rewrite as a structured 60-minute session with two interviewers. Candidates diagnose a real metric movement from a dashboard, propose causes, and say what they'd measure next. Score against a written rubric on 1–4 scales. A score of 3 or higher is required to hire. A score of 2 or lower is a no-hire unless you sign a written override.
4. Presentation: Fold it into the metric round as a communication check, so candidates spend less time on it, and don't weight it separately in the decision.
5. VP Engineering conversation: Keep it for a pass/fail check on serious collaboration red flags. Don't score it or count it in the decision, given how it has tracked performance.
6. Decision process: Every interviewer submits written scores before the panel meets, so no one anchors on the first opinion. The hiring manager still decides, but must write down why if they hire against the metric-round rule or reject someone who met it.
7. Data: Record the interviewer for every score, withdrawal stage and reason, and 6- and 12-month ratings for every hire. Keep scoring the old rounds next quarter even though they won't be weighted, so we keep building the dataset.

What I need from you

• Approval for the metric-round rubric and the override rule.
• Agreement on a review date. Four hires next quarter won't settle this. I'd treat the next two quarters as calibration, with a 6-month review of the four new hires and a full review once we have about twelve hires on the new loop.
• A decision on whether to pay for a small outside check on our rubric, such as having a PM coach double-score a sample of recordings.

To: Elif Demir, VP Product From: Director of Product Subject: PM interview loop: what the evidence supports and what to change

Recommendation

For next quarter’s four hires, retain and standardise the behavioural metric round; replace the week-long strategy deck and presentation with a bounded, realistic work sample; and replace “technical credibility” with a structured engineering-partnership interview.

The evidence supports changing what we reward—not claiming we have discovered a validated hiring formula.

What the data says

We have 52 finalists: nine hired, 33 rejected and ten withdrawn. Only the nine hires have performance outcomes. Their average 12-month rating is 3.33, close to the company PM average of 3.4. Three received a 2; none received a 1. That average masks the individual misses Elif is concerned about, but does not establish that the loop underperforms an alternative.

Among those nine hires:

Descriptive Pearson correlations on ordinal scores; not validated predictive estimates.

The behavioural pattern is particularly clear: the three hires scoring 2 all received performance ratings of 2; those scoring 3 averaged 3.33; those scoring 4 averaged 4.67. Probing actual ownership, measurement and learning appears promising.

Conversely, all three hires rated 2 had scored 4 on both presentation and VP Engineering. Two also scored 4 on the take-home. These rounds clearly failed to screen out our weakest performers. They may reward polish, familiarity or pedigree rather than effective PM work—but the data does not establish that mechanism.

We cannot evaluate the recruiter screen: it has neither scores nor data on candidates eliminated before finals. Nor can we isolate the value of panel discussion or hiring-manager judgment without decision rationales.

How confident should we be?

Enough to change the process; not enough to treat these correlations as durable prediction.

There are four important limitations:

• Nine outcomes are very few. One or two different hires could materially change the estimates. We cannot credibly estimate each round’s independent contribution.
• We observe performance only for selected candidates. Hiring already depended on these interviews. Conditioning on selection can distort associations; we do not know how rejected or withdrawn candidates would have performed.
• Background and scores are entangled. All four big-tech hires received VP Engineering scores of 4 and averaged 2.25 in performance; the three startup hires averaged 4.67. That does not establish a background effect—or justify preferring startup candidates. It prevents us from separating interview signal, background and role fit.
• Performance ratings are imperfect outcomes. We lack information on manager calibration, assignment difficulty, onboarding and opportunities to deliver results.

Accordingly, I would prioritise the behavioural round, but would not reverse-score presentations or engineering interviews, adopt background preferences, or derive numerical weights from nine people. The negative correlations show that those scores were not useful positive signals here; they do not prove the underlying skills are unimportant.

The loop I would run next quarter

1. Structured recruiter screen — 25 minutes. Check role requirements, relevant scope, motivation and practical constraints against a common checklist. Do not use employer prestige as a proxy for capability. Explain the process and time commitment.

2. Realistic work sample — 75 minutes, replacing both deck and presentation. Give every candidate the same bounded, fictional product problem, customer evidence, metric definitions and engineering constraints. Ask them to diagnose the problem, prioritise a response, define success and revise their recommendation when new evidence arrives. No slide deck or advance homework.

Score problem framing, evidence use, prioritisation, measurement and adaptation. Assess communication through clarity of reasoning, not presentation polish. Offer reasonable accommodations and equivalent administration conditions.

3. Structured behavioural metric interview — 50 minutes. Retain the current core question, with consistent probes: baseline, personal contribution, alternatives considered, causal evidence, trade-offs, setbacks and learning. Use a second example to distinguish repeatable judgment from one well-rehearsed story. Anchor scores to demonstrated ownership and reasoning, not the absolute size of a metric increase.

4. Engineering partnership interview — 40 minutes. An engineering leader uses a common scenario involving feasibility, scope, reliability and disagreement. Score how candidates uncover constraints and negotiate trade-offs—not jargon, coding knowledge or perceived “credibility.”

Decision process. Before interviewing, agree on competency definitions, anchored 1–4 ratings and minimum acceptable evidence for critical competencies. Interviewers submit scores and supporting observations independently before debriefing. Discuss discrepancies against the rubric; do not mechanically average scores. The hiring manager records the decision, unresolved risks and any exception to the agreed bar.

How we will learn

Record scores, assessor identity, decision reasons, candidate time and withdrawal stage for every finalist. Audit scoring consistency and progression by background and, where appropriately collected, demographic group.

For hires, collect consistently defined 3-, 6- and 12-month outcomes, with role context and performance assessment kept independent of interview scores where feasible. Review process quality after the next four hires, but accumulate more cohorts before claiming predictive validity. Four additional hires will test whether we can run a better loop—not prove that it works.

To: Elif Demir, VP Product From: Director of Product Subject: PM interview loop: what the data says and what to change next quarter

What seems to predict performance

We have records for 52 final-round candidates, including 8 hires with 12-month ratings. Those hires averaged 3.5, versus a company PM average of 3.4; four were below 3.4, including two rated 2.

Among the eight hires, the clearest signal is the behavioural metric round. Its score has a strong positive association with 12-month ratings (exploratory Pearson r ≈ +0.93). The four hires rated 4–5 averaged 3.75 on this round, versus 2.5 for the four rated 2–3. This is promising evidence for probing how candidates actually move metrics and what they learn—not proof that the round is a reliable predictor.

The other scored rounds show no positive relationship with ratings in this small hired sample:

The negative estimates for presentation and VP Engineering should not be read as evidence that those skills hurt PM performance. With only eight hires, these figures are highly unstable. They do tell us that our current scores offer no reassuring evidence that those rounds predict success.

How confident we should be

Low confidence overall; modest confidence that the behavioural round is the best lead to investigate. Eight outcomes are far too few to set reliable weights or cutoffs, and interview scores are ordinal ratings, so the correlations are descriptive only. We also observe performance only for people we hired: panel decisions restrict the range of scores and can distort associations. Non-hires have no performance outcome for comparison.

We lack recruiter-screen scores, interviewer-level data, consistent evidence of scoring reliability, and context about role, manager, or rating differences. The company-average comparison is also a coarse measure. The data can guide a better experiment; it cannot establish causation or validate the loop.

Loop I’d run next quarter

1. Keep the recruiter screen for eligibility, motivation, and logistics, but don’t treat it as a predictive assessment. We have no screen-score data to evaluate.
2. Replace the week-long take-home deck and separate presentation with one time-boxed, job-relevant work sample. Use a live or short-prep product problem, not a polished deck. Score problem framing, user evidence, prioritization and tradeoffs, success measures, and communication against anchored criteria. This reduces candidate burden and stops us treating two related presentation exercises as independent evidence.
3. Keep and strengthen the behavioural round. Ask for a specific metric the candidate owned or influenced; probe the baseline, their actions, evidence of impact, attribution, tradeoffs, and what they learned. Use the same questions and anchored scorecard for every candidate.
4. Refocus the VP Engineering conversation on PM–engineering collaboration and concrete technical tradeoffs—not general “technical credibility.” Use a structured scenario and score it consistently; don’t make an unstructured impression a veto.
5. Have interviewers score independently before the panel discussion. The hiring manager can decide after reviewing evidence against the role criteria, with any disagreement recorded. Do not introduce numerical weights based on these eight hires.

For each of the four hires, preserve round-level scores and interviewer notes, then collect comparable 6- and 12-month outcomes with role and manager context. Review the evidence after the cohort has matured. Next quarter’s loop should be a more structured, lower-burden test—not a claim that we have already found a validated predictor.