Hire a PM

To: Elif Demir, VP Product From: Director of Product Re: What our PM loop predicts, and the loop I'd run next quarter

Bottom line

The behavioural metric round is the only part of the loop that tracks how hires perform. The presentation and VP Engineering rounds seem to drive hiring decisions without predicting performance. All three hires who struggled scored 4 on VP Engineering and 2 on the metric round. I'd make the metric round the centre of the loop and stop weighting the others as heavily. The sample is small, so this is a strong hypothesis to test, not a settled answer.

What the data covers

• 52 candidates reached the final rounds over 18 months: 9 hired, 33 rejected, 10 withdrew.
• Only the 9 hires have 12-month ratings. We can't tell whether our rejections were right, so we can't measure false negatives.
• We don't record withdrawal stage or reason, or which interviewer gave each score, so we can't check consistency across panels.

What the data shows

Specifics:

• Metric round: Hires who scored 2 all rated 2 at 12 months (3 of 3). Hires who scored 3 or 4 averaged 4.0. Its average gap between hired and rejected candidates is the smallest of the four, so the panel leaned on the other rounds more.
• The three struggling hires (rated 2 after 12 months) all scored 4 on VP Engineering and 4 on presentation, and scored 3–4 on the take-home. They looked strong everywhere except the metric round, which was the signal that predicted their problems.
• VP Engineering: All four hires who scored 4 here were rated 2 or 3 at 12 months.
• Withdrawals: 8 of the 10 candidates who withdrew scored 3 or higher on the metric round. We are losing strong candidates, and we don't know where or why.
• Background (hypothesis only): Big-tech hires (4) averaged 2.25 at 12 months; startup hires (3) averaged 4.7. The numbers are far too small to act on, but we should track it.

How confident we should be

• Metric round as the best predictor: moderate. A correlation of about 0.9 across nine hires is striking, and it fits what PM work involves: diagnosing a metric and deciding what to do about it. But with nine people, one or two different ratings would move the result a lot.
• Negative readings for presentation and VP Engineering: low. We only see hires, so the range is restricted, and the 1–4 scale gives few distinct values. The defensible claim is that these rounds did not predict success. I wouldn't claim they predict failure.
• Take-home as a screen-out: low to moderate. Zero hires among ten candidates who scored 1 is consistent with the take-home filtering out weak work, but that's a small group.
• Anything about rejected candidates: no confidence. We have no data on them.

The loop I'd run next quarter

1. Recruiter screen: No change.
2. Take-home: Keep it as a gate. Candidates who score 1 don't advance without a written override. Cap the time we ask for, since we're losing strong candidates and may be losing some to the length of the process.
3. Metric round (anchor): Rewrite as a structured 60-minute session with two interviewers. Candidates diagnose a real metric movement from a dashboard, propose causes, and say what they'd measure next. Score against a written rubric on 1–4 scales. A score of 3 or higher is required to hire. A score of 2 or lower is a no-hire unless you sign a written override.
4. Presentation: Fold it into the metric round as a communication check, so candidates spend less time on it, and don't weight it separately in the decision.
5. VP Engineering conversation: Keep it for a pass/fail check on serious collaboration red flags. Don't score it or count it in the decision, given how it has tracked performance.
6. Decision process: Every interviewer submits written scores before the panel meets, so no one anchors on the first opinion. The hiring manager still decides, but must write down why if they hire against the metric-round rule or reject someone who met it.
7. Data: Record the interviewer for every score, withdrawal stage and reason, and 6- and 12-month ratings for every hire. Keep scoring the old rounds next quarter even though they won't be weighted, so we keep building the dataset.

What I need from you

• Approval for the metric-round rubric and the override rule.
• Agreement on a review date. Four hires next quarter won't settle this. I'd treat the next two quarters as calibration, with a 6-month review of the four new hires and a full review once we have about twelve hires on the new loop.
• A decision on whether to pay for a small outside check on our rubric, such as having a PM coach double-score a sample of recordings.

To: Elif Demir, VP Product From: Director of Product Subject: PM interview loop: what the data says and what to change next quarter

What seems to predict performance

We have records for 52 final-round candidates, including 8 hires with 12-month ratings. Those hires averaged 3.5, versus a company PM average of 3.4; four were below 3.4, including two rated 2.

Among the eight hires, the clearest signal is the behavioural metric round. Its score has a strong positive association with 12-month ratings (exploratory Pearson r ≈ +0.93). The four hires rated 4–5 averaged 3.75 on this round, versus 2.5 for the four rated 2–3. This is promising evidence for probing how candidates actually move metrics and what they learn—not proof that the round is a reliable predictor.

The other scored rounds show no positive relationship with ratings in this small hired sample:

The negative estimates for presentation and VP Engineering should not be read as evidence that those skills hurt PM performance. With only eight hires, these figures are highly unstable. They do tell us that our current scores offer no reassuring evidence that those rounds predict success.

How confident we should be

Low confidence overall; modest confidence that the behavioural round is the best lead to investigate. Eight outcomes are far too few to set reliable weights or cutoffs, and interview scores are ordinal ratings, so the correlations are descriptive only. We also observe performance only for people we hired: panel decisions restrict the range of scores and can distort associations. Non-hires have no performance outcome for comparison.

We lack recruiter-screen scores, interviewer-level data, consistent evidence of scoring reliability, and context about role, manager, or rating differences. The company-average comparison is also a coarse measure. The data can guide a better experiment; it cannot establish causation or validate the loop.

Loop I’d run next quarter

1. Keep the recruiter screen for eligibility, motivation, and logistics, but don’t treat it as a predictive assessment. We have no screen-score data to evaluate.
2. Replace the week-long take-home deck and separate presentation with one time-boxed, job-relevant work sample. Use a live or short-prep product problem, not a polished deck. Score problem framing, user evidence, prioritization and tradeoffs, success measures, and communication against anchored criteria. This reduces candidate burden and stops us treating two related presentation exercises as independent evidence.
3. Keep and strengthen the behavioural round. Ask for a specific metric the candidate owned or influenced; probe the baseline, their actions, evidence of impact, attribution, tradeoffs, and what they learned. Use the same questions and anchored scorecard for every candidate.
4. Refocus the VP Engineering conversation on PM–engineering collaboration and concrete technical tradeoffs—not general “technical credibility.” Use a structured scenario and score it consistently; don’t make an unstructured impression a veto.
5. Have interviewers score independently before the panel discussion. The hiring manager can decide after reviewing evidence against the role criteria, with any disagreement recorded. Do not introduce numerical weights based on these eight hires.

For each of the four hires, preserve round-level scores and interviewer notes, then collect comparable 6- and 12-month outcomes with role and manager context. Review the evidence after the cohort has matured. Next quarter’s loop should be a more structured, lower-burden test—not a claim that we have already found a validated predictor.

To: Elif Demir, VP Product From: Director of Product Re: What in our PM loop predicts success, and the loop for next quarter's four hires

Bottom line

Of the four loop scores, only the behavioural round ("tell me about a metric you moved") tracks how PMs perform after hire. The take-home, the presentation and the VP Engineering conversation show no positive relationship, and the last two are negative among the people we hired. The sample is very small, so treat this as a strong lead rather than proof. I recommend making the behavioural evidence the anchor of the loop, shrinking the deck work, and turning the VP Eng conversation into a pass/fail check.

What we have

We have 52 final-round candidates: 9 hired, 33 rejected and 10 who withdrew. Only the 9 hires have 12-month ratings (company average 3.4). The three who struggled (C107, C132, C148) each scored 2, and C104 and C134 scored 3, also below average. All five below-average hires are the same kind of candidate: strong on the deck and the presentation, and scored 3–4 by the VP Eng.

What the data shows

Correlation with 12-month rating, among the 9 hires:

The behavioural pattern is clean. Hires scoring 4 averaged 4.7 (ratings 5, 5, 4), hires scoring 3 averaged 3.3, and all three hires scoring 2 were rated 2. Had we required a behavioural score of at least 3, we would have avoided all three struggling hires and kept every hire rated 3 or above. The pattern also holds within background: among the 4 Big tech hires, the one scoring 3 outperformed the three scoring 2, and among the other 5 hires the 4s outperformed the 3s.

The other three signals deserve less trust than the negative numbers suggest, for two reasons:

• Selection effects. We only see people we hired. If hiring managers pushed through candidates who were strong on the deck and the VP Eng score but weaker elsewhere, those scores will look negatively related to outcomes among hires even if they carry some real signal.
• The VP Eng score works as a gate. Every candidate scoring 1 was rejected or withdrew, so we have no outcome data on that group and cannot say whether the gate screens out good people.

Two further observations about the process:

• The take-home and presentation are the same artifact scored twice, so the deck gets double weight in the panel's discussion.
• Decisions look inconsistent. C135 (3/4/4/3) was rejected while C134 (4/4/3/3) was hired, and five candidates scoring 4 on the behavioural round were rejected. Three of those five had a VP Eng score of 1. With no written decision rule, we can't tell what drove these calls.

How confident we can be

• High confidence: the three struggling hires share one profile (Big tech, polished deck, VP Eng 4, behavioural 2), and our loop rewarded exactly those traits.
• Moderate confidence: the behavioural round carries real signal. The relationship is strong and consistent within subgroups, but it rests on 9 people and one outcome measure.
• Low confidence: any conclusion that the take-home, presentation or VP Eng conversation is useless. Selection effects and range restriction mean the data can't tell us either way. Our evidence only shows they haven't helped.
• Measurement problems:
• The 12-month rating comes from one source, likely the manager who chose the hire, so it may be biased toward justifying that choice.
• Behavioural scores and background are entangled: Big tech hires scored lower (2–3) than the others (3–4). I can't fully separate "behavioural score predicts performance" from "Big tech backgrounds fit us less well," though the within-group pattern favours the former.
• We have only a single 1–4 behavioural score per candidate, with no record of who scored it or how.
• Two or three outliers would change several of these numbers.

The loop I'd run next quarter

1. Recruiter screen. Keep it, with written criteria applied the same way for every candidate.
2. Two structured past-work interviews, replacing the single behavioural round. Different interviewers run them, each using a written rubric. They probe individual contribution versus team credit, the specific decisions the candidate made, what went wrong and what they would do differently, and how they worked with engineering. Interviewers score independently before any discussion. Two scores per candidate also lets us check whether the scoring is consistent.
3. Shorter work sample, replacing the one-week deck. Two to three hours, followed by a 30-minute live discussion, producing one combined score on a rubric. This cuts the double-counting and the candidate burden (10 of 52 withdrew). I'm keeping a work sample because we have no evidence it's useless, but it gets no more weight than any other single signal.
4. VP Eng conversation as a pass/fail risk check. The VP Eng answers one question: "Is there a specific, documented concern about this person's ability to work with engineering?" The 1–4 score no longer counts toward the decision, and a veto needs a written reason.
5. A written decision rule. A candidate must average at least 3 across the two past-work interviews to be eligible. Among those who clear that bar, the hiring manager decides and records the rationale against the rubric. This is a provisional threshold from 9 data points, so we should revisit it after this round.
6. Better outcome data. Run a structured 6- and 12-month review with defined criteria and a second reviewer, rather than a single manager rating, and log every candidate's scores in one place. Four hires bring us to 13 data points, still small, so decisions should stay modest in confidence until we have more.

What I'm not recommending

I'm not recommending a background filter (for example, avoiding Big tech candidates). The sample can't support it, and strong Big tech candidates may well do fine under a better-calibrated behavioural round. Interviewers should be told that polished decks and fluent technical conversation have not predicted success here.

I can have the rubrics and interviewer assignments ready within two weeks if you agree with the direction.