Hire a PM

MEMO

To: Elif Demir, VP Product From: Director of Product Subject: Fixing the PM Interview Loop Before Next Quarter’s Hiring

Elif,

I analyzed the data from the 52 PM candidates who reached our final rounds over the last 18 months—specifically looking at the 9 we hired, the 33 we rejected, and the 10 who withdrew.

The headline is clear: We are over-indexing on polished presentation skills and Big Tech pedigrees, while actively ignoring our single best predictor of on-the-job success.

Here is what the data shows, how confident we can be in these findings, and the revised interview loop I recommend for next quarter.

---

1. What Predicts Success (and What Doesn’t)

Of the 9 PMs we hired, their average 12-month performance rating is 3.33 (slightly below the company PM average of 3.4). Four hires met or exceeded expectations (ratings of 4 and 5), two were below average (rating of 3), and three struggled significantly (rating of 2).

When we compare their interview scores to their 12-month performance ratings, the patterns are striking:

• The Behavioural ("Metric You Moved") Round is our only strong positive predictor ($r = +0.87$).
• Every hired PM who scored a 4 in this round (C101, C141, C149) went on to become a top performer (ratings of 5, 5, and 4).
• Every hired PM who scored a 2 in this round (C107, C132, C148) struggled on the job (all three received a rating of 2).
• Despite this, we treated this round as optional: we hired three candidates who scored a 2 here because we were dazzled by their decks and pedigrees. Conversely, we rejected 5 candidates who scored a 4 in this round.
• The Take-Home and Presentation are negatively correlated with performance ($r = -0.57$ and $r = -0.84$).
• Our three struggling hires (all rated 2) scored 4s on their presentations, and two of them scored 4s on the take-home.
• Meanwhile, our top performers (rated 4 and 5) scored 2s and 3s on the take-home and presentation.
• A one-week strategy deck tests a candidate’s ability to craft polished slides in a vacuum with unlimited time—a skill that does not translate to shipping product at Kestrel.
• The VP Engineering Round is heavily biased toward Big Tech and negatively correlates with success ($r = -0.68$).
• Across all 52 candidates, Big Tech PMs averaged a 3.44 with the VP of Engineering, while Startup, Agency, and Consulting PMs averaged 2.16, 1.70, and 1.93, respectively.
• Every single Big Tech PM we hired scored a 4 in this round, yet they averaged a 2.25 performance rating at 12 months. Meanwhile, top performers from Startup and Agency backgrounds (C101, C149) scored 2s in this round. The unstructured conversation is indexing on Big Tech jargon rather than the scrappy technical collaboration we actually need.
• The Process Causes High Candidate Drop-Off.
• 10 of our 52 finalists (19%) withdrew. Notably, 3 of the 8 candidates who scored a 4 on our best predictor (the Behavioural round) withdrew. A one-week take-home followed by a 4-stage loop is too slow and burdensome for top-tier, in-demand talent.

---

2. How Confident Can We Be?

We should treat these findings as directionally vital, though statistically limited: 1. Small sample size ($n = 9$): Nine hires is a small dataset. A correlation of $+0.87$ on 9 data points is remarkably clean, but we shouldn't treat the exact coefficients as immutable laws. 2. Survivorship and selection bias: We don't know how the 33 rejected candidates would have performed. Furthermore, the negative correlation on the presentation and VP Eng rounds is partially an artifact of our hiring decisions: when candidates had a weak Behavioural score (2), we only hired them because they had 4s in the other rounds. 3. The "Big Tech" confounding variable: All three struggling hires came from Big Tech, while all four top performers came from Startups or Agencies. Big Tech PMs aced the deck and VP Eng rounds, while Startup/Agency PMs aced the metric-execution round.

Even with $n=9$, the qualitative logic backs the data: past execution on moving metrics predicts future execution at Kestrel; making pretty slides and name-dropping Big Tech infrastructure does not.

---

3. The Proposed Loop for Next Quarter

To hire four great PMs next quarter—and stop losing top candidates to withdrawal—we need to cut the time-to-hire, eliminate the take-home deck, and re-center the loop on execution and structured collaboration.

1. Recruiter Screen (30 mins) * No change, except: Source more heavily from Startup and Agency backgrounds, and screen out Big Tech candidates who haven't owned end-to-end execution.

2. Behavioural Execution Deep-Dive: "A Metric You Moved" (60 mins) * Change: Move this to the very first round after the recruiter screen and make a score of 3 or 4 a hard requirement to advance. * Why: It is our strongest predictor of success. Moving it up filters out "talkers" early and saves panel bandwidth.

3. Live Product Case & Prioritization Workshop (60 mins) * Change: Kill the 1-week take-home and panel presentation. Replace them with a zero-prep, live working session with two PMs using a real, messy Kestrel problem. * Why: The take-home selects for slide-making and causes nearly 20% of finalists to withdraw. A live case tests how candidates structure ambiguity, incorporate feedback, and think on their feet—without requiring 15 hours of unpaid homework.

4. Structured Engineering Collaboration Round (45 mins) * Change: Replace the unstructured VP Engineering "technical credibility" chat with a structured rubric led by a Senior Engineering Manager or Tech Lead. Prompt: "Tell me about a time you had to make a tough scope vs. technical-debt tradeoff with your engineering team." * Why: Eliminates the Big Tech halo effect and tests how the PM actually partners with engineers day-to-day.

5. Hiring Decision & Bar-Raising * Change: Replace open-ended panel discussion with independent score submission prior to debrief. Institute a hard rule: No candidate can be hired with a score below 3 on the Behavioural Execution round, regardless of how well they interview elsewhere.

If you're aligned, I will work with Recruiting and Engineering this week to build the new rubrics and live-case prompts so we are ready for next quarter's reqs.

To: Elif Demir, VP Product From: Director of Product Re: What in our PM loop predicts success, and the loop for next quarter's four hires

Bottom line

Of the four loop scores, only the behavioural round ("tell me about a metric you moved") tracks how PMs perform after hire. The take-home, the presentation and the VP Engineering conversation show no positive relationship, and the last two are negative among the people we hired. The sample is very small, so treat this as a strong lead rather than proof. I recommend making the behavioural evidence the anchor of the loop, shrinking the deck work, and turning the VP Eng conversation into a pass/fail check.

What we have

We have 52 final-round candidates: 9 hired, 33 rejected and 10 who withdrew. Only the 9 hires have 12-month ratings (company average 3.4). The three who struggled (C107, C132, C148) each scored 2, and C104 and C134 scored 3, also below average. All five below-average hires are the same kind of candidate: strong on the deck and the presentation, and scored 3–4 by the VP Eng.

What the data shows

Correlation with 12-month rating, among the 9 hires:

The behavioural pattern is clean. Hires scoring 4 averaged 4.7 (ratings 5, 5, 4), hires scoring 3 averaged 3.3, and all three hires scoring 2 were rated 2. Had we required a behavioural score of at least 3, we would have avoided all three struggling hires and kept every hire rated 3 or above. The pattern also holds within background: among the 4 Big tech hires, the one scoring 3 outperformed the three scoring 2, and among the other 5 hires the 4s outperformed the 3s.

The other three signals deserve less trust than the negative numbers suggest, for two reasons:

• Selection effects. We only see people we hired. If hiring managers pushed through candidates who were strong on the deck and the VP Eng score but weaker elsewhere, those scores will look negatively related to outcomes among hires even if they carry some real signal.
• The VP Eng score works as a gate. Every candidate scoring 1 was rejected or withdrew, so we have no outcome data on that group and cannot say whether the gate screens out good people.

Two further observations about the process:

• The take-home and presentation are the same artifact scored twice, so the deck gets double weight in the panel's discussion.
• Decisions look inconsistent. C135 (3/4/4/3) was rejected while C134 (4/4/3/3) was hired, and five candidates scoring 4 on the behavioural round were rejected. Three of those five had a VP Eng score of 1. With no written decision rule, we can't tell what drove these calls.

How confident we can be

• High confidence: the three struggling hires share one profile (Big tech, polished deck, VP Eng 4, behavioural 2), and our loop rewarded exactly those traits.
• Moderate confidence: the behavioural round carries real signal. The relationship is strong and consistent within subgroups, but it rests on 9 people and one outcome measure.
• Low confidence: any conclusion that the take-home, presentation or VP Eng conversation is useless. Selection effects and range restriction mean the data can't tell us either way. Our evidence only shows they haven't helped.
• Measurement problems:
• The 12-month rating comes from one source, likely the manager who chose the hire, so it may be biased toward justifying that choice.
• Behavioural scores and background are entangled: Big tech hires scored lower (2–3) than the others (3–4). I can't fully separate "behavioural score predicts performance" from "Big tech backgrounds fit us less well," though the within-group pattern favours the former.
• We have only a single 1–4 behavioural score per candidate, with no record of who scored it or how.
• Two or three outliers would change several of these numbers.

The loop I'd run next quarter

1. Recruiter screen. Keep it, with written criteria applied the same way for every candidate.
2. Two structured past-work interviews, replacing the single behavioural round. Different interviewers run them, each using a written rubric. They probe individual contribution versus team credit, the specific decisions the candidate made, what went wrong and what they would do differently, and how they worked with engineering. Interviewers score independently before any discussion. Two scores per candidate also lets us check whether the scoring is consistent.
3. Shorter work sample, replacing the one-week deck. Two to three hours, followed by a 30-minute live discussion, producing one combined score on a rubric. This cuts the double-counting and the candidate burden (10 of 52 withdrew). I'm keeping a work sample because we have no evidence it's useless, but it gets no more weight than any other single signal.
4. VP Eng conversation as a pass/fail risk check. The VP Eng answers one question: "Is there a specific, documented concern about this person's ability to work with engineering?" The 1–4 score no longer counts toward the decision, and a veto needs a written reason.
5. A written decision rule. A candidate must average at least 3 across the two past-work interviews to be eligible. Among those who clear that bar, the hiring manager decides and records the rationale against the rubric. This is a provisional threshold from 9 data points, so we should revisit it after this round.
6. Better outcome data. Run a structured 6- and 12-month review with defined criteria and a second reviewer, rather than a single manager rating, and log every candidate's scores in one place. Four hires bring us to 13 data points, still small, so decisions should stay modest in confidence until we have more.

What I'm not recommending

I'm not recommending a background filter (for example, avoiding Big tech candidates). The sample can't support it, and strong Big tech candidates may well do fine under a better-calibrated behavioural round. Interviewers should be told that polished decks and fluent technical conversation have not predicted success here.

I can have the rubrics and interviewer assignments ready within two weeks if you agree with the direction.

To: Elif Demir, VP Product From: Director of Product Re: What our PM loop predicts, and the loop for next quarter

Summary

Of the five stages in our loop, only the behavioural "metric you moved" round tracks how hires perform after 12 months. The take-home, the presentation and the VP Engineering conversation all point the wrong way: hires who scored higher on them performed worse. The evidence is thin (nine hires), so I'd treat this as a strong hint rather than proof. Even so, it is consistent enough to change the loop now and to measure the new loop properly.

What the data shows

Over 18 months, 52 candidates reached the final rounds. We hired 9, rejected 33, and 10 withdrew. Our hires averaged 3.3 at 12 months, slightly below the PM average of 3.4. The three who struggled are C107, C132 and C148, all rated 2.

Average 12-month rating of hires, by interview score

The three strugglers share an identical profile:

• 4 on the presentation
• 4 on VP Engineering
• 3 or 4 on the take-home
• 2 on the behavioural round
• Big-tech background

Our two strongest hires, C101 and C141 (both rated 5), had middling presentation and VP Engineering scores but scored 4 on the behavioural round.

In other words, we have been hiring for polish and for technical fluency, and overriding the one signal that worked.

Background is tangled up in this. All four big-tech hires were rated 2 or 3, averaging 2.25. The five others averaged 4.2. The VP Engineering round looks especially exposed to this:

• Every VP Engineering score of 4 in the dataset went to a big-tech candidate (9 of 11 big-tech candidates).
• No other candidate scored 4.

The round may be measuring familiarity with big-tech engineering culture rather than how well someone will work with our engineers.

The behavioural result holds within each group. This is the most reassuring part of the analysis.

• Among non-big-tech hires, the behavioural score still tracks performance. Those who scored 4 were rated 5, 5 and 4. Those who scored 3 were rated 3 and 4.
• Among big-tech hires, the one who scored 3 was rated 3 and the three who scored 2 were rated 2.

So the behavioural signal isn't simply a stand-in for background. Within non-big-tech hires, the presentation and take-home scores still lean negative.

How confident we can be

I'm only moderately confident, for five reasons:

• Nine data points. One or two different outcomes could change the picture, and none of these correlations would survive a strict significance test.
• We only see outcomes for people we hired. Hires were selected partly on high presentation and VP Engineering scores, so we can't see how low scorers on those stages would have done. We also don't know about strong behavioural candidates we rejected (C121, C123, C131, C135, C146), or about withdrawals. C135, for example, was rejected despite scoring 3, 4, 4 and 3, among the strongest sets of scores in the pool.
• Background is confounded. With four big-tech hires, I can't fully separate "big-tech PMs struggle here" from "our loop rewards the wrong things in big-tech PMs." Either way, the fix is to the instruments, not to filter on background.
• 12-month ratings are one manager's judgement, and they also reflect team and scope.
• Interviewer effects. We don't know who scored which candidates, so some of this may come from individual raters rather than from the stages themselves.

What I am confident about is narrower. The take-home, presentation and VP Engineering scores give us no evidence that they help. Given what they cost candidates and the panel, that is reason enough to stop relying on them.

The loop I'd run next quarter

1. Recruiter screen. Unchanged.

2. Structured behavioural round, now the core of the loop (two interviewers, about 60 minutes). - Keep "a metric you moved," and add a second prompt: "a decision you got wrong and what you changed." - Use a written rubric that scores: - specificity about the candidate's own actions; - how they diagnosed the problem; - the trade-offs they made; - what they learned. - Score the substance, not the delivery. Each interviewer scores independently.

3. Replace the week-long take-home and presentation with a 90-minute live working session. - Give the candidate a real, anonymised Kestrel problem with messy data, and work through it with a PM and an engineer. - Score how they reason, prioritise and respond to pushback, not how finished the output looks. - This also cuts the candidate time we ask for. Ten of 52 finalists withdrew; I can't say the week-long take-home caused that, but it won't have helped.

4. Redesign the VP Engineering round as a structured "working with engineering" round. - Base it on a scenario such as a scope cut, a technical-debt trade-off or an estimate dispute, with a rubric agreed with Engineering. - Score it pass/concern only until we have evidence that it predicts anything, so it can flag real problems without rewarding big-tech vocabulary.

5. Change how we decide. - All scores and written evidence go in before the debrief, so nobody anchors on the loudest voice. - No hire with a behavioural score below 3. - The hiring manager still decides. If they hire against the behavioural score, they write down why.

6. Measure it this time. - Log every interviewer's scores, not just the panel's. - Collect 6-month and 12-month ratings against a consistent rubric. - With four more hires we'll have 13 to analyse. We should review the data in two quarters, not two years. - I'd also like a quick look back at the strong behavioural candidates we rejected, to see whether the debrief was systematically discounting that round.

The ask

I'd like your sign-off to:

• pilot this loop for the four upcoming hires;
• get two hours of interviewer calibration on the new rubrics;
• secure the VP Engineering's agreement to redesign their round.

I can have the rubrics and the working-session problem ready within three weeks.