Hire a PM

To: Elif Demir, VP Product From: Director of Product Re: What our PM loop predicts, and the loop for next quarter

Summary

Of the five stages in our loop, only the behavioural "metric you moved" round tracks how hires perform after 12 months. The take-home, the presentation and the VP Engineering conversation all point the wrong way: hires who scored higher on them performed worse. The evidence is thin (nine hires), so I'd treat this as a strong hint rather than proof. Even so, it is consistent enough to change the loop now and to measure the new loop properly.

What the data shows

Over 18 months, 52 candidates reached the final rounds. We hired 9, rejected 33, and 10 withdrew. Our hires averaged 3.3 at 12 months, slightly below the PM average of 3.4. The three who struggled are C107, C132 and C148, all rated 2.

Average 12-month rating of hires, by interview score

The three strugglers share an identical profile:

• 4 on the presentation
• 4 on VP Engineering
• 3 or 4 on the take-home
• 2 on the behavioural round
• Big-tech background

Our two strongest hires, C101 and C141 (both rated 5), had middling presentation and VP Engineering scores but scored 4 on the behavioural round.

In other words, we have been hiring for polish and for technical fluency, and overriding the one signal that worked.

Background is tangled up in this. All four big-tech hires were rated 2 or 3, averaging 2.25. The five others averaged 4.2. The VP Engineering round looks especially exposed to this:

• Every VP Engineering score of 4 in the dataset went to a big-tech candidate (9 of 11 big-tech candidates).
• No other candidate scored 4.

The round may be measuring familiarity with big-tech engineering culture rather than how well someone will work with our engineers.

The behavioural result holds within each group. This is the most reassuring part of the analysis.

• Among non-big-tech hires, the behavioural score still tracks performance. Those who scored 4 were rated 5, 5 and 4. Those who scored 3 were rated 3 and 4.
• Among big-tech hires, the one who scored 3 was rated 3 and the three who scored 2 were rated 2.

So the behavioural signal isn't simply a stand-in for background. Within non-big-tech hires, the presentation and take-home scores still lean negative.

How confident we can be

I'm only moderately confident, for five reasons:

• Nine data points. One or two different outcomes could change the picture, and none of these correlations would survive a strict significance test.
• We only see outcomes for people we hired. Hires were selected partly on high presentation and VP Engineering scores, so we can't see how low scorers on those stages would have done. We also don't know about strong behavioural candidates we rejected (C121, C123, C131, C135, C146), or about withdrawals. C135, for example, was rejected despite scoring 3, 4, 4 and 3, among the strongest sets of scores in the pool.
• Background is confounded. With four big-tech hires, I can't fully separate "big-tech PMs struggle here" from "our loop rewards the wrong things in big-tech PMs." Either way, the fix is to the instruments, not to filter on background.
• 12-month ratings are one manager's judgement, and they also reflect team and scope.
• Interviewer effects. We don't know who scored which candidates, so some of this may come from individual raters rather than from the stages themselves.

What I am confident about is narrower. The take-home, presentation and VP Engineering scores give us no evidence that they help. Given what they cost candidates and the panel, that is reason enough to stop relying on them.

The loop I'd run next quarter

1. Recruiter screen. Unchanged.

2. Structured behavioural round, now the core of the loop (two interviewers, about 60 minutes). - Keep "a metric you moved," and add a second prompt: "a decision you got wrong and what you changed." - Use a written rubric that scores: - specificity about the candidate's own actions; - how they diagnosed the problem; - the trade-offs they made; - what they learned. - Score the substance, not the delivery. Each interviewer scores independently.

3. Replace the week-long take-home and presentation with a 90-minute live working session. - Give the candidate a real, anonymised Kestrel problem with messy data, and work through it with a PM and an engineer. - Score how they reason, prioritise and respond to pushback, not how finished the output looks. - This also cuts the candidate time we ask for. Ten of 52 finalists withdrew; I can't say the week-long take-home caused that, but it won't have helped.

4. Redesign the VP Engineering round as a structured "working with engineering" round. - Base it on a scenario such as a scope cut, a technical-debt trade-off or an estimate dispute, with a rubric agreed with Engineering. - Score it pass/concern only until we have evidence that it predicts anything, so it can flag real problems without rewarding big-tech vocabulary.

5. Change how we decide. - All scores and written evidence go in before the debrief, so nobody anchors on the loudest voice. - No hire with a behavioural score below 3. - The hiring manager still decides. If they hire against the behavioural score, they write down why.

6. Measure it this time. - Log every interviewer's scores, not just the panel's. - Collect 6-month and 12-month ratings against a consistent rubric. - With four more hires we'll have 13 to analyse. We should review the data in two quarters, not two years. - I'd also like a quick look back at the strong behavioural candidates we rejected, to see whether the debrief was systematically discounting that round.

The ask

I'd like your sign-off to:

• pilot this loop for the four upcoming hires;
• get two hours of interviewer calibration on the new rubrics;
• secure the VP Engineering's agreement to redesign their round.

I can have the rubrics and the working-session problem ready within three weeks.

MEMORANDUM

TO: Elif Demir, VP Product FROM: Director of Product DATE: October 24, 2023 SUBJECT: Analysis of PM Hiring Loop Performance and Proposed Redesign for Q1 Hiring

*

Executive Summary

Over the past 18 months, we evaluated 52 PM finalists and hired 9. Three hires struggled significantly (rated 2/5 after 12 months, well below our 3.4 company average).

Our data reveals a clear and counter-intuitive pattern: 1. The only reliable predictor of success is the Behavioural Metric Round ("Tell me about a metric you moved"). 2. The Take-Home Deck, Presentation Round, and VP Engineering Conversation are negatively correlated with performance. They reward presentation polish, shared jargon, and Big Tech pedigree over actual execution. 3. The 1-week take-home deck is causing talent leakage: 19% of final-round candidates withdrew, including top performers. 4. All three struggling hires share the exact same profile: Big Tech background, high presentation and VP Eng scores (4/4), and poor behavioural metric scores (2/4). We over-indexed on polish and excused weak operational substance.

Below is an analysis of our interview signals, our statistical confidence, and a redesigned loop for next quarter.

---

What the Data Tells Us

#### 1. What Predicts Performance: The Behavioural Metric Round Candidate performance on the behavioural round correlates almost perfectly with 12-month post-hire ratings: * Score 4 (3 hires): Average rating 4.67 (two 5s, one 4). * Score 3 (3 hires): Average rating 3.33 (one 4, two 3s). * Score 2 (3 hires): Average rating 2.00 (all three are our struggling hires: C107, C132, C148).

PMs who could drill into the mechanics of moving a metric—causality, counter-metrics, execution trade-offs, and failure lessons—excelled. Those who could not struggled immediately in our environment.

#### 2. What Misleads: Presentation, Take-Home, and VP Engineering Counter-intuitively, scoring at the top of these three stages predicted lower job performance: * Presentation Score: Hires scoring 4/4 averaged a 2.25 rating; hires scoring 2 or 3 averaged 4.20. * VP Engineering Score: Hires scoring 4/4 averaged 2.25; hires scoring 2 or 3 averaged 4.20. * Take-Home Score: Hires scoring 4/4 averaged 2.50; hires scoring 2 or 3 averaged 4.00.

Why are these rounds misleading us? * Deck Polish $\neq$ Product Sense: The take-home strategy deck and presentation test slide design and storytelling. Candidates with Big Tech or consulting backgrounds excel at creating slick artifacts, which creates an "executive presence" halo effect. * Flawed Engineering Rubric: "Technical credibility" currently measures whether a PM speaks Big Tech architectural jargon, not whether they partner effectively with engineers on scoping, trade-offs, and technical debt. * The "Big Tech Halo": All 4 Big Tech hires received top VP Eng scores (4/4), but 3 received failing behavioural scores (2/4). Panel discussions dismissed the poor metric scores because the candidates "interviewed so well." Consequently, Big Tech hires averaged a 2.25 post-hire rating, whereas Startup hires averaged 4.67 (C101: 5, C141: 5, C151: 4).

#### 3. Funnel Inefficiency and Dropout Of 52 candidates reaching final stages, 10 withdrew (19.2%). A one-week take-home assignment imposes an asymmetric burden that disproportionately deters active, highly sought-after PMs (e.g., C117 and C138, both of whom withdrew despite scoring 4 on the behavioural round).

---

How Confident Can We Be?

• Sample Size Constraints ($N=9$): Our post-hire sample is small. Standard statistical models on 9 points are vulnerable to sample variance, and we cannot observe post-hire performance for rejected candidates (selection bias).
• Directional Certainty: While we should not rely on micro-variations, our qualitative and directional confidence is very high. The three struggling hires did not randomly underperform; they share an identical score footprint (TH 3–4, Pres 4, Beh 2, VPE 4). Similarly, zero candidates with a Behavioural score of 2 succeeded, and zero candidates with a Behavioural score of 4 struggled.
• Verdict: We have more than enough empirical and qualitative evidence to eliminate rounds that waste candidate time and bias hiring decisions.

---

Recommended Loop for Next Quarter

To hire four strong PMs next quarter, we must eliminate pedigree bias, reduce dropout, and center the evaluation on rigorous product execution.

``` [Stage 1] Recruiter Screen (30 min) │ [Stage 2] Hiring Manager Screen: Problem Framing & Track Record (45 min) │ [Stage 3] Virtual Onsite Loop (3 Rounds, Same Day) ├── 1. Product Execution & Metrics (60 min) ├── 2. Live Product Design & Problem-Solving (60 min — replaces take-home) └── 3. Engineering & Cross-Functional Partnership (45 min) │ [Stage 4] Structured Debrief & Decision Rule ```

#### Key Adjustments & Rubric Changes

1. Kill the Take-Home Deck and Presentation Round: * Replace with a 60-minute Live Product Design Case (e.g., scoping a feature for an ambiguous user problem). This assesses real-time product thinking, hypothesis generation, and adaptability without requiring 15 hours of slide deck preparation. 2. Elevate the Metrics Round to a Core Gate: * Retain and standardize the Product Execution & Metrics interview. Probe deeply on unit economics, input vs. output metrics, experimentation, and cross-functional friction. 3. Refactor the VP Engineering Round into "Engineering Partnership": * Change the prompt from "technical credibility" to cross-functional execution: how the candidate navigates technical trade-offs, works through debt vs. feature priorities, and writes crisp specs. Pair the VP Eng with an Engineering Manager or Staff Engineer using a standardized rubric. 4. Enforce Non-Negotiable Decision Rules: * Hard Stop on Execution: Any candidate scoring $<3$ on the Metrics/Execution round is an automatic "No Hire," regardless of deck polish or pedigree. Hiring managers cannot override this rule. * Blind Background Debrief: Panelists submit written scorecards before debriefing to prevent seniority or company pedigree from anchoring the room.

This restructured loop eliminates candidate dropouts, shortens time-to-hire, and directly targets the operational execution our PM team needs to hit our growth targets next quarter.

MEMORANDUM

TO: Elif Demir, VP of Product FROM: Director of Product DATE: October 24, 2023 SUBJECT: PM Interview Loop Analysis & Recommendations for Q1 Hiring

*

Elif, you are entirely right to flag this. Looking back at our last 18 months of hiring data, out of our 6 hired PMs who completed their first year, half (C107, C132, C148) are struggling with 12-month performance ratings of 2 (well below our company average of 3.4). Meanwhile, our top performers (C101, C141 with ratings of 5, and C149 with a rating of 4) represent the standard we actually need.

To prep for hiring four more PMs next quarter, I’ve analyzed the data for all 52 candidates who reached our final rounds. Here is what works, what is completely broken, how confident we can be, and how we should redesign the loop.

---

1. What Predicts Performance? (And What Doesn't)

By analyzing the correlation between interview stage scores and 12-month post-hire ratings for our 6 completed hires, we see stark contrasts in signal quality:

* Take-Home Strategy Deck (Zero to Negative Correlation): Our take-home exam is a massive waste of candidate time and offers zero predictive value. Our struggling hires (C107, C132, C148) averaged a 3.7 out of 4 on the take-home. Conversely, our top performers averaged a 2.7. High scores on a 1-week unsupervised deck merely signal that a candidate has free time, a supportive current employer, or an agency background—not that they can execute at Kestrel. * Take-Home Presentation Panel (Weak / Misleading Signal): Similar to the take-home itself, presentation scores do not align with success. C107 and C148 both scored 4s on their presentation and turned out to be 12-month "2" ratings. * VP Engineering Conversation (Inverted Signal): Surprisingly, the VP Engineering score shows a negative relationship with success. Candidates who scored high with Engineering often struggled on the job, while our top performers (C101, C141, C149) mixed moderate-to-low engineering scores (2 to 3) with high marks elsewhere. While technical credibility matters, our current technical screen appears to favor academic/theoretical answers over pragmatic product-engineering partnership. * Behavioural Metric Round (Our Only Real Signal): The behavioural round ("Tell me about a metric you moved") is the only component in our current loop that correlates with on-the-job success. * Our three struggling hires averaged 2.0 on this round. * Our three successful hires averaged 4.0 on this round. This is our sole bright spot. It tests past execution, ownership, and learning—things that actually matter day-to-day at Kestrel.

---

2. Statistical Confidence

We must temper our confidence. We have only 6 data points of complete hire-and-tenure data. While the directional trend on the behavioural round is unmistakable (+2.0 delta between high and low performers), running statistical significance tests on $n=6$ is risky.

However, we can look at broader hiring funnel data across all 52 candidates to understand drop-offs. Notably, 13 candidates withdrew from the process, heavily skewed toward candidates who scored well on the take-home (e.g., C100, C113, C142, C147 all scored 4s on the take-home and walked away). Our heavy, one-week take-home is actively driving away top-tier passive talent who refuse to do free consulting work for a week, leaving us to select from a pool of people willing to endure it.

---

3. The Proposed Loop for Next Quarter

To scale our team to four new PMs without repeating past mistakes, we need to strip away vanity metrics (the take-home deck) and lean heavily into what actually predicts success: rigorous behavioural assessment of past execution, combined with a real-time collaborative product session.

Here is the 4-stage loop I propose we deploy immediately:

1. Recruiter Screen (30 mins): Standard fit, compensation alignment, and basic background check.
2. Product Craft & Execution Interview (45 mins - Live Video): Replaces the take-home. Instead of a week-long deck, we give the candidate a real Kestrel product challenge 48 hours in advance (requiring no more than 2 hours of prep) and spend the interview deep-diving into their hypotheses, prioritization trade-offs, and customer empathy.
3. Behavioural & Metric Ownership Round (45 mins): Double down on what works. Retain our behavioural round focusing on metrics moved, but expand the rubric to explicitly test for accountability, cross-functional conflict resolution, and handling failure.
4. Product-Engineering Partnership Round (45 mins): Refocused Engineering screen. Instead of "technical credibility" evaluated in a vacuum, this is a joint session with the VP of Engineering or a Senior Tech Lead to evaluate how the PM collaborates with engineering to ship product, scope down MVPs, and manage technical debt.

Decision Gate: Drop the consensus panel meeting where the loudest voice wins. The hiring manager makes the final call, but only if the candidate scores a predefined threshold on the Behavioural and Product Craft rounds.

Summary

By eliminating the 1-week take-home, we will stop scaring away top-tier talent, shorten our time-to-hire, and refocus our interview panel on real-world execution rather than polished slide decks. Let me know if you are aligned, and I will draft the new interview rubrics by Friday.