Hire a PM

MEMO

To: Elif Demir, VP Product From: Director of Product Subject: Fixing the PM Interview Loop Before Next Quarter’s Hiring

Elif,

I analyzed the data from the 52 PM candidates who reached our final rounds over the last 18 months—specifically looking at the 9 we hired, the 33 we rejected, and the 10 who withdrew.

The headline is clear: We are over-indexing on polished presentation skills and Big Tech pedigrees, while actively ignoring our single best predictor of on-the-job success.

Here is what the data shows, how confident we can be in these findings, and the revised interview loop I recommend for next quarter.

---

1. What Predicts Success (and What Doesn’t)

Of the 9 PMs we hired, their average 12-month performance rating is 3.33 (slightly below the company PM average of 3.4). Four hires met or exceeded expectations (ratings of 4 and 5), two were below average (rating of 3), and three struggled significantly (rating of 2).

When we compare their interview scores to their 12-month performance ratings, the patterns are striking:

• The Behavioural ("Metric You Moved") Round is our only strong positive predictor ($r = +0.87$).
• Every hired PM who scored a 4 in this round (C101, C141, C149) went on to become a top performer (ratings of 5, 5, and 4).
• Every hired PM who scored a 2 in this round (C107, C132, C148) struggled on the job (all three received a rating of 2).
• Despite this, we treated this round as optional: we hired three candidates who scored a 2 here because we were dazzled by their decks and pedigrees. Conversely, we rejected 5 candidates who scored a 4 in this round.
• The Take-Home and Presentation are negatively correlated with performance ($r = -0.57$ and $r = -0.84$).
• Our three struggling hires (all rated 2) scored 4s on their presentations, and two of them scored 4s on the take-home.
• Meanwhile, our top performers (rated 4 and 5) scored 2s and 3s on the take-home and presentation.
• A one-week strategy deck tests a candidate’s ability to craft polished slides in a vacuum with unlimited time—a skill that does not translate to shipping product at Kestrel.
• The VP Engineering Round is heavily biased toward Big Tech and negatively correlates with success ($r = -0.68$).
• Across all 52 candidates, Big Tech PMs averaged a 3.44 with the VP of Engineering, while Startup, Agency, and Consulting PMs averaged 2.16, 1.70, and 1.93, respectively.
• Every single Big Tech PM we hired scored a 4 in this round, yet they averaged a 2.25 performance rating at 12 months. Meanwhile, top performers from Startup and Agency backgrounds (C101, C149) scored 2s in this round. The unstructured conversation is indexing on Big Tech jargon rather than the scrappy technical collaboration we actually need.
• The Process Causes High Candidate Drop-Off.
• 10 of our 52 finalists (19%) withdrew. Notably, 3 of the 8 candidates who scored a 4 on our best predictor (the Behavioural round) withdrew. A one-week take-home followed by a 4-stage loop is too slow and burdensome for top-tier, in-demand talent.

---

2. How Confident Can We Be?

We should treat these findings as directionally vital, though statistically limited: 1. Small sample size ($n = 9$): Nine hires is a small dataset. A correlation of $+0.87$ on 9 data points is remarkably clean, but we shouldn't treat the exact coefficients as immutable laws. 2. Survivorship and selection bias: We don't know how the 33 rejected candidates would have performed. Furthermore, the negative correlation on the presentation and VP Eng rounds is partially an artifact of our hiring decisions: when candidates had a weak Behavioural score (2), we only hired them because they had 4s in the other rounds. 3. The "Big Tech" confounding variable: All three struggling hires came from Big Tech, while all four top performers came from Startups or Agencies. Big Tech PMs aced the deck and VP Eng rounds, while Startup/Agency PMs aced the metric-execution round.

Even with $n=9$, the qualitative logic backs the data: past execution on moving metrics predicts future execution at Kestrel; making pretty slides and name-dropping Big Tech infrastructure does not.

---

3. The Proposed Loop for Next Quarter

To hire four great PMs next quarter—and stop losing top candidates to withdrawal—we need to cut the time-to-hire, eliminate the take-home deck, and re-center the loop on execution and structured collaboration.

1. Recruiter Screen (30 mins) * No change, except: Source more heavily from Startup and Agency backgrounds, and screen out Big Tech candidates who haven't owned end-to-end execution.

2. Behavioural Execution Deep-Dive: "A Metric You Moved" (60 mins) * Change: Move this to the very first round after the recruiter screen and make a score of 3 or 4 a hard requirement to advance. * Why: It is our strongest predictor of success. Moving it up filters out "talkers" early and saves panel bandwidth.

3. Live Product Case & Prioritization Workshop (60 mins) * Change: Kill the 1-week take-home and panel presentation. Replace them with a zero-prep, live working session with two PMs using a real, messy Kestrel problem. * Why: The take-home selects for slide-making and causes nearly 20% of finalists to withdraw. A live case tests how candidates structure ambiguity, incorporate feedback, and think on their feet—without requiring 15 hours of unpaid homework.

4. Structured Engineering Collaboration Round (45 mins) * Change: Replace the unstructured VP Engineering "technical credibility" chat with a structured rubric led by a Senior Engineering Manager or Tech Lead. Prompt: "Tell me about a time you had to make a tough scope vs. technical-debt tradeoff with your engineering team." * Why: Eliminates the Big Tech halo effect and tests how the PM actually partners with engineers day-to-day.

5. Hiring Decision & Bar-Raising * Change: Replace open-ended panel discussion with independent score submission prior to debrief. Institute a hard rule: No candidate can be hired with a score below 3 on the Behavioural Execution round, regardless of how well they interview elsewhere.

If you're aligned, I will work with Recruiting and Engineering this week to build the new rubrics and live-case prompts so we are ready for next quarter's reqs.

MEMORANDUM

TO: Elif Demir, VP Product FROM: Director of Product DATE: October 24, 2023 SUBJECT: Analysis of PM Hiring Loop Performance and Proposed Redesign for Q1 Hiring

*

Executive Summary

Over the past 18 months, we evaluated 52 PM finalists and hired 9. Three hires struggled significantly (rated 2/5 after 12 months, well below our 3.4 company average).

Our data reveals a clear and counter-intuitive pattern: 1. The only reliable predictor of success is the Behavioural Metric Round ("Tell me about a metric you moved"). 2. The Take-Home Deck, Presentation Round, and VP Engineering Conversation are negatively correlated with performance. They reward presentation polish, shared jargon, and Big Tech pedigree over actual execution. 3. The 1-week take-home deck is causing talent leakage: 19% of final-round candidates withdrew, including top performers. 4. All three struggling hires share the exact same profile: Big Tech background, high presentation and VP Eng scores (4/4), and poor behavioural metric scores (2/4). We over-indexed on polish and excused weak operational substance.

Below is an analysis of our interview signals, our statistical confidence, and a redesigned loop for next quarter.

---

What the Data Tells Us

#### 1. What Predicts Performance: The Behavioural Metric Round Candidate performance on the behavioural round correlates almost perfectly with 12-month post-hire ratings: * Score 4 (3 hires): Average rating 4.67 (two 5s, one 4). * Score 3 (3 hires): Average rating 3.33 (one 4, two 3s). * Score 2 (3 hires): Average rating 2.00 (all three are our struggling hires: C107, C132, C148).

PMs who could drill into the mechanics of moving a metric—causality, counter-metrics, execution trade-offs, and failure lessons—excelled. Those who could not struggled immediately in our environment.

#### 2. What Misleads: Presentation, Take-Home, and VP Engineering Counter-intuitively, scoring at the top of these three stages predicted lower job performance: * Presentation Score: Hires scoring 4/4 averaged a 2.25 rating; hires scoring 2 or 3 averaged 4.20. * VP Engineering Score: Hires scoring 4/4 averaged 2.25; hires scoring 2 or 3 averaged 4.20. * Take-Home Score: Hires scoring 4/4 averaged 2.50; hires scoring 2 or 3 averaged 4.00.

Why are these rounds misleading us? * Deck Polish $\neq$ Product Sense: The take-home strategy deck and presentation test slide design and storytelling. Candidates with Big Tech or consulting backgrounds excel at creating slick artifacts, which creates an "executive presence" halo effect. * Flawed Engineering Rubric: "Technical credibility" currently measures whether a PM speaks Big Tech architectural jargon, not whether they partner effectively with engineers on scoping, trade-offs, and technical debt. * The "Big Tech Halo": All 4 Big Tech hires received top VP Eng scores (4/4), but 3 received failing behavioural scores (2/4). Panel discussions dismissed the poor metric scores because the candidates "interviewed so well." Consequently, Big Tech hires averaged a 2.25 post-hire rating, whereas Startup hires averaged 4.67 (C101: 5, C141: 5, C151: 4).

#### 3. Funnel Inefficiency and Dropout Of 52 candidates reaching final stages, 10 withdrew (19.2%). A one-week take-home assignment imposes an asymmetric burden that disproportionately deters active, highly sought-after PMs (e.g., C117 and C138, both of whom withdrew despite scoring 4 on the behavioural round).

---

How Confident Can We Be?

• Sample Size Constraints ($N=9$): Our post-hire sample is small. Standard statistical models on 9 points are vulnerable to sample variance, and we cannot observe post-hire performance for rejected candidates (selection bias).
• Directional Certainty: While we should not rely on micro-variations, our qualitative and directional confidence is very high. The three struggling hires did not randomly underperform; they share an identical score footprint (TH 3–4, Pres 4, Beh 2, VPE 4). Similarly, zero candidates with a Behavioural score of 2 succeeded, and zero candidates with a Behavioural score of 4 struggled.
• Verdict: We have more than enough empirical and qualitative evidence to eliminate rounds that waste candidate time and bias hiring decisions.

---

Recommended Loop for Next Quarter

To hire four strong PMs next quarter, we must eliminate pedigree bias, reduce dropout, and center the evaluation on rigorous product execution.

``` [Stage 1] Recruiter Screen (30 min) │ [Stage 2] Hiring Manager Screen: Problem Framing & Track Record (45 min) │ [Stage 3] Virtual Onsite Loop (3 Rounds, Same Day) ├── 1. Product Execution & Metrics (60 min) ├── 2. Live Product Design & Problem-Solving (60 min — replaces take-home) └── 3. Engineering & Cross-Functional Partnership (45 min) │ [Stage 4] Structured Debrief & Decision Rule ```

#### Key Adjustments & Rubric Changes

1. Kill the Take-Home Deck and Presentation Round: * Replace with a 60-minute Live Product Design Case (e.g., scoping a feature for an ambiguous user problem). This assesses real-time product thinking, hypothesis generation, and adaptability without requiring 15 hours of slide deck preparation. 2. Elevate the Metrics Round to a Core Gate: * Retain and standardize the Product Execution & Metrics interview. Probe deeply on unit economics, input vs. output metrics, experimentation, and cross-functional friction. 3. Refactor the VP Engineering Round into "Engineering Partnership": * Change the prompt from "technical credibility" to cross-functional execution: how the candidate navigates technical trade-offs, works through debt vs. feature priorities, and writes crisp specs. Pair the VP Eng with an Engineering Manager or Staff Engineer using a standardized rubric. 4. Enforce Non-Negotiable Decision Rules: * Hard Stop on Execution: Any candidate scoring $<3$ on the Metrics/Execution round is an automatic "No Hire," regardless of deck polish or pedigree. Hiring managers cannot override this rule. * Blind Background Debrief: Panelists submit written scorecards before debriefing to prevent seniority or company pedigree from anchoring the room.

This restructured loop eliminates candidate dropouts, shortens time-to-hire, and directly targets the operational execution our PM team needs to hit our growth targets next quarter.

To: Elif Demir, VP Product From: Director of Product Subject: PM interview loop: what the evidence supports and what to change next quarter

Recommendation

For the next four hires, make demonstrated product judgment, metric ownership, and cross-functional execution the center of the loop. Replace the week-long strategy deck and separate presentation with a short, standardized work sample; retain and strengthen the behavioral metric interview; replace the unstructured “technical credibility” conversation with a scored collaboration exercise.

These are evidence-informed changes, not a validated hiring formula. Our strongest signal comes from just nine hires.

What appears to predict performance

We have 52 finalists: nine hired, 33 rejected, and ten withdrawn. Only the nine hires have performance outcomes. Their average 12-month rating is 3.33, versus the company PM average of 3.4. Three received a 2; none received a 1. The similar averages do not negate the three struggling hires, but neither establish that the entire loop is failing.

Among hires:

Descriptive Pearson correlations on ordinal scores, based on nine hires—not validated predictive estimates.

The behavioral round shows a particularly clear pattern: the three hires scoring 2 all subsequently received performance ratings of 2; those scoring 3 averaged 3.33; those scoring 4 averaged 4.67. It plausibly tests something relevant to the job: whether candidates can explain their own contribution, reason about outcomes, and learn from results.

Conversely, three hires—C107, C132, and C148—received presentation and VP Engineering scores of 4, but later performance ratings of 2. C101 scored 2 in both those rounds and subsequently received a 5. We should stop treating polish or this version of technical credibility as sufficient evidence of PM effectiveness.

We have no scored evidence for the recruiter screen, and cannot assess the independent value of panel discussion or hiring-manager judgment from these files.

How confident should we be?

Confident about the observed pattern; low confidence that it will generalize.

• Nine outcomes are too few to estimate stable predictive relationships or optimize weights and cutoffs. A few cases could materially change the results.
• Selection limits the analysis. We observe performance only for people our existing process hired. Rejected and withdrawn candidates are not failed PMs; their missing outcomes prevent us from measuring false negatives or assessing validity across all finalists.
• Background and scores are entangled. The four big-tech hires averaged 2.25 in performance; the three startup hires averaged 4.67. Background also overlaps with interview-score patterns. We cannot isolate interview effects from background, assignments, support, or other differences—and should not turn this into a background preference.
• The rounds are not independent. The deck and presentation share content, while ratings may reflect overlapping impressions. We cannot establish which round adds value beyond another.
• Performance ratings are an imperfect criterion. We lack role scope, manager, onboarding, and rating-calibration information. The company average is not a matched comparison group.

Negative correlations therefore do not mean we should prefer poor presenters or candidates with low technical scores. They mean the current versions have not demonstrated useful positive signal.

The loop I would run next quarter

1. Structured recruiter screen. Confirm role expectations, relevant experience, motivation, and logistics using consistent questions. Explain the process and preparation requirements upfront.

2. Standardized product work sample, replacing the deck and presentation. Use a fictional product brief and small dataset, with a roughly 60–90-minute total candidate commitment. Ask candidates to diagnose a problem, choose an outcome metric, prioritize options, and explain tradeoffs. Introduce a new constraint during discussion to test adaptation, not slide polish. Use equivalent prompts, consistent preparation conditions, and accommodations where needed.

3. Structured behavioral metric interview. Keep the strongest observed signal. Probe the baseline, candidate’s personal actions, alternatives considered, attribution versus correlation, unintended effects, and lessons learned. Score ownership, analytical judgment, and learning separately using behaviorally anchored rubrics.

4. Structured engineering collaboration interview. Replace the standalone credibility conversation with a realistic scenario involving delivery constraints, instrumentation, and competing priorities. An engineering interviewer scores clarification, tradeoffs, and collaboration—not jargon, pedigree, or conversational rapport.

5. Evidence-led decision. Set role-specific competencies and minimum requirements before interviewing. Interviewers submit independent scores and concrete evidence before discussion. The hiring manager documents the decision and any override. Presentation polish must not compensate for missing evidence on critical competencies. Do not derive mechanical cutoffs from these nine hires.

How we will know whether this is better

Preserve round-level scores, interviewer identities, decisions, overrides, withdrawals, and candidate feedback. Define consistent, role-adjusted performance measures at 3, 6, and 12 months; where feasible, have reviewers assess performance without seeing interview scores.

Review candidate burden and scoring consistency immediately. Treat outcomes from the next four hires as additional evidence—not proof of success. Build a larger cohort before claiming predictive validity or tuning numerical weights.