Hire a PM

MEMORANDUM

TO: Elif Demir, VP of Product FROM: Director of Product DATE: October 24, 2023 SUBJECT: PM Interview Loop Analysis & Recommendations for Q1 Hiring

*

Elif, you are entirely right to flag this. Looking back at our last 18 months of hiring data, out of our 6 hired PMs who completed their first year, half (C107, C132, C148) are struggling with 12-month performance ratings of 2 (well below our company average of 3.4). Meanwhile, our top performers (C101, C141 with ratings of 5, and C149 with a rating of 4) represent the standard we actually need.

To prep for hiring four more PMs next quarter, I’ve analyzed the data for all 52 candidates who reached our final rounds. Here is what works, what is completely broken, how confident we can be, and how we should redesign the loop.

---

1. What Predicts Performance? (And What Doesn't)

By analyzing the correlation between interview stage scores and 12-month post-hire ratings for our 6 completed hires, we see stark contrasts in signal quality:

* Take-Home Strategy Deck (Zero to Negative Correlation): Our take-home exam is a massive waste of candidate time and offers zero predictive value. Our struggling hires (C107, C132, C148) averaged a 3.7 out of 4 on the take-home. Conversely, our top performers averaged a 2.7. High scores on a 1-week unsupervised deck merely signal that a candidate has free time, a supportive current employer, or an agency background—not that they can execute at Kestrel. * Take-Home Presentation Panel (Weak / Misleading Signal): Similar to the take-home itself, presentation scores do not align with success. C107 and C148 both scored 4s on their presentation and turned out to be 12-month "2" ratings. * VP Engineering Conversation (Inverted Signal): Surprisingly, the VP Engineering score shows a negative relationship with success. Candidates who scored high with Engineering often struggled on the job, while our top performers (C101, C141, C149) mixed moderate-to-low engineering scores (2 to 3) with high marks elsewhere. While technical credibility matters, our current technical screen appears to favor academic/theoretical answers over pragmatic product-engineering partnership. * Behavioural Metric Round (Our Only Real Signal): The behavioural round ("Tell me about a metric you moved") is the only component in our current loop that correlates with on-the-job success. * Our three struggling hires averaged 2.0 on this round. * Our three successful hires averaged 4.0 on this round. This is our sole bright spot. It tests past execution, ownership, and learning—things that actually matter day-to-day at Kestrel.

---

2. Statistical Confidence

We must temper our confidence. We have only 6 data points of complete hire-and-tenure data. While the directional trend on the behavioural round is unmistakable (+2.0 delta between high and low performers), running statistical significance tests on $n=6$ is risky.

However, we can look at broader hiring funnel data across all 52 candidates to understand drop-offs. Notably, 13 candidates withdrew from the process, heavily skewed toward candidates who scored well on the take-home (e.g., C100, C113, C142, C147 all scored 4s on the take-home and walked away). Our heavy, one-week take-home is actively driving away top-tier passive talent who refuse to do free consulting work for a week, leaving us to select from a pool of people willing to endure it.

---

3. The Proposed Loop for Next Quarter

To scale our team to four new PMs without repeating past mistakes, we need to strip away vanity metrics (the take-home deck) and lean heavily into what actually predicts success: rigorous behavioural assessment of past execution, combined with a real-time collaborative product session.

Here is the 4-stage loop I propose we deploy immediately:

1. Recruiter Screen (30 mins): Standard fit, compensation alignment, and basic background check.
2. Product Craft & Execution Interview (45 mins - Live Video): Replaces the take-home. Instead of a week-long deck, we give the candidate a real Kestrel product challenge 48 hours in advance (requiring no more than 2 hours of prep) and spend the interview deep-diving into their hypotheses, prioritization trade-offs, and customer empathy.
3. Behavioural & Metric Ownership Round (45 mins): Double down on what works. Retain our behavioural round focusing on metrics moved, but expand the rubric to explicitly test for accountability, cross-functional conflict resolution, and handling failure.
4. Product-Engineering Partnership Round (45 mins): Refocused Engineering screen. Instead of "technical credibility" evaluated in a vacuum, this is a joint session with the VP of Engineering or a Senior Tech Lead to evaluate how the PM collaborates with engineering to ship product, scope down MVPs, and manage technical debt.

Decision Gate: Drop the consensus panel meeting where the loudest voice wins. The hiring manager makes the final call, but only if the candidate scores a predefined threshold on the Behavioural and Product Craft rounds.

Summary

By eliminating the 1-week take-home, we will stop scaring away top-tier talent, shorten our time-to-hire, and refocus our interview panel on real-world execution rather than polished slide decks. Let me know if you are aligned, and I will draft the new interview rubrics by Friday.

To: Elif Demir, VP Product From: Director of Product Re: What our PM loop predicts, and the loop I'd run next quarter

Bottom line

The behavioural metric round is the only part of the loop that tracks how hires perform. The presentation and VP Engineering rounds seem to drive hiring decisions without predicting performance. All three hires who struggled scored 4 on VP Engineering and 2 on the metric round. I'd make the metric round the centre of the loop and stop weighting the others as heavily. The sample is small, so this is a strong hypothesis to test, not a settled answer.

What the data covers

• 52 candidates reached the final rounds over 18 months: 9 hired, 33 rejected, 10 withdrew.
• Only the 9 hires have 12-month ratings. We can't tell whether our rejections were right, so we can't measure false negatives.
• We don't record withdrawal stage or reason, or which interviewer gave each score, so we can't check consistency across panels.

What the data shows

Specifics:

• Metric round: Hires who scored 2 all rated 2 at 12 months (3 of 3). Hires who scored 3 or 4 averaged 4.0. Its average gap between hired and rejected candidates is the smallest of the four, so the panel leaned on the other rounds more.
• The three struggling hires (rated 2 after 12 months) all scored 4 on VP Engineering and 4 on presentation, and scored 3–4 on the take-home. They looked strong everywhere except the metric round, which was the signal that predicted their problems.
• VP Engineering: All four hires who scored 4 here were rated 2 or 3 at 12 months.
• Withdrawals: 8 of the 10 candidates who withdrew scored 3 or higher on the metric round. We are losing strong candidates, and we don't know where or why.
• Background (hypothesis only): Big-tech hires (4) averaged 2.25 at 12 months; startup hires (3) averaged 4.7. The numbers are far too small to act on, but we should track it.

How confident we should be

• Metric round as the best predictor: moderate. A correlation of about 0.9 across nine hires is striking, and it fits what PM work involves: diagnosing a metric and deciding what to do about it. But with nine people, one or two different ratings would move the result a lot.
• Negative readings for presentation and VP Engineering: low. We only see hires, so the range is restricted, and the 1–4 scale gives few distinct values. The defensible claim is that these rounds did not predict success. I wouldn't claim they predict failure.
• Take-home as a screen-out: low to moderate. Zero hires among ten candidates who scored 1 is consistent with the take-home filtering out weak work, but that's a small group.
• Anything about rejected candidates: no confidence. We have no data on them.

The loop I'd run next quarter

1. Recruiter screen: No change.
2. Take-home: Keep it as a gate. Candidates who score 1 don't advance without a written override. Cap the time we ask for, since we're losing strong candidates and may be losing some to the length of the process.
3. Metric round (anchor): Rewrite as a structured 60-minute session with two interviewers. Candidates diagnose a real metric movement from a dashboard, propose causes, and say what they'd measure next. Score against a written rubric on 1–4 scales. A score of 3 or higher is required to hire. A score of 2 or lower is a no-hire unless you sign a written override.
4. Presentation: Fold it into the metric round as a communication check, so candidates spend less time on it, and don't weight it separately in the decision.
5. VP Engineering conversation: Keep it for a pass/fail check on serious collaboration red flags. Don't score it or count it in the decision, given how it has tracked performance.
6. Decision process: Every interviewer submits written scores before the panel meets, so no one anchors on the first opinion. The hiring manager still decides, but must write down why if they hire against the metric-round rule or reject someone who met it.
7. Data: Record the interviewer for every score, withdrawal stage and reason, and 6- and 12-month ratings for every hire. Keep scoring the old rounds next quarter even though they won't be weighted, so we keep building the dataset.

What I need from you

• Approval for the metric-round rubric and the override rule.
• Agreement on a review date. Four hires next quarter won't settle this. I'd treat the next two quarters as calibration, with a 6-month review of the four new hires and a full review once we have about twelve hires on the new loop.
• A decision on whether to pay for a small outside check on our rubric, such as having a PM coach double-score a sample of recordings.

To: Elif Demir, VP Product From: Director of Product Subject: PM interview loop: what the evidence supports and what to change next quarter

Recommendation

For the next four hires, make demonstrated product judgment, metric ownership, and cross-functional execution the center of the loop. Replace the week-long strategy deck and separate presentation with a short, standardized work sample; retain and strengthen the behavioral metric interview; replace the unstructured “technical credibility” conversation with a scored collaboration exercise.

These are evidence-informed changes, not a validated hiring formula. Our strongest signal comes from just nine hires.

What appears to predict performance

We have 52 finalists: nine hired, 33 rejected, and ten withdrawn. Only the nine hires have performance outcomes. Their average 12-month rating is 3.33, versus the company PM average of 3.4. Three received a 2; none received a 1. The similar averages do not negate the three struggling hires, but neither establish that the entire loop is failing.

Among hires:

Descriptive Pearson correlations on ordinal scores, based on nine hires—not validated predictive estimates.

The behavioral round shows a particularly clear pattern: the three hires scoring 2 all subsequently received performance ratings of 2; those scoring 3 averaged 3.33; those scoring 4 averaged 4.67. It plausibly tests something relevant to the job: whether candidates can explain their own contribution, reason about outcomes, and learn from results.

Conversely, three hires—C107, C132, and C148—received presentation and VP Engineering scores of 4, but later performance ratings of 2. C101 scored 2 in both those rounds and subsequently received a 5. We should stop treating polish or this version of technical credibility as sufficient evidence of PM effectiveness.

We have no scored evidence for the recruiter screen, and cannot assess the independent value of panel discussion or hiring-manager judgment from these files.

How confident should we be?

Confident about the observed pattern; low confidence that it will generalize.

• Nine outcomes are too few to estimate stable predictive relationships or optimize weights and cutoffs. A few cases could materially change the results.
• Selection limits the analysis. We observe performance only for people our existing process hired. Rejected and withdrawn candidates are not failed PMs; their missing outcomes prevent us from measuring false negatives or assessing validity across all finalists.
• Background and scores are entangled. The four big-tech hires averaged 2.25 in performance; the three startup hires averaged 4.67. Background also overlaps with interview-score patterns. We cannot isolate interview effects from background, assignments, support, or other differences—and should not turn this into a background preference.
• The rounds are not independent. The deck and presentation share content, while ratings may reflect overlapping impressions. We cannot establish which round adds value beyond another.
• Performance ratings are an imperfect criterion. We lack role scope, manager, onboarding, and rating-calibration information. The company average is not a matched comparison group.

Negative correlations therefore do not mean we should prefer poor presenters or candidates with low technical scores. They mean the current versions have not demonstrated useful positive signal.

The loop I would run next quarter

1. Structured recruiter screen. Confirm role expectations, relevant experience, motivation, and logistics using consistent questions. Explain the process and preparation requirements upfront.

2. Standardized product work sample, replacing the deck and presentation. Use a fictional product brief and small dataset, with a roughly 60–90-minute total candidate commitment. Ask candidates to diagnose a problem, choose an outcome metric, prioritize options, and explain tradeoffs. Introduce a new constraint during discussion to test adaptation, not slide polish. Use equivalent prompts, consistent preparation conditions, and accommodations where needed.

3. Structured behavioral metric interview. Keep the strongest observed signal. Probe the baseline, candidate’s personal actions, alternatives considered, attribution versus correlation, unintended effects, and lessons learned. Score ownership, analytical judgment, and learning separately using behaviorally anchored rubrics.

4. Structured engineering collaboration interview. Replace the standalone credibility conversation with a realistic scenario involving delivery constraints, instrumentation, and competing priorities. An engineering interviewer scores clarification, tradeoffs, and collaboration—not jargon, pedigree, or conversational rapport.

5. Evidence-led decision. Set role-specific competencies and minimum requirements before interviewing. Interviewers submit independent scores and concrete evidence before discussion. The hiring manager documents the decision and any override. Presentation polish must not compensate for missing evidence on critical competencies. Do not derive mechanical cutoffs from these nine hires.

How we will know whether this is better

Preserve round-level scores, interviewer identities, decisions, overrides, withdrawals, and candidate feedback. Define consistent, role-adjusted performance measures at 3, 6, and 12 months; where feasible, have reviewers assess performance without seeing interview scores.

Review candidate burden and scoring consistency immediately. Treat outcomes from the next four hires as additional evidence—not proof of success. Build a larger cohort before claiming predictive validity or tuning numerical weights.