Hire a PM

To: Elif Demir, VP Product From: Director of Product Subject: PM interview loop: what the evidence supports and what to change next quarter

Recommendation

For the next four hires, make demonstrated product judgment, metric ownership, and cross-functional execution the center of the loop. Replace the week-long strategy deck and separate presentation with a short, standardized work sample; retain and strengthen the behavioral metric interview; replace the unstructured “technical credibility” conversation with a scored collaboration exercise.

These are evidence-informed changes, not a validated hiring formula. Our strongest signal comes from just nine hires.

What appears to predict performance

We have 52 finalists: nine hired, 33 rejected, and ten withdrawn. Only the nine hires have performance outcomes. Their average 12-month rating is 3.33, versus the company PM average of 3.4. Three received a 2; none received a 1. The similar averages do not negate the three struggling hires, but neither establish that the entire loop is failing.

Among hires:

Descriptive Pearson correlations on ordinal scores, based on nine hires—not validated predictive estimates.

The behavioral round shows a particularly clear pattern: the three hires scoring 2 all subsequently received performance ratings of 2; those scoring 3 averaged 3.33; those scoring 4 averaged 4.67. It plausibly tests something relevant to the job: whether candidates can explain their own contribution, reason about outcomes, and learn from results.

Conversely, three hires—C107, C132, and C148—received presentation and VP Engineering scores of 4, but later performance ratings of 2. C101 scored 2 in both those rounds and subsequently received a 5. We should stop treating polish or this version of technical credibility as sufficient evidence of PM effectiveness.

We have no scored evidence for the recruiter screen, and cannot assess the independent value of panel discussion or hiring-manager judgment from these files.

How confident should we be?

Confident about the observed pattern; low confidence that it will generalize.

• Nine outcomes are too few to estimate stable predictive relationships or optimize weights and cutoffs. A few cases could materially change the results.
• Selection limits the analysis. We observe performance only for people our existing process hired. Rejected and withdrawn candidates are not failed PMs; their missing outcomes prevent us from measuring false negatives or assessing validity across all finalists.
• Background and scores are entangled. The four big-tech hires averaged 2.25 in performance; the three startup hires averaged 4.67. Background also overlaps with interview-score patterns. We cannot isolate interview effects from background, assignments, support, or other differences—and should not turn this into a background preference.
• The rounds are not independent. The deck and presentation share content, while ratings may reflect overlapping impressions. We cannot establish which round adds value beyond another.
• Performance ratings are an imperfect criterion. We lack role scope, manager, onboarding, and rating-calibration information. The company average is not a matched comparison group.

Negative correlations therefore do not mean we should prefer poor presenters or candidates with low technical scores. They mean the current versions have not demonstrated useful positive signal.

The loop I would run next quarter

1. Structured recruiter screen. Confirm role expectations, relevant experience, motivation, and logistics using consistent questions. Explain the process and preparation requirements upfront.

2. Standardized product work sample, replacing the deck and presentation. Use a fictional product brief and small dataset, with a roughly 60–90-minute total candidate commitment. Ask candidates to diagnose a problem, choose an outcome metric, prioritize options, and explain tradeoffs. Introduce a new constraint during discussion to test adaptation, not slide polish. Use equivalent prompts, consistent preparation conditions, and accommodations where needed.

3. Structured behavioral metric interview. Keep the strongest observed signal. Probe the baseline, candidate’s personal actions, alternatives considered, attribution versus correlation, unintended effects, and lessons learned. Score ownership, analytical judgment, and learning separately using behaviorally anchored rubrics.

4. Structured engineering collaboration interview. Replace the standalone credibility conversation with a realistic scenario involving delivery constraints, instrumentation, and competing priorities. An engineering interviewer scores clarification, tradeoffs, and collaboration—not jargon, pedigree, or conversational rapport.

5. Evidence-led decision. Set role-specific competencies and minimum requirements before interviewing. Interviewers submit independent scores and concrete evidence before discussion. The hiring manager documents the decision and any override. Presentation polish must not compensate for missing evidence on critical competencies. Do not derive mechanical cutoffs from these nine hires.

How we will know whether this is better

Preserve round-level scores, interviewer identities, decisions, overrides, withdrawals, and candidate feedback. Define consistent, role-adjusted performance measures at 3, 6, and 12 months; where feasible, have reviewers assess performance without seeing interview scores.

Review candidate burden and scoring consistency immediately. Treat outcomes from the next four hires as additional evidence—not proof of success. Build a larger cohort before claiming predictive validity or tuning numerical weights.

To: Elif Demir, VP Product From: Director of Product Subject: PM interview loop: what the evidence supports and what to change

Recommendation

For next quarter’s four hires, retain and standardise the behavioural metric round; replace the week-long strategy deck and presentation with a bounded, realistic work sample; and replace “technical credibility” with a structured engineering-partnership interview.

The evidence supports changing what we reward—not claiming we have discovered a validated hiring formula.

What the data says

We have 52 finalists: nine hired, 33 rejected and ten withdrawn. Only the nine hires have performance outcomes. Their average 12-month rating is 3.33, close to the company PM average of 3.4. Three received a 2; none received a 1. That average masks the individual misses Elif is concerned about, but does not establish that the loop underperforms an alternative.

Among those nine hires:

Descriptive Pearson correlations on ordinal scores; not validated predictive estimates.

The behavioural pattern is particularly clear: the three hires scoring 2 all received performance ratings of 2; those scoring 3 averaged 3.33; those scoring 4 averaged 4.67. Probing actual ownership, measurement and learning appears promising.

Conversely, all three hires rated 2 had scored 4 on both presentation and VP Engineering. Two also scored 4 on the take-home. These rounds clearly failed to screen out our weakest performers. They may reward polish, familiarity or pedigree rather than effective PM work—but the data does not establish that mechanism.

We cannot evaluate the recruiter screen: it has neither scores nor data on candidates eliminated before finals. Nor can we isolate the value of panel discussion or hiring-manager judgment without decision rationales.

How confident should we be?

Enough to change the process; not enough to treat these correlations as durable prediction.

There are four important limitations:

• Nine outcomes are very few. One or two different hires could materially change the estimates. We cannot credibly estimate each round’s independent contribution.
• We observe performance only for selected candidates. Hiring already depended on these interviews. Conditioning on selection can distort associations; we do not know how rejected or withdrawn candidates would have performed.
• Background and scores are entangled. All four big-tech hires received VP Engineering scores of 4 and averaged 2.25 in performance; the three startup hires averaged 4.67. That does not establish a background effect—or justify preferring startup candidates. It prevents us from separating interview signal, background and role fit.
• Performance ratings are imperfect outcomes. We lack information on manager calibration, assignment difficulty, onboarding and opportunities to deliver results.

Accordingly, I would prioritise the behavioural round, but would not reverse-score presentations or engineering interviews, adopt background preferences, or derive numerical weights from nine people. The negative correlations show that those scores were not useful positive signals here; they do not prove the underlying skills are unimportant.

The loop I would run next quarter

1. Structured recruiter screen — 25 minutes. Check role requirements, relevant scope, motivation and practical constraints against a common checklist. Do not use employer prestige as a proxy for capability. Explain the process and time commitment.

2. Realistic work sample — 75 minutes, replacing both deck and presentation. Give every candidate the same bounded, fictional product problem, customer evidence, metric definitions and engineering constraints. Ask them to diagnose the problem, prioritise a response, define success and revise their recommendation when new evidence arrives. No slide deck or advance homework.

Score problem framing, evidence use, prioritisation, measurement and adaptation. Assess communication through clarity of reasoning, not presentation polish. Offer reasonable accommodations and equivalent administration conditions.

3. Structured behavioural metric interview — 50 minutes. Retain the current core question, with consistent probes: baseline, personal contribution, alternatives considered, causal evidence, trade-offs, setbacks and learning. Use a second example to distinguish repeatable judgment from one well-rehearsed story. Anchor scores to demonstrated ownership and reasoning, not the absolute size of a metric increase.

4. Engineering partnership interview — 40 minutes. An engineering leader uses a common scenario involving feasibility, scope, reliability and disagreement. Score how candidates uncover constraints and negotiate trade-offs—not jargon, coding knowledge or perceived “credibility.”

Decision process. Before interviewing, agree on competency definitions, anchored 1–4 ratings and minimum acceptable evidence for critical competencies. Interviewers submit scores and supporting observations independently before debriefing. Discuss discrepancies against the rubric; do not mechanically average scores. The hiring manager records the decision, unresolved risks and any exception to the agreed bar.

How we will learn

Record scores, assessor identity, decision reasons, candidate time and withdrawal stage for every finalist. Audit scoring consistency and progression by background and, where appropriately collected, demographic group.

For hires, collect consistently defined 3-, 6- and 12-month outcomes, with role context and performance assessment kept independent of interview scores where feasible. Review process quality after the next four hires, but accumulate more cohorts before claiming predictive validity. Four additional hires will test whether we can run a better loop—not prove that it works.

To: Elif Demir, VP Product From: Director of Product Re: What in our PM loop predicts success, and the loop for next quarter's four hires

Bottom line

Of the four loop scores, only the behavioural round ("tell me about a metric you moved") tracks how PMs perform after hire. The take-home, the presentation and the VP Engineering conversation show no positive relationship, and the last two are negative among the people we hired. The sample is very small, so treat this as a strong lead rather than proof. I recommend making the behavioural evidence the anchor of the loop, shrinking the deck work, and turning the VP Eng conversation into a pass/fail check.

What we have

We have 52 final-round candidates: 9 hired, 33 rejected and 10 who withdrew. Only the 9 hires have 12-month ratings (company average 3.4). The three who struggled (C107, C132, C148) each scored 2, and C104 and C134 scored 3, also below average. All five below-average hires are the same kind of candidate: strong on the deck and the presentation, and scored 3–4 by the VP Eng.

What the data shows

Correlation with 12-month rating, among the 9 hires:

The behavioural pattern is clean. Hires scoring 4 averaged 4.7 (ratings 5, 5, 4), hires scoring 3 averaged 3.3, and all three hires scoring 2 were rated 2. Had we required a behavioural score of at least 3, we would have avoided all three struggling hires and kept every hire rated 3 or above. The pattern also holds within background: among the 4 Big tech hires, the one scoring 3 outperformed the three scoring 2, and among the other 5 hires the 4s outperformed the 3s.

The other three signals deserve less trust than the negative numbers suggest, for two reasons:

• Selection effects. We only see people we hired. If hiring managers pushed through candidates who were strong on the deck and the VP Eng score but weaker elsewhere, those scores will look negatively related to outcomes among hires even if they carry some real signal.
• The VP Eng score works as a gate. Every candidate scoring 1 was rejected or withdrew, so we have no outcome data on that group and cannot say whether the gate screens out good people.

Two further observations about the process:

• The take-home and presentation are the same artifact scored twice, so the deck gets double weight in the panel's discussion.
• Decisions look inconsistent. C135 (3/4/4/3) was rejected while C134 (4/4/3/3) was hired, and five candidates scoring 4 on the behavioural round were rejected. Three of those five had a VP Eng score of 1. With no written decision rule, we can't tell what drove these calls.

How confident we can be

• High confidence: the three struggling hires share one profile (Big tech, polished deck, VP Eng 4, behavioural 2), and our loop rewarded exactly those traits.
• Moderate confidence: the behavioural round carries real signal. The relationship is strong and consistent within subgroups, but it rests on 9 people and one outcome measure.
• Low confidence: any conclusion that the take-home, presentation or VP Eng conversation is useless. Selection effects and range restriction mean the data can't tell us either way. Our evidence only shows they haven't helped.
• Measurement problems:
• The 12-month rating comes from one source, likely the manager who chose the hire, so it may be biased toward justifying that choice.
• Behavioural scores and background are entangled: Big tech hires scored lower (2–3) than the others (3–4). I can't fully separate "behavioural score predicts performance" from "Big tech backgrounds fit us less well," though the within-group pattern favours the former.
• We have only a single 1–4 behavioural score per candidate, with no record of who scored it or how.
• Two or three outliers would change several of these numbers.

The loop I'd run next quarter

1. Recruiter screen. Keep it, with written criteria applied the same way for every candidate.
2. Two structured past-work interviews, replacing the single behavioural round. Different interviewers run them, each using a written rubric. They probe individual contribution versus team credit, the specific decisions the candidate made, what went wrong and what they would do differently, and how they worked with engineering. Interviewers score independently before any discussion. Two scores per candidate also lets us check whether the scoring is consistent.
3. Shorter work sample, replacing the one-week deck. Two to three hours, followed by a 30-minute live discussion, producing one combined score on a rubric. This cuts the double-counting and the candidate burden (10 of 52 withdrew). I'm keeping a work sample because we have no evidence it's useless, but it gets no more weight than any other single signal.
4. VP Eng conversation as a pass/fail risk check. The VP Eng answers one question: "Is there a specific, documented concern about this person's ability to work with engineering?" The 1–4 score no longer counts toward the decision, and a veto needs a written reason.
5. A written decision rule. A candidate must average at least 3 across the two past-work interviews to be eligible. Among those who clear that bar, the hiring manager decides and records the rationale against the rubric. This is a provisional threshold from 9 data points, so we should revisit it after this round.
6. Better outcome data. Run a structured 6- and 12-month review with defined criteria and a second reviewer, rather than a single manager rating, and log every candidate's scores in one place. Four hires bring us to 13 data points, still small, so decisions should stay modest in confidence until we have more.

What I'm not recommending

I'm not recommending a background filter (for example, avoiding Big tech candidates). The sample can't support it, and strong Big tech candidates may well do fine under a better-calibrated behavioural round. Interviewers should be told that polished decks and fluent technical conversation have not predicted success here.

I can have the rubrics and interviewer assignments ready within two weeks if you agree with the direction.