Tasks / Metrics & Experimentation

Draft OKRs

Can the model turn a team's wish list into a few outcome-based OKRs that add up to the company's goals?

Measures the modelTask type v1.1 · 3 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 7 models. 63% were usable with at most a quick edit.

Reliably right

  1. Focuses on the big rock100% pass
    The output reduces the draft to one objective with three KRs, explicitly drops dark mode, AI summaries, and NPS, and explains why.
    GPT-6.1 Sol · API · Eleven key results and a bonus
  2. Builds the key results on the data100% pass
    Key results 2 and 3 are built directly on the first-week joining data and time-to-shared-note data, with baselines from the brief and targets.
    GPT-6.1 Sol · API · Eleven key results and a bonus
  3. Aims the teams with the churn data100% pass
    Uses churn reasons and segment data to aim Payroll at errors, Integrations at 3+ integration adoption, and Core HR at churn reduction and price/value risk, with explicit links.
    GPT-6.1 Sol · API · The goals that don't add up

Where it slips

  1. Flags the bonus link61% pass
    It names the risk of safe targets but only proposes asking leadership how a 70% score will be read, not a concrete fix like separating bonuses from OKR scores.
    Opus 5.5 · Claude · Eleven key results and a bonus
  2. Identifies material uncertainty70% pass
    The output does not name any specific unknowns that could change the OKRs or how they would be resolved.
    GPT-6 Luna · API · Eleven key results and a bonus
  3. Uses the supplied evidence correctly75% pass
    Claims that the gap is correlation and that the safe play is to pick easy KRs are not supported by the brief or arithmetic.
    Sonnet 5.5 · API · Eleven key results and a bonus

How it’s graded

The checks come from what the best product leaders have said about doing this job well on Lenny’s Podcast. Each one names the guest it comes from: follow a name to the idea on the Lenny’s Podcast wiki.

  1. Key results are outcomes, not output

    Is every key result a measurable change in customer or business behaviour, rather than something the team will ship or do?

    Passes when Every key result names a metric with a baseline and a target; shipping work appears only as initiatives that serve them.

  2. Focuses on the big rock

    Does the output cut the goals down to the few that matter most, rather than covering everything the team could do?

    Passes when Few objectives, each with a small number of key results, and says what was dropped and why.

  3. Shows how the goals add up

    Does the output show how each team-level key result contributes to the company-level goals, and flag any that don't?

    Passes when Every key result is linked to a company goal, with the reasoning or data that connects them, and any that serve no company goal are flagged or cut.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a Staff PM working for Hannah Iyer, Quarry's CPO. The CEO has set the company's OKRs for next quarter, and the four product teams have proposed theirs. Write a memo for Hannah, in no more than 1,200 words: what each team should keep or change so their goals add up to the company's (with the revised key results), and your recommendation on the process questions below. The pack below is everything we have. Not all of it matters equally.

What the model was given6 items: About Quarry, Company OKRs (set by the CEO), Why customers leave (exit surveys, 212 responses), Churn by segment, Team proposals, Process questions
About QuarryHR software for companies with 50 to 500 employees. 420 staff, 2,600 customers. The core HR product is sold per employee; Payroll is a paid add-on.
Company OKRs (set by the CEO)Objective: grow by keeping and expanding the customers we have. Key results: net revenue retention from 101% to 108%; annual gross churn from 14% to 10%; Payroll attach rate from 18% to 25%.
Why customers leave (exit surveys, 212 responses)Payroll errors or complexity: 38%. Price: 24%. Missing integrations: 19%. Moved to an all-in-one competitor: 11%. Other: 8%.
Churn by segmentCustomers using Payroll: 8% a year. Customers without it: 16%. Customers with three or more integrations connected: 6%.
Team proposalsCore HR. Objective: build the best HR platform. Key results: ship the org-chart redesign; ship custom fields v2; raise CSAT from 4.2 to 4.5. Payroll. Objective: make payroll effortless. Key results: cut payroll-error tickets by 40%; raise Payroll attach from 18% to 25%. Integrations. Objective: connect everything. Key result: ship 12 new integrations. Growth. Objective: win more customers. Key results: new logos up 20%; trial conversion from 9% to 12%.
Process questionsThe CEO wants every team to have three objectives 'so everything is covered'. The CFO wants OKR scores to set PM bonuses from next quarter. Last year teams scored 0.95 on average. Planning last quarter took five of the quarter's thirteen weeks.
What a strong answer doesThe answer key the graders mark against

Rewrites each team's goals so they add up to the company's. Core HR's key results are outputs; it should aim at a churn driver it owns or justify its place. Payroll's are close (error tickets connect to the biggest churn reason, and attach links to lower churn), so keep them and make the link explicit. Integrations' 'ship 12' becomes an outcome such as the share of customers with three or more integrations connected (churn 6%). Growth's new-logo goal serves none of the company's key results; re-point it at expansion (such as Payroll attach in trials or upsell) or name it as a deliberate exception. Pushes back on three objectives per team (focus on the few that matter), and on tying OKR scores to bonuses: a 0.95 average already suggests safe targets, and pay would make it worse. Suggests a lighter, shorter planning process.

Critical failures (cap the score at 40)
  • Endorses tying OKR scores to PM bonuses without flagging the risk
Task

Staff-level · v1.0 · synthetic · B2B SaaS, goals, staff-level

The outputs

7 setups answered this brief. Pick up to three to read side by side.

Usable with a quick edit11 of 11 checks passed

The memo is clear, evidence-based, and actionable; a PM could use it with only minor wording tweaks, without re-checking facts or redoing reasoning.

Output

To: Hannah Iyer, CPO From: Staff PM Subject: Align next quarter’s product goals to retention and expansion

Recommendation

Give each team one focused objective, supported by two or three measurable results. Replace acquisition and shipping commitments with customer outcomes that support 108% NRR, 10% annual gross churn, and 25% Payroll attach.

The strongest signals favor fixing Payroll and improving useful integration adoption. Payroll errors or complexity account for 38% of surveyed departures; missing integrations account for another 19%. Price, at 24%, also warrants attention to perceived value and commercial policy.

Payroll customers and customers with three or more integrations churn less, but these are correlations—not proof that attaching either product causes retention. We should improve adoption and reliability while testing their effect on retention, not force adoption to hit a number.

Revised team OKRs

Core HR: Help existing customers realize enough value to stay

Keep: CSAT improvement as a supporting indicator. Change: “Build the best HR platform” is too broad. Remove the org-chart redesign and custom fields v2 from the KRs; retain them as candidate initiatives only if evidence connects them to renewal risk or meaningful customer value.

Revised KRs: 1. Reduce annual gross churn from 14% to 10%, as a shared company result. Core HR is the coordinating product owner, not the sole contributor. 2. Raise CSAT from 4.2 to 4.5, using the same question, sampling approach, and customer population. 3. Improve retention at renewal by 10 percentage points in a pre-defined price/value-risk cohort, versus its baseline. Lock cohort criteria before intervention, include every eligible account, and establish the baseline in week one.

The third target is a proposed stretch target, not a forecast. Core should work with Customer Success and Finance on value realization and price objections. Product improvements alone may not address price sensitivity; unrestricted discounting would also undermine NRR.

Payroll: Make Payroll reliable and easy enough to adopt and keep

Keep: The objective’s intent and the focus on errors. Change: Make the error measure volume-adjusted, add a direct measure of complexity, and move accountability for attach to Growth. Payroll remains responsible for readiness and activation quality.

Revised KRs: 1. Reduce payroll-error tickets per 100 payroll runs by 40% against the previous-quarter baseline, with consistent ticket classification. 2. Reduce median customer time to complete a payroll run by 20%, comparing similar payroll complexity; establish the baseline in week one.

Track error severity and support-contact behavior alongside ticket volume. Fewer tickets are not success if errors persist or customers stop reporting them. Growth should not scale attach ahead of Payroll’s ability to deliver a dependable experience.

Integrations: Make the connections customers need dependable and useful

Keep: Investment in integrations. Change: Replace “connect everything” and “ship 12 integrations.” Twelve low-demand releases could satisfy the proposal without retaining anyone. Prioritize missing connections linked to renewal risk and demand among existing customers.

Revised KRs: 1. Increase by 20% the share of active customers using at least three healthy integrations, relative to a week-one baseline. “Using” must require successful recurring data exchange, not merely installation. 2. Reduce failed scheduled syncs per 1,000 sync attempts by 30%, versus the previous quarter.

Treat both numerical targets as planning proposals to validate against the baseline and capacity. Report retention for newly adopting customers against a comparable cohort. The observed 6% churn rate among customers with three or more integrations is a useful signal, not a guaranteed outcome for new adopters.

Growth: Expand revenue from existing customers through successful Payroll adoption

Keep: Experimentation and conversion discipline. Change: Replace “win more customers,” new-logo growth, and trial conversion. They support acquisition, not this quarter’s stated company objective. Necessary acquisition work can continue as business-as-usual; it should not dominate these OKRs.

Revised KRs: 1. Increase Payroll attach from 18% to 25%, measured on the same active-customer denominator as the company metric. 2. Increase NRR from 101% to 108%, as a shared company result, with Growth accountable for coordinating the expansion plan.

Growth owns targeting, commercial conversion, and the adoption funnel; Payroll owns product readiness. Track successful first payroll and subsequent usage so paid-but-unused attachments do not masquerade as progress. Evaluate incentives against margin, cancellations, and net retained revenue.

Make the goals add up financially

In week one, Finance and Analytics should produce a single retained-revenue bridge: opening recurring revenue, churn, contraction, expansion, and closing retained revenue. Size Payroll expansion and other expansion opportunities against the gap to 108% NRR.

We cannot infer that moving churn down four points and attach up seven points automatically achieves NRR. We lack account revenue, Payroll pricing, contraction, and the precise churn definition. Do not treat customer churn and revenue churn as interchangeable.

Preserve the company metrics’ existing definitions and reporting windows. Annual churn and NRR will respond slowly; review renewal outcomes, payroll quality, and adoption weekly without substituting those leading indicators for the company targets. Shared KRs should appear once on the executive scorecard, with named coordinating owners—not be counted repeatedly as independent contributions.

Process decisions

Reject three objectives per team. Coverage belongs in roadmaps and operating plans. A mandatory count encourages teams to invent goals and dilute attention. One objective per team is sufficient here.

Do not tie next-quarter PM bonuses mechanically to OKR scores. Teams share outcomes, annual metrics lag, and direct compensation linkage encourages sandbagging and metric manipulation. Use OKRs as evidence in a broader performance review covering impact, judgment, collaboration, and learning. Keep incentive decisions separate from stretch-goal scoring.

Investigate last year’s 0.95 average. It may indicate conservative targets, output-heavy KRs, or generous scoring—not exceptional performance. Calibrate committed versus stretch goals and review supporting evidence rather than imposing an arbitrary desired average score.

Time-box planning to two weeks. Five weeks consumed 38% of the quarter. Use week one for baselines, the revenue bridge, dependencies, and target feasibility; week two for trade-offs and Hannah’s approval. Review progress monthly and adjust initiatives as evidence changes, without quietly rewriting targets or definitions.

Grades and run details

Decision model 100 · LLM judge 12 of 12 checks
Decision model checks
  • passUses the supplied evidence correctly45%
  • passAddresses the actual decision77%
  • passRespects explicit constraints40%
  • passIdentifies material uncertainty86%
  • passAvoids unsupported claims65%
  • passProduces the required deliverable65%
  • passAims the teams with the churn data95%
  • passReads the 0.95 average for what it is51%
  • passKey results are outcomes, not output91%
  • passFocuses on the big rock95%
  • passShows how the goals add up43%
Run
Run
#1
API response time
69 s
Submitted
5 Oct 2026

Usable with a quick edit11 of 11 checks passed

The memo is clear, actionable, and covers all required elements; a PM could use it with only minor wording adjustments.

Output

To: Hannah Iyer, CPO Subject: Next-quarter OKRs: focus teams on retention and expansion

The CEO’s objective is clear, but the team proposals do not yet add up to it: Growth is optimizing new logos, while Core HR and Integrations mostly measure shipping. I recommend one objective per team, with a small number of measurable results tied to the company outcomes. Keep delivery plans such as “ship 12 integrations” as initiatives, not proof of impact.

Team recommendations

Core HR — change the objective and replace output KRs. “Build the best HR platform” is too broad. Org-chart redesign, custom fields, and CSAT may be useful, but the pack does not show that shipping them will reduce churn or grow revenue. Prioritize work against the customer problems behind churn—including price/value, which needs a product and pricing response rather than a feature assumption.

Revised objective: Improve retention of core HR customers. - Reduce annualized gross churn for customers without Payroll from 16% to at most 10.7%. This is a first-pass allocation: if Payroll attach reaches 25%, Payroll churn stays at 8%, and the segment rates are customer-weighted, non-Payroll churn needs to be about 10.7% for overall churn to reach 10%. Validate definitions and denominators before locking this target. - Keep CSAT at or above 4.2 as a guardrail, rather than making a score increase the main success measure.

Treat the two proposed launches as candidate initiatives, to be prioritized only if evidence links them to retention or expansion.

Payroll — keep the strongest proposal; clarify the outcome. Payroll directly supports both the company’s attach and churn goals. Payroll customers churn at half the rate of non-Payroll customers, and payroll errors or complexity are the most common reported reason for leaving.

Revised objective: Make Payroll reliable and grow adoption among current customers. - Increase Payroll attach from 18% to 25%. - Reduce payroll-error tickets by 40%. - Keep annual Payroll customer churn at 8% or lower.

Define error-ticket counting consistently, and pair the reduction with a quality guardrail so the metric cannot improve by discouraging customers from reporting problems.

Integrations — keep the problem area; replace “ship 12” as the key result. The evidence supports this area: missing integrations account for 19% of exit-survey responses, and customers with three or more integrations have 6% annual churn. But shipping integrations does not show that customers adopt them or stay.

Revised objective: Reduce integration-related customer loss. - Reduce integrations’ share of exit-survey reasons for leaving from 19% to 14%; treat this as a directional diagnostic, since survey shares are noisy. - Increase the number of customers with three or more active integrations. Establish the baseline in week one and set a numeric quarter target before plans are finalized; the pack does not provide enough data to invent one. - Keep “12 integrations shipped” as a delivery commitment only where customer demand and expected adoption justify it.

Growth — change the objective and stop optimizing acquisition this quarter. New logos and trial conversion do not support the stated company objective. Reorient Growth toward expansion in the installed base, working with Payroll on customer targeting and adoption.

Revised objective: Grow revenue from existing customers. - Increase company NRR from 101% to 108%; Growth is accountable for the expansion motion, with other teams contributing to the shared result. - Set a numeric target, after establishing a baseline, for expansion revenue from existing customers excluding Payroll attach. This makes Growth’s contribution visible without double-counting Payroll’s attach KR.

Process recommendations

  • Do not require three objectives per team. Coverage is not a reason to dilute focus. Use one objective and a few KRs per team; capture other responsibilities as initiatives or guardrails. Make cross-team ownership explicit, with one directly accountable owner for each company KR.
  • Do not use OKR scores to set PM bonuses next quarter. Bonuses would encourage teams to choose safe targets, avoid shared work, or optimize the score instead of customer outcomes. An average score of 0.95 last year is already a reason to examine target-setting and scoring calibration—not to add financial stakes. Keep performance and compensation decisions separate from quarterly OKR scores.
  • Cut planning from five weeks to two. Five weeks consumes nearly 40% of the quarter. Timebox baseline checks and target-setting, then revisit assumptions during the quarter. Use the missing integration and expansion baselines to sharpen targets early, not to prolong planning.

Finally, treat the company’s annual churn and NRR measures as lagging outcomes: review leading indicators during the quarter, but keep the company outcomes as the test of whether the portfolio is working.

Grades and run details

Decision model 95 · LLM judge 12 of 12 checks
Decision model checks
  • passUses the supplied evidence correctly33%
  • passAddresses the actual decision68%
  • passRespects explicit constraints54%
  • passIdentifies material uncertainty65%
  • passAvoids unsupported claims30%
  • partialProduces the required deliverable52%
  • passAims the teams with the churn data94%
  • passReads the 0.95 average for what it is91%
  • passKey results are outcomes, not output70%
  • passFocuses on the big rock97%
  • passShows how the goals add up57%
Run
Run
#1
API response time
57 s
Submitted
5 Oct 2026

Needs more than a quick edit7 of 11 checks passed

The output invents baseline metrics not in the supplied evidence, so a PM would need to re-check and replace those numbers before using it. The reasoning is otherwise sound, but the factual gaps prevent immediate use.

Output

MEMORANDUM

TO: Hannah Iyer, Chief Product Officer FROM: Staff Product Manager DATE: October 24, 2023 SUBJECT: Q1 OKR Review & Planning Process Recommendations

---

Part 1: Strategic Alignment of Team OKRs

The CEO’s company-level OKR is clear: Grow by keeping and expanding the customers we have.

Our current metrics show the path forward clearly. Customers without Payroll churn at 16% annually, while those with it churn at 8%. Integrations act as a powerful retention anchor (churn drops to 6% for those with $\ge 3$ integrations). Exit surveys pinpoint our vulnerabilities: 38% leave due to payroll errors/complexity, and 19% due to missing integrations.

To hit a Net Revenue Retention (NRR) of 108%, gross churn of 10%, and a Payroll attach rate of 25%, our product teams cannot operate as independent feature factories. Their proposals currently suffer from output-focus (shipping lists of features) rather than outcome-focus.

Below is my assessment of what each team must keep or change, along with revised Key Results (KRs).

---

#### 1. Core HR * Current Proposal: Obj: Build the best HR platform. KRs: Ship org-chart redesign; ship custom fields v2; raise CSAT from 4.2 to 4.5. * Critique: "Build the best HR platform" is a vague umbrella. Org charts and custom fields are outputs, not outcomes. They do not directly map to NRR, churn reduction, or Payroll attach. * What to Keep: The focus on CSAT. Core HR is our baseline; low satisfaction bleeds into overall retention. * What to Change: Core HR must directly support the retention of non-Payroll customers (who churn at 16%) and lay the groundwork for Payroll discovery. * Revised OKRs: * Objective: Make Core HR indispensable to mid-market operations. * KR1: Raise Core HR CSAT from 4.2 to 4.5. * KR2: Increase 30-day post-signup feature adoption for accounts without Payroll from 40% to 65%. (Drives engagement to lower baseline churn). * KR3: Increase Payroll feature discovery interactions (clicks on Payroll upgrade prompts in Core HR) by 50%. (Supports company attach-rate goal).

---

#### 2. Payroll * Current Proposal: Obj: Make payroll effortless. KRs: Cut payroll-error tickets by 40%; raise Payroll attach from 18% to 25%. * Critique: Excellent start. It directly addresses our #1 churn driver (38% leave due to payroll errors) and the company attach-rate goal. * What to Keep: Both KRs are exceptional. They tie directly to business metrics. * What to Change: The team needs a third KR focused on expansion revenue or reducing the complexity cited in exit surveys, rather than just error ticket volume. * Revised OKRs: * Objective: Make payroll effortless, reliable, and irresistible. * KR1: Cut payroll-error support tickets by 40%. * KR2: Increase Payroll attach rate from 18% to 25%. * KR3: Reduce time-to-run-payroll (median minutes per run) from 18 minutes to 10 minutes. (Directly attacks "complexity" cited in 38% of churned exit surveys).

---

#### 3. Integrations * Current Proposal: Obj: Connect everything. KR: Ship 12 new integrations. * Critique: "Ship 12 integrations" is a classic output trap. We could ship 12 unused integrations and see zero impact on churn. Customers with $\ge 3$ integrations churn at just 6%; 19% leave due to missing integrations. * What to Keep: The expansion of our ecosystem footprint. * What to Change: Pivot from counting shipped integrations to measuring active connection rates and their downstream impact on customer retention. * Revised OKRs: * Objective: Embed Quarry into our customers' existing tech stacks. * KR1: Increase the percentage of customers with 3+ active integrations connected from 22% to 35%. * KR2: Drive integration adoption in the top 5 most-requested categories (Accounting, Time-Tracking) to 40% of our customer base. * KR3: Reduce churn among customers missing key integrations by 30%.

---

#### 4. Growth * Current Proposal: Obj: Win more customers. KRs: New logos up 20%; trial conversion from 9% to 12%. * Critique: The company OKR explicitly shifts focus from pure acquisition to keeping and expanding what we have. A 20% increase in new logos conflicts with our need to fix retention and payroll friction. Furthermore, acquisition velocity without solved payroll/integration churn just feeds a leaky bucket. * What to Keep: Trial conversion optimization (efficient acquisition). * What to Change: Drop the new logo quota. Shift focus to acquiring the right logos (companies predisposed to adopting Payroll) and improving land-and-expand motions. * Revised OKRs: * Objective: Acquire and land high-LTV customers efficiently. * KR1: Increase free-to-paid trial conversion rate from 9% to 12%. * KR2: Increase trials starting with Payroll bundled or selected from 15% to 25%. * KR3: Reduce Day-90 logo churn for new cohorts from 5% to 3%.

---

Part 2: Recommendations on Process Questions

#### 1. The CEO’s request for three objectives per team * Recommendation: Push back firmly. * Rationale: Requiring three objectives per team (across four teams = 12 objectives) creates cognitive overload and diffuses focus. It forces teams to invent tertiary goals just to fill quota, leading to the exact output-driven KRs we are trying to eliminate. One sharp, outcome-oriented objective per team—tightly wound to the company’s NRR, gross churn, and attach-rate goals—is far more effective. If the CEO insists on structural symmetry, permit sub-objectives, but ensure each team has only one primary North Star objective for the quarter.

#### 2. The CFO’s proposal to use OKR scores for PM bonuses * Recommendation: Strongly advise against this. * Rationale: Tying OKR scores directly to compensation induces the "sandbagging effect." PMs will lobby for hockey-stick inputs disguised as KRs or set artificially low baselines to guarantee a 1.0 score (as evidenced by last year's suspicious 0.95 average). Furthermore, product outcomes depend on engineering execution, market conditions, and sales—factors outside a PM's absolute control. * Alternative: Use OKRs strictly for learning, strategic alignment, and performance calibration (what habits and rigor were applied), separating them entirely from bonus calculations. If bonuses must be tied to metrics, tie them to lagging company-level outcomes (like NRR or ARR), not team-level OKR self-assessments.

#### 3. Planning timeline (last quarter took 5 of 13 weeks) * Recommendation: Institute a strict 2-week timebox for Q1 planning, utilizing a Top-Down/Bottom-Up framework. * Rationale: Spending nearly 40% of a quarter on planning is a massive waste of R&D capital. The paralysis last quarter came from teams guessing what mattered because company goals weren't translated early enough. * Proposed Rules for Next Quarter: * Week -2 (led by CPO/CEO): Company OKRs locked and communicated (done). * Week 1 (led by PMs): Draft team OKRs against company constraints in 5 days. No multi-round negotiations. * Week 2 (led by CPO/Finance): Final cross-functional review, dependency mapping, and sign-off. * Pencils down by the end of Week 2. Any missing alignment is resolved by you (the CPO) rather than endless consensus-seeking.

Claims the judge couldn’t find in the brief

  • Core HR is our baseline.
  • The paralysis last quarter came from teams guessing what mattered because company goals weren't translated early enough.
  • Current 30-day post-signup feature adoption for accounts without Payroll is 40%.
  • Current time-to-run-payroll median is 18 minutes.
  • Current Day-90 logo churn for new cohorts is 5%.
  • Current percentage of customers with 3+ active integrations connected is 22%.
  • There are Payroll upgrade prompts in Core HR with measurable interactions.

Grades and run details

Decision model 73 · LLM judge 7 of 12 checks
Decision model checks
  • failUses the supplied evidence correctly85%
  • passAddresses the actual decision56%
  • passRespects explicit constraints36%
  • failIdentifies material uncertainty68%
  • partialAvoids unsupported claims59%
  • passProduces the required deliverable69%
  • passAims the teams with the churn data97%
  • passReads the 0.95 average for what it is99%
  • passKey results are outcomes, not output35%
  • passFocuses on the big rock51%
  • partialShows how the goals add up30%
Run
Run
#1
Time to output
9 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Uses the supplied evidence correctlyRightRightWrong
GPT-6.1 Sol · API

All statements about the current situation are taken directly from the brief or context, or follow from them; no invented facts.

GPT-6 Luna · API

All factual claims about the current situation are directly supported by the supplied context or simple arithmetic.

Gemini 3.5 Flash-Lite · Gemini

The output invents several current-state metrics and product features (e.g., 40% feature adoption, 18-minute payroll run, 5% Day-90 churn, 22% with 3+ integrations, Payroll upgrade prompts) that are not in the brief or derivable from it.

Addresses the actual decisionRightRightMixed
GPT-6.1 Sol · API

The memo commits to clear revised OKRs and process recommendations for Hannah, and notes what would change the call (testing retention effects, validating baselines).

GPT-6 Luna · API

The memo commits to clear revised key results for each team and explicit process recommendations, framed for Hannah.

Gemini 3.5 Flash-Lite · Gemini

The output gives clear recommendations but does not state what result or condition would change them, as required by the criterion.

Identifies material uncertaintyRightRightWrong
GPT-6.1 Sol · API

It identifies correlation vs. causation, missing data, and the need to validate targets, and says how to resolve them (retained-revenue bridge, testing, baselines).

GPT-6 Luna · API

It names missing baselines and definitional unknowns, and says to establish baselines in week one and validate definitions before locking targets.

Gemini 3.5 Flash-Lite · Gemini

The output does not name any unknowns that could change the decision or say how they would be resolved.

Avoids unsupported claimsRightRightWrong
GPT-6.1 Sol · API

Interpretations are labelled as such (correlations, signals, may indicate), and confident claims are supported by the evidence.

GPT-6 Luna · API

Interpretations and forecasts are clearly labelled or are directly supported by the evidence; no unsupported factual claims are presented as established.

Gemini 3.5 Flash-Lite · Gemini

It presents interpretations (e.g., cause of planning paralysis, Core HR as baseline) and invented baseline metrics as established facts without labelling them as hypotheses.

All got right 7

Respects explicit constraintsRightRightRight
GPT-6.1 Sol · API

The output is a memo to Hannah, under 1,200 words, and respects the requested form and reader.

GPT-6 Luna · API

The output is a memo under 1,200 words, addresses the CPO, and respects the requested form.

Gemini 3.5 Flash-Lite · Gemini

The output is a memo to Hannah, within 1,200 words, and covers the requested team OKR revisions and process recommendations.

Produces the required deliverableRightRightRight
GPT-6.1 Sol · API

The memo provides complete revised KRs and process recommendations, within length, and is directly usable by Hannah.

GPT-6 Luna · API

The memo provides revised team goals with key results and process recommendations, and is usable with light edits.

Gemini 3.5 Flash-Lite · Gemini

The memo is complete, in the right form, for the right reader, within the word limit, and a PM could act on it with light edits to the invented numbers.

Aims the teams with the churn dataRightRightRight
GPT-6.1 Sol · API

Uses churn reasons and segment data to aim Payroll at errors, Integrations at 3+ integration adoption, and Core HR at churn reduction and price/value risk, with explicit links.

GPT-6 Luna · API

It uses the churn reasons and segment data to aim Payroll at errors, Integrations at 3+ integrations, and Core HR at non-Payroll churn, with explicit links.

Gemini 3.5 Flash-Lite · Gemini

It uses the churn reasons and churn-by-segment data to aim Payroll at errors, Integrations at 3+ integrations, and Core HR at non-Payroll retention, with explicit links.

Reads the 0.95 average for what it isRightRightRight
GPT-6.1 Sol · API

Recognizes the 0.95 average suggests safe targets, and advises against tying bonuses to OKR scores because it would encourage sandbagging.

GPT-6 Luna · API

It flags the 0.95 average as a sign of safe targets and recommends against tying bonuses to scores to avoid further sandbagging.

Gemini 3.5 Flash-Lite · Gemini

It calls the 0.95 average 'suspicious', says it suggests safe targets, and argues tying bonuses would push targets lower, recommending against it.

Key results are outcomes, not outputRightRightRight
GPT-6.1 Sol · API

Every key result is a measurable change in customer or business behavior with a baseline and target; no shipping outputs appear as KRs.

GPT-6 Luna · API

Every key result is a measurable outcome (churn, attach, error tickets, CSAT guardrail, exit-survey share, 3+ integrations, NRR, expansion revenue); shipping appears only as initiatives.

Gemini 3.5 Flash-Lite · Gemini

All revised key results are measurable changes in customer or business behaviour with baselines and targets; no shipping outputs appear as KRs.

Focuses on the big rockRightRightRight
GPT-6.1 Sol · API

Cuts each team to one objective with a few KRs, drops shipping and acquisition goals, and explains why each was dropped.

GPT-6 Luna · API

It cuts each team to one objective with a few KRs, drops shipping KRs, and explains what was dropped and why.

Gemini 3.5 Flash-Lite · Gemini

It pushes back on three objectives per team, recommends one primary objective each, and explicitly drops new logos and output-based KRs like shipping 12 integrations.

Shows how the goals add upRightRightRight
GPT-6.1 Sol · API

Every KR is linked to a company goal (churn, NRR, attach) with reasoning, and the new-logo goal is explicitly cut because it serves no company goal.

GPT-6 Luna · API

Each team-level KR is linked to a company goal with reasoning or data, and the new-logo goal is flagged as not supporting the company objective and reoriented.

Gemini 3.5 Flash-Lite · Gemini

Every team's KRs are linked to company goals (NRR, churn, attach rate) with reasoning, and the new-logo goal is flagged as serving no company goal and cut.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Provisional The checklist is still being calibrated against our PM.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6.1 SolwithAPI100.0100.02None
2GPT-6 AstrawithChatGPT98.5100.03None
3Sonnet 5.5withAPI90.991.72None
4GPT-6 LunawithAPI93.283.32None
5Opus 5.5withClaude90.986.13None
6Gemini 3.8 FlashwithAPI68.270.82None
7Gemini 3.5 Flash-LitewithGemini77.362.52None

About the task

The PM job

Setting a team's goals for the quarter.

Why it matters

Most OKRs are task lists in disguise. Teams ship everything on them and nothing moves.

What good looks like

  • Key results are outcomes, not things to ship
  • Few enough to focus on
  • Each team's goals visibly add up to the company's
  • Kept apart from performance ratings

Deliberately not measured

  • OKR software or formatting
Capability tested

Goal setting

The failure we’re looking for

A task list dressed up as key results

Grading

Decision model and LLM judge, calibrated against a blind PM review