Tasks / Metrics & Experimentation

Draft OKRs

Can the model turn a team's wish list into a few outcome-based OKRs that add up to the company's goals?

Measures the modelTask type v1.1 · 3 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 20 graded outputs by 7 models. 65% were usable with at most a quick edit.

Reliably right

Builds the key results on the data100% pass
Key results 2 and 3 are built directly on the first-week joining data and time-to-shared-note data, with baselines from the brief and targets.
GPT-6.1 Sol · API · Eleven key results and a bonus
Aims the teams with the churn data100% pass
Uses churn reasons and segment data to aim Payroll at errors, Integrations at 3+ integration adoption, and Core HR at churn reduction and price/value risk, with explicit links.
GPT-6.1 Sol · API · The goals that don't add up
Reads the 0.95 average for what it is100% pass
Recognizes the 0.95 average suggests safe targets, and advises against tying bonuses to OKR scores because it would encourage sandbagging.
GPT-6.1 Sol · API · The goals that don't add up

Where it slips

Flags the bonus link61% pass
It names the risk of safe targets but only proposes asking leadership how a 70% score will be read, not a concrete fix like separating bonuses from OKR scores.
Opus 5.5 · Claude · Eleven key results and a bonus
Identifies material uncertainty70% pass
The output does not name any specific unknowns that could change the OKRs or how they would be resolved.
GPT-6 Luna · API · Eleven key results and a bonus
Uses the supplied evidence correctly75% pass
Claims that the gap is correlation and that the safe play is to pick easy KRs are not supported by the brief or arithmetic.
Sonnet 5.5 · API · Eleven key results and a bonus

How it’s graded

The checks come from what the best product leaders have said about doing this job well on Lenny’s Podcast. Each one names the guest it comes from: follow a name to the idea on the Lenny’s Podcast wiki.

Key results are outcomes, not output
Is every key result a measurable change in customer or business behaviour, rather than something the team will ship or do?
Passes when Every key result names a metric with a baseline and a target; shipping work appears only as initiatives that serve them.
Matt LeMayLenny’s Podcast · OKRs
Focuses on the big rock
Does the output cut the goals down to the few that matter most, rather than covering everything the team could do?
Passes when Few objectives, each with a small number of key results, and says what was dropped and why.
Christina WodtkeLenny’s Podcast · OKRs
Shows how the goals add up
Does the output show how each team-level key result contributes to the company-level goals, and flag any that don't?
Passes when Every key result is linked to a company goal, with the reasoning or data that connects them, and any that serve no company goal are flagged or cut.
Matt LeMayLenny’s Podcast · OKRs Claire Hughes JohnsonLenny’s Podcast · OKRs

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a Staff PM at Pathwise, working for the CPO, Grace Mensah. Last quarter's OKR scorecard is below, along with the company goal it was meant to serve. Grace wants a memo of no more than 1,100 words: what last quarter's OKRs actually tell us (not just the scores), and next quarter's OKRs for the Learner app, Admin and Growth teams, each with baselines and targets, that add up to the company goal.\n\nThe files are attached.

What the model was given5 items: About Pathwise, company_okr_q3.md, okr_scorecard_q3.csv (team_label is the type each team gave its key result), metric_dictionary.md, churn_reasons_q3.md

About PathwiseOnline learning for companies: courses employees take, and tools for the HR and L&D teams who buy it. Sold per seat on annual contracts.

company_okr_q3.md7 lines · Download

# Company OKR, Q3 (set by the CEO)

Objective: grow by keeping and expanding the customers we have.
- Net revenue retention: from 103% to 108%. Actual: 102%.
- Quarterly logo churn: from 3.1% to 2.5%. Actual: 3.4%.

Q3 average key result score across product teams: 0.81. The all-hands slide said "a great quarter".

okr_scorecard_q3.csv (team_label is the type each team gave its key result)team,objective,key_result,team_label,baseline,target,actual,score Content,Best course library in L&D,Publish 40 new courses,output,0,40,41,1.00 Content,Best course library in L&D,Course completion rate,outcome,41,55,58,1.00 Content,Best course library in L&D,Translate top 20 courses into 3 languages,output,0,60,60,1.00 Learner app,Learning people love,Ship mobile offline mode,output,0,1,1,1.00 Learner app,Learning people love,Weekly active learners per seat,outcome,0.31,0.4,0.29,0.00 Learner app,Learning people love,Ship streaks and badges,output,0,1,1,1.00 Learner app,Learning people love,App store rating,outcome,4.2,4.6,4.5,0.75 Admin,Admins get value fast,Launch 3 HRIS integrations,outcome,0,3,3,1.00 Admin,Admins get value fast,Days from contract to first learner,outcome,34,14,31,0.15 Admin,Admins get value fast,Ship new reporting dashboard,output,0,1,1,1.00 Growth,Expand inside accounts,Seats added in existing accounts,outcome,9200,14000,8100,0.00 Growth,Expand inside accounts,Run 12 expansion experiments,output,0,12,13,1.00 Growth,Expand inside accounts,Launch in-app upgrade prompts,output,0,1,1,1.00 Platform,Reliable at scale,Uptime,outcome,99.5,99.9,99.93,1.00 Platform,Reliable at scale,Migrate to new video CDN,output,0,1,1,1.00 Platform,Reliable at scale,Close 150 bugs,output,0,150,162,1.00

metric_dictionary.md5 lines · Download

# Metric dictionary (excerpt)

- Course completion rate: share of course enrolments marked complete. Until 31 July "complete" meant every module finished. From 1 August it means 80% of modules finished (changed so completion matches the certificate rule).
- Weekly active learners per seat: learners who start or continue a course in a week, divided by paid seats.
- Days from contract to first learner: calendar days from contract signature to the first learner starting a course.

churn_reasons_q3.md8 lines · Download

# Why accounts left in Q3 (exit interviews, 38 accounts)

- "Our people didn't use it": 17
- Budget cut: 9
- Moved to the LMS that came with our HR system: 7
- Content didn't fit our industry: 5

…

What a strong answer doesThe answer key the graders mark against

Reads the scorecard for what it is: the average is 0.81, but every output key result scored 1.00 (things shipped) while outcome key results averaged about 0.48, and the company missed both goals (NRR 102% against 108%, churn 3.4% against 2.5%). Two scores flatter: 'Launch 3 HRIS integrations' is labelled an outcome but is output, and course completion only 'beat' its target because completion was redefined on 1 August (all modules to 80%), so the result can't be compared with its baseline. The outcome key results that matter for churn moved backwards or barely (weekly active learners per seat 0.31 to 0.29; days to first learner 34 to 31; seats added 8,100 against 9,200). Builds next quarter's OKRs on the churn evidence: low usage is the top exit reason, and accounts reaching a first learner within 14 days churn at a third of the rate. So Admin owns days from contract to first learner (from 31), Learner app owns weekly active learners per seat (from 0.29), Growth owns seats added in existing accounts or net expansion, each with a baseline from the files and a target, with shipping work as initiatives. Notes the completion metric needs restating on one definition before anyone uses it.

Critical failures (cap the score at 40)

Describes last quarter as a success on the strength of the 0.81 average

Task

Staff-level · v1.0 · synthetic · B2B SaaS, goals, data files, staff-level

The outputs

6 setups answered this brief. Pick up to three to read side by side.

GPT-6 AstraChatGPT100Usable with a quick edit GPT-6.1 SolAPI98Usable with a quick edit Sonnet 5.5API94Usable with a quick edit GPT-6 LunaAPI90Usable with a quick edit Opus 5.5Claude85Needs more work Gemini 3.5 Flash-LiteGemini31Needs more work

Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit2 of 11 checks passed

The memo invents baselines for new metrics not in the supplied data, and includes output key results that are things to ship, requiring rework of the OKRs and verification of baselines before use.

Output

MEMORANDUM

TO: Grace Mensah, Chief Product Officer FROM: Staff PM, Product DATE: October 2, 2023 SUBJECT: Q3 OKR Post-Mortem & Q4 Product OKRs

---

Part 1: What Q3 Actually Tells Us (Beyond the 0.81 Score)

On paper, Q3 looked like a triumph. The company-wide average key result score across product teams was 0.81, prompting celebratory remarks at the all-hands. However, a closer look at the data reveals a stark disconnect between our product output and the company’s core objective: growth by keeping and expanding the customers we have.

The company's Q3 goals failed. Net Revenue Retention (NRR) dropped from our baseline of 103% to an actual of 102% (target: 108%), and quarterly logo churn rose from 3.1% to 3.4% (target: 2.5%).

Why did we miss our financial and retention targets while hitting an 0.81 KR score? The data points to a systemic structural flaw in how we set and measure OKRs: we rewarded teams for shipping outputs, while outcomes that actually drive retention and expansion flatlined or missed.

1. The Output-Outcome Illusion: Look at the team scorecards. Every single output KR scored 1.00. Content published 41 courses (target 40), translated 60 items, shipped mobile offline mode, shipped streaks and badges, launched three HRIS integrations, shipped a new reporting dashboard, ran 13 expansion experiments, launched in-app upgrade prompts, migrated a video CDN, and closed 162 bugs. We were exceptionally busy. Conversely, our critical outcomes largely failed: * Learner App: Weekly active learners per seat plummeted from 0.31 to an actual of 0.29 (target: 0.40, score: 0.00). * Admin: Days from contract to first learner barely budged, moving from 34 to 31 days (target: 14 days, score: 0.15). * Growth: Seats added in existing accounts fell from a baseline of 9,200 to 8,100 (target: 14,000, score: 0.00).

2. The Root Causes of Churn: Our Q3 exit interviews across 38 churned accounts provide the smoking gun. The leading reason for churn—cited by nearly half (17 out of 38)—was simple: "Our people didn't use it." Furthermore, the data reveals a powerful operational lever we have ignored: accounts that reached their first learner within 14 days of contract signature churned at one-third of the rate of the rest. Yet, our Admin team’s initiative to reduce this time stalled at 31 days.

3. Metric Distortions: Even our apparent wins require scrutiny. Content’s course completion rate hit 1.00 (rising from 41% to 58%), but the metric dictionary reveals a rule change on August 1st: the definition of "complete" was quietly lowered from 100% of modules finished to 80%. The "success" is largely an artifact of moving the goalposts.

The Takeaway: In Q3, product engineering functioned as an effective feature factory, but those features did not translate into user engagement, rapid time-to-value, or account expansion. To fix NRR and logo churn in Q4, we must completely realign product incentives away from output delivery and lock them onto core engagement and adoption metrics.

---

Part 2: Q4 OKRs for Learner App, Admin, and Growth Teams

To directly support the CEO's ongoing company objective—grow by keeping and expanding the customers we have—we are restructuring Q4 OKRs. Every team’s objectives are anchored to lagging retention drivers (engagement, time-to-value, and expansion).

#### 1. Learner App Team * Context & Rationale: With 17 out of 38 churned accounts citing lack of employee usage, the Learner App team must pivot away from vanity features (streaks and badges shipped in Q3 did nothing to stop the drop) and focus entirely on driving habitual weekly usage. * Objective: Drive habitual engagement to secure employee adoption and prevent account-level churn. * Key Results: * KR 1 (Outcome): Increase weekly active learners per seat from 0.29 to 0.36. (Baseline: 0.29, Target: 0.36) * KR 2 (Outcome): Increase 30-day course retention rate for newly enrolled learners from 22% to 35%. (Baseline: 22%, Target: 35%) * KR 3 (Output): Ship personalized "Next Step" AI recommendations on the home screen to streamline the path from login to learning. (Baseline: 0, Target: 1)

#### 2. Admin Team * Context & Rationale: Exit data proves that getting the first learner active within 14 days slashes churn by two-thirds. In Q3, we focused on HRIS integrations (output), but customers still languished for 31 days before their employees started learning. Admin must own the friction reduction between contract signature and organizational activation. * Objective: Accelerate time-to-value so customer organizations realize immediate ROI. * Key Results: * KR 1 (Outcome): Reduce days from contract signature to first learner from 31 days to 14 days. (Baseline: 31, Target: 14) * KR 2 (Outcome): Increase self-serve HRIS integration completion rate (admins successfully connecting without support intervention) from 40% to 75%. (Baseline: 40%, Target: 75%) * KR 3 (Output): Launch automated "Nudge Admin" workflow triggers when zero learners are active 7 days post-contract. (Baseline: 0, Target: 1)

#### 3. Growth Team * Context & Rationale: Q3 saw expansion seats drop to 8,100 against a 14,000 target. Running 12 experiments and launching upgrade prompts generated output, but no revenue expansion. Growth must transition from running experiments to driving net-new seat expansion inside existing healthy accounts. * Objective: Expand footprint and seat count within existing healthy accounts. * Key Results: * KR 1 (Outcome): Increase net new seats added in existing accounts from 8,100 (Q3 actual) to 12,500. (Baseline: 8,100, Target: 12,500) * KR 2 (Outcome): Increase conversion rate of in-app upgrade prompts from trial/capped tiers to paid seat expansions from 1.2% to 3.5%. (Baseline: 1.2%, Target: 3.5%) * KR 3 (Output): Launch automated team-invitation workflows for departmental managers in accounts nearing seat capacity limits. (Baseline: 0, Target: 1)

Claims the judge couldn’t find in the brief

30-day course retention rate for newly enrolled learners baseline is 22%
Self-serve HRIS integration completion rate baseline is 40%
Conversion rate of in-app upgrade prompts baseline is 1.2%
Streaks and badges shipped in Q3 did nothing to stop the drop in weekly active learners
Product engineering functioned as an effective feature factory

Grades and run details

Decision model 41 · LLM judge 3 of 12 checks

Decision model checks

failUses the supplied evidence correctly86%
passAddresses the actual decision42%
failRespects explicit constraints12%
failIdentifies material uncertainty36%
failAvoids unsupported claims36%
partialProduces the required deliverable48%
passReads the scorecard for what it is98%
partialCatches the redefined key result95%
partialKey results are outcomes, not output55%
partialFocuses on the big rock30%
partialShows how the goals add up87%

Run

Run: #1
Time to output: 8 s
Submitted: 5 Oct 2026

Opus 5.5Claude

Needs more than a quick edit10 of 11 checks passed

The memo contains a factual error in the average score calculation (0.74 vs 0.81), which would require re-checking the evidence before use.

Output

To: Grace Mensah, CPO From: Staff PM, Product Re: What Q3's OKRs tell us, and proposed Q4 OKRs for Learner app, Admin and Growth

---

1. Bottom line

Q3 was not a great quarter. Both company key results moved backwards:

Net revenue retention (NRR) fell from 103% to 102%, against a target of 108%.
Logo churn rose from 3.1% to 3.4%, against a target of 2.5%.

The 0.81 average hid this because it mixed outputs, which we control, with outcomes, which customers control. We shipped everything we said we would. The customer behaviours those launches were meant to change didn't move, or got worse.

2. What Q3 actually tells us

Outputs scored perfectly. Outcomes didn't.

All 10 output key results (KRs) scored 1.00. I'm counting "Launch 3 HRIS integrations" as an output; it was labelled an outcome, but it's a launch.
The 6 genuine outcomes averaged 0.48.
The three outcomes that link most directly to the company goal averaged 0.05: weekly active learners per seat (0.00), days to first learner (0.15) and seats added in existing accounts (0.00).

The 0.81 doesn't reproduce. The 16 KRs on the scorecard average 0.74. Averaging by team gives 0.75. We should find out how 0.81 was calculated before it's quoted again.

Launches didn't move the metrics they targeted.

Team	What shipped	What happened
Learner app	Offline mode, streaks and badges	Weekly active learners per seat fell from 0.31 to 0.29
Growth	13 experiments, in-app upgrade prompts	Seats added fell from 9,200 to 8,100
Admin	3 HRIS integrations, new dashboard	Days to first learner improved only from 34 to 31, against a target of 14

One "win" is probably a definition change. Course completion rose from 41% to 58%. But on 1 August, "complete" changed from 100% of modules finished to 80%. Completion climbing while weekly activity fell suggests most of the gain came from the new definition. We should restate Q3 on both definitions before claiming it.

Scoring rewarded going backwards no more harshly than standing still. Weekly activity and seats added both ended below their baselines, yet each scored 0.00, the same as no change. The scorecard couldn't show that we lost ground.

The churn data points to adoption. Of 38 exit interviews:

Reason for leaving	Accounts	Share
"Our people didn't use it"	17	45%
Budget cut	9	24%
Moved to the LMS bundled with their HR system	7	18%
Content didn't fit their industry	5	13%

Accounts that reached a first learner within 14 days churned at one third the rate of the rest. That is a correlation, not proof: healthier buyers may simply onboard faster. Still, it is our strongest signal. Adoption is the lever. More features and more experiments are not, on Q3's evidence.

3. Rule changes for Q4

KRs are outcomes only. Launches go in a separate initiatives list and are not scored.
Score against the baseline, and allow negative scores. Ending below the baseline should look worse than standing still.
Freeze metric definitions for the quarter. Any change gets restated on both the old and new definition.
Every KR names the company KR it serves.

4. Proposed Q4 OKRs

Baselines are Q3 actuals where we have them. Three baselines are marked TBC: we don't yet track those metrics, and Data will deliver them by the end of week 2. Each target is set as a fixed increase on its baseline, so the target is fixed as soon as the baseline is known.

Learner app. Objective: the seats customers pay for get used. This serves both churn and NRR.

KR	Baseline	Target
Weekly active learners per paid seat (all accounts)	0.29	0.33
Weekly active learners per seat in accounts renewing in Q4 or Q1	TBC	Baseline + 0.05
Guardrail: app store rating	4.5	Stays at or above 4.5

The second KR matters because it targets the accounts whose renewal decision is being made now.

Admin. Objective: every new customer has learners in their first two weeks. This serves churn.

KR	Baseline	Target
Median days from contract to first learner	31 (Q3 figure; confirm it is a median, not a mean)	20
Share of new accounts with a first learner within 14 days	TBC	Baseline + 25 points
Share of new accounts with an HRIS sync live by day 14	TBC	50%

The HRIS sync KR turns the integrations we shipped into usage. It also addresses the 7 accounts that left for their HR system's bundled LMS.

Growth. Objective: expand where usage proves value, and stop seat losses at renewal. This serves NRR.

KR	Baseline	Target
Seats added in existing accounts	8,100	10,000
Seat retention at renewal (seats renewed ÷ seats up for renewal)	TBC (Finance)	Baseline + 3 points

Growth should focus expansion on accounts above a usage threshold, to be agreed with Learner app. Q3 showed that prompting low-usage accounts to upgrade doesn't work.

5. How these add up to the company goal, and where they don't

Churn (3.4% → 2.5%). This works through two levers:

Q4–Q1 renewal accounts. Contracts are annual, so churn this quarter is mostly decided by how accounts used us over the past year. The only lever with a near-term effect is raising usage in accounts renewing soon. That is the Learner app's second KR.
New-account onboarding. Admin's onboarding KRs act on new accounts, which won't come up for renewal for a year.

To illustrate the size of the onboarding effect: suppose 20% of accounts currently reach a first learner within 14 days, and those accounts churn at one third the rate of the rest. Moving that share to 50% would cut churn from 3.4% to about 2.6%. That only holds once the whole customer base has passed through the new onboarding, and only if the relationship is causal.

Honest read: we will not reach 2.5% churn in Q4. Admin's onboarding work is what gets us there in 2025.

NRR (102% → 108%). NRR gains come from two places: fewer churned and contracted seats (Growth's seat retention KR, plus the Learner app's usage work) and more seats added (Growth's expansion KR).

I can't yet show the arithmetic that closes the 6-point gap. That needs Finance's figures for revenue up for renewal and average price per seat. Even with the 10,000 seats target, I expect Q4 to land nearer 104–105% than 108%.

The remaining 6 churned accounts don't map to these KRs, though the HRIS work partly addresses the bundled-LMS losses:

Budget cuts (9 accounts) need commercial responses, such as flexible tiers.
Content fit (5 accounts) belongs to the Content team. Their Q4 KR should be usage of newly published courses, not the number of courses published.

6. Decisions I need from you

Approve the four scoring rules, and correct the 0.81 figure with the CEO.
Discuss the company targets with the CEO. I suggest keeping 108% NRR and 2.5% churn as the year-end destination, with Q4 checkpoints of 104–105% NRR and 3.0% churn, so the scorecard tells the truth.
Ask Finance and Data to deliver the TBC baselines and the NRR arithmetic by the end of week 2. I'll lock the targets the same day.

Claims the judge couldn’t find in the brief

The 16 KRs on the scorecard average 0.74, not 0.81.

Grades and run details

Decision model 95 · LLM judge 10 of 12 checks

Decision model checks

passUses the supplied evidence correctly28%
passAddresses the actual decision82%
passRespects explicit constraints17%
passIdentifies material uncertainty95%
passAvoids unsupported claims47%
passProduces the required deliverable39%
passReads the scorecard for what it is100%
passCatches the redefined key result100%
passKey results are outcomes, not output79%
passFocuses on the big rock67%
partialShows how the goals add up25%

Run

Run: #1
Time to output: 65 s
Submitted: 5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.