Tasks / Metrics & Experimentation

Draft OKRs

Can the model turn a team's wish list into a few outcome-based OKRs that add up to the company's goals?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 50% were usable with at most a quick edit.

Reliably right

Builds the key results on the data100% pass
Key results 2 and 3 are built directly on the first-week joining data and time-to-shared-note data, with baselines from the brief and targets.
GPT-6.1 Sol · API · Eleven key results and a bonus
Respects explicit constraints99% pass
The output is a memo to Grace Mensah, within the 1,100-word limit, and includes the requested diagnosis and team OKRs with baselines and targets.
GPT-6 Astra · ChatGPT · A great quarter, on paper
Produces the required deliverable99% pass
The memo is complete, in the right form for the CPO, and a PM could act on it with only light edits.
GPT-6 Astra · ChatGPT · A great quarter, on paper

Where it slips

Flags the bonus link47% pass
Mentions bonus multipliers only to justify outcome KRs; does not name the risk of safe targets or suggest decoupling OKR scores from bonuses.
Haiku 5.5 · API · Eleven key results and a bonus
Avoids unsupported claims67% pass
Presents 'the gap is correlation' and 'the safe play is to pick easy KRs' as facts without labeling them as hypotheses.
Sonnet 5.5 · API · Eleven key results and a bonus
Identifies material uncertainty68% pass
The output does not name any specific unknowns that could change the OKRs or how they would be resolved.
GPT-6 Luna · API · Eleven key results and a bonus

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Key results are outcomes, not output
Is every key result a measurable change in customer or business behaviour, rather than something the team will ship or do?
Passes when Every key result names a metric with a baseline and a target; shipping work appears only as initiatives that serve them.
Matt LeMayLenny’s Podcast
Focuses on the big rock
Does the output cut the goals down to the few that matter most, rather than covering everything the team could do?
Passes when Few objectives, each with a small number of key results, and says what was dropped and why.
Christina WodtkeLenny’s Podcast
Shows how the goals add up
Does the output show how each team-level key result contributes to the company-level goals, and flag any that don't?
Passes when Every key result is linked to a company goal, with the reasoning or data that connects them, and any that serve no company goal are flagged or cut.
Matt LeMayLenny’s Podcast Claire Hughes JohnsonLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a Staff PM at Pathwise, working for the CPO, Grace Mensah. Last quarter's OKR scorecard is below, along with the company goal it was meant to serve. Grace wants a memo of no more than 1,100 words: what last quarter's OKRs actually tell us (not just the scores), and next quarter's OKRs for the Learner app, Admin and Growth teams, each with baselines and targets, that add up to the company goal.\n\nThe files are attached.

What the model was given5 items: About Pathwise, company_okr_q3.md, okr_scorecard_q3.csv (team_label is the type each team gave its key result), metric_dictionary.md, churn_reasons_q3.md

About PathwiseOnline learning for companies: courses employees take, and tools for the HR and L&D teams who buy it. Sold per seat on annual contracts.

company_okr_q3.md7 lines · Download

# Company OKR, Q3 (set by the CEO)

Objective: grow by keeping and expanding the customers we have.
- Net revenue retention: from 103% to 108%. Actual: 102%.
- Quarterly logo churn: from 3.1% to 2.5%. Actual: 3.4%.

Q3 average key result score across product teams: 0.81. The all-hands slide said "a great quarter".

okr_scorecard_q3.csv (team_label is the type each team gave its key result)team,objective,key_result,team_label,baseline,target,actual,score Content,Best course library in L&D,Publish 40 new courses,output,0,40,41,1.00 Content,Best course library in L&D,Course completion rate,outcome,41,55,58,1.00 Content,Best course library in L&D,Translate top 20 courses into 3 languages,output,0,60,60,1.00 Learner app,Learning people love,Ship mobile offline mode,output,0,1,1,1.00 Learner app,Learning people love,Weekly active learners per seat,outcome,0.31,0.4,0.29,0.00 Learner app,Learning people love,Ship streaks and badges,output,0,1,1,1.00 Learner app,Learning people love,App store rating,outcome,4.2,4.6,4.5,0.75 Admin,Admins get value fast,Launch 3 HRIS integrations,outcome,0,3,3,1.00 Admin,Admins get value fast,Days from contract to first learner,outcome,34,14,31,0.15 Admin,Admins get value fast,Ship new reporting dashboard,output,0,1,1,1.00 Growth,Expand inside accounts,Seats added in existing accounts,outcome,9200,14000,8100,0.00 Growth,Expand inside accounts,Run 12 expansion experiments,output,0,12,13,1.00 Growth,Expand inside accounts,Launch in-app upgrade prompts,output,0,1,1,1.00 Platform,Reliable at scale,Uptime,outcome,99.5,99.9,99.93,1.00 Platform,Reliable at scale,Migrate to new video CDN,output,0,1,1,1.00 Platform,Reliable at scale,Close 150 bugs,output,0,150,162,1.00

metric_dictionary.md5 lines · Download

# Metric dictionary (excerpt)

- Course completion rate: share of course enrolments marked complete. Until 31 July "complete" meant every module finished. From 1 August it means 80% of modules finished (changed so completion matches the certificate rule).
- Weekly active learners per seat: learners who start or continue a course in a week, divided by paid seats.
- Days from contract to first learner: calendar days from contract signature to the first learner starting a course.

churn_reasons_q3.md8 lines · Download

# Why accounts left in Q3 (exit interviews, 38 accounts)

- "Our people didn't use it": 17
- Budget cut: 9
- Moved to the LMS that came with our HR system: 7
- Content didn't fit our industry: 5

…

What a strong answer doesThe answer key the graders mark against

Reads the scorecard for what it is: the average is 0.81, but every output key result scored 1.00 (things shipped) while outcome key results averaged about 0.48, and the company missed both goals (NRR 102% against 108%, churn 3.4% against 2.5%). Two scores flatter: 'Launch 3 HRIS integrations' is labelled an outcome but is output, and course completion only 'beat' its target because completion was redefined on 1 August (all modules to 80%), so the result can't be compared with its baseline. The outcome key results that matter for churn moved backwards or barely (weekly active learners per seat 0.31 to 0.29; days to first learner 34 to 31; seats added 8,100 against 9,200). Builds next quarter's OKRs on the churn evidence: low usage is the top exit reason, and accounts reaching a first learner within 14 days churn at a third of the rate. So Admin owns days from contract to first learner (from 31), Learner app owns weekly active learners per seat (from 0.29), Growth owns seats added in existing accounts or net expansion, each with a baseline from the files and a target, with shipping work as initiatives. Notes the completion metric needs restating on one definition before anyone uses it.

Critical failures (cap the score at 40)

Describes last quarter as a success on the strength of the 0.81 average

Task

Staff-level · v1.1

The outputs

9 setups answered this brief.

GPT-6 AstraChatGPT100Usable with a quick edit GPT-6.1 SolAPI100Usable with a quick edit Sonnet 5.5API100Usable with a quick edit Haiku 5.5API96Usable with a quick edit Opus 5.5Claude88Needs more work GPT-6 LunaAPI85Usable with a quick edit Gemini 4 ArgonAPI77Needs more work Gemini 3.8 FlashAPI58Needs more work Gemini 3.5 Flash-LiteGemini33Needs more work

Gemini 4 ArgonAPI

Needs more than a quick edit8 of 11 checks passed

The unsupported causal claims (e.g., about gamification and budget cut causes) would need verification or rewriting before a PM could use it without re-checking evidence or reasoning.

Output

MEMO To: Grace Mensah, CPO From: Staff PM Date: Start of Q4 Subject: Q3 OKR Post-Mortem and Proposed Q4 Product OKRs

1. What Q3’s OKRs Actually Tell Us

At the all-hands, we celebrated a 0.81 average OKR score while the company failed its only two goals: Net Revenue Retention (NRR) dropped from 103% to 102% (target: 108%), and logo churn rose from 3.1% to 3.4% (target: 2.5%).

We achieved high team scores while losing customers because our OKR system rewards shipping features over changing customer behavior.

Outputs disguised as outcomes inflated our score. Ten of our 16 KRs were binary tasks (shipping features, running experiments, closing bugs). The Admin team even mislabeled "Launch 3 HRIS integrations" as an outcome. All 10 output KRs scored 1.00. If we strip out outputs and look only at true outcomes, our average score drops to 0.48. If we further remove technical uptime, the score for customer-behavior outcomes is 0.38.

The commercial outcomes we actually needed to move went backward. * Learner Engagement (0.00): Weekly active learners per seat (WAL/seat) fell from 0.31 to 0.29. Over 70% of paid seats sit unused each week. * Onboarding Speed (0.15): Days from contract to first learner barely budged from 34 to 31 days (target: 14). * Account Expansion (0.00): Seats added in existing accounts fell from 9,200 to 8,100 (target: 14,000).

Our three outcome wins were disconnected from retention: 1. Course completion rate (41% to 58%, score 1.00): This was largely a mirage. On August 1, we lowered the definition of "complete" from 100% of modules finished to 80% to match our certificate rule. We moved the goalposts mid-quarter and claimed a win. 2. App store rating (4.2 to 4.5, score 0.75): A vanity metric. It takes a tiny fraction of happy users to rate an app, which masks the fact that overall weekly usage shrank. 3. Uptime (99.93%, score 1.00): Good hygiene from Platform, but high availability doesn’t stop churn if nobody logs in.

Our outputs failed to move the outcomes they sat next to. * Admin shipped 3 HRIS integrations, yet time-to-first-learner stayed stuck at a month. Shipping an integration means nothing if admins don't adopt it during onboarding. * Learner shipped offline mode, streaks, and badges, yet weekly active usage dropped. Gamification only engages users who are already active; it doesn't get dormant seats in the door. * Growth ran 13 experiments and shipped upgrade prompts, yet added fewer seats than in Q2. You cannot prompt an admin to buy more seats when 71% of the seats they already pay for are gathering dust.

The Root Cause of Q3 Churn Exit interviews from 38 churned accounts make the causal chain clear: * 45% left because "our people didn't use it" (17/38). Another 24% (9/38) cited budget cuts, and 18% (7/38) consolidated into their HR system’s bundled LMS—both of which happen when an L&D team cannot prove ROI on Pathwise usage. * Crucially, accounts that reached their first learner within 14 days churned at a third of the rate of the rest. Right now, our average account takes 31 days—meaning the majority of new cohorts start life in the high-churn danger zone.

---

2. How Q4 OKRs Add Up to the Company Goal

To hit 108% NRR and 2.5% logo churn, Product must fix the sequence of value creation for an annual seat-based business:

Admin must compress onboarding under the 14-day threshold and get seats assigned, cutting early structural churn.
Learner must turn assigned seats into active weekly learners—both to stop the #1 churn reason ("people didn't use it") and to eat up seat capacity.
Growth must convert high seat utilization into mid-contract seat expansion and give L&D buyers the ROI proof they need to defend their budget at renewal.

---

3. Proposed Q4 OKRs (Learner, Admin, Growth)

Every KR below is a measurable customer behavior or business outcome. No roadmaps, features, or experiment counts are permitted on the scorecard.

#### Admin Team Role in Company OKR: Cut logo churn by getting accounts past the 14-day survival mark and driving seat provisioning.

Objective: Get every new account to fast, broad adoption. * KR 1: Reduce average days from contract signature to first learner from 31 days to 14 days. * KR 2: Increase the share of new accounts reaching their first active learner within 14 days from 28% (est. Q3 baseline) to 75%. (Note: Admin to confirm exact Q3 percentage in week 1). * KR 3: Increase average seat assignment rate (paid seats assigned to a named employee via HRIS or invite) within 30 days of contract from 52% (est. baseline) to 85%.

#### Learner App Team Role in Company OKR: Eliminate the #1 churn driver ("people didn't use it") by activating dormant seats and building a weekly learning habit.

Objective: Turn paid seats into weekly active learners. * KR 1: Increase Weekly Active Learners per paid seat (WAL/seat) from 0.29 to 0.42. * KR 2: Increase 30-day assigned-seat activation rate (share of newly assigned seats that start at least one course within 30 days) from 45% (est. baseline) to 70%. * KR 3: Increase course completion rate—held strictly to the constant 80%-module definition—from 58% to 68%.

#### Growth Team Role in Company OKR: Drive NRR from 102% to 108% by expanding capacity-constrained accounts and arming buyers to prevent renewal down-sell.

Objective: Expand high-usage accounts and lock in renewals. * KR 1: Increase quarterly seats added in existing accounts from 8,100 to 14,500 (sized to make up the Q3 shortfall and hit 108% NRR). * KR 2: Increase the quarterly seat-expansion conversion rate among "at-capacity" accounts (accounts with >80% of paid seats assigned and active) from 12% (est. baseline) to 28%. * KR 3: Reduce gross revenue contraction from seat reductions at renewal from 4.5% (est. Q3 baseline) to 2.0% of renewable ARR (e.g., by proving ROI via the new reporting dashboard before renewal conversations).

---

4. Guardrails for Q4 Execution

Freeze the Metric Dictionary: No metric definition may change mid-quarter without CPO sign-off and a retroactive restatement of the baseline.
Track Outputs in Sprint Reviews, Not OKRs: Shipping features, running 12 experiments, or launching integrations are how we try to move KRs. If an experiment ships and the metric doesn’t move, the score is 0.00—which tells us in week 4 to pivot our approach before the quarter is lost.
Confirm Baselines by Friday: Where Q3 tracking didn't capture cohort rates (Admin KR2/KR3, Learner KR2, Growth KR2/KR3), Data and PM leads will pull exact Q3 actuals by Friday so our baselines are locked before Q4 kickoff.

Grades and run details

Decision model 77 · LLM judge 83

Decision model checks

failUses the supplied evidence correctly76%
failAddresses the actual decision20%
passRespects explicit constraints28%
passIdentifies material uncertainty54%
partialAvoids unsupported claims43%
passProduces the required deliverable74%
passReads the scorecard for what it is100%
passCatches the redefined key result98%
passKey results are outcomes, not output97%
passFocuses on the big rock49%
passShows how the goals add up62%

Run

Run: #1
API response time: 22 s
Submitted: 13 Oct 2026

GPT-6 LunaAPI

Usable with a quick edit9 of 11 checks passed

The memo is clear, evidence-based, and actionable; a PM could use it with minor wording tweaks to emphasize the output/outcome score split.

Output

Memo

To: Grace Mensah, CPO Subject: What Q3 OKRs tell us—and Q4 OKRs tied to retention and expansion

What Q3 tells us

Q3 was not a “great quarter” against the company goal. Net revenue retention fell to 102%, below the 108% goal, and logo churn rose to 3.4%, above the 2.5% goal. The 0.81 average team score obscures that miss: many high-scoring KRs measured shipping or activity, not customer value.

The clearest warning is usage. Learner app weekly active learners per paid seat declined from 0.31 to 0.29, missing its 0.40 target. In exit interviews, “our people didn’t use it” was the most common reason for leaving (17 of 38 accounts). That makes engagement a credible retention lever to investigate—not proof that low app usage caused those departures.

Onboarding is another promising lever: time to first learner was 31 days, versus a 14-day target. Accounts reaching a first learner within 14 days churned at one-third the rate of the rest. That association supports prioritizing faster activation, but does not establish causation. Seven of the 38 departing accounts moved to an LMS bundled with their HR system, so integrations may also matter; Q3’s three launches are an output, not evidence of customer adoption or retention impact.

Expansion also underperformed: existing-account seats added were 8,100, below both the 14,000 target and the 9,200 baseline. Running 13 experiments and shipping upgrade prompts did not translate into the intended result. Meanwhile, Content’s completion result is hard to interpret: the definition changed on 1 August, from every module to 80% of modules, so 58% cannot be fairly compared with the 55% target without a consistent-definition view. Strong uptime and course delivery are useful foundations, but do not offset the missed retention and expansion outcomes.

Proposed Q4 OKRs

Keep the company outcome explicit and shared: NRR 102% → 108%; quarterly logo churn 3.4% → 2.5%. These are not numbers that team KRs can literally be summed to produce. They are the outcomes the teams collectively support; the team KRs below are leading indicators. Review company outcomes alongside them, using consistent cohort and revenue definitions.

Team / objective	Key result: Q3 baseline → Q4 target
Learner app — Make learning a regular habit for more paid learners	Weekly active learners per paid seat: 0.29 → 0.40
Admin — Get new customers to learner value sooner	Calendar days from contract signature to first learner starting a course: 31 → 14 or fewer
Growth — Expand paid seats in existing accounts	Paid seats added in existing accounts: 8,100 → 14,000

Use the existing metric definitions and reporting windows for these comparisons; do not count shipped features, experiments, or prompts as substitutes for the outcomes. The Q3 data provided does not show how expansion seats translate into NRR, or how team-level changes affect logo churn. For Q4 reviews, Finance and the teams should make the NRR bridge visible—expansion, contraction, and churn—and check whether faster activation and higher learner activity precede improved retention. That will let us judge whether these team results are actually adding up to the company goal.

Grades and run details

Decision model 77 · LLM judge 92

Decision model checks

passUses the supplied evidence correctly89%
failAddresses the actual decision2%
passRespects explicit constraints83%
partialIdentifies material uncertainty28%
passAvoids unsupported claims89%
passProduces the required deliverable62%
partialReads the scorecard for what it is83%
partialCatches the redefined key result64%
passKey results are outcomes, not output98%
passFocuses on the big rock74%
passShows how the goals add up80%

Run

Run: #1
API response time: 24 s
Submitted: 5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 3

Uses the supplied evidence correctlyMixedRight

Gemini 4 Argon · API

All factual statements about the current situation are sourced from the provided context or follow directly by arithmetic.

GPT-6 Luna · API

All factual claims about the current situation are directly from the supplied context or arithmetic.

Avoids unsupported claimsWrongRight

Gemini 4 Argon · API

Several causal claims are presented as settled fact without evidence (e.g., 'gamification only engages users who are already active', 'both [budget cuts and consolidation] happen when an L&D team cannot prove ROI').

GPT-6 Luna · API

Hypotheses are clearly labelled as not proven, and interpretations are cautious.

Reads the scorecard for what it isRightWrong

Gemini 4 Argon · API

It separates output and outcome scores, shows outcome average 0.48, and sets them against the missed company goals.

GPT-6 Luna · API

Does not split the scores by output and outcome with numbers; only says 'many high-scoring KRs measured shipping or activity' without quantifying the perfect output scores versus poor outcome scores.

All mixed 1

Addresses the actual decisionMixedMixed

Gemini 4 Argon · API

The memo commits to specific Q4 OKRs for each team, framed for the CPO, and conditionally relies on baseline confirmation to finalise.

GPT-6 Luna · API

Commits to specific Q4 OKRs with baselines and targets, framed for the CPO, and suggests what to monitor to validate them.

All got right 7

Respects explicit constraintsRightRight

Gemini 4 Argon · API

The memo is well within 1,100 words, addresses the CPO as requested, and delivers the required analysis and OKRs.

GPT-6 Luna · API

Memo format, under 1,100 words, addresses the three requested teams and ties to the company goal.

Identifies material uncertaintyRightRight

Gemini 4 Argon · API

It identifies estimated baselines and requires exact Q3 actuals by week 1, acknowledging uncertainty in those numbers.

GPT-6 Luna · API

Identifies causation uncertainty, metric redefinition, and missing NRR bridge, and says how to resolve them in Q4 reviews.

Produces the required deliverableRightRight

Gemini 4 Argon · API

The memo is a complete deliverable with analysis and team-level OKRs with baselines and targets, usable as-is.

GPT-6 Luna · API

Provides a complete memo with analysis and next-quarter OKRs as requested, usable as is.

Catches the redefined key resultRightRight

Gemini 4 Argon · API

It flags both the course completion definition change and the mislabeled HRIS integration output, explaining how they inflated scores.

GPT-6 Luna · API

Catches both the redefined completion metric and the output-labelled-as-outcome integrations KR, and explains the implications for interpreting scores.

Key results are outcomes, not outputRightRight

Gemini 4 Argon · API

All proposed key results are outcome metrics with baselines and targets; no shipping or activity-based KRs appear on the scorecard.

GPT-6 Luna · API

All proposed key results are outcome metrics with baselines and targets; no shipping work appears as a KR.

Focuses on the big rockRightRight

Gemini 4 Argon · API

Each team has one objective and three key results, with explicit reasoning that previous output KRs are dropped because they failed to move outcomes.

GPT-6 Luna · API

Focuses on three teams with one KR each, implicitly dropping Content and Platform as not directly driving the retention goal.

Shows how the goals add upRightRight

Gemini 4 Argon · API

The memo explicitly links each team's KRs to the company goal of NRR and churn, with a clear laddering narrative.

GPT-6 Luna · API

Links each team KR to the company retention/expansion goal, acknowledges data gaps, and suggests how to verify the connection.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 100% of 12 checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	GPT-6 AstrawithChatGPT	97.7	100.0	2	None
2	GPT-6.1 SolwithAPI	97.7	100.0	2	None
3	Sonnet 5.5withAPI	100.0	91.7	2	None
4	Haiku 5.5withAPI	93.2	95.8	2	None
5	Opus 5.5withClaude	95.5	79.2	2	None
6	GPT-6 LunawithAPI	79.5	79.2	2	None
7	Gemini 4 ArgonwithAPI	77.3	70.8	2	None
8	Gemini 3.8 FlashwithAPI	61.4	62.5	2	None
9	Gemini 3.5 Flash-LitewithGemini	59.1	45.8	2	None

What you get for the cost

Haiku 5.5 · API scores within 4 points of GPT-6 Astra · ChatGPT at about a thirtieth of the cost.

Best valueNobody beats these on both score and cost

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
GPT-6 Luna · API	78	1.0×
Gemini 3.5 Flash-Lite · Gemini	50	2.1×
Haiku 5.5 · API	95	2.1×
Gemini 3.8 Flash · API	59	13×
Gemini 4 Argon · API	71	13×
GPT-6.1 Sol · API	99	16×
Sonnet 5.5 · API	96	44×
Opus 5.5 · Claude	83	78×
GPT-6 Astra · ChatGPT	99	83×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Setting a team's goals for the quarter.

Why it matters

Most OKRs are task lists in disguise. Teams ship everything on them and nothing moves.

What good looks like

Key results are outcomes, not things to ship
Few enough to focus on
Each team's goals visibly add up to the company's
Kept apart from performance ratings

Deliberately not measured

OKR software or formatting

Capability tested

Goal setting

The failure we’re looking for

A task list dressed up as key results

Grading

Decision model and LLM judge, calibrated against a blind PM review

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

Key results are outcomes, not output

Focuses on the big rock

Shows how the goals add up

Plus our standard checks

The tasks

The outputs

Output

1. What Q3’s OKRs Actually Tell Us

2. How Q4 OKRs Add Up to the Company Goal

3. Proposed Q4 OKRs (Learner, Admin, Growth)

4. Guardrails for Q4 Execution

Grades and run details

Decision model checks

Run

Output

What Q3 tells us

Proposed Q4 OKRs

Grades and run details

Decision model checks

Run

Check by check

Where they differ 3

All mixed 1

All got right 7

Results

What you get for the cost

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured