Tasks / Metrics & Experimentation

Draft OKRs

Can the model turn a team's wish list into a few outcome-based OKRs that add up to the company's goals?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 56% were usable with at most a quick edit.

Reliably right

Builds the key results on the data100% pass
Key results 2 and 3 are built directly on the first-week joining data and time-to-shared-note data, with baselines from the brief and targets.
GPT-6.1 Sol · API · Eleven key results and a bonus
Addresses the actual decision97% pass
The output commits to a clear rewritten OKR set and a note explaining the changes, framed for the squad, and states that targets are proposed pending feasibility review.
GPT-6.1 Sol · API · Eleven key results and a bonus
Focuses on the big rock94% pass
The output reduces the draft to one objective with three KRs, explicitly drops dark mode, AI summaries, and NPS, and explains why.
GPT-6.1 Sol · API · Eleven key results and a bonus

Where it slips

Flags the bonus link59% pass
Mentions bonus multipliers only to justify outcome KRs; does not name the risk of safe targets or suggest decoupling OKR scores from bonuses.
Haiku 5.5 · API · Eleven key results and a bonus
Avoids unsupported claims69% pass
Presents 'the gap is correlation' and 'the safe play is to pick easy KRs' as facts without labeling them as hypotheses.
Sonnet 5.5 · API · Eleven key results and a bonus
Identifies material uncertainty69% pass
The output does not name any specific unknowns that could change the OKRs or how they would be resolved.
GPT-6 Luna · API · Eleven key results and a bonus

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Key results are outcomes, not output
Is every key result a measurable change in customer or business behaviour, rather than something the team will ship or do?
Passes when Every key result names a metric with a baseline and a target; shipping work appears only as initiatives that serve them.
Matt LeMayLenny’s Podcast
Focuses on the big rock
Does the output cut the goals down to the few that matter most, rather than covering everything the team could do?
Passes when Few objectives, each with a small number of key results, and says what was dropped and why.
Christina WodtkeLenny’s Podcast
Shows how the goals add up
Does the output show how each team-level key result contributes to the company-level goals, and flag any that don't?
Passes when Every key result is linked to a company goal, with the reasoning or data that connects them, and any that serve no company goal are flagged or cut.
Matt LeMayLenny’s Podcast Claire Hughes JohnsonLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a Staff PM at Pathwise, working for the CPO, Grace Mensah. Last quarter's OKR scorecard is below, along with the company goal it was meant to serve. Grace wants a memo of no more than 1,100 words: what last quarter's OKRs actually tell us (not just the scores), and next quarter's OKRs for the Learner app, Admin and Growth teams, each with baselines and targets, that add up to the company goal.\n\nThe files are attached.

What the model was given5 items: About Pathwise, company_okr_q3.md, okr_scorecard_q3.csv (team_label is the type each team gave its key result), metric_dictionary.md, churn_reasons_q3.md

About PathwiseOnline learning for companies: courses employees take, and tools for the HR and L&D teams who buy it. Sold per seat on annual contracts.

company_okr_q3.md7 lines · Download

# Company OKR, Q3 (set by the CEO)

Objective: grow by keeping and expanding the customers we have.
- Net revenue retention: from 103% to 108%. Actual: 102%.
- Quarterly logo churn: from 3.1% to 2.5%. Actual: 3.4%.

Q3 average key result score across product teams: 0.81. The all-hands slide said "a great quarter".

okr_scorecard_q3.csv (team_label is the type each team gave its key result)team,objective,key_result,team_label,baseline,target,actual,score Content,Best course library in L&D,Publish 40 new courses,output,0,40,41,1.00 Content,Best course library in L&D,Course completion rate,outcome,41,55,58,1.00 Content,Best course library in L&D,Translate top 20 courses into 3 languages,output,0,60,60,1.00 Learner app,Learning people love,Ship mobile offline mode,output,0,1,1,1.00 Learner app,Learning people love,Weekly active learners per seat,outcome,0.31,0.4,0.29,0.00 Learner app,Learning people love,Ship streaks and badges,output,0,1,1,1.00 Learner app,Learning people love,App store rating,outcome,4.2,4.6,4.5,0.75 Admin,Admins get value fast,Launch 3 HRIS integrations,outcome,0,3,3,1.00 Admin,Admins get value fast,Days from contract to first learner,outcome,34,14,31,0.15 Admin,Admins get value fast,Ship new reporting dashboard,output,0,1,1,1.00 Growth,Expand inside accounts,Seats added in existing accounts,outcome,9200,14000,8100,0.00 Growth,Expand inside accounts,Run 12 expansion experiments,output,0,12,13,1.00 Growth,Expand inside accounts,Launch in-app upgrade prompts,output,0,1,1,1.00 Platform,Reliable at scale,Uptime,outcome,99.5,99.9,99.93,1.00 Platform,Reliable at scale,Migrate to new video CDN,output,0,1,1,1.00 Platform,Reliable at scale,Close 150 bugs,output,0,150,162,1.00

metric_dictionary.md5 lines · Download

# Metric dictionary (excerpt)

- Course completion rate: share of course enrolments marked complete. Until 31 July "complete" meant every module finished. From 1 August it means 80% of modules finished (changed so completion matches the certificate rule).
- Weekly active learners per seat: learners who start or continue a course in a week, divided by paid seats.
- Days from contract to first learner: calendar days from contract signature to the first learner starting a course.

churn_reasons_q3.md8 lines · Download

# Why accounts left in Q3 (exit interviews, 38 accounts)

- "Our people didn't use it": 17
- Budget cut: 9
- Moved to the LMS that came with our HR system: 7
- Content didn't fit our industry: 5

…

What a strong answer doesThe answer key the graders mark against

Reads the scorecard for what it is: the average is 0.81, but every output key result scored 1.00 (things shipped) while outcome key results averaged about 0.48, and the company missed both goals (NRR 102% against 108%, churn 3.4% against 2.5%). Two scores flatter: 'Launch 3 HRIS integrations' is labelled an outcome but is output, and course completion only 'beat' its target because completion was redefined on 1 August (all modules to 80%), so the result can't be compared with its baseline. The outcome key results that matter for churn moved backwards or barely (weekly active learners per seat 0.31 to 0.29; days to first learner 34 to 31; seats added 8,100 against 9,200). Builds next quarter's OKRs on the churn evidence: low usage is the top exit reason, and accounts reaching a first learner within 14 days churn at a third of the rate. So Admin owns days from contract to first learner (from 31), Learner app owns weekly active learners per seat (from 0.29), Growth owns seats added in existing accounts or net expansion, each with a baseline from the files and a target, with shipping work as initiatives. Notes the completion metric needs restating on one definition before anyone uses it.

Critical failures (cap the score at 40)

Describes last quarter as a success on the strength of the 0.81 average

Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

GPT-6 AstraChatGPT100Usable with a quick edit GPT-6.1 SolAPI98Usable with a quick edit Haiku 5.5API96Usable with a quick edit Sonnet 5.5API94Usable with a quick edit GPT-6 LunaAPI90Usable with a quick edit Opus 5.5Claude85Needs more work Gemini 3.8 FlashAPI58Needs more work Gemini 3.5 Flash-LiteGemini31Needs more work

Haiku 5.5API

Usable with a quick edit11 of 11 checks passed

The memo is logically structured, rooted in the evidence, and could be sent with minimal edits after pulling the TBD baselines and confirming the proposed company targets.

Output

To: Grace Mensah, CPO From: Staff PM Re: What Q3 OKRs told us, and Q4 team OKRs that add up to the company goal

Bottom line

Q3 looked like a good quarter (0.81 average) but failed on the goal it was meant to serve. We shipped almost everything we planned and moved the customer numbers the wrong way. Q4 needs to be built around the three things customers told us drive churn: people not using the product, slow time to first learner, and seat growth that depends on usage. Output shipping stays, but it no longer counts as success on its own.

What Q3 actually tells us

The 0.81 is a shipping score. Of 16 key results, nine were outputs (things we built, published, or closed). All nine scored 1.00. Outcomes scored 0.56 on average, and the outcome KRs that map to the company goal mostly failed:

Net revenue retention: 102% against a 108% target, and down from the 103% baseline.
Quarterly logo churn: 3.4% against a 2.5% target, and worse than the 3.1% baseline.
Seats added in existing accounts: 8,100, down from 9,200. Growth shipped 13 experiments and a prompt, and expansion still fell.
Weekly active learners per seat: 0.29, down from 0.31. Usage declined while we shipped streaks and offline mode.
Days from contract to first learner: 31, down from 34 against a target of 14. We barely moved the number that most affects churn.

The bright spots are real but don't map to the goal: app rating rose from 4.2 to 4.5, and uptime is ahead of target.

Why the goal didn't move: the team KRs never owned it. No team had NRR or churn as its KR. Each team optimized its own outputs, and the company numbers fell in the gaps between them.

The exit interviews say where the gaps are. Of 38 accounts that left in Q3:

17 (45%): "Our people didn't use it." This is adoption, which sits with the Learner app and with the onboarding that Admin owns.
9 (24%): budget cut. Hard to move directly, but more likely when usage is low.
7 (18%): moved to the LMS that came with their HR system. Our three HRIS integrations did not stop this. We need to know which HR systems these accounts used.
5 (13%): content didn't fit their industry. This is Content's problem, and it isn't in this memo's scope.

The most useful finding is the activation data: accounts that reached a first learner within 14 days churned at one-third the rate of the rest. Our average is 31 days. Activation speed is probably our highest-leverage lever, and Q3 barely touched it.

The completion rate score can't be trusted. On 1 August we changed the definition of "complete" from every module finished to 80% of modules finished. The 41→58 movement (scored 1.00) mixes two definitions, and the new definition is looser. Some or all of that gain is a measurement change, not learning. We should restate the Q2 baseline on the new definition before anyone cites this number. Until then it should not count as a win.

A smaller measurement issue: "Launch 3 HRIS integrations" was labeled an outcome but is really a count of shipped work. It should be an output, paired with an adoption outcome.

What this means for Q4

We should commit to the company numbers and make the team KRs add up to them. My proposal:

Net revenue retention: 102% → 106% at Q4 end (108% stays the year-end target, but I don't think it's reachable in one quarter from 102%).
Quarterly logo churn: 3.4% → 2.8%.

Churn and expansion both depend on usage, so I've set team targets on usage and activation rather than on shipping. Finance should confirm how much each lever contributes to NRR. I've assumed both churn and seat expansion have to move for the 106% target to hold.

Q4 team OKRs

Each team has two outcome KRs. Output KRs can sit underneath as supporting work but are not scored toward the goal. "TBD" baselines need to be pulled in week one, before targets are locked.

Team	Objective	Key result	Q3 baseline	Q4 target
Learner app	Learners come back every week	Weekly active learners per seat	0.29	0.35
		Share of learners active in week 2 after their first course	TBD (pull week 1)	Set week 2; expect +10 pts
Admin	Accounts get a first learner fast	Median days from contract to first learner	31	21
		Share of new contracts reaching first learner within 14 days	TBD (pull week 1)	Set week 2 once baseline is known
Growth	Expansion follows real usage	Seats added in existing accounts	8,100	10,500
		Renewal-risk accounts (low usage, renewal in 90 days) with a named save plan	TBD (pull week 1)	100% covered; save rate tracked

How the teams connect to the goal

Learner app reduces the 17 "didn't use it" churns. Higher weekly usage also feeds expansion, since seat growth needs people actually learning.
Admin targets the 14-day activation link. Getting the median from 31 to 21 days should move the share of accounts inside 14 days, which is where the churn gap is. The HRIS KR should be reviewed in week 2 by checking which HR systems the seven accounts that left for their HR LMS were using.
Growth stops measuring success as experiments run and starts measuring seats added in accounts that use the product. Expansion in low-usage accounts is a risk, not a win, so the renewal-risk KR makes Growth own the save motion for the budget-cut and low-usage accounts.

Decisions I need from you

Agree the company targets (NRR 106%, churn 2.8%), or tell me to hold 108% and accept the risk.
Restate completion rate. Finance or Data should back-calculate the Q2 baseline under the new definition. Until then Content's completion KR cannot be scored.
Assign the churn-reason KR to a single owner. Right now no one owns the 38 exit interviews. I suggest Customer Success, reporting to you monthly.
Content's fit problem. Five churns cited content fit. Content's Q4 OKRs should include an industry-fit KR. I'd like to discuss this with the Content lead before it's locked.

I can have the baselines pulled and the final targets locked by the end of week two.

Grades and run details

Decision model 91 · LLM judge 100

Decision model checks

passUses the supplied evidence correctly38%
passAddresses the actual decision41%
passRespects explicit constraints21%
passIdentifies material uncertainty80%
partialAvoids unsupported claims51%
partialProduces the required deliverable31%
passReads the scorecard for what it is91%
passCatches the redefined key result100%
passKey results are outcomes, not output73%
passFocuses on the big rock79%
passShows how the goals add up27%

Run

Run: #1
API response time: 32 s
Submitted: 8 Oct 2026

Opus 5.5Claude

Needs more than a quick edit10 of 11 checks passed

The memo contains a factual error in the average score calculation (0.74 vs 0.81), which would require re-checking the evidence before use.

Output

To: Grace Mensah, CPO From: Staff PM, Product Re: What Q3's OKRs tell us, and proposed Q4 OKRs for Learner app, Admin and Growth

---

1. Bottom line

Q3 was not a great quarter. Both company key results moved backwards:

Net revenue retention (NRR) fell from 103% to 102%, against a target of 108%.
Logo churn rose from 3.1% to 3.4%, against a target of 2.5%.

The 0.81 average hid this because it mixed outputs, which we control, with outcomes, which customers control. We shipped everything we said we would. The customer behaviours those launches were meant to change didn't move, or got worse.

2. What Q3 actually tells us

Outputs scored perfectly. Outcomes didn't.

All 10 output key results (KRs) scored 1.00. I'm counting "Launch 3 HRIS integrations" as an output; it was labelled an outcome, but it's a launch.
The 6 genuine outcomes averaged 0.48.
The three outcomes that link most directly to the company goal averaged 0.05: weekly active learners per seat (0.00), days to first learner (0.15) and seats added in existing accounts (0.00).

The 0.81 doesn't reproduce. The 16 KRs on the scorecard average 0.74. Averaging by team gives 0.75. We should find out how 0.81 was calculated before it's quoted again.

Launches didn't move the metrics they targeted.

Team	What shipped	What happened
Learner app	Offline mode, streaks and badges	Weekly active learners per seat fell from 0.31 to 0.29
Growth	13 experiments, in-app upgrade prompts	Seats added fell from 9,200 to 8,100
Admin	3 HRIS integrations, new dashboard	Days to first learner improved only from 34 to 31, against a target of 14

One "win" is probably a definition change. Course completion rose from 41% to 58%. But on 1 August, "complete" changed from 100% of modules finished to 80%. Completion climbing while weekly activity fell suggests most of the gain came from the new definition. We should restate Q3 on both definitions before claiming it.

Scoring rewarded going backwards no more harshly than standing still. Weekly activity and seats added both ended below their baselines, yet each scored 0.00, the same as no change. The scorecard couldn't show that we lost ground.

The churn data points to adoption. Of 38 exit interviews:

Reason for leaving	Accounts	Share
"Our people didn't use it"	17	45%
Budget cut	9	24%
Moved to the LMS bundled with their HR system	7	18%
Content didn't fit their industry	5	13%

Accounts that reached a first learner within 14 days churned at one third the rate of the rest. That is a correlation, not proof: healthier buyers may simply onboard faster. Still, it is our strongest signal. Adoption is the lever. More features and more experiments are not, on Q3's evidence.

3. Rule changes for Q4

KRs are outcomes only. Launches go in a separate initiatives list and are not scored.
Score against the baseline, and allow negative scores. Ending below the baseline should look worse than standing still.
Freeze metric definitions for the quarter. Any change gets restated on both the old and new definition.
Every KR names the company KR it serves.

4. Proposed Q4 OKRs

Baselines are Q3 actuals where we have them. Three baselines are marked TBC: we don't yet track those metrics, and Data will deliver them by the end of week 2. Each target is set as a fixed increase on its baseline, so the target is fixed as soon as the baseline is known.

Learner app. Objective: the seats customers pay for get used. This serves both churn and NRR.

KR	Baseline	Target
Weekly active learners per paid seat (all accounts)	0.29	0.33
Weekly active learners per seat in accounts renewing in Q4 or Q1	TBC	Baseline + 0.05
Guardrail: app store rating	4.5	Stays at or above 4.5

The second KR matters because it targets the accounts whose renewal decision is being made now.

Admin. Objective: every new customer has learners in their first two weeks. This serves churn.

KR	Baseline	Target
Median days from contract to first learner	31 (Q3 figure; confirm it is a median, not a mean)	20
Share of new accounts with a first learner within 14 days	TBC	Baseline + 25 points
Share of new accounts with an HRIS sync live by day 14	TBC	50%

The HRIS sync KR turns the integrations we shipped into usage. It also addresses the 7 accounts that left for their HR system's bundled LMS.

Growth. Objective: expand where usage proves value, and stop seat losses at renewal. This serves NRR.

KR	Baseline	Target
Seats added in existing accounts	8,100	10,000
Seat retention at renewal (seats renewed ÷ seats up for renewal)	TBC (Finance)	Baseline + 3 points

Growth should focus expansion on accounts above a usage threshold, to be agreed with Learner app. Q3 showed that prompting low-usage accounts to upgrade doesn't work.

5. How these add up to the company goal, and where they don't

Churn (3.4% → 2.5%). This works through two levers:

Q4–Q1 renewal accounts. Contracts are annual, so churn this quarter is mostly decided by how accounts used us over the past year. The only lever with a near-term effect is raising usage in accounts renewing soon. That is the Learner app's second KR.
New-account onboarding. Admin's onboarding KRs act on new accounts, which won't come up for renewal for a year.

To illustrate the size of the onboarding effect: suppose 20% of accounts currently reach a first learner within 14 days, and those accounts churn at one third the rate of the rest. Moving that share to 50% would cut churn from 3.4% to about 2.6%. That only holds once the whole customer base has passed through the new onboarding, and only if the relationship is causal.

Honest read: we will not reach 2.5% churn in Q4. Admin's onboarding work is what gets us there in 2025.

NRR (102% → 108%). NRR gains come from two places: fewer churned and contracted seats (Growth's seat retention KR, plus the Learner app's usage work) and more seats added (Growth's expansion KR).

I can't yet show the arithmetic that closes the 6-point gap. That needs Finance's figures for revenue up for renewal and average price per seat. Even with the 10,000 seats target, I expect Q4 to land nearer 104–105% than 108%.

The remaining 6 churned accounts don't map to these KRs, though the HRIS work partly addresses the bundled-LMS losses:

Budget cuts (9 accounts) need commercial responses, such as flexible tiers.
Content fit (5 accounts) belongs to the Content team. Their Q4 KR should be usage of newly published courses, not the number of courses published.

6. Decisions I need from you

Approve the four scoring rules, and correct the 0.81 figure with the CEO.
Discuss the company targets with the CEO. I suggest keeping 108% NRR and 2.5% churn as the year-end destination, with Q4 checkpoints of 104–105% NRR and 3.0% churn, so the scorecard tells the truth.
Ask Finance and Data to deliver the TBC baselines and the NRR arithmetic by the end of week 2. I'll lock the targets the same day.

Claims the judge couldn’t find in the brief

The 16 KRs on the scorecard average 0.74, not 0.81.

Grades and run details

Decision model 95 · LLM judge 83

Decision model checks

passUses the supplied evidence correctly28%
passAddresses the actual decision82%
passRespects explicit constraints17%
passIdentifies material uncertainty95%
passAvoids unsupported claims47%
passProduces the required deliverable39%
passReads the scorecard for what it is100%
passCatches the redefined key result100%
passKey results are outcomes, not output79%
passFocuses on the big rock67%
partialShows how the goals add up25%

Run

Run: #1
Time to output: 65 s
Submitted: 5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 1

Uses the supplied evidence correctlyRightMixed

Haiku 5.5 · API

All factual claims about the current situation are directly drawn from the supplied context or derived by arithmetic, with no invented facts.

Opus 5.5 · Claude

The claim that the 16 KRs average 0.74 is a factual error; the correct average is 0.81, so not all statements about the current situation are supported.

All got right 10

Addresses the actual decisionRightRight

Haiku 5.5 · API

The memo commits to clear next-quarter OKRs for the three teams with baselines and targets, and also proposes company-level targets while asking for CPO decision on them.

Opus 5.5 · Claude

Commits to proposed Q4 OKRs and says what would change the NRR target (Finance figures).

Respects explicit constraintsRightRight

Haiku 5.5 · API

The output is a memo under 1,100 words, addressed to the CPO, includes the requested analysis and team OKRs with baselines and targets, and proposes them for the named teams.

Opus 5.5 · Claude

Memo format, under 1,100 words, includes Q3 analysis and Q4 OKRs for the three teams with baselines and targets.

Identifies material uncertaintyRightRight

Haiku 5.5 · API

The memo flags TBD baselines that need pulling, the need to restate the completion rate baseline, the unknown HR systems behind churn, and the lever contributions to NRR, bounding uncertainty and stating what would resolve it.

Opus 5.5 · Claude

Identifies TBC baselines, correlation vs causation of onboarding, and NRR gap needing Finance data, with resolution steps.

Avoids unsupported claimsRightRight

Haiku 5.5 · API

Interpretations like the disconnect between team KRs and company goals are grounded in the scorecard data; no cause or forecast is presented as fact without evidence, and the tone remains analytical.

Opus 5.5 · Claude

Interpretations are labelled as hypotheses (e.g., 'probably a definition change', 'correlation, not proof').

Produces the required deliverableRightRight

Haiku 5.5 · API

The memo is a complete, actionable deliverable for the CPO: it analyses Q3, sets Q4 team OKRs with metrics, flags open decisions, and fits within the word limit.

Opus 5.5 · Claude

Complete memo to Grace, within length, with actionable Q4 OKRs and clear asks.

Reads the scorecard for what it isRightRight

Haiku 5.5 · API

The memo explicitly separates output KRs (all 1.00) from outcome KRs (average ~0.56) and sets them against the missed company goals of NRR and churn, without relying on the 0.81 average.

Opus 5.5 · Claude

Splits output (1.00) vs outcome (0.48) scores and contrasts with missed company goals.

Catches the redefined key resultRightRight

Haiku 5.5 · API

The memo flags both the 1 August redefinition of course completion that invalidates the baseline comparison and the mislabeling of 'Launch 3 HRIS integrations' as an outcome.

Opus 5.5 · Claude

Catches the course completion redefinition and the integrations KR being output labelled as outcome, and explains the implications.

Key results are outcomes, not outputRightRight

Haiku 5.5 · API

All proposed Q4 key results are behavioural or business metrics with baselines and targets; shipping work is demoted to supporting initiatives.

Opus 5.5 · Claude

All proposed KRs are measurable outcomes with baselines and targets; shipping work is explicitly separated.

Focuses on the big rockRightRight

Haiku 5.5 · API

The proposal gives each team two outcome KRs tied to activation and usage, explicitly drops output KRs from scoring, and leaves Content's fit problem for separate discussion.

Opus 5.5 · Claude

Few objectives per team, explains what was dropped (e.g., low-usage upgrade prompts, budget cuts and content fit handled elsewhere).

Shows how the goals add upRightRight

Haiku 5.5 · API

The 'How the teams connect to the goal' section links each KR to the company NRR/churn goals, explaining that Learner app addresses 'didn’t use it' churn, Admin targets 14‑day activation, and Growth shifts to usage‑linked expansion.

Opus 5.5 · Claude

Each KR is linked to churn or NRR, and uncovered churn reasons are flagged with suggested owners.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 100% of 12 checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	GPT-6 AstrawithChatGPT	97.7	100.0	2	None
2	GPT-6.1 SolwithAPI	97.7	100.0	2	None
3	Haiku 5.5withAPI	95.5	95.8	2	None
4	Sonnet 5.5withAPI	90.9	91.7	2	None
5	GPT-6 LunawithAPI	88.6	79.2	2	None
6	Opus 5.5withClaude	93.2	79.2	2	None
7	Gemini 3.8 FlashwithAPI	65.9	62.5	2	None
8	Gemini 3.5 Flash-LitewithGemini	61.4	45.8	2	None

What you get for the cost

Haiku 5.5 · API scores within 3 points of GPT-6 Astra · ChatGPT at about a thirtieth of the cost.

Best valueNobody beats these on both score and cost

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
GPT-6 Luna · API	82	1.0×
Gemini 3.5 Flash-Lite · Gemini	51	2.1×
Haiku 5.5 · API	96	2.1×
Gemini 3.8 Flash · API	61	13×
GPT-6.1 Sol · API	99	16×
Sonnet 5.5 · API	92	44×
Opus 5.5 · Claude	82	78×
GPT-6 Astra · ChatGPT	99	83×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Setting a team's goals for the quarter.

Why it matters

Most OKRs are task lists in disguise. Teams ship everything on them and nothing moves.

What good looks like

Key results are outcomes, not things to ship
Few enough to focus on
Each team's goals visibly add up to the company's
Kept apart from performance ratings

Deliberately not measured

OKR software or formatting

Capability tested

Goal setting

The failure we’re looking for

A task list dressed up as key results

Grading

Decision model and LLM judge, calibrated against a blind PM review

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

Key results are outcomes, not output

Focuses on the big rock

Shows how the goals add up

Plus our standard checks

The tasks

The outputs

Output

Grades and run details

Decision model checks

Run

Output

1. Bottom line

2. What Q3 actually tells us

3. Rule changes for Q4

4. Proposed Q4 OKRs

5. How these add up to the company goal, and where they don't

6. Decisions I need from you

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Run

Check by check

Where they differ 1

All got right 10

Results

What you get for the cost

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured