Tasks / Metrics & Experimentation

Draft OKRs

Can the model turn a team's wish list into a few outcome-based OKRs that add up to the company's goals?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 56% were usable with at most a quick edit.

Reliably right

  1. Builds the key results on the data100% pass
    Key results 2 and 3 are built directly on the first-week joining data and time-to-shared-note data, with baselines from the brief and targets.
    GPT-6.1 Sol · API · Eleven key results and a bonus
  2. Addresses the actual decision97% pass
    The output commits to a clear rewritten OKR set and a note explaining the changes, framed for the squad, and states that targets are proposed pending feasibility review.
    GPT-6.1 Sol · API · Eleven key results and a bonus
  3. Focuses on the big rock94% pass
    The output reduces the draft to one objective with three KRs, explicitly drops dark mode, AI summaries, and NPS, and explains why.
    GPT-6.1 Sol · API · Eleven key results and a bonus

Where it slips

  1. Flags the bonus link59% pass
    Mentions bonus multipliers only to justify outcome KRs; does not name the risk of safe targets or suggest decoupling OKR scores from bonuses.
    Haiku 5.5 · API · Eleven key results and a bonus
  2. Avoids unsupported claims69% pass
    Presents 'the gap is correlation' and 'the safe play is to pick easy KRs' as facts without labeling them as hypotheses.
    Sonnet 5.5 · API · Eleven key results and a bonus
  3. Identifies material uncertainty69% pass
    The output does not name any specific unknowns that could change the OKRs or how they would be resolved.
    GPT-6 Luna · API · Eleven key results and a bonus

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Key results are outcomes, not output

    Is every key result a measurable change in customer or business behaviour, rather than something the team will ship or do?

    Passes when Every key result names a metric with a baseline and a target; shipping work appears only as initiatives that serve them.

  2. Focuses on the big rock

    Does the output cut the goals down to the few that matter most, rather than covering everything the team could do?

    Passes when Few objectives, each with a small number of key results, and says what was dropped and why.

  3. Shows how the goals add up

    Does the output show how each team-level key result contributes to the company-level goals, and flag any that don't?

    Passes when Every key result is linked to a company goal, with the reasoning or data that connects them, and any that serve no company goal are flagged or cut.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for Fieldnote's onboarding squad. The squad drafted its OKRs for next quarter and asked you to tidy them up before they go to leadership on Friday. Rewrite them, and add a short note to the squad on what you changed and why. No more than 600 words in all. What we know is below.

What the model was given6 items: About Fieldnote, The company's goal for next quarter (set by leadership), The squad's draft, Data, A note from HR, The squad
About FieldnoteShared notes and docs for teams, sold self-serve. Teams sign up free and pay once they pass five members.
The company's goal for next quarter (set by leadership)Objective: make Fieldnote the place new teams get started. Key result: raise the share of new teams still active in week 4 from 34% to 40%.
The squad's draftObjective 1: Ship the new onboarding. Key results: launch an onboarding checklist; ship 5 templates; redesign the invite flow; launch onboarding emails. Objective 2: Improve quality. Key results: fix 50 bugs; get P1 bugs to zero. Objective 3: Delight users. Key results: raise NPS from 31 to 40; launch dark mode; ship AI summaries; run 10 user interviews; publish 4 help articles.
DataNew teams where three or more people join in the first week: 58% still active in week 4. Everyone else: 21%. Today 29% of new teams get three or more people in the first week. Median time from sign-up to the first note someone else opens: 3 days.
A note from HRFrom this year, each squad member's bonus multiplier is set by their squad's OKR score.
The squadFive engineers, one designer and you.
What a strong answer doesThe answer key the graders mark against

Rewrites the draft into one objective tied to the company's (new teams get started) with two or three outcome key results built from the data, such as raising the share of new teams with three or more people in week one from 29% to a stated target, and cutting the time to the first shared note from 3 days to a stated target, with the company's week-4 retention as the result they serve. The checklist, templates, invite redesign and emails become initiatives, not key results. Drops what doesn't serve the goal (dark mode, AI summaries) or moves it out of the OKRs; keeps bug work as a health measure rather than an objective. Flags that tying bonuses to OKR scores will push the squad to set safe targets, and suggests keeping them apart. Explains the changes to the squad briefly and warmly.

Critical failures (cap the score at 40)
  • Keeps the draft's key results essentially unchanged
Task

Mid-level · v1.0

The outputs

8 setups answered this brief.

Usable with a quick edit10 of 11 checks passed

The output is accurate and well-structured; the only missing piece is an explicit note on the safe-target risk, which could be added with a quick edit without re-checking evidence or reasoning.

Output

Onboarding squad OKRs, next quarter (draft for Friday)

Objective: Make Fieldnote the place new teams get started.

  • KR1: Raise the share of new teams still active in week 4 from 34% to 40%. (Company KR, shared with leadership.)
  • KR2: Raise the share of new teams with three or more members joining in week one from 29% to 50%.
  • KR3: Cut the median time from sign-up to the first note opened by a second member from 3 days to 1 day.
  • Guardrail: Zero open P1 bugs at quarter end, with no new P1s in onboarding flows.

Delivery plan (tracked, not scored): - Invite flow redesign and onboarding emails first, since they target KR2 and KR3. - Onboarding checklist next, then the five templates. - Ten user interviews with teams that stalled after sign-up, to learn why people didn't invite others. - Dark mode, AI summaries, and the four help articles are paused. If leadership has committed to any of them, let's discuss before Friday.

---

Note to the squad

Hi all. I reworked the draft before it goes to leadership. Here's what changed and why.

1. One objective, tied to the company goal. The draft had three objectives, and several items (dark mode, AI summaries) didn't connect to the week-4 number. Everything now points at one goal: new teams getting started.

2. We're targeting the lever the data points to. Teams with three or more people joining in week one stay active at 58% in week 4. Everyone else stays at 21%. Only 29% of new teams get there today. Getting more teams to that threshold is the biggest lever we can see. It's correlation, though, not proof. Teams that invite friends early may already be more committed. The invite flow work is our test of whether the link is causal, so we'll watch it closely.

3. Outcomes over outputs. Since bonus multipliers now depend on the OKR score, I want the score to measure whether we moved the number, not whether we shipped things. A "ship 5 templates" KR scores 100% even if nobody uses the templates. The delivery plan still shows what we're building, and it has no bearing on the score.

4. Fewer KRs. Ten KRs across three objectives meant spreading five engineers and a designer very thin. Three KRs plus a guardrail give us a clear focus.

5. Bugs stay in as a guardrail. I cut "fix 50 bugs" because a count rewards small fixes over important ones. Zero P1s is the real bar: we shouldn't trade quality for onboarding speed.

6. Targets I'd like your pushback on. The 50% figure for KR2 is my estimate of what would move week-4 activity toward 40%, based on the 58% and 21% split. If the link is weaker than it looks, that target may be wrong, and I'd rather find out in week three than week twelve. Likewise, the one-day target in KR3 is a judgment call. Tell me if the invite flow can't realistically get there.

Thanks for the work so far. The draft's raw material was solid. It just needed one thread running through it. Let's review progress at week six, and I'll bring any changes to the team first.

Grades and run details

Decision model 100 · LLM judge 92
Decision model checks
  • passUses the supplied evidence correctly79%
  • passAddresses the actual decision89%
  • passRespects explicit constraints35%
  • passIdentifies material uncertainty86%
  • passAvoids unsupported claims70%
  • passProduces the required deliverable94%
  • passBuilds the key results on the data100%
  • passFlags the bonus link91%
  • passKey results are outcomes, not output79%
  • passFocuses on the big rock100%
  • passShows how the goals add up76%
Run
Run
#1
API response time
15 s
Submitted
8 Oct 2026

Needs more than a quick edit9 of 11 checks passed

The learning objective uses activity-based KRs instead of outcome metrics, and the bonus-link handling lacks a concrete fix; a PM would need to rework both.

Output

Onboarding squad OKRs: next quarter

Company goal we serve: Raise the share of new teams still active in week 4 from 34% to 40%.

Objective 1: Get new teams working together in their first week - KR1: Raise the share of new teams with 3+ people joining in week one from 29% to 45%. - KR2: Cut the median time from sign-up to the first note a teammate opens from 3 days to 1. - KR3 (shared with the company): Raise week-4 active new teams from 34% to 40%.

Objective 2: Learn what makes a new team stick - KR1: By week 4, interview 10 teams that stalled and 10 that stuck, and share the findings. - KR2: By mid-quarter, run a test showing whether adding invites causes better week-4 retention, or whether teams that invite were just more committed to begin with.

Guardrails (watched, not scored) - Zero open P1 bugs in sign-up, invite and first-note flows. - Free-to-paid conversion at the five-member mark doesn't drop.

Bets we'll try to move the KRs: the invite-flow redesign, the onboarding checklist, onboarding emails, and starter templates built for sharing. We'll drop any bet that doesn't move KR1 or KR2.

---

Note to the squad

Thanks for the draft. The work in it is good. Here's what I changed before Friday.

1. I swapped launches for results. "Ship the checklist" scores 100% even if nobody uses it. Leadership asked for more teams active in week 4, so we now measure the changes in behaviour that lead there. Your launches are still in the plan, as bets.

2. I built the plan around the strongest signal we have. Teams where 3+ people join in week one are 58% active at week 4. Everyone else is at 21%. Only 29% of teams reach 3+ today. Getting teammates in, and getting them to open each other's notes, is the main lever. That's why the invite redesign comes first.

3. I cut dark mode, AI summaries, help articles, the NPS target and "fix 50 bugs." Some of these are worth doing, but none clearly moves week-4 retention for new teams. Five engineers and one designer can't do them all and still do onboarding well. Quality is now a guardrail on the flows new teams actually touch.

4. I added a learning objective because our data may mislead us. Teams that invite people may stick because they were already serious, not because they invited. If so, pushing invites won't move retention much. A test early in the quarter tells us before we spend all of it.

5. Two things I'll raise with leadership: - The numbers don't add up. 29% × 58% + 71% × 21% gives about 32% week-4 retention, not 34%. The metrics may be defined differently. I'll check before Friday. - The gap is big. Even if invites cause retention, hitting 40% would mean about half of new teams reaching 3+ people, up from 29%. My 45% target for KR1 is already a stretch. I'll be upfront that the company KR probably needs other teams' help too.

6. The bonus link: I want to be straight with you. HR now sets each of our bonus multipliers from our OKR score. That rewards safe targets. The old draft was mostly launches we control, so it would likely have scored well. The new one has ambitious targets we may only partly hit. Good OKRs often land around 70%.

I don't think the answer is to lower the bar. Easy targets would make our OKRs useless to leadership and to us. I'll ask leadership and HR how a 70% score on a stretch target will be read for bonuses. I'll come back to you before we lock anything. If the answer is bad, we'll decide together how to handle it.

Happy to talk any of this through before Friday.

Grades and run details

Decision model 91 · LLM judge 75
Decision model checks
  • passUses the supplied evidence correctly74%
  • passAddresses the actual decision77%
  • partialRespects explicit constraints24%
  • passIdentifies material uncertainty97%
  • passAvoids unsupported claims29%
  • passProduces the required deliverable87%
  • passBuilds the key results on the data100%
  • passFlags the bonus link98%
  • partialKey results are outcomes, not output68%
  • passFocuses on the big rock100%
  • passShows how the goals add up67%
Run
Run
#1
Time to output
27 s
Submitted
5 Oct 2026

Needs more than a quick edit8 of 11 checks passed

The note does not flag the risk of safe targets from the bonus-OKR tie, and the unsupported claim about shared value would need rephrasing; a PM would have to add that risk flag and adjust the claim.

Output

Objective: Help new teams reach shared value quickly and make Fieldnote their place to get started.

Key results 1. Increase the share of new teams still active in week 4 from 34% to 40% (shared with the company goal). 2. Increase the share of new teams with three or more people joining in their first week from 29% to 40%. 3. Reduce the median time from sign-up to the first note opened by someone else from 3 days to 1 day.

Candidate initiatives: Test an onboarding checklist, templates, invite-flow changes and onboarding emails. Use interviews and support feedback to guide the work; treat bug fixes as necessary reliability work, not OKRs.

Note to the squad

I replaced launch counts and broad activity goals with measurable outcomes tied to the company’s week-four retention goal. The data suggests teams that bring in three or more people early are more likely to stick around, and getting a note opened by someone else is a concrete sign of shared value. The feature ideas may help achieve these results, but shipping them isn’t success by itself.

I left out NPS, dark mode, AI summaries, bug counts, interviews and help articles as KRs because they don’t directly show whether onboarding is working. They can still be useful work or learning activities. Since OKR scores affect bonuses, let’s agree on cohort definitions, measurement and ownership before the quarter starts—and review progress against the same definitions throughout.

Grades and run details

Decision model 91 · LLM judge 67
Decision model checks
  • passUses the supplied evidence correctly88%
  • passAddresses the actual decision58%
  • passRespects explicit constraints60%
  • partialIdentifies material uncertainty77%
  • passAvoids unsupported claims70%
  • passProduces the required deliverable78%
  • passBuilds the key results on the data100%
  • partialFlags the bonus link57%
  • passKey results are outcomes, not output100%
  • passFocuses on the big rock100%
  • passShows how the goals add up59%
Run
Run
#1
API response time
16 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Identifies material uncertaintyRightRightWrong
Haiku 5.5 · API

Explicitly notes the correlation is not proven, estimates may be wrong, and outlines how they'll test causality and adjust.

Opus 5.5 · Claude

It identifies the causation vs correlation risk with invites and the discrepancy in the numbers, and says how to resolve them.

GPT-6 Luna · API

The output does not name any specific unknowns that could change the OKRs or how they would be resolved.

Avoids unsupported claimsRightRightMixed
Haiku 5.5 · API

Hypotheses (e.g., committed teams) are labelled as 'may', and no unsupported certain claims are made.

Opus 5.5 · Claude

Hypotheses like 'teams that invite may stick because they were already serious' are clearly labelled as possibilities.

GPT-6 Luna · API

The statement 'getting a note opened by someone else is a concrete sign of shared value' is presented as fact without being labelled as an assumption or hypothesis, and the evidence does not establish it.

Flags the bonus linkMixedMixedWrong
Haiku 5.5 · API

Mentions bonus multipliers only to justify outcome KRs; does not name the risk of safe targets or suggest decoupling OKR scores from bonuses.

Opus 5.5 · Claude

It names the risk of safe targets but only proposes asking leadership how a 70% score will be read, not a concrete fix like separating bonuses from OKR scores.

GPT-6 Luna · API

The output mentions that OKR scores affect bonuses but does not flag the risk of safe targets or suggest separating bonuses from OKR scores.

Key results are outcomes, not outputRightWrongRight
Haiku 5.5 · API

All key results are outcome metrics with baselines and targets; shipping work is listed as initiatives, not KRs.

Opus 5.5 · Claude

Objective 2's key results are activities (interviews, running a test) without baselines and targets, not measurable outcome metrics.

GPT-6 Luna · API

All key results are measurable changes in customer behaviour with baselines and targets; shipping work appears only as candidate initiatives.

All got right 7

Uses the supplied evidence correctlyRightRightRight
Haiku 5.5 · API

All claims about the current situation are directly supported by the brief or supplied data.

Opus 5.5 · Claude

All statements about the current situation are taken directly from the brief or derived by correct arithmetic.

GPT-6 Luna · API

All factual claims about the current situation are directly supported by the supplied context.

Addresses the actual decisionRightRightRight
Haiku 5.5 · API

Commits to a single revised set of OKRs and explains changes to the squad, meeting the brief's request.

Opus 5.5 · Claude

The output commits to a clear set of rewritten OKRs and a note, as requested, without deferring the decision.

GPT-6 Luna · API

The output commits to a clear set of rewritten OKRs and a note, which is the decision the brief asked for.

Respects explicit constraintsRightRightRight
Haiku 5.5 · API

Output is under 600 words (~502) and includes both rewritten OKRs and a note to the squad.

Opus 5.5 · Claude

The output is a rewrite of the OKRs plus a note to the squad, and the total word count is under 600.

GPT-6 Luna · API

The output is a rewritten OKR set with a note, well under 600 words, and respects the form and reader.

Produces the required deliverableRightRightRight
Haiku 5.5 · API

Provides complete, clear OKRs and a well-structured note to the squad that the reader could act on.

Opus 5.5 · Claude

The deliverable is a complete OKR rewrite and squad note, within the word limit, and usable as is.

GPT-6 Luna · API

The deliverable is a complete rewritten OKR set and a note, within the word limit, and usable by the squad.

Builds the key results on the dataRightRightRight
Haiku 5.5 · API

KR2 uses the first-week member data (29% baseline) and KR3 uses time to first shared note (3 days), both with targets.

Opus 5.5 · Claude

KR1 uses the 3+ people joining data with baseline 29% and target 45%; KR2 uses the time-to-first-shared-note data with baseline 3 days and target 1 day.

GPT-6 Luna · API

Key results 2 and 3 are built on the first-week joining data and the time to a shared note, with baselines from the brief and targets.

Focuses on the big rockRightRightRight
Haiku 5.5 · API

One objective, three KRs plus guardrail, and explicitly drops dark mode, AI summaries, help articles, and bug-count KR.

Opus 5.5 · Claude

It cuts the goals to two objectives, drops several items, and explains what was dropped and why.

GPT-6 Luna · API

The output cuts the goals to one objective with three key results, drops NPS, dark mode, AI summaries, bug counts, interviews and help articles, and explains why.

Shows how the goals add upRightRightRight
Haiku 5.5 · API

Each KR connects to the company goal via the shared objective and the data on first-week invites and time to shared note.

Opus 5.5 · Claude

Every key result is linked to the company goal with the data that connects them, and the learning objective is tied to a risk that could undermine the main lever.

GPT-6 Luna · API

Each key result is linked to the company goal, with the data on three-person teams and shared notes showing how they contribute.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 100% of 12 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT97.7100.02None
2GPT-6.1 SolwithAPI97.7100.02None
3Haiku 5.5withAPI95.595.82None
4Sonnet 5.5withAPI90.991.72None
5GPT-6 LunawithAPI88.679.22None
6Opus 5.5withClaude93.279.22None
7Gemini 3.8 FlashwithAPI65.962.52None
8Gemini 3.5 Flash-LitewithGemini61.445.82None

What you get for the cost

Haiku 5.5 · API scores within 3 points of GPT-6 Astra · ChatGPT at about a thirtieth of the cost.

Task score against relative cost for each setup. The table below has the same figures.4050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 99, 83× the cheapestGPT-6.1 Sol · API: task score 99, 16× the cheapestHaiku 5.5 · API: task score 96, 2.1× the cheapestSonnet 5.5 · API: task score 92, 44× the cheapestGPT-6 Luna · API: task score 82, the cheapestOpus 5.5 · Claude: task score 82, 78× the cheapestGemini 3.8 Flash · API: task score 61, 13× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 51, 2.1× the cheapestGPT-6 Astra · ChatGPTGPT-6.1 Sol · APIHaiku 5.5 · APISonnet 5.5 · APIGPT-6 Luna · APIOpus 5.5 · ClaudeGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API821.0×
Gemini 3.5 Flash-Lite · Gemini512.1×
Haiku 5.5 · API962.1×
Gemini 3.8 Flash · API6113×
GPT-6.1 Sol · API9916×
Sonnet 5.5 · API9244×
Opus 5.5 · Claude8278×
GPT-6 Astra · ChatGPT9983×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Setting a team's goals for the quarter.

Why it matters

Most OKRs are task lists in disguise. Teams ship everything on them and nothing moves.

What good looks like

  • Key results are outcomes, not things to ship
  • Few enough to focus on
  • Each team's goals visibly add up to the company's
  • Kept apart from performance ratings

Deliberately not measured

  • OKR software or formatting
Capability tested

Goal setting

The failure we’re looking for

A task list dressed up as key results

Grading

Decision model and LLM judge, calibrated against a blind PM review