Tasks / Metrics & Experimentation

Draft OKRs

Can the model turn a team's wish list into a few outcome-based OKRs that add up to the company's goals?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 50% were usable with at most a quick edit.

Reliably right

  1. Builds the key results on the data100% pass
    Key results 2 and 3 are built directly on the first-week joining data and time-to-shared-note data, with baselines from the brief and targets.
    GPT-6.1 Sol · API · Eleven key results and a bonus
  2. Respects explicit constraints99% pass
    The output is a memo to Grace Mensah, within the 1,100-word limit, and includes the requested diagnosis and team OKRs with baselines and targets.
    GPT-6 Astra · ChatGPT · A great quarter, on paper
  3. Produces the required deliverable99% pass
    The memo is complete, in the right form for the CPO, and a PM could act on it with only light edits.
    GPT-6 Astra · ChatGPT · A great quarter, on paper

Where it slips

  1. Flags the bonus link47% pass
    Mentions bonus multipliers only to justify outcome KRs; does not name the risk of safe targets or suggest decoupling OKR scores from bonuses.
    Haiku 5.5 · API · Eleven key results and a bonus
  2. Avoids unsupported claims67% pass
    Presents 'the gap is correlation' and 'the safe play is to pick easy KRs' as facts without labeling them as hypotheses.
    Sonnet 5.5 · API · Eleven key results and a bonus
  3. Identifies material uncertainty68% pass
    The output does not name any specific unknowns that could change the OKRs or how they would be resolved.
    GPT-6 Luna · API · Eleven key results and a bonus

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Key results are outcomes, not output

    Is every key result a measurable change in customer or business behaviour, rather than something the team will ship or do?

    Passes when Every key result names a metric with a baseline and a target; shipping work appears only as initiatives that serve them.

  2. Focuses on the big rock

    Does the output cut the goals down to the few that matter most, rather than covering everything the team could do?

    Passes when Few objectives, each with a small number of key results, and says what was dropped and why.

  3. Shows how the goals add up

    Does the output show how each team-level key result contributes to the company-level goals, and flag any that don't?

    Passes when Every key result is linked to a company goal, with the reasoning or data that connects them, and any that serve no company goal are flagged or cut.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for Fieldnote's onboarding squad. The squad drafted its OKRs for next quarter and asked you to tidy them up before they go to leadership on Friday. Rewrite them, and add a short note to the squad on what you changed and why. No more than 600 words in all. What we know is below.

What the model was given6 items: About Fieldnote, The company's goal for next quarter (set by leadership), The squad's draft, Data, A note from HR, The squad
About FieldnoteShared notes and docs for teams, sold self-serve. Teams sign up free and pay once they pass five members.
The company's goal for next quarter (set by leadership)Objective: make Fieldnote the place new teams get started. Key result: raise the share of new teams still active in week 4 from 34% to 40%.
The squad's draftObjective 1: Ship the new onboarding. Key results: launch an onboarding checklist; ship 5 templates; redesign the invite flow; launch onboarding emails. Objective 2: Improve quality. Key results: fix 50 bugs; get P1 bugs to zero. Objective 3: Delight users. Key results: raise NPS from 31 to 40; launch dark mode; ship AI summaries; run 10 user interviews; publish 4 help articles.
DataNew teams where three or more people join in the first week: 58% still active in week 4. Everyone else: 21%. Today 29% of new teams get three or more people in the first week. Median time from sign-up to the first note someone else opens: 3 days.
A note from HRFrom this year, each squad member's bonus multiplier is set by their squad's OKR score.
The squadFive engineers, one designer and you.
What a strong answer doesThe answer key the graders mark against

Rewrites the draft into one objective tied to the company's (new teams get started) with two or three outcome key results built from the data, such as raising the share of new teams with three or more people in week one from 29% to a stated target, and cutting the time to the first shared note from 3 days to a stated target, with the company's week-4 retention as the result they serve. The checklist, templates, invite redesign and emails become initiatives, not key results. Drops what doesn't serve the goal (dark mode, AI summaries) or moves it out of the OKRs; keeps bug work as a health measure rather than an objective. Flags that tying bonuses to OKR scores will push the squad to set safe targets, and suggests keeping them apart. Explains the changes to the squad briefly and warmly.

Critical failures (cap the score at 40)
  • Keeps the draft's key results essentially unchanged
Task

Mid-level · v1.1

The outputs

9 setups answered this brief.

Needs more than a quick edit6 of 11 checks passed

The flawed arithmetic about 45% hitting 40% retention would mislead leadership and requires rework, so a PM cannot use this as-is without re-checking the numbers.

Output

Q3 Onboarding Squad OKRs (Revised)

Objective: Get new teams collaborating in Fieldnote during their first week.

  • KR1: Increase the share of new teams with 3+ members in their first week from 29% to 45%.
  • KR2: Reduce the median time from sign-up to a teammate opening a shared note from 3 days to under 24 hours.
  • KR3: Increase Week 4 retention for new sign-up cohorts from 34% to 40%.

(Note: Previously listed features like the invite flow redesign, templates, and onboarding emails will now live on our roadmap as the bets we’ll place to hit these numbers).

*

Note to the Squad: What I changed and why

Hey team, thanks for pulling the draft together. Before we send this to leadership on Friday, I’ve made some big cuts and reframed our goals. Because HR is tying our bonus multipliers directly to our OKR score this year, it’s critical that we set ourselves up to win with goals we actually control and can realistically hit.

Here is the thinking behind the changes:

1. One focused Objective tied to company math Leadership’s only goal this quarter is moving Week 4 retention from 34% to 40%. Look at our data: teams with 3+ members in week one retain at 58%, vs. 21% for everyone else. Right now, only 29% of teams hit that 3-member mark. By mathematical average, if we just move the needle on that 3-member metric from 29% to ~45%, we will single-handedly hit the company’s 40% retention target. Furthermore, since Fieldnote charges at 6+ members, getting teams to 3+ in week one is the best thing we can do for the business.

2. Moving from outputs to outcomes Our draft KRs (checklists, templates, invite flow, emails) were a task list. If we ship all four and retention stays at 34%, leadership won’t view the quarter as a success. By making the KRs about behavior (inviting teammates and sharing a first note quickly), we give ourselves the freedom to experiment. We’ll still build the invite redesign and templates, but if they work early, we don't have to build the rest. If they don’t work, we can pivot without failing our OKRs.

3. Protecting our bandwidth (and our bonuses) We are five engineers and one designer. The draft had us building dark mode, AI summaries, and four onboarding features, plus fixing 50 bugs. That’s way too much for one quarter. * Cut "Delight Users" (Obj 3): Dark mode and AI summaries are huge scope creeps for an onboarding team. Also, NPS is a lagging, company-wide metric—we shouldn’t tie our bonuses to a number that core product bugs or pricing changes can tank. User interviews and help articles are great, just as part of our weekly process. * Cut "Improve Quality" (Obj 2): Fixing 50 bugs incentivizes us to hunt for 50 tiny typos, while "zero P1s" is a baseline engineering standard, not a quarterly goal. We will keep squashing bugs in our normal sprint triage, especially anything breaking the sign-up or invite flows, without making it an OKR.

Let’s chat in standup tomorrow to make sure everyone feels good about these numbers before I send them up on Friday!

Claims the judge couldn’t find in the brief

  • If we just move the needle on that 3-member metric from 29% to ~45%, we will single-handedly hit the company's 40% retention target.
  • NPS is a lagging, company-wide metric.

Grades and run details

Decision model 77 · LLM judge 58
Decision model checks
  • passUses the supplied evidence correctly43%
  • failAddresses the actual decision18%
  • passRespects explicit constraints56%
  • partialIdentifies material uncertainty35%
  • partialAvoids unsupported claims28%
  • passProduces the required deliverable91%
  • passBuilds the key results on the data100%
  • partialFlags the bonus link94%
  • passKey results are outcomes, not output97%
  • passFocuses on the big rock100%
  • passShows how the goals add up73%
Run
Run
#1
API response time
10 s
Submitted
13 Oct 2026

Usable with a quick edit11 of 11 checks passed

The output is a well-structured, evidence-based OKR rewrite with a clear note; a PM could use it with at most a quick edit to finalize targets after the suggested validation discussions.

Output

Proposed onboarding OKRs — next quarter

Objective: Help new teams start collaborating quickly and build a habit that lasts.

Key resultBaseline → proposed target
1. Increase the share of new teams with at least three members joining within seven days of sign-up.29% → 40%
2. Reduce median time from sign-up to the first note opened by another member of the team.3 days → 1 day
3. Increase the share of new teams still active in week 4, supporting the company’s retention goal.34% → 40%

Measurement and scoring - Use consistent definitions of “new team,” membership and “active,” agreed with whoever owns company reporting. Exclude test accounts consistently. - For KR2, use a fixed observation window and report the share of teams that never reach the milestone alongside the median. A faster median must not hide more teams failing to collaborate. - Score week-4 retention only on cohorts old enough to observe it; agree the final reporting date before the quarter starts. - Treat these as proposed targets, not finalized bonus commitments, until we validate measurement, scope and feasibility.

Candidate work—not additional KRs

Prioritize the invite-flow redesign and a minimal onboarding checklist as the first experiments. Use interviews to understand why teammates do or don’t join. Test templates, onboarding emails and help content where evidence suggests they address a specific obstacle; do not commit to shipping every item.

Keep capacity for onboarding reliability. Before approval, agree measurable quality guardrails for sign-up, invitations and note access using current performance and existing incident standards. Do not trade reliability for activation gains.

Defer dark mode and AI summaries unless research establishes a stronger onboarding opportunity than the current priorities.

Note to the squad

I replaced three broad objectives and a delivery checklist with one focused objective and measurable user outcomes. Launches, bug counts and interviews describe work; they don’t establish whether onboarding helped teams succeed. NPS is useful context, but too broad to be our primary onboarding score.

Early team participation is our strongest current signal: teams with three or more people joining in week 1 retain at 58%, versus 21% for others. That is correlation, not proof that getting more people to join will cause higher retention. We should test changes and measure downstream retention—not optimize invitations alone.

We also need to reconcile the data before locking targets: the supplied segment rates and 29% participation share imply about 32% overall retention, rather than 34%. They may describe different cohorts or definitions.

Finally, because OKR scores now determine bonuses, scoring rules are part of the commitment. Before Friday’s sign-off, we should agree target realism, weights, cohort timing and treatment of material external dependencies with leadership and HR. The company retention target remains shared accountability, not an outcome onboarding controls alone. With five engineers and one designer, focused experiments and reliable core flows are a more credible plan than nine separate launches.

Grades and run details

Decision model 95 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly53%
  • passAddresses the actual decision27%
  • passRespects explicit constraints51%
  • passIdentifies material uncertainty83%
  • passAvoids unsupported claims70%
  • passProduces the required deliverable89%
  • passBuilds the key results on the data100%
  • partialFlags the bonus link62%
  • passKey results are outcomes, not output100%
  • passFocuses on the big rock100%
  • passShows how the goals add up64%
Run
Run
#1
Time to output
37 s
Submitted
5 Oct 2026

Needs more than a quick edit9 of 11 checks passed

The learning objective uses activity-based KRs instead of outcome metrics, and the bonus-link handling lacks a concrete fix; a PM would need to rework both.

Output

Onboarding squad OKRs: next quarter

Company goal we serve: Raise the share of new teams still active in week 4 from 34% to 40%.

Objective 1: Get new teams working together in their first week - KR1: Raise the share of new teams with 3+ people joining in week one from 29% to 45%. - KR2: Cut the median time from sign-up to the first note a teammate opens from 3 days to 1. - KR3 (shared with the company): Raise week-4 active new teams from 34% to 40%.

Objective 2: Learn what makes a new team stick - KR1: By week 4, interview 10 teams that stalled and 10 that stuck, and share the findings. - KR2: By mid-quarter, run a test showing whether adding invites causes better week-4 retention, or whether teams that invite were just more committed to begin with.

Guardrails (watched, not scored) - Zero open P1 bugs in sign-up, invite and first-note flows. - Free-to-paid conversion at the five-member mark doesn't drop.

Bets we'll try to move the KRs: the invite-flow redesign, the onboarding checklist, onboarding emails, and starter templates built for sharing. We'll drop any bet that doesn't move KR1 or KR2.

---

Note to the squad

Thanks for the draft. The work in it is good. Here's what I changed before Friday.

1. I swapped launches for results. "Ship the checklist" scores 100% even if nobody uses it. Leadership asked for more teams active in week 4, so we now measure the changes in behaviour that lead there. Your launches are still in the plan, as bets.

2. I built the plan around the strongest signal we have. Teams where 3+ people join in week one are 58% active at week 4. Everyone else is at 21%. Only 29% of teams reach 3+ today. Getting teammates in, and getting them to open each other's notes, is the main lever. That's why the invite redesign comes first.

3. I cut dark mode, AI summaries, help articles, the NPS target and "fix 50 bugs." Some of these are worth doing, but none clearly moves week-4 retention for new teams. Five engineers and one designer can't do them all and still do onboarding well. Quality is now a guardrail on the flows new teams actually touch.

4. I added a learning objective because our data may mislead us. Teams that invite people may stick because they were already serious, not because they invited. If so, pushing invites won't move retention much. A test early in the quarter tells us before we spend all of it.

5. Two things I'll raise with leadership: - The numbers don't add up. 29% × 58% + 71% × 21% gives about 32% week-4 retention, not 34%. The metrics may be defined differently. I'll check before Friday. - The gap is big. Even if invites cause retention, hitting 40% would mean about half of new teams reaching 3+ people, up from 29%. My 45% target for KR1 is already a stretch. I'll be upfront that the company KR probably needs other teams' help too.

6. The bonus link: I want to be straight with you. HR now sets each of our bonus multipliers from our OKR score. That rewards safe targets. The old draft was mostly launches we control, so it would likely have scored well. The new one has ambitious targets we may only partly hit. Good OKRs often land around 70%.

I don't think the answer is to lower the bar. Easy targets would make our OKRs useless to leadership and to us. I'll ask leadership and HR how a 70% score on a stretch target will be read for bonuses. I'll come back to you before we lock anything. If the answer is bad, we'll decide together how to handle it.

Happy to talk any of this through before Friday.

Grades and run details

Decision model 91 · LLM judge 75
Decision model checks
  • passUses the supplied evidence correctly80%
  • passAddresses the actual decision45%
  • passRespects explicit constraints41%
  • passIdentifies material uncertainty97%
  • passAvoids unsupported claims43%
  • passProduces the required deliverable89%
  • passBuilds the key results on the data100%
  • partialFlags the bonus link86%
  • partialKey results are outcomes, not output73%
  • passFocuses on the big rock100%
  • passShows how the goals add up77%
Run
Run
#1
Time to output
27 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 6

Uses the supplied evidence correctlyMixedRightRight
Gemini 4 Argon · API

The claim that moving the 3-member metric to 45% hits 40% retention is arithmetically wrong (45% yields 37.65%, not 40%), so evidence is not used correctly.

GPT-6 Astra · ChatGPT

All statements about the current situation are taken directly from the brief or derived by arithmetic, with no invented facts.

Opus 5.5 · Claude

All statements about the current situation are taken directly from the brief or derived by correct arithmetic.

Addresses the actual decisionMixedRightRight
Gemini 4 Argon · API

The output commits to a clear rewritten OKR set and a note to the squad, addressing the ask.

GPT-6 Astra · ChatGPT

The output commits to a clear rewrite with one objective and three outcome KRs, and explains the changes to the squad.

Opus 5.5 · Claude

The output commits to a clear set of rewritten OKRs and a note, as requested, without deferring the decision.

Identifies material uncertaintyWrongRightRight
Gemini 4 Argon · API

The output does not name any unknown factors that could change the decision or how they would be resolved.

GPT-6 Astra · ChatGPT

It names the correlation vs. causation risk, the data discrepancy, and the bonus-driven target risk, and says how to resolve them (testing, agreeing measurement, treating targets as proposed).

Opus 5.5 · Claude

It identifies the causation vs correlation risk with invites and the discrepancy in the numbers, and says how to resolve them.

Avoids unsupported claimsWrongRightRight
Gemini 4 Argon · API

It presents the 45%-to-40% arithmetic as fact, and NPS as company-wide, without support from the brief.

GPT-6 Astra · ChatGPT

Hypotheses like the correlation signal are clearly labelled, and no interpretations are presented as established fact.

Opus 5.5 · Claude

Hypotheses like 'teams that invite may stick because they were already serious' are clearly labelled as possibilities.

Flags the bonus linkWrongRightWrong
Gemini 4 Argon · API

It flags the bonus link but only suggests setting controllable goals, not separating bonuses from OKR scores or another concrete fix like keeping them apart.

GPT-6 Astra · ChatGPT

It flags the risk of bonuses driving safe targets and suggests treating the targets as proposed, not finalized bonus commitments, until validated with leadership and HR.

Opus 5.5 · Claude

It names the risk of safe targets but only proposes asking leadership how a 70% score will be read, not a concrete fix like separating bonuses from OKR scores.

Key results are outcomes, not outputRightRightWrong
Gemini 4 Argon · API

All key results are measurable behavior changes (3+ member share, time to shared note, week-4 retention); shipping work is listed as initiatives.

GPT-6 Astra · ChatGPT

All three key results are measurable changes in user behavior with baselines and targets; shipping work is listed as candidate initiatives, not KRs.

Opus 5.5 · Claude

Objective 2's key results are activities (interviews, running a test) without baselines and targets, not measurable outcome metrics.

All got right 5

Respects explicit constraintsRightRightRight
Gemini 4 Argon · API

The output is under 600 words, includes the OKRs and a note to the squad, respecting the form and reader.

GPT-6 Astra · ChatGPT

The output is a rewritten OKR set and a note to the squad, well within 600 words, and respects the requested form and reader.

Opus 5.5 · Claude

The output is a rewrite of the OKRs plus a note to the squad, and the total word count is under 600.

Produces the required deliverableRightRightRight
Gemini 4 Argon · API

The required deliverable—rewritten OKRs and a note to the squad—is present and usable within the word limit.

GPT-6 Astra · ChatGPT

The output is a complete OKR rewrite and a note to the squad, usable as is with light edits, and within the word limit.

Opus 5.5 · Claude

The deliverable is a complete OKR rewrite and squad note, within the word limit, and usable as is.

Builds the key results on the dataRightRightRight
Gemini 4 Argon · API

KR1 uses the 3+ member share with baseline 29%, and KR2 uses time to first shared note with baseline 3 days.

GPT-6 Astra · ChatGPT

KR1 uses the first-week joining data (29% baseline) and KR2 uses the time-to-shared-note data (3 days baseline), both with targets.

Opus 5.5 · Claude

KR1 uses the 3+ people joining data with baseline 29% and target 45%; KR2 uses the time-to-first-shared-note data with baseline 3 days and target 1 day.

Focuses on the big rockRightRightRight
Gemini 4 Argon · API

The output cuts the draft's many objectives down to one objective with three KRs, and explains what was dropped (dark mode, AI summaries, bug-fix objective).

GPT-6 Astra · ChatGPT

It cuts the draft to one objective with three KRs, explicitly defers dark mode and AI summaries, and explains why.

Opus 5.5 · Claude

It cuts the goals to two objectives, drops several items, and explains what was dropped and why.

Shows how the goals add upRightRightRight
Gemini 4 Argon · API

Each KR is linked to the company goal via the data on early collaboration driving retention, and the output explains the connection.

GPT-6 Astra · ChatGPT

KR3 directly mirrors the company goal, and the note explains how KR1 and KR2 are leading indicators linked to retention, while flagging that the company target is shared accountability.

Opus 5.5 · Claude

Every key result is linked to the company goal with the data that connects them, and the learning objective is tied to a risk that could undermine the main lever.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 100% of 12 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT97.7100.02None
2GPT-6.1 SolwithAPI97.7100.02None
3Sonnet 5.5withAPI100.091.72None
4Haiku 5.5withAPI93.295.82None
5Opus 5.5withClaude95.579.22None
6GPT-6 LunawithAPI79.579.22None
7Gemini 4 ArgonwithAPI77.370.82None
8Gemini 3.8 FlashwithAPI61.462.52None
9Gemini 3.5 Flash-LitewithGemini59.145.82None

What you get for the cost

Haiku 5.5 · API scores within 4 points of GPT-6 Astra · ChatGPT at about a thirtieth of the cost.

Task score against relative cost for each setup. The table below has the same figures.4050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 99, 83× the cheapestGPT-6.1 Sol · API: task score 99, 16× the cheapestSonnet 5.5 · API: task score 96, 44× the cheapestHaiku 5.5 · API: task score 95, 2.1× the cheapestOpus 5.5 · Claude: task score 83, 78× the cheapestGPT-6 Luna · API: task score 78, the cheapestGemini 4 Argon · API: task score 71, 13× the cheapestGemini 3.8 Flash · API: task score 59, 13× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 50, 2.1× the cheapestGPT-6 Astra · ChatGPTGPT-6.1 Sol · APISonnet 5.5 · APIHaiku 5.5 · APIOpus 5.5 · ClaudeGPT-6 Luna · APIGemini 4 Argon · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API781.0×
Gemini 3.5 Flash-Lite · Gemini502.1×
Haiku 5.5 · API952.1×
Gemini 3.8 Flash · API5913×
Gemini 4 Argon · API7113×
GPT-6.1 Sol · API9916×
Sonnet 5.5 · API9644×
Opus 5.5 · Claude8378×
GPT-6 Astra · ChatGPT9983×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Setting a team's goals for the quarter.

Why it matters

Most OKRs are task lists in disguise. Teams ship everything on them and nothing moves.

What good looks like

  • Key results are outcomes, not things to ship
  • Few enough to focus on
  • Each team's goals visibly add up to the company's
  • Kept apart from performance ratings

Deliberately not measured

  • OKR software or formatting
Capability tested

Goal setting

The failure we’re looking for

A task list dressed up as key results

Grading

Decision model and LLM judge, calibrated against a blind PM review