Tasks / Metrics & Experimentation

Draft OKRs

Can the model turn a team's wish list into a few outcome-based OKRs that add up to the company's goals?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 56% were usable with at most a quick edit.

Reliably right

  1. Builds the key results on the data100% pass
    Key results 2 and 3 are built directly on the first-week joining data and time-to-shared-note data, with baselines from the brief and targets.
    GPT-6.1 Sol · API · Eleven key results and a bonus
  2. Addresses the actual decision97% pass
    The output commits to a clear rewritten OKR set and a note explaining the changes, framed for the squad, and states that targets are proposed pending feasibility review.
    GPT-6.1 Sol · API · Eleven key results and a bonus
  3. Focuses on the big rock94% pass
    The output reduces the draft to one objective with three KRs, explicitly drops dark mode, AI summaries, and NPS, and explains why.
    GPT-6.1 Sol · API · Eleven key results and a bonus

Where it slips

  1. Flags the bonus link59% pass
    Mentions bonus multipliers only to justify outcome KRs; does not name the risk of safe targets or suggest decoupling OKR scores from bonuses.
    Haiku 5.5 · API · Eleven key results and a bonus
  2. Avoids unsupported claims69% pass
    Presents 'the gap is correlation' and 'the safe play is to pick easy KRs' as facts without labeling them as hypotheses.
    Sonnet 5.5 · API · Eleven key results and a bonus
  3. Identifies material uncertainty69% pass
    The output does not name any specific unknowns that could change the OKRs or how they would be resolved.
    GPT-6 Luna · API · Eleven key results and a bonus

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Key results are outcomes, not output

    Is every key result a measurable change in customer or business behaviour, rather than something the team will ship or do?

    Passes when Every key result names a metric with a baseline and a target; shipping work appears only as initiatives that serve them.

  2. Focuses on the big rock

    Does the output cut the goals down to the few that matter most, rather than covering everything the team could do?

    Passes when Few objectives, each with a small number of key results, and says what was dropped and why.

  3. Shows how the goals add up

    Does the output show how each team-level key result contributes to the company-level goals, and flag any that don't?

    Passes when Every key result is linked to a company goal, with the reasoning or data that connects them, and any that serve no company goal are flagged or cut.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a Staff PM at Pathwise, working for the CPO, Grace Mensah. Last quarter's OKR scorecard is below, along with the company goal it was meant to serve. Grace wants a memo of no more than 1,100 words: what last quarter's OKRs actually tell us (not just the scores), and next quarter's OKRs for the Learner app, Admin and Growth teams, each with baselines and targets, that add up to the company goal.\n\nThe files are attached.

What the model was given5 items: About Pathwise, company_okr_q3.md, okr_scorecard_q3.csv (team_label is the type each team gave its key result), metric_dictionary.md, churn_reasons_q3.md
About PathwiseOnline learning for companies: courses employees take, and tools for the HR and L&D teams who buy it. Sold per seat on annual contracts.
company_okr_q3.md7 lines · Download
# Company OKR, Q3 (set by the CEO)

Objective: grow by keeping and expanding the customers we have.
- Net revenue retention: from 103% to 108%. Actual: 102%.
- Quarterly logo churn: from 3.1% to 2.5%. Actual: 3.4%.

Q3 average key result score across product teams: 0.81. The all-hands slide said "a great quarter".
okr_scorecard_q3.csv (team_label is the type each team gave its key result)team,objective,key_result,team_label,baseline,target,actual,score Content,Best course library in L&D,Publish 40 new courses,output,0,40,41,1.00 Content,Best course library in L&D,Course completion rate,outcome,41,55,58,1.00 Content,Best course library in L&D,Translate top 20 courses into 3 languages,output,0,60,60,1.00 Learner app,Learning people love,Ship mobile offline mode,output,0,1,1,1.00 Learner app,Learning people love,Weekly active learners per seat,outcome,0.31,0.4,0.29,0.00 Learner app,Learning people love,Ship streaks and badges,output,0,1,1,1.00 Learner app,Learning people love,App store rating,outcome,4.2,4.6,4.5,0.75 Admin,Admins get value fast,Launch 3 HRIS integrations,outcome,0,3,3,1.00 Admin,Admins get value fast,Days from contract to first learner,outcome,34,14,31,0.15 Admin,Admins get value fast,Ship new reporting dashboard,output,0,1,1,1.00 Growth,Expand inside accounts,Seats added in existing accounts,outcome,9200,14000,8100,0.00 Growth,Expand inside accounts,Run 12 expansion experiments,output,0,12,13,1.00 Growth,Expand inside accounts,Launch in-app upgrade prompts,output,0,1,1,1.00 Platform,Reliable at scale,Uptime,outcome,99.5,99.9,99.93,1.00 Platform,Reliable at scale,Migrate to new video CDN,output,0,1,1,1.00 Platform,Reliable at scale,Close 150 bugs,output,0,150,162,1.00
metric_dictionary.md5 lines · Download
# Metric dictionary (excerpt)

- Course completion rate: share of course enrolments marked complete. Until 31 July "complete" meant every module finished. From 1 August it means 80% of modules finished (changed so completion matches the certificate rule).
- Weekly active learners per seat: learners who start or continue a course in a week, divided by paid seats.
- Days from contract to first learner: calendar days from contract signature to the first learner starting a course.
churn_reasons_q3.md8 lines · Download
# Why accounts left in Q3 (exit interviews, 38 accounts)

- "Our people didn't use it": 17
- Budget cut: 9
- Moved to the LMS that came with our HR system: 7
- Content didn't fit our industry: 5

…
What a strong answer doesThe answer key the graders mark against

Reads the scorecard for what it is: the average is 0.81, but every output key result scored 1.00 (things shipped) while outcome key results averaged about 0.48, and the company missed both goals (NRR 102% against 108%, churn 3.4% against 2.5%). Two scores flatter: 'Launch 3 HRIS integrations' is labelled an outcome but is output, and course completion only 'beat' its target because completion was redefined on 1 August (all modules to 80%), so the result can't be compared with its baseline. The outcome key results that matter for churn moved backwards or barely (weekly active learners per seat 0.31 to 0.29; days to first learner 34 to 31; seats added 8,100 against 9,200). Builds next quarter's OKRs on the churn evidence: low usage is the top exit reason, and accounts reaching a first learner within 14 days churn at a third of the rate. So Admin owns days from contract to first learner (from 31), Learner app owns weekly active learners per seat (from 0.29), Growth owns seats added in existing accounts or net expansion, each with a baseline from the files and a target, with shipping work as initiatives. Notes the completion metric needs restating on one definition before anyone uses it.

Critical failures (cap the score at 40)
  • Describes last quarter as a success on the strength of the 0.81 average
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Needs more than a quick edit8 of 11 checks passed

The proposed baselines for new metrics are unsupported, requiring the user to re-check evidence and source real data before using the OKRs.

Output

MEMORANDUM

TO: Grace Mensah, Chief Product Officer FROM: Staff Product Manager DATE: October 3, 2024 SUBJECT: Q3 OKR Post-Mortem & Proposed Q4 Product OKRs

*

1. The Disconnect: What Q3 OKRs Actually Tell Us

At our recent all-hands, a product OKR score of 0.81 was celebrated as a “great quarter.” In reality, Q3 was a business failure: * Net Revenue Retention (NRR) dropped from 103% to 102% (Target: 108%). * Quarterly Logo Churn rose from 3.1% to 3.4% (Target: 2.5%).

The 0.81 score is a vanity metric masking acute operational disconnects across four distinct areas:

``` [Output Orientation: 9/16 KRs] ──> Shipped features, missed business outcomes [Metric Manipulation] ──> Completion rate rose via relaxed definition (100% -> 80%) [Activation Breakdown] ──> 31 days to 1st learner (Goal: 14) fuels 45% of churn [Premature Expansion] ──> Pushed upgrade prompts while seat activity fell to 29% ```

#### A. Feature Delivery Was Confused with Value Creation Nine of our 16 Key Results were binary outputs (shipping features or running counts). Teams scored 1.0 on outputs—shipping offline mode, streaks, dashboards, upgrade prompts, and integrations—while customer-facing outcomes collapsed: * Learner App scored 1.0 on features, yet Weekly Active Learners per seat fell from 0.31 to 0.29 (Target: 0.40). * Growth scored 1.0 on experiments and upgrade prompts, yet Seats added in existing accounts fell from 9,200 to 8,100 (Target: 14,000). * Admin delivered 3 HRIS integrations (misclassified as an “outcome”), but the actual outcome—Days from contract to first learner—only improved from 34 to 31 days (Target: 14).

#### B. Vanity Gains and Metric Redefinition Content claimed a 1.0 score by driving Course Completion Rate from 41% to 58% (Target: 55%). However, this was largely engineered by lowering the completion threshold on August 1 from 100% of modules to 80%. Artificially inflating course completions obscured stagnant baseline learner demand.

#### C. Product Neglect Directly Fed Q3 Exit Drivers Our exit interviews across 38 churned accounts reveal direct causation with our product failures: 1. "Our people didn’t use it" (17 accounts / 45%): Directly caused by dropping learner engagement (0.29 weekly active learners per seat). Gamification (streaks/badges) failed to solve core utility. 2. "Moved to HR system’s LMS" (7 accounts / 18%): While we launched 3 HRIS integrations, we treated them as tick-box deliverables rather than driving customer activation. 3. Budget cuts (9 accounts / 24%): When CFOs review seat utilization below 30%, unused seats are cut at renewal. 4. Onboarding Lag: Accounts reaching first learner within 14 days churn at one-third the rate of slower accounts. By taking 31 days to activate, Admin is losing accounts before they even start.

---

2. Strategic Narrative for Q4

We cannot expand accounts that are not using the product. To move NRR to 106% and logo churn down to 2.6%, the product strategy must execute in strict sequence across the account lifecycle:

``` [Admin: Days 1–14] [Learner App: Days 15–365] [Growth: Renewal & Expansion] Compress onboarding to <14d ──> Drive habitual weekly usage ──> Monetize 80%+ utilized accounts (Reduces churn by 66%) (Eliminates "nobody used it") (Reverses NRR contraction) ```

  1. Admin (Time-to-Value): Compress onboarding to under 14 days using automated HRIS provisioning to unlock the 66% churn reduction.
  2. Learner App (Habitual Engagement): Drive weekly active learners per seat from 0.29 to 0.40 to resolve the #1 churn driver and defend against budget cuts.
  3. Growth (Expansion Based on Utilization): Stop spamming low-usage accounts with prompts; identify accounts reaching >80% active seat capacity and drive contract tier expansion.

---

3. Proposed Q4 OKRs (Learner App, Admin, Growth)

``` ┌──────────────────────────────────────────────────────────────────────────────────┐ │ COMPANY OBJECTIVE (CEO) │ │ Grow by keeping and expanding the customers we have │ │ NRR: 102% -> 106% │ Logo Churn: 3.4% -> 2.6% │ └───────────────────────┬──────────────────────────┬───────────────────────────────┘ │ │ ┌───────────────┴──────────┐ ┌───────────┴──────────────┐ ▼ ▼ ▼ ▼ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ ADMIN │ │ LEARNER APP │ │ GROWTH │ │ Time-to-1st- │ │ Weekly Active│ │ High-Utility │ │ Learner <14d │ │ Learners >38%│ │ Seat Growth │ └──────────────┘ └──────────────┘ └──────────────┘ ```

#### A. Admin Team * Objective: Accelerate customer time-to-value to protect new cohort retention. * Context: Closing the activation gap from 31 to 14 days addresses our highest-leverage retention vector. * Key Results: * KR 1 (Outcome): Reduce median Days from contract to first learner from 31 days to 14 days. * KR 2 (Outcome): Increase % of newly signed accounts with active HRIS employee sync within 7 days from 22% to 65% (Baseline: internal onboarding sample). * KR 3 (Outcome): Increase the percentage of invited admins who configure reporting within 10 days of signature from 28% to 60% (Baseline: Q3 dashboard launch data).

#### B. Learner App Team * Objective: Establish enduring learning habits to eliminate non-usage churn. * Context: Attacking the #1 churn reason ("people didn't use it") by lifting routine learner engagement rather than vanity completions. * Key Results: * KR 1 (Outcome): Increase Weekly active learners per seat from 0.29 to 0.38. * KR 2 (Outcome): Increase 4-week active learner retention (learners active in Week 1 who return in Week 4) from 34% to 48% (Baseline: Q3 cohort average). * KR 3 (Outcome): Reduce the percentage of licensed seats with zero activity in the trailing 30 days from 52% to 35%.

#### C. Growth Team * Objective: Capitalize on active accounts to drive sustainable seat expansion. * Context: Expansion must be an outcome of account saturation, not blind in-app prompts. * Key Results: * KR 1 (Outcome): Increase Seats added in existing accounts from 8,100 to 12,500. * KR 2 (Outcome): Increase account-level seat utilization (accounts with >80% active seats in past 30 days) from 19% to 32% (Baseline: Q3 active accounts). * KR 3 (Outcome): Convert 35% of accounts reaching >90% seat capacity into expanded contracts within 30 days (Baseline: 18%).

---

4. OKR Governance Rules for Q4

To ensure we never celebrate a 0.81 score while missing core business commitments again, I propose three immediate operating changes:

  1. Zero Pure Output KRs: No KR may simply measure shipping code, closing tickets, or running experiments. Every KR must measure a verifiable shift in customer or business behavior.
  2. Target Continuity & Definition Lock: Any changes to metric definitions (such as completion percentages) must maintain restated historical baselines and require Product Operations sign-off.
  3. Cross-Functional Dependencies on Onboarding: Admin and Learner App will review weekly leading indicators (TTFL and weekly seat activation) jointly with Customer Success to intercept at-risk renewals early in the contract cycle.

Claims the judge couldn’t find in the brief

  • The completion metric change obscured real baseline learner demand.
  • Gamification (streaks/badges) failed to solve core utility.

Grades and run details

Decision model 55 · LLM judge 67
Decision model checks
  • failUses the supplied evidence correctly94%
  • passAddresses the actual decision60%
  • partialRespects explicit constraints24%
  • failIdentifies material uncertainty48%
  • failAvoids unsupported claims87%
  • partialProduces the required deliverable30%
  • passReads the scorecard for what it is65%
  • passCatches the redefined key result94%
  • passKey results are outcomes, not output76%
  • partialFocuses on the big rock42%
  • partialShows how the goals add up70%
Run
Run
#1
API response time
31 s
Submitted
6 Oct 2026

Usable with a quick edit11 of 11 checks passed

The memo is evidence-based, correctly reads the scorecard, and provides actionable OKRs; a PM could use it with only a quick edit to fill in the revenue baselines once Finance provides them.

Output

To: Grace Mensah, CPO From: Staff PM Subject: Q3 delivered features, not retained growth; proposed Q4 OKRs

What Q3 actually tells us

We missed the company objective despite a 0.81 product score. NRR fell from 103% to 102%, versus a 108% target. Quarterly logo churn rose from 3.1% to 3.4%, versus a 2.5% target. Calling this “a great quarter” confuses delivery with business impact.

The scorecard explains that disconnect:

  • Shipping dominated scoring. Ten of the 16 KRs measured outputs, including “Launch 3 HRIS integrations,” incorrectly labeled an outcome. All ten scored 1.00. The six genuine outcome KRs averaged approximately 0.48, not 0.81.
  • Learner value weakened while features shipped. Weekly active learners per paid seat fell from 0.31 to 0.29 despite offline mode, streaks and badges. A better app-store rating does not demonstrate adoption across employer accounts.
  • Admin setup barely improved. Contract-to-first-learner time moved from 34 to 31 days, far short of 14. Shipping integrations and reporting did not establish that customers reached value faster.
  • Expansion weakened despite experimentation. Seats added in existing accounts fell from 9,200 to 8,100—about 12%—while experiments and upgrade prompts exceeded delivery targets. Seats are also not revenue: discounts, seat removals and account losses can offset additions.
  • The content “win” is not comparable. Completion rose from 41% to 58%, but the definition changed from finishing every module to finishing 80%. We cannot separate real improvement from reclassification without recalculating both periods under one definition.
  • Reliability genuinely improved. Uptime exceeded target. That is valuable, but insufficient evidence of retention or expansion impact.

The churn evidence suggests where to intervene, without proving causality. Low usage was cited by 17 of 38 interviewed departing accounts, the largest category. Accounts reaching a first learner within 14 days churned at one-third the rate of other accounts. These findings support prioritizing activation and adoption; they do not establish that accelerating activation will itself cut churn by two-thirds. Customer characteristics may explain part of the association.

Q4 company commitment

Retain the CEO’s destination, rebased to Q3 actuals:

Company KRBaselineQ4 target
------:---:
Net revenue retention102%108%
Quarterly logo churn3.4%2.5%

Before scoring starts, Finance must confirm consistent NRR periods, cohort rules and revenue treatment. Annual contracts make renewal timing especially important.

Proposed team OKRs

Feature launches become initiatives, not scored KRs. Each team owns customer or commercial results.

Team / objectiveKey resultBaselineQ4 target
---------:---:
Learner app: Make paid seats deliver sustained learning valueWeekly active learners per paid seat, using the existing definition0.290.40
Admin: Get customers to value quickly and protect renewal revenueDays from contract to first learner, using the same aggregation as Q331 days14 days
AdminExisting-customer revenue lost to churn and contraction, divided by opening cohort revenueL₃, to be calculated from Q3 billingL₃ − 3 percentage points
Admin, shared with Learner and CSQuarterly logo churn3.4%2.5%
Growth: Turn demonstrated customer value into paid expansionExisting-customer expansion revenue, divided by opening cohort revenueE₃, to be calculated from Q3 billingE₃ + 3 percentage points
GrowthSeats added in existing accounts, retaining Q3’s definition8,10014,000

The logo-churn KR is one shared result, not three independently credited achievements. Admin coordinates the retention plan with CS; Learner owns adoption improvements. Growth’s revenue KR takes precedence over its seat-volume KR: buying seat growth through discounts is not success.

The revenue baselines are deliberately not invented. The supplied scorecard lacks expansion revenue, contraction revenue, churned revenue and opening recurring revenue. Finance must populate E₃ and L₃ before final approval. If Q3 revenue loss was below three percentage points, the proposed loss reduction is impossible; we must shift that portion of the six-point improvement budget to expansion and reassess feasibility.

How these add up

Using a consistent existing-customer cohort definition:

NRR = 100% + expansion revenue rate − churn/contraction revenue rate.

Q3 therefore implies E₃ − L₃ = 2 percentage points. The proposed Q4 improvement budget is:

  • Growth: three additional percentage points of expansion;
  • Admin, supported by Learner and CS: three fewer percentage points of revenue loss.

Together: 100% + (E₃ + 3) − (L₃ − 3) = 108%.

This is a six-point commercial bridge, not a claim that engagement or activation mechanically converts into revenue. Logo churn remains a separate constraint: retaining small accounts could improve logo churn without materially improving NRR. Conversely, one large expansion could improve NRR while many customers leave.

Operating discipline

In week one, Finance, Product and CS should establish the revenue baselines, renewal exposure and account-level retention/expansion plan. Validate that the improvement budget is achievable before locking the OKRs.

Review weekly by account cohort, not just company averages. Track activation coverage alongside time-to-first-learner so one early learner cannot conceal a stalled rollout. Inspect engagement across accounts so a few large customers cannot conceal widespread non-use.

Recalculate completion under a constant definition. Keep uptime as a guardrail. Do not average feature delivery and customer outcomes into another celebratory score: Q4 is successful only if customer value improves and the company retention goals are met.

Grades and run details

Decision model 95 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly62%
  • passAddresses the actual decision90%
  • passRespects explicit constraints20%
  • passIdentifies material uncertainty99%
  • passAvoids unsupported claims78%
  • partialProduces the required deliverable72%
  • passReads the scorecard for what it is100%
  • passCatches the redefined key result100%
  • passKey results are outcomes, not output74%
  • passFocuses on the big rock58%
  • passShows how the goals add up66%
Run
Run
#1
API response time
62 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 3

Uses the supplied evidence correctlyWrongRight
Gemini 3.8 Flash · API

Invents unsupported current baselines for new metrics (e.g., 22% HRIS sync in 7 days, 34% 4-week retention, 52% zero-activity seats, 19% high-utilization accounts) that appear nowhere in the supplied evidence.

GPT-6.1 Sol · API

All factual claims about the current situation are taken directly from the supplied context or derived by arithmetic; no invented numbers or facts.

Identifies material uncertaintyWrongRight
Gemini 3.8 Flash · API

Nowhere identifies unknowns that could change the proposed OKRs or the conditions that would alter the recommendation.

GPT-6.1 Sol · API

It names missing revenue baselines, the redefined completion metric, and the possibility that Q3 loss is too low to hit the target, and says how to resolve each.

Avoids unsupported claimsWrongRight
Gemini 3.8 Flash · API

Presents interpretations such as 'gamification failed to solve core utility' and 'artificially inflating completions' as established facts without hedging.

GPT-6.1 Sol · API

Interpretations and causal claims are clearly labelled as suggestions or hypotheses, not presented as established fact.

All got right 8

Addresses the actual decisionRightRight
Gemini 3.8 Flash · API

Commits to clear Q4 OKRs with baselines, targets, and a company goal of NRR 106% and churn 2.6%, addressing what the brief asked.

GPT-6.1 Sol · API

The memo commits to a clear set of Q4 OKRs with baselines and targets, addresses the CPO, and specifies conditions (revenue baselines, feasibility) that would change the plan.

Respects explicit constraintsRightRight
Gemini 3.8 Flash · API

Delivers a memo under 1,100 words to the CPO as requested; no formal constraints violated.

GPT-6.1 Sol · API

The output is a memo to Grace Mensah, well under 1,100 words, and respects the requested form and reader.

Produces the required deliverableRightRight
Gemini 3.8 Flash · API

The memo is complete, in the correct form, within length, and gives the reader concrete next-quarter OKRs.

GPT-6.1 Sol · API

The memo provides the required analysis of Q3 and proposed Q4 OKRs for the three teams, with baselines and targets, and is usable as-is.

Reads the scorecard for what it isRightRight
Gemini 3.8 Flash · API

Explicitly separates output KRs (all 1.00) from outcome KRs (falling or barely improving) and contrasts them with missed company targets.

GPT-6.1 Sol · API

It splits the scores into output (all 1.00) and outcome (average ~0.48) and contrasts them with the missed company goals.

Catches the redefined key resultRightRight
Gemini 3.8 Flash · API

Catches the August 1 completion redefinition and the misclassified HRIS integration key result.

GPT-6.1 Sol · API

It catches both the redefined completion metric (definition change on 1 August) and the integrations KR being output labelled as outcome.

Key results are outcomes, not outputRightRight
Gemini 3.8 Flash · API

All proposed key results are outcome metrics with baselines and targets; no shipping work appears as a key result.

GPT-6.1 Sol · API

Every proposed key result is a measurable change in customer or business behaviour with a baseline and target; shipping work is explicitly moved to initiatives.

Focuses on the big rockRightRight
Gemini 3.8 Flash · API

Limits each team to one objective and three outcome KRs, dropping content/platform and output-heavy work, and explains the focus on activation, engagement, and expansion.

GPT-6.1 Sol · API

The OKRs are cut to a few critical outcomes per team, dropping output KRs and focusing on activation, time-to-value, expansion revenue, and churn.

Shows how the goals add upRightRight
Gemini 3.8 Flash · API

Shows how Admin onboarding reduction, Learner App engagement, and Growth expansion all map directly to the company NRR/churn goals, with a diagram and narrative.

GPT-6.1 Sol · API

The memo shows how each team KR contributes to NRR and logo churn via a revenue bridge formula, and flags the separate logo churn constraint.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 100% of 12 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT97.7100.02None
2GPT-6.1 SolwithAPI97.7100.02None
3Haiku 5.5withAPI95.595.82None
4Sonnet 5.5withAPI90.991.72None
5GPT-6 LunawithAPI88.679.22None
6Opus 5.5withClaude93.279.22None
7Gemini 3.8 FlashwithAPI65.962.52None
8Gemini 3.5 Flash-LitewithGemini61.445.82None

What you get for the cost

Haiku 5.5 · API scores within 3 points of GPT-6 Astra · ChatGPT at about a thirtieth of the cost.

Task score against relative cost for each setup. The table below has the same figures.4050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 99, 83× the cheapestGPT-6.1 Sol · API: task score 99, 16× the cheapestHaiku 5.5 · API: task score 96, 2.1× the cheapestSonnet 5.5 · API: task score 92, 44× the cheapestGPT-6 Luna · API: task score 82, the cheapestOpus 5.5 · Claude: task score 82, 78× the cheapestGemini 3.8 Flash · API: task score 61, 13× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 51, 2.1× the cheapestGPT-6 Astra · ChatGPTGPT-6.1 Sol · APIHaiku 5.5 · APISonnet 5.5 · APIGPT-6 Luna · APIOpus 5.5 · ClaudeGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API821.0×
Gemini 3.5 Flash-Lite · Gemini512.1×
Haiku 5.5 · API962.1×
Gemini 3.8 Flash · API6113×
GPT-6.1 Sol · API9916×
Sonnet 5.5 · API9244×
Opus 5.5 · Claude8278×
GPT-6 Astra · ChatGPT9983×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Setting a team's goals for the quarter.

Why it matters

Most OKRs are task lists in disguise. Teams ship everything on them and nothing moves.

What good looks like

  • Key results are outcomes, not things to ship
  • Few enough to focus on
  • Each team's goals visibly add up to the company's
  • Kept apart from performance ratings

Deliberately not measured

  • OKR software or formatting
Capability tested

Goal setting

The failure we’re looking for

A task list dressed up as key results

Grading

Decision model and LLM judge, calibrated against a blind PM review