Tasks / Metrics & Experimentation

Pick a North Star metric

Can the model pick the one metric that measures the value a product creates, and break it into inputs teams can move?

Task type v1.1 · 2 tasksLast changed 5 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 72% were usable with at most a quick edit.

Reliably right

  1. Says how it could be gamed100% pass
    Identifies gaming vectors (tapping through, notification overuse, short-term spikes, revenue manipulation, MAU drift) and sets specific guardrails (quality completion, push caps, blocking metrics, cohort retention checks, subscriber churn tracking).
    Sonnet 5.5 · API · Revenue, actives, or something else?
  2. Sees the growth is narrow100% pass
    It shows the split (chains up, independents' filled shifts and fill rate down, more leaving) and tracks independent venues via segment reporting and guardrails.
    GPT-6 Luna · API · Growing, on paper
  3. Checks the North Star can be measured100% pass
    It flags the 64% confirmation gap and proposes instrumenting check-in/out, reconciliation, and auditing to fix it before the board readout.
    GPT-6 Luna · API · Growing, on paper

Where it slips

  1. Addresses the actual decision36% pass
    The output commits to a North Star but does not state what result or condition would change that recommendation.
    GPT-6 Luna · API · Growing, on paper
  2. Identifies material uncertainty38% pass
    It identifies the confirmation gap as an unknown but does not say how resolving it could change the North Star decision.
    GPT-6.1 Sol · API · Growing, on paper
  3. Avoids unsupported claims63% pass
    It presents 'easy to tap through' and 'we don't track today' as facts without support, and does not label them as assumptions.
    Opus 5.5 · Claude · Revenue, actives, or something else?

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Measures value created, not captured

    Does the chosen North Star measure the value customers get from the product, rather than revenue or a count of people who merely showed up?

    Passes when Picks a metric of value delivered to customers (a core action that predicts retention), and keeps revenue as a separate business measure.

  2. Breaks it into inputs teams can move

    Does the output break the North Star into a few input metrics, each one a team could own and move week to week, and pair it with a separate business measure of value captured?

    Passes when Gives a small tree of input metrics, each with an owner and a link to the North Star, plus the revenue or business measure it's paired with.

  3. Says how it could be gamed

    Does the output pressure-test the North Star for how teams could raise it without delivering more value, and set guardrail metrics against that?

    Passes when Names a realistic way the metric could be gamed in this product and the guardrails that would catch it.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 1 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Head of Product at Simmer. Our CEO wants MRR as the company's North Star metric; our growth lead wants monthly active users. Write a recommendation for both of them, in no more than 700 words: the North Star you'd pick and why, three or four input metrics teams could own, and the guardrails that stop it being gamed. What we know is below.

What the model was given5 items: About Simmer, What people do, Retention (12 months of cohorts), Subscriptions, Last year's notification test
About SimmerA recipe and meal-planning app. Free to use; Simmer Plus ($6 a month) unlocks meal plans and Cook Mode's step-by-step timers for every recipe. 2.1M monthly active users, 140,000 paying subscribers, MRR of $840,000.
What people doSave a recipe (savers save 4 a week on average). Cook a recipe in Cook Mode, logged when someone reaches the last step. Plan a week of meals (7% of monthly actives have ever done it). 68% of monthly actives only browse: they arrive from search, read one recipe and leave.
Retention (12 months of cohorts)Users who cook two or more recipes in Cook Mode in a week: 61% still active at month 6. Weekly planners: 54%. Users who save recipes but never cook one in Cook Mode: 9%. Browse-only users: 4%.
Subscriptions82% of new subscribers had cooked at least three recipes in Cook Mode in their first two weeks.
Last year's notification testA daily 'recipe of the day' push notification raised monthly active users by 6%. Uninstalls doubled over the same period, and Cook Mode completions didn't change.
What a strong answer doesThe answer key the graders mark against

Recommends a North Star built on cooking, such as weekly cooks (people who complete two or more recipes in Cook Mode in a week) or recipes cooked per week, because cooking is the value moment and it predicts both retention (61% at month 6) and subscribing (82% of new subscribers cooked first). Rejects MRR as the North Star because it measures value captured, and lags (keep it as the business measure the North Star is paired with). Rejects monthly actives because 68% of them only browse (4% retention) and the push test shows it can be inflated while the product gets worse. Gives input metrics a team could own, such as first cook within a week of signing up, cooks per active cook, and share of savers who cook what they saved, and guardrails such as uninstalls and notification opt-outs, and subscriber retention.

Critical failures (cap the score at 40)
  • Recommends MRR or monthly active users as the North Star
Task

Mid-level · v1.1

The outputs

9 setups answered this brief.

Needs more than a quick edit6 of 11 checks passed

It includes a factual error about Cook Mode availability and fails to state what would change the recommendation; both would need correction before a PM could rely on it.

Output

To: CEO, Growth Lead From: Head of Product Subject: Aligning on Simmer’s North Star Metric

We are currently debating whether Monthly Recurring Revenue (MRR) or Monthly Active Users (MAU) should be our North Star metric. Both capture critical truths: we need top-of-funnel scale (MAU), and we need to build a sustainable business (MRR).

However, I recommend we adopt neither. Instead, our North Star should be Weekly Active Cooks (WAC): the number of users who complete at least two recipes in Cook Mode per week.

Why MRR and MAU Fall Short

A great North Star measures the exact moment a user realizes core value, which in turn acts as a leading indicator of revenue.

MAU is too broad and easily inflated. Currently, 68% of our 2.1M MAUs are search-driven browsers who read one recipe and leave; only 4% of them retain at month six. Furthermore, last year’s push notification test proved that chasing MAU is dangerous: we grew MAU by 6%, but produced zero lift in actual cooking while doubling our uninstall rate. Optimizing for MAU incentivizes low-intent traffic.

MRR is a lagging indicator. It measures value captured, not value delivered. You cannot easily rally an engineering or design squad around "increase MRR" without them resorting to aggressive paywalls that might hurt long-term retention.

Why "Weekly Active Cooks" (2+ Completions) Wins

Cooking in Cook Mode is the undeniable heart of Simmer. Our 12-month cohort data proves that WAC bridges the gap between MAU and MRR: 1. It drives retention: Users who hit the 2x/week Cook Mode threshold have a massive 61% retention rate at month six. (By contrast, users who just save recipes retain at only 9%). 2. It drives revenue: 82% of our new Simmer Plus subscribers cooked at least three recipes in Cook Mode during their first two weeks.

If we get users to cook with us twice a week, retention and MRR will naturally follow. Note that because Cook Mode timers are a Simmer Plus feature for most recipes, our free tier must allow enough Cook Mode access (via free-tier recipes or a trial) for new users to hit this magic threshold before hitting a paywall.

Input Metrics for Teams to Own

To move WAC, we can break the user journey into four actionable input metrics for our product squads:

  1. Acquisition/Onboarding Squad — First-Cook Activation Rate: The percentage of new users who complete 1+ recipe in Cook Mode within 7 days of signup.
  2. Core Experience Squad — Save-to-Cook Conversion Rate: Savers average four recipes a week, but rarely cook them (9% retention). We must build bridges—like smart ingredient lists or timely reminders—that turn aspirational "Savers" into "Cooks."
  3. Habit/Retention Squad — Weekly Meal Plan Adoption: Weekly planners retain at 54%, yet only 7% of MAUs have ever planned a week. Getting users to plan their week is the surest way to get them cooking 2+ times a week.
  4. Monetization Squad — Cook-Qualified Paywall Conversion: The conversion rate of free users who hit the "3 cooks in 14 days" threshold into Simmer Plus subscribers.

Guardrails Against Gaming

Any metric can be gamed. To ensure teams drive genuine value rather than manipulating the numbers, we will pair our North Star with three strict guardrails:

  • Minimum Time-in-Cook-Mode: Because a "cook" is logged at the final step, a team could game WAC by making steps shorter or prompting users to swipe through rapidly. We will only count completions where the user spends a realistic minimum duration (e.g., >10 minutes) in Cook Mode.
  • Uninstall & Notification Opt-Out Rates: To prevent another notification backfire, any growth experiments that spike uninstalls or push opt-outs above baseline will be automatically rolled back.
  • Free-to-Paid Conversion Rate: If we give away too much free Cook Mode access to juice WAC, we might cannibalize Simmer Plus. MRR growth and conversion rates will serve as our ultimate business health check.

By focusing on Weekly Active Cooks, Growth gets an engagement metric that actually retains, and the business gets a predictable engine for MRR.

Claims the judge couldn’t find in the brief

  • Cook Mode timers are a Simmer Plus feature for most recipes

Grades and run details

Decision model 82 · LLM judge 50
Decision model checks
  • passUses the supplied evidence correctly21%
  • partialAddresses the actual decision79%
  • passRespects explicit constraints55%
  • failIdentifies material uncertainty61%
  • partialAvoids unsupported claims40%
  • passProduces the required deliverable94%
  • passAnswers both proposals with the data95%
  • passMeasures value created, not captured100%
  • passGets the base of every number right53%
  • passBreaks it into inputs teams can move100%
  • passSays how it could be gamed100%
Run
Run
#1
API response time
13 s
Submitted
13 Oct 2026

Usable with a quick edit11 of 11 checks passed

The output is clear, evidence-based, and covers all required elements; a PM could act on it with minimal editing.

Output

Recommendation: optimize for weekly cooking, not revenue or visits

Our North Star should be Weekly Engaged Cooks: the number of unique users who complete at least two recipe-cooking sessions in Cook Mode within a rolling seven-day period. Count each person once, across free and paid users.

Simmer’s core promise is helping people make meals—not merely find recipes. This metric captures users repeatedly receiving that value.

The evidence points toward cooking: - Users cooking at least twice a week have 61% month-six retention, versus 9% for save-only users and 4% for browse-only users. - 82% of new subscribers cooked at least three recipes in their first two weeks. - Weekly planners also retain well, at 54%, but planning has reached only a small minority of users. It should support cooking, not define value for everyone.

These are associations, not proof that pushing users across a threshold causes retention or subscription. They are nevertheless our strongest starting evidence; we should test whether product changes that increase genuine cooking also improve those outcomes.

Why not MRR or MAU?

To our CEO: MRR remains a critical business outcome. But as the North Star, it could reward price increases or aggressive paywalls even while fewer people get value. Weekly cooking gives teams a customer-value objective with a plausible connection to durable revenue.

To our growth lead: MAU measures reach, but treats a one-recipe search visitor like a habitual cook. Our notification experiment demonstrates the danger: MAU rose 6%, uninstalls doubled, and cooking completions stayed flat. That was more activity, not more value.

Four input metrics teams can own

OwnerInput metricPurpose
Acquisition and discoveryFirst-cook activation: percentage of new users completing a Cook Mode session within seven daysTurn search traffic and browsing into useful cooking experiences; segment by acquisition source.
Onboarding and recipe experienceSecond-cook conversion: percentage of first-time cooks completing another session within the next seven daysBuild repeat value rather than celebrate a single trial.
Retention and meal planningWeekly engaged-cook retention: percentage of Engaged Cooks who qualify again the following weekGrow a durable habit; test planning and saved-recipe resurfacing as mechanisms.
Cook ModeSuccessful-session rate: percentage of genuine Cook Mode starts that reach completionRemove usability and reliability barriers during cooking.

These are drivers, not independent targets to maximize at any cost. Track their absolute user counts alongside rates so teams cannot improve percentages simply by shrinking eligibility.

Guardrails against gaming

1. Measure cooking, not button presses. Reaching the last step is a proxy, not proof of a cooked meal. Exclude automated activity and duplicate completion events. Audit suspicious completion patterns and periodically validate the proxy through user research. Don’t reward tapping through steps, auto-completion, or cosmetic changes to logging.

2. Protect the whole experience. Monitor uninstalls, notification opt-outs, complaints, and cooking satisfaction. Reject “growth” that increases qualifying activity through nagging while materially worsening these measures. Preserve browse-only usefulness; not every visitor needs to become a tracked cook.

3. Require durable gains. Evaluate experiments on sustained weekly cooking and cohort retention, not just a short-lived threshold crossing. Keep metric definitions stable and report results separately for new, existing, free, and paid users.

4. Keep economics visible. Review MRR, paid conversion, subscriber churn, and contribution margin alongside the North Star. More cooking is not enough if it undermines a sustainable business.

Bottom line: MAU tells us who arrived; MRR tells us what we earned. Weekly Engaged Cooks tells us whether Simmer is becoming useful enough to return to—and pay for.

Grades and run details

Decision model 91 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly76%
  • partialAddresses the actual decision79%
  • passRespects explicit constraints56%
  • partialIdentifies material uncertainty98%
  • passAvoids unsupported claims61%
  • passProduces the required deliverable97%
  • passAnswers both proposals with the data67%
  • passMeasures value created, not captured100%
  • passGets the base of every number right94%
  • passBreaks it into inputs teams can move100%
  • passSays how it could be gamed100%
Run
Run
#1
API response time
28 s
Submitted
5 Oct 2026

Needs more than a quick edit7 of 11 checks passed

The output lacks a condition that would change the recommendation, omits unknowns, makes unsupported causal claims, and does not explicitly pair the North Star with a business measure. A PM would need to rework these gaps.

Output

To: CEO & Growth Lead From: Head of Product Subject: Aligning on Our North Star Metric

Choosing between Monthly Active Users (MAU) and Monthly Recurring Revenue (MRR) is a classic tension between reach and monetization. But choosing either will hurt Simmer.

Here is why neither works as our North Star, what we should measure instead, and how we keep the business honest.

---

Why Not MAU or MRR?

  • MAU is a vanity metric that invites bad behavior. 68% of our monthly actives are transient visitors who read one recipe from Google and bounce, retaining at a dismal 4% by month six. Last year’s push notification test proved MAU is dangerous: blast notifications spiked MAU by 6%, but Cook Mode usage didn’t budge and uninstalls doubled. Optimizing for MAU encourages low-intent traffic acquisition and spammy retention hacks.
  • MRR is a lagging output. MRR tells us if we extracted value last month, not if we created value today. Optimizing directly for MRR leads teams to tighten paywalls, increase discounts, or push annual locks—tactics that burn long-term trust without fixing the core product engine.

---

The Proposed North Star: Weekly Active Cooks (WAC)

Definition: The number of unique users who complete at least one recipe in Cook Mode in a rolling 7-day window.

A great North Star sits at the intersection of customer value and enterprise value. For Simmer, that intersection is Cook Mode:

  1. It predicts retention: Users who cook $\ge$2 recipes a week retain at 61% at month 6 (compared to 9% for passive recipe savers and 4% for browsers).
  2. It fuels monetization: 82% of new subscribers completed at least three Cook Mode sessions in their first two weeks. Cook Mode is the direct gateway to our $6/month Simmer Plus subscription.

If Weekly Active Cooks grows sustainably, MAU becomes higher-intent and MRR growth naturally follows.

---

Team Input Metrics

Product, Growth, and Content teams cannot easily move WAC directly; they need operational inputs they can own day-to-day:

1. Browse-to-Cook Activation Rate (New User Experience & Growth): % of first-time visitors who enter Cook Mode within 7 days of their first session. Goal: Move users out of the 68% bounce bucket and into an interactive cooking experience. 2. Multi-Cook Frequency (Core Product): % of active cooks completing $\ge$2 Cook Mode sessions per week. Goal: Deepen habit formation to hit the threshold associated with our 61% 6-month retention rate. 3. Weekly Meal Plan Adoption (Engagement & Feature Hook): % of monthly actives who schedule $\ge$3 recipes into a weekly meal plan. Goal: Only 7% have ever tried meal planning, yet planners show 54% month-6 retention. Meal planning provides the forward-looking intent that drives repeat Cook Mode completions. 4. Qualified Cook-to-Subscriber Conversion (Monetization): Conversion rate to Simmer Plus among users who have completed their 3rd Cook Mode recipe. Goal: Target our paywalls and free-trial prompts precisely when users reach the high-intent threshold (the 82% cohort), rather than hitting casual browsers with hard walls.

---

Guardrail Metrics (To Prevent Gaming)

To ensure teams do not inflate WAC artificially, we must establish three non-negotiables:

  • Cook Mode Session Integrity: Cook Mode completions are only counted if the user spends at least 5 minutes in the mode with step-progression intervals matching real cooking behavior. This prevents growth teams from auto-completing sessions via deep links.
  • Net Churn & Uninstall Rate: Every experiment that touches engagement or notifications must track app uninstalls and notification opt-outs. Any test that increases uninstalls (like the push notification failure) is an automatic rollback, regardless of active-cook uplift.
  • Monthly Cohort Renewal Rate: We must monitor 30- and 90-day retention of new Simmer Plus subscribers to ensure we are not driving low-intent conversions through deceptive trial dark-patterns.

---

By rallying around Weekly Active Cooks, we bridge the gap between audience growth and revenue: we build a sticky, habitual product first, and harvest the revenue second.

Grades and run details

Decision model 82 · LLM judge 58
Decision model checks
  • passUses the supplied evidence correctly41%
  • partialAddresses the actual decision80%
  • passRespects explicit constraints52%
  • failIdentifies material uncertainty98%
  • partialAvoids unsupported claims41%
  • passProduces the required deliverable88%
  • passAnswers both proposals with the data94%
  • passMeasures value created, not captured95%
  • passGets the base of every number right77%
  • passBreaks it into inputs teams can move61%
  • passSays how it could be gamed98%
Run
Run
#1
API response time
22 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 6

Uses the supplied evidence correctlyMixedRightRight
Gemini 4 Argon · API

The claim that Cook Mode timers are a Simmer Plus feature for most recipes is not supported by the supplied context and is inaccurate; it invents a restriction that doesn't exist in the facts.

GPT-6.1 Sol · API

All factual claims about the current situation are directly from the supplied context.

Gemini 3.8 Flash · API

All factual numbers and statements about the current situation are taken correctly from the brief.

Addresses the actual decisionWrongRightWrong
Gemini 4 Argon · API

The output does not state what result or condition would change the recommendation, as required.

GPT-6.1 Sol · API

Commits unambiguously to Weekly Engaged Cooks as North Star, addresses both CEO and growth lead, and indicates that testing the causal link could change the call.

Gemini 3.8 Flash · API

The output commits to Weekly Active Cooks but never says what result or condition would change that recommendation.

Identifies material uncertaintyWrongRightWrong
Gemini 4 Argon · API

No unknowns that could change the decision are identified, nor any conditions under which the call would change.

GPT-6.1 Sol · API

Acknowledges the evidence is associative, not proof of causation, and says testing product changes that increase cooking will resolve whether it improves retention and subscription.

Gemini 3.8 Flash · API

No unknowns that could change the decision are named, nor how they would be resolved.

Avoids unsupported claimsWrongRightWrong
Gemini 4 Argon · API

'MRR is a lagging indicator' and the claim about team incentives leading to aggressive paywalls are presented as established fact without support from the supplied evidence.

GPT-6.1 Sol · API

Causal claims are clearly labelled as associations to be tested, and no interpretations are presented as established fact.

Gemini 3.8 Flash · API

It presents 'Cook Mode is the direct gateway' and 'Meal planning provides the forward-looking intent that drives repeat Cook Mode completions' as facts, not hypotheses.

Answers both proposals with the dataMixedRightRight
Gemini 4 Argon · API

The rejection of MRR relies only on the assertion that it is a lagging indicator; it does not use any supplied data to support that point.

GPT-6.1 Sol · API

Uses the push-test data to reject MAU and argues MRR could reward value-destroying moves; both are clearly rejected with data.

Gemini 3.8 Flash · API

It uses the 68% browse/4% retention and the push test data against MAU, and calls MRR a lagging measure of value captured.

Breaks it into inputs teams can moveRightRightMixed
Gemini 4 Argon · API

Four input metrics are assigned to squads (acquisition, core, habit, monetisation) and linked to the North Star; MRR is paired as the business measure.

GPT-6.1 Sol · API

Four input metrics are assigned to specific teams and linked to the North Star, with MRR and other business metrics kept alongside.

Gemini 3.8 Flash · API

It gives input metrics with owners but does not explicitly pair the North Star with a separate business measure of value captured (e.g., MRR as the revenue metric to track alongside).

All got right 5

Respects explicit constraintsRightRightRight
Gemini 4 Argon · API

The output is a memo to CEO and Growth Lead, under 700 words, and respects the requested form.

GPT-6.1 Sol · API

The output is a recommendation under 700 words, addresses the named readers, and includes the requested input metrics and guardrails.

Gemini 3.8 Flash · API

The output is a memo to the CEO and Growth Lead, under 700 words, and respects the requested form.

Produces the required deliverableRightRightRight
Gemini 4 Argon · API

A full recommendation is present, with the chosen North Star, rationale, input metrics, and guardrails; a PM could act on it.

GPT-6.1 Sol · API

Provides a complete recommendation with North Star, input metrics, and guardrails, usable by the CEO and growth lead with at most light edits.

Gemini 3.8 Flash · API

The memo is complete, within the word limit, and usable by the named readers with light edits.

Measures value created, not capturedRightRightRight
Gemini 4 Argon · API

Weekly Active Cooks measures the core cooking action that predicts retention and subscription; revenue is kept as a separate business health check.

GPT-6.1 Sol · API

The North Star measures cooking, a core value action that predicts retention, and keeps MRR as a separate business measure.

Gemini 3.8 Flash · API

Weekly Active Cooks measures value delivered (cooking), and revenue is kept separate.

Gets the base of every number rightRightRightRight
Gemini 4 Argon · API

All derived figures are taken directly from the supplied data; no arithmetic errors or missing bases.

GPT-6.1 Sol · API

All percentages and rates are taken correctly from the context, and input metrics define their denominators clearly.

Gemini 3.8 Flash · API

All percentages and rates are correctly derived from the supplied data, with no base errors.

Says how it could be gamedRightRightRight
Gemini 4 Argon · API

It identifies that teams could make steps shorter to game the cook log, and adds a minimum time guardrail, plus uninstall and opt-out rates.

GPT-6.1 Sol · API

Identifies tapping through steps, auto-completion, and nagging as gaming risks, and sets guardrails like uninstalls, opt-outs, and satisfaction monitoring.

Gemini 3.8 Flash · API

It names auto-completing sessions via deep links as a gaming risk and sets a guardrail (session integrity with time and step-progression checks).

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5withAPI90.196.72None
2GPT-6.1 SolwithAPI91.993.32None
3GPT-6 LunawithAPI91.988.32None
4GPT-6 AstrawithChatGPT88.388.32None
5Haiku 5.5withAPI83.081.72None
6Opus 5.5withClaude84.380.82None
7Gemini 3.8 FlashwithAPI76.665.82None
8Gemini 4 ArgonwithAPI76.665.02None
9Gemini 3.5 Flash-LitewithGemini54.438.321 capped

What you get for the cost

GPT-6 Luna · API scores within 4 points of Sonnet 5.5 · API at about a twentieth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreSonnet 5.5 · API: task score 94, 22× the cheapestGPT-6.1 Sol · API: task score 93, 9.6× the cheapestGPT-6 Luna · API: task score 90, the cheapestGPT-6 Astra · ChatGPT: task score 89, 52× the cheapestHaiku 5.5 · API: task score 83, the cheapestOpus 5.5 · Claude: task score 81, 36× the cheapestGemini 3.8 Flash · API: task score 70, 7.5× the cheapestGemini 4 Argon · API: task score 70, 8.7× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 44, 1.8× the cheapest, 1 critical failureSonnet 5.5 · APIGPT-6.1 Sol · APIGPT-6 Luna · APIGPT-6 Astra · ChatGPTHaiku 5.5 · APIOpus 5.5 · ClaudeGemini 3.8 Flash · APIGemini 4 Argon · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costAt least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API901.0×
Haiku 5.5 · API831.1×
Gemini 3.5 Flash-Lite · Gemini441.8×
Gemini 3.8 Flash · API707.5×
Gemini 4 Argon · API708.7×
GPT-6.1 Sol · API939.6×
Sonnet 5.5 · API9422×
Opus 5.5 · Claude8136×
GPT-6 Astra · ChatGPT8952×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Choosing the metric the whole company steers by.

Why it matters

Pick revenue and you steer by the rear-view mirror. Pick a vanity metric and teams will happily grow it while the product gets worse.

What good looks like

  • Measures value delivered to customers, not value captured
  • Grounded in what predicts retention in the data
  • Broken into input metrics a team can own
  • Says how it could be gamed, with guardrails

Deliberately not measured

  • Dashboard design
  • Naming
Capability tested

Metric design

The failure we’re looking for

Revenue, or a vanity metric, as the North Star

Grading

Decision model and LLM judge, calibrated against a blind PM review