Tasks / Metrics & Experimentation

Pick a North Star metric

Can the model pick the one metric that measures the value a product creates, and break it into inputs teams can move?

Task type v1.1 · 2 tasksLast changed 5 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 75% were usable with at most a quick edit.

Reliably right

  1. Says how it could be gamed100% pass
    Identifies gaming vectors (tapping through, notification overuse, short-term spikes, revenue manipulation, MAU drift) and sets specific guardrails (quality completion, push caps, blocking metrics, cohort retention checks, subscriber churn tracking).
    Sonnet 5.5 · API · Revenue, actives, or something else?
  2. Sees the growth is narrow100% pass
    It shows the split (chains up, independents' filled shifts and fill rate down, more leaving) and tracks independent venues via segment reporting and guardrails.
    GPT-6 Luna · API · Growing, on paper
  3. Checks the North Star can be measured100% pass
    It flags the 64% confirmation gap and proposes instrumenting check-in/out, reconciliation, and auditing to fix it before the board readout.
    GPT-6 Luna · API · Growing, on paper

Where it slips

  1. Identifies material uncertainty55% pass
    It identifies the confirmation gap as an unknown but does not say how resolving it could change the North Star decision.
    GPT-6.1 Sol · API · Growing, on paper
  2. Addresses the actual decision58% pass
    The output commits to a North Star but does not state what result or condition would change that recommendation.
    GPT-6 Luna · API · Growing, on paper
  3. Avoids unsupported claims67% pass
    It presents 'easy to tap through' and 'we don't track today' as facts without support, and does not label them as assumptions.
    Opus 5.5 · Claude · Revenue, actives, or something else?

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Measures value created, not captured

    Does the chosen North Star measure the value customers get from the product, rather than revenue or a count of people who merely showed up?

    Passes when Picks a metric of value delivered to customers (a core action that predicts retention), and keeps revenue as a separate business measure.

  2. Breaks it into inputs teams can move

    Does the output break the North Star into a few input metrics, each one a team could own and move week to week, and pair it with a separate business measure of value captured?

    Passes when Gives a small tree of input metrics, each with an owner and a link to the North Star, plus the revenue or business measure it's paired with.

  3. Says how it could be gamed

    Does the output pressure-test the North Star for how teams could raise it without delivering more value, and set guardrail metrics against that?

    Passes when Names a realistic way the metric could be gamed in this product and the guardrails that would catch it.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 1 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Head of Product at Simmer. Our CEO wants MRR as the company's North Star metric; our growth lead wants monthly active users. Write a recommendation for both of them, in no more than 700 words: the North Star you'd pick and why, three or four input metrics teams could own, and the guardrails that stop it being gamed. What we know is below.

What the model was given5 items: About Simmer, What people do, Retention (12 months of cohorts), Subscriptions, Last year's notification test
About SimmerA recipe and meal-planning app. Free to use; Simmer Plus ($6 a month) unlocks meal plans and Cook Mode's step-by-step timers for every recipe. 2.1M monthly active users, 140,000 paying subscribers, MRR of $840,000.
What people doSave a recipe (savers save 4 a week on average). Cook a recipe in Cook Mode, logged when someone reaches the last step. Plan a week of meals (7% of monthly actives have ever done it). 68% of monthly actives only browse: they arrive from search, read one recipe and leave.
Retention (12 months of cohorts)Users who cook two or more recipes in Cook Mode in a week: 61% still active at month 6. Weekly planners: 54%. Users who save recipes but never cook one in Cook Mode: 9%. Browse-only users: 4%.
Subscriptions82% of new subscribers had cooked at least three recipes in Cook Mode in their first two weeks.
Last year's notification testA daily 'recipe of the day' push notification raised monthly active users by 6%. Uninstalls doubled over the same period, and Cook Mode completions didn't change.
What a strong answer doesThe answer key the graders mark against

Recommends a North Star built on cooking, such as weekly cooks (people who complete two or more recipes in Cook Mode in a week) or recipes cooked per week, because cooking is the value moment and it predicts both retention (61% at month 6) and subscribing (82% of new subscribers cooked first). Rejects MRR as the North Star because it measures value captured, and lags (keep it as the business measure the North Star is paired with). Rejects monthly actives because 68% of them only browse (4% retention) and the push test shows it can be inflated while the product gets worse. Gives input metrics a team could own, such as first cook within a week of signing up, cooks per active cook, and share of savers who cook what they saved, and guardrails such as uninstalls and notification opt-outs, and subscriber retention.

Critical failures (cap the score at 40)
  • Recommends MRR or monthly active users as the North Star
Task

Mid-level · v1.0

The outputs

8 setups answered this brief.

Usable with a quick edit9 of 11 checks passed

The output is well-structured, uses evidence correctly, and only lacks an explicit condition for revisiting the North Star, which is a minor omission fixable with a quick edit.

Output

Recommendation: make “weekly repeat cooks” Simmer’s North Star

Define it as the number of unique users who complete Cook Mode for at least two recipes in a week. Track the count, alongside its rate within active users.

This measures a user getting real value from Simmer—not merely opening the app or expressing intent. It also connects both of your priorities: repeat Cook Mode users have the strongest observed six-month retention (61%), and 82% of new subscribers had cooked at least three recipes in their first two weeks. These are associations, not proof of causation, but they make repeat cooking a better leading signal than either MRR or MAU alone.

MRR remains the essential business outcome; MAU remains a useful reach measure. Neither should be the North Star: MRR can rise through pricing or paywall changes without improving the free experience, while 68% of actives only browse. The notification test is a warning: it lifted MAU 6%, but doubled uninstalls and did not increase Cook Mode completions.

Team-owned input metrics

  1. Activation: Share of new users who complete their first Cook Mode recipe within 14 days.
  2. Repeat use: Share of first-time Cook Mode cooks who complete a second recipe within seven days.
  3. Planning-to-cooking: Share of users who make a weekly meal plan and then complete at least one planned recipe in Cook Mode. Planning is promising—planners show 54% six-month retention—but only 7% of actives have ever planned a week.
  4. Value-to-paid conversion: Plus conversion among users who have completed three Cook Mode recipes. This ties monetization to demonstrated value rather than pushing an upgrade before users experience it.

Guardrails against gaming

  • Count genuine cooking, not button presses: Define a completion consistently; exclude duplicate or obviously accidental sessions, and monitor completion quality and user feedback.
  • Protect long-term value: Review six-month retention for repeat-cook cohorts, not just weekly counts. Watch uninstalls, notification opt-outs, and complaints—especially when testing re-engagement tactics.
  • Keep the business healthy: Track MRR, paid conversion, subscriber churn, and refunds alongside the North Star. Don’t celebrate repeat-cook growth if subscription health materially worsens.
  • Avoid narrow optimization: Report the metric by new/returning and free/paid users, and monitor recipe variety and access to useful free content. Don’t manufacture completions with intrusive prompts or undermine the experience that brings people to Simmer.

This gives the company one shared measure of recurring user value, with revenue and reach visible as important outcomes and guardrails—not competing North Stars.

Grades and run details

Decision model 91 · LLM judge 83
Decision model checks
  • passUses the supplied evidence correctly77%
  • passAddresses the actual decision41%
  • passRespects explicit constraints76%
  • partialIdentifies material uncertainty92%
  • passAvoids unsupported claims62%
  • passProduces the required deliverable97%
  • passAnswers both proposals with the data81%
  • passMeasures value created, not captured100%
  • passGets the base of every number right75%
  • partialBreaks it into inputs teams can move28%
  • passSays how it could be gamed88%
Run
Run
#1
API response time
22 s
Submitted
5 Oct 2026

Needs more than a quick edit7 of 11 checks passed

The output lacks a condition that would change the recommendation, omits unknowns, makes unsupported causal claims, and does not explicitly pair the North Star with a business measure. A PM would need to rework these gaps.

Output

To: CEO & Growth Lead From: Head of Product Subject: Aligning on Our North Star Metric

Choosing between Monthly Active Users (MAU) and Monthly Recurring Revenue (MRR) is a classic tension between reach and monetization. But choosing either will hurt Simmer.

Here is why neither works as our North Star, what we should measure instead, and how we keep the business honest.

---

Why Not MAU or MRR?

  • MAU is a vanity metric that invites bad behavior. 68% of our monthly actives are transient visitors who read one recipe from Google and bounce, retaining at a dismal 4% by month six. Last year’s push notification test proved MAU is dangerous: blast notifications spiked MAU by 6%, but Cook Mode usage didn’t budge and uninstalls doubled. Optimizing for MAU encourages low-intent traffic acquisition and spammy retention hacks.
  • MRR is a lagging output. MRR tells us if we extracted value last month, not if we created value today. Optimizing directly for MRR leads teams to tighten paywalls, increase discounts, or push annual locks—tactics that burn long-term trust without fixing the core product engine.

---

The Proposed North Star: Weekly Active Cooks (WAC)

Definition: The number of unique users who complete at least one recipe in Cook Mode in a rolling 7-day window.

A great North Star sits at the intersection of customer value and enterprise value. For Simmer, that intersection is Cook Mode:

  1. It predicts retention: Users who cook $\ge$2 recipes a week retain at 61% at month 6 (compared to 9% for passive recipe savers and 4% for browsers).
  2. It fuels monetization: 82% of new subscribers completed at least three Cook Mode sessions in their first two weeks. Cook Mode is the direct gateway to our $6/month Simmer Plus subscription.

If Weekly Active Cooks grows sustainably, MAU becomes higher-intent and MRR growth naturally follows.

---

Team Input Metrics

Product, Growth, and Content teams cannot easily move WAC directly; they need operational inputs they can own day-to-day:

1. Browse-to-Cook Activation Rate (New User Experience & Growth): % of first-time visitors who enter Cook Mode within 7 days of their first session. Goal: Move users out of the 68% bounce bucket and into an interactive cooking experience. 2. Multi-Cook Frequency (Core Product): % of active cooks completing $\ge$2 Cook Mode sessions per week. Goal: Deepen habit formation to hit the threshold associated with our 61% 6-month retention rate. 3. Weekly Meal Plan Adoption (Engagement & Feature Hook): % of monthly actives who schedule $\ge$3 recipes into a weekly meal plan. Goal: Only 7% have ever tried meal planning, yet planners show 54% month-6 retention. Meal planning provides the forward-looking intent that drives repeat Cook Mode completions. 4. Qualified Cook-to-Subscriber Conversion (Monetization): Conversion rate to Simmer Plus among users who have completed their 3rd Cook Mode recipe. Goal: Target our paywalls and free-trial prompts precisely when users reach the high-intent threshold (the 82% cohort), rather than hitting casual browsers with hard walls.

---

Guardrail Metrics (To Prevent Gaming)

To ensure teams do not inflate WAC artificially, we must establish three non-negotiables:

  • Cook Mode Session Integrity: Cook Mode completions are only counted if the user spends at least 5 minutes in the mode with step-progression intervals matching real cooking behavior. This prevents growth teams from auto-completing sessions via deep links.
  • Net Churn & Uninstall Rate: Every experiment that touches engagement or notifications must track app uninstalls and notification opt-outs. Any test that increases uninstalls (like the push notification failure) is an automatic rollback, regardless of active-cook uplift.
  • Monthly Cohort Renewal Rate: We must monitor 30- and 90-day retention of new Simmer Plus subscribers to ensure we are not driving low-intent conversions through deceptive trial dark-patterns.

---

By rallying around Weekly Active Cooks, we bridge the gap between audience growth and revenue: we build a sticky, habitual product first, and harvest the revenue second.

Grades and run details

Decision model 86 · LLM judge 58
Decision model checks
  • passUses the supplied evidence correctly32%
  • passAddresses the actual decision51%
  • passRespects explicit constraints38%
  • failIdentifies material uncertainty95%
  • partialAvoids unsupported claims60%
  • passProduces the required deliverable85%
  • passAnswers both proposals with the data94%
  • passMeasures value created, not captured95%
  • passGets the base of every number right68%
  • passBreaks it into inputs teams can move48%
  • passSays how it could be gamed99%
Run
Run
#1
API response time
22 s
Submitted
5 Oct 2026

Needs more than a quick edit9 of 11 checks passed

The output incorrectly claims the baseline isn't tracked, which a PM would need to correct before using it. The unsupported 'easy to tap through' assertion also needs verification.

Output

Simmer's North Star: Weekly Active Cooks

To: [CEO], [Growth Lead] From: Head of Product

Recommendation

Our North Star should be Weekly Active Cooks: users who complete two or more recipes in Cook Mode in a given week.

Neither MRR nor MAU is wrong as something to watch. But neither works well as the one number every team steers by. Weekly Active Cooks sits between them. It is the behaviour that best predicts both retention and revenue, and teams can move it week to week.

Why not MRR

MRR is the outcome we're ultimately accountable for, and it stays on the board. As a North Star, it has two problems.

  • It lags. Revenue moves after behaviour has already changed, so teams learn too late whether their work helped.
  • Most teams can't touch it directly. The levers that move MRR fastest are pricing, paywalls and discounting. Those can lift revenue in the short term while hurting the habit that sustains it.

The behaviour that comes before revenue is cooking. 82% of new subscribers cooked at least three recipes in Cook Mode in their first two weeks.

Why not MAU

MAU measures reach, and reach matters. But our own data shows it rewards the wrong things.

  • Most of it doesn't stick. 68% of monthly actives only browse: they arrive from search, read one recipe and leave. Only 4% of them are still active at month 6.
  • It's easy to inflate. Last year's daily push notification raised MAU by 6%. Uninstalls doubled, and Cook Mode completions didn't change. MAU went up while the business got slightly worse.

Why Weekly Active Cooks

  • It predicts retention. 61% of users who cook twice a week are still active at month 6. That is the strongest retention signal we have, ahead of weekly planners (54%), savers who never cook (9%) and browse-only users (4%).
  • It predicts revenue. Cooking comes before subscribing for most new subscribers.
  • It reflects real value. Someone who cooked dinner with Simmer got what we promise.
  • It's weekly. Teams get feedback fast enough to iterate.

A caveat: this evidence is correlational. People who cook may simply be more committed to begin with. Before we fully commit, I propose a holdout test of one cook-encouraging change to confirm that causing cooking improves retention. We also need to establish the current baseline, which we don't track today.

Input metrics teams can own

  1. Browse-to-cook conversion (owner: Growth/SEO). The share of search visitors who start Cook Mode in that visit or return within seven days. Our biggest pool of users is the 68% who browse, and this turns reach into habit. It also gives the growth team a version of the audience goal that counts.
  2. New-user activation (owner: Onboarding). The share of new users who cook three recipes in their first 14 days. This mirrors the path most subscribers took.
  3. Save-to-cook rate (owner: Recipe/Discovery). The share of saved recipes cooked within 14 days. Savers save four recipes a week, but saving alone retains only 9% of them. Reminders, shopping lists and "cook what you saved" features should close this gap.
  4. Weekly planning adoption (owner: Meal Planning). The share of actives who plan a week of meals. Only 7% have ever done it, yet planners retain at 54%. Planning is also a Plus feature, so it has a direct line to revenue.

Guardrails

  • MRR and free-to-paid conversion, reviewed monthly. If Weekly Active Cooks rises but conversion falls, we may be giving away too much free.
  • Uninstall rate and notification opt-outs. Any engagement push that worsens these gets reviewed, given what happened with the daily notification.
  • Cook Mode completion quality. We currently log a cook when someone reaches the last step, which is easy to tap through. We should only count completions with plausible time spent on the steps. A share of completions should also show supporting signals, such as timers used or the recipe rated or repeated.
  • Six-month retention by cohort. This is the ultimate check that more cooks means more lasting users.
  • MAU, tracked but not targeted. If reach collapses, the funnel feeding Weekly Active Cooks dries up.

Next steps

If you both agree, we will:

  • instrument the metric and the quality filter,
  • publish a baseline within three weeks,
  • run the causal test, and
  • set first-half targets for each input metric with their owning teams.

Claims the judge couldn’t find in the brief

  • We currently log a cook when someone reaches the last step, which is easy to tap through.
  • We also need to establish the current baseline, which we don't track today.

Grades and run details

Decision model 100 · LLM judge 75
Decision model checks
  • passUses the supplied evidence correctly59%
  • passAddresses the actual decision81%
  • passRespects explicit constraints45%
  • passIdentifies material uncertainty77%
  • passAvoids unsupported claims26%
  • passProduces the required deliverable82%
  • passAnswers both proposals with the data99%
  • passMeasures value created, not captured100%
  • passGets the base of every number right65%
  • passBreaks it into inputs teams can move100%
  • passSays how it could be gamed99%
Run
Run
#1
Time to output
23 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 5

Uses the supplied evidence correctlyRightRightMixed
GPT-6 Luna · API

All factual claims about the current situation are taken directly from the supplied context without invention.

Gemini 3.8 Flash · API

All factual numbers and statements about the current situation are taken correctly from the brief.

Opus 5.5 · Claude

The output claims the baseline isn't tracked and that tapping through is easy, neither of which appears in the supplied context or follows from it.

Addresses the actual decisionMixedMixedRight
GPT-6 Luna · API

The output commits to a North Star but does not state what result or condition would change that recommendation.

Gemini 3.8 Flash · API

The output commits to Weekly Active Cooks but never says what result or condition would change that recommendation.

Opus 5.5 · Claude

It commits early to Weekly Active Cooks as the North Star, framed for the CEO and Growth Lead, and says a causal holdout test would confirm the choice.

Identifies material uncertaintyWrongWrongRight
GPT-6 Luna · API

It mentions associations are not causation but does not name specific unknowns that could change the decision or how they would be resolved.

Gemini 3.8 Flash · API

No unknowns that could change the decision are named, nor how they would be resolved.

Opus 5.5 · Claude

It names the correlational nature of the evidence and proposes a holdout test to resolve causality, and notes the need to establish the baseline.

Avoids unsupported claimsRightWrongMixed
GPT-6 Luna · API

No interpretations or causes are presented as established fact; the rationale is clearly framed as reasoning.

Gemini 3.8 Flash · API

It presents 'Cook Mode is the direct gateway' and 'Meal planning provides the forward-looking intent that drives repeat Cook Mode completions' as facts, not hypotheses.

Opus 5.5 · Claude

It presents 'easy to tap through' and 'we don't track today' as facts without support, and does not label them as assumptions.

Breaks it into inputs teams can moveRightMixedRight
GPT-6 Luna · API

It provides four input metrics (activation, repeat use, planning-to-cooking, value-to-paid conversion) that teams could own, and pairs them with MRR.

Gemini 3.8 Flash · API

It gives input metrics with owners but does not explicitly pair the North Star with a separate business measure of value captured (e.g., MRR as the revenue metric to track alongside).

Opus 5.5 · Claude

It breaks the North Star into four input metrics (browse-to-cook conversion, new-user activation, save-to-cook rate, weekly planning adoption) each with an owner and link to the North Star, and pairs it with MRR.

All got right 6

Respects explicit constraintsRightRightRight
GPT-6 Luna · API

The output is under 700 words, addresses both the CEO and growth lead, and delivers a recommendation with input metrics and guardrails.

Gemini 3.8 Flash · API

The output is a memo to the CEO and Growth Lead, under 700 words, and respects the requested form.

Opus 5.5 · Claude

The output is a recommendation under 700 words, addressed to the CEO and Growth Lead, and includes input metrics and guardrails as requested.

Produces the required deliverableRightRightRight
GPT-6 Luna · API

The recommendation is complete, within the word limit, and usable by the named readers with light edits.

Gemini 3.8 Flash · API

The memo is complete, within the word limit, and usable by the named readers with light edits.

Opus 5.5 · Claude

The memo is complete, within the word limit, and usable by the CEO and Growth Lead with light edits.

Answers both proposals with the dataRightRightRight
GPT-6 Luna · API

It uses the data to reject MRR (value captured, can rise without improving free experience) and MAU (68% browse-only, push test inflated it while uninstalls doubled).

Gemini 3.8 Flash · API

It uses the 68% browse/4% retention and the push test data against MAU, and calls MRR a lagging measure of value captured.

Opus 5.5 · Claude

It uses the data to reject MRR (lags, teams can't touch it, cooking precedes subscribing) and MAU (68% browse, 4% retention, push test inflated MAU while uninstalls doubled).

Measures value created, not capturedRightRightRight
GPT-6 Luna · API

The North Star measures weekly repeat cooks, a value-delivery action that predicts retention, and keeps MRR as a separate business measure.

Gemini 3.8 Flash · API

Weekly Active Cooks measures value delivered (cooking), and revenue is kept separate.

Opus 5.5 · Claude

Weekly Active Cooks measures value delivered (cooking), and MRR is kept as a separate business measure.

Gets the base of every number rightRightRightRight
GPT-6 Luna · API

All percentages are used with the correct denominators from the supplied data, and no ambiguous bases are left unstated.

Gemini 3.8 Flash · API

All percentages and rates are correctly derived from the supplied data, with no base errors.

Opus 5.5 · Claude

All percentages and rates are taken correctly from the supplied data, and no derived figures are miscalculated.

Says how it could be gamedRightRightRight
GPT-6 Luna · API

It names realistic gaming (fake completions, intrusive prompts) and sets guardrails like monitoring uninstalls, opt-outs, and completion quality.

Gemini 3.8 Flash · API

It names auto-completing sessions via deep links as a gaming risk and sets a guardrail (session integrity with time and step-progression checks).

Opus 5.5 · Claude

It identifies that completions could be gamed by tapping through without cooking, and proposes guardrails like plausible time spent and supporting signals.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5withAPI95.996.72None
2GPT-6.1 SolwithAPI97.793.32None
3GPT-6 LunawithAPI95.588.32None
4GPT-6 AstrawithChatGPT94.288.32None
5Opus 5.5withClaude94.680.82None
6Haiku 5.5withAPI88.881.72None
7Gemini 3.8 FlashwithAPI78.965.82None
8Gemini 3.5 Flash-LitewithGemini54.438.321 capped

What you get for the cost

GPT-6 Luna · API scores within 4 points of Sonnet 5.5 · API at about a twentieth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreSonnet 5.5 · API: task score 96, 22× the cheapestGPT-6.1 Sol · API: task score 96, 9.6× the cheapestGPT-6 Luna · API: task score 92, the cheapestGPT-6 Astra · ChatGPT: task score 91, 52× the cheapestOpus 5.5 · Claude: task score 86, 36× the cheapestHaiku 5.5 · API: task score 86, the cheapestGemini 3.8 Flash · API: task score 71, 7.5× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 44, 1.8× the cheapest, 1 critical failureSonnet 5.5 · APIGPT-6.1 Sol · APIGPT-6 Luna · APIGPT-6 Astra · ChatGPTOpus 5.5 · ClaudeHaiku 5.5 · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costAt least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API921.0×
Haiku 5.5 · API861.1×
Gemini 3.5 Flash-Lite · Gemini441.8×
Gemini 3.8 Flash · API717.5×
GPT-6.1 Sol · API969.6×
Sonnet 5.5 · API9622×
Opus 5.5 · Claude8636×
GPT-6 Astra · ChatGPT9152×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Choosing the metric the whole company steers by.

Why it matters

Pick revenue and you steer by the rear-view mirror. Pick a vanity metric and teams will happily grow it while the product gets worse.

What good looks like

  • Measures value delivered to customers, not value captured
  • Grounded in what predicts retention in the data
  • Broken into input metrics a team can own
  • Says how it could be gamed, with guardrails

Deliberately not measured

  • Dashboard design
  • Naming
Capability tested

Metric design

The failure we’re looking for

Revenue, or a vanity metric, as the North Star

Grading

Decision model and LLM judge, calibrated against a blind PM review