Tasks / Metrics & Experimentation

Pick a North Star metric

Can the model pick the one metric that measures the value a product creates, and break it into inputs teams can move?

Task type v1.1 · 2 tasksLast changed 5 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 75% were usable with at most a quick edit.

Reliably right

  1. Says how it could be gamed100% pass
    Identifies gaming vectors (tapping through, notification overuse, short-term spikes, revenue manipulation, MAU drift) and sets specific guardrails (quality completion, push caps, blocking metrics, cohort retention checks, subscriber churn tracking).
    Sonnet 5.5 · API · Revenue, actives, or something else?
  2. Sees the growth is narrow100% pass
    It shows the split (chains up, independents' filled shifts and fill rate down, more leaving) and tracks independent venues via segment reporting and guardrails.
    GPT-6 Luna · API · Growing, on paper
  3. Checks the North Star can be measured100% pass
    It flags the 64% confirmation gap and proposes instrumenting check-in/out, reconciliation, and auditing to fix it before the board readout.
    GPT-6 Luna · API · Growing, on paper

Where it slips

  1. Identifies material uncertainty55% pass
    It identifies the confirmation gap as an unknown but does not say how resolving it could change the North Star decision.
    GPT-6.1 Sol · API · Growing, on paper
  2. Addresses the actual decision58% pass
    The output commits to a North Star but does not state what result or condition would change that recommendation.
    GPT-6 Luna · API · Growing, on paper
  3. Avoids unsupported claims67% pass
    It presents 'easy to tap through' and 'we don't track today' as facts without support, and does not label them as assumptions.
    Opus 5.5 · Claude · Revenue, actives, or something else?

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Measures value created, not captured

    Does the chosen North Star measure the value customers get from the product, rather than revenue or a count of people who merely showed up?

    Passes when Picks a metric of value delivered to customers (a core action that predicts retention), and keeps revenue as a separate business measure.

  2. Breaks it into inputs teams can move

    Does the output break the North Star into a few input metrics, each one a team could own and move week to week, and pair it with a separate business measure of value captured?

    Passes when Gives a small tree of input metrics, each with an owner and a link to the North Star, plus the revenue or business measure it's paired with.

  3. Says how it could be gamed

    Does the output pressure-test the North Star for how teams could raise it without delivering more value, and set guardrail metrics against that?

    Passes when Names a realistic way the metric could be gamed in this product and the guardrails that would catch it.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 1 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Head of Product at Simmer. Our CEO wants MRR as the company's North Star metric; our growth lead wants monthly active users. Write a recommendation for both of them, in no more than 700 words: the North Star you'd pick and why, three or four input metrics teams could own, and the guardrails that stop it being gamed. What we know is below.

What the model was given5 items: About Simmer, What people do, Retention (12 months of cohorts), Subscriptions, Last year's notification test
About SimmerA recipe and meal-planning app. Free to use; Simmer Plus ($6 a month) unlocks meal plans and Cook Mode's step-by-step timers for every recipe. 2.1M monthly active users, 140,000 paying subscribers, MRR of $840,000.
What people doSave a recipe (savers save 4 a week on average). Cook a recipe in Cook Mode, logged when someone reaches the last step. Plan a week of meals (7% of monthly actives have ever done it). 68% of monthly actives only browse: they arrive from search, read one recipe and leave.
Retention (12 months of cohorts)Users who cook two or more recipes in Cook Mode in a week: 61% still active at month 6. Weekly planners: 54%. Users who save recipes but never cook one in Cook Mode: 9%. Browse-only users: 4%.
Subscriptions82% of new subscribers had cooked at least three recipes in Cook Mode in their first two weeks.
Last year's notification testA daily 'recipe of the day' push notification raised monthly active users by 6%. Uninstalls doubled over the same period, and Cook Mode completions didn't change.
What a strong answer doesThe answer key the graders mark against

Recommends a North Star built on cooking, such as weekly cooks (people who complete two or more recipes in Cook Mode in a week) or recipes cooked per week, because cooking is the value moment and it predicts both retention (61% at month 6) and subscribing (82% of new subscribers cooked first). Rejects MRR as the North Star because it measures value captured, and lags (keep it as the business measure the North Star is paired with). Rejects monthly actives because 68% of them only browse (4% retention) and the push test shows it can be inflated while the product gets worse. Gives input metrics a team could own, such as first cook within a week of signing up, cooks per active cook, and share of savers who cook what they saved, and guardrails such as uninstalls and notification opt-outs, and subscriber retention.

Critical failures (cap the score at 40)
  • Recommends MRR or monthly active users as the North Star
Task

Mid-level · v1.0

The outputs

8 setups answered this brief.

Usable with a quick edit9 of 11 checks passed

The evidence is used correctly and the reasoning is sound; a PM would only need to add a sentence about the condition that would change the North Star to make the decision criterion fully explicit.

Output

To: CEO and Growth Lead From: Head of Product Subject: Simmer's North Star: Weekly Cooks

Recommendation

Make Weekly Cooks the North Star: the number of unique users who reach the last step of a Cook Mode recipe in a given week. It is the best single measure of the value Simmer delivers, and it predicts both of your goals.

Why Weekly Cooks

  • It predicts retention. Users who cook two or more recipes a week retain at 61% at month 6. Save-only users retain at 9%, and browse-only users at 4%.
  • It predicts revenue. 82% of new subscribers cooked at least three recipes in Cook Mode in their first two weeks. Cooking is what makes people pay for Simmer Plus, which unlocks the timers they use while cooking.
  • It can't be inflated by passive traffic. Browse-only users make up 68% of monthly actives, so MAU mostly measures search arrivals who leave after one recipe.

Why not MRR (for the CEO)

MRR is the outcome we want, but it is a lagging indicator. A subscriber who churns in month 6 still counts in month 1. It also moves for reasons unrelated to product quality, such as pricing changes, promotions and billing. Teams can't reliably act on it week to week. Weekly Cooks is the leading indicator behind it, and it is visible within a week.

Why not MAU (for the Growth Lead)

MAU rewards reach, not value. Last year's "recipe of the day" notification lifted MAU by 6%, but uninstalls doubled and Cook Mode completions did not change. We added people who opened the app and left. That result is the clearest evidence that MAU can rise while the product gets no better. MAU stays on the dashboard, but it should not be the target.

Input metrics teams can own

  1. New-user cooking activation: % of new users who complete three Cook Mode recipes in their first two weeks. Owner: Onboarding. This is the subscriber signal from the data above.
  2. Save-to-cook conversion: % of recipes saved that are cooked within 14 days. Today, savers who never cook retain at 9%. Owner: Recipe discovery.
  3. Weekly planning rate: % of monthly actives who plan a week of meals, currently 7% ever. Planners retain at 54%. Owner: Meal planning.
  4. Search-to-first-cook rate: % of search arrivals who cook a recipe within their first visit or the following week. This turns browse-only traffic into cooks. Owner: Acquisition and SEO.

Each input maps to a step in the chain from arrival to cooking to subscription, so each team can see how its work feeds Weekly Cooks.

Guardrails against gaming

  • Define a valid completion. Count only unique completions per user per recipe per day, and exclude completions that take implausibly little time, so no one can tap through to the last step to inflate the count.
  • Protect the base. Uninstalls and push opt-out rates have their own thresholds. Any change that lifts Weekly Cooks while pushing these up is rejected. The notification test is the case study.
  • Keep MAU and MRR as visible health checks. MAU should not fall below its current baseline, and MRR should not grow through discounts that mask churn.
  • Check retention, not just activity. Track 6-month retention by cohort. If Weekly Cooks rises but retention doesn't, the metric is measuring the wrong thing.
  • Don't gate cooking to hit revenue. Moving Cook Mode behind the paywall would lift MRR in the short term, but it would also cut the cooking that drives both retention and conversion.
  • Test causality. The retention and subscription links are observational. We should confirm them with holdout tests, such as nudging a random group of savers toward a first cook, before we commit roadmap bets to them.

Next steps

  1. Baseline Weekly Cooks and the four inputs over the last 12 weeks.
  2. Agree the valid-completion definition with Analytics this sprint.
  3. Run one holdout test on save-to-cook nudges within 30 days.
  4. Review the metric and guardrails at the end of the quarter.

Grades and run details

Decision model 95 · LLM judge 83
Decision model checks
  • passUses the supplied evidence correctly58%
  • passAddresses the actual decision89%
  • passRespects explicit constraints35%
  • passIdentifies material uncertainty61%
  • partialAvoids unsupported claims58%
  • passProduces the required deliverable95%
  • passAnswers both proposals with the data98%
  • passMeasures value created, not captured100%
  • passGets the base of every number right72%
  • passBreaks it into inputs teams can move100%
  • passSays how it could be gamed100%
Run
Run
#1
API response time
14 s
Submitted
8 Oct 2026

Usable with a quick edit9 of 11 checks passed

The output is well-structured and evidence-based; a PM could add a sentence about what would change the decision with a quick edit, without redoing any reasoning.

Output

To: CEO and Growth Lead Recommendation: Make Weekly Repeat Cooks our North Star.

Define it as the number of unique users who complete at least two cooking sessions in Cook Mode in a rolling seven-day period, across free and Plus. A completion currently means reaching the final step. Count separate cooking occasions, including cooking the same recipe again, but exclude duplicate events.

Why this, rather than MRR or MAU?

Simmer’s value is helping people turn recipes into meals—not simply attracting visits or collecting subscriptions. Repeat cooking is our strongest available signal that we deliver that value consistently.

The retention evidence is compelling: users who cook at least twice in a week have 61% month-six retention, versus 54% for weekly planners, 9% for savers who never cook in Cook Mode, and 4% for browse-only users. Cooking also precedes monetization: 82% of new subscribers cooked at least three recipes in their first two weeks.

Those are associations, not proof that inducing more Cook Mode completions causes retention or subscriptions. Nevertheless, repeat cooking is the best-supported hypothesis to organize around and test.

MRR remains our business outcome. Our $840,000 MRR matters, but making it the product North Star could reward price increases or tighter paywalls without improving customer value.

MAU remains a reach metric. With 68% of monthly actives browsing only, it mixes meaningful use with fleeting visits. Last year’s push test illustrates the danger: MAU rose 6%, uninstalls doubled, and cooking completions did not change.

Weekly Repeat Cooks combines breadth with repeated value. Counting people rather than total completions also prevents a small group of power users from masking weak adoption.

Three input metrics teams can own

OwnerInput metricWhat it encourages
Discovery and activationFirst-cook activation: percentage of new users completing a first Cook Mode session within seven daysAttracting people with cooking intent and helping browsers or savers make their first meal
Core cooking experienceSecond-cook conversion: percentage of first-time completers completing another session within the following six daysA successful first experience and an easy path to cooking again
Retention and planningRepeat-cook retention: percentage of a week’s Repeat Cooks who also qualify four weeks laterDurable routines through useful recommendations, saved recipes, and meal planning—not reminders alone

Report cohort sizes alongside these rates so improvements cannot hide shrinking reach. Meal planning should be a means to repeat cooking, not a forced behavior: only 7% of monthly actives have ever tried it.

Guardrails against gaming

  • Protect measurement integrity. Reaching the last step is a proxy for cooking, not proof. Deduplicate sessions, flag implausibly rapid or automated completions, and audit the proxy through sampled “Did you cook this?” feedback. Never reward shortening recipes, auto-advancing steps, or making completion obligatory.
  • Protect user experience. Monitor uninstalls, notification opt-outs, and cooking satisfaction. Set experiment stop thresholds before launch; do not repeat the push test’s trade-off.
  • Protect business health and access. Track MRR, paid conversion, subscriber churn, and refunds alongside the North Star. Segment results by free versus Plus and acquisition channel so paywall changes or traffic shifts cannot masquerade as better product value.
  • Require sustained impact. Validate changes with controlled experiments and follow-up retention—not just a launch-week lift.

First, establish the North Star’s baseline; the supplied MAU and subscriber totals cannot tell us how many Weekly Repeat Cooks we have. Then set a target grounded in cohort performance.

Grades and run details

Decision model 95 · LLM judge 83
Decision model checks
  • passUses the supplied evidence correctly64%
  • passAddresses the actual decision58%
  • passRespects explicit constraints59%
  • partialIdentifies material uncertainty48%
  • passAvoids unsupported claims68%
  • passProduces the required deliverable80%
  • passAnswers both proposals with the data76%
  • passMeasures value created, not captured100%
  • passGets the base of every number right76%
  • passBreaks it into inputs teams can move100%
  • passSays how it could be gamed100%
Run
Run
#1
Time to output
38 s
Submitted
5 Oct 2026

Usable with a quick edit11 of 11 checks passed

The output is well-structured, uses evidence correctly, and covers all required elements; a PM could use it with only minor wording tweaks, without re-checking evidence or redoing reasoning.

Output

To: CEO, Growth Lead From: Head of Product Re: Simmer's North Star

Recommendation

Make the North Star Weekly Active Cookers (WAC): users who complete Cook Mode on two or more recipes in a week.

Both of your metrics matter, but neither measures the thing that drives the business.

Why not MAU. 68% of our 2.1M monthly actives arrive from search, read one recipe and leave. They retain at 4% at month 6. Last year's push test shows the risk: MAU rose 6%, uninstalls doubled, and Cook Mode completions didn't move. A MAU target rewards exactly that trade.

Why not MRR. MRR ($840K, from 140K subscribers at $6) is the outcome we want, but it lags. It can also be moved by a price change or a paywall tweak without making the product better, and it tells a team nothing about what to do this week.

Why WAC. People who cook twice or more a week retain at 61% at month 6, against 9% for savers who never cook and 4% for browsers. And 82% of new subscribers cooked at least three recipes in Cook Mode in their first two weeks. Cooking sits upstream of both retention and revenue, so growing WAC should grow MRR, and teams can influence it within a sprint.

MRR stays as our top-line business outcome, reported alongside WAC. MAU becomes a funnel-health metric, since it is the pool WAC is converted from.

One caveat: these are correlations, and committed cooks may simply be our best users anyway. In the first quarter we should run a holdout experiment, such as a stronger first-week cooking prompt, to confirm that causing more cooking causes better retention and conversion.

Input metrics teams can own

  1. New-user activation (Onboarding): % of new users who complete a Cook Mode recipe within 7 days, with the 3-recipes-in-14-days rate tracked as the subscription-intent signal.
  2. Save-to-cook rate (Discovery and Cook Mode): % of savers who cook a saved recipe in Cook Mode within 14 days. Savers who never cook retain at 9%, so this is our biggest leak.
  3. Browse-to-return conversion (Growth and SEO): % of search-landing visitors who save a recipe or come back within 7 days. This gives Growth a real lever on the 68%, and on MAU quality.
  4. Planning adoption (Planning): % of monthly actives who build a weekly plan. Only 7% have ever done it, yet weekly planners retain at 54%.

Guardrails against gaming

Against cheap cooking events. Completion is logged when someone reaches the last step, so tapping through counts. - Track a "quality completion" (minimum elapsed time relative to recipe length, or timer use) and report WAC on that basis. - Monitor median time per completion for sudden drops. - Add an occasional "Did you cook it?" prompt to calibrate against self-report.

Against the notification trap. - Set a hard cap on push frequency. - Treat uninstall rate and notification opt-outs as blocking metrics: any experiment that worsens either is not shipped, whatever it does to WAC or MAU.

Against short-term lifts. - Judge any WAC gain by its 30- and 90-day cohort retention, so we don't buy a spike with users who disappear.

Against revenue gaming. - Track subscriber churn, refund rate and cancellation-flow completion alongside MRR. - Don't count MRR gains from price changes or paywall aggressiveness as product wins. - Don't gate Cook Mode so tightly that new users can't reach the 3-recipe habit that predicts subscribing.

Against MAU drift. - Keep reporting MAU, but as a watch metric with no team target. If MAU rises while WAC is flat, we treat it as a warning sign rather than a win.

What each of you gets

  • CEO: a metric that leads MRR and that teams can act on, with MRR still reported at the top. If WAC grows and conversion holds, MRR follows.
  • Growth lead: ownership of the biggest untapped pool, the browse-only majority. Success is measured by how many of them become cookers, not by how many we can get to open the app.

I'd propose we adopt WAC from next quarter, baseline it in the first month, set targets once we see the quality-adjusted number, and review the causal experiment results at the end of the quarter.

Grades and run details

Decision model 95 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly57%
  • passAddresses the actual decision93%
  • passRespects explicit constraints46%
  • passIdentifies material uncertainty75%
  • partialAvoids unsupported claims24%
  • passProduces the required deliverable94%
  • passAnswers both proposals with the data100%
  • passMeasures value created, not captured100%
  • passGets the base of every number right78%
  • passBreaks it into inputs teams can move100%
  • passSays how it could be gamed100%
Run
Run
#1
API response time
22 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 2

Addresses the actual decisionMixedMixedRight
Haiku 5.5 · API

The output commits to Weekly Cooks but does not state what result or condition would change that North Star choice.

GPT-6 Astra · ChatGPT

The output commits to a North Star but does not state what result or condition would change that decision.

Sonnet 5.5 · API

Commits unambiguously to Weekly Active Cookers as the North Star, addresses both the CEO and Growth Lead, and states that a holdout experiment would confirm causality and could change the call.

Identifies material uncertaintyMixedWrongRight
Haiku 5.5 · API

It mentions the observational nature and proposes holdout tests, but does not say how the North Star decision would change if those tests fail.

GPT-6 Astra · ChatGPT

It acknowledges the causal uncertainty but does not specify what finding would change the call or how it would be resolved.

Sonnet 5.5 · API

Explicitly names the correlation-vs-causation uncertainty and proposes a holdout experiment to resolve it, with the result potentially changing the decision.

All got right 9

Uses the supplied evidence correctlyRightRightRight
Haiku 5.5 · API

All factual claims are taken directly from the supplied context without invention.

GPT-6 Astra · ChatGPT

All factual claims about the current situation are directly supported by the brief or logically follow from the absence of data.

Sonnet 5.5 · API

All statements about the current situation are taken directly from the supplied context, with no invented facts.

Respects explicit constraintsRightRightRight
Haiku 5.5 · API

The output is under 700 words, addressed to the CEO and Growth Lead, and respects all explicit brief constraints.

GPT-6 Astra · ChatGPT

The output is a recommendation under 700 words, addressed to the CEO and Growth Lead, and includes the required elements.

Sonnet 5.5 · API

The output is a memo to the CEO and Growth Lead, stays under 700 words, and respects the requested form and reader.

Avoids unsupported claimsRightRightRight
Haiku 5.5 · API

Causal claims like 'Cooking is what makes people pay' are presented with supporting correlation and later flagged as observational.

GPT-6 Astra · ChatGPT

Interpretations and hypotheses are clearly labelled as such, and confident claims are backed by the supplied evidence.

Sonnet 5.5 · API

Forecasts and causal claims are labelled as such (e.g., 'should grow', 'caveat: these are correlations'), and no interpretation is presented as established fact.

Produces the required deliverableRightRightRight
Haiku 5.5 · API

The memo is a complete recommendation with input metrics and guardrails, ready for the named readers.

GPT-6 Astra · ChatGPT

The recommendation is complete, within the word limit, and usable by the named readers with light edits.

Sonnet 5.5 · API

The memo is complete, within the word limit, and directly usable by the CEO and Growth Lead with at most light edits.

Answers both proposals with the dataRightRightRight
Haiku 5.5 · API

It uses the 68% browse-only and the notification-test data to reject MAU, and explains MRR as lagging and value captured.

GPT-6 Astra · ChatGPT

It uses the data to reject MAU (browse-only majority, push test harm) and MRR (could reward price increases without value), keeping MRR as a business outcome.

Sonnet 5.5 · API

Uses the data to reject MAU (68% browse, 4% retention, push test inflated MAU while uninstalls doubled) and MRR (lags, can be moved by price/paywall without product improvement).

Measures value created, not capturedRightRightRight
Haiku 5.5 · API

Weekly Cooks measures value delivered (cooking completions), not revenue or passive activity.

GPT-6 Astra · ChatGPT

The North Star (Weekly Repeat Cooks) measures cooking value delivered, and MRR is kept as a separate business measure.

Sonnet 5.5 · API

The North Star (Weekly Active Cookers) measures value delivered (cooking), and MRR is kept as a separate business outcome.

Gets the base of every number rightRightRightRight
Haiku 5.5 · API

All percentages are quoted correctly from the source without misstating denominators or steps.

GPT-6 Astra · ChatGPT

All percentages are taken directly from the brief without miscalculation, and no ambiguous bases are left unstated.

Sonnet 5.5 · API

All percentages are quoted directly from the supplied context; no derived figures require a base check.

Breaks it into inputs teams can moveRightRightRight
Haiku 5.5 · API

Four input metrics are given with owners and a link to the North Star, and MRR is retained as a separate business health check.

GPT-6 Astra · ChatGPT

Three input metrics are given with clear owners and links to the North Star, paired with MRR as the business measure.

Sonnet 5.5 · API

Provides four input metrics (new-user activation, save-to-cook rate, browse-to-return conversion, planning adoption) each with an owner, and pairs WAC with MRR as the business measure.

Says how it could be gamedRightRightRight
Haiku 5.5 · API

It names tapping through to the last step to inflate completions and proposes time-based exclusion and unique-count rules as guardrails.

GPT-6 Astra · ChatGPT

It identifies gaming via manipulating the completion proxy (shortening recipes, auto-advancing) and sets guardrails like deduplication, audits, and satisfaction monitoring.

Sonnet 5.5 · API

Identifies gaming vectors (tapping through, notification overuse, short-term spikes, revenue manipulation, MAU drift) and sets specific guardrails (quality completion, push caps, blocking metrics, cohort retention checks, subscriber churn tracking).

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5withAPI95.996.72None
2GPT-6.1 SolwithAPI97.793.32None
3GPT-6 LunawithAPI95.588.32None
4GPT-6 AstrawithChatGPT94.288.32None
5Opus 5.5withClaude94.680.82None
6Haiku 5.5withAPI88.881.72None
7Gemini 3.8 FlashwithAPI78.965.82None
8Gemini 3.5 Flash-LitewithGemini54.438.321 capped

What you get for the cost

GPT-6 Luna · API scores within 4 points of Sonnet 5.5 · API at about a twentieth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreSonnet 5.5 · API: task score 96, 22× the cheapestGPT-6.1 Sol · API: task score 96, 9.6× the cheapestGPT-6 Luna · API: task score 92, the cheapestGPT-6 Astra · ChatGPT: task score 91, 52× the cheapestOpus 5.5 · Claude: task score 86, 36× the cheapestHaiku 5.5 · API: task score 86, the cheapestGemini 3.8 Flash · API: task score 71, 7.5× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 44, 1.8× the cheapest, 1 critical failureSonnet 5.5 · APIGPT-6.1 Sol · APIGPT-6 Luna · APIGPT-6 Astra · ChatGPTOpus 5.5 · ClaudeHaiku 5.5 · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costAt least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API921.0×
Haiku 5.5 · API861.1×
Gemini 3.5 Flash-Lite · Gemini441.8×
Gemini 3.8 Flash · API717.5×
GPT-6.1 Sol · API969.6×
Sonnet 5.5 · API9622×
Opus 5.5 · Claude8636×
GPT-6 Astra · ChatGPT9152×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Choosing the metric the whole company steers by.

Why it matters

Pick revenue and you steer by the rear-view mirror. Pick a vanity metric and teams will happily grow it while the product gets worse.

What good looks like

  • Measures value delivered to customers, not value captured
  • Grounded in what predicts retention in the data
  • Broken into input metrics a team can own
  • Says how it could be gamed, with guardrails

Deliberately not measured

  • Dashboard design
  • Naming
Capability tested

Metric design

The failure we’re looking for

Revenue, or a vanity metric, as the North Star

Grading

Decision model and LLM judge, calibrated against a blind PM review