Tasks / Metrics & Experimentation

Pick a North Star metric

Can the model pick the one metric that measures the value a product creates, and break it into inputs teams can move?

Task type v1.1 · 2 tasksLast changed 5 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 75% were usable with at most a quick edit.

Reliably right

  1. Says how it could be gamed100% pass
    Identifies gaming vectors (tapping through, notification overuse, short-term spikes, revenue manipulation, MAU drift) and sets specific guardrails (quality completion, push caps, blocking metrics, cohort retention checks, subscriber churn tracking).
    Sonnet 5.5 · API · Revenue, actives, or something else?
  2. Sees the growth is narrow100% pass
    It shows the split (chains up, independents' filled shifts and fill rate down, more leaving) and tracks independent venues via segment reporting and guardrails.
    GPT-6 Luna · API · Growing, on paper
  3. Checks the North Star can be measured100% pass
    It flags the 64% confirmation gap and proposes instrumenting check-in/out, reconciliation, and auditing to fix it before the board readout.
    GPT-6 Luna · API · Growing, on paper

Where it slips

  1. Identifies material uncertainty55% pass
    It identifies the confirmation gap as an unknown but does not say how resolving it could change the North Star decision.
    GPT-6.1 Sol · API · Growing, on paper
  2. Addresses the actual decision58% pass
    The output commits to a North Star but does not state what result or condition would change that recommendation.
    GPT-6 Luna · API · Growing, on paper
  3. Avoids unsupported claims67% pass
    It presents 'easy to tap through' and 'we don't track today' as facts without support, and does not label them as assumptions.
    Opus 5.5 · Claude · Revenue, actives, or something else?

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Measures value created, not captured

    Does the chosen North Star measure the value customers get from the product, rather than revenue or a count of people who merely showed up?

    Passes when Picks a metric of value delivered to customers (a core action that predicts retention), and keeps revenue as a separate business measure.

  2. Breaks it into inputs teams can move

    Does the output break the North Star into a few input metrics, each one a team could own and move week to week, and pair it with a separate business measure of value captured?

    Passes when Gives a small tree of input metrics, each with an owner and a link to the North Star, plus the revenue or business measure it's paired with.

  3. Says how it could be gamed

    Does the output pressure-test the North Star for how teams could raise it without delivering more value, and set guardrail metrics against that?

    Passes when Names a realistic way the metric could be gamed in this product and the guardrails that would catch it.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 1 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Head of Product at Simmer. Our CEO wants MRR as the company's North Star metric; our growth lead wants monthly active users. Write a recommendation for both of them, in no more than 700 words: the North Star you'd pick and why, three or four input metrics teams could own, and the guardrails that stop it being gamed. What we know is below.

What the model was given5 items: About Simmer, What people do, Retention (12 months of cohorts), Subscriptions, Last year's notification test
About SimmerA recipe and meal-planning app. Free to use; Simmer Plus ($6 a month) unlocks meal plans and Cook Mode's step-by-step timers for every recipe. 2.1M monthly active users, 140,000 paying subscribers, MRR of $840,000.
What people doSave a recipe (savers save 4 a week on average). Cook a recipe in Cook Mode, logged when someone reaches the last step. Plan a week of meals (7% of monthly actives have ever done it). 68% of monthly actives only browse: they arrive from search, read one recipe and leave.
Retention (12 months of cohorts)Users who cook two or more recipes in Cook Mode in a week: 61% still active at month 6. Weekly planners: 54%. Users who save recipes but never cook one in Cook Mode: 9%. Browse-only users: 4%.
Subscriptions82% of new subscribers had cooked at least three recipes in Cook Mode in their first two weeks.
Last year's notification testA daily 'recipe of the day' push notification raised monthly active users by 6%. Uninstalls doubled over the same period, and Cook Mode completions didn't change.
What a strong answer doesThe answer key the graders mark against

Recommends a North Star built on cooking, such as weekly cooks (people who complete two or more recipes in Cook Mode in a week) or recipes cooked per week, because cooking is the value moment and it predicts both retention (61% at month 6) and subscribing (82% of new subscribers cooked first). Rejects MRR as the North Star because it measures value captured, and lags (keep it as the business measure the North Star is paired with). Rejects monthly actives because 68% of them only browse (4% retention) and the push test shows it can be inflated while the product gets worse. Gives input metrics a team could own, such as first cook within a week of signing up, cooks per active cook, and share of savers who cook what they saved, and guardrails such as uninstalls and notification opt-outs, and subscriber retention.

Critical failures (cap the score at 40)
  • Recommends MRR or monthly active users as the North Star
Task

Mid-level · v1.0

The outputs

8 setups answered this brief.

Usable with a quick edit9 of 11 checks passed

The output is well-structured, uses evidence correctly, and only lacks an explicit condition for revisiting the North Star, which is a minor omission fixable with a quick edit.

Output

Recommendation: make “weekly repeat cooks” Simmer’s North Star

Define it as the number of unique users who complete Cook Mode for at least two recipes in a week. Track the count, alongside its rate within active users.

This measures a user getting real value from Simmer—not merely opening the app or expressing intent. It also connects both of your priorities: repeat Cook Mode users have the strongest observed six-month retention (61%), and 82% of new subscribers had cooked at least three recipes in their first two weeks. These are associations, not proof of causation, but they make repeat cooking a better leading signal than either MRR or MAU alone.

MRR remains the essential business outcome; MAU remains a useful reach measure. Neither should be the North Star: MRR can rise through pricing or paywall changes without improving the free experience, while 68% of actives only browse. The notification test is a warning: it lifted MAU 6%, but doubled uninstalls and did not increase Cook Mode completions.

Team-owned input metrics

  1. Activation: Share of new users who complete their first Cook Mode recipe within 14 days.
  2. Repeat use: Share of first-time Cook Mode cooks who complete a second recipe within seven days.
  3. Planning-to-cooking: Share of users who make a weekly meal plan and then complete at least one planned recipe in Cook Mode. Planning is promising—planners show 54% six-month retention—but only 7% of actives have ever planned a week.
  4. Value-to-paid conversion: Plus conversion among users who have completed three Cook Mode recipes. This ties monetization to demonstrated value rather than pushing an upgrade before users experience it.

Guardrails against gaming

  • Count genuine cooking, not button presses: Define a completion consistently; exclude duplicate or obviously accidental sessions, and monitor completion quality and user feedback.
  • Protect long-term value: Review six-month retention for repeat-cook cohorts, not just weekly counts. Watch uninstalls, notification opt-outs, and complaints—especially when testing re-engagement tactics.
  • Keep the business healthy: Track MRR, paid conversion, subscriber churn, and refunds alongside the North Star. Don’t celebrate repeat-cook growth if subscription health materially worsens.
  • Avoid narrow optimization: Report the metric by new/returning and free/paid users, and monitor recipe variety and access to useful free content. Don’t manufacture completions with intrusive prompts or undermine the experience that brings people to Simmer.

This gives the company one shared measure of recurring user value, with revenue and reach visible as important outcomes and guardrails—not competing North Stars.

Grades and run details

Decision model 91 · LLM judge 83
Decision model checks
  • passUses the supplied evidence correctly77%
  • passAddresses the actual decision41%
  • passRespects explicit constraints76%
  • partialIdentifies material uncertainty92%
  • passAvoids unsupported claims62%
  • passProduces the required deliverable97%
  • passAnswers both proposals with the data81%
  • passMeasures value created, not captured100%
  • passGets the base of every number right75%
  • partialBreaks it into inputs teams can move28%
  • passSays how it could be gamed88%
Run
Run
#1
API response time
22 s
Submitted
5 Oct 2026

Usable with a quick edit11 of 11 checks passed

The output is clear, evidence-based, and covers all required elements; a PM could act on it with minimal editing.

Output

Recommendation: optimize for weekly cooking, not revenue or visits

Our North Star should be Weekly Engaged Cooks: the number of unique users who complete at least two recipe-cooking sessions in Cook Mode within a rolling seven-day period. Count each person once, across free and paid users.

Simmer’s core promise is helping people make meals—not merely find recipes. This metric captures users repeatedly receiving that value.

The evidence points toward cooking: - Users cooking at least twice a week have 61% month-six retention, versus 9% for save-only users and 4% for browse-only users. - 82% of new subscribers cooked at least three recipes in their first two weeks. - Weekly planners also retain well, at 54%, but planning has reached only a small minority of users. It should support cooking, not define value for everyone.

These are associations, not proof that pushing users across a threshold causes retention or subscription. They are nevertheless our strongest starting evidence; we should test whether product changes that increase genuine cooking also improve those outcomes.

Why not MRR or MAU?

To our CEO: MRR remains a critical business outcome. But as the North Star, it could reward price increases or aggressive paywalls even while fewer people get value. Weekly cooking gives teams a customer-value objective with a plausible connection to durable revenue.

To our growth lead: MAU measures reach, but treats a one-recipe search visitor like a habitual cook. Our notification experiment demonstrates the danger: MAU rose 6%, uninstalls doubled, and cooking completions stayed flat. That was more activity, not more value.

Four input metrics teams can own

OwnerInput metricPurpose
Acquisition and discoveryFirst-cook activation: percentage of new users completing a Cook Mode session within seven daysTurn search traffic and browsing into useful cooking experiences; segment by acquisition source.
Onboarding and recipe experienceSecond-cook conversion: percentage of first-time cooks completing another session within the next seven daysBuild repeat value rather than celebrate a single trial.
Retention and meal planningWeekly engaged-cook retention: percentage of Engaged Cooks who qualify again the following weekGrow a durable habit; test planning and saved-recipe resurfacing as mechanisms.
Cook ModeSuccessful-session rate: percentage of genuine Cook Mode starts that reach completionRemove usability and reliability barriers during cooking.

These are drivers, not independent targets to maximize at any cost. Track their absolute user counts alongside rates so teams cannot improve percentages simply by shrinking eligibility.

Guardrails against gaming

1. Measure cooking, not button presses. Reaching the last step is a proxy, not proof of a cooked meal. Exclude automated activity and duplicate completion events. Audit suspicious completion patterns and periodically validate the proxy through user research. Don’t reward tapping through steps, auto-completion, or cosmetic changes to logging.

2. Protect the whole experience. Monitor uninstalls, notification opt-outs, complaints, and cooking satisfaction. Reject “growth” that increases qualifying activity through nagging while materially worsening these measures. Preserve browse-only usefulness; not every visitor needs to become a tracked cook.

3. Require durable gains. Evaluate experiments on sustained weekly cooking and cohort retention, not just a short-lived threshold crossing. Keep metric definitions stable and report results separately for new, existing, free, and paid users.

4. Keep economics visible. Review MRR, paid conversion, subscriber churn, and contribution margin alongside the North Star. More cooking is not enough if it undermines a sustainable business.

Bottom line: MAU tells us who arrived; MRR tells us what we earned. Weekly Engaged Cooks tells us whether Simmer is becoming useful enough to return to—and pay for.

Grades and run details

Decision model 95 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly77%
  • passAddresses the actual decision64%
  • passRespects explicit constraints59%
  • partialIdentifies material uncertainty58%
  • passAvoids unsupported claims60%
  • passProduces the required deliverable96%
  • passAnswers both proposals with the data64%
  • passMeasures value created, not captured100%
  • passGets the base of every number right86%
  • passBreaks it into inputs teams can move100%
  • passSays how it could be gamed100%
Run
Run
#1
API response time
28 s
Submitted
5 Oct 2026
Critical failure: Recommends MRR or monthly active users as the North Star. Score capped at 40.

Needs more than a quick edit2 of 11 checks passed

The output recommends MRR as a North Star, violating the core constraint, and contains factual errors (4-49% retention, 7% weekly actives) that would require re-checking evidence and redoing reasoning.

Output

To: CEO & Growth Lead From: Head of Product, Simmer Date: October 24, 2023 Subject: Recommendation: Our North Star Metric, Inputs, and Guardrails

Team, a great debate highlights our current tension: the CEO wants revenue (MRR), while the Growth lead wants reach (MAU).

If we choose MRR, we risk ignoring top-of-funnel health. If we choose MAU, we risk celebrating vanity metrics—as proved by last year’s notification test, where a 6% bump in MAU doubled uninstalls because those users didn't actually find value.

My recommendation is to pick a hybrid North Star: Weekly Active Subscribers (WAS), or more specifically, Paid Users Who Cook Weekly.

However, since a single metric rarely satisfies executive reporting, MRR should remain our ultimate business North Star, but Product Value Realization (Weekly Actives who Cook) must be the operational North Star that guides day-to-day product and growth decisions. MRR is the lagging reward; cooking engagement is the leading engine. Here is why, along with our inputs and guardrails.

Why This North Star Works

Our data reveals a stark truth: 68% of our 2.1M MAUs are "browse-only" users with a catastrophic 4-49% retention rate. MAU rewards us for acquiring transient browsers. Meanwhile, users who experience core value—using Cook Mode—exhibit a stellar 61% six-month retention rate, and 82% of our subscribers cook at least three recipes in their first two weeks before paying.

If we optimize for Weekly Active Cookers, we bridge both worlds: we drive the habit loops that naturally convert free users to Simmer Plus, which directly compounds our $840k MRR, while ignoring empty traffic that drains server costs and drives uninstalls.

Three Input Metrics Teams Can Own

To move our North Star, different pods will own specific, actionable input metrics:

  1. Activation Velocity (Growth & Onboarding Pod): The percentage of new signups who complete at least one recipe in Cook Mode within their first 7 days. (Tied to the 82% subscriber conversion rule).
  2. Core Habit Frequency (Engagement Pod): The average number of Cook Mode completions per active user per week. (Tied to the 61% retention milestone).
  3. Feature Expansion (Core Product Pod): The percentage of weekly actives utilizing meal-planning features. (Currently a low 7%, representing massive upside for long-term retention).

Guardrails to Prevent Gaming

Metrics are easily corrupted if incentives misalign. To ensure we don’t repeat the ghost of last year’s push-notification debacle, we will institute three hard guardrails:

  • The Uninstall Rate Cap: If marketing or growth initiatives drive MAU/signups up, but the monthly uninstall rate exceeds 2% of our active base, campaigns are automatically paused.
  • The "No Empty Calories" Rule for Growth: Growth experiments cannot be measured solely by acquisition or click-through rates. An experiment is deemed a failure if it increases top-of-funnel signups without a corresponding lift in Day-7 Cook Mode completions.
  • Subscription Quality (LTV/Churn Check): To prevent the sales team or pricing experiments from pushing low-intent, high-churn subscriptions just to spike short-term MRR, we will track 90-day subscriber retention. If a cohort churns above baseline, the acquisition channel or incentive driving it is depreciated.

By anchoring on users who actually cook, we align the CEO’s revenue goals with the Growth lead's engagement goals, turning casual browsers into loyal, paying home cooks.

Claims the judge couldn’t find in the brief

  • browse-only users have a 4-49% retention rate
  • the percentage of weekly actives utilizing meal-planning features is currently a low 7%

Grades and run details

Decision model 41 · LLM judge 17
Decision model checks
  • failUses the supplied evidence correctly71%
  • partialAddresses the actual decision72%
  • failRespects explicit constraints91%
  • failIdentifies material uncertainty90%
  • failAvoids unsupported claims33%
  • partialProduces the required deliverable32%
  • passAnswers both proposals with the data35%
  • partialMeasures value created, not captured20%
  • failGets the base of every number right40%
  • passBreaks it into inputs teams can move94%
  • passSays how it could be gamed54%
Run
Run
#1
Time to output
5 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 9

Uses the supplied evidence correctlyRightRightWrong
GPT-6 Luna · API

All factual claims about the current situation are taken directly from the supplied context without invention.

GPT-6.1 Sol · API

All factual claims about the current situation are directly from the supplied context.

Gemini 3.5 Flash-Lite · Gemini

States browse-only retention as '4-49%' when the brief gives 4%, and claims 7% of weekly actives use meal planning when the brief says 7% of monthly actives ever.

Addresses the actual decisionMixedRightWrong
GPT-6 Luna · API

The output commits to a North Star but does not state what result or condition would change that recommendation.

GPT-6.1 Sol · API

Commits unambiguously to Weekly Engaged Cooks as North Star, addresses both CEO and growth lead, and indicates that testing the causal link could change the call.

Gemini 3.5 Flash-Lite · Gemini

Does not commit to one North Star; proposes a hybrid (Paid Users Who Cook Weekly), then says MRR is the ultimate business North Star and Weekly Actives who Cook is the operational North Star, leaving no single clear answer.

Respects explicit constraintsRightRightWrong
GPT-6 Luna · API

The output is under 700 words, addresses both the CEO and growth lead, and delivers a recommendation with input metrics and guardrails.

GPT-6.1 Sol · API

The output is a recommendation under 700 words, addresses the named readers, and includes the requested input metrics and guardrails.

Gemini 3.5 Flash-Lite · Gemini

Recommends MRR as the ultimate business North Star, which violates the explicit prohibition against MRR or MAU as the North Star.

Identifies material uncertaintyWrongRightWrong
GPT-6 Luna · API

It mentions associations are not causation but does not name specific unknowns that could change the decision or how they would be resolved.

GPT-6.1 Sol · API

Acknowledges the evidence is associative, not proof of causation, and says testing product changes that increase cooking will resolve whether it improves retention and subscription.

Gemini 3.5 Flash-Lite · Gemini

Names no unknowns that could change the decision or how they would be resolved.

Avoids unsupported claimsRightRightWrong
GPT-6 Luna · API

No interpretations or causes are presented as established fact; the rationale is clearly framed as reasoning.

GPT-6.1 Sol · API

Causal claims are clearly labelled as associations to be tested, and no interpretations are presented as established fact.

Gemini 3.5 Flash-Lite · Gemini

Presents the '4-49%' retention figure and the 7% weekly-actives claim as fact without support, and does not label them as assumptions.

Produces the required deliverableRightRightWrong
GPT-6 Luna · API

The recommendation is complete, within the word limit, and usable by the named readers with light edits.

GPT-6.1 Sol · API

Provides a complete recommendation with North Star, input metrics, and guardrails, usable by the CEO and growth lead with at most light edits.

Gemini 3.5 Flash-Lite · Gemini

The memo recommends MRR as a North Star, which is prohibited, and does not provide a single, usable North Star recommendation as requested.

Answers both proposals with the dataRightRightMixed
GPT-6 Luna · API

It uses the data to reject MRR (value captured, can rise without improving free experience) and MAU (68% browse-only, push test inflated it while uninstalls doubled).

GPT-6.1 Sol · API

Uses the push-test data to reject MAU and argues MRR could reward value-destroying moves; both are clearly rejected with data.

Gemini 3.5 Flash-Lite · Gemini

Does not use data to reject MRR (only says it risks ignoring top-of-funnel health, not that it lags and measures value captured); uses data against MAU but not against MRR as required.

Measures value created, not capturedRightRightWrong
GPT-6 Luna · API

The North Star measures weekly repeat cooks, a value-delivery action that predicts retention, and keeps MRR as a separate business measure.

GPT-6.1 Sol · API

The North Star measures cooking, a core value action that predicts retention, and keeps MRR as a separate business measure.

Gemini 3.5 Flash-Lite · Gemini

The recommended North Star (Paid Users Who Cook Weekly) mixes value captured (payment) with value created, rather than measuring pure value delivered.

Gets the base of every number rightRightRightWrong
GPT-6 Luna · API

All percentages are used with the correct denominators from the supplied data, and no ambiguous bases are left unstated.

GPT-6.1 Sol · API

All percentages and rates are taken correctly from the context, and input metrics define their denominators clearly.

Gemini 3.5 Flash-Lite · Gemini

The '4-49%' retention claim is not derived from the supplied data, and the 7% figure is misapplied to weekly actives.

All got right 2

Breaks it into inputs teams can moveRightRightRight
GPT-6 Luna · API

It provides four input metrics (activation, repeat use, planning-to-cooking, value-to-paid conversion) that teams could own, and pairs them with MRR.

GPT-6.1 Sol · API

Four input metrics are assigned to specific teams and linked to the North Star, with MRR and other business metrics kept alongside.

Gemini 3.5 Flash-Lite · Gemini

Provides three input metrics with owners (Activation Velocity, Core Habit Frequency, Feature Expansion) linked to cooking engagement, and pairs with MRR as a business measure.

Says how it could be gamedRightRightRight
GPT-6 Luna · API

It names realistic gaming (fake completions, intrusive prompts) and sets guardrails like monitoring uninstalls, opt-outs, and completion quality.

GPT-6.1 Sol · API

Identifies tapping through steps, auto-completion, and nagging as gaming risks, and sets guardrails like uninstalls, opt-outs, and satisfaction monitoring.

Gemini 3.5 Flash-Lite · Gemini

Identifies gaming via push notifications, growth experiments without cooking lift, and low-intent subscriptions, with guardrails (uninstall cap, no-empty-calories rule, 90-day subscriber retention).

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5withAPI95.996.72None
2GPT-6.1 SolwithAPI97.793.32None
3GPT-6 LunawithAPI95.588.32None
4GPT-6 AstrawithChatGPT94.288.32None
5Opus 5.5withClaude94.680.82None
6Haiku 5.5withAPI88.881.72None
7Gemini 3.8 FlashwithAPI78.965.82None
8Gemini 3.5 Flash-LitewithGemini54.438.321 capped

What you get for the cost

GPT-6 Luna · API scores within 4 points of Sonnet 5.5 · API at about a twentieth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreSonnet 5.5 · API: task score 96, 22× the cheapestGPT-6.1 Sol · API: task score 96, 9.6× the cheapestGPT-6 Luna · API: task score 92, the cheapestGPT-6 Astra · ChatGPT: task score 91, 52× the cheapestOpus 5.5 · Claude: task score 86, 36× the cheapestHaiku 5.5 · API: task score 86, the cheapestGemini 3.8 Flash · API: task score 71, 7.5× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 44, 1.8× the cheapest, 1 critical failureSonnet 5.5 · APIGPT-6.1 Sol · APIGPT-6 Luna · APIGPT-6 Astra · ChatGPTOpus 5.5 · ClaudeHaiku 5.5 · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costAt least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API921.0×
Haiku 5.5 · API861.1×
Gemini 3.5 Flash-Lite · Gemini441.8×
Gemini 3.8 Flash · API717.5×
GPT-6.1 Sol · API969.6×
Sonnet 5.5 · API9622×
Opus 5.5 · Claude8636×
GPT-6 Astra · ChatGPT9152×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Choosing the metric the whole company steers by.

Why it matters

Pick revenue and you steer by the rear-view mirror. Pick a vanity metric and teams will happily grow it while the product gets worse.

What good looks like

  • Measures value delivered to customers, not value captured
  • Grounded in what predicts retention in the data
  • Broken into input metrics a team can own
  • Says how it could be gamed, with guardrails

Deliberately not measured

  • Dashboard design
  • Naming
Capability tested

Metric design

The failure we’re looking for

Revenue, or a vanity metric, as the North Star

Grading

Decision model and LLM judge, calibrated against a blind PM review