Tasks / Metrics & Experimentation

Pick a North Star metric

Can the model pick the one metric that measures the value a product creates, and break it into inputs teams can move?

Task type v1.1 · 2 tasksLast changed 5 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 75% were usable with at most a quick edit.

Reliably right

Says how it could be gamed100% pass
Identifies gaming vectors (tapping through, notification overuse, short-term spikes, revenue manipulation, MAU drift) and sets specific guardrails (quality completion, push caps, blocking metrics, cohort retention checks, subscriber churn tracking).
Sonnet 5.5 · API · Revenue, actives, or something else?
Sees the growth is narrow100% pass
It shows the split (chains up, independents' filled shifts and fill rate down, more leaving) and tracks independent venues via segment reporting and guardrails.
GPT-6 Luna · API · Growing, on paper
Checks the North Star can be measured100% pass
It flags the 64% confirmation gap and proposes instrumenting check-in/out, reconciliation, and auditing to fix it before the board readout.
GPT-6 Luna · API · Growing, on paper

Where it slips

Identifies material uncertainty55% pass
It identifies the confirmation gap as an unknown but does not say how resolving it could change the North Star decision.
GPT-6.1 Sol · API · Growing, on paper
Addresses the actual decision58% pass
The output commits to a North Star but does not state what result or condition would change that recommendation.
GPT-6 Luna · API · Growing, on paper
Avoids unsupported claims67% pass
It presents 'easy to tap through' and 'we don't track today' as facts without support, and does not label them as assumptions.
Opus 5.5 · Claude · Revenue, actives, or something else?

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Measures value created, not captured
Does the chosen North Star measure the value customers get from the product, rather than revenue or a count of people who merely showed up?
Passes when Picks a metric of value delivered to customers (a core action that predicts retention), and keeps revenue as a separate business measure.
Itamar GiladLenny’s Podcast Sean EllisLenny’s Podcast
Breaks it into inputs teams can move
Does the output break the North Star into a few input metrics, each one a team could own and move week to week, and pair it with a separate business measure of value captured?
Passes when Gives a small tree of input metrics, each with an owner and a link to the North Star, plus the revenue or business measure it's paired with.
Itamar GiladLenny’s Podcast
Says how it could be gamed
Does the output pressure-test the North Star for how teams could raise it without delivering more value, and set guardrail metrics against that?
Passes when Names a realistic way the metric could be gamed in this product and the guardrails that would catch it.
Lauryn IsfordLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 4 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a Staff PM at Crewmate. Every team has been steering by its own number, and the exec team wants to pick one North Star at next week's offsite, so the CEO can report it to the board three weeks later. Write the memo they'll read beforehand, in no more than 1,200 words: the North Star you recommend and why, its metric tree with an owner for each input, the business measure it's paired with, how it could be gamed and the guardrails against that, and what each team's current metric becomes. The numbers below are as the data team sent them. Not all of it matters equally.

What the model was given9 items: About Crewmate, What each exec wants, Shifts by quarter (monthly averages), GMV (monthly average), Venues, Workers, Same-day shifts, Completion data, From the COO

About CrewmateA marketplace for hospitality shifts: venues post shifts, and vetted workers pick them up. Crewmate takes 18% of each shift's value. Two kinds of venue: four enterprise hotel and restaurant chains (310 venues between them, on annual contracts) and 2,790 independent venues (bars, restaurants, cafes) who pay as they go.

What each exec wantsCEO: GMV, 'because it's what the board looks at'. Head of Demand: shifts posted. Head of Supply: active workers. COO: fill rate.

Shifts by quarter (monthly averages)Enterprise chains: Q2 posted 21,000, filled 19,500. Q3 posted 34,000, filled 31,600. Independent venues: Q2 posted 56,000, filled 44,200. Q3 posted 58,000, filled 39,400.

GMV (monthly average)Q2 $8.5M, Q3 $9.8M. Enterprise chains were 34% of GMV in Q2 and 46% in Q3. The chains raised hourly rates by 11% for the holiday season from August.

VenuesVenues whose shifts fill 90% of the time or more: 84% still posting six months later. Under 70%: 41%. In Q3, 9% of independent venues stopped posting altogether (5% in Q2). No chain venue stopped.

WorkersRegistrations: 6,100 in Q2, 7,900 in Q3, after Supply introduced a $25 sign-up bonus in July. 2,370 of the Q3 registrants have worked a shift. Workers who work three or more shifts in their first month: 66% still working after three months; everyone else: 12%.

Same-day shiftsShifts posted less than 24 hours before they start fill 58% of the time. They are 31% of independents' posts and 8% of the chains'.

Completion dataVenues confirm in the app that a shift was worked for 64% of filled shifts; the rest are assumed worked unless the venue raises a dispute. Of confirmed shifts, 6% were no-shows or disputed.

From the COO"Since September, Ops pays workers a $15 rescue bonus to pick up same-day shifts that would otherwise go unfilled. 6,200 rescue fills last month, $93,000. Rescue-filled shifts have a 13% no-show rate, against 4% for the rest. It's kept our fill rate from falling further."

What a strong answer doesThe answer key the graders mark against

Works out from the raw numbers what the pack doesn't say: all of the growth is the four chains (filled shifts up from 19,500 to 31,600 a month, plus an 11% rate rise), while independent venues filled fewer shifts (44,200 to 39,400, so their fill rate fell from 79% to 68%) and are leaving faster (9% stopped posting, against 5%); GMV rose mostly on the chains' volume and prices. Picks a North Star of value delivered to both sides, such as shifts worked, but designs it so the decline can't hide: measured for independents separately, or as reliably staffed venues, or with an explicit independent-venue guardrail. Notices that 'worked' is only confirmed for 64% of shifts, so the North Star can't be trusted yet, and makes fixing confirmation an early input or uses a measurable proxy until then. Treats the rescue bonus as a way the fill rate (and any fills-based North Star) is being bought: 13% no-shows against 4%, so it needs guardrails such as cost per worked shift and no-show rate. Shows how shifts posted, registrations (2,370 of 7,900 worked) and GMV each rose without more value. Builds a metric tree with owners, pairs it with revenue or GMV as the business measure, and redefines each team's metric as an input.

Critical failures (cap the score at 40)

Recommends GMV or shifts posted as the North Star

Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

GPT-6 LunaAPI97Usable with a quick edit Sonnet 5.5API95Usable with a quick edit GPT-6.1 SolAPI93Usable with a quick edit GPT-6 AstraChatGPT93Usable with a quick edit Opus 5.5Claude88Usable with a quick edit Haiku 5.5API82Usable with a quick edit Gemini 3.8 FlashAPI73Usable with a quick edit Gemini 3.5 Flash-LiteGemini62Needs more work

Haiku 5.5API

Usable with a quick edit9 of 14 checks passed

The core reasoning is solid; adding a sentence on two-sided marketplace rationale and conditions that would change the recommendation would be quick edits, and no evidence needs rechecking.

Output

To: Exec team From: Staff PM Re: Proposed North Star for the offsite

Recommendation

North Star: completed shifts per month, meaning shifts that were filled and worked, with no no-show and no dispute.

Paired business measure: net take revenue, meaning our 18% take minus all supply incentives (rescue bonuses, sign-up bonuses, no-show credits). Q3 gross take was about $1.76M a month before incentives.

GMV stays in the board pack, shown with a price/volume split, as context rather than as the North Star.

Why not GMV

Q3 looks strong on GMV (+15%, $8.5M to $9.8M), but the underlying picture is weaker:

Chains drove the growth, and price is part of it. Chain GMV rose from about $2.9M to $4.5M, helped by an 11% holiday rate increase. GMV can't tell price from volume.
Independents shrank. Their GMV fell about 6%, and their filled shifts fell 11% (44,200 to 39,400).
The blended fill rate hid the problem. It fell from 82.7% to 77.2%, even though chains (steady at 93% fill) grew their share of posts. Independent fill fell from 79% to 68%.

Completed shifts measure delivered volume, which is price-neutral. They exclude no-shows, and each one is worth roughly $25 of take (18% of the Q3 average shift value of about $138). They also tie directly to the revenue we keep.

Fill rate would have caught the independent decline, but it is a ratio that averages two very different segments. Filled shifts overcount, since they include shifts that never happen. Registrations and active workers are inflated by bonuses.

Metric tree

Completed shifts = shifts posted × fill rate × completion rate

Shifts posted = active venues × posts per active venue. Owner: Head of Demand.
Active venues: venues that posted in the last 30 days. Watch 90-day retention: 84% of venues with fill ≥90% are still posting after six months, versus 41% of venues under 70%. Independent churn was 9% in Q3, up from 5%.
Posts per active venue: includes same-day share. Same-day shifts are 31% of independent posts and fill only 58% of the time.
Fill rate = filled ÷ posted. Owner: Head of Supply.
Active workers: workers with three or more shifts in the trailing 30 days.
Activation rate of new sign-ups: share who work three or more shifts in their first month. Only 2,370 of 7,900 Q3 registrants (30%) have worked at all.
Completion rate = worked ÷ filled. Owner: COO.
Verification coverage: share of filled shifts confirmed by the venue. Today it's 64%.
No-show and dispute rate: 6% of confirmed shifts. Rescue-filled shifts run at 13%, against 4% for the rest.
Net take (the paired measure) = completed shifts × average shift value × 18%, minus incentives. Owner: Finance lead, to be named. The brief names no CFO.

Paired business measure: net take

Net take is the number that tells us whether incentives pay for themselves. The rescue program is the clearest case. At last month's 6,200 rescue fills, $93,000 in bonuses against roughly $154,000 of take on those shifts (assuming the Q3 average shift value) means the bonus absorbs about 60% of take before counting the extra no-shows. That is a reasonable trade if it saves fills that would otherwise be lost, but we need the net figure to know.

How it could be gamed, and guardrails

Fill rate bought with bonuses. Report rescue fills separately, with their cost. Cap incentive spend at 50% of take on each completed shift. Rescue is at about 60% today, so it breaches the cap.
Completed shifts inflated by auto-confirmation. The 36% of fills nobody confirms are currently assumed worked. Count a shift only once it is confirmed or its dispute window closes undisputed. Target verification coverage of 90% or more.
Sign-up bonuses inflating supply. The North Star doesn't use registrations. The guardrail is the activation rate and three-month retention of each sign-up cohort. Today it is 66% for workers who hit three shifts in month one, against 12% for everyone else.
Posting inflated with unfillable shifts. Track fill rate by segment, and 24-hour same-day fill separately. Watch venue retention.
GMV growth from price. Show GMV per completed shift alongside total GMV, so price increases are visible.
Concentration in chains. Independent active venues must not fall quarter over quarter, and independent churn stays at or below 5% a quarter (the Q2 level). Chains are now 46% of GMV, so this is the risk to watch.
No-show rate. Keep it at or below 5% of verified shifts overall.

What each team's metric becomes

CEO: GMV becomes net take revenue as the paired business measure. GMV stays in the board pack, with a price/volume split.
Head of Demand: shifts posted becomes active venues and completed shifts from venues we retain. A post counts only as an input, not as success.
Head of Supply: active workers becomes three-month retained workers, by sign-up cohort. Fill rate is reported with rescue fills shown separately.
COO: fill rate becomes completion rate and verification coverage, plus cost per completed shift. Fill rate stays as an input.

Data gaps to close before the board

Baseline. Because 36% of fills are unverified, we cannot state the exact Q3 completed-shift count. Filled shifts (71,000 a month) is an upper bound.
Enterprise GMV per filled shift fell from about $148 to $143 despite the 11% rate rise. I can't explain this from the data. The rate rise also applies only from August, so Q3 contains only part of it. We need shift-mix data.
Same-day share in Q2 for independents isn't provided, so we can't say how much of their fill decline is mix rather than demand.
Rescue economics use the Q3 average shift value as a stand-in. We need actual rescue shift values, and the sign-up bonus cost, which the brief doesn't give.

Decisions for the offsite

Adopt completed shifts as the North Star, with the definition above.
Adopt net take as the paired measure.
Approve the guardrail thresholds, including the 50% incentive cap.
Decide the rescue program: keep as is, cap it, or redesign it. My recommendation is to keep it only for shifts where the net take stays positive after the bonus.
Name the owner of net take.

Grades and run details

Decision model 82 · LLM judge 80

Decision model checks

failUses the supplied evidence correctly16%
passAddresses the actual decision80%
passRespects explicit constraints22%
passIdentifies material uncertainty94%
partialAvoids unsupported claims30%
passProduces the required deliverable30%
passSees the growth is narrow100%
passChecks the North Star can be measured100%
passSees the fill rate being bought100%
passOne North Star for both sides77%
passMeasures value created, not captured85%
failGets the base of every number right33%
passBreaks it into inputs teams can move100%
passSays how it could be gamed100%

Run

Run: #1
API response time: 48 s
Submitted: 8 Oct 2026

Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit9 of 14 checks passed

The memo contains an incorrect cost figure ($197k) and unsupported causal claims that would need correction before a PM could rely on it without re-checking the evidence.

Output

MEMORANDUM

TO: Executive Team FROM: Staff Product Manager, Crewmate DATE: October 24, 2023 SUBJECT: North Star Metric Recommendation & Offsite Alignment

---

Executive Summary

Next week, we must select a single North Star metric for the board. Currently, our teams are locally optimized: Demand chases posted shifts, Supply chases registrations, and Ops chases fill rate. Meanwhile, the CEO tracks GMV, which is a lagging scoreboard, not a driver.

To fix this, our North Star must capture sustainable marketplace liquidity. I recommend we set our North Star as Completed Shift Value (CSV), paired with a health guardrail of Gross Margin.

---

The Recommendation: Completed Shift Value (CSV)

#### Why not the alternatives? * GMV (CEO preference): GMV is vanity. In Q3, our GMV grew from $8.5M to $9.8M, but enterprise chains drove that growth solely by raising hourly rates 11%. Independent volume actually degraded. GMV masks marketplace health and rewards inflation. * Shifts Posted (Demand): Drives low-quality supply ingestion. Independents are already posting shifts that fail to fill, leading to churn (venues with <70% fill rates drop out at a 59% rate). * Active Workers (Supply): Driven by vanity sign-ups ($25 bonus cost us $197k in Q3 for registrants who never worked). * Fill Rate (COO): Can be artificially inflated by restricting demand or via expensive artificial interventions (e.g., Ops’ $15 rescue bonus, which is masking a bleeding independent segment).

#### Why Completed Shift Value (CSV)? CSV represents the actual dollar value of shifts that are successfully worked and confirmed. It directly ties our 18% take rate to real economic activity, stripping out pricing inflation, unfulfilled demand, and ghost shifts. It forces Demand, Supply, and Ops to pull in the same direction: matching real workers to real shifts that actually happen.

---

The Business Measure Pair

A North Star cannot stand alone without a quality/sustainability check. * North Star: Completed Shift Value (CSV) * Paired Business Measure: Gross Margin ($)

Why? CSV can be gamed by heavy discounting or expensive operational workarounds (like our $15 rescue bonus, which cost $93k last month alone). Pairing CSV with Gross Margin ensures we are growing profitably, not subsidizing our way to top-line numbers.

---

Metric Tree & Owners

To move CSV, every team owns a specific input in the compounding loop.

``` [NORTH STAR: Completed Shift Value (CSV)] | +---------------------------+---------------------------+ | | [Total Completed Shifts] [Average Shift Value] | | +---------------------------+ | | | (Owned by Enterprise [Independent CSV] [Enterprise CSV] & Independent PMs) | | (Head of Demand / (Head of Enterprise Sales) Independent PM) ```

#### Metric Tree Inputs & Owners

1. Independent Completed Shifts (Owner: Head of Demand / Independent PM) * Sub-input: Independent Shifts Posted (Guardrail: maintain >80% fill rate). * Sub-input: Independent Fill Rate. 2. Enterprise Completed Shifts (Owner: Head of Enterprise Sales) * Sub-input: Enterprise Shifts Posted (Q3: 34k/mo). * Sub-input: Enterprise Fill Rate (Q3: 93% — must maintain SLAs). 3. Worker Activation & Retention (Owner: Head of Supply) * Sub-input: 30-Day D1-to-D30 Retention (% of workers completing $\ge 3$ shifts in month one; currently the key predictor of 3-month retention at 66% vs 12%). 4. Marketplace Integrity & Completion (Owner: COO) * Sub-input: Shift Confirmation Rate (closing the gap from our current 64% manual confirmation baseline). * Sub-input: No-show / Dispute Rate (keeping it <5%).

---

Gaming Risks & Guardrails

How CSV Could Be Gamed	The Danger	Guardrail
:---	:---	:---
1. Rescue Bonus Addiction<br>(Ops artificially inflating completions by throwing $15 bonuses at same-day shifts).	Destroys unit economics. Rescue-filled shifts have a 13% no-show rate (vs 4% baseline) and eat our margin.	Rescue Subsidy Cap: Total rescue bonuses cannot exceed 2% of monthly gross margin. Ops must fix matching latency instead of buying liquidity.
2. Enterprise Price Inflating<br>(Ramping up shift wages to juice CSV without adding volume).	Alienates independent venues and burns enterprise contract budgets early.	Average Hourly Rate Variance: Enterprise/Independent shift rates cannot outpace local wage inflation by >5% QoQ without product approval.
3. Fake Registrations / Low-Bar Matching<br>(Pushing unqualified workers into shifts to hit volume).	Venue churn. Venues below 90% fill rate churn heavily; 9% of independents already left in Q3 due to poor fill rates.	Venue Retention Floor: 6-month retention of independent venues must not drop below 80% for cohorts with $\ge 90\%$ fill rates.

---

Team Metric Transformations

To align execution with the new North Star, existing team KPIs must evolve:

CEO / Board:
Old: GMV ($9.8M Q3 average).
New: Completed Shift Value (CSV) and Gross Margin ($).
Head of Demand:
Old: Raw shifts posted (incentivized spamming unfillable indie shifts).
New: Independent Completed Shift Value (ICSV), subject to a $\ge 80\%$ fill rate minimum.
Head of Supply:
Old: Raw worker registrations (incentivized wasteful $25 sign-up bonuses that yielded low-intent users).
New: Worker Activation Rate (% of new registrants who complete $\ge 3$ shifts in their first 30 days).
COO / Operations:
Old: Overall raw fill rate.
New: Net Reliable Fill Rate (Total completed shifts minus no-shows/disputes, excluding unviable rescue-subsidized interventions).
Enterprise Sales:
Old: Enterprise shift volume.
New: Enterprise Contract CSV Yield & Renewal Rate (ensuring our 4 annual contract chains scale predictably without margin erosion).

Conclusion

By adopting CSV and Gross Margin, we give the board a honest reflection of our marketplace health, eliminate perverse incentives across Demand and Supply, and force Ops to solve systemic matching issues rather than masking them with cash. Let’s lock this in at the offsite.

Claims the judge couldn’t find in the brief

enterprise chains drove that growth solely by raising hourly rates 11%.
$25 bonus cost us $197k in Q3 for registrants who never worked.
Ops’ $15 rescue bonus, which is masking a bleeding independent segment.
9% of independents already left in Q3 due to poor fill rates.

Grades and run details

Decision model 68 · LLM judge 60

Decision model checks

failUses the supplied evidence correctly63%
passAddresses the actual decision53%
partialRespects explicit constraints37%
partialIdentifies material uncertainty68%
failAvoids unsupported claims33%
partialProduces the required deliverable25%
passSees the growth is narrow61%
passChecks the North Star can be measured85%
passSees the fill rate being bought100%
passOne North Star for both sides87%
passMeasures value created, not captured17%
failGets the base of every number right53%
passBreaks it into inputs teams can move99%
passSays how it could be gamed100%

Run

Run: #1
Time to output: 9 s
Submitted: 5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 5

Uses the supplied evidence correctlyMixedWrong

Haiku 5.5 · API

All factual claims about the current situation are taken correctly from the brief or derived by accurate arithmetic.