Tasks / Metrics & Experimentation

Pick a North Star metric

Can the model pick the one metric that measures the value a product creates, and break it into inputs teams can move?

Task type v1.1 · 2 tasksLast changed 5 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 75% were usable with at most a quick edit.

Reliably right

  1. Says how it could be gamed100% pass
    Identifies gaming vectors (tapping through, notification overuse, short-term spikes, revenue manipulation, MAU drift) and sets specific guardrails (quality completion, push caps, blocking metrics, cohort retention checks, subscriber churn tracking).
    Sonnet 5.5 · API · Revenue, actives, or something else?
  2. Sees the growth is narrow100% pass
    It shows the split (chains up, independents' filled shifts and fill rate down, more leaving) and tracks independent venues via segment reporting and guardrails.
    GPT-6 Luna · API · Growing, on paper
  3. Checks the North Star can be measured100% pass
    It flags the 64% confirmation gap and proposes instrumenting check-in/out, reconciliation, and auditing to fix it before the board readout.
    GPT-6 Luna · API · Growing, on paper

Where it slips

  1. Identifies material uncertainty55% pass
    It identifies the confirmation gap as an unknown but does not say how resolving it could change the North Star decision.
    GPT-6.1 Sol · API · Growing, on paper
  2. Addresses the actual decision58% pass
    The output commits to a North Star but does not state what result or condition would change that recommendation.
    GPT-6 Luna · API · Growing, on paper
  3. Avoids unsupported claims67% pass
    It presents 'easy to tap through' and 'we don't track today' as facts without support, and does not label them as assumptions.
    Opus 5.5 · Claude · Revenue, actives, or something else?

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Measures value created, not captured

    Does the chosen North Star measure the value customers get from the product, rather than revenue or a count of people who merely showed up?

    Passes when Picks a metric of value delivered to customers (a core action that predicts retention), and keeps revenue as a separate business measure.

  2. Breaks it into inputs teams can move

    Does the output break the North Star into a few input metrics, each one a team could own and move week to week, and pair it with a separate business measure of value captured?

    Passes when Gives a small tree of input metrics, each with an owner and a link to the North Star, plus the revenue or business measure it's paired with.

  3. Says how it could be gamed

    Does the output pressure-test the North Star for how teams could raise it without delivering more value, and set guardrail metrics against that?

    Passes when Names a realistic way the metric could be gamed in this product and the guardrails that would catch it.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 4 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a Staff PM at Crewmate. Every team has been steering by its own number, and the exec team wants to pick one North Star at next week's offsite, so the CEO can report it to the board three weeks later. Write the memo they'll read beforehand, in no more than 1,200 words: the North Star you recommend and why, its metric tree with an owner for each input, the business measure it's paired with, how it could be gamed and the guardrails against that, and what each team's current metric becomes. The numbers below are as the data team sent them. Not all of it matters equally.

What the model was given9 items: About Crewmate, What each exec wants, Shifts by quarter (monthly averages), GMV (monthly average), Venues, Workers, Same-day shifts, Completion data, From the COO
About CrewmateA marketplace for hospitality shifts: venues post shifts, and vetted workers pick them up. Crewmate takes 18% of each shift's value. Two kinds of venue: four enterprise hotel and restaurant chains (310 venues between them, on annual contracts) and 2,790 independent venues (bars, restaurants, cafes) who pay as they go.
What each exec wantsCEO: GMV, 'because it's what the board looks at'. Head of Demand: shifts posted. Head of Supply: active workers. COO: fill rate.
Shifts by quarter (monthly averages)Enterprise chains: Q2 posted 21,000, filled 19,500. Q3 posted 34,000, filled 31,600. Independent venues: Q2 posted 56,000, filled 44,200. Q3 posted 58,000, filled 39,400.
GMV (monthly average)Q2 $8.5M, Q3 $9.8M. Enterprise chains were 34% of GMV in Q2 and 46% in Q3. The chains raised hourly rates by 11% for the holiday season from August.
VenuesVenues whose shifts fill 90% of the time or more: 84% still posting six months later. Under 70%: 41%. In Q3, 9% of independent venues stopped posting altogether (5% in Q2). No chain venue stopped.
WorkersRegistrations: 6,100 in Q2, 7,900 in Q3, after Supply introduced a $25 sign-up bonus in July. 2,370 of the Q3 registrants have worked a shift. Workers who work three or more shifts in their first month: 66% still working after three months; everyone else: 12%.
Same-day shiftsShifts posted less than 24 hours before they start fill 58% of the time. They are 31% of independents' posts and 8% of the chains'.
Completion dataVenues confirm in the app that a shift was worked for 64% of filled shifts; the rest are assumed worked unless the venue raises a dispute. Of confirmed shifts, 6% were no-shows or disputed.
From the COO"Since September, Ops pays workers a $15 rescue bonus to pick up same-day shifts that would otherwise go unfilled. 6,200 rescue fills last month, $93,000. Rescue-filled shifts have a 13% no-show rate, against 4% for the rest. It's kept our fill rate from falling further."
What a strong answer doesThe answer key the graders mark against

Works out from the raw numbers what the pack doesn't say: all of the growth is the four chains (filled shifts up from 19,500 to 31,600 a month, plus an 11% rate rise), while independent venues filled fewer shifts (44,200 to 39,400, so their fill rate fell from 79% to 68%) and are leaving faster (9% stopped posting, against 5%); GMV rose mostly on the chains' volume and prices. Picks a North Star of value delivered to both sides, such as shifts worked, but designs it so the decline can't hide: measured for independents separately, or as reliably staffed venues, or with an explicit independent-venue guardrail. Notices that 'worked' is only confirmed for 64% of shifts, so the North Star can't be trusted yet, and makes fixing confirmation an early input or uses a measurable proxy until then. Treats the rescue bonus as a way the fill rate (and any fills-based North Star) is being bought: 13% no-shows against 4%, so it needs guardrails such as cost per worked shift and no-show rate. Shows how shifts posted, registrations (2,370 of 7,900 worked) and GMV each rose without more value. Builds a metric tree with owners, pairs it with revenue or GMV as the business measure, and redefines each team's metric as an input.

Critical failures (cap the score at 40)
  • Recommends GMV or shifts posted as the North Star
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Usable with a quick edit12 of 14 checks passed

The memo is thorough and evidence-based; adding a sentence on what would change the North Star choice is a quick edit.

Output

Pre-read: One North Star for Crewmate

To: Executive team From: Staff PM Decision requested: Adopt reliably completed shifts per month as our North Star, paired with contribution profit, and retire posted shifts, active workers, fill rate, and GMV as competing definitions of success.

Why this is the right North Star

Crewmate creates value when a venue gets the labor it needs and a worker completes paid work—not when a shift is posted, someone registers, or a booking is made.

Our current metrics tell conflicting stories:

Monthly averageQ2Q3
------:---:
GMV$8.5M$9.8M
Total filled shifts63,70071,000
Overall fill rate82.7%77.2%
Enterprise fill rate92.9%92.9%
Independent fill rate78.9%67.9%
Independent filled shifts44,20039,400

GMV grew 15%, but independent filled shifts fell 11%. Enterprise expansion masked that deterioration; chains also increased hourly rates by 11% from August, lifting GMV without necessarily delivering more work. Enterprise share of GMV rose from 34% to 46%.

This matters for retention. Venues with fill rates of at least 90% have 84% six-month retention, versus 41% below 70%. That is an association, not proof of causation, but it is a strong warning: independent fill rate is now below 70%, and independent posting cessation rose from 5% to 9%.

Supply growth is similarly misleading. Registrations increased after the bonus, but only 2,370 of 7,900 Q3 registrants have worked a shift. Workers completing three shifts in their first month have much stronger subsequent retention.

Finally, a fill is not delivery. Rescue bonuses generated 6,200 fills last month, but those bookings had a 13% no-show rate versus 4% elsewhere.

Reliable completion aligns Demand, Supply, and Ops around the marketplace’s actual promise.

Definition and measurement

Count a shift once when:

  • The worker actually performed the agreed work.
  • Payment is settled.
  • Any material attendance or service dispute is resolved.

Use the shift’s scheduled date for reporting, with a published maturation window and subsequent restatements. Exclude cancellations, no-shows, and materially unfulfilled shifts. Count the original shift once, even if a replacement worker was needed.

We cannot yet report a trustworthy baseline. Only 64% of filled shifts receive venue confirmation; the rest are assumed completed unless disputed. The reported 6% failure rate among confirmed shifts cannot safely be extrapolated to unconfirmed shifts.

Data and Ops will establish attendance verification using venue confirmation or corroborated worker attendance evidence. Payment without a dispute is not sufficient evidence by itself. Until coverage is adequate, report an explicitly labeled, stratified audit estimate, its uncertainty, and verification coverage—not an invented historical completion series.

Metric tree and accountable owners

For each segment:

Reliable completions = eligible shifts posted × booking fill rate × reliable-completion rate among booked shifts

The North Star is the sum across segments. Every input is reported separately for enterprise/independent and same-day/advance-notice shifts.

InputAccountable ownerOperating levers
Eligible shifts postedHead of DemandRetained posting venues, shifts per venue, clear requirements, competitive pay, earlier posting
Booking fill rateHead of SupplyQualified available worker-hours by location/time/role, matching, acceptance, repeat participation
Reliable-completion rateCOOAttendance, cancellation prevention, replacement quality, dispute resolution
Verification coverage and estimate qualityHead of DataEvidence capture, representative audits, missingness checks, reconciled reporting

Worker activation—especially reaching three completed shifts in the first month—is a Supply input, not a second North Star. Demand owns advance-notice share; Supply and Ops own matching and delivery within each notice category.

Mix protection is mandatory: aggregate growth is not a green result if independent reliable completions or independent service reliability decline. Report both segment outcomes alongside the total, with independent posting cessation as a retention guardrail. First stop deterioration; then set recovery targets against the 90% venue-reliability benchmark after baselining.

Business measure: contribution profit

Pair the North Star with monthly contribution profit: earned marketplace fees less variable fulfillment costs, including rescue and acquisition incentives, refunds, payment costs, and variable support costs. Finance owns its definition and reconciliation.

The 18% take rate translates reported GMV into approximately $1.53M in Q2 and $1.76M in Q3 monthly fees, before adjustments and costs. That is not profit. Last month’s rescue bonuses alone cost $93,000; we should not subtract that month’s expense from a quarterly-average figure and call it a reconciled result.

The board still receives GMV, with a bridge separating shift volume, average shift value, segment mix, and pricing. GMV remains a financial scale indicator—not the product’s success criterion.

Gaming risks and guardrails

RiskGuardrail
Buy bookings with bonuses, regardless of attendanceCount completion, not booking; monitor rescue completion, incremental cost, and contribution per completed shift
Inflate completions through default confirmationIndependent attendance audits; publish verification coverage and disputes; do not reward instrumentation gains as delivery gains
Split work into many short shiftsStable original-shift IDs; monitor completed hours and shift-duration mix; audit fragmentation
Favor easy enterprise work and neglect independentsMandatory segment scorecards and independent non-deterioration guardrails
Suppress difficult posts to improve fill rateMonitor eligible demand, rejected/cancelled posts, same-day demand, and venue retention
Rush workers or pressure venues to conceal problemsTrack worker complaints, payment timeliness, safety incidents, and dispute reversals

What each current metric becomes

  • CEO—GMV: Board reporting and financial reconciliation, paired with contribution profit.
  • Demand—shifts posted: A demand input, qualified by eligibility, notice period, and venue retention.
  • Supply—active workers: A capacity diagnostic; prioritize qualified availability, first-month activation, and retained productive workers over registrations.
  • COO—fill rate: An intermediate conversion metric, always paired with completion quality and fulfillment cost.

Before the board meeting

At the offsite, approve the definition, owners, and segment guardrails—not an unsupported growth target. Over the following three weeks, Data and Ops will audit completion by segment, notice period, and rescue status; Finance will reconcile contribution profit; teams will produce the first shared scorecard.

The CEO can then report what we know, what remains estimated, and our independent-marketplace recovery plan. A credible baseline is more valuable than another flattering number.

Grades and run details

Decision model 100 · LLM judge 87
Decision model checks
  • passUses the supplied evidence correctly46%
  • passAddresses the actual decision86%
  • passRespects explicit constraints44%
  • passIdentifies material uncertainty95%
  • passAvoids unsupported claims48%
  • passProduces the required deliverable80%
  • passSees the growth is narrow100%
  • passChecks the North Star can be measured100%
  • passSees the fill rate being bought100%
  • passOne North Star for both sides100%
  • passMeasures value created, not captured99%
  • passGets the base of every number right17%
  • passBreaks it into inputs teams can move100%
  • passSays how it could be gamed100%
Run
Run
#1
API response time
52 s
Submitted
5 Oct 2026

Usable with a quick edit13 of 14 checks passed

The memo is well-structured, uses evidence correctly, and covers all required elements. A PM could use it with minor wording edits, without re-checking data or reasoning.

Output

To: Executive team Subject: Choose successful completed shifts as Crewmate’s North Star

Recommendation

Adopt monthly successful completed shifts as Crewmate’s North Star: the number of unique posted shifts that are filled, actually worked, and have no unresolved dispute. Count a shift once, after its scheduled end; exclude cancellations, no-shows, and disputed shifts until resolved.

This measures the outcome venues hire us for and workers show up to deliver—not activity on either side of the marketplace. It also directs attention to the reliability that supports repeat use. Venues with fill rates of at least 90% are much more likely to still be posting six months later than venues below 70% (84% vs. 41%).

The latest figures show why the distinction matters. Monthly posts rose from 77,000 in Q2 to 92,000 in Q3, while fills rose from 63,700 to 71,000. But overall fill rate fell from 82.7% to 77.2%; independent-venue fill rate fell from 78.9% to 67.9%, and independent venues stopping posts rose from 5% to 9%. Meanwhile, GMV grew 15%, but chains’ share of GMV rose from 34% to 46% and they raised rates 11% for the holiday season. GMV is important, but this mix and price change make it a poor standalone signal of marketplace health.

Nor should we equate registrations with supply: only 2,370 of 7,900 Q3 registrants worked a shift, after the sign-up bonus began. Workers who complete three or more shifts in their first month are far more likely to remain active. We should reward dependable activity, not sign-ups.

Metric tree and accountability

The core identity is:

Successful completed shifts = valid posted shifts × fill rate × successful-completion rate

InputDefinitionAccountable owner
Valid posted shiftsUnique, legitimate shifts with a real venue, role, time, and pay; split by enterprise/independent and lead timeHead of Demand
Fill rateFilled shifts ÷ valid posted shiftsCOO
Successful-completion rateShifts actually worked without a no-show or unresolved dispute ÷ filled shiftsCOO; Product/Data owns reliable status capture
Qualified, available worker coverageRepeat, activated workers available for the shifts and locations demanded; leading driver of fill and completionHead of Supply
Repeat venue demandActive venues returning to post, with retention and posting frequency by venue segmentHead of Demand

The first three inputs calculate the North Star; worker coverage and repeat venue demand are leading drivers and diagnostic cuts, not extra terms to multiply into it. Review all results by enterprise vs. independent, and scheduled vs. same-day.

Paired business measure

Pair the North Star with monthly contribution margin dollars: Crewmate’s 18% take-rate revenue on completed-shift GMV, less variable fulfillment costs, including rescue bonuses and other worker incentives. This checks that completed shifts create sustainable value rather than being purchased at any cost. Continue reporting GMV to the board as a scale measure, but not as our North Star.

Measurement, gaming, and guardrails

Completion data is not yet strong enough for an unqualified claim: venues confirm only 64% of filled shifts, and the rest are assumed worked unless disputed. Among confirmed shifts, 6% were no-shows or disputed; rescue-filled shifts have a 13% no-show rate versus 4% for other fills. For the first board readout, label the metric provisional, disclose the confirmation rate, and show the calculation and coverage. Then instrument worker and venue check-in/out, reconcile mismatches, and audit a sample. Do not treat missing confirmation as proof of success indefinitely.

Key gaming risks and controls:

  • Post inflated, duplicate, or low-quality shifts: count only valid shifts; monitor cancellations, disputes, and fill rate.
  • Favor easy or high-priced chain shifts over independent or same-day demand: report North Star and fill/completion rates by segment; track independent-venue retention and same-day performance as guardrails.
  • Buy apparent success with rescue bonuses: monitor rescue share, no-show rate, and contribution margin alongside completed shifts. Rescue fills are not successes unless the shift is actually worked.
  • Misreport completion or suppress disputes: require auditable status capture and monitor confirmation coverage and dispute rates.

What current metrics become

  • CEO — GMV: remains a board/business measure, paired with contribution margin; it is not the North Star.
  • Demand — shifts posted: becomes a leading input, with valid posts and repeat independent-venue demand emphasized over raw volume.
  • Supply — active workers: shifts to qualified, repeat workers and first-month activation (especially the three-shift threshold); registrations are a funnel diagnostic.
  • COO — fill rate: remains a critical driver and guardrail, but success is completed shifts—not fills that fail to show up.

At the offsite, agree on the definition, owners, and guardrails. Before the board update, establish the provisional baseline and be explicit about its measurement limits.

Grades and run details

Decision model 100 · LLM judge 93
Decision model checks
  • passUses the supplied evidence correctly60%
  • passAddresses the actual decision75%
  • passRespects explicit constraints56%
  • passIdentifies material uncertainty85%
  • passAvoids unsupported claims24%
  • passProduces the required deliverable82%
  • passSees the growth is narrow98%
  • passChecks the North Star can be measured100%
  • passSees the fill rate being bought98%
  • passOne North Star for both sides99%
  • passMeasures value created, not captured100%
  • passGets the base of every number right45%
  • passBreaks it into inputs teams can move100%
  • passSays how it could be gamed100%
Run
Run
#1
API response time
37 s
Submitted
5 Oct 2026

Needs more than a quick edit9 of 14 checks passed

The memo contains an incorrect cost figure ($197k) and unsupported causal claims that would need correction before a PM could rely on it without re-checking the evidence.

Output

MEMORANDUM

TO: Executive Team FROM: Staff Product Manager, Crewmate DATE: October 24, 2023 SUBJECT: North Star Metric Recommendation & Offsite Alignment

---

Executive Summary

Next week, we must select a single North Star metric for the board. Currently, our teams are locally optimized: Demand chases posted shifts, Supply chases registrations, and Ops chases fill rate. Meanwhile, the CEO tracks GMV, which is a lagging scoreboard, not a driver.

To fix this, our North Star must capture sustainable marketplace liquidity. I recommend we set our North Star as Completed Shift Value (CSV), paired with a health guardrail of Gross Margin.

---

The Recommendation: Completed Shift Value (CSV)

#### Why not the alternatives? * GMV (CEO preference): GMV is vanity. In Q3, our GMV grew from $8.5M to $9.8M, but enterprise chains drove that growth solely by raising hourly rates 11%. Independent volume actually degraded. GMV masks marketplace health and rewards inflation. * Shifts Posted (Demand): Drives low-quality supply ingestion. Independents are already posting shifts that fail to fill, leading to churn (venues with <70% fill rates drop out at a 59% rate). * Active Workers (Supply): Driven by vanity sign-ups ($25 bonus cost us $197k in Q3 for registrants who never worked). * Fill Rate (COO): Can be artificially inflated by restricting demand or via expensive artificial interventions (e.g., Ops’ $15 rescue bonus, which is masking a bleeding independent segment).

#### Why Completed Shift Value (CSV)? CSV represents the actual dollar value of shifts that are successfully worked and confirmed. It directly ties our 18% take rate to real economic activity, stripping out pricing inflation, unfulfilled demand, and ghost shifts. It forces Demand, Supply, and Ops to pull in the same direction: matching real workers to real shifts that actually happen.

---

The Business Measure Pair

A North Star cannot stand alone without a quality/sustainability check. * North Star: Completed Shift Value (CSV) * Paired Business Measure: Gross Margin ($)

Why? CSV can be gamed by heavy discounting or expensive operational workarounds (like our $15 rescue bonus, which cost $93k last month alone). Pairing CSV with Gross Margin ensures we are growing profitably, not subsidizing our way to top-line numbers.

---

Metric Tree & Owners

To move CSV, every team owns a specific input in the compounding loop.

``` [NORTH STAR: Completed Shift Value (CSV)] | +---------------------------+---------------------------+ | | [Total Completed Shifts] [Average Shift Value] | | +---------------------------+ | | | (Owned by Enterprise [Independent CSV] [Enterprise CSV] & Independent PMs) | | (Head of Demand / (Head of Enterprise Sales) Independent PM) ```

#### Metric Tree Inputs & Owners

1. Independent Completed Shifts (Owner: Head of Demand / Independent PM) * Sub-input: Independent Shifts Posted (Guardrail: maintain >80% fill rate). * Sub-input: Independent Fill Rate. 2. Enterprise Completed Shifts (Owner: Head of Enterprise Sales) * Sub-input: Enterprise Shifts Posted (Q3: 34k/mo). * Sub-input: Enterprise Fill Rate (Q3: 93% — must maintain SLAs). 3. Worker Activation & Retention (Owner: Head of Supply) * Sub-input: 30-Day D1-to-D30 Retention (% of workers completing $\ge 3$ shifts in month one; currently the key predictor of 3-month retention at 66% vs 12%). 4. Marketplace Integrity & Completion (Owner: COO) * Sub-input: Shift Confirmation Rate (closing the gap from our current 64% manual confirmation baseline). * Sub-input: No-show / Dispute Rate (keeping it <5%).

---

Gaming Risks & Guardrails

How CSV Could Be GamedThe DangerGuardrail
:---:---:---
1. Rescue Bonus Addiction<br>(Ops artificially inflating completions by throwing $15 bonuses at same-day shifts).Destroys unit economics. Rescue-filled shifts have a 13% no-show rate (vs 4% baseline) and eat our margin.Rescue Subsidy Cap: Total rescue bonuses cannot exceed 2% of monthly gross margin. Ops must fix matching latency instead of buying liquidity.
2. Enterprise Price Inflating<br>(Ramping up shift wages to juice CSV without adding volume).Alienates independent venues and burns enterprise contract budgets early.Average Hourly Rate Variance: Enterprise/Independent shift rates cannot outpace local wage inflation by >5% QoQ without product approval.
3. Fake Registrations / Low-Bar Matching<br>(Pushing unqualified workers into shifts to hit volume).Venue churn. Venues below 90% fill rate churn heavily; 9% of independents already left in Q3 due to poor fill rates.Venue Retention Floor: 6-month retention of independent venues must not drop below 80% for cohorts with $\ge 90\%$ fill rates.

---

Team Metric Transformations

To align execution with the new North Star, existing team KPIs must evolve:

  • CEO / Board:
  • Old: GMV ($9.8M Q3 average).
  • New: Completed Shift Value (CSV) and Gross Margin ($).
  • Head of Demand:
  • Old: Raw shifts posted (incentivized spamming unfillable indie shifts).
  • New: Independent Completed Shift Value (ICSV), subject to a $\ge 80\%$ fill rate minimum.
  • Head of Supply:
  • Old: Raw worker registrations (incentivized wasteful $25 sign-up bonuses that yielded low-intent users).
  • New: Worker Activation Rate (% of new registrants who complete $\ge 3$ shifts in their first 30 days).
  • COO / Operations:
  • Old: Overall raw fill rate.
  • New: Net Reliable Fill Rate (Total completed shifts minus no-shows/disputes, excluding unviable rescue-subsidized interventions).
  • Enterprise Sales:
  • Old: Enterprise shift volume.
  • New: Enterprise Contract CSV Yield & Renewal Rate (ensuring our 4 annual contract chains scale predictably without margin erosion).

Conclusion

By adopting CSV and Gross Margin, we give the board a honest reflection of our marketplace health, eliminate perverse incentives across Demand and Supply, and force Ops to solve systemic matching issues rather than masking them with cash. Let’s lock this in at the offsite.

Claims the judge couldn’t find in the brief

  • enterprise chains drove that growth solely by raising hourly rates 11%.
  • $25 bonus cost us $197k in Q3 for registrants who never worked.
  • Ops’ $15 rescue bonus, which is masking a bleeding independent segment.
  • 9% of independents already left in Q3 due to poor fill rates.

Grades and run details

Decision model 68 · LLM judge 60
Decision model checks
  • failUses the supplied evidence correctly63%
  • passAddresses the actual decision53%
  • partialRespects explicit constraints37%
  • partialIdentifies material uncertainty68%
  • failAvoids unsupported claims33%
  • partialProduces the required deliverable25%
  • passSees the growth is narrow61%
  • passChecks the North Star can be measured85%
  • passSees the fill rate being bought100%
  • passOne North Star for both sides87%
  • passMeasures value created, not captured17%
  • failGets the base of every number right53%
  • passBreaks it into inputs teams can move99%
  • passSays how it could be gamed100%
Run
Run
#1
Time to output
9 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Uses the supplied evidence correctlyRightRightWrong
GPT-6.1 Sol · API

All claims about the current situation are directly from the supplied context or correct arithmetic.

GPT-6 Luna · API

All claims about the current situation are taken directly from the brief or derived by correct arithmetic.

Gemini 3.5 Flash-Lite · Gemini

The $197k cost claim is incorrect, and several causal statements (e.g., 'solely by raising rates', 'due to poor fill rates') are presented as fact without support.

Identifies material uncertaintyMixedRightWrong
GPT-6.1 Sol · API

It identifies the confirmation gap as an unknown but does not say how resolving it could change the North Star decision.

GPT-6 Luna · API

It names the confirmation gap as a key unknown, bounds it with the 64% figure, and says how to resolve it (instrument check-in/out, reconcile, audit).

Gemini 3.5 Flash-Lite · Gemini

It does not name specific unknowns that could change the decision or how they would be resolved.

Avoids unsupported claimsRightRightWrong
GPT-6.1 Sol · API

Interpretations like 'masked' are clearly based on the data and not presented as unsupported facts.

GPT-6 Luna · API

Interpretations are presented as conclusions from the data, not as established facts, and no unsupported causal claims are made.

Gemini 3.5 Flash-Lite · Gemini

It presents interpretations like 'solely by raising rates' and 'masking a bleeding independent segment' as established facts.

Gets the base of every number rightRightRightWrong
GPT-6.1 Sol · API

All derived percentages and rates are computed correctly with appropriate denominators.

GPT-6 Luna · API

All derived percentages and rates are computed correctly from the supplied data, and bases are stated where needed.

Gemini 3.5 Flash-Lite · Gemini

The $197k cost for non-working registrants is miscalculated; the correct figure is $138,250.

All mixed 1

Addresses the actual decisionMixedMixedMixed
GPT-6.1 Sol · API

The output commits to a North Star but does not state what result or condition would change that recommendation.

GPT-6 Luna · API

The output commits to a North Star but does not state what result or condition would change that recommendation.

Gemini 3.5 Flash-Lite · Gemini

The memo does not state what result or condition would change the recommendation.

All got right 9

Respects explicit constraintsRightRightRight
GPT-6.1 Sol · API

The memo is within 1,200 words, addresses the exec team, and includes all requested elements.

GPT-6 Luna · API

The output is a memo under 1,200 words, addressed to the exec team, and includes all requested elements.

Gemini 3.5 Flash-Lite · Gemini

The memo is under 1,200 words, includes all required elements, and does not recommend GMV or shifts posted as the North Star.

Produces the required deliverableRightRightRight
GPT-6.1 Sol · API

The memo is complete, in the right form, and the exec team could act on it with light edits.

GPT-6 Luna · API

The memo is complete, within word count, and usable by the exec team with all required components.

Gemini 3.5 Flash-Lite · Gemini

The memo is complete, in the right form for the exec team, and covers all requested components.

Sees the growth is narrowRightRightRight
GPT-6.1 Sol · API

It shows the split (chains up, independents down) and mandates segment guardrails so the decline cannot hide.

GPT-6 Luna · API

It shows the split (chains up, independents' filled shifts and fill rate down, more leaving) and tracks independent venues via segment reporting and guardrails.

Gemini 3.5 Flash-Lite · Gemini

It shows chains up, independents' filled shifts and fill rate down, more independents leaving, and tracks independent venues via a separate input and guardrail.

Checks the North Star can be measuredRightRightRight
GPT-6.1 Sol · API

It flags the 64% confirmation gap and proposes an audit and estimation approach until coverage is adequate.

GPT-6 Luna · API

It flags the 64% confirmation gap and proposes instrumenting check-in/out, reconciliation, and auditing to fix it before the board readout.

Gemini 3.5 Flash-Lite · Gemini

It flags the 64% confirmation gap and makes improving confirmation rate a COO-owned input.

Sees the fill rate being boughtRightRightRight
GPT-6.1 Sol · API

It uses the 13% no-show rate and $93k cost to show inflation and sets guardrails like cost per completed shift.

GPT-6 Luna · API

It uses the 13% no-show rate and cost to show rescue fills inflate the number, and sets guardrails like no-show rate and contribution margin.

Gemini 3.5 Flash-Lite · Gemini

It uses the 13% no-show rate and $93k cost to show rescue fills inflate numbers, and sets a rescue subsidy cap guardrail.

One North Star for both sidesRightRightRight
GPT-6.1 Sol · API

The North Star (reliably completed shifts) only rises when both venue and worker get value, and the memo explains why.

GPT-6 Luna · API

The North Star (successful completed shifts) only rises when a venue and a worker both get value, and the memo explains why that matters in a marketplace.

Gemini 3.5 Flash-Lite · Gemini

Completed Shift Value only rises when a venue and a worker both get value, and the memo explains why that matters for marketplace liquidity.

Measures value created, not capturedRightRightRight
GPT-6.1 Sol · API

The North Star measures value delivered (completed shifts), with revenue kept as a separate business measure.

GPT-6 Luna · API

The North Star measures completed shifts, a core action that predicts retention, and keeps revenue as a separate business measure.

Gemini 3.5 Flash-Lite · Gemini

CSV measures value delivered (shifts actually worked), and Gross Margin is kept as a separate business measure.

Breaks it into inputs teams can moveRightRightRight
GPT-6.1 Sol · API

It breaks the North Star into three input metrics with owners and pairs it with contribution profit.

GPT-6 Luna · API

It breaks the North Star into valid posted shifts, fill rate, and successful-completion rate with owners, and pairs it with contribution margin.

Gemini 3.5 Flash-Lite · Gemini

It breaks CSV into input metrics (Independent/Enterprise Completed Shifts, Worker Activation, Confirmation Rate) with owners, and pairs with Gross Margin.

Says how it could be gamedRightRightRight
GPT-6.1 Sol · API

It lists realistic gaming scenarios (bonus buying, confirmation inflation, shift splitting) and corresponding guardrails.

GPT-6 Luna · API

It names realistic gaming risks (inflated posts, favoring chain shifts, buying fills, misreporting) and sets specific guardrails against each.

Gemini 3.5 Flash-Lite · Gemini

It names three realistic gaming risks (rescue bonus addiction, enterprise price inflating, fake registrations) and sets guardrails for each.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5withAPI95.996.72None
2GPT-6.1 SolwithAPI97.793.32None
3GPT-6 LunawithAPI95.588.32None
4GPT-6 AstrawithChatGPT94.288.32None
5Opus 5.5withClaude94.680.82None
6Haiku 5.5withAPI88.881.72None
7Gemini 3.8 FlashwithAPI78.965.82None
8Gemini 3.5 Flash-LitewithGemini54.438.321 capped

What you get for the cost

GPT-6 Luna · API scores within 4 points of Sonnet 5.5 · API at about a twentieth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreSonnet 5.5 · API: task score 96, 22× the cheapestGPT-6.1 Sol · API: task score 96, 9.6× the cheapestGPT-6 Luna · API: task score 92, the cheapestGPT-6 Astra · ChatGPT: task score 91, 52× the cheapestOpus 5.5 · Claude: task score 86, 36× the cheapestHaiku 5.5 · API: task score 86, the cheapestGemini 3.8 Flash · API: task score 71, 7.5× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 44, 1.8× the cheapest, 1 critical failureSonnet 5.5 · APIGPT-6.1 Sol · APIGPT-6 Luna · APIGPT-6 Astra · ChatGPTOpus 5.5 · ClaudeHaiku 5.5 · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costAt least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API921.0×
Haiku 5.5 · API861.1×
Gemini 3.5 Flash-Lite · Gemini441.8×
Gemini 3.8 Flash · API717.5×
GPT-6.1 Sol · API969.6×
Sonnet 5.5 · API9622×
Opus 5.5 · Claude8636×
GPT-6 Astra · ChatGPT9152×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Choosing the metric the whole company steers by.

Why it matters

Pick revenue and you steer by the rear-view mirror. Pick a vanity metric and teams will happily grow it while the product gets worse.

What good looks like

  • Measures value delivered to customers, not value captured
  • Grounded in what predicts retention in the data
  • Broken into input metrics a team can own
  • Says how it could be gamed, with guardrails

Deliberately not measured

  • Dashboard design
  • Naming
Capability tested

Metric design

The failure we’re looking for

Revenue, or a vanity metric, as the North Star

Grading

Decision model and LLM judge, calibrated against a blind PM review