Tasks / Metrics & Experimentation

Pick a North Star metric

Can the model pick the one metric that measures the value a product creates, and break it into inputs teams can move?

Task type v1.1 · 2 tasksLast changed 5 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 72% were usable with at most a quick edit.

Reliably right

  1. Says how it could be gamed100% pass
    Identifies gaming vectors (tapping through, notification overuse, short-term spikes, revenue manipulation, MAU drift) and sets specific guardrails (quality completion, push caps, blocking metrics, cohort retention checks, subscriber churn tracking).
    Sonnet 5.5 · API · Revenue, actives, or something else?
  2. Sees the growth is narrow100% pass
    It shows the split (chains up, independents' filled shifts and fill rate down, more leaving) and tracks independent venues via segment reporting and guardrails.
    GPT-6 Luna · API · Growing, on paper
  3. Checks the North Star can be measured100% pass
    It flags the 64% confirmation gap and proposes instrumenting check-in/out, reconciliation, and auditing to fix it before the board readout.
    GPT-6 Luna · API · Growing, on paper

Where it slips

  1. Addresses the actual decision36% pass
    The output commits to a North Star but does not state what result or condition would change that recommendation.
    GPT-6 Luna · API · Growing, on paper
  2. Identifies material uncertainty38% pass
    It identifies the confirmation gap as an unknown but does not say how resolving it could change the North Star decision.
    GPT-6.1 Sol · API · Growing, on paper
  3. Avoids unsupported claims63% pass
    It presents 'easy to tap through' and 'we don't track today' as facts without support, and does not label them as assumptions.
    Opus 5.5 · Claude · Revenue, actives, or something else?

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Measures value created, not captured

    Does the chosen North Star measure the value customers get from the product, rather than revenue or a count of people who merely showed up?

    Passes when Picks a metric of value delivered to customers (a core action that predicts retention), and keeps revenue as a separate business measure.

  2. Breaks it into inputs teams can move

    Does the output break the North Star into a few input metrics, each one a team could own and move week to week, and pair it with a separate business measure of value captured?

    Passes when Gives a small tree of input metrics, each with an owner and a link to the North Star, plus the revenue or business measure it's paired with.

  3. Says how it could be gamed

    Does the output pressure-test the North Star for how teams could raise it without delivering more value, and set guardrail metrics against that?

    Passes when Names a realistic way the metric could be gamed in this product and the guardrails that would catch it.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 4 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a Staff PM at Crewmate. Every team has been steering by its own number, and the exec team wants to pick one North Star at next week's offsite, so the CEO can report it to the board three weeks later. Write the memo they'll read beforehand, in no more than 1,200 words: the North Star you recommend and why, its metric tree with an owner for each input, the business measure it's paired with, how it could be gamed and the guardrails against that, and what each team's current metric becomes. The numbers below are as the data team sent them. Not all of it matters equally.

What the model was given9 items: About Crewmate, What each exec wants, Shifts by quarter (monthly averages), GMV (monthly average), Venues, Workers, Same-day shifts, Completion data, From the COO
About CrewmateA marketplace for hospitality shifts: venues post shifts, and vetted workers pick them up. Crewmate takes 18% of each shift's value. Two kinds of venue: four enterprise hotel and restaurant chains (310 venues between them, on annual contracts) and 2,790 independent venues (bars, restaurants, cafes) who pay as they go.
What each exec wantsCEO: GMV, 'because it's what the board looks at'. Head of Demand: shifts posted. Head of Supply: active workers. COO: fill rate.
Shifts by quarter (monthly averages)Enterprise chains: Q2 posted 21,000, filled 19,500. Q3 posted 34,000, filled 31,600. Independent venues: Q2 posted 56,000, filled 44,200. Q3 posted 58,000, filled 39,400.
GMV (monthly average)Q2 $8.5M, Q3 $9.8M. Enterprise chains were 34% of GMV in Q2 and 46% in Q3. The chains raised hourly rates by 11% for the holiday season from August.
VenuesVenues whose shifts fill 90% of the time or more: 84% still posting six months later. Under 70%: 41%. In Q3, 9% of independent venues stopped posting altogether (5% in Q2). No chain venue stopped.
WorkersRegistrations: 6,100 in Q2, 7,900 in Q3, after Supply introduced a $25 sign-up bonus in July. 2,370 of the Q3 registrants have worked a shift. Workers who work three or more shifts in their first month: 66% still working after three months; everyone else: 12%.
Same-day shiftsShifts posted less than 24 hours before they start fill 58% of the time. They are 31% of independents' posts and 8% of the chains'.
Completion dataVenues confirm in the app that a shift was worked for 64% of filled shifts; the rest are assumed worked unless the venue raises a dispute. Of confirmed shifts, 6% were no-shows or disputed.
From the COO"Since September, Ops pays workers a $15 rescue bonus to pick up same-day shifts that would otherwise go unfilled. 6,200 rescue fills last month, $93,000. Rescue-filled shifts have a 13% no-show rate, against 4% for the rest. It's kept our fill rate from falling further."
What a strong answer doesThe answer key the graders mark against

Works out from the raw numbers what the pack doesn't say: all of the growth is the four chains (filled shifts up from 19,500 to 31,600 a month, plus an 11% rate rise), while independent venues filled fewer shifts (44,200 to 39,400, so their fill rate fell from 79% to 68%) and are leaving faster (9% stopped posting, against 5%); GMV rose mostly on the chains' volume and prices. Picks a North Star of value delivered to both sides, such as shifts worked, but designs it so the decline can't hide: measured for independents separately, or as reliably staffed venues, or with an explicit independent-venue guardrail. Notices that 'worked' is only confirmed for 64% of shifts, so the North Star can't be trusted yet, and makes fixing confirmation an early input or uses a measurable proxy until then. Treats the rescue bonus as a way the fill rate (and any fills-based North Star) is being bought: 13% no-shows against 4%, so it needs guardrails such as cost per worked shift and no-show rate. Shows how shifts posted, registrations (2,370 of 7,900 worked) and GMV each rose without more value. Builds a metric tree with owners, pairs it with revenue or GMV as the business measure, and redefines each team's metric as an input.

Critical failures (cap the score at 40)
  • Recommends GMV or shifts posted as the North Star
Task

Staff-level · v1.1

The outputs

9 setups answered this brief.

Usable with a quick edit9 of 14 checks passed

The memo is evidence-based and usable with very light edits to label causal interpretations and add a condition that could change the recommendation.

Output

MEMO

To: Executive Team From: Staff PM Date: October 24 Subject: Pre-Read: Selecting Crewmate’s North Star Metric

The Diagnosis: Why Our Current Metrics Are Failing Us

On the surface, Q3 looks like a triumph: monthly GMV grew 15% to $9.8M, posted shifts hit 92,000, and worker registrations jumped 30%.

Beneath the surface, our core marketplace is cannibalizing itself because every team is optimizing for its own siloed number:

  1. GMV is masking an independent venue exodus. Enterprise chains grew from 34% to 46% of GMV in Q3, buoyed by a massive spike in posts and an 11% seasonal wage hike. This pulled worker supply away from Independent venues. Independent fill rates plunged from 78.9% to 67.9%. Waitstaff didn't disappear; they went to higher-paying, earlier-posted Enterprise shifts. Because venues with a <70% fill rate only retain at 41% (versus 84% for >90% fill rate), independent venue churn nearly doubled from 5% to 9% in Q3. We are trading our diversified, high-margin core for concentration risk in just four Enterprise clients.
  2. Demand is chasing volume over fillability. Demand hit 92,000 posted shifts, but 31% of Independent shifts are posted with less than 24 hours' notice (filling at just 58%, compared to 8% same-day posts for Enterprise). Counting unfillable, last-minute shifts as "Demand success" sets Ops up to fail.
  3. Supply is buying hollow registrations. Driven by a $25 sign-up bonus, Supply registered 7,900 workers in Q3, but 70% of them never worked a single shift. We are paying for top-of-funnel vanity while starving the marketplace of actual labor.
  4. Ops is subsidizing "fake" fills. To defend Fill Rate against same-day posts and low active supply, Ops spent $93,000 last month ($15/shift) on 6,200 "rescue fills." However, rescue shifts have a 13% no-show rate (vs. 4% normally). Worse, without those 6,200 subsidized fills, our Q3 fill rate would have been 70.4%, not 77.2%.

Most importantly: a "filled" shift is not a worked shift. Across confirmed shifts, 6% result in no-shows or disputes. A venue with a no-show has an empty floor; they don’t care that our dashboard marked the shift "filled."

---

The Recommendation: Monthly Completed Shifts (MCS)

I recommend our North Star Metric be Monthly Completed Shifts (MCS)—defined as shifts that are posted, claimed, and actually worked without a no-show or venue dispute.

Why MCS? A North Star must capture the exact moment we deliver value to both sides of the marketplace and earn our 18% take rate. * Why not GMV? GMV is an output, not an input. It is artificially inflated by wage inflation (Enterprise's 11% bump) and shift duration, masking underlying marketplace decay. * Why not Fill Rate? Fill rate is a ratio. You can achieve a 100% fill rate by firing all Independent venues and shrinking the business. Furthermore, "filled" ignores our 6% no-show problem. * Why not Shifts Posted or Active Workers? Each represents only half the marketplace. Unfilled posts cause venue churn; idle workers cause supply churn.

MCS forces alignment: Demand must source fillable shifts, Supply must activate reliable workers, and Ops must ensure completion, not just claims.

(Note on instrumentation: Currently, 36% of shifts are "assumed worked" unless disputed. Product will immediately build frictionless check-in/out to close this blind spot, though the silent-dispute baseline is accurate enough to begin tracking MCS today).

---

The Metric Tree & Ownership

$$\text{Monthly Completed Shifts} = \text{Posted Shifts} \times \text{Fill Rate} \times \text{Completion Rate}$$

Using Q3 baselines (92,000 posted $\times$ 77.2% fill $\times$ ~94% completion), we are currently at ~66,750 MCS. Here is how we break the tree down by owner:

1. Head of Demand: Quality Posted Shifts * Inputs: Active Venues $\times$ Shifts/Venue $\times$ % Posted >24hrs in Advance. * Focus: Stop Incentivizing raw post volume. Work with Independents to post schedules earlier (reducing their 31% same-day post rate) and price shifts competitively against Enterprise.

2. Head of Supply: Activated & Retained Workers * Inputs: New Worker Activation (% hitting $\ge$3 shifts in Month 1) + Retained Active Workers. * Focus: Our data shows workers who complete $\ge$3 shifts in their first month retain at 66% at month three, vs. 12% for everyone else. Supply must shift the $25 bonus from registration to completing the 3rd shift.

3. COO (Ops) & Product: Match Efficiency & Reliability * Inputs: Fill Rate (segmented by Enterprise vs. Independent) $\times$ Completion Rate ($1 - \text{No-Show/Dispute Rate}$). * Focus: Replace blanket $15 rescue bonuses with algorithmic reliability scoring (penalizing no-shows, rewarding streaks) and better matching to keep Independent venues above the critical 90% fill threshold.

---

The Paired Business Measure: Net Contribution Margin (or Net Revenue)

To satisfy the Board and prevent us from buying growth unprofitably, MCS must be paired with Net Contribution Margin (Gross Revenue minus direct variable incentives like the $15 rescue and $25 sign-up bonuses).

In Q3, we averaged $9.8M in GMV, yielding ~$1.76M in gross revenue (at 18%). However, accounting for the ~6% no-show rate (~$105k in uncollected fees) and ~$160k/month in worker bonuses ($93k rescue + ~$66k sign-up), we burned over 15% of our gross revenue on flawed incentives. Pairing MCS with Net Contribution Margin proves to the board that operational discipline drives durable GMV.

---

Gaming the Metric & Our Guardrails

Every metric can be gamed. Here is how teams might game MCS, and the guardrails we will lock in to prevent it:

  • Gaming Risk 1: Over-indexing on Enterprise. Because Enterprise shifts fill at 93% with low same-day rates, teams could abandon Independents entirely to juice MCS.
  • Guardrail: % of Independent Venues with $\ge$80% Completion Rate. We must protect the long-tail marketplace. If Independent health drops, we fail.
  • Gaming Risk 2: Splitting shifts. Demand could ask venues to turn one 8-hour shift into two 4-hour shifts to double MCS.
  • Guardrail: Average Hours (or GMV) per Shift. Must remain stable within historical bands.
  • Gaming Risk 3: Buying completions. Ops and Supply could ramp up $15 and $25 bonuses to force completions at a financial loss.
  • Guardrail: Incentive Spend as a % of Gross Revenue (capped via our paired business metric).

---

Summary: What Your Current Metric Becomes

For the Board and our weekly operating reviews, here is how our scorecard evolves:

ExecutiveOld MetricWhat It Becomes Under MCS
:---:---:---
CEOGMVBoard Financial Output. Reported alongside MCS and Net Contribution Margin as the financial byproduct of completed shifts.
DemandShifts PostedLead-Time Qualified Posts. Evaluated on growing shifts posted >24 hours in advance, segmented by Independent vs. Enterprise.
SupplyActive WorkersActivated Workers ($\ge$3 Shifts). Evaluated on new workers reaching the 3-shift retention tipping point in Month 1, and total retained cohort size.
COOFill RateCompletion Rate & Venue Reliability. Evaluated on minimizing the 6% no-show gap and maximizing the % of venues hitting the 90% completion retention threshold.

Grades and run details

Decision model 71 · LLM judge 80
Decision model checks
  • failUses the supplied evidence correctly46%
  • partialAddresses the actual decision79%
  • passRespects explicit constraints41%
  • partialIdentifies material uncertainty77%
  • failAvoids unsupported claims41%
  • passProduces the required deliverable83%
  • passSees the growth is narrow98%
  • passChecks the North Star can be measured98%
  • passSees the fill rate being bought100%
  • passOne North Star for both sides100%
  • passMeasures value created, not captured98%
  • failGets the base of every number right46%
  • passBreaks it into inputs teams can move100%
  • passSays how it could be gamed100%
Run
Run
#1
API response time
20 s
Submitted
13 Oct 2026

Usable with a quick edit13 of 14 checks passed

The memo is well-structured, uses evidence correctly, and covers all required elements. A PM could use it with minor wording edits, without re-checking data or reasoning.

Output

To: Executive team Subject: Choose successful completed shifts as Crewmate’s North Star

Recommendation

Adopt monthly successful completed shifts as Crewmate’s North Star: the number of unique posted shifts that are filled, actually worked, and have no unresolved dispute. Count a shift once, after its scheduled end; exclude cancellations, no-shows, and disputed shifts until resolved.

This measures the outcome venues hire us for and workers show up to deliver—not activity on either side of the marketplace. It also directs attention to the reliability that supports repeat use. Venues with fill rates of at least 90% are much more likely to still be posting six months later than venues below 70% (84% vs. 41%).

The latest figures show why the distinction matters. Monthly posts rose from 77,000 in Q2 to 92,000 in Q3, while fills rose from 63,700 to 71,000. But overall fill rate fell from 82.7% to 77.2%; independent-venue fill rate fell from 78.9% to 67.9%, and independent venues stopping posts rose from 5% to 9%. Meanwhile, GMV grew 15%, but chains’ share of GMV rose from 34% to 46% and they raised rates 11% for the holiday season. GMV is important, but this mix and price change make it a poor standalone signal of marketplace health.

Nor should we equate registrations with supply: only 2,370 of 7,900 Q3 registrants worked a shift, after the sign-up bonus began. Workers who complete three or more shifts in their first month are far more likely to remain active. We should reward dependable activity, not sign-ups.

Metric tree and accountability

The core identity is:

Successful completed shifts = valid posted shifts × fill rate × successful-completion rate

InputDefinitionAccountable owner
Valid posted shiftsUnique, legitimate shifts with a real venue, role, time, and pay; split by enterprise/independent and lead timeHead of Demand
Fill rateFilled shifts ÷ valid posted shiftsCOO
Successful-completion rateShifts actually worked without a no-show or unresolved dispute ÷ filled shiftsCOO; Product/Data owns reliable status capture
Qualified, available worker coverageRepeat, activated workers available for the shifts and locations demanded; leading driver of fill and completionHead of Supply
Repeat venue demandActive venues returning to post, with retention and posting frequency by venue segmentHead of Demand

The first three inputs calculate the North Star; worker coverage and repeat venue demand are leading drivers and diagnostic cuts, not extra terms to multiply into it. Review all results by enterprise vs. independent, and scheduled vs. same-day.

Paired business measure

Pair the North Star with monthly contribution margin dollars: Crewmate’s 18% take-rate revenue on completed-shift GMV, less variable fulfillment costs, including rescue bonuses and other worker incentives. This checks that completed shifts create sustainable value rather than being purchased at any cost. Continue reporting GMV to the board as a scale measure, but not as our North Star.

Measurement, gaming, and guardrails

Completion data is not yet strong enough for an unqualified claim: venues confirm only 64% of filled shifts, and the rest are assumed worked unless disputed. Among confirmed shifts, 6% were no-shows or disputed; rescue-filled shifts have a 13% no-show rate versus 4% for other fills. For the first board readout, label the metric provisional, disclose the confirmation rate, and show the calculation and coverage. Then instrument worker and venue check-in/out, reconcile mismatches, and audit a sample. Do not treat missing confirmation as proof of success indefinitely.

Key gaming risks and controls:

  • Post inflated, duplicate, or low-quality shifts: count only valid shifts; monitor cancellations, disputes, and fill rate.
  • Favor easy or high-priced chain shifts over independent or same-day demand: report North Star and fill/completion rates by segment; track independent-venue retention and same-day performance as guardrails.
  • Buy apparent success with rescue bonuses: monitor rescue share, no-show rate, and contribution margin alongside completed shifts. Rescue fills are not successes unless the shift is actually worked.
  • Misreport completion or suppress disputes: require auditable status capture and monitor confirmation coverage and dispute rates.

What current metrics become

  • CEO — GMV: remains a board/business measure, paired with contribution margin; it is not the North Star.
  • Demand — shifts posted: becomes a leading input, with valid posts and repeat independent-venue demand emphasized over raw volume.
  • Supply — active workers: shifts to qualified, repeat workers and first-month activation (especially the three-shift threshold); registrations are a funnel diagnostic.
  • COO — fill rate: remains a critical driver and guardrail, but success is completed shifts—not fills that fail to show up.

At the offsite, agree on the definition, owners, and guardrails. Before the board update, establish the provisional baseline and be explicit about its measurement limits.

Grades and run details

Decision model 93 · LLM judge 93
Decision model checks
  • passUses the supplied evidence correctly51%
  • partialAddresses the actual decision79%
  • passRespects explicit constraints76%
  • partialIdentifies material uncertainty94%
  • passAvoids unsupported claims40%
  • passProduces the required deliverable86%
  • passSees the growth is narrow98%
  • passChecks the North Star can be measured100%
  • passSees the fill rate being bought98%
  • passOne North Star for both sides100%
  • passMeasures value created, not captured100%
  • passGets the base of every number right53%
  • passBreaks it into inputs teams can move100%
  • passSays how it could be gamed100%
Run
Run
#1
API response time
37 s
Submitted
5 Oct 2026

Usable with a quick edit9 of 14 checks passed

The memo is comprehensive and well-reasoned; the only error is a minor misstatement about bonus spend allocation, which can be fixed with a quick edit.

Output

MEMORANDUM

TO: Executive Team FROM: Staff Product Manager DATE: October 24, 2024 SUBJECT: Crewmate North Star Metric Recommendation for Offsite & Board Reporting

---

1. Executive Summary: The False Comfort of Top-Line Growth

In Q3, GMV grew 15.3% to $9.8M. On the surface, the marketplace looks healthy. In reality, that growth masks an accelerating decay in our core business.

The GMV increase was driven almost entirely by four enterprise clients: they expanded shift volume by 62% and raised hourly rates by 11%, increasing their share of GMV from 34% to 46%. Meanwhile, our 2,790 independent venues—the diversified bedrock of our marketplace—are failing: * Independent shift volume dropped by 4,800 fills/month (-11%) despite posts increasing to 58,000. * Independent fill rate plunged from 78.9% to 67.9%, falling below the critical 70% threshold where 6-month retention collapses from 84% to 41%. * Independent venue quarterly churn nearly doubled from 5% to 9%. * Operations spent $93,000 last month ($1.1M annualized) on $15 "rescue bonuses" to prop up same-day fill rates, which carry an unacceptable 13% no-show rate. * Supply spent heavily on $25 sign-up bonuses; 70% of those new sign-ups never worked a single shift.

Each team is currently optimizing for a siloed proxy that actively harms the broader platform. We must align around a single, un-gameable measure of marketplace health.

---

2. Recommended North Star Metric: Successfully Completed Shifts (SCS)

Definition: The count of shifts that are matched, worked, and completed without a dispute or no-show.

$$\text{Successfully Completed Shifts (SCS)} = \text{Shifts Claimed} \times (1 - \text{No-Show / Dispute Rate})$$

#### Why This Metric? 1. Captures Mutual Value Exchange: Venues don't pay for posted shifts or claimed shifts; they pay for labor that shows up. Workers don't benefit from registrations; they benefit from earned wages. SCS represents the exact transaction where value is realized and Crewmate earns its 18% fee. 2. Exposes False Liquidity: Fills that result in no-shows (currently 13% for rescue shifts and 6% baseline) destroy venue trust. Fill rate treats a no-show as a success until a dispute occurs; SCS counts only real completions. 3. Corrects Cross-Functional Misalignment: To grow SCS, Demand cannot post unfillable shifts, Supply cannot recruit ghost workers, and Ops cannot paper over structural matching issues with subsidy-driven, unreliable fills.

---

3. Metric Tree and Input Ownership

``` [North Star: Successfully Completed Shifts (SCS)] | +--------------------------------+--------------------------------+ | | [Fill Volume] [Fulfillment Reliability] | | +-----+-------------------------+ +-------------+-------------+ | | | | [Fillable Shift Demand] [Retained Worker Supply] [Attendance Rate] [Dispute-Free Rate] (Demand Team) (Supply Team) (Operations) (Product/Ops) ```

Each tier-one input has a single owner accountable for driving it:

Input MetricOwnerStrategic Objective
:---:---:---
1. Fillable Shift DemandHead of DemandMaximize shifts posted with $>24\text{h}$ lead time. Shift distribution currently sets supply up to fail: 31% of indie posts are same-day ($<24\text{h}$), converting at an abysmal 58% fill rate.
2. Activated Core WorkersHead of SupplyWorkers completing $\ge 3$ shifts in month one. Workers reaching this threshold have a 66% 3-month retention rate, compared to just 12% for those who do not. Vanity registrations must be abandoned.
3. Match Efficiency & Fill RateHead of ProductMatch speed and algorithmic discovery. Product must improve worker routing to high-churn independent venues without relying on manual operational interventions.
4. Fulfillment ReliabilityHead of OperationsEliminate no-shows and unconfirmed shifts. Currently, 36% of shifts require passive auto-confirmation, and rescue-bonus fills have more than triple the baseline no-show rate (13% vs. 4%).

---

4. Paired Business Measure: Net Contribution Margin per Shift

A North Star can encourage excessive spending to generate volume. To ensure sustainable, profitable growth, SCS must be paired with Net Contribution Margin per Completed Shift.

$$\text{Net Contribution Margin} = (\text{Take Rate} \times \text{Shift Value}) - (\text{Acquisition Subsidies} + \text{Rescue Bonuses} + \text{Dispute Costs})$$

  • Why it matters: In Q3, Ops spent $93,000/month on rescue fills, and Supply spent $197,500 on registration bonuses for workers who never activated. Subsidies directly cannibalized the marketplace margin.
  • The Constraint: Teams cannot buy SCS growth through operational bailouts or uneconomic supply acquisition. Every completed shift must generate positive net platform margin.

---

5. Gaming Vectors and Guardrails

VectorHow It Could Be GamedMandatory Guardrail
:---:---:---
The "Rescue" LoopholeOps artificially inflates SCS by deploying bonuses to rescue low-lead-time shifts at high unit cost.Max 3% Subsidized Shifts: Subsidized shifts capped at 3% of total fills; rescue fills with $>5\%$ no-show rate forfeit bonus payouts.
Enterprise BiasTeams focus exclusively on the 4 enterprise accounts to hit raw SCS volume, abandoning the higher-margin independent base.Independent Floor: Independent venue SCS must account for $\ge 55\%$ of total SCS; independent venue 6-month retention must stay $\ge 80\%$.
Passive ConfirmationsTeams ignore verification because unconfirmed shifts (36%) default to "worked" status after a dispute window.Verification Coverage: $\ge 85\%$ active venue verification in-app before a shift is classified as completed in the primary SCS reporting.

---

6. Re-anchoring the Executive Team: What Current Metrics Become

To establish SCS as our operational standard, the executive team must reframe their existing target metrics from headline drivers to supporting inputs:

#### CEO: GMV $\rightarrow$ The Enterprise Valuation Output * What it becomes: A downstream financial output metric for quarterly investor updates. * Why: GMV inflated by rate hikes and enterprise concentration gave false confidence while 9% of our independent client base vanished. GMV will grow healthily only if SCS grows sustainably.

#### Head of Demand: Shifts Posted $\rightarrow$ Liquidity-Adjusted Posts * What it becomes: Advance-notice shifts posted ($>24\text{h}$ lead time) with market-clearing rates. * Why: Unfilled posts drive churn. Posting low-lead-time, below-market shifts destroys venue trust when fill rates sit at 58%. Demand's target is shifts that can realistically clear.

#### Head of Supply: Active Workers $\rightarrow$ Activated Core Workers ($3+$ shifts/month) * What it becomes: Cohort activation rate within 30 days of onboarding. * Why: Raw registrations jumped to 7,900 via a $25 bonus, but only 30% ever worked. Focusing on workers reaching 3+ shifts increases 90-day retention by $5.5\times$ (66% vs. 12%), creating dependable supply.

#### COO: Gross Fill Rate $\rightarrow$ Net Verified Completion Rate * What it becomes: $\text{Completed Shifts} \div \text{Posted Shifts}$ net of cancellations, no-shows, and rescue subsidies. * Why: Buying fill rate with a 13% no-show rate and a $93,000 monthly subsidy masks operational deficits. Operations must optimize for clean, reliable shifts, not emergency rescue band-aids.

---

7. Recommendation for the Offsite

At next week's offsite, we should formally adopt Successfully Completed Shifts (SCS) as our sole operational North Star, paired with Net Contribution Margin per Shift.

Adopting SCS unifies product, supply, demand, and operations behind marketplace quality, directly targets our independent churn crisis, and presents the Board with a repeatable growth engine rather than masked churn.

Claims the judge couldn’t find in the brief

  • Supply spent $197,500 on registration bonuses for workers who never activated.

Grades and run details

Decision model 71 · LLM judge 73
Decision model checks
  • failUses the supplied evidence correctly72%
  • partialAddresses the actual decision77%
  • passRespects explicit constraints35%
  • partialIdentifies material uncertainty37%
  • failAvoids unsupported claims28%
  • passProduces the required deliverable64%
  • passSees the growth is narrow100%
  • passChecks the North Star can be measured98%
  • passSees the fill rate being bought100%
  • passOne North Star for both sides100%
  • passMeasures value created, not captured98%
  • failGets the base of every number right34%
  • passBreaks it into inputs teams can move100%
  • passSays how it could be gamed100%
Run
Run
#1
API response time
29 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Uses the supplied evidence correctlyMixedRightWrong
Gemini 4 Argon · API

All factual claims about the current situation stem directly from the supplied context or correct arithmetic on it.

GPT-6 Luna · API

All claims about the current situation are taken directly from the brief or derived by correct arithmetic.

Gemini 3.8 Flash · API

The claim that Supply spent $197,500 on registration bonuses for workers who never activated is not supported; the total spend was $197,500, but some of those workers did activate.

Identifies material uncertaintyWrongRightWrong
Gemini 4 Argon · API

It does not name specific unknowns (e.g., board insistence on GMV, independent churn acceleration) that could change the decision, nor explain how they would be resolved.

GPT-6 Luna · API

It names the confirmation gap as a key unknown, bounds it with the 64% figure, and says how to resolve it (instrument check-in/out, reconcile, audit).

Gemini 3.8 Flash · API

The output does not identify any unknowns that could change the decision or how they would be resolved.

Avoids unsupported claimsWrongRightWrong
Gemini 4 Argon · API

Causal claims like 'This pulled worker supply away' and 'Driven by a $25 sign-up bonus... hollow registrations' are presented as fact, not labelled as hypotheses.

GPT-6 Luna · API

Interpretations are presented as conclusions from the data, not as established facts, and no unsupported causal claims are made.

Gemini 3.8 Flash · API

The output presents the $197,500 as spent on non-activated workers as fact, which is not established by the evidence.

Gets the base of every number rightMixedRightMixed
Gemini 4 Argon · API

All derived figures (fill rates, churn, no-show impact, MCS calculation) are computed correctly from the given data and clear steps.

GPT-6 Luna · API

All derived percentages and rates are computed correctly from the supplied data, and bases are stated where needed.

Gemini 3.8 Flash · API

All derived percentages and rates are correctly computed from the supplied data.

All got wrong 1

Addresses the actual decisionWrongWrongWrong
Gemini 4 Argon · API

The memo firmly recommends Monthly Completed Shifts as the North Star but never states what result or condition would change that recommendation.

GPT-6 Luna · API

The output commits to a North Star but does not state what result or condition would change that recommendation.

Gemini 3.8 Flash · API

The output does not state what result or condition would change the recommendation.

All got right 9

Respects explicit constraintsRightRightRight
Gemini 4 Argon · API

The response is a memo within the word limit, includes all required elements, and is addressed to the executive team.

GPT-6 Luna · API

The output is a memo under 1,200 words, addressed to the exec team, and includes all requested elements.

Gemini 3.8 Flash · API

The memo is within 1,200 words, in memo form, and addresses all requested elements.

Produces the required deliverableRightRightRight
Gemini 4 Argon · API

It delivers a complete, actionable memo with North Star, tree, business metric, guardrails, and team metric redefinitions, usable by the exec team.

GPT-6 Luna · API

The memo is complete, within word count, and usable by the exec team with all required components.

Gemini 3.8 Flash · API

The memo includes the North Star, metric tree with owners, business measure, gaming and guardrails, and redefined team metrics, and is usable.

Sees the growth is narrowRightRightRight
Gemini 4 Argon · API

It clearly breaks out the independent venue decline (filled shifts down, fill rate drop, higher churn) and adds an explicit independent-venue guardrail.

GPT-6 Luna · API

It shows the split (chains up, independents' filled shifts and fill rate down, more leaving) and tracks independent venues via segment reporting and guardrails.

Gemini 3.8 Flash · API

The memo clearly shows the split between enterprise growth and independent decline, and includes an independent floor guardrail.

Checks the North Star can be measuredRightRightRight
Gemini 4 Argon · API

It flags that 36% of shifts are unconfirmed and proposes building check-in/out to close the gap, while noting the current proxy is adequate to start.

GPT-6 Luna · API

It flags the 64% confirmation gap and proposes instrumenting check-in/out, reconciliation, and auditing to fix it before the board readout.

Gemini 3.8 Flash · API

The memo flags the 36% unconfirmed shifts and proposes a verification coverage guardrail to ensure measurability.

Sees the fill rate being boughtRightRightRight
Gemini 4 Argon · API

It uses the 13% vs 4% no-show rates and $93k cost to show rescue fills inflate the fill number and sets an incentive-spend guardrail.

GPT-6 Luna · API

It uses the 13% no-show rate and cost to show rescue fills inflate the number, and sets guardrails like no-show rate and contribution margin.

Gemini 3.8 Flash · API

The memo uses the 13% no-show rate and cost to highlight the issue and sets a guardrail capping subsidized shifts and no-show rate.

One North Star for both sidesRightRightRight
Gemini 4 Argon · API

MCS requires a shift to be completed, capturing value for both venue and worker, and the memo explicitly argues this marketplace value.

GPT-6 Luna · API

The North Star (successful completed shifts) only rises when a venue and a worker both get value, and the memo explains why that matters in a marketplace.

Gemini 3.8 Flash · API

SCS captures value for both venues and workers, as it requires a completed shift without dispute.

Measures value created, not capturedRightRightRight
Gemini 4 Argon · API

Completed shifts measure actual work delivered, not revenue or mere presence, and revenue is kept as a paired business measure.

GPT-6 Luna · API

The North Star measures completed shifts, a core action that predicts retention, and keeps revenue as a separate business measure.

Gemini 3.8 Flash · API

SCS measures value delivered (completed shifts), while Net Contribution Margin is the separate business measure.

Breaks it into inputs teams can moveRightRightRight
Gemini 4 Argon · API

It breaks MCS into owned inputs (demand, supply, match/reliability) with sub-metrics and pairs it with Net Contribution Margin as the business measure.

GPT-6 Luna · API

It breaks the North Star into valid posted shifts, fill rate, and successful-completion rate with owners, and pairs it with contribution margin.

Gemini 3.8 Flash · API

The metric tree breaks SCS into fill volume and fulfillment reliability, with owners for each input, and pairs with Net Contribution Margin.

Says how it could be gamedRightRightRight
Gemini 4 Argon · API

Three gaming scenarios (over-index enterprise, split shifts, buy completions) are identified with corresponding guardrail metrics.

GPT-6 Luna · API

It names realistic gaming risks (inflated posts, favoring chain shifts, buying fills, misreporting) and sets specific guardrails against each.

Gemini 3.8 Flash · API

The memo identifies gaming vectors like rescue loophole, enterprise bias, and passive confirmations, with specific guardrails.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5withAPI90.196.72None
2GPT-6.1 SolwithAPI91.993.32None
3GPT-6 LunawithAPI91.988.32None
4GPT-6 AstrawithChatGPT88.388.32None
5Haiku 5.5withAPI83.081.72None
6Opus 5.5withClaude84.380.82None
7Gemini 3.8 FlashwithAPI76.665.82None
8Gemini 4 ArgonwithAPI76.665.02None
9Gemini 3.5 Flash-LitewithGemini54.438.321 capped

What you get for the cost

GPT-6 Luna · API scores within 4 points of Sonnet 5.5 · API at about a twentieth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreSonnet 5.5 · API: task score 94, 22× the cheapestGPT-6.1 Sol · API: task score 93, 9.6× the cheapestGPT-6 Luna · API: task score 90, the cheapestGPT-6 Astra · ChatGPT: task score 89, 52× the cheapestHaiku 5.5 · API: task score 83, the cheapestOpus 5.5 · Claude: task score 81, 36× the cheapestGemini 3.8 Flash · API: task score 70, 7.5× the cheapestGemini 4 Argon · API: task score 70, 8.7× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 44, 1.8× the cheapest, 1 critical failureSonnet 5.5 · APIGPT-6.1 Sol · APIGPT-6 Luna · APIGPT-6 Astra · ChatGPTHaiku 5.5 · APIOpus 5.5 · ClaudeGemini 3.8 Flash · APIGemini 4 Argon · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costAt least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API901.0×
Haiku 5.5 · API831.1×
Gemini 3.5 Flash-Lite · Gemini441.8×
Gemini 3.8 Flash · API707.5×
Gemini 4 Argon · API708.7×
GPT-6.1 Sol · API939.6×
Sonnet 5.5 · API9422×
Opus 5.5 · Claude8136×
GPT-6 Astra · ChatGPT8952×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Choosing the metric the whole company steers by.

Why it matters

Pick revenue and you steer by the rear-view mirror. Pick a vanity metric and teams will happily grow it while the product gets worse.

What good looks like

  • Measures value delivered to customers, not value captured
  • Grounded in what predicts retention in the data
  • Broken into input metrics a team can own
  • Says how it could be gamed, with guardrails

Deliberately not measured

  • Dashboard design
  • Naming
Capability tested

Metric design

The failure we’re looking for

Revenue, or a vanity metric, as the North Star

Grading

Decision model and LLM judge, calibrated against a blind PM review