Tasks / Metrics & Experimentation

Pick a North Star metric

Can the model pick the one metric that measures the value a product creates, and break it into inputs teams can move?

Task type v1.1 · 2 tasksLast changed 5 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 75% were usable with at most a quick edit.

Reliably right

  1. Says how it could be gamed100% pass
    Identifies gaming vectors (tapping through, notification overuse, short-term spikes, revenue manipulation, MAU drift) and sets specific guardrails (quality completion, push caps, blocking metrics, cohort retention checks, subscriber churn tracking).
    Sonnet 5.5 · API · Revenue, actives, or something else?
  2. Sees the growth is narrow100% pass
    It shows the split (chains up, independents' filled shifts and fill rate down, more leaving) and tracks independent venues via segment reporting and guardrails.
    GPT-6 Luna · API · Growing, on paper
  3. Checks the North Star can be measured100% pass
    It flags the 64% confirmation gap and proposes instrumenting check-in/out, reconciliation, and auditing to fix it before the board readout.
    GPT-6 Luna · API · Growing, on paper

Where it slips

  1. Identifies material uncertainty55% pass
    It identifies the confirmation gap as an unknown but does not say how resolving it could change the North Star decision.
    GPT-6.1 Sol · API · Growing, on paper
  2. Addresses the actual decision58% pass
    The output commits to a North Star but does not state what result or condition would change that recommendation.
    GPT-6 Luna · API · Growing, on paper
  3. Avoids unsupported claims67% pass
    It presents 'easy to tap through' and 'we don't track today' as facts without support, and does not label them as assumptions.
    Opus 5.5 · Claude · Revenue, actives, or something else?

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Measures value created, not captured

    Does the chosen North Star measure the value customers get from the product, rather than revenue or a count of people who merely showed up?

    Passes when Picks a metric of value delivered to customers (a core action that predicts retention), and keeps revenue as a separate business measure.

  2. Breaks it into inputs teams can move

    Does the output break the North Star into a few input metrics, each one a team could own and move week to week, and pair it with a separate business measure of value captured?

    Passes when Gives a small tree of input metrics, each with an owner and a link to the North Star, plus the revenue or business measure it's paired with.

  3. Says how it could be gamed

    Does the output pressure-test the North Star for how teams could raise it without delivering more value, and set guardrail metrics against that?

    Passes when Names a realistic way the metric could be gamed in this product and the guardrails that would catch it.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 4 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a Staff PM at Crewmate. Every team has been steering by its own number, and the exec team wants to pick one North Star at next week's offsite, so the CEO can report it to the board three weeks later. Write the memo they'll read beforehand, in no more than 1,200 words: the North Star you recommend and why, its metric tree with an owner for each input, the business measure it's paired with, how it could be gamed and the guardrails against that, and what each team's current metric becomes. The numbers below are as the data team sent them. Not all of it matters equally.

What the model was given9 items: About Crewmate, What each exec wants, Shifts by quarter (monthly averages), GMV (monthly average), Venues, Workers, Same-day shifts, Completion data, From the COO
About CrewmateA marketplace for hospitality shifts: venues post shifts, and vetted workers pick them up. Crewmate takes 18% of each shift's value. Two kinds of venue: four enterprise hotel and restaurant chains (310 venues between them, on annual contracts) and 2,790 independent venues (bars, restaurants, cafes) who pay as they go.
What each exec wantsCEO: GMV, 'because it's what the board looks at'. Head of Demand: shifts posted. Head of Supply: active workers. COO: fill rate.
Shifts by quarter (monthly averages)Enterprise chains: Q2 posted 21,000, filled 19,500. Q3 posted 34,000, filled 31,600. Independent venues: Q2 posted 56,000, filled 44,200. Q3 posted 58,000, filled 39,400.
GMV (monthly average)Q2 $8.5M, Q3 $9.8M. Enterprise chains were 34% of GMV in Q2 and 46% in Q3. The chains raised hourly rates by 11% for the holiday season from August.
VenuesVenues whose shifts fill 90% of the time or more: 84% still posting six months later. Under 70%: 41%. In Q3, 9% of independent venues stopped posting altogether (5% in Q2). No chain venue stopped.
WorkersRegistrations: 6,100 in Q2, 7,900 in Q3, after Supply introduced a $25 sign-up bonus in July. 2,370 of the Q3 registrants have worked a shift. Workers who work three or more shifts in their first month: 66% still working after three months; everyone else: 12%.
Same-day shiftsShifts posted less than 24 hours before they start fill 58% of the time. They are 31% of independents' posts and 8% of the chains'.
Completion dataVenues confirm in the app that a shift was worked for 64% of filled shifts; the rest are assumed worked unless the venue raises a dispute. Of confirmed shifts, 6% were no-shows or disputed.
From the COO"Since September, Ops pays workers a $15 rescue bonus to pick up same-day shifts that would otherwise go unfilled. 6,200 rescue fills last month, $93,000. Rescue-filled shifts have a 13% no-show rate, against 4% for the rest. It's kept our fill rate from falling further."
What a strong answer doesThe answer key the graders mark against

Works out from the raw numbers what the pack doesn't say: all of the growth is the four chains (filled shifts up from 19,500 to 31,600 a month, plus an 11% rate rise), while independent venues filled fewer shifts (44,200 to 39,400, so their fill rate fell from 79% to 68%) and are leaving faster (9% stopped posting, against 5%); GMV rose mostly on the chains' volume and prices. Picks a North Star of value delivered to both sides, such as shifts worked, but designs it so the decline can't hide: measured for independents separately, or as reliably staffed venues, or with an explicit independent-venue guardrail. Notices that 'worked' is only confirmed for 64% of shifts, so the North Star can't be trusted yet, and makes fixing confirmation an early input or uses a measurable proxy until then. Treats the rescue bonus as a way the fill rate (and any fills-based North Star) is being bought: 13% no-shows against 4%, so it needs guardrails such as cost per worked shift and no-show rate. Shows how shifts posted, registrations (2,370 of 7,900 worked) and GMV each rose without more value. Builds a metric tree with owners, pairs it with revenue or GMV as the business measure, and redefines each team's metric as an input.

Critical failures (cap the score at 40)
  • Recommends GMV or shifts posted as the North Star
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Usable with a quick edit12 of 14 checks passed

The memo is thorough and evidence-based; adding a sentence on what would change the North Star choice is a quick edit.

Output

Pre-read: One North Star for Crewmate

To: Executive team From: Staff PM Decision requested: Adopt reliably completed shifts per month as our North Star, paired with contribution profit, and retire posted shifts, active workers, fill rate, and GMV as competing definitions of success.

Why this is the right North Star

Crewmate creates value when a venue gets the labor it needs and a worker completes paid work—not when a shift is posted, someone registers, or a booking is made.

Our current metrics tell conflicting stories:

Monthly averageQ2Q3
------:---:
GMV$8.5M$9.8M
Total filled shifts63,70071,000
Overall fill rate82.7%77.2%
Enterprise fill rate92.9%92.9%
Independent fill rate78.9%67.9%
Independent filled shifts44,20039,400

GMV grew 15%, but independent filled shifts fell 11%. Enterprise expansion masked that deterioration; chains also increased hourly rates by 11% from August, lifting GMV without necessarily delivering more work. Enterprise share of GMV rose from 34% to 46%.

This matters for retention. Venues with fill rates of at least 90% have 84% six-month retention, versus 41% below 70%. That is an association, not proof of causation, but it is a strong warning: independent fill rate is now below 70%, and independent posting cessation rose from 5% to 9%.

Supply growth is similarly misleading. Registrations increased after the bonus, but only 2,370 of 7,900 Q3 registrants have worked a shift. Workers completing three shifts in their first month have much stronger subsequent retention.

Finally, a fill is not delivery. Rescue bonuses generated 6,200 fills last month, but those bookings had a 13% no-show rate versus 4% elsewhere.

Reliable completion aligns Demand, Supply, and Ops around the marketplace’s actual promise.

Definition and measurement

Count a shift once when:

  • The worker actually performed the agreed work.
  • Payment is settled.
  • Any material attendance or service dispute is resolved.

Use the shift’s scheduled date for reporting, with a published maturation window and subsequent restatements. Exclude cancellations, no-shows, and materially unfulfilled shifts. Count the original shift once, even if a replacement worker was needed.

We cannot yet report a trustworthy baseline. Only 64% of filled shifts receive venue confirmation; the rest are assumed completed unless disputed. The reported 6% failure rate among confirmed shifts cannot safely be extrapolated to unconfirmed shifts.

Data and Ops will establish attendance verification using venue confirmation or corroborated worker attendance evidence. Payment without a dispute is not sufficient evidence by itself. Until coverage is adequate, report an explicitly labeled, stratified audit estimate, its uncertainty, and verification coverage—not an invented historical completion series.

Metric tree and accountable owners

For each segment:

Reliable completions = eligible shifts posted × booking fill rate × reliable-completion rate among booked shifts

The North Star is the sum across segments. Every input is reported separately for enterprise/independent and same-day/advance-notice shifts.

InputAccountable ownerOperating levers
Eligible shifts postedHead of DemandRetained posting venues, shifts per venue, clear requirements, competitive pay, earlier posting
Booking fill rateHead of SupplyQualified available worker-hours by location/time/role, matching, acceptance, repeat participation
Reliable-completion rateCOOAttendance, cancellation prevention, replacement quality, dispute resolution
Verification coverage and estimate qualityHead of DataEvidence capture, representative audits, missingness checks, reconciled reporting

Worker activation—especially reaching three completed shifts in the first month—is a Supply input, not a second North Star. Demand owns advance-notice share; Supply and Ops own matching and delivery within each notice category.

Mix protection is mandatory: aggregate growth is not a green result if independent reliable completions or independent service reliability decline. Report both segment outcomes alongside the total, with independent posting cessation as a retention guardrail. First stop deterioration; then set recovery targets against the 90% venue-reliability benchmark after baselining.

Business measure: contribution profit

Pair the North Star with monthly contribution profit: earned marketplace fees less variable fulfillment costs, including rescue and acquisition incentives, refunds, payment costs, and variable support costs. Finance owns its definition and reconciliation.

The 18% take rate translates reported GMV into approximately $1.53M in Q2 and $1.76M in Q3 monthly fees, before adjustments and costs. That is not profit. Last month’s rescue bonuses alone cost $93,000; we should not subtract that month’s expense from a quarterly-average figure and call it a reconciled result.

The board still receives GMV, with a bridge separating shift volume, average shift value, segment mix, and pricing. GMV remains a financial scale indicator—not the product’s success criterion.

Gaming risks and guardrails

RiskGuardrail
Buy bookings with bonuses, regardless of attendanceCount completion, not booking; monitor rescue completion, incremental cost, and contribution per completed shift
Inflate completions through default confirmationIndependent attendance audits; publish verification coverage and disputes; do not reward instrumentation gains as delivery gains
Split work into many short shiftsStable original-shift IDs; monitor completed hours and shift-duration mix; audit fragmentation
Favor easy enterprise work and neglect independentsMandatory segment scorecards and independent non-deterioration guardrails
Suppress difficult posts to improve fill rateMonitor eligible demand, rejected/cancelled posts, same-day demand, and venue retention
Rush workers or pressure venues to conceal problemsTrack worker complaints, payment timeliness, safety incidents, and dispute reversals

What each current metric becomes

  • CEO—GMV: Board reporting and financial reconciliation, paired with contribution profit.
  • Demand—shifts posted: A demand input, qualified by eligibility, notice period, and venue retention.
  • Supply—active workers: A capacity diagnostic; prioritize qualified availability, first-month activation, and retained productive workers over registrations.
  • COO—fill rate: An intermediate conversion metric, always paired with completion quality and fulfillment cost.

Before the board meeting

At the offsite, approve the definition, owners, and segment guardrails—not an unsupported growth target. Over the following three weeks, Data and Ops will audit completion by segment, notice period, and rescue status; Finance will reconcile contribution profit; teams will produce the first shared scorecard.

The CEO can then report what we know, what remains estimated, and our independent-marketplace recovery plan. A credible baseline is more valuable than another flattering number.

Grades and run details

Decision model 100 · LLM judge 87
Decision model checks
  • passUses the supplied evidence correctly46%
  • passAddresses the actual decision86%
  • passRespects explicit constraints44%
  • passIdentifies material uncertainty95%
  • passAvoids unsupported claims48%
  • passProduces the required deliverable80%
  • passSees the growth is narrow100%
  • passChecks the North Star can be measured100%
  • passSees the fill rate being bought100%
  • passOne North Star for both sides100%
  • passMeasures value created, not captured99%
  • passGets the base of every number right17%
  • passBreaks it into inputs teams can move100%
  • passSays how it could be gamed100%
Run
Run
#1
API response time
52 s
Submitted
5 Oct 2026

Usable with a quick edit9 of 14 checks passed

The memo is comprehensive and well-reasoned; the only error is a minor misstatement about bonus spend allocation, which can be fixed with a quick edit.

Output

MEMORANDUM

TO: Executive Team FROM: Staff Product Manager DATE: October 24, 2024 SUBJECT: Crewmate North Star Metric Recommendation for Offsite & Board Reporting

---

1. Executive Summary: The False Comfort of Top-Line Growth

In Q3, GMV grew 15.3% to $9.8M. On the surface, the marketplace looks healthy. In reality, that growth masks an accelerating decay in our core business.

The GMV increase was driven almost entirely by four enterprise clients: they expanded shift volume by 62% and raised hourly rates by 11%, increasing their share of GMV from 34% to 46%. Meanwhile, our 2,790 independent venues—the diversified bedrock of our marketplace—are failing: * Independent shift volume dropped by 4,800 fills/month (-11%) despite posts increasing to 58,000. * Independent fill rate plunged from 78.9% to 67.9%, falling below the critical 70% threshold where 6-month retention collapses from 84% to 41%. * Independent venue quarterly churn nearly doubled from 5% to 9%. * Operations spent $93,000 last month ($1.1M annualized) on $15 "rescue bonuses" to prop up same-day fill rates, which carry an unacceptable 13% no-show rate. * Supply spent heavily on $25 sign-up bonuses; 70% of those new sign-ups never worked a single shift.

Each team is currently optimizing for a siloed proxy that actively harms the broader platform. We must align around a single, un-gameable measure of marketplace health.

---

2. Recommended North Star Metric: Successfully Completed Shifts (SCS)

Definition: The count of shifts that are matched, worked, and completed without a dispute or no-show.

$$\text{Successfully Completed Shifts (SCS)} = \text{Shifts Claimed} \times (1 - \text{No-Show / Dispute Rate})$$

#### Why This Metric? 1. Captures Mutual Value Exchange: Venues don't pay for posted shifts or claimed shifts; they pay for labor that shows up. Workers don't benefit from registrations; they benefit from earned wages. SCS represents the exact transaction where value is realized and Crewmate earns its 18% fee. 2. Exposes False Liquidity: Fills that result in no-shows (currently 13% for rescue shifts and 6% baseline) destroy venue trust. Fill rate treats a no-show as a success until a dispute occurs; SCS counts only real completions. 3. Corrects Cross-Functional Misalignment: To grow SCS, Demand cannot post unfillable shifts, Supply cannot recruit ghost workers, and Ops cannot paper over structural matching issues with subsidy-driven, unreliable fills.

---

3. Metric Tree and Input Ownership

``` [North Star: Successfully Completed Shifts (SCS)] | +--------------------------------+--------------------------------+ | | [Fill Volume] [Fulfillment Reliability] | | +-----+-------------------------+ +-------------+-------------+ | | | | [Fillable Shift Demand] [Retained Worker Supply] [Attendance Rate] [Dispute-Free Rate] (Demand Team) (Supply Team) (Operations) (Product/Ops) ```

Each tier-one input has a single owner accountable for driving it:

Input MetricOwnerStrategic Objective
:---:---:---
1. Fillable Shift DemandHead of DemandMaximize shifts posted with $>24\text{h}$ lead time. Shift distribution currently sets supply up to fail: 31% of indie posts are same-day ($<24\text{h}$), converting at an abysmal 58% fill rate.
2. Activated Core WorkersHead of SupplyWorkers completing $\ge 3$ shifts in month one. Workers reaching this threshold have a 66% 3-month retention rate, compared to just 12% for those who do not. Vanity registrations must be abandoned.
3. Match Efficiency & Fill RateHead of ProductMatch speed and algorithmic discovery. Product must improve worker routing to high-churn independent venues without relying on manual operational interventions.
4. Fulfillment ReliabilityHead of OperationsEliminate no-shows and unconfirmed shifts. Currently, 36% of shifts require passive auto-confirmation, and rescue-bonus fills have more than triple the baseline no-show rate (13% vs. 4%).

---

4. Paired Business Measure: Net Contribution Margin per Shift

A North Star can encourage excessive spending to generate volume. To ensure sustainable, profitable growth, SCS must be paired with Net Contribution Margin per Completed Shift.

$$\text{Net Contribution Margin} = (\text{Take Rate} \times \text{Shift Value}) - (\text{Acquisition Subsidies} + \text{Rescue Bonuses} + \text{Dispute Costs})$$

  • Why it matters: In Q3, Ops spent $93,000/month on rescue fills, and Supply spent $197,500 on registration bonuses for workers who never activated. Subsidies directly cannibalized the marketplace margin.
  • The Constraint: Teams cannot buy SCS growth through operational bailouts or uneconomic supply acquisition. Every completed shift must generate positive net platform margin.

---

5. Gaming Vectors and Guardrails

VectorHow It Could Be GamedMandatory Guardrail
:---:---:---
The "Rescue" LoopholeOps artificially inflates SCS by deploying bonuses to rescue low-lead-time shifts at high unit cost.Max 3% Subsidized Shifts: Subsidized shifts capped at 3% of total fills; rescue fills with $>5\%$ no-show rate forfeit bonus payouts.
Enterprise BiasTeams focus exclusively on the 4 enterprise accounts to hit raw SCS volume, abandoning the higher-margin independent base.Independent Floor: Independent venue SCS must account for $\ge 55\%$ of total SCS; independent venue 6-month retention must stay $\ge 80\%$.
Passive ConfirmationsTeams ignore verification because unconfirmed shifts (36%) default to "worked" status after a dispute window.Verification Coverage: $\ge 85\%$ active venue verification in-app before a shift is classified as completed in the primary SCS reporting.

---

6. Re-anchoring the Executive Team: What Current Metrics Become

To establish SCS as our operational standard, the executive team must reframe their existing target metrics from headline drivers to supporting inputs:

#### CEO: GMV $\rightarrow$ The Enterprise Valuation Output * What it becomes: A downstream financial output metric for quarterly investor updates. * Why: GMV inflated by rate hikes and enterprise concentration gave false confidence while 9% of our independent client base vanished. GMV will grow healthily only if SCS grows sustainably.

#### Head of Demand: Shifts Posted $\rightarrow$ Liquidity-Adjusted Posts * What it becomes: Advance-notice shifts posted ($>24\text{h}$ lead time) with market-clearing rates. * Why: Unfilled posts drive churn. Posting low-lead-time, below-market shifts destroys venue trust when fill rates sit at 58%. Demand's target is shifts that can realistically clear.

#### Head of Supply: Active Workers $\rightarrow$ Activated Core Workers ($3+$ shifts/month) * What it becomes: Cohort activation rate within 30 days of onboarding. * Why: Raw registrations jumped to 7,900 via a $25 bonus, but only 30% ever worked. Focusing on workers reaching 3+ shifts increases 90-day retention by $5.5\times$ (66% vs. 12%), creating dependable supply.

#### COO: Gross Fill Rate $\rightarrow$ Net Verified Completion Rate * What it becomes: $\text{Completed Shifts} \div \text{Posted Shifts}$ net of cancellations, no-shows, and rescue subsidies. * Why: Buying fill rate with a 13% no-show rate and a $93,000 monthly subsidy masks operational deficits. Operations must optimize for clean, reliable shifts, not emergency rescue band-aids.

---

7. Recommendation for the Offsite

At next week's offsite, we should formally adopt Successfully Completed Shifts (SCS) as our sole operational North Star, paired with Net Contribution Margin per Shift.

Adopting SCS unifies product, supply, demand, and operations behind marketplace quality, directly targets our independent churn crisis, and presents the Board with a repeatable growth engine rather than masked churn.

Claims the judge couldn’t find in the brief

  • Supply spent $197,500 on registration bonuses for workers who never activated.

Grades and run details

Decision model 71 · LLM judge 73
Decision model checks
  • failUses the supplied evidence correctly64%
  • passAddresses the actual decision60%
  • partialRespects explicit constraints26%
  • partialIdentifies material uncertainty43%
  • failAvoids unsupported claims29%
  • passProduces the required deliverable53%
  • passSees the growth is narrow100%
  • passChecks the North Star can be measured98%
  • passSees the fill rate being bought100%
  • passOne North Star for both sides100%
  • passMeasures value created, not captured99%
  • failGets the base of every number right39%
  • passBreaks it into inputs teams can move98%
  • passSays how it could be gamed100%
Run
Run
#1
API response time
29 s
Submitted
5 Oct 2026

Usable with a quick edit13 of 14 checks passed

The memo is thorough, covers all required elements, and could be used by a PM with only minor wording edits, without re-checking evidence or redoing reasoning.

Output

Memo: Choosing Crewmate's North Star

To: Exec team | From: Staff PM | For: Offsite decision; CEO reports to the board three weeks later

Recommendation

North Star: Completed shifts, meaning shifts filled by a vetted worker and worked (no no-show, no dispute), per month.

Paired business measure: Net revenue, which is our 18% take minus worker incentives (rescue and sign-up bonuses). GMV stays in the board deck as context, shown at constant rates and split by segment.

Why not the four numbers on the table

Each can rise while the business gets worse. Q2 to Q3, monthly averages:

  • Shifts posted rose 19% (77k to 92k), but filled shifts rose only 11%. Independents posted 4% more and filled 11% fewer (44.2k to 39.4k). Fill fell from 79% to 68%.
  • GMV rose 15% ($8.5M to $9.8M), but all the growth is chains. Chain GMV went from about $2.9M to $4.5M (+56%) and independent GMV from about $5.6M to $5.3M (−6%). The chains' 11% rate rise from August also inflates it. That is price, not service.
  • Active workers is the most gameable. Registrations rose 30% (6,100 to 7,900) after the $25 bonus, but only 2,370 of the Q3 registrants (30%) have worked a shift.
  • Fill rate is now partly bought. Chains hold at 93% and independents have dropped to 68%. Ops spends $93k a month on $15 rescue bonuses, and those fills no-show at 13% against 4% elsewhere. That is not a criticism of Ops. The fill rate simply can't tell a durable fill from a purchased one.

Completed shifts needs demand, supply and reliability to all work, and price can't move it. It also tracks what keeps venues: those filling 90% or more retain at 84% after six months, against 41% below 70%. The average independent is now at 68%, and 9% of independents stopped posting in Q3 (5% in Q2).

Caveats you should know: 1. The headline will look healthy. I estimate completed shifts rose about 11%, from ~60k to ~67k a month, applying the 6% failure rate to filled shifts. Chains are carrying that, so the independent segment must be reported next to it (see guardrails). 2. The baseline is provisional. Venues confirm only 64% of filled shifts. The 6% failure rate comes from that confirmed subset, and the other 36% are assumed worked. Our first board number should be labeled an estimate, with confirmation coverage disclosed.

Metric tree

Completed shifts = Posted shifts × Fill rate × Show-up rate

InputOwnerLeading drivers (same owner)
Posted shifts, by segmentHead of DemandRetained independent venues (churn 9% in Q3); posts per venue
Fill rate, reported with and without rescue fillsCOOSame-day fill (58% vs. far higher for planned shifts); time to fill; rescue spend per completed shift
Show-up rate (filled shifts not no-show or disputed)Head of SupplyWorker reliability; no-show rate of rescue fills (13%)
Worker capacity: activated workers (3+ shifts in first month)Head of SupplyActivation of new registrants; 3-month retention (66% for activated workers vs. 12% for the rest)
Confirmation coverage (share of filled shifts with a verified outcome)Product (me), with OpsIn-app confirmation rate; worker clock-in
Net revenue (paired measure)CEO/FinanceTake rate; incentive spend as % of take

Supply drives fill, so the Head of Supply's activation target is a stated input to the COO's fill target. Owners agree those two numbers jointly.

How it could be gamed, and the guardrails

Gaming riskGuardrail
Unverified "completions." 36% of fills are assumed worked, so sloppy confirmation inflates the count.Target 90% confirmation coverage within two quarters. Add worker clock-in as a second signal. Audit a monthly sample. The board number states its coverage.
Buying fills. Rescue bonuses lift fill but 13% no-show. Rescue costs $15 against average take of about $25 a shift.Count only completed shifts. Pay rescue bonuses after the shift is confirmed worked, not at pickup. Cap incentive spend as a % of take (now ~5% from rescue alone). Net revenue exposes the cost.
Chains masking independents. Headline rises while independents decay.Report independent completed shifts, fill rate and 90-day venue churn beside the headline. Trigger a review if independent churn exceeds 5% (the Q2 level) or independent fill is below 70%.
Shift-splitting. Posting more, shorter shifts inflates the count.Report hours worked and average shift length next to the count. If they diverge, move to hours.
Registration farming. Chasing sign-up bonuses.Registrations aren't in the tree. Pay the sign-up bonus on the third completed shift, not at registration.
Easy-shift cherry-picking. Ops or Supply prioritizes easy fills and lets same-day shifts lapse.Track fill by segment and by same-day vs. planned. Same-day is 31% of independents' posts, so neglecting it shows up quickly.
Phantom demand. Venues post shifts and cancel late to flatter posted counts.Track late cancellations (<24h) as a Demand guardrail.

What each team's current metric becomes

TodayBecomes
CEO: GMVBoard context, reported at constant rates and split chain/independent. The paired measure is net revenue. GMV is a mix of volume and price, so it can't be the North Star.
Demand: shifts postedInput to the tree, with churn and late-cancellation guardrails. The goal is posts that get worked, not posts.
Supply: active workersReplaced by activated workers (3+ shifts in first month) and 3-month retention. Registrations become a diagnostic only.
COO: fill rateInput to the tree, always shown with and without rescue fills, plus incentive cost per completed shift.

What I'm asking the exec team to decide at the offsite

  1. Adopt completed shifts as the North Star, paired with net revenue.
  2. Agree the definition: filled, not no-show, not disputed. The board gets the estimated baseline (~60k in Q2, ~67k in Q3) with confirmation coverage disclosed.
  3. Confirm the owners above.
  4. Change incentive payouts. Rescue bonuses are paid on completion, and the sign-up bonus on the third shift. I'd keep rescue itself, because fill matters for retention. We just shouldn't count it as organic.
  5. Fund confirmation coverage, in-app confirmation prompts and worker clock-in, as the first Product deliverable. Until it lands, the North Star carries a known error band.

What I'm not claiming

I haven't shown that rescue or the sign-up bonus is net-negative. The data doesn't isolate their effects, and rescue covers only about one month of Q3. The point is that neither should be able to move our headline number without proving it produced a completed shift. Over the next quarter, Product and Finance should model the retention value of rescue fills against their cost and no-show rate.

Grades and run details

Decision model 96 · LLM judge 93
Decision model checks
  • passUses the supplied evidence correctly12%
  • passAddresses the actual decision86%
  • passRespects explicit constraints38%
  • passIdentifies material uncertainty97%
  • passAvoids unsupported claims32%
  • passProduces the required deliverable88%
  • passSees the growth is narrow100%
  • passChecks the North Star can be measured100%
  • passSees the fill rate being bought100%
  • passOne North Star for both sides95%
  • passMeasures value created, not captured97%
  • partialGets the base of every number right6%
  • passBreaks it into inputs teams can move100%
  • passSays how it could be gamed100%
Run
Run
#1
API response time
68 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Uses the supplied evidence correctlyRightWrongRight
GPT-6.1 Sol · API

All claims about the current situation are directly from the supplied context or correct arithmetic.

Gemini 3.8 Flash · API

The claim that Supply spent $197,500 on registration bonuses for workers who never activated is not supported; the total spend was $197,500, but some of those workers did activate.

Sonnet 5.5 · API

All factual claims about the current situation are taken directly from the supplied context or derived by correct arithmetic.

Identifies material uncertaintyMixedWrongRight
GPT-6.1 Sol · API

It identifies the confirmation gap as an unknown but does not say how resolving it could change the North Star decision.

Gemini 3.8 Flash · API

The output does not identify any unknowns that could change the decision or how they would be resolved.

Sonnet 5.5 · API

Identifies the confirmation gap and unmodeled incentive effects as material unknowns, and says how to resolve them (fund confirmation coverage, model retention value).

Avoids unsupported claimsRightWrongRight
GPT-6.1 Sol · API

Interpretations like 'masked' are clearly based on the data and not presented as unsupported facts.

Gemini 3.8 Flash · API

The output presents the $197,500 as spent on non-activated workers as fact, which is not established by the evidence.

Sonnet 5.5 · API

Estimates and assumptions are clearly labeled; no interpretations are presented as established fact.

Gets the base of every number rightRightMixedRight
GPT-6.1 Sol · API

All derived percentages and rates are computed correctly with appropriate denominators.

Gemini 3.8 Flash · API

All derived percentages and rates are correctly computed from the supplied data.

Sonnet 5.5 · API

All derived figures (chain/independent GMV, fill rates, growth rates) use correct denominators and arithmetic; the completed-shifts estimate is caveated.

All mixed 1

Addresses the actual decisionMixedMixedMixed
GPT-6.1 Sol · API

The output commits to a North Star but does not state what result or condition would change that recommendation.

Gemini 3.8 Flash · API

The output does not state what result or condition would change the recommendation.

Sonnet 5.5 · API

The output commits to a clear North Star recommendation but does not state what result or condition would change that answer.

All got right 9

Respects explicit constraintsRightRightRight
GPT-6.1 Sol · API

The memo is within 1,200 words, addresses the exec team, and includes all requested elements.

Gemini 3.8 Flash · API

The memo is within 1,200 words, in memo form, and addresses all requested elements.

Sonnet 5.5 · API

The memo is within 1,200 words, includes all required elements (North Star, metric tree, business measure, gaming/guardrails, team metric changes), and is addressed to the exec team.

Produces the required deliverableRightRightRight
GPT-6.1 Sol · API

The memo is complete, in the right form, and the exec team could act on it with light edits.

Gemini 3.8 Flash · API

The memo includes the North Star, metric tree with owners, business measure, gaming and guardrails, and redefined team metrics, and is usable.

Sonnet 5.5 · API

The memo is complete, in the right form for the exec team, and usable as-is with light edits.

Sees the growth is narrowRightRightRight
GPT-6.1 Sol · API

It shows the split (chains up, independents down) and mandates segment guardrails so the decline cannot hide.

Gemini 3.8 Flash · API

The memo clearly shows the split between enterprise growth and independent decline, and includes an independent floor guardrail.

Sonnet 5.5 · API

Shows that growth came from chains while independents' filled shifts and fill rate fell and churn rose, and sets guardrails to track independent completed shifts and churn separately.

Checks the North Star can be measuredRightRightRight
GPT-6.1 Sol · API

It flags the 64% confirmation gap and proposes an audit and estimation approach until coverage is adequate.

Gemini 3.8 Flash · API

The memo flags the 36% unconfirmed shifts and proposes a verification coverage guardrail to ensure measurability.

Sonnet 5.5 · API

Flags the 64% confirmation gap and proposes fixes (worker clock-in, 90% coverage target, board disclosure) before the North Star goes to the board.

Sees the fill rate being boughtRightRightRight
GPT-6.1 Sol · API

It uses the 13% no-show rate and $93k cost to show inflation and sets guardrails like cost per completed shift.

Gemini 3.8 Flash · API

The memo uses the 13% no-show rate and cost to highlight the issue and sets a guardrail capping subsidized shifts and no-show rate.

Sonnet 5.5 · API

Uses the 13% no-show rate vs 4% and the $93k cost to show fill is being bought, and sets guardrails like paying on completion and capping incentive spend.

One North Star for both sidesRightRightRight
GPT-6.1 Sol · API

The North Star (reliably completed shifts) only rises when both venue and worker get value, and the memo explains why.

Gemini 3.8 Flash · API

SCS captures value for both venues and workers, as it requires a completed shift without dispute.

Sonnet 5.5 · API

Completed shifts require both a venue posting and a worker working, so the metric only rises when both sides get value.

Measures value created, not capturedRightRightRight
GPT-6.1 Sol · API

The North Star measures value delivered (completed shifts), with revenue kept as a separate business measure.

Gemini 3.8 Flash · API

SCS measures value delivered (completed shifts), while Net Contribution Margin is the separate business measure.

Sonnet 5.5 · API

The North Star measures completed shifts (value delivered to customers), and net revenue is kept as a separate business measure.

Breaks it into inputs teams can moveRightRightRight
GPT-6.1 Sol · API

It breaks the North Star into three input metrics with owners and pairs it with contribution profit.

Gemini 3.8 Flash · API

The metric tree breaks SCS into fill volume and fulfillment reliability, with owners for each input, and pairs with Net Contribution Margin.

Sonnet 5.5 · API

Breaks the North Star into posted shifts, fill rate, show-up rate, and worker capacity, each with an owner, and pairs it with net revenue.

Says how it could be gamedRightRightRight
GPT-6.1 Sol · API

It lists realistic gaming scenarios (bonus buying, confirmation inflation, shift splitting) and corresponding guardrails.

Gemini 3.8 Flash · API

The memo identifies gaming vectors like rescue loophole, enterprise bias, and passive confirmations, with specific guardrails.

Sonnet 5.5 · API

Lists realistic gaming risks (unverified completions, buying fills, shift-splitting, registration farming, cherry-picking, phantom demand) and specific guardrails.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5withAPI95.996.72None
2GPT-6.1 SolwithAPI97.793.32None
3GPT-6 LunawithAPI95.588.32None
4GPT-6 AstrawithChatGPT94.288.32None
5Opus 5.5withClaude94.680.82None
6Haiku 5.5withAPI88.881.72None
7Gemini 3.8 FlashwithAPI78.965.82None
8Gemini 3.5 Flash-LitewithGemini54.438.321 capped

What you get for the cost

GPT-6 Luna · API scores within 4 points of Sonnet 5.5 · API at about a twentieth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreSonnet 5.5 · API: task score 96, 22× the cheapestGPT-6.1 Sol · API: task score 96, 9.6× the cheapestGPT-6 Luna · API: task score 92, the cheapestGPT-6 Astra · ChatGPT: task score 91, 52× the cheapestOpus 5.5 · Claude: task score 86, 36× the cheapestHaiku 5.5 · API: task score 86, the cheapestGemini 3.8 Flash · API: task score 71, 7.5× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 44, 1.8× the cheapest, 1 critical failureSonnet 5.5 · APIGPT-6.1 Sol · APIGPT-6 Luna · APIGPT-6 Astra · ChatGPTOpus 5.5 · ClaudeHaiku 5.5 · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costAt least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API921.0×
Haiku 5.5 · API861.1×
Gemini 3.5 Flash-Lite · Gemini441.8×
Gemini 3.8 Flash · API717.5×
GPT-6.1 Sol · API969.6×
Sonnet 5.5 · API9622×
Opus 5.5 · Claude8636×
GPT-6 Astra · ChatGPT9152×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Choosing the metric the whole company steers by.

Why it matters

Pick revenue and you steer by the rear-view mirror. Pick a vanity metric and teams will happily grow it while the product gets worse.

What good looks like

  • Measures value delivered to customers, not value captured
  • Grounded in what predicts retention in the data
  • Broken into input metrics a team can own
  • Says how it could be gamed, with guardrails

Deliberately not measured

  • Dashboard design
  • Naming
Capability tested

Metric design

The failure we’re looking for

Revenue, or a vanity metric, as the North Star

Grading

Decision model and LLM judge, calibrated against a blind PM review