Tasks / Metrics & Experimentation

Pick a North Star metric

Can the model pick the one metric that measures the value a product creates, and break it into inputs teams can move?

Task type v1.1 · 2 tasksLast changed 5 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 72% were usable with at most a quick edit.

Reliably right

  1. Says how it could be gamed100% pass
    Identifies gaming vectors (tapping through, notification overuse, short-term spikes, revenue manipulation, MAU drift) and sets specific guardrails (quality completion, push caps, blocking metrics, cohort retention checks, subscriber churn tracking).
    Sonnet 5.5 · API · Revenue, actives, or something else?
  2. Sees the growth is narrow100% pass
    It shows the split (chains up, independents' filled shifts and fill rate down, more leaving) and tracks independent venues via segment reporting and guardrails.
    GPT-6 Luna · API · Growing, on paper
  3. Checks the North Star can be measured100% pass
    It flags the 64% confirmation gap and proposes instrumenting check-in/out, reconciliation, and auditing to fix it before the board readout.
    GPT-6 Luna · API · Growing, on paper

Where it slips

  1. Addresses the actual decision36% pass
    The output commits to a North Star but does not state what result or condition would change that recommendation.
    GPT-6 Luna · API · Growing, on paper
  2. Identifies material uncertainty38% pass
    It identifies the confirmation gap as an unknown but does not say how resolving it could change the North Star decision.
    GPT-6.1 Sol · API · Growing, on paper
  3. Avoids unsupported claims63% pass
    It presents 'easy to tap through' and 'we don't track today' as facts without support, and does not label them as assumptions.
    Opus 5.5 · Claude · Revenue, actives, or something else?

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Measures value created, not captured

    Does the chosen North Star measure the value customers get from the product, rather than revenue or a count of people who merely showed up?

    Passes when Picks a metric of value delivered to customers (a core action that predicts retention), and keeps revenue as a separate business measure.

  2. Breaks it into inputs teams can move

    Does the output break the North Star into a few input metrics, each one a team could own and move week to week, and pair it with a separate business measure of value captured?

    Passes when Gives a small tree of input metrics, each with an owner and a link to the North Star, plus the revenue or business measure it's paired with.

  3. Says how it could be gamed

    Does the output pressure-test the North Star for how teams could raise it without delivering more value, and set guardrail metrics against that?

    Passes when Names a realistic way the metric could be gamed in this product and the guardrails that would catch it.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 4 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a Staff PM at Crewmate. Every team has been steering by its own number, and the exec team wants to pick one North Star at next week's offsite, so the CEO can report it to the board three weeks later. Write the memo they'll read beforehand, in no more than 1,200 words: the North Star you recommend and why, its metric tree with an owner for each input, the business measure it's paired with, how it could be gamed and the guardrails against that, and what each team's current metric becomes. The numbers below are as the data team sent them. Not all of it matters equally.

What the model was given9 items: About Crewmate, What each exec wants, Shifts by quarter (monthly averages), GMV (monthly average), Venues, Workers, Same-day shifts, Completion data, From the COO
About CrewmateA marketplace for hospitality shifts: venues post shifts, and vetted workers pick them up. Crewmate takes 18% of each shift's value. Two kinds of venue: four enterprise hotel and restaurant chains (310 venues between them, on annual contracts) and 2,790 independent venues (bars, restaurants, cafes) who pay as they go.
What each exec wantsCEO: GMV, 'because it's what the board looks at'. Head of Demand: shifts posted. Head of Supply: active workers. COO: fill rate.
Shifts by quarter (monthly averages)Enterprise chains: Q2 posted 21,000, filled 19,500. Q3 posted 34,000, filled 31,600. Independent venues: Q2 posted 56,000, filled 44,200. Q3 posted 58,000, filled 39,400.
GMV (monthly average)Q2 $8.5M, Q3 $9.8M. Enterprise chains were 34% of GMV in Q2 and 46% in Q3. The chains raised hourly rates by 11% for the holiday season from August.
VenuesVenues whose shifts fill 90% of the time or more: 84% still posting six months later. Under 70%: 41%. In Q3, 9% of independent venues stopped posting altogether (5% in Q2). No chain venue stopped.
WorkersRegistrations: 6,100 in Q2, 7,900 in Q3, after Supply introduced a $25 sign-up bonus in July. 2,370 of the Q3 registrants have worked a shift. Workers who work three or more shifts in their first month: 66% still working after three months; everyone else: 12%.
Same-day shiftsShifts posted less than 24 hours before they start fill 58% of the time. They are 31% of independents' posts and 8% of the chains'.
Completion dataVenues confirm in the app that a shift was worked for 64% of filled shifts; the rest are assumed worked unless the venue raises a dispute. Of confirmed shifts, 6% were no-shows or disputed.
From the COO"Since September, Ops pays workers a $15 rescue bonus to pick up same-day shifts that would otherwise go unfilled. 6,200 rescue fills last month, $93,000. Rescue-filled shifts have a 13% no-show rate, against 4% for the rest. It's kept our fill rate from falling further."
What a strong answer doesThe answer key the graders mark against

Works out from the raw numbers what the pack doesn't say: all of the growth is the four chains (filled shifts up from 19,500 to 31,600 a month, plus an 11% rate rise), while independent venues filled fewer shifts (44,200 to 39,400, so their fill rate fell from 79% to 68%) and are leaving faster (9% stopped posting, against 5%); GMV rose mostly on the chains' volume and prices. Picks a North Star of value delivered to both sides, such as shifts worked, but designs it so the decline can't hide: measured for independents separately, or as reliably staffed venues, or with an explicit independent-venue guardrail. Notices that 'worked' is only confirmed for 64% of shifts, so the North Star can't be trusted yet, and makes fixing confirmation an early input or uses a measurable proxy until then. Treats the rescue bonus as a way the fill rate (and any fills-based North Star) is being bought: 13% no-shows against 4%, so it needs guardrails such as cost per worked shift and no-show rate. Shows how shifts posted, registrations (2,370 of 7,900 worked) and GMV each rose without more value. Builds a metric tree with owners, pairs it with revenue or GMV as the business measure, and redefines each team's metric as an input.

Critical failures (cap the score at 40)
  • Recommends GMV or shifts posted as the North Star
Task

Staff-level · v1.1

The outputs

9 setups answered this brief.

Usable with a quick edit9 of 14 checks passed

The memo is evidence-based and usable with very light edits to label causal interpretations and add a condition that could change the recommendation.

Output

MEMO

To: Executive Team From: Staff PM Date: October 24 Subject: Pre-Read: Selecting Crewmate’s North Star Metric

The Diagnosis: Why Our Current Metrics Are Failing Us

On the surface, Q3 looks like a triumph: monthly GMV grew 15% to $9.8M, posted shifts hit 92,000, and worker registrations jumped 30%.

Beneath the surface, our core marketplace is cannibalizing itself because every team is optimizing for its own siloed number:

  1. GMV is masking an independent venue exodus. Enterprise chains grew from 34% to 46% of GMV in Q3, buoyed by a massive spike in posts and an 11% seasonal wage hike. This pulled worker supply away from Independent venues. Independent fill rates plunged from 78.9% to 67.9%. Waitstaff didn't disappear; they went to higher-paying, earlier-posted Enterprise shifts. Because venues with a <70% fill rate only retain at 41% (versus 84% for >90% fill rate), independent venue churn nearly doubled from 5% to 9% in Q3. We are trading our diversified, high-margin core for concentration risk in just four Enterprise clients.
  2. Demand is chasing volume over fillability. Demand hit 92,000 posted shifts, but 31% of Independent shifts are posted with less than 24 hours' notice (filling at just 58%, compared to 8% same-day posts for Enterprise). Counting unfillable, last-minute shifts as "Demand success" sets Ops up to fail.
  3. Supply is buying hollow registrations. Driven by a $25 sign-up bonus, Supply registered 7,900 workers in Q3, but 70% of them never worked a single shift. We are paying for top-of-funnel vanity while starving the marketplace of actual labor.
  4. Ops is subsidizing "fake" fills. To defend Fill Rate against same-day posts and low active supply, Ops spent $93,000 last month ($15/shift) on 6,200 "rescue fills." However, rescue shifts have a 13% no-show rate (vs. 4% normally). Worse, without those 6,200 subsidized fills, our Q3 fill rate would have been 70.4%, not 77.2%.

Most importantly: a "filled" shift is not a worked shift. Across confirmed shifts, 6% result in no-shows or disputes. A venue with a no-show has an empty floor; they don’t care that our dashboard marked the shift "filled."

---

The Recommendation: Monthly Completed Shifts (MCS)

I recommend our North Star Metric be Monthly Completed Shifts (MCS)—defined as shifts that are posted, claimed, and actually worked without a no-show or venue dispute.

Why MCS? A North Star must capture the exact moment we deliver value to both sides of the marketplace and earn our 18% take rate. * Why not GMV? GMV is an output, not an input. It is artificially inflated by wage inflation (Enterprise's 11% bump) and shift duration, masking underlying marketplace decay. * Why not Fill Rate? Fill rate is a ratio. You can achieve a 100% fill rate by firing all Independent venues and shrinking the business. Furthermore, "filled" ignores our 6% no-show problem. * Why not Shifts Posted or Active Workers? Each represents only half the marketplace. Unfilled posts cause venue churn; idle workers cause supply churn.

MCS forces alignment: Demand must source fillable shifts, Supply must activate reliable workers, and Ops must ensure completion, not just claims.

(Note on instrumentation: Currently, 36% of shifts are "assumed worked" unless disputed. Product will immediately build frictionless check-in/out to close this blind spot, though the silent-dispute baseline is accurate enough to begin tracking MCS today).

---

The Metric Tree & Ownership

$$\text{Monthly Completed Shifts} = \text{Posted Shifts} \times \text{Fill Rate} \times \text{Completion Rate}$$

Using Q3 baselines (92,000 posted $\times$ 77.2% fill $\times$ ~94% completion), we are currently at ~66,750 MCS. Here is how we break the tree down by owner:

1. Head of Demand: Quality Posted Shifts * Inputs: Active Venues $\times$ Shifts/Venue $\times$ % Posted >24hrs in Advance. * Focus: Stop Incentivizing raw post volume. Work with Independents to post schedules earlier (reducing their 31% same-day post rate) and price shifts competitively against Enterprise.

2. Head of Supply: Activated & Retained Workers * Inputs: New Worker Activation (% hitting $\ge$3 shifts in Month 1) + Retained Active Workers. * Focus: Our data shows workers who complete $\ge$3 shifts in their first month retain at 66% at month three, vs. 12% for everyone else. Supply must shift the $25 bonus from registration to completing the 3rd shift.

3. COO (Ops) & Product: Match Efficiency & Reliability * Inputs: Fill Rate (segmented by Enterprise vs. Independent) $\times$ Completion Rate ($1 - \text{No-Show/Dispute Rate}$). * Focus: Replace blanket $15 rescue bonuses with algorithmic reliability scoring (penalizing no-shows, rewarding streaks) and better matching to keep Independent venues above the critical 90% fill threshold.

---

The Paired Business Measure: Net Contribution Margin (or Net Revenue)

To satisfy the Board and prevent us from buying growth unprofitably, MCS must be paired with Net Contribution Margin (Gross Revenue minus direct variable incentives like the $15 rescue and $25 sign-up bonuses).

In Q3, we averaged $9.8M in GMV, yielding ~$1.76M in gross revenue (at 18%). However, accounting for the ~6% no-show rate (~$105k in uncollected fees) and ~$160k/month in worker bonuses ($93k rescue + ~$66k sign-up), we burned over 15% of our gross revenue on flawed incentives. Pairing MCS with Net Contribution Margin proves to the board that operational discipline drives durable GMV.

---

Gaming the Metric & Our Guardrails

Every metric can be gamed. Here is how teams might game MCS, and the guardrails we will lock in to prevent it:

  • Gaming Risk 1: Over-indexing on Enterprise. Because Enterprise shifts fill at 93% with low same-day rates, teams could abandon Independents entirely to juice MCS.
  • Guardrail: % of Independent Venues with $\ge$80% Completion Rate. We must protect the long-tail marketplace. If Independent health drops, we fail.
  • Gaming Risk 2: Splitting shifts. Demand could ask venues to turn one 8-hour shift into two 4-hour shifts to double MCS.
  • Guardrail: Average Hours (or GMV) per Shift. Must remain stable within historical bands.
  • Gaming Risk 3: Buying completions. Ops and Supply could ramp up $15 and $25 bonuses to force completions at a financial loss.
  • Guardrail: Incentive Spend as a % of Gross Revenue (capped via our paired business metric).

---

Summary: What Your Current Metric Becomes

For the Board and our weekly operating reviews, here is how our scorecard evolves:

ExecutiveOld MetricWhat It Becomes Under MCS
:---:---:---
CEOGMVBoard Financial Output. Reported alongside MCS and Net Contribution Margin as the financial byproduct of completed shifts.
DemandShifts PostedLead-Time Qualified Posts. Evaluated on growing shifts posted >24 hours in advance, segmented by Independent vs. Enterprise.
SupplyActive WorkersActivated Workers ($\ge$3 Shifts). Evaluated on new workers reaching the 3-shift retention tipping point in Month 1, and total retained cohort size.
COOFill RateCompletion Rate & Venue Reliability. Evaluated on minimizing the 6% no-show gap and maximizing the % of venues hitting the 90% completion retention threshold.

Grades and run details

Decision model 71 · LLM judge 80
Decision model checks
  • failUses the supplied evidence correctly46%
  • partialAddresses the actual decision79%
  • passRespects explicit constraints41%
  • partialIdentifies material uncertainty77%
  • failAvoids unsupported claims41%
  • passProduces the required deliverable83%
  • passSees the growth is narrow98%
  • passChecks the North Star can be measured98%
  • passSees the fill rate being bought100%
  • passOne North Star for both sides100%
  • passMeasures value created, not captured98%
  • failGets the base of every number right46%
  • passBreaks it into inputs teams can move100%
  • passSays how it could be gamed100%
Run
Run
#1
API response time
20 s
Submitted
13 Oct 2026

Usable with a quick edit9 of 14 checks passed

The core reasoning is solid; adding a sentence on two-sided marketplace rationale and conditions that would change the recommendation would be quick edits, and no evidence needs rechecking.

Output

To: Exec team From: Staff PM Re: Proposed North Star for the offsite

Recommendation

North Star: completed shifts per month, meaning shifts that were filled and worked, with no no-show and no dispute.

Paired business measure: net take revenue, meaning our 18% take minus all supply incentives (rescue bonuses, sign-up bonuses, no-show credits). Q3 gross take was about $1.76M a month before incentives.

GMV stays in the board pack, shown with a price/volume split, as context rather than as the North Star.

Why not GMV

Q3 looks strong on GMV (+15%, $8.5M to $9.8M), but the underlying picture is weaker:

  • Chains drove the growth, and price is part of it. Chain GMV rose from about $2.9M to $4.5M, helped by an 11% holiday rate increase. GMV can't tell price from volume.
  • Independents shrank. Their GMV fell about 6%, and their filled shifts fell 11% (44,200 to 39,400).
  • The blended fill rate hid the problem. It fell from 82.7% to 77.2%, even though chains (steady at 93% fill) grew their share of posts. Independent fill fell from 79% to 68%.

Completed shifts measure delivered volume, which is price-neutral. They exclude no-shows, and each one is worth roughly $25 of take (18% of the Q3 average shift value of about $138). They also tie directly to the revenue we keep.

Fill rate would have caught the independent decline, but it is a ratio that averages two very different segments. Filled shifts overcount, since they include shifts that never happen. Registrations and active workers are inflated by bonuses.

Metric tree

Completed shifts = shifts posted × fill rate × completion rate

  • Shifts posted = active venues × posts per active venue. Owner: Head of Demand.
  • Active venues: venues that posted in the last 30 days. Watch 90-day retention: 84% of venues with fill ≥90% are still posting after six months, versus 41% of venues under 70%. Independent churn was 9% in Q3, up from 5%.
  • Posts per active venue: includes same-day share. Same-day shifts are 31% of independent posts and fill only 58% of the time.
  • Fill rate = filled ÷ posted. Owner: Head of Supply.
  • Active workers: workers with three or more shifts in the trailing 30 days.
  • Activation rate of new sign-ups: share who work three or more shifts in their first month. Only 2,370 of 7,900 Q3 registrants (30%) have worked at all.
  • Completion rate = worked ÷ filled. Owner: COO.
  • Verification coverage: share of filled shifts confirmed by the venue. Today it's 64%.
  • No-show and dispute rate: 6% of confirmed shifts. Rescue-filled shifts run at 13%, against 4% for the rest.
  • Net take (the paired measure) = completed shifts × average shift value × 18%, minus incentives. Owner: Finance lead, to be named. The brief names no CFO.

Paired business measure: net take

Net take is the number that tells us whether incentives pay for themselves. The rescue program is the clearest case. At last month's 6,200 rescue fills, $93,000 in bonuses against roughly $154,000 of take on those shifts (assuming the Q3 average shift value) means the bonus absorbs about 60% of take before counting the extra no-shows. That is a reasonable trade if it saves fills that would otherwise be lost, but we need the net figure to know.

How it could be gamed, and guardrails

  1. Fill rate bought with bonuses. Report rescue fills separately, with their cost. Cap incentive spend at 50% of take on each completed shift. Rescue is at about 60% today, so it breaches the cap.
  2. Completed shifts inflated by auto-confirmation. The 36% of fills nobody confirms are currently assumed worked. Count a shift only once it is confirmed or its dispute window closes undisputed. Target verification coverage of 90% or more.
  3. Sign-up bonuses inflating supply. The North Star doesn't use registrations. The guardrail is the activation rate and three-month retention of each sign-up cohort. Today it is 66% for workers who hit three shifts in month one, against 12% for everyone else.
  4. Posting inflated with unfillable shifts. Track fill rate by segment, and 24-hour same-day fill separately. Watch venue retention.
  5. GMV growth from price. Show GMV per completed shift alongside total GMV, so price increases are visible.
  6. Concentration in chains. Independent active venues must not fall quarter over quarter, and independent churn stays at or below 5% a quarter (the Q2 level). Chains are now 46% of GMV, so this is the risk to watch.
  7. No-show rate. Keep it at or below 5% of verified shifts overall.

What each team's metric becomes

  • CEO: GMV becomes net take revenue as the paired business measure. GMV stays in the board pack, with a price/volume split.
  • Head of Demand: shifts posted becomes active venues and completed shifts from venues we retain. A post counts only as an input, not as success.
  • Head of Supply: active workers becomes three-month retained workers, by sign-up cohort. Fill rate is reported with rescue fills shown separately.
  • COO: fill rate becomes completion rate and verification coverage, plus cost per completed shift. Fill rate stays as an input.

Data gaps to close before the board

  • Baseline. Because 36% of fills are unverified, we cannot state the exact Q3 completed-shift count. Filled shifts (71,000 a month) is an upper bound.
  • Enterprise GMV per filled shift fell from about $148 to $143 despite the 11% rate rise. I can't explain this from the data. The rate rise also applies only from August, so Q3 contains only part of it. We need shift-mix data.
  • Same-day share in Q2 for independents isn't provided, so we can't say how much of their fill decline is mix rather than demand.
  • Rescue economics use the Q3 average shift value as a stand-in. We need actual rescue shift values, and the sign-up bonus cost, which the brief doesn't give.

Decisions for the offsite

  1. Adopt completed shifts as the North Star, with the definition above.
  2. Adopt net take as the paired measure.
  3. Approve the guardrail thresholds, including the 50% incentive cap.
  4. Decide the rescue program: keep as is, cap it, or redesign it. My recommendation is to keep it only for shifts where the net take stays positive after the bonus.
  5. Name the owner of net take.

Grades and run details

Decision model 75 · LLM judge 80
Decision model checks
  • failUses the supplied evidence correctly7%
  • partialAddresses the actual decision69%
  • passRespects explicit constraints34%
  • partialIdentifies material uncertainty82%
  • partialAvoids unsupported claims23%
  • passProduces the required deliverable40%
  • passSees the growth is narrow98%
  • passChecks the North Star can be measured100%
  • passSees the fill rate being bought100%
  • passOne North Star for both sides73%
  • passMeasures value created, not captured86%
  • failGets the base of every number right4%
  • passBreaks it into inputs teams can move100%
  • passSays how it could be gamed100%
Run
Run
#1
API response time
48 s
Submitted
8 Oct 2026

Usable with a quick edit12 of 14 checks passed

The memo is complete and actionable; a PM could add a sentence about what would change the North Star choice as a quick edit, without rechecking evidence or redoing reasoning.

Output

Decision memo: Crewmate’s North Star

To: Executive team From: Staff PM Decision for next week’s offsite: Adopt successful shifts worked per month as our North Star, paired with marketplace contribution dollars. Publish both with mandatory enterprise/independent breakdowns.

Why this, not today’s competing metrics?

Crewmate creates value when a venue gets the staffing it requested and a worker completes paid work—not when someone posts, registers, or accepts a shift.

Our headline growth currently masks deteriorating service:

Monthly averageQ2Q3
------:---:
Total filled shifts63,70071,000
Overall fill rate82.7%77.2%
Enterprise fill rate92.9%92.9%
Independent fill rate78.9%67.9%

GMV rose 15.3%, but independent GMV fell approximately 5.7%, from $5.61M to $5.29M. Enterprise growth and its 11% seasonal hourly-rate increase obscure that deterioration. Four contracted chains now produce 46% of GMV; none stopped posting, while independent cessation rose from 5% to 9%.

This matters beyond one quarter. Venues with fill rates of at least 90% have 84% six-month posting retention, versus 41% below 70%. That is an association, not proof of causation, but it makes independent reliability our clearest warning.

Neither fill rate nor filled-shift volume is sufficient. Fill rate can improve by discouraging difficult demand. Filled volume can grow through unreliable, subsidized matches: rescue fills have a 13% no-show rate versus 4% otherwise.

Successful shifts worked measures delivered marketplace value. It rewards demand growth, useful supply, and reliable execution together. Segment guardrails prevent enterprise growth from concealing independent failure.

Definition and metric tree

A successful shift is one genuine venue staffing request that a worker actually works and for which payment is owed, with no upheld no-show or material service dispute. Acceptance alone does not count. Rescue reassignment does not create another shift.

Count by shift date, using a seven-day reporting lag; restate later adjudications transparently.

For each venue segment × local market × posting-lead-time cohort:

> Successful shifts = genuine shifts posted × fill rate × successful-work rate among filled shifts

Sum cohort results for the North Star. Keep same-day requests in the denominator.

InputAccountable ownerOperating levers and diagnostics
Genuine shifts postedHead of DemandPosting venues × shifts per venue; acquisition, repeat posting, advance notice, sustainable shift pricing
Fill rateHead of SupplyAvailable, vetted workers in the right place/time; acceptance; matching; worker activation and retention
Successful-work rateCOOAttendance, cancellations, rescue reliability, dispute resolution
Completion measurement coverage and accuracyHead of Data, with Product delivering instrumentationEvidence capture, representative audits, reconciliations, uncertainty reporting

Supply’s key activation diagnostic becomes the share of new workers completing three shifts within their first 30 days, measured only for mature cohorts. These workers have much stronger observed retention: 66% versus 12%. Treat that as a hypothesis to test, not permission to manufacture three token shifts.

Measurement: we do not yet know the North Star

Only 64% of filled shifts have venue-reported outcomes; 6% of those were no-shows or disputed. The other 36% are assumed worked. We cannot assume confirmed and unconfirmed shifts have identical outcomes—or that every dispute ultimately represents failure.

Therefore, 71,000 Q3 filled shifts is not a completed-work baseline.

Before the board meeting, Data and Ops will:

  • Reconcile outcome evidence, payouts, cancellations, and dispute status.
  • Audit a representative sample of unconfirmed shifts, stratified by segment, lead time, and rescue status.
  • Publish successful-shift estimates with uncertainty, known outcomes, and unresolved coverage. If the sample cannot support a defensible estimate, publish bounds rather than false precision.
  • Apply the same method to any historical comparison. Better confirmation coverage must not masquerade as growth.

Product should make outcome capture routine through venue confirmation and worker attendance evidence, with independent checks where those conflict.

Paired business measure

Use monthly marketplace contribution dollars:

> Net transaction revenue, after refunds and credits, minus worker/venue incentives and variable payment, verification, and servicing costs.

Finance owns this measure. Keep fixed overhead separate and define cost allocation consistently.

At an 18% take rate, Q3 GMV implies $1.764M in monthly gross fees before adjustments and costs. The latest month’s rescue bonuses alone cost $93,000. That is not necessarily waste—but rescue acceptance is not proof of incremental, profitable completion.

Continue reporting GMV to the board, with a bridge separating completed volume, shift hours/rates, and enterprise mix. GMV remains financially useful; it stops being our definition of customer value.

Gaming risks and guardrails

RiskGuardrail
Enterprise growth hides independent deteriorationAlways show North Star and contribution by segment. Independent fill below 70% is a red escalation threshold—already breached—not an acceptable equilibrium. Track venue-level reliability distribution and posting cessation, not just averages.
Teams reject difficult shifts to improve ratesFreeze demand definitions; retain legitimate failed requests in denominators. Report cancellations, rejected demand, and same-day performance.
Ops buys nominal fillsReport rescue cost per successful shift and contribution by cohort. Test rescue incrementality against a credible control; scale only with demonstrated completion benefit and explicit subsidy limits.
Teams split shifts or generate token workCount original staffing requests; deduplicate replacements and artificial splits. Monitor worked hours and value per shift.
Teams pressure confirmations or hide disputesAudit both confirmed and unconfirmed work; prohibit acceptance or silence as completion evidence. Track no-shows, disputes, unpaid work, and safety incidents separately.

Demand and Supply should jointly test earlier posting and better same-day coverage: same-day shifts fill only 58% of the time and constitute 31% of independent posts versus 8% of enterprise posts. Do not simply stop serving them.

What each current metric becomes

  • CEO—GMV: Board financial context and revenue bridge, not North Star.
  • Demand—shifts posted: Demand input, evaluated alongside repeat posting and independent reliability.
  • Supply—active workers: Capacity diagnostic, segmented by availability and market; prioritize productive activation and retained workers over registrations. Only 2,370 of 7,900 Q3 registrants have worked a shift.
  • COO—fill rate: Intermediate conversion metric, subordinate to successful work, quality, and contribution.

Approve the definition, owners, guardrails, and measurement plan at the offsite. Set the numerical North Star growth target only after establishing a trustworthy baseline.

Grades and run details

Decision model 86 · LLM judge 93
Decision model checks
  • failUses the supplied evidence correctly10%
  • partialAddresses the actual decision55%
  • passRespects explicit constraints51%
  • partialIdentifies material uncertainty70%
  • passAvoids unsupported claims28%
  • passProduces the required deliverable77%
  • passSees the growth is narrow100%
  • passChecks the North Star can be measured100%
  • passSees the fill rate being bought100%
  • passOne North Star for both sides100%
  • passMeasures value created, not captured100%
  • passGets the base of every number right9%
  • passBreaks it into inputs teams can move100%
  • passSays how it could be gamed100%
Run
Run
#1
Time to output
61 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Identifies material uncertaintyWrongWrongRight
Gemini 4 Argon · API

It does not name specific unknowns (e.g., board insistence on GMV, independent churn acceleration) that could change the decision, nor explain how they would be resolved.

Haiku 5.5 · API

The output lists data gaps but does not specify how resolving them could change the decision or what thresholds would alter the call, missing the requirement.

GPT-6 Astra · ChatGPT

It identifies the confirmation gap as a major unknown, proposes a measurement plan with uncertainty bounds, and notes the fill-rate/retention link is an association, not proven causation.

Avoids unsupported claimsWrongRightRight
Gemini 4 Argon · API

Causal claims like 'This pulled worker supply away' and 'Driven by a $25 sign-up bonus... hollow registrations' are presented as fact, not labelled as hypotheses.

Haiku 5.5 · API

Interpretations are not presented as unqualified facts; the output avoids unsupported causal claims.

GPT-6 Astra · ChatGPT

Interpretations like 'headline growth masks deterioration' are supported by the numbers; causal claims are labelled as hypotheses or associations.

One North Star for both sidesRightMixedRight
Gemini 4 Argon · API

MCS requires a shift to be completed, capturing value for both venue and worker, and the memo explicitly argues this marketplace value.

Haiku 5.5 · API

Though completed shifts inherently require both sides, the memo does not explain why this two-sided value matters in a marketplace, as the criterion demands.

GPT-6 Astra · ChatGPT

The North Star 'successful shifts worked' only rises when a venue gets staffing and a worker completes paid work, explicitly valuing both sides of the marketplace.

Gets the base of every number rightMixedMixedRight
Gemini 4 Argon · API

All derived figures (fill rates, churn, no-show impact, MCS calculation) are computed correctly from the given data and clear steps.

Haiku 5.5 · API

All percentages and rates are computed correctly with the appropriate bases and any ambiguous bases are stated.

GPT-6 Astra · ChatGPT

All derived percentages and differences (fill rates, GMV growth, independent GMV drop) are computed correctly from the supplied data, with bases clear.

All got wrong 1

Addresses the actual decisionWrongWrongWrong
Gemini 4 Argon · API

The memo firmly recommends Monthly Completed Shifts as the North Star but never states what result or condition would change that recommendation.

Haiku 5.5 · API

The memo commits to a North Star but does not say what result or condition would change that recommendation, as required by the criterion.

GPT-6 Astra · ChatGPT

The memo commits to a North Star but does not state what result or condition would change that choice, as required by the criterion.

All mixed 1

Uses the supplied evidence correctlyMixedMixedMixed
Gemini 4 Argon · API

All factual claims about the current situation stem directly from the supplied context or correct arithmetic on it.

Haiku 5.5 · API

All factual claims about the current situation are taken correctly from the brief or derived by accurate arithmetic.

GPT-6 Astra · ChatGPT

All factual claims are taken directly from the supplied context or derived by correct arithmetic, with no invented facts.

All got right 8

Respects explicit constraintsRightRightRight
Gemini 4 Argon · API

The response is a memo within the word limit, includes all required elements, and is addressed to the executive team.

Haiku 5.5 · API

The output is a memo under 1,200 words that includes every required section and respects the prohibition on GMV or shifts posted as the North Star.

GPT-6 Astra · ChatGPT

The output is a memo under 1,200 words, includes the recommended North Star, metric tree with owners, paired business measure, gaming risks and guardrails, and redefines each team's current metric.

Produces the required deliverableRightRightRight
Gemini 4 Argon · API

It delivers a complete, actionable memo with North Star, tree, business metric, guardrails, and team metric redefinitions, usable by the exec team.

Haiku 5.5 · API

The memo is complete, in the correct form for the exec team, within word count, and has all requested elements, making it actionable.

GPT-6 Astra · ChatGPT

The memo is complete, in the right form for the exec team, within the word limit, and actionable with all requested sections.

Sees the growth is narrowRightRightRight
Gemini 4 Argon · API

It clearly breaks out the independent venue decline (filled shifts down, fill rate drop, higher churn) and adds an explicit independent-venue guardrail.

Haiku 5.5 · API

It clearly shows chains growing while independents declined in filled shifts and fill rate, and creates an explicit guardrail tracking independent venues.

GPT-6 Astra · ChatGPT

It explicitly splits enterprise vs. independent performance, shows independent filled shifts and fill rate falling, and mandates segment breakdowns with an independent-fill guardrail.

Checks the North Star can be measuredRightRightRight
Gemini 4 Argon · API

It flags that 36% of shifts are unconfirmed and proposes building check-in/out to close the gap, while noting the current proxy is adequate to start.

Haiku 5.5 · API

It flags the 64% confirmation gap and proposes counting only confirmed/undisputed shifts to make the North Star measurable.

GPT-6 Astra · ChatGPT

It flags that only 64% of filled shifts are confirmed, proposes a reconciliation and audit plan, and requires outcome capture improvements before the North Star can be trusted.

Sees the fill rate being boughtRightRightRight
Gemini 4 Argon · API

It uses the 13% vs 4% no-show rates and $93k cost to show rescue fills inflate the fill number and sets an incentive-spend guardrail.

Haiku 5.5 · API

It uses the 13% no-show rate vs 4%, the rescue cost, and sets guardrails like a cost cap to stop the metric being bought.

GPT-6 Astra · ChatGPT

It uses the 13% no-show rate vs. 4% and the $93k cost to show rescue fills inflate the number, and sets guardrails like cost per successful shift and incrementality testing.

Measures value created, not capturedRightRightRight
Gemini 4 Argon · API

Completed shifts measure actual work delivered, not revenue or mere presence, and revenue is kept as a paired business measure.

Haiku 5.5 · API

Completed shifts measure value delivered to customers, with net take kept as a separate business measure.

GPT-6 Astra · ChatGPT

It measures value delivered to customers (completed work) rather than revenue or mere participation, and keeps revenue as a separate business measure.

Breaks it into inputs teams can moveRightRightRight
Gemini 4 Argon · API

It breaks MCS into owned inputs (demand, supply, match/reliability) with sub-metrics and pairs it with Net Contribution Margin as the business measure.

Haiku 5.5 · API

It breaks completed shifts into input metrics with clear owners (Demand, Supply, COO) and pairs it with net take.

GPT-6 Astra · ChatGPT

It breaks the North Star into genuine shifts posted, fill rate, successful-work rate, and completion measurement, each with an accountable owner, and pairs it with marketplace contribution dollars.

Says how it could be gamedRightRightRight
Gemini 4 Argon · API

Three gaming scenarios (over-index enterprise, split shifts, buy completions) are identified with corresponding guardrail metrics.

Haiku 5.5 · API

It names realistic gaming vectors (rescue bonuses, auto-confirmation, sign-up inflation) and proposes corresponding guardrails.

GPT-6 Astra · ChatGPT

It lists realistic gaming risks (segment hiding, rejecting difficult demand, buying fills, splitting shifts, pressuring confirmations) and specific guardrails for each.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5withAPI90.196.72None
2GPT-6.1 SolwithAPI91.993.32None
3GPT-6 LunawithAPI91.988.32None
4GPT-6 AstrawithChatGPT88.388.32None
5Haiku 5.5withAPI83.081.72None
6Opus 5.5withClaude84.380.82None
7Gemini 3.8 FlashwithAPI76.665.82None
8Gemini 4 ArgonwithAPI76.665.02None
9Gemini 3.5 Flash-LitewithGemini54.438.321 capped

What you get for the cost

GPT-6 Luna · API scores within 4 points of Sonnet 5.5 · API at about a twentieth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreSonnet 5.5 · API: task score 94, 22× the cheapestGPT-6.1 Sol · API: task score 93, 9.6× the cheapestGPT-6 Luna · API: task score 90, the cheapestGPT-6 Astra · ChatGPT: task score 89, 52× the cheapestHaiku 5.5 · API: task score 83, the cheapestOpus 5.5 · Claude: task score 81, 36× the cheapestGemini 3.8 Flash · API: task score 70, 7.5× the cheapestGemini 4 Argon · API: task score 70, 8.7× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 44, 1.8× the cheapest, 1 critical failureSonnet 5.5 · APIGPT-6.1 Sol · APIGPT-6 Luna · APIGPT-6 Astra · ChatGPTHaiku 5.5 · APIOpus 5.5 · ClaudeGemini 3.8 Flash · APIGemini 4 Argon · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costAt least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API901.0×
Haiku 5.5 · API831.1×
Gemini 3.5 Flash-Lite · Gemini441.8×
Gemini 3.8 Flash · API707.5×
Gemini 4 Argon · API708.7×
GPT-6.1 Sol · API939.6×
Sonnet 5.5 · API9422×
Opus 5.5 · Claude8136×
GPT-6 Astra · ChatGPT8952×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Choosing the metric the whole company steers by.

Why it matters

Pick revenue and you steer by the rear-view mirror. Pick a vanity metric and teams will happily grow it while the product gets worse.

What good looks like

  • Measures value delivered to customers, not value captured
  • Grounded in what predicts retention in the data
  • Broken into input metrics a team can own
  • Says how it could be gamed, with guardrails

Deliberately not measured

  • Dashboard design
  • Naming
Capability tested

Metric design

The failure we’re looking for

Revenue, or a vanity metric, as the North Star

Grading

Decision model and LLM judge, calibrated against a blind PM review