Humanity’s Last PRD
Latest drop: Haiku 5.5 · 8 Oct
Tasks / Metrics & Experimentation

Pick a North Star metric

Can the model pick the one metric that measures the value a product creates, and break it into inputs teams can move?

Task type v1.1 · 2 tasksLast changed 5 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 75% were usable with at most a quick edit.

Reliably right

  1. Says how it could be gamed100% pass
    Identifies gaming vectors (tapping through, notification overuse, short-term spikes, revenue manipulation, MAU drift) and sets specific guardrails (quality completion, push caps, blocking metrics, cohort retention checks, subscriber churn tracking).
    Sonnet 5.5 · API · Revenue, actives, or something else?
  2. Sees the growth is narrow100% pass
    It shows the split (chains up, independents' filled shifts and fill rate down, more leaving) and tracks independent venues via segment reporting and guardrails.
    GPT-6 Luna · API · Growing, on paper
  3. Checks the North Star can be measured100% pass
    It flags the 64% confirmation gap and proposes instrumenting check-in/out, reconciliation, and auditing to fix it before the board readout.
    GPT-6 Luna · API · Growing, on paper

Where it slips

  1. Identifies material uncertainty55% pass
    It identifies the confirmation gap as an unknown but does not say how resolving it could change the North Star decision.
    GPT-6.1 Sol · API · Growing, on paper
  2. Addresses the actual decision58% pass
    The output commits to a North Star but does not state what result or condition would change that recommendation.
    GPT-6 Luna · API · Growing, on paper
  3. Avoids unsupported claims67% pass
    It presents 'easy to tap through' and 'we don't track today' as facts without support, and does not label them as assumptions.
    Opus 5.5 · Claude · Revenue, actives, or something else?

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Measures value created, not captured

    Does the chosen North Star measure the value customers get from the product, rather than revenue or a count of people who merely showed up?

    Passes when Picks a metric of value delivered to customers (a core action that predicts retention), and keeps revenue as a separate business measure.

    Itamar GiladLenny’s PodcastSean EllisLenny’s Podcast

  2. Breaks it into inputs teams can move

    Does the output break the North Star into a few input metrics, each one a team could own and move week to week, and pair it with a separate business measure of value captured?

    Passes when Gives a small tree of input metrics, each with an owner and a link to the North Star, plus the revenue or business measure it's paired with.

    Itamar GiladLenny’s Podcast

  3. Says how it could be gamed

    Does the output pressure-test the North Star for how teams could raise it without delivering more value, and set guardrail metrics against that?

    Passes when Names a realistic way the metric could be gamed in this product and the guardrails that would catch it.

    Lauryn IsfordLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 4 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a Staff PM at Crewmate. Every team has been steering by its own number, and the exec team wants to pick one North Star at next week's offsite, so the CEO can report it to the board three weeks later. Write the memo they'll read beforehand, in no more than 1,200 words: the North Star you recommend and why, its metric tree with an owner for each input, the business measure it's paired with, how it could be gamed and the guardrails against that, and what each team's current metric becomes. The numbers below are as the data team sent them. Not all of it matters equally.

What the model was given9 items: About Crewmate, What each exec wants, Shifts by quarter (monthly averages), GMV (monthly average), Venues, Workers, Same-day shifts, Completion data, From the COO
About CrewmateA marketplace for hospitality shifts: venues post shifts, and vetted workers pick them up. Crewmate takes 18% of each shift's value. Two kinds of venue: four enterprise hotel and restaurant chains (310 venues between them, on annual contracts) and 2,790 independent venues (bars, restaurants, cafes) who pay as they go.
What each exec wantsCEO: GMV, 'because it's what the board looks at'. Head of Demand: shifts posted. Head of Supply: active workers. COO: fill rate.
Shifts by quarter (monthly averages)Enterprise chains: Q2 posted 21,000, filled 19,500. Q3 posted 34,000, filled 31,600. Independent venues: Q2 posted 56,000, filled 44,200. Q3 posted 58,000, filled 39,400.
GMV (monthly average)Q2 $8.5M, Q3 $9.8M. Enterprise chains were 34% of GMV in Q2 and 46% in Q3. The chains raised hourly rates by 11% for the holiday season from August.
VenuesVenues whose shifts fill 90% of the time or more: 84% still posting six months later. Under 70%: 41%. In Q3, 9% of independent venues stopped posting altogether (5% in Q2). No chain venue stopped.
WorkersRegistrations: 6,100 in Q2, 7,900 in Q3, after Supply introduced a $25 sign-up bonus in July. 2,370 of the Q3 registrants have worked a shift. Workers who work three or more shifts in their first month: 66% still working after three months; everyone else: 12%.
Same-day shiftsShifts posted less than 24 hours before they start fill 58% of the time. They are 31% of independents' posts and 8% of the chains'.
Completion dataVenues confirm in the app that a shift was worked for 64% of filled shifts; the rest are assumed worked unless the venue raises a dispute. Of confirmed shifts, 6% were no-shows or disputed.
From the COO"Since September, Ops pays workers a $15 rescue bonus to pick up same-day shifts that would otherwise go unfilled. 6,200 rescue fills last month, $93,000. Rescue-filled shifts have a 13% no-show rate, against 4% for the rest. It's kept our fill rate from falling further."
What a strong answer doesThe answer key the graders mark against

Works out from the raw numbers what the pack doesn't say: all of the growth is the four chains (filled shifts up from 19,500 to 31,600 a month, plus an 11% rate rise), while independent venues filled fewer shifts (44,200 to 39,400, so their fill rate fell from 79% to 68%) and are leaving faster (9% stopped posting, against 5%); GMV rose mostly on the chains' volume and prices. Picks a North Star of value delivered to both sides, such as shifts worked, but designs it so the decline can't hide: measured for independents separately, or as reliably staffed venues, or with an explicit independent-venue guardrail. Notices that 'worked' is only confirmed for 64% of shifts, so the North Star can't be trusted yet, and makes fixing confirmation an early input or uses a measurable proxy until then. Treats the rescue bonus as a way the fill rate (and any fills-based North Star) is being bought: 13% no-shows against 4%, so it needs guardrails such as cost per worked shift and no-show rate. Shows how shifts posted, registrations (2,370 of 7,900 worked) and GMV each rose without more value. Builds a metric tree with owners, pairs it with revenue or GMV as the business measure, and redefines each team's metric as an input.

Critical failures (cap the score at 40)
  • Recommends GMV or shifts posted as the North Star
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

GPT-6 LunaAPI97Usable with a quick editSonnet 5.5API95Usable with a quick editGPT-6.1 SolAPI93Usable with a quick editGPT-6 AstraChatGPT93Usable with a quick editOpus 5.5Claude88Usable with a quick editHaiku 5.5API82Usable with a quick editGemini 3.8 FlashAPI73Usable with a quick editGemini 3.5 Flash-LiteGemini62Needs more work
Sonnet 5.5API

Usable with a quick edit13 of 14 checks passed

The memo is thorough, covers all required elements, and could be used by a PM with only minor wording edits, without re-checking evidence or redoing reasoning.

Output

Memo: Choosing Crewmate's North Star

To: Exec team | From: Staff PM | For: Offsite decision; CEO reports to the board three weeks later

Recommendation

North Star: Completed shifts, meaning shifts filled by a vetted worker and worked (no no-show, no dispute), per month.

Paired business measure: Net revenue, which is our 18% take minus worker incentives (rescue and sign-up bonuses). GMV stays in the board deck as context, shown at constant rates and split by segment.

Why not the four numbers on the table

Each can rise while the business gets worse. Q2 to Q3, monthly averages:

  • Shifts posted rose 19% (77k to 92k), but filled shifts rose only 11%. Independents posted 4% more and filled 11% fewer (44.2k to 39.4k). Fill fell from 79% to 68%.
  • GMV rose 15% ($8.5M to $9.8M), but all the growth is chains. Chain GMV went from about $2.9M to $4.5M (+56%) and independent GMV from about $5.6M to $5.3M (−6%). The chains' 11% rate rise from August also inflates it. That is price, not service.
  • Active workers is the most gameable. Registrations rose 30% (6,100 to 7,900) after the $25 bonus, but only 2,370 of the Q3 registrants (30%) have worked a shift.
  • Fill rate is now partly bought. Chains hold at 93% and independents have dropped to 68%. Ops spends $93k a month on $15 rescue bonuses, and those fills no-show at 13% against 4% elsewhere. That is not a criticism of Ops. The fill rate simply can't tell a durable fill from a purchased one.

Completed shifts needs demand, supply and reliability to all work, and price can't move it. It also tracks what keeps venues: those filling 90% or more retain at 84% after six months, against 41% below 70%. The average independent is now at 68%, and 9% of independents stopped posting in Q3 (5% in Q2).

Caveats you should know: 1. The headline will look healthy. I estimate completed shifts rose about 11%, from ~60k to ~67k a month, applying the 6% failure rate to filled shifts. Chains are carrying that, so the independent segment must be reported next to it (see guardrails). 2. The baseline is provisional. Venues confirm only 64% of filled shifts. The 6% failure rate comes from that confirmed subset, and the other 36% are assumed worked. Our first board number should be labeled an estimate, with confirmation coverage disclosed.

Metric tree

Completed shifts = Posted shifts × Fill rate × Show-up rate

InputOwnerLeading drivers (same owner)
Posted shifts, by segmentHead of DemandRetained independent venues (churn 9% in Q3); posts per venue
Fill rate, reported with and without rescue fillsCOOSame-day fill (58% vs. far higher for planned shifts); time to fill; rescue spend per completed shift
Show-up rate (filled shifts not no-show or disputed)Head of SupplyWorker reliability; no-show rate of rescue fills (13%)
Worker capacity: activated workers (3+ shifts in first month)Head of SupplyActivation of new registrants; 3-month retention (66% for activated workers vs. 12% for the rest)
Confirmation coverage (share of filled shifts with a verified outcome)Product (me), with OpsIn-app confirmation rate; worker clock-in
Net revenue (paired measure)CEO/FinanceTake rate; incentive spend as % of take

Supply drives fill, so the Head of Supply's activation target is a stated input to the COO's fill target. Owners agree those two numbers jointly.

How it could be gamed, and the guardrails

Gaming riskGuardrail
Unverified "completions." 36% of fills are assumed worked, so sloppy confirmation inflates the count.Target 90% confirmation coverage within two quarters. Add worker clock-in as a second signal. Audit a monthly sample. The board number states its coverage.
Buying fills. Rescue bonuses lift fill but 13% no-show. Rescue costs $15 against average take of about $25 a shift.Count only completed shifts. Pay rescue bonuses after the shift is confirmed worked, not at pickup. Cap incentive spend as a % of take (now ~5% from rescue alone). Net revenue exposes the cost.
Chains masking independents. Headline rises while independents decay.Report independent completed shifts, fill rate and 90-day venue churn beside the headline. Trigger a review if independent churn exceeds 5% (the Q2 level) or independent fill is below 70%.
Shift-splitting. Posting more, shorter shifts inflates the count.Report hours worked and average shift length next to the count. If they diverge, move to hours.
Registration farming. Chasing sign-up bonuses.Registrations aren't in the tree. Pay the sign-up bonus on the third completed shift, not at registration.
Easy-shift cherry-picking. Ops or Supply prioritizes easy fills and lets same-day shifts lapse.Track fill by segment and by same-day vs. planned. Same-day is 31% of independents' posts, so neglecting it shows up quickly.
Phantom demand. Venues post shifts and cancel late to flatter posted counts.Track late cancellations (<24h) as a Demand guardrail.

What each team's current metric becomes

TodayBecomes
CEO: GMVBoard context, reported at constant rates and split chain/independent. The paired measure is net revenue. GMV is a mix of volume and price, so it can't be the North Star.
Demand: shifts postedInput to the tree, with churn and late-cancellation guardrails. The goal is posts that get worked, not posts.
Supply: active workersReplaced by activated workers (3+ shifts in first month) and 3-month retention. Registrations become a diagnostic only.
COO: fill rateInput to the tree, always shown with and without rescue fills, plus incentive cost per completed shift.

What I'm asking the exec team to decide at the offsite

  1. Adopt completed shifts as the North Star, paired with net revenue.
  2. Agree the definition: filled, not no-show, not disputed. The board gets the estimated baseline (~60k in Q2, ~67k in Q3) with confirmation coverage disclosed.
  3. Confirm the owners above.
  4. Change incentive payouts. Rescue bonuses are paid on completion, and the sign-up bonus on the third shift. I'd keep rescue itself, because fill matters for retention. We just shouldn't count it as organic.
  5. Fund confirmation coverage, in-app confirmation prompts and worker clock-in, as the first Product deliverable. Until it lands, the North Star carries a known error band.

What I'm not claiming

I haven't shown that rescue or the sign-up bonus is net-negative. The data doesn't isolate their effects, and rescue covers only about one month of Q3. The point is that neither should be able to move our headline number without proving it produced a completed shift. Over the next quarter, Product and Finance should model the retention value of rescue fills against their cost and no-show rate.

Grades and run details

Decision model 96 · LLM judge 93
Decision model checks
  • passUses the supplied evidence correctly12%
  • passAddresses the actual decision86%
  • passRespects explicit constraints38%
  • passIdentifies material uncertainty97%
  • passAvoids unsupported claims32%
  • passProduces the required deliverable88%
  • passSees the growth is narrow100%
  • passChecks the North Star can be measured100%
  • passSees the fill rate being bought100%
  • passOne North Star for both sides95%
  • passMeasures value created, not captured97%
  • partialGets the base of every number right6%
  • passBreaks it into inputs teams can move100%
  • passSays how it could be gamed100%
Run
Run
#1
API response time
68 s
Submitted
5 Oct 2026
Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit9 of 14 checks passed

The memo contains an incorrect cost figure ($197k) and unsupported causal claims that would need correction before a PM could rely on it without re-checking the evidence.

Output

MEMORANDUM

TO: Executive Team FROM: Staff Product Manager, Crewmate DATE: October 24, 2023 SUBJECT: North Star Metric Recommendation & Offsite Alignment

---

Executive Summary

Next week, we must select a single North Star metric for the board. Currently, our teams are locally optimized: Demand chases posted shifts, Supply chases registrations, and Ops chases fill rate. Meanwhile, the CEO tracks GMV, which is a lagging scoreboard, not a driver.

To fix this, our North Star must capture sustainable marketplace liquidity. I recommend we set our North Star as Completed Shift Value (CSV), paired with a health guardrail of Gross Margin.

---

The Recommendation: Completed Shift Value (CSV)

#### Why not the alternatives? * GMV (CEO preference): GMV is vanity. In Q3, our GMV grew from $8.5M to $9.8M, but enterprise chains drove that growth solely by raising hourly rates 11%. Independent volume actually degraded. GMV masks marketplace health and rewards inflation. * Shifts Posted (Demand): Drives low-quality supply ingestion. Independents are already posting shifts that fail to fill, leading to churn (venues with <70% fill rates drop out at a 59% rate). * Active Workers (Supply): Driven by vanity sign-ups ($25 bonus cost us $197k in Q3 for registrants who never worked). * Fill Rate (COO): Can be artificially inflated by restricting demand or via expensive artificial interventions (e.g., Ops’ $15 rescue bonus, which is masking a bleeding independent segment).

#### Why Completed Shift Value (CSV)? CSV represents the actual dollar value of shifts that are successfully worked and confirmed. It directly ties our 18% take rate to real economic activity, stripping out pricing inflation, unfulfilled demand, and ghost shifts. It forces Demand, Supply, and Ops to pull in the same direction: matching real workers to real shifts that actually happen.

---

The Business Measure Pair

A North Star cannot stand alone without a quality/sustainability check. * North Star: Completed Shift Value (CSV) * Paired Business Measure: Gross Margin ($)

Why? CSV can be gamed by heavy discounting or expensive operational workarounds (like our $15 rescue bonus, which cost $93k last month alone). Pairing CSV with Gross Margin ensures we are growing profitably, not subsidizing our way to top-line numbers.

---

Metric Tree & Owners

To move CSV, every team owns a specific input in the compounding loop.

``` [NORTH STAR: Completed Shift Value (CSV)] | +---------------------------+---------------------------+ | | [Total Completed Shifts] [Average Shift Value] | | +---------------------------+ | | | (Owned by Enterprise [Independent CSV] [Enterprise CSV] & Independent PMs) | | (Head of Demand / (Head of Enterprise Sales) Independent PM) ```

#### Metric Tree Inputs & Owners

1. Independent Completed Shifts (Owner: Head of Demand / Independent PM) * Sub-input: Independent Shifts Posted (Guardrail: maintain >80% fill rate). * Sub-input: Independent Fill Rate. 2. Enterprise Completed Shifts (Owner: Head of Enterprise Sales) * Sub-input: Enterprise Shifts Posted (Q3: 34k/mo). * Sub-input: Enterprise Fill Rate (Q3: 93% — must maintain SLAs). 3. Worker Activation & Retention (Owner: Head of Supply) * Sub-input: 30-Day D1-to-D30 Retention (% of workers completing $\ge 3$ shifts in month one; currently the key predictor of 3-month retention at 66% vs 12%). 4. Marketplace Integrity & Completion (Owner: COO) * Sub-input: Shift Confirmation Rate (closing the gap from our current 64% manual confirmation baseline). * Sub-input: No-show / Dispute Rate (keeping it <5%).

---

Gaming Risks & Guardrails

How CSV Could Be GamedThe DangerGuardrail
:---:---:---
1. Rescue Bonus Addiction<br>(Ops artificially inflating completions by throwing $15 bonuses at same-day shifts).Destroys unit economics. Rescue-filled shifts have a 13% no-show rate (vs 4% baseline) and eat our margin.Rescue Subsidy Cap: Total rescue bonuses cannot exceed 2% of monthly gross margin. Ops must fix matching latency instead of buying liquidity.
2. Enterprise Price Inflating<br>(Ramping up shift wages to juice CSV without adding volume).Alienates independent venues and burns enterprise contract budgets early.Average Hourly Rate Variance: Enterprise/Independent shift rates cannot outpace local wage inflation by >5% QoQ without product approval.
3. Fake Registrations / Low-Bar Matching<br>(Pushing unqualified workers into shifts to hit volume).Venue churn. Venues below 90% fill rate churn heavily; 9% of independents already left in Q3 due to poor fill rates.Venue Retention Floor: 6-month retention of independent venues must not drop below 80% for cohorts with $\ge 90\%$ fill rates.

---

Team Metric Transformations

To align execution with the new North Star, existing team KPIs must evolve:

  • CEO / Board:
  • Old: GMV ($9.8M Q3 average).
  • New: Completed Shift Value (CSV) and Gross Margin ($).
  • Head of Demand:
  • Old: Raw shifts posted (incentivized spamming unfillable indie shifts).
  • New: Independent Completed Shift Value (ICSV), subject to a $\ge 80\%$ fill rate minimum.
  • Head of Supply:
  • Old: Raw worker registrations (incentivized wasteful $25 sign-up bonuses that yielded low-intent users).
  • New: Worker Activation Rate (% of new registrants who complete $\ge 3$ shifts in their first 30 days).
  • COO / Operations:
  • Old: Overall raw fill rate.
  • New: Net Reliable Fill Rate (Total completed shifts minus no-shows/disputes, excluding unviable rescue-subsidized interventions).
  • Enterprise Sales:
  • Old: Enterprise shift volume.
  • New: Enterprise Contract CSV Yield & Renewal Rate (ensuring our 4 annual contract chains scale predictably without margin erosion).

Conclusion

By adopting CSV and Gross Margin, we give the board a honest reflection of our marketplace health, eliminate perverse incentives across Demand and Supply, and force Ops to solve systemic matching issues rather than masking them with cash. Let’s lock this in at the offsite.

Claims the judge couldn’t find in the brief

  • enterprise chains drove that growth solely by raising hourly rates 11%.
  • $25 bonus cost us $197k in Q3 for registrants who never worked.
  • Ops’ $15 rescue bonus, which is masking a bleeding independent segment.
  • 9% of independents already left in Q3 due to poor fill rates.

Grades and run details

Decision model 68 · LLM judge 60
Decision model checks
  • failUses the supplied evidence correctly63%
  • passAddresses the actual decision53%
  • partialRespects explicit constraints37%
  • partialIdentifies material uncertainty68%
  • failAvoids unsupported claims33%
  • partialProduces the required deliverable25%
  • passSees the growth is narrow61%
  • passChecks the North Star can be measured85%
  • passSees the fill rate being bought100%
  • passOne North Star for both sides87%
  • passMeasures value created, not captured17%
  • failGets the base of every number right53%
  • passBreaks it into inputs teams can move99%
  • passSays how it could be gamed100%
Run
Run
#1
Time to output
9 s
Submitted
5 Oct 2026
GPT-6 AstraChatGPT

Usable with a quick edit12 of 14 checks passed

The memo is complete and actionable; a PM could add a sentence about what would change the North Star choice as a quick edit, without rechecking evidence or redoing reasoning.

Output

Decision memo: Crewmate’s North Star

To: Executive team From: Staff PM Decision for next week’s offsite: Adopt successful shifts worked per month as our North Star, paired with marketplace contribution dollars. Publish both with mandatory enterprise/independent breakdowns.

Why this, not today’s competing metrics?

Crewmate creates value when a venue gets the staffing it requested and a worker completes paid work—not when someone posts, registers, or accepts a shift.

Our headline growth currently masks deteriorating service:

Monthly averageQ2Q3
------:---:
Total filled shifts63,70071,000
Overall fill rate82.7%77.2%
Enterprise fill rate92.9%92.9%
Independent fill rate78.9%67.9%

GMV rose 15.3%, but independent GMV fell approximately 5.7%, from $5.61M to $5.29M. Enterprise growth and its 11% seasonal hourly-rate increase obscure that deterioration. Four contracted chains now produce 46% of GMV; none stopped posting, while independent cessation rose from 5% to 9%.

This matters beyond one quarter. Venues with fill rates of at least 90% have 84% six-month posting retention, versus 41% below 70%. That is an association, not proof of causation, but it makes independent reliability our clearest warning.

Neither fill rate nor filled-shift volume is sufficient. Fill rate can improve by discouraging difficult demand. Filled volume can grow through unreliable, subsidized matches: rescue fills have a 13% no-show rate versus 4% otherwise.

Successful shifts worked measures delivered marketplace value. It rewards demand growth, useful supply, and reliable execution together. Segment guardrails prevent enterprise growth from concealing independent failure.

Definition and metric tree

A successful shift is one genuine venue staffing request that a worker actually works and for which payment is owed, with no upheld no-show or material service dispute. Acceptance alone does not count. Rescue reassignment does not create another shift.

Count by shift date, using a seven-day reporting lag; restate later adjudications transparently.

For each venue segment × local market × posting-lead-time cohort:

> Successful shifts = genuine shifts posted × fill rate × successful-work rate among filled shifts

Sum cohort results for the North Star. Keep same-day requests in the denominator.

InputAccountable ownerOperating levers and diagnostics
Genuine shifts postedHead of DemandPosting venues × shifts per venue; acquisition, repeat posting, advance notice, sustainable shift pricing
Fill rateHead of SupplyAvailable, vetted workers in the right place/time; acceptance; matching; worker activation and retention
Successful-work rateCOOAttendance, cancellations, rescue reliability, dispute resolution
Completion measurement coverage and accuracyHead of Data, with Product delivering instrumentationEvidence capture, representative audits, reconciliations, uncertainty reporting

Supply’s key activation diagnostic becomes the share of new workers completing three shifts within their first 30 days, measured only for mature cohorts. These workers have much stronger observed retention: 66% versus 12%. Treat that as a hypothesis to test, not permission to manufacture three token shifts.

Measurement: we do not yet know the North Star

Only 64% of filled shifts have venue-reported outcomes; 6% of those were no-shows or disputed. The other 36% are assumed worked. We cannot assume confirmed and unconfirmed shifts have identical outcomes—or that every dispute ultimately represents failure.

Therefore, 71,000 Q3 filled shifts is not a completed-work baseline.

Before the board meeting, Data and Ops will:

  • Reconcile outcome evidence, payouts, cancellations, and dispute status.
  • Audit a representative sample of unconfirmed shifts, stratified by segment, lead time, and rescue status.
  • Publish successful-shift estimates with uncertainty, known outcomes, and unresolved coverage. If the sample cannot support a defensible estimate, publish bounds rather than false precision.
  • Apply the same method to any historical comparison. Better confirmation coverage must not masquerade as growth.

Product should make outcome capture routine through venue confirmation and worker attendance evidence, with independent checks where those conflict.

Paired business measure

Use monthly marketplace contribution dollars:

> Net transaction revenue, after refunds and credits, minus worker/venue incentives and variable payment, verification, and servicing costs.

Finance owns this measure. Keep fixed overhead separate and define cost allocation consistently.

At an 18% take rate, Q3 GMV implies $1.764M in monthly gross fees before adjustments and costs. The latest month’s rescue bonuses alone cost $93,000. That is not necessarily waste—but rescue acceptance is not proof of incremental, profitable completion.

Continue reporting GMV to the board, with a bridge separating completed volume, shift hours/rates, and enterprise mix. GMV remains financially useful; it stops being our definition of customer value.

Gaming risks and guardrails

RiskGuardrail
Enterprise growth hides independent deteriorationAlways show North Star and contribution by segment. Independent fill below 70% is a red escalation threshold—already breached—not an acceptable equilibrium. Track venue-level reliability distribution and posting cessation, not just averages.
Teams reject difficult shifts to improve ratesFreeze demand definitions; retain legitimate failed requests in denominators. Report cancellations, rejected demand, and same-day performance.
Ops buys nominal fillsReport rescue cost per successful shift and contribution by cohort. Test rescue incrementality against a credible control; scale only with demonstrated completion benefit and explicit subsidy limits.
Teams split shifts or generate token workCount original staffing requests; deduplicate replacements and artificial splits. Monitor worked hours and value per shift.
Teams pressure confirmations or hide disputesAudit both confirmed and unconfirmed work; prohibit acceptance or silence as completion evidence. Track no-shows, disputes, unpaid work, and safety incidents separately.

Demand and Supply should jointly test earlier posting and better same-day coverage: same-day shifts fill only 58% of the time and constitute 31% of independent posts versus 8% of enterprise posts. Do not simply stop serving them.

What each current metric becomes

  • CEO—GMV: Board financial context and revenue bridge, not North Star.
  • Demand—shifts posted: Demand input, evaluated alongside repeat posting and independent reliability.
  • Supply—active workers: Capacity diagnostic, segmented by availability and market; prioritize productive activation and retained workers over registrations. Only 2,370 of 7,900 Q3 registrants have worked a shift.
  • COO—fill rate: Intermediate conversion metric, subordinate to successful work, quality, and contribution.

Approve the definition, owners, guardrails, and measurement plan at the offsite. Set the numerical North Star growth target only after establishing a trustworthy baseline.

Grades and run details

Decision model 93 · LLM judge 93
Decision model checks
  • passUses the supplied evidence correctly12%
  • passAddresses the actual decision86%
  • passRespects explicit constraints43%
  • passIdentifies material uncertainty99%
  • passAvoids unsupported claims35%
  • passProduces the required deliverable74%
  • passSees the growth is narrow100%
  • passChecks the North Star can be measured100%
  • passSees the fill rate being bought100%
  • passOne North Star for both sides100%
  • passMeasures value created, not captured99%
  • failGets the base of every number right16%
  • passBreaks it into inputs teams can move100%
  • passSays how it could be gamed100%
Run
Run
#1
Time to output
61 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Uses the supplied evidence correctlyRightWrongRight
Sonnet 5.5 · API

All factual claims about the current situation are taken directly from the supplied context or derived by correct arithmetic.

Gemini 3.5 Flash-Lite · Gemini

The $197k cost claim is incorrect, and several causal statements (e.g., 'solely by raising rates', 'due to poor fill rates') are presented as fact without support.

GPT-6 Astra · ChatGPT

All factual claims are taken directly from the supplied context or derived by correct arithmetic, with no invented facts.

Identifies material uncertaintyRightWrongRight
Sonnet 5.5 · API

Identifies the confirmation gap and unmodeled incentive effects as material unknowns, and says how to resolve them (fund confirmation coverage, model retention value).

Gemini 3.5 Flash-Lite · Gemini

It does not name specific unknowns that could change the decision or how they would be resolved.

GPT-6 Astra · ChatGPT

It identifies the confirmation gap as a major unknown, proposes a measurement plan with uncertainty bounds, and notes the fill-rate/retention link is an association, not proven causation.

Avoids unsupported claimsRightWrongRight
Sonnet 5.5 · API

Estimates and assumptions are clearly labeled; no interpretations are presented as established fact.

Gemini 3.5 Flash-Lite · Gemini

It presents interpretations like 'solely by raising rates' and 'masking a bleeding independent segment' as established facts.

GPT-6 Astra · ChatGPT

Interpretations like 'headline growth masks deterioration' are supported by the numbers; causal claims are labelled as hypotheses or associations.

Gets the base of every number rightRightWrongMixed
Sonnet 5.5 · API

All derived figures (chain/independent GMV, fill rates, growth rates) use correct denominators and arithmetic; the completed-shifts estimate is caveated.

Gemini 3.5 Flash-Lite · Gemini

The $197k cost for non-working registrants is miscalculated; the correct figure is $138,250.

GPT-6 Astra · ChatGPT

All derived percentages and differences (fill rates, GMV growth, independent GMV drop) are computed correctly from the supplied data, with bases clear.

All mixed 1

Addresses the actual decisionMixedMixedMixed
Sonnet 5.5 · API

The output commits to a clear North Star recommendation but does not state what result or condition would change that answer.

Gemini 3.5 Flash-Lite · Gemini

The memo does not state what result or condition would change the recommendation.

GPT-6 Astra · ChatGPT

The memo commits to a North Star but does not state what result or condition would change that choice, as required by the criterion.

All got right 9

Respects explicit constraintsRightRightRight
Sonnet 5.5 · API

The memo is within 1,200 words, includes all required elements (North Star, metric tree, business measure, gaming/guardrails, team metric changes), and is addressed to the exec team.

Gemini 3.5 Flash-Lite · Gemini

The memo is under 1,200 words, includes all required elements, and does not recommend GMV or shifts posted as the North Star.

GPT-6 Astra · ChatGPT

The output is a memo under 1,200 words, includes the recommended North Star, metric tree with owners, paired business measure, gaming risks and guardrails, and redefines each team's current metric.

Produces the required deliverableRightRightRight
Sonnet 5.5 · API

The memo is complete, in the right form for the exec team, and usable as-is with light edits.

Gemini 3.5 Flash-Lite · Gemini

The memo is complete, in the right form for the exec team, and covers all requested components.

GPT-6 Astra · ChatGPT

The memo is complete, in the right form for the exec team, within the word limit, and actionable with all requested sections.

Sees the growth is narrowRightRightRight
Sonnet 5.5 · API

Shows that growth came from chains while independents' filled shifts and fill rate fell and churn rose, and sets guardrails to track independent completed shifts and churn separately.

Gemini 3.5 Flash-Lite · Gemini

It shows chains up, independents' filled shifts and fill rate down, more independents leaving, and tracks independent venues via a separate input and guardrail.

GPT-6 Astra · ChatGPT

It explicitly splits enterprise vs. independent performance, shows independent filled shifts and fill rate falling, and mandates segment breakdowns with an independent-fill guardrail.

Checks the North Star can be measuredRightRightRight
Sonnet 5.5 · API

Flags the 64% confirmation gap and proposes fixes (worker clock-in, 90% coverage target, board disclosure) before the North Star goes to the board.

Gemini 3.5 Flash-Lite · Gemini

It flags the 64% confirmation gap and makes improving confirmation rate a COO-owned input.

GPT-6 Astra · ChatGPT

It flags that only 64% of filled shifts are confirmed, proposes a reconciliation and audit plan, and requires outcome capture improvements before the North Star can be trusted.

Sees the fill rate being boughtRightRightRight
Sonnet 5.5 · API

Uses the 13% no-show rate vs 4% and the $93k cost to show fill is being bought, and sets guardrails like paying on completion and capping incentive spend.

Gemini 3.5 Flash-Lite · Gemini

It uses the 13% no-show rate and $93k cost to show rescue fills inflate numbers, and sets a rescue subsidy cap guardrail.

GPT-6 Astra · ChatGPT

It uses the 13% no-show rate vs. 4% and the $93k cost to show rescue fills inflate the number, and sets guardrails like cost per successful shift and incrementality testing.

One North Star for both sidesRightRightRight
Sonnet 5.5 · API

Completed shifts require both a venue posting and a worker working, so the metric only rises when both sides get value.

Gemini 3.5 Flash-Lite · Gemini

Completed Shift Value only rises when a venue and a worker both get value, and the memo explains why that matters for marketplace liquidity.

GPT-6 Astra · ChatGPT

The North Star 'successful shifts worked' only rises when a venue gets staffing and a worker completes paid work, explicitly valuing both sides of the marketplace.

Measures value created, not capturedRightRightRight
Sonnet 5.5 · API

The North Star measures completed shifts (value delivered to customers), and net revenue is kept as a separate business measure.

Gemini 3.5 Flash-Lite · Gemini

CSV measures value delivered (shifts actually worked), and Gross Margin is kept as a separate business measure.

GPT-6 Astra · ChatGPT

It measures value delivered to customers (completed work) rather than revenue or mere participation, and keeps revenue as a separate business measure.

Breaks it into inputs teams can moveRightRightRight
Sonnet 5.5 · API

Breaks the North Star into posted shifts, fill rate, show-up rate, and worker capacity, each with an owner, and pairs it with net revenue.

Gemini 3.5 Flash-Lite · Gemini

It breaks CSV into input metrics (Independent/Enterprise Completed Shifts, Worker Activation, Confirmation Rate) with owners, and pairs with Gross Margin.

GPT-6 Astra · ChatGPT

It breaks the North Star into genuine shifts posted, fill rate, successful-work rate, and completion measurement, each with an accountable owner, and pairs it with marketplace contribution dollars.

Says how it could be gamedRightRightRight
Sonnet 5.5 · API

Lists realistic gaming risks (unverified completions, buying fills, shift-splitting, registration farming, cherry-picking, phantom demand) and specific guardrails.

Gemini 3.5 Flash-Lite · Gemini

It names three realistic gaming risks (rescue bonus addiction, enterprise price inflating, fake registrations) and sets guardrails for each.

GPT-6 Astra · ChatGPT

It lists realistic gaming risks (segment hiding, rejecting difficult demand, buying fills, splitting shifts, pressuring confirmations) and specific guardrails for each.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5API95.996.72None
2GPT-6.1 SolAPI97.793.32None
3GPT-6 LunaAPI95.588.32None
4GPT-6 AstraChatGPT94.288.32None
5Opus 5.5Claude94.680.82None
6Haiku 5.5API88.881.72None
7Gemini 3.8 FlashAPI78.965.82None
8Gemini 3.5 Flash-LiteGemini54.438.321 capped

What you get for the cost

GPT-6 Luna · API scores within 4 points of Sonnet 5.5 · API at about a twentieth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreSonnet 5.5 · API: task score 96, 22× the cheapestGPT-6.1 Sol · API: task score 96, 9.6× the cheapestGPT-6 Luna · API: task score 92, the cheapestGPT-6 Astra · ChatGPT: task score 91, 52× the cheapestOpus 5.5 · Claude: task score 86, 36× the cheapestHaiku 5.5 · API: task score 86, the cheapestGemini 3.8 Flash · API: task score 71, 7.5× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 44, 1.8× the cheapest, 1 critical failureSonnet 5.5 · APIGPT-6.1 Sol · APIGPT-6 Luna · APIGPT-6 Astra · ChatGPTOpus 5.5 · ClaudeHaiku 5.5 · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costAt least one critical failure

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Choosing the metric the whole company steers by.

Why it matters

Pick revenue and you steer by the rear-view mirror. Pick a vanity metric and teams will happily grow it while the product gets worse.

What good looks like

  • Measures value delivered to customers, not value captured
  • Grounded in what predicts retention in the data
  • Broken into input metrics a team can own
  • Says how it could be gamed, with guardrails

Deliberately not measured

  • Dashboard design
  • Naming
Capability tested

Metric design

The failure we’re looking for

Revenue, or a vanity metric, as the North Star

Grading

Decision model and LLM judge, calibrated against a blind PM review