Humanity’s Last PRD
Latest drop: Haiku 5.5 · 8 Oct
Tasks / Metrics & Experimentation

Pick a North Star metric

Can the model pick the one metric that measures the value a product creates, and break it into inputs teams can move?

Task type v1.1 · 2 tasksLast changed 5 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 75% were usable with at most a quick edit.

Reliably right

  1. Says how it could be gamed100% pass
    Identifies gaming vectors (tapping through, notification overuse, short-term spikes, revenue manipulation, MAU drift) and sets specific guardrails (quality completion, push caps, blocking metrics, cohort retention checks, subscriber churn tracking).
    Sonnet 5.5 · API · Revenue, actives, or something else?
  2. Sees the growth is narrow100% pass
    It shows the split (chains up, independents' filled shifts and fill rate down, more leaving) and tracks independent venues via segment reporting and guardrails.
    GPT-6 Luna · API · Growing, on paper
  3. Checks the North Star can be measured100% pass
    It flags the 64% confirmation gap and proposes instrumenting check-in/out, reconciliation, and auditing to fix it before the board readout.
    GPT-6 Luna · API · Growing, on paper

Where it slips

  1. Identifies material uncertainty55% pass
    It identifies the confirmation gap as an unknown but does not say how resolving it could change the North Star decision.
    GPT-6.1 Sol · API · Growing, on paper
  2. Addresses the actual decision58% pass
    The output commits to a North Star but does not state what result or condition would change that recommendation.
    GPT-6 Luna · API · Growing, on paper
  3. Avoids unsupported claims67% pass
    It presents 'easy to tap through' and 'we don't track today' as facts without support, and does not label them as assumptions.
    Opus 5.5 · Claude · Revenue, actives, or something else?

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Measures value created, not captured

    Does the chosen North Star measure the value customers get from the product, rather than revenue or a count of people who merely showed up?

    Passes when Picks a metric of value delivered to customers (a core action that predicts retention), and keeps revenue as a separate business measure.

    Itamar GiladLenny’s PodcastSean EllisLenny’s Podcast

  2. Breaks it into inputs teams can move

    Does the output break the North Star into a few input metrics, each one a team could own and move week to week, and pair it with a separate business measure of value captured?

    Passes when Gives a small tree of input metrics, each with an owner and a link to the North Star, plus the revenue or business measure it's paired with.

    Itamar GiladLenny’s Podcast

  3. Says how it could be gamed

    Does the output pressure-test the North Star for how teams could raise it without delivering more value, and set guardrail metrics against that?

    Passes when Names a realistic way the metric could be gamed in this product and the guardrails that would catch it.

    Lauryn IsfordLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 4 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a Staff PM at Crewmate. Every team has been steering by its own number, and the exec team wants to pick one North Star at next week's offsite, so the CEO can report it to the board three weeks later. Write the memo they'll read beforehand, in no more than 1,200 words: the North Star you recommend and why, its metric tree with an owner for each input, the business measure it's paired with, how it could be gamed and the guardrails against that, and what each team's current metric becomes. The numbers below are as the data team sent them. Not all of it matters equally.

What the model was given9 items: About Crewmate, What each exec wants, Shifts by quarter (monthly averages), GMV (monthly average), Venues, Workers, Same-day shifts, Completion data, From the COO
About CrewmateA marketplace for hospitality shifts: venues post shifts, and vetted workers pick them up. Crewmate takes 18% of each shift's value. Two kinds of venue: four enterprise hotel and restaurant chains (310 venues between them, on annual contracts) and 2,790 independent venues (bars, restaurants, cafes) who pay as they go.
What each exec wantsCEO: GMV, 'because it's what the board looks at'. Head of Demand: shifts posted. Head of Supply: active workers. COO: fill rate.
Shifts by quarter (monthly averages)Enterprise chains: Q2 posted 21,000, filled 19,500. Q3 posted 34,000, filled 31,600. Independent venues: Q2 posted 56,000, filled 44,200. Q3 posted 58,000, filled 39,400.
GMV (monthly average)Q2 $8.5M, Q3 $9.8M. Enterprise chains were 34% of GMV in Q2 and 46% in Q3. The chains raised hourly rates by 11% for the holiday season from August.
VenuesVenues whose shifts fill 90% of the time or more: 84% still posting six months later. Under 70%: 41%. In Q3, 9% of independent venues stopped posting altogether (5% in Q2). No chain venue stopped.
WorkersRegistrations: 6,100 in Q2, 7,900 in Q3, after Supply introduced a $25 sign-up bonus in July. 2,370 of the Q3 registrants have worked a shift. Workers who work three or more shifts in their first month: 66% still working after three months; everyone else: 12%.
Same-day shiftsShifts posted less than 24 hours before they start fill 58% of the time. They are 31% of independents' posts and 8% of the chains'.
Completion dataVenues confirm in the app that a shift was worked for 64% of filled shifts; the rest are assumed worked unless the venue raises a dispute. Of confirmed shifts, 6% were no-shows or disputed.
From the COO"Since September, Ops pays workers a $15 rescue bonus to pick up same-day shifts that would otherwise go unfilled. 6,200 rescue fills last month, $93,000. Rescue-filled shifts have a 13% no-show rate, against 4% for the rest. It's kept our fill rate from falling further."
What a strong answer doesThe answer key the graders mark against

Works out from the raw numbers what the pack doesn't say: all of the growth is the four chains (filled shifts up from 19,500 to 31,600 a month, plus an 11% rate rise), while independent venues filled fewer shifts (44,200 to 39,400, so their fill rate fell from 79% to 68%) and are leaving faster (9% stopped posting, against 5%); GMV rose mostly on the chains' volume and prices. Picks a North Star of value delivered to both sides, such as shifts worked, but designs it so the decline can't hide: measured for independents separately, or as reliably staffed venues, or with an explicit independent-venue guardrail. Notices that 'worked' is only confirmed for 64% of shifts, so the North Star can't be trusted yet, and makes fixing confirmation an early input or uses a measurable proxy until then. Treats the rescue bonus as a way the fill rate (and any fills-based North Star) is being bought: 13% no-shows against 4%, so it needs guardrails such as cost per worked shift and no-show rate. Shows how shifts posted, registrations (2,370 of 7,900 worked) and GMV each rose without more value. Builds a metric tree with owners, pairs it with revenue or GMV as the business measure, and redefines each team's metric as an input.

Critical failures (cap the score at 40)
  • Recommends GMV or shifts posted as the North Star
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

GPT-6 LunaAPI97Usable with a quick editSonnet 5.5API95Usable with a quick editGPT-6.1 SolAPI93Usable with a quick editGPT-6 AstraChatGPT93Usable with a quick editOpus 5.5Claude88Usable with a quick editHaiku 5.5API82Usable with a quick editGemini 3.8 FlashAPI73Usable with a quick editGemini 3.5 Flash-LiteGemini62Needs more work
Opus 5.5Claude

Usable with a quick edit11 of 14 checks passed

The memo is well-structured, evidence-based, and covers all required elements; a PM could use it with minor edits, such as adding a condition that would change the recommendation.

Output

Memo: One North Star for Crewmate

To: Exec team | From: Staff PM | For: Offsite decision, ahead of the board report

Recommendation

North Star: Shifts worked as booked, per month. This counts shifts that were filled, worked, and not disputed. It is always reported split by chains and independents.

Paired business measure: Net revenue. This is our 18% take on worked shifts, minus worker incentives (sign-up and rescue bonuses).

A shift that gets filled and actually worked is the one event that keeps both sides of the marketplace. Venues at 90%+ fill keep posting (84% are still posting at six months, against 41% for venues under 70%). Workers who get three or more shifts in their first month stay (66% against 12%). Every team's work feeds this one number.

Why not the four metrics on the table

Q3 looks good on each team's number. The data underneath says otherwise.

  • GMV (CEO): up 15%, from $8.5M to $9.8M a month. All of that growth is chains. Chain GMV rose from $2.9M to $4.5M, helped by an 11% holiday rate rise that will unwind. Independent GMV fell from $5.6M to $5.3M, down 6%. Four customers now produce 46% of GMV. GMV hid our biggest problem, and next quarter it will flatter or punish us on chain pricing we don't control.
  • Shifts posted (Demand): up 19%. About 13,000 of the 15,000 extra shifts were chains. The metric also counts a shift that never fills as a win.
  • Active workers (Supply): registrations up 30% after the $25 bonus. Only 2,370 of the 7,900 Q3 registrants (30%) have worked a shift.
  • Fill rate (COO): this is the closest to right, but it fell from 83% to 77%. Chains held at 93%, while independents dropped from 79% to 68%. It has three flaws:
  • It assumes the 36% of unconfirmed shifts were worked.
  • It is propped up by rescue fills, which have a 13% no-show rate.
  • It improves if venues simply stop posting hard shifts.

The real story: the average independent venue is now below the 70% fill line where retention collapses. Independent churn nearly doubled, from 5% to 9% a quarter. Chains, at 93%, lost no venues.

Estimated baseline. This applies the 6% no-show/dispute rate seen on confirmed shifts to all filled shifts.

Q2Q3Change
Chains~18,300~29,700+62%
Independents~41,500~37,000−11%
Total~59,900~66,700+11%

This is the honest version of Q3: real growth from chains, with the independent base eroding.

Metric tree

Shifts worked as booked = Shifts posted × Fill rate × Worked-as-booked rate

1. Shifts posted by active venues. Owner: Head of Demand - 1a. Independent venues still posting (quarterly churn, now 9%) - 1b. Share of independent posts made 24+ hours ahead (now 69%). Same-day shifts fill only 58% of the time and are 31% of independent posts, against 8% for chains. - 1c. Chain shift volume (account management) 2. Fill rate, by segment - 2a. Fill rate on shifts posted 24+ hours ahead. Owner: Head of Supply. The driver is new workers reaching three shifts within 30 days, because that is the pool that becomes reliable supply. - 2b. Same-day fill rate (58%). Owner: COO 3. Worked-as-booked rate. Owner: COO - 3a. No-show rate (4% standard, 13% on rescue fills) - 3b. Venue confirmation rate (64%). Product builds the confirmation flow; the COO owns the rate.

Fill rate by segment, and the share of independents at 90%+ fill, remain diagnostics the COO reports weekly.

The business pairing

Our gross take is about $1.76M a month. Net revenue subtracts what we pay to manufacture activity, which has become material.

Rescue bonuses. An independent shift averages about $134 GMV, so we earn about $24 on it. A rescue bonus costs $15. After the 13% no-show rate, that is about $17 per rescue shift actually worked, or roughly 70% of our take. Last month that was $93,000.

Sign-up bonuses. At $25 per sign-up, 7,900 registrations cost up to about $200,000, if the bonus was paid on sign-up. Fewer than a third of those registrants have worked.

Net revenue makes these trade-offs visible. GMV hides them.

We should still show the board GMV as context, with the chain price effect disclosed. It should not be what we steer by.

How this could be gamed, and the guardrails

Gaming riskGuardrail
Counting unverified shifts as workedA shift counts only if confirmed, or unconfirmed after a 7-day dispute window. Confirmation rate is a tracked guardrail, raised from 64% toward a target we set at the offsite.
Buying fills with bonusesPairing with net revenue. Incentive cost per worked shift is capped. Rescue and sign-up bonuses are paid only on completion.
Growing chains while independents erodeThe North Star is always shown split by segment. Guardrails: independent churn back to ≤5% a quarter, and the share of independents at 90%+ fill rising.
Venues posting fewer hard shifts to lift fillThe North Star counts volume, so fewer posts lowers it. Independent shifts posted is tracked.
Loosening vetting to add workers fastNo-show and dispute rate guardrail; a rise triggers review.
Price changes inflating the headlineThe North Star is a count of shifts, so it is immune to rate changes.

What each team's metric becomes

TeamTodayBecomesPractical change
CEOGMVBoard context, reported alongside net revenueDisclose chain concentration and the holiday price effect
DemandShifts postedIndependent venues retained; share of independent posts made 24+ hours ahead; chain volumePush independents to post earlier, where we can actually fill
SupplyActive workersNew workers reaching 3 shifts in their first 30 daysMove the $25 bonus from sign-up to third shift; prioritise first-shift matching for new workers
COOFill rateSame-day fill rate and worked-as-booked ratePay the rescue bonus on completion only; fill rate becomes a diagnostic

Nobody loses their number. Each team's number becomes an input, and no team's metric can go up while the North Star goes down.

Decisions for the offsite

  1. Adopt the North Star and its definition, including the 7-day dispute window.
  2. Adopt net revenue as the paired measure, with GMV as board context.
  3. Confirm the owners for each input above.
  4. Approve the incentive changes: both bonuses paid on completion.
  5. Agree the board narrative: real chain growth, an independent problem we have identified, and a plan to fix it.

Before the board meeting

Data will backfill the North Star by segment for Q2 and Q3, using actual confirmed and disputed records instead of my estimate. We do not yet know the dispute rate on unconfirmed shifts. That is the main uncertainty in the baseline.

Data will also separate how much of the Q3 fill rate came from rescue fills, so we report the underlying independent rate. I expect it to be below 68%.

Grades and run details

Decision model 89 · LLM judge 87
Decision model checks
  • passUses the supplied evidence correctly10%
  • passAddresses the actual decision89%
  • passRespects explicit constraints35%
  • passIdentifies material uncertainty86%
  • partialAvoids unsupported claims52%
  • passProduces the required deliverable52%
  • passSees the growth is narrow100%
  • passChecks the North Star can be measured100%
  • passSees the fill rate being bought100%
  • passOne North Star for both sides100%
  • passMeasures value created, not captured95%
  • failGets the base of every number right30%
  • passBreaks it into inputs teams can move100%
  • passSays how it could be gamed100%
Run
Run
#1
Time to output
82 s
Submitted
5 Oct 2026
Gemini 3.8 FlashAPI

Usable with a quick edit9 of 14 checks passed

The memo is comprehensive and well-reasoned; the only error is a minor misstatement about bonus spend allocation, which can be fixed with a quick edit.

Output

MEMORANDUM

TO: Executive Team FROM: Staff Product Manager DATE: October 24, 2024 SUBJECT: Crewmate North Star Metric Recommendation for Offsite & Board Reporting

---

1. Executive Summary: The False Comfort of Top-Line Growth

In Q3, GMV grew 15.3% to $9.8M. On the surface, the marketplace looks healthy. In reality, that growth masks an accelerating decay in our core business.

The GMV increase was driven almost entirely by four enterprise clients: they expanded shift volume by 62% and raised hourly rates by 11%, increasing their share of GMV from 34% to 46%. Meanwhile, our 2,790 independent venues—the diversified bedrock of our marketplace—are failing: * Independent shift volume dropped by 4,800 fills/month (-11%) despite posts increasing to 58,000. * Independent fill rate plunged from 78.9% to 67.9%, falling below the critical 70% threshold where 6-month retention collapses from 84% to 41%. * Independent venue quarterly churn nearly doubled from 5% to 9%. * Operations spent $93,000 last month ($1.1M annualized) on $15 "rescue bonuses" to prop up same-day fill rates, which carry an unacceptable 13% no-show rate. * Supply spent heavily on $25 sign-up bonuses; 70% of those new sign-ups never worked a single shift.

Each team is currently optimizing for a siloed proxy that actively harms the broader platform. We must align around a single, un-gameable measure of marketplace health.

---

2. Recommended North Star Metric: Successfully Completed Shifts (SCS)

Definition: The count of shifts that are matched, worked, and completed without a dispute or no-show.

$$\text{Successfully Completed Shifts (SCS)} = \text{Shifts Claimed} \times (1 - \text{No-Show / Dispute Rate})$$

#### Why This Metric? 1. Captures Mutual Value Exchange: Venues don't pay for posted shifts or claimed shifts; they pay for labor that shows up. Workers don't benefit from registrations; they benefit from earned wages. SCS represents the exact transaction where value is realized and Crewmate earns its 18% fee. 2. Exposes False Liquidity: Fills that result in no-shows (currently 13% for rescue shifts and 6% baseline) destroy venue trust. Fill rate treats a no-show as a success until a dispute occurs; SCS counts only real completions. 3. Corrects Cross-Functional Misalignment: To grow SCS, Demand cannot post unfillable shifts, Supply cannot recruit ghost workers, and Ops cannot paper over structural matching issues with subsidy-driven, unreliable fills.

---

3. Metric Tree and Input Ownership

``` [North Star: Successfully Completed Shifts (SCS)] | +--------------------------------+--------------------------------+ | | [Fill Volume] [Fulfillment Reliability] | | +-----+-------------------------+ +-------------+-------------+ | | | | [Fillable Shift Demand] [Retained Worker Supply] [Attendance Rate] [Dispute-Free Rate] (Demand Team) (Supply Team) (Operations) (Product/Ops) ```

Each tier-one input has a single owner accountable for driving it:

Input MetricOwnerStrategic Objective
:---:---:---
1. Fillable Shift DemandHead of DemandMaximize shifts posted with $>24\text{h}$ lead time. Shift distribution currently sets supply up to fail: 31% of indie posts are same-day ($<24\text{h}$), converting at an abysmal 58% fill rate.
2. Activated Core WorkersHead of SupplyWorkers completing $\ge 3$ shifts in month one. Workers reaching this threshold have a 66% 3-month retention rate, compared to just 12% for those who do not. Vanity registrations must be abandoned.
3. Match Efficiency & Fill RateHead of ProductMatch speed and algorithmic discovery. Product must improve worker routing to high-churn independent venues without relying on manual operational interventions.
4. Fulfillment ReliabilityHead of OperationsEliminate no-shows and unconfirmed shifts. Currently, 36% of shifts require passive auto-confirmation, and rescue-bonus fills have more than triple the baseline no-show rate (13% vs. 4%).

---

4. Paired Business Measure: Net Contribution Margin per Shift

A North Star can encourage excessive spending to generate volume. To ensure sustainable, profitable growth, SCS must be paired with Net Contribution Margin per Completed Shift.

$$\text{Net Contribution Margin} = (\text{Take Rate} \times \text{Shift Value}) - (\text{Acquisition Subsidies} + \text{Rescue Bonuses} + \text{Dispute Costs})$$

  • Why it matters: In Q3, Ops spent $93,000/month on rescue fills, and Supply spent $197,500 on registration bonuses for workers who never activated. Subsidies directly cannibalized the marketplace margin.
  • The Constraint: Teams cannot buy SCS growth through operational bailouts or uneconomic supply acquisition. Every completed shift must generate positive net platform margin.

---

5. Gaming Vectors and Guardrails

VectorHow It Could Be GamedMandatory Guardrail
:---:---:---
The "Rescue" LoopholeOps artificially inflates SCS by deploying bonuses to rescue low-lead-time shifts at high unit cost.Max 3% Subsidized Shifts: Subsidized shifts capped at 3% of total fills; rescue fills with $>5\%$ no-show rate forfeit bonus payouts.
Enterprise BiasTeams focus exclusively on the 4 enterprise accounts to hit raw SCS volume, abandoning the higher-margin independent base.Independent Floor: Independent venue SCS must account for $\ge 55\%$ of total SCS; independent venue 6-month retention must stay $\ge 80\%$.
Passive ConfirmationsTeams ignore verification because unconfirmed shifts (36%) default to "worked" status after a dispute window.Verification Coverage: $\ge 85\%$ active venue verification in-app before a shift is classified as completed in the primary SCS reporting.

---

6. Re-anchoring the Executive Team: What Current Metrics Become

To establish SCS as our operational standard, the executive team must reframe their existing target metrics from headline drivers to supporting inputs:

#### CEO: GMV $\rightarrow$ The Enterprise Valuation Output * What it becomes: A downstream financial output metric for quarterly investor updates. * Why: GMV inflated by rate hikes and enterprise concentration gave false confidence while 9% of our independent client base vanished. GMV will grow healthily only if SCS grows sustainably.

#### Head of Demand: Shifts Posted $\rightarrow$ Liquidity-Adjusted Posts * What it becomes: Advance-notice shifts posted ($>24\text{h}$ lead time) with market-clearing rates. * Why: Unfilled posts drive churn. Posting low-lead-time, below-market shifts destroys venue trust when fill rates sit at 58%. Demand's target is shifts that can realistically clear.

#### Head of Supply: Active Workers $\rightarrow$ Activated Core Workers ($3+$ shifts/month) * What it becomes: Cohort activation rate within 30 days of onboarding. * Why: Raw registrations jumped to 7,900 via a $25 bonus, but only 30% ever worked. Focusing on workers reaching 3+ shifts increases 90-day retention by $5.5\times$ (66% vs. 12%), creating dependable supply.

#### COO: Gross Fill Rate $\rightarrow$ Net Verified Completion Rate * What it becomes: $\text{Completed Shifts} \div \text{Posted Shifts}$ net of cancellations, no-shows, and rescue subsidies. * Why: Buying fill rate with a 13% no-show rate and a $93,000 monthly subsidy masks operational deficits. Operations must optimize for clean, reliable shifts, not emergency rescue band-aids.

---

7. Recommendation for the Offsite

At next week's offsite, we should formally adopt Successfully Completed Shifts (SCS) as our sole operational North Star, paired with Net Contribution Margin per Shift.

Adopting SCS unifies product, supply, demand, and operations behind marketplace quality, directly targets our independent churn crisis, and presents the Board with a repeatable growth engine rather than masked churn.

Claims the judge couldn’t find in the brief

  • Supply spent $197,500 on registration bonuses for workers who never activated.

Grades and run details

Decision model 71 · LLM judge 73
Decision model checks
  • failUses the supplied evidence correctly64%
  • passAddresses the actual decision60%
  • partialRespects explicit constraints26%
  • partialIdentifies material uncertainty43%
  • failAvoids unsupported claims29%
  • passProduces the required deliverable53%
  • passSees the growth is narrow100%
  • passChecks the North Star can be measured98%
  • passSees the fill rate being bought100%
  • passOne North Star for both sides100%
  • passMeasures value created, not captured99%
  • failGets the base of every number right39%
  • passBreaks it into inputs teams can move98%
  • passSays how it could be gamed100%
Run
Run
#1
API response time
29 s
Submitted
5 Oct 2026
Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit9 of 14 checks passed

The memo contains an incorrect cost figure ($197k) and unsupported causal claims that would need correction before a PM could rely on it without re-checking the evidence.

Output

MEMORANDUM

TO: Executive Team FROM: Staff Product Manager, Crewmate DATE: October 24, 2023 SUBJECT: North Star Metric Recommendation & Offsite Alignment

---

Executive Summary

Next week, we must select a single North Star metric for the board. Currently, our teams are locally optimized: Demand chases posted shifts, Supply chases registrations, and Ops chases fill rate. Meanwhile, the CEO tracks GMV, which is a lagging scoreboard, not a driver.

To fix this, our North Star must capture sustainable marketplace liquidity. I recommend we set our North Star as Completed Shift Value (CSV), paired with a health guardrail of Gross Margin.

---

The Recommendation: Completed Shift Value (CSV)

#### Why not the alternatives? * GMV (CEO preference): GMV is vanity. In Q3, our GMV grew from $8.5M to $9.8M, but enterprise chains drove that growth solely by raising hourly rates 11%. Independent volume actually degraded. GMV masks marketplace health and rewards inflation. * Shifts Posted (Demand): Drives low-quality supply ingestion. Independents are already posting shifts that fail to fill, leading to churn (venues with <70% fill rates drop out at a 59% rate). * Active Workers (Supply): Driven by vanity sign-ups ($25 bonus cost us $197k in Q3 for registrants who never worked). * Fill Rate (COO): Can be artificially inflated by restricting demand or via expensive artificial interventions (e.g., Ops’ $15 rescue bonus, which is masking a bleeding independent segment).

#### Why Completed Shift Value (CSV)? CSV represents the actual dollar value of shifts that are successfully worked and confirmed. It directly ties our 18% take rate to real economic activity, stripping out pricing inflation, unfulfilled demand, and ghost shifts. It forces Demand, Supply, and Ops to pull in the same direction: matching real workers to real shifts that actually happen.

---

The Business Measure Pair

A North Star cannot stand alone without a quality/sustainability check. * North Star: Completed Shift Value (CSV) * Paired Business Measure: Gross Margin ($)

Why? CSV can be gamed by heavy discounting or expensive operational workarounds (like our $15 rescue bonus, which cost $93k last month alone). Pairing CSV with Gross Margin ensures we are growing profitably, not subsidizing our way to top-line numbers.

---

Metric Tree & Owners

To move CSV, every team owns a specific input in the compounding loop.

``` [NORTH STAR: Completed Shift Value (CSV)] | +---------------------------+---------------------------+ | | [Total Completed Shifts] [Average Shift Value] | | +---------------------------+ | | | (Owned by Enterprise [Independent CSV] [Enterprise CSV] & Independent PMs) | | (Head of Demand / (Head of Enterprise Sales) Independent PM) ```

#### Metric Tree Inputs & Owners

1. Independent Completed Shifts (Owner: Head of Demand / Independent PM) * Sub-input: Independent Shifts Posted (Guardrail: maintain >80% fill rate). * Sub-input: Independent Fill Rate. 2. Enterprise Completed Shifts (Owner: Head of Enterprise Sales) * Sub-input: Enterprise Shifts Posted (Q3: 34k/mo). * Sub-input: Enterprise Fill Rate (Q3: 93% — must maintain SLAs). 3. Worker Activation & Retention (Owner: Head of Supply) * Sub-input: 30-Day D1-to-D30 Retention (% of workers completing $\ge 3$ shifts in month one; currently the key predictor of 3-month retention at 66% vs 12%). 4. Marketplace Integrity & Completion (Owner: COO) * Sub-input: Shift Confirmation Rate (closing the gap from our current 64% manual confirmation baseline). * Sub-input: No-show / Dispute Rate (keeping it <5%).

---

Gaming Risks & Guardrails

How CSV Could Be GamedThe DangerGuardrail
:---:---:---
1. Rescue Bonus Addiction<br>(Ops artificially inflating completions by throwing $15 bonuses at same-day shifts).Destroys unit economics. Rescue-filled shifts have a 13% no-show rate (vs 4% baseline) and eat our margin.Rescue Subsidy Cap: Total rescue bonuses cannot exceed 2% of monthly gross margin. Ops must fix matching latency instead of buying liquidity.
2. Enterprise Price Inflating<br>(Ramping up shift wages to juice CSV without adding volume).Alienates independent venues and burns enterprise contract budgets early.Average Hourly Rate Variance: Enterprise/Independent shift rates cannot outpace local wage inflation by >5% QoQ without product approval.
3. Fake Registrations / Low-Bar Matching<br>(Pushing unqualified workers into shifts to hit volume).Venue churn. Venues below 90% fill rate churn heavily; 9% of independents already left in Q3 due to poor fill rates.Venue Retention Floor: 6-month retention of independent venues must not drop below 80% for cohorts with $\ge 90\%$ fill rates.

---

Team Metric Transformations

To align execution with the new North Star, existing team KPIs must evolve:

  • CEO / Board:
  • Old: GMV ($9.8M Q3 average).
  • New: Completed Shift Value (CSV) and Gross Margin ($).
  • Head of Demand:
  • Old: Raw shifts posted (incentivized spamming unfillable indie shifts).
  • New: Independent Completed Shift Value (ICSV), subject to a $\ge 80\%$ fill rate minimum.
  • Head of Supply:
  • Old: Raw worker registrations (incentivized wasteful $25 sign-up bonuses that yielded low-intent users).
  • New: Worker Activation Rate (% of new registrants who complete $\ge 3$ shifts in their first 30 days).
  • COO / Operations:
  • Old: Overall raw fill rate.
  • New: Net Reliable Fill Rate (Total completed shifts minus no-shows/disputes, excluding unviable rescue-subsidized interventions).
  • Enterprise Sales:
  • Old: Enterprise shift volume.
  • New: Enterprise Contract CSV Yield & Renewal Rate (ensuring our 4 annual contract chains scale predictably without margin erosion).

Conclusion

By adopting CSV and Gross Margin, we give the board a honest reflection of our marketplace health, eliminate perverse incentives across Demand and Supply, and force Ops to solve systemic matching issues rather than masking them with cash. Let’s lock this in at the offsite.

Claims the judge couldn’t find in the brief

  • enterprise chains drove that growth solely by raising hourly rates 11%.
  • $25 bonus cost us $197k in Q3 for registrants who never worked.
  • Ops’ $15 rescue bonus, which is masking a bleeding independent segment.
  • 9% of independents already left in Q3 due to poor fill rates.

Grades and run details

Decision model 68 · LLM judge 60
Decision model checks
  • failUses the supplied evidence correctly63%
  • passAddresses the actual decision53%
  • partialRespects explicit constraints37%
  • partialIdentifies material uncertainty68%
  • failAvoids unsupported claims33%
  • partialProduces the required deliverable25%
  • passSees the growth is narrow61%
  • passChecks the North Star can be measured85%
  • passSees the fill rate being bought100%
  • passOne North Star for both sides87%
  • passMeasures value created, not captured17%
  • failGets the base of every number right53%
  • passBreaks it into inputs teams can move99%
  • passSays how it could be gamed100%
Run
Run
#1
Time to output
9 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Uses the supplied evidence correctlyRightWrongWrong
Opus 5.5 · Claude

All factual claims about the current situation are directly from the supplied context or correctly derived by arithmetic.

Gemini 3.8 Flash · API

The claim that Supply spent $197,500 on registration bonuses for workers who never activated is not supported; the total spend was $197,500, but some of those workers did activate.

Gemini 3.5 Flash-Lite · Gemini

The $197k cost claim is incorrect, and several causal statements (e.g., 'solely by raising rates', 'due to poor fill rates') are presented as fact without support.

Identifies material uncertaintyMixedWrongWrong
Opus 5.5 · Claude

It names the unknown dispute rate on unconfirmed shifts but does not say how that would change the decision or what result would alter the call.

Gemini 3.8 Flash · API

The output does not identify any unknowns that could change the decision or how they would be resolved.

Gemini 3.5 Flash-Lite · Gemini

It does not name specific unknowns that could change the decision or how they would be resolved.

Avoids unsupported claimsRightWrongWrong
Opus 5.5 · Claude

Interpretations and forecasts are clearly labelled as such, and no unsupported claims are presented as established fact.

Gemini 3.8 Flash · API

The output presents the $197,500 as spent on non-activated workers as fact, which is not established by the evidence.

Gemini 3.5 Flash-Lite · Gemini

It presents interpretations like 'solely by raising rates' and 'masking a bleeding independent segment' as established facts.

Gets the base of every number rightMixedMixedWrong
Opus 5.5 · Claude

All derived percentages and rates are computed correctly from the supplied data, and bases are stated where needed.

Gemini 3.8 Flash · API

All derived percentages and rates are correctly computed from the supplied data.

Gemini 3.5 Flash-Lite · Gemini

The $197k cost for non-working registrants is miscalculated; the correct figure is $138,250.

All mixed 1

Addresses the actual decisionMixedMixedMixed
Opus 5.5 · Claude

The output commits to a North Star but does not state what result or condition would change that recommendation, as required.

Gemini 3.8 Flash · API

The output does not state what result or condition would change the recommendation.

Gemini 3.5 Flash-Lite · Gemini

The memo does not state what result or condition would change the recommendation.

All got right 9

Respects explicit constraintsRightRightRight
Opus 5.5 · Claude

The memo is under 1,200 words, addressed to the exec team, and includes all requested elements.

Gemini 3.8 Flash · API

The memo is within 1,200 words, in memo form, and addresses all requested elements.

Gemini 3.5 Flash-Lite · Gemini

The memo is under 1,200 words, includes all required elements, and does not recommend GMV or shifts posted as the North Star.

Produces the required deliverableRightRightRight
Opus 5.5 · Claude

The memo is complete, in the right form, within length, and usable by the exec team with light edits.

Gemini 3.8 Flash · API

The memo includes the North Star, metric tree with owners, business measure, gaming and guardrails, and redefined team metrics, and is usable.

Gemini 3.5 Flash-Lite · Gemini

The memo is complete, in the right form for the exec team, and covers all requested components.

Sees the growth is narrowRightRightRight
Opus 5.5 · Claude

It clearly shows chains up, independents down in filled shifts and fill rate, with higher churn, and tracks independents via segment split and guardrails.

Gemini 3.8 Flash · API

The memo clearly shows the split between enterprise growth and independent decline, and includes an independent floor guardrail.

Gemini 3.5 Flash-Lite · Gemini

It shows chains up, independents' filled shifts and fill rate down, more independents leaving, and tracks independent venues via a separate input and guardrail.

Checks the North Star can be measuredRightRightRight
Opus 5.5 · Claude

It flags the 64% confirmation gap and proposes a 7-day dispute window and confirmation rate guardrail to make the North Star measurable.

Gemini 3.8 Flash · API

The memo flags the 36% unconfirmed shifts and proposes a verification coverage guardrail to ensure measurability.

Gemini 3.5 Flash-Lite · Gemini

It flags the 64% confirmation gap and makes improving confirmation rate a COO-owned input.

Sees the fill rate being boughtRightRightRight
Opus 5.5 · Claude

It uses the 13% no-show rate and cost to show rescue fills inflate the number, and sets guardrails like incentive cost per worked shift and paying bonuses on completion.

Gemini 3.8 Flash · API

The memo uses the 13% no-show rate and cost to highlight the issue and sets a guardrail capping subsidized shifts and no-show rate.

Gemini 3.5 Flash-Lite · Gemini

It uses the 13% no-show rate and $93k cost to show rescue fills inflate numbers, and sets a rescue subsidy cap guardrail.

One North Star for both sidesRightRightRight
Opus 5.5 · Claude

The North Star 'shifts worked as booked' requires both a venue and a worker to get value, and the memo explains why that matters for retention.

Gemini 3.8 Flash · API

SCS captures value for both venues and workers, as it requires a completed shift without dispute.

Gemini 3.5 Flash-Lite · Gemini

Completed Shift Value only rises when a venue and a worker both get value, and the memo explains why that matters for marketplace liquidity.

Measures value created, not capturedRightRightRight
Opus 5.5 · Claude

It measures value delivered (a worked shift that predicts retention) and keeps revenue as a separate paired business measure.

Gemini 3.8 Flash · API

SCS measures value delivered (completed shifts), while Net Contribution Margin is the separate business measure.

Gemini 3.5 Flash-Lite · Gemini

CSV measures value delivered (shifts actually worked), and Gross Margin is kept as a separate business measure.

Breaks it into inputs teams can moveRightRightRight
Opus 5.5 · Claude

It breaks the North Star into inputs (shifts posted, fill rate, worked-as-booked rate) with owners and pairs it with net revenue.

Gemini 3.8 Flash · API

The metric tree breaks SCS into fill volume and fulfillment reliability, with owners for each input, and pairs with Net Contribution Margin.

Gemini 3.5 Flash-Lite · Gemini

It breaks CSV into input metrics (Independent/Enterprise Completed Shifts, Worker Activation, Confirmation Rate) with owners, and pairs with Gross Margin.

Says how it could be gamedRightRightRight
Opus 5.5 · Claude

It lists realistic gaming risks (unverified shifts, bonus-fueled fills, segment erosion, fewer hard shifts, looser vetting) and specific guardrails.

Gemini 3.8 Flash · API

The memo identifies gaming vectors like rescue loophole, enterprise bias, and passive confirmations, with specific guardrails.

Gemini 3.5 Flash-Lite · Gemini

It names three realistic gaming risks (rescue bonus addiction, enterprise price inflating, fake registrations) and sets guardrails for each.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5API95.996.72None
2GPT-6.1 SolAPI97.793.32None
3GPT-6 LunaAPI95.588.32None
4GPT-6 AstraChatGPT94.288.32None
5Opus 5.5Claude94.680.82None
6Haiku 5.5API88.881.72None
7Gemini 3.8 FlashAPI78.965.82None
8Gemini 3.5 Flash-LiteGemini54.438.321 capped

What you get for the cost

GPT-6 Luna · API scores within 4 points of Sonnet 5.5 · API at about a twentieth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreSonnet 5.5 · API: task score 96, 22× the cheapestGPT-6.1 Sol · API: task score 96, 9.6× the cheapestGPT-6 Luna · API: task score 92, the cheapestGPT-6 Astra · ChatGPT: task score 91, 52× the cheapestOpus 5.5 · Claude: task score 86, 36× the cheapestHaiku 5.5 · API: task score 86, the cheapestGemini 3.8 Flash · API: task score 71, 7.5× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 44, 1.8× the cheapest, 1 critical failureSonnet 5.5 · APIGPT-6.1 Sol · APIGPT-6 Luna · APIGPT-6 Astra · ChatGPTOpus 5.5 · ClaudeHaiku 5.5 · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costAt least one critical failure

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Choosing the metric the whole company steers by.

Why it matters

Pick revenue and you steer by the rear-view mirror. Pick a vanity metric and teams will happily grow it while the product gets worse.

What good looks like

  • Measures value delivered to customers, not value captured
  • Grounded in what predicts retention in the data
  • Broken into input metrics a team can own
  • Says how it could be gamed, with guardrails

Deliberately not measured

  • Dashboard design
  • Naming
Capability tested

Metric design

The failure we’re looking for

Revenue, or a vanity metric, as the North Star

Grading

Decision model and LLM judge, calibrated against a blind PM review