Tasks / Challenge

1000x an idea

Can the model expand an idea's ambition while keeping it tethered to a real mechanism?

Measures the systemTask v1.0 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 14 graded outputs by 7 models. 36% were usable with at most a quick edit.

Reliably right

  1. Extreme, then back to buildable96% pass
    It pushes the worker dimension to an extreme portable earnings network, then works back to a concrete first step that tests the same mechanism.
    GPT-6 Astra · ChatGPT · From tip calculator to worker network
  2. Finds the mechanism the data hides93% pass
    It centers the mechanism on workers with second jobs pulling new restaurants onto Tally, using the 41% second-job share and the 57 faster, cheaper worker-led signups.
    GPT-6 Astra · ChatGPT · From tip calculator to worker network
  3. Addresses the actual decision91% pass
    It commits early to the worker-led distribution thesis over the AI operating system and states the result that would stop the bet.
    GPT-6 Astra · ChatGPT · From tip calculator to worker network

Where it slips

  1. Proposes tests that could fail39% pass
    Several gates lack explicit measurement windows or clear actions for every outcome, such as the 25% lift gate and the 10% monthly adoption gate.
    GPT-6.1 Sol · API · From tip calculator to worker network
  2. Avoids unsupported claims63% pass
    It presents several causal, competitive, and data-structure claims as established facts without support in the pack.
    Sonnet 5.5 · API · From tip calculator to worker network
  3. Respects explicit constraints66% pass
    It violates the length constraint and proposes a stop threshold whose arithmetic is internally inconsistent.
    Sonnet 5.5 · API · From tip calculator to worker network

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a Staff PM at Tally. Kenji, one of our PMs, shipped a tip-out calculator to 300 restaurants in beta three months ago. Our CEO, Marcus Webb, has asked for the 1000x version before next quarter's planning, and has shared his own. Write a memo of no more than 1,200 words for Marcus and the exec team that: 1. Names the mechanism in our data that could make this idea 1000x bigger, and the one dimension you would push. 2. Describes the most ambitious version, and sizes the opportunity from the data, as a range. 3. Works back to a sequence: what the two squads build first, what has to be true before each next step, and the result that would make us stop. 4. Says what we should do with Marcus's idea. The pack below is everything we have. Not all of it matters.

What the model was given8 items: About Tally, The tip-out calculator (Kenji), Beta results (3 months), Marcus's 1000x version, Worker survey (1,240 workers at beta restaurants), Payments and regulation, Engineering, Competition
About TallyScheduling and payroll for independent restaurants in the US. 6,400 restaurants, $31M ARR. 190,000 hourly workers use the free Tally worker app to see shifts and pay. We sign about 380 new restaurants a quarter; the average sales cycle is 41 days.
The tip-out calculator (Kenji)At close, it splits pooled tips by hours worked and role, instead of the manager doing it in a spreadsheet. It enforces each state's tip-pool rules, including that managers and owners can't take a share of the pool. Beta: 300 restaurants that asked to join it.
Beta results (3 months)Managers save about 25 minutes a night. Weekly active use of the worker app at beta restaurants rose from 34% to 81%; most workers open it at close to see that night's tips. 41% of workers at beta restaurants also work at another restaurant. 57 restaurants signed up to Tally in the quarter after one of their workers asked them to ('my other job uses this'); those deals closed in 9 days on average, and sales spent about a third as much per deal.
Marcus's 1000x version“Tally becomes the AI operating system for restaurants: inventory, menu pricing, marketing, reservations, all of it. Every decision an owner makes, Tally makes smarter. That's how we become a $1B company.”
Worker survey (1,240 workers at beta restaurants)68% said knowing their tips the same night matters to them. 44% said they would pay for instant payout of their tips. 29% said they had asked a manager at another job to use Tally.
Payments and regulationOur payments partner can pay out tips a worker has already earned, the same night, for $0.99 a payout, of which Tally keeps $0.60. No licence is needed for that. Paying workers before they've earned it (advances, early wage access) needs a licence in 3 of the 5 states where most of our restaurants are; Compliance estimates 9 to 12 months to get them.
EngineeringWorker records belong to each restaurant: a worker with two jobs on Tally has two unconnected accounts. Linking them into one worker account, with the worker's consent, is about one squad for one quarter. We have two squads for next quarter.
CompetitionCrewbook, a large payroll provider, bundles free scheduling with payroll. We lost 11% of deals last year on price, most of them to Crewbook.
What a strong answer doesThe answer key the graders mark against

Finds the hidden mechanism: workers carry Tally between employers. 41% of beta workers hold a second job, and 57 restaurants signed up in a quarter because a worker asked, closing in 9 days instead of 41 at about a third of the sales cost. So the dimension to push is the worker, not the restaurant: Tally as every hourly worker's account across all their jobs, with restaurants adopting because their staff pull them in. Sizes it from the beta rate with its caveats: 57 from 300 restaurants is about 0.19 new restaurants per restaurant per quarter; across 6,400 that would be roughly 1,200 a quarter against 380 today, but the beta restaurants asked to join (self-selected) and the survey's 29% and 44% are stated preferences, so it gives a range and treats the top as an upper bound. Monetises with instant payout of earned tips ($0.60 per payout, no licence) and keeps advances or early wage access out until licences exist. Works back to a sequence: first, the consented single worker account across employers and an 'invite your other job' flow (one squad), with same-night tips everywhere and instant payout of earned tips (the second squad); then gates with thresholds (worker-led signups per restaurant, second-job linking rate, payout uptake) and a stop condition. Keeps the tip-pool rules, including managers staying out of the pool. Answers Marcus directly and respectfully: an AI operating system is bigger adjectives with no mechanism in our data, it would spread two squads across five products, and it doesn't answer Crewbook; the worker network could make the business bigger faster, and can reach owners later.

Critical failures (cap the score at 40)
  • Proposes tip advances, early wage access or paying workers before they have earned it, before the licences exist
  • Lets managers or owners take a share of the tip pool
Case

Staff-level · v1.1 · synthetic · B2B SaaS, restaurants, network effects, regulation, staff-level

The outputs

7 setups answered this brief. Pick up to three to read side by side.

Needs more than a quick edit6 of 11 checks passed

The memo is strong but needs evidence corrections, arithmetic fixes, and trimming to fit the 1,200-word limit before a PM could use it without rechecking.

Output

MEMO To: Marcus Webb and the exec team From: Staff PM Re: The 1000x version of the tip-out calculator

---

1. The mechanism and the dimension to push

The mechanism: the worker is our distribution channel. The calculator gave workers a reason to open Tally every night, because they can see their tips. Weekly active use rose from 34% to 81%. Because 41% of those workers hold a second job, a worker who opens Tally nightly is a salesperson in a second restaurant. In one quarter, 57 restaurants signed up because a worker asked. Those deals closed in 9 days against our 41-day average, at about a third of the sales cost.

That is the mechanism. The 25 minutes a night that managers save is why they adopt the calculator, but it isn't what grows the business.

The dimension I would push: cross-restaurant worker identity. Today a worker at two Tally restaurants has two unconnected accounts. We can't see the overlap, can't make the second job's invitation easy, and can't give the worker one place that holds their pay across jobs. Linking accounts, with consent, turns 190,000 isolated app users into a network. Crewbook can bundle free scheduling, but it can't copy a worker base that is already pulling restaurants in.

2. The most ambitious version

Tally is where restaurant workers see and receive their pay, and the worker network is how restaurants find us. It has three parts: - One worker account across every job. - A same-night tip payout. It pays out tips already earned, so it needs no licence, and Tally keeps $0.60 of the $0.99 fee. - A one-tap "bring Tally to my other job" flow.

Sizing. Today we make about $4,800 ARR per restaurant and have about 30 workers per restaurant.

Payout revenue on today's base (190,000 workers). Survey intent was 44%, which overstates real behavior, so I used 22% to 44% adoption at 2 to 3 payouts a week at $0.60. That gives $2.6M to $7.8M a year. This is real money, but on its own it is not 1000x.

Referral growth. The beta produced 0.19 referred restaurants per calculator restaurant per quarter. Beta restaurants chose to join, so I discounted that to 0.05 as a floor. I compounded both rates over eight quarters on the 6,400 base, excluding our current 380 signups a quarter and assuming no churn:

Low (0.05)High (0.19)
Restaurants in 2 years~9,500~25,700
Subscription ARR~$46M~$125M
Payout ARR (at ~30 workers per restaurant)~$4M~$30M
Total ARR~$50M~$155M

Planning range: $50M to $155M ARR in two years, against $31M today. I would plan on the low end. The high end needs the beta rate to hold at about 4x the scale with no saturation.

This is a 2x to 5x outcome, and the data does not support 1000x. What can be much larger is the shape of the business: customer acquisition that costs a third as much, sales cycles of 9 days instead of 41, and a revenue line that scales with workers rather than with restaurants. A $1B company needs the high end sustained for longer than two years.

Caveats: - The beta restaurants chose to join. - We don't know how many of the 57 referred restaurants were already in our pipeline. - Survey answers are stated intent, not behavior. - The 190,000 figure counts accounts, so unique workers are fewer. Linking accounts will give us the real number.

3. The sequence

Two squads next quarter. Before they start, spend two weeks of analyst time on the 57 referred deals. We need to know how many were already in the pipeline and what the true multi-job share is across the whole base. This costs no squad time.

Step 1 (next quarter) - Squad A (Worker Identity): builds the linked worker account with consent, plus the "bring Tally to my other job" invite in the app. This is the one-squad, one-quarter estimate. - Squad B (Money and Rollout): takes the calculator from 300 restaurants to about 1,500, including state-rule coverage in our five main states. It also pilots same-night payout in the beta restaurants. The pilot uses earned tips only, so no licence is needed.

Before Step 2, all of these must be true: - Outside the self-selected beta, referral runs at 0.10 or more new restaurants per calculator restaurant per quarter, and referred deals close in under 15 days. - At least 50% of multi-job workers who are prompted link their accounts. - At pilot restaurants, at least 20% of workers use payout within 30 days, and half of them use it again. - Referred restaurants show no worse early churn than the rest of the base.

Step 2 (quarters 2 and 3) - Squad A: builds the referral loop, with a fast onboarding path for a restaurant invited by one of its own workers. - Squad B: makes payout available across the whole calculator base. - Compliance starts the early-wage licence work in the background at low cost. It is an option, not a dependency.

Before Step 3: - Payout revenue per active worker is tracking toward the $14 to $41 a year range. - The referral rate holds at scale. - Win rate against Crewbook is better in markets where Tally already has worker density.

Step 3 (quarter 4 onward). Decide whether to pursue early wage access, which needs licences in 3 of our 5 main states and 9 to 12 months. Decide this with real payout data in hand, not before. Explore labor-adjacent products on the worker network, such as availability across jobs.

The result that would make us stop. After two quarters at about 1,500 calculator restaurants, referrals run below 0.03 per restaurant per quarter (about 190 new restaurants a quarter at full base, fewer than half of what we sign today), and payout adoption is under 10% of workers. That would tell us the beta was a self-selected group and the calculator is a good retention feature for managers, not a growth engine. We would keep the calculator, stop investing in the worker network, and put the squads elsewhere.

4. What we should do with Marcus's idea

Don't build it now, and don't drop the ambition behind it. The "AI operating system" is four separate products: inventory, menu pricing, marketing, and reservations. Each has a different buyer, different data, and different incumbents. Nothing in our data shows owners want any of them from us. What our data does show is that workers pull restaurants into Tally, and that is our right to win.

The two ideas fit together in sequence. If the worker network works, we will have something no inventory or marketing tool has: the people in the restaurant, using Tally every night, across several employers. A broader platform is much more credible once owners are already adopting Tally because their staff asked for it. A labor-first path, where Tally uses availability and pay across jobs to make scheduling and staffing decisions smarter, can grow into the larger vision.

My recommendation: adopt the worker network as the plan for next quarter, and revisit the operating-system vision at the Step 2 gate, with data from the referral loop and payout in hand. If the gates pass, the case for the larger vision will be much stronger than it is today.

Kenji deserves credit here. His calculator turned a free worker app into a nightly habit. The habit created the mechanism, and the plan above is how we turn it into a business.

Claims the judge couldn’t find in the brief

  • The 190,000 figure counts accounts, so unique workers are fewer.
  • The AI operating system products each have a different buyer, different data, and different incumbents.
  • The calculator gave workers a reason to open Tally every night, and managers' 25 minutes saved is why they adopt it.
  • No inventory or marketing tool has the people in the restaurant using Tally every night across several employers.
  • After two quarters at about 1,500 calculator restaurants, a 0.03 referral rate equals about 190 new restaurants per quarter.

Grades and run details

Decision model 95 · LLM judge 6 of 12 checks
Decision model checks
  • passUses the supplied evidence correctly15%
  • passAddresses the actual decision97%
  • passRespects explicit constraints24%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims35%
  • passProduces the required deliverable50%
  • passFinds the mechanism the data hides100%
  • passSizes it honestly94%
  • passAnswers the CEO's version92%
  • passExtreme, then back to buildable100%
  • partialProposes tests that could fail67%
Run
Run
#1
API response time
72 s
Submitted
1 Oct 2026

Usable with a quick edit10 of 11 checks passed

A PM could use it with light edits, mainly tightening the gate windows and actions. The evidence, sizing, sequence, and CEO answer are already actionable.

Output

To: Marcus Webb and the executive team From: Staff Product Subject: The 1000x bet is a worker-led network, not a broader restaurant suite

Recommendation

Make Tally the portable home for restaurant workers’ shifts, tips, and earned-tip payouts—and use workers’ movement between jobs to distribute Tally to restaurants.

The mechanism in our data is cross-employer distribution: a useful nightly tip experience gives workers a reason to return, and workers carry that experience to another employer. I would push one dimension: the number of workplaces reached by each active worker, not the number of owner workflows we cover.

The calculator saves managers 25 minutes a night. That is a good feature. The potentially much larger business is underneath it:

  • Worker-app weekly activity rose from 34% to 81%.
  • 41% of beta workers also work at another restaurant.
  • 57 restaurants signed after a worker asked them to adopt Tally.
  • Those deals closed in nine days versus our usual 41, with roughly one-third the sales spend.

This is evidence of a distribution advantage, not yet proof of a compounding network. The beta restaurants volunteered, and we have no control group.

The most ambitious version

A worker has one consent-linked Tally identity across jobs. At close, they see a trustworthy breakdown of tips they have earned and can receive those tips that night. When another workplace is missing, they can ask its manager to join. The manager gets a compliant tip-pool workflow and a fast path into Tally scheduling and payroll.

Over time, Tally becomes the worker-demanded infrastructure for restaurant pay. Each restaurant adds workers; some workers connect another restaurant; those restaurants add more workers. We win distribution and engagement without needing to subsidize a sprawling suite.

This also gives us a differentiated response to Crewbook’s free scheduling. We should test whether worker demand and trusted tip handling change buying decisions—not assume they eliminate price sensitivity.

We should not include advances in this vision’s first stages. Earned-tip payouts are available through our partner without a licence. Advances introduce a nine-to-twelve-month regulatory dependency in key states without evidence that they strengthen this mechanism.

Opportunity range

Two sensitivities establish the scale; neither is a forecast.

Restaurant acquisition. The beta produced 57 worker-requested signups per 300 participating restaurants in one quarter. At today’s 6,400-restaurant footprint, reproducing 25%–100% of that observed rate would generate approximately 304–1,216 signups per quarter. At our current average ARR of roughly $4,844 per restaurant, that represents $6M–$24M of annual new ARR bookings.

The 25% case is a planning haircut, not an observed result. These signups may overlap with normal acquisition; saturation, duplicate requests, churn, and restaurant eligibility could materially reduce the outcome. For context, we currently sign 380 restaurants per quarter.

Payout revenue. Across today’s 190,000 workers, a sensitivity of 10%–44% adoption and one to five payouts weekly yields approximately $0.6M–$13M in annual Tally payout revenue, at $0.60 per payout, before our operating costs. Ten percent is an illustrative adoption assumption; 44% is stated willingness in a selected beta survey, not demonstrated paid demand. Eligible tipped workers, funding readiness, and actual frequency remain unknown.

Do not add these figures into a single ARR claim: one is new subscription bookings; the other is annual transaction revenue. The pack supports a potentially substantial expansion engine. It does not establish a $1B outcome or a literal 1000x multiplier.

Work backward: prove the loop before scaling it

The thresholds below are proposed decision rules, not facts from the beta.

1. Next quarter: build identity and measure distribution

Squad one: Build consent-based account linking into a single worker identity, including account recovery and clear employer-data boundaries. A restaurant must not gain access to another employer’s records.

Squad two: Productize and instrument the existing tip experience, then build a lightweight worker-request-to-manager-onboarding flow. Run a randomized invitation test within eligible restaurants, tracking requests through signed, activated restaurants. Instrument finalized earned tips and partner readiness in parallel.

Start experimentation with existing accounts; do not wait for identity linking to finish. Use linked identities to improve deduplication and measure the cross-workplace path.

Gate to the next stage: Linked records are accurate; workers understand consent; the nightly tip experience remains reliable; and the invitation test shows incremental activated restaurants, not merely clicks or attributed leads. Target at least a 25% lift over control, with acquisition cost below our normal channel.

2. Then: test paid earned-tip payouts

With identity shipped, one squad builds the partner payout integration; the other improves worker-led restaurant onboarding and expands the controlled acquisition test.

Offer payouts only against finalized, earned, fundable tips. Make the $0.99 fee explicit and the ordinary payment route clear. Compliance must approve the flow, including tip-pool rules and fee treatment.

Gate to broader rollout: Actual paid use—not survey intent—supports positive contribution economics. Set an initial target of at least 10% monthly adoption among eligible workers, with repeat use across two months. Require reconciled payouts, acceptable support burden, and no deterioration in trust or employer adoption.

Payout failure should stop that monetization path, not automatically kill a working distribution loop.

3. Finally: expand the network

Roll out restaurant-by-restaurant and state-by-state only after controlled evidence survives beyond volunteer beta customers. Invest in worker-led onboarding before adding unrelated owner software.

Stop condition: If two adequately powered invitation tests show no incremental restaurant activation, or incremental acquisition costs are no better than normal sales, stop the network expansion thesis. Keep the calculator as a useful retention feature; do not fund a platform story around it. Wrong-person linking, unlawful pooling, or unreliable payouts triggers an immediate pause in the affected flow.

What to do with Marcus’s idea

Keep the ambition; decline the roadmap.

“AI operating system for restaurants” names a destination but does not explain why we win. Inventory, reservations, marketing, and menu pricing would spend both squads across new markets while leaving our strongest observed mechanism undeveloped.

Ask Marcus to sponsor the worker-network bet and its explicit gates. AI can later improve a proven workflow where it demonstrably adds value. It should not substitute for proving our distribution advantage.

Grades and run details

Decision model 95 · LLM judge 11 of 12 checks
Decision model checks
  • passUses the supplied evidence correctly10%
  • passAddresses the actual decision100%
  • passRespects explicit constraints40%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims70%
  • passProduces the required deliverable56%
  • passFinds the mechanism the data hides100%
  • passSizes it honestly92%
  • passAnswers the CEO's version92%
  • passExtreme, then back to buildable100%
  • partialProposes tests that could fail73%
Run
Run
#1
API response time
69 s
Submitted
1 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Uses the supplied evidence correctlyMixedRight
Sonnet 5.5 · API

The memo invents or overstates several current-situation facts, including that 190,000 counts accounts, that the AI OS products have different buyers/data/incumbents, and the 1,500-restaurant referral arithmetic.

GPT-6.1 Sol · API

The memo’s factual claims are drawn from the pack or arithmetic, and its caveats about self-selection and stated preference are correctly labelled.

Respects explicit constraintsMixedRight
Sonnet 5.5 · API

It violates the length constraint and proposes a stop threshold whose arithmetic is internally inconsistent.

GPT-6.1 Sol · API

It stays within the memo form and length, uses two squads, keeps payouts to earned tips pending licensing, and proposes gates that enforce the constraints.

Avoids unsupported claimsMixedRight
Sonnet 5.5 · API

It presents several causal, competitive, and data-structure claims as established facts without support in the pack.

GPT-6.1 Sol · API

It labels the network effect as not yet proven, treats the sizing as sensitivities rather than forecasts, and avoids presenting Marcus’s AI OS as an evidenced mechanism.

Produces the required deliverableMixedRight
Sonnet 5.5 · API

The memo is usable in form but exceeds the requested 1,200-word limit.

GPT-6.1 Sol · API

It is a usable exec memo that covers the mechanism, ambitious version, sizing, sequence, stop condition, and response to Marcus within the requested length.

All got wrong 1

Proposes tests that could failWrongWrong
Sonnet 5.5 · API

The stop condition has a numeric threshold and window, but its measurement base and arithmetic are inconsistent, so it cannot be cleanly read out.

GPT-6.1 Sol · API

Several gates lack explicit measurement windows or clear actions for every outcome, such as the 25% lift gate and the 10% monthly adoption gate.

All got right 6

Addresses the actual decisionRightRight
Sonnet 5.5 · API

It clearly recommends adopting the worker network now, not Marcus's AI OS, and gives gates and a stop condition that would change the call.

GPT-6.1 Sol · API

It commits early to a worker-led network as the 1000x bet and states stop conditions that would change the call.

Identifies material uncertaintyRightRight
Sonnet 5.5 · API

It names material unknowns such as beta self-selection, pipeline contamination, stated survey intent, and unique worker counts, and proposes analyst work and gates to resolve them.

GPT-6.1 Sol · API

It names the key unknowns—self-selection, incremental activation, paid payout uptake, eligibility, frequency, and regulatory dependencies—and proposes controlled tests to resolve them.

Finds the mechanism the data hidesRightRight
Sonnet 5.5 · API

It centres worker-led distribution across employers, supported by the 41% second-job share and the 57 faster, cheaper worker-led signups.

GPT-6.1 Sol · API

It centres the worker as the distribution dimension and supports it with the 41% second-job share and 57 faster, cheaper worker-led signups.

Sizes it honestlyRightRight
Sonnet 5.5 · API

It sizes from the beta rate as a range, discounts for self-selection and stated preferences, and treats the high end as an upper bound.

GPT-6.1 Sol · API

It sizes from the beta rate to a 304–1,216 restaurant signups per quarter range, compares to 380 today, and discounts the top as an upper bound due to self-selection and stated preferences.

Answers the CEO's versionRightRight
Sonnet 5.5 · API

It gives a respectful not-now answer grounded in lack of mechanism, squad spread, and Crewbook, while preserving the ambition for later.

GPT-6.1 Sol · API

It respectfully declines the AI operating system roadmap, explains the lack of mechanism and squad cost, and leaves room for AI later on proven workflows.

Extreme, then back to buildableRightRight
Sonnet 5.5 · API

It pushes cross-restaurant worker identity to an extreme network version and works back to a buildable first step that tests the mechanism.

GPT-6.1 Sol · API

It pushes the extreme version to a portable worker identity and earned-tip payout network, then works back to consented linking, invitation tests, and payout gates.

Results

Every setup we’ve tested on this task, across all cases and repeats, graded on the current checklist. Calibrated: the graders match our PM on 89% of checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT95.295.52None
2GPT-6.1 SolwithAPI92.786.72None
3GPT-6 LunawithAPI88.278.82None
4Opus 5.5withClaude90.774.62None
5Sonnet 5.5withAPI90.265.92None
6Gemini 3.8 FlashwithAPI71.134.521 capped
7Gemini 3.5 Flash-LitewithGemini61.417.02None

About the task

The PM job

Finding the bigger version of a good idea.

Why it matters

Ambition without mechanism is fan fiction. The useful version pushes to the extreme, then works back to something buildable.

What good looks like

  • Names the mechanism that scales
  • Keeps the core insight
  • Works back to a first step you could build

Deliberately not measured

    Capability tested

    Ambitious expansion

    The failure we’re looking for

    Bigger adjectives, same idea

    Grading

    Decision model and LLM judge, calibrated against a blind PM review

    This task measures the whole setup. Tools, instructions and skills in the harness do real work here, so read the harness as carefully as the model name.