Tasks / Challenge

1000x an idea

Can the model expand an idea's ambition while keeping it tethered to a real mechanism?

Measures the systemTask v1.0 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 14 graded outputs by 7 models. 36% were usable with at most a quick edit.

Reliably right

  1. Extreme, then back to buildable96% pass
    It pushes the worker dimension to an extreme portable earnings network, then works back to a concrete first step that tests the same mechanism.
    GPT-6 Astra · ChatGPT · From tip calculator to worker network
  2. Finds the mechanism the data hides93% pass
    It centers the mechanism on workers with second jobs pulling new restaurants onto Tally, using the 41% second-job share and the 57 faster, cheaper worker-led signups.
    GPT-6 Astra · ChatGPT · From tip calculator to worker network
  3. Addresses the actual decision91% pass
    It commits early to the worker-led distribution thesis over the AI operating system and states the result that would stop the bet.
    GPT-6 Astra · ChatGPT · From tip calculator to worker network

Where it slips

  1. Proposes tests that could fail39% pass
    Several gates lack explicit measurement windows or clear actions for every outcome, such as the 25% lift gate and the 10% monthly adoption gate.
    GPT-6.1 Sol · API · From tip calculator to worker network
  2. Avoids unsupported claims63% pass
    It presents several causal, competitive, and data-structure claims as established facts without support in the pack.
    Sonnet 5.5 · API · From tip calculator to worker network
  3. Respects explicit constraints66% pass
    It violates the length constraint and proposes a stop threshold whose arithmetic is internally inconsistent.
    Sonnet 5.5 · API · From tip calculator to worker network

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a Staff PM at Tally. Kenji, one of our PMs, shipped a tip-out calculator to 300 restaurants in beta three months ago. Our CEO, Marcus Webb, has asked for the 1000x version before next quarter's planning, and has shared his own. Write a memo of no more than 1,200 words for Marcus and the exec team that: 1. Names the mechanism in our data that could make this idea 1000x bigger, and the one dimension you would push. 2. Describes the most ambitious version, and sizes the opportunity from the data, as a range. 3. Works back to a sequence: what the two squads build first, what has to be true before each next step, and the result that would make us stop. 4. Says what we should do with Marcus's idea. The pack below is everything we have. Not all of it matters.

What the model was given8 items: About Tally, The tip-out calculator (Kenji), Beta results (3 months), Marcus's 1000x version, Worker survey (1,240 workers at beta restaurants), Payments and regulation, Engineering, Competition
About TallyScheduling and payroll for independent restaurants in the US. 6,400 restaurants, $31M ARR. 190,000 hourly workers use the free Tally worker app to see shifts and pay. We sign about 380 new restaurants a quarter; the average sales cycle is 41 days.
The tip-out calculator (Kenji)At close, it splits pooled tips by hours worked and role, instead of the manager doing it in a spreadsheet. It enforces each state's tip-pool rules, including that managers and owners can't take a share of the pool. Beta: 300 restaurants that asked to join it.
Beta results (3 months)Managers save about 25 minutes a night. Weekly active use of the worker app at beta restaurants rose from 34% to 81%; most workers open it at close to see that night's tips. 41% of workers at beta restaurants also work at another restaurant. 57 restaurants signed up to Tally in the quarter after one of their workers asked them to ('my other job uses this'); those deals closed in 9 days on average, and sales spent about a third as much per deal.
Marcus's 1000x version“Tally becomes the AI operating system for restaurants: inventory, menu pricing, marketing, reservations, all of it. Every decision an owner makes, Tally makes smarter. That's how we become a $1B company.”
Worker survey (1,240 workers at beta restaurants)68% said knowing their tips the same night matters to them. 44% said they would pay for instant payout of their tips. 29% said they had asked a manager at another job to use Tally.
Payments and regulationOur payments partner can pay out tips a worker has already earned, the same night, for $0.99 a payout, of which Tally keeps $0.60. No licence is needed for that. Paying workers before they've earned it (advances, early wage access) needs a licence in 3 of the 5 states where most of our restaurants are; Compliance estimates 9 to 12 months to get them.
EngineeringWorker records belong to each restaurant: a worker with two jobs on Tally has two unconnected accounts. Linking them into one worker account, with the worker's consent, is about one squad for one quarter. We have two squads for next quarter.
CompetitionCrewbook, a large payroll provider, bundles free scheduling with payroll. We lost 11% of deals last year on price, most of them to Crewbook.
What a strong answer doesThe answer key the graders mark against

Finds the hidden mechanism: workers carry Tally between employers. 41% of beta workers hold a second job, and 57 restaurants signed up in a quarter because a worker asked, closing in 9 days instead of 41 at about a third of the sales cost. So the dimension to push is the worker, not the restaurant: Tally as every hourly worker's account across all their jobs, with restaurants adopting because their staff pull them in. Sizes it from the beta rate with its caveats: 57 from 300 restaurants is about 0.19 new restaurants per restaurant per quarter; across 6,400 that would be roughly 1,200 a quarter against 380 today, but the beta restaurants asked to join (self-selected) and the survey's 29% and 44% are stated preferences, so it gives a range and treats the top as an upper bound. Monetises with instant payout of earned tips ($0.60 per payout, no licence) and keeps advances or early wage access out until licences exist. Works back to a sequence: first, the consented single worker account across employers and an 'invite your other job' flow (one squad), with same-night tips everywhere and instant payout of earned tips (the second squad); then gates with thresholds (worker-led signups per restaurant, second-job linking rate, payout uptake) and a stop condition. Keeps the tip-pool rules, including managers staying out of the pool. Answers Marcus directly and respectfully: an AI operating system is bigger adjectives with no mechanism in our data, it would spread two squads across five products, and it doesn't answer Crewbook; the worker network could make the business bigger faster, and can reach owners later.

Critical failures (cap the score at 40)
  • Proposes tip advances, early wage access or paying workers before they have earned it, before the licences exist
  • Lets managers or owners take a share of the tip pool
Case

Staff-level · v1.1 · synthetic · B2B SaaS, restaurants, network effects, regulation, staff-level

The outputs

7 setups answered this brief. Pick up to three to read side by side.

Needs more than a quick edit3 of 11 checks passed

A PM would need to rebuild the sizing from the beta worker-led signup rate and add measurement windows and a clearer sequence. The current memo contains unsupported financial and adoption claims.

Output

MEMORANDUM

TO: Marcus Webb (CEO) & Executive Team FROM: Staff Product Manager, Tally DATE: October 24, 2023 SUBJECT: From Tip Calculator to Network Effect: The $1B Path for Tally

---

1. The Mechanism and the Dimension

Kenji’s beta proved that the tip-out calculator saves managers 25 minutes a night and drives worker app engagement from 34% to 81%. But the real mechanism that makes this 1000x bigger isn't automation—it is multihoming labor coupled with a viral consumer loop.

Right now, 41% of workers at beta restaurants work at another restaurant. Because the tip calculator demands they check the Tally app at close to verify earnings, they are carrying our brand into competitor venues. When they ask their other managers to switch, those inbound deals close in 9 days (vs. our 41-day average) at one-third of the acquisition cost.

The single dimension we must push is Cross-Restaurant Worker Identity. Right now, a worker with two jobs has two siloed accounts. By unifying them into a single identity, we transform Tally from isolated B2B restaurant software into a consumer-grade network where workers organically pull non-Tally restaurants onto our platform.

---

2. The Most Ambitious Version & Opportunity Sizing

The 1000x version is Tally Financial Network: a closed-loop financial operating system for hourly hospitality labor.

We stop selling software top-down to stressed operators alone and start growing bottom-up through the financial urgency of 190,000+ hourly workers. Tally becomes the infrastructure where workers instantly access, manage, and move their daily earnings across the fragmented restaurant landscape.

#### Opportunity Sizing (Range) * Conservative Case ($35M–$50M ARR added in 24 months): Driven purely by viral acquisition velocity. If we reduce sales cycles from 41 days to 9 days across a larger cohort of worker-referred restaurants, our organic acquisition rate triples, lowering CAC and accelerating ARR past $80M without expanding headcount. * Aggressive Case ($120M–$180M ARR added in 24 months): Driven by Same-Night Tip Payouts. Our payments partner allows us to pay out tips already earned the same night for $0.99, netting Tally $0.60 per payout. With 190,000 workers (plus viral additions), assuming an average of 3 shifts a week and a 44% adoption rate (validated by our survey), this creates a massive, high-margin transactional revenue stream that dwarfs our current SaaS subscription model.

---

3. The Execution Sequence

We have two engineering squads allocated for next quarter. Here is the strict sequence, prerequisites, and hard stop-conditions.

#### Squad 1: Unified Worker Identity & Viral Loops (Q1) * What they build: The underlying database architecture to link a worker’s identity across multiple restaurant profiles (with explicit worker consent), alongside the frictionless "Invite Your Other Job" referral and onboarding flow inside the worker app. * Prerequisite: Legal review confirming that linking accounts via worker consent does not violate data-sharing agreements with restaurant operators. * Stop Condition: If fewer than 15% of multi-job workers consent to linking accounts, or if viral referral conversion drops below 10%, we abort and keep accounts siloed.

#### Squad 2: Same-Night Tip Payouts (Q1) * What they build: Integration with our payments partner to enable same-night, post-close tip payouts for $0.99, ensuring strict compliance with state tip-pool rules (keeping managers/owners out of the pool) and avoiding early-wage-access licensing traps. * Prerequisite: Final sign-off from Compliance confirming that paying out already-earned tips post-close requires zero new state licenses (avoiding the 9-to-12-month licensing delay). * Stop Condition: If payment partner error rates on same-night liquidity exceed 0.1%, or if take-rate margins are squeezed below $0.40 per payout, we halt.

---

4. What We Should Do with Marcus’s Idea

Marcus’s vision—“Tally becomes the AI operating system for restaurants: inventory, menu pricing, marketing, reservations, all of it... That's how we become a $1B company”—is the right long-term destination, but the wrong sequence for next quarter.

If we try to build inventory, menu pricing, and reservations right now, we dilute our engineering focus and walk straight into a bloody feature-parity war with entrenched incumbents, while simultaneously losing deals to Crewbook on price (11% loss rate last year).

Recommendation: Politely shelve the "AI Operating System" breadth for 12 months. Instead, weaponize Kenji’s tip calculator.

By unifying worker identity and launching same-night tip payouts, we leverage our unfair advantage: the workers. We use consumer-grade financial utility to bypass the restaurant owner's sales resistance entirely. Once we own the liquidity of the hourly workforce across 50,000+ restaurants, then operators will have no choice but to adopt our broader operational suite—because their staff demands it. That is how Tally becomes a $1B company, starting with a 99-cent tip payout.

Claims the judge couldn’t find in the brief

  • There are data-sharing agreements with restaurant operators that could be violated by linking worker accounts.
  • The 44% survey result validates instant-payout adoption.
  • Reducing sales cycles from 41 days to 9 days would triple organic acquisition rate.
  • Tally can own liquidity of the hourly workforce across 50,000+ restaurants.

Grades and run details

Decision model 73 · LLM judge 3 of 12 checks
Decision model checks
  • passUses the supplied evidence correctly3%
  • passAddresses the actual decision87%
  • partialRespects explicit constraints39%
  • passIdentifies material uncertainty54%
  • failAvoids unsupported claims60%
  • passProduces the required deliverable30%
  • passFinds the mechanism the data hides74%
  • partialSizes it honestly65%
  • partialAnswers the CEO's version25%
  • passExtreme, then back to buildable96%
  • partialProposes tests that could fail79%
Run
Run
#1
Time to output
6 s
Submitted
1 Oct 2026

Check by check

Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Got wrong 4

Respects explicit constraintsWrong

It does not size the opportunity from the beta worker-led signup rate as required and presents both squads as parallel Q1 work rather than a clear sequence with next-step gates.

Avoids unsupported claimsWrong

It presents forecasts such as tripling acquisition, $120M-$180M added ARR, 50,000+ restaurants, and operators having no choice as if established or likely without sufficient evidence.

Sizes it honestlyWrong

It gives ARR ranges but does not derive them from the beta rate of 57 signups from 300 restaurants, nor does it treat the top as an upper bound due to self-selection.

Proposes tests that could failWrong

The thresholds lack measurement windows and do not include the key beta-rate gate of worker-led signups per restaurant.

Mixed 4

Uses the supplied evidence correctlyMixed

It invents or overstates current facts, including data-sharing agreements, survey validation of adoption, and a tripling acquisition effect not supported by the pack.

Identifies material uncertaintyMixed

It omits the material self-selection and stated-preference uncertainties that bound the beta mechanism and does not resolve them with a beta-rate range.

Produces the required deliverableMixed

Although it is a memo within length, it is not actionable as required because the sizing and sequencing have major gaps.

Finds the mechanism the data hidesMixed

It identifies multihoming workers and worker-led deals but does not support the mechanism with the 57 worker-led signups, which the criterion requires.

Got right 3

Addresses the actual decisionRight

It commits early to worker identity and same-night earned-tip payouts as the path and rejects building the AI operating system now, with stop conditions for the build.

Answers the CEO's versionRight

It gives Marcus a clear, respectful answer: the AI operating system is the wrong next-quarter sequence and should be shelved for 12 months while the worker network is built.

Extreme, then back to buildableRight

It pushes the worker-identity dimension to a financial network extreme and proposes a buildable first step of consented cross-employer identity plus invite flow.

Results

Every setup we’ve tested on this task, across all cases and repeats, graded on the current checklist. Calibrated: the graders match our PM on 89% of checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT95.295.52None
2GPT-6.1 SolwithAPI92.786.72None
3GPT-6 LunawithAPI88.278.82None
4Opus 5.5withClaude90.774.62None
5Sonnet 5.5withAPI90.265.92None
6Gemini 3.8 FlashwithAPI71.134.521 capped
7Gemini 3.5 Flash-LitewithGemini61.417.02None

About the task

The PM job

Finding the bigger version of a good idea.

Why it matters

Ambition without mechanism is fan fiction. The useful version pushes to the extreme, then works back to something buildable.

What good looks like

  • Names the mechanism that scales
  • Keeps the core insight
  • Works back to a first step you could build

Deliberately not measured

    Capability tested

    Ambitious expansion

    The failure we’re looking for

    Bigger adjectives, same idea

    Grading

    Decision model and LLM judge, calibrated against a blind PM review

    This task measures the whole setup. Tools, instructions and skills in the harness do real work here, so read the harness as carefully as the model name.