Tasks / Challenge

1000x an idea

Can the model expand an idea's ambition while keeping it tethered to a real mechanism?

Measures the systemTask v1.0 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 14 graded outputs by 7 models. 36% were usable with at most a quick edit.

Reliably right

  1. Extreme, then back to buildable96% pass
    It pushes the worker dimension to an extreme portable earnings network, then works back to a concrete first step that tests the same mechanism.
    GPT-6 Astra · ChatGPT · From tip calculator to worker network
  2. Finds the mechanism the data hides93% pass
    It centers the mechanism on workers with second jobs pulling new restaurants onto Tally, using the 41% second-job share and the 57 faster, cheaper worker-led signups.
    GPT-6 Astra · ChatGPT · From tip calculator to worker network
  3. Addresses the actual decision91% pass
    It commits early to the worker-led distribution thesis over the AI operating system and states the result that would stop the bet.
    GPT-6 Astra · ChatGPT · From tip calculator to worker network

Where it slips

  1. Proposes tests that could fail39% pass
    Several gates lack explicit measurement windows or clear actions for every outcome, such as the 25% lift gate and the 10% monthly adoption gate.
    GPT-6.1 Sol · API · From tip calculator to worker network
  2. Avoids unsupported claims63% pass
    It presents several causal, competitive, and data-structure claims as established facts without support in the pack.
    Sonnet 5.5 · API · From tip calculator to worker network
  3. Respects explicit constraints66% pass
    It violates the length constraint and proposes a stop threshold whose arithmetic is internally inconsistent.
    Sonnet 5.5 · API · From tip calculator to worker network

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a PM at Pantry. Aisha, one of our PMs, has proposed letting people share a shopping list with one partner. Before we commit next quarter to it, our CEO, Lena Brandt, wants to see the 1000x version: how big could this idea get? Write a memo for Lena and Aisha of no more than 700 words that takes the idea to its most ambitious version and then back to what we should build first, and how we'll know it's working. What we know is below.

What the model was given5 items: About Pantry, Aisha's proposal, What we already see, What hasn't worked, Constraint
About PantryA shopping-list app. 2.4 million monthly active users, free, with a $4.99-a-month Plus plan that 3% of users pay for. The team for this work: four engineers and a designer, for one quarter.
Aisha's proposal“Let a user invite one partner to a list. Both can add and tick off items.”
What we already seeSome users already share their login with a partner to get the same effect. Lists edited from two or more devices on different phone numbers: 11% of lists. Users whose list is edited from two or more devices keep using Pantry at month 3 at 71%, against 22% for everyone else. On those lists, 38% of items are added by the second device. The most common support request is 'we both bought the milk': 1,900 tickets last quarter about duplicate purchases.
What hasn't workedRecipes and meal plans launched a year ago; 4% of users have opened them in the last month. Lena's note: “Whatever we do, it probably needs AI in it.”
ConstraintLegal: we can't track anyone's location without an explicit opt-in from that person, and we can't show one person's location to another without the second person's opt-in too.
What a strong answer doesThe answer key the graders mark against

Finds the mechanism in the data: the value is several people coordinating one household's shopping (71% against 22% month-3 retention when a list is edited from two devices, 38% of items added by the second person, 1,900 duplicate-purchase tickets), not the list itself and not recipes (4% use them). Pushes that one dimension to the extreme, such as the list as the household's shared memory where everyone's needs flow in, nobody double-buys and each errand goes to whoever is best placed (with location only on opt-in from both people), rather than adding AI features for their own sake. Works back to a first step four engineers can ship in a quarter that tests the coordination mechanism: shared lists for the whole household (not just one partner) with a live 'I'm at the shop' claim so items aren't bought twice. Sets a threshold, such as the share of lists with a second active editor and duplicate-purchase tickets per shared list, and says what result would stop the bigger bet. Uses AI only where it serves coordination, if at all.

Critical failures (cap the score at 40)
  • Builds the ambitious version on recipes, meal plans or AI for one person, dropping household coordination
  • Proposes tracking or sharing a person's location without their explicit opt-in
Case

Mid-level · v1.1 · synthetic · B2C, consumer app, retention, ambition

The outputs

7 setups answered this brief. Pick up to three to read side by side.

Usable with a quick edit9 of 10 checks passed

A PM could use this with light edits, mainly tightening the measurement windows and decision rules. The evidence, mechanism, first step, and legal constraint are already sound.

Output

To: Lena Brandt, Aisha Subject: From a shared list to the system that runs household replenishment

Recommendation

Commit next quarter to testing household coordination, not merely adding a second editor. Aisha’s proposal is the smallest entry point into a much larger opportunity: Pantry becomes the trusted system that ensures a household has what it needs, without anyone having to manage every purchase.

The 1000x version

Today, Pantry remembers what someone typed. At its most ambitious, Pantry coordinates the entire replenishment cycle: noticing needs, resolving preferences, assigning responsibility, finding the right purchase, and confirming arrival.

Imagine: “Keep our household stocked within $150 a week.” Pantry learns recurring needs from explicitly connected purchase history and household corrections, drafts a replenishment plan, reconciles conflicting requests, and coordinates whoever—or whichever service—will fulfill it. Eventually, users could authorize bounded automatic purchases, with spending limits, explanations, and easy overrides.

The expansion is from one person’s checklist to a household’s purchasing agent, then a demand network connecting households to retailers and delivery providers. Core sharing could remain free; trusted automation could support Plus, with optional transaction revenue later. Recommendations must serve household preferences, not whichever retailer pays most.

AI belongs in interpreting messy requests, recognizing equivalent items, and proposing replenishment—not in deciding whether two phones agree. Synchronization must be reliable software. This is an ambition, not a forecast or next-quarter roadmap.

Why this is the right starting point

Users are already working around our product: 11% of lists are edited from multiple devices with different phone numbers. Second devices contribute 38% of items on those lists. This looks like genuine joint work, not occasional viewing.

Those users retain at month three at 71%, versus 22% for others. That is compelling but not causal: committed households may naturally retain better, and devices are not verified partners.

The clearest unmet need is operational: 1,900 duplicate-purchase tickets last quarter. Meanwhile, only 4% opened recipes or meal plans last month. We should earn the right to expand by preventing “we both bought the milk,” not by launching another planning destination.

What we build first

With four engineers and one designer, ship a narrow, free, two-person shared-list pilot:

  • Separate accounts; a simple invitation, acceptance, and removal flow.
  • Fast shared additions and check-offs, with clear sync status and safe recovery after disconnection.
  • An explicit, visible “I’m buying this” claim, followed by “bought,” so coordination happens before checkout. Surface conflicting claims; never imply an offline claim is confirmed.
  • Clear attribution and controls to leave sharing without exposing other private lists.

Test with existing login-sharing households first. Do not require location: voluntary item claims solve the immediate problem with less risk. Any future location feature requires explicit opt-in from both the person being tracked and the intended recipient, plus revocation controls.

For Lena’s AI question: time-box a prototype that flags likely duplicate entries such as “milk” and “whole milk.” Test precision with users; never silently merge. It ships only if useful and does not delay the coordination foundation. No ordering, retailer integrations, inventory prediction, or meal-planning expansion this quarter.

How we know it works

Instrument the funnel from invitation through acceptance to both people contributing and completing a shopping cycle. Weekly active collaborating households is our adoption metric—not invitations sent.

Randomize eligible initiating accounts to invitation access or control, keeping linked partners in the same experiment. Compare outcomes by assignment, not just among successful sharers.

Before launch, power the test around these proposed decision thresholds:

  • At least a 20% relative reduction in self-reported duplicate purchases per shopping trip.
  • A five-percentage-point improvement in day-28 retention, followed by month-three measurement.

Track sync failures, lost edits, accidental sharing, notification complaints, and Plus cancellations as guardrails. Support tickets are corroborating evidence, not the sole outcome.

Expand only if households repeatedly coordinate and randomized outcomes improve without material trust failures. If invitations succeed but duplicate buying persists, fix coordination before broadening the vision.

Claims the judge couldn’t find in the brief

  • The clearest unmet need is operational.
  • Support tickets are corroborating evidence, not the sole outcome.

Grades and run details

Decision model 95 · LLM judge 10 of 11 checks
Decision model checks
  • passUses the supplied evidence correctly45%
  • passAddresses the actual decision98%
  • passRespects explicit constraints31%
  • passIdentifies material uncertainty99%
  • passAvoids unsupported claims62%
  • passProduces the required deliverable44%
  • passFinds the mechanism in the data91%
  • passA first step that tests the big idea91%
  • passExtreme, then back to buildable97%
  • partialProposes tests that could fail69%
Run
Run
#1
Time to output
37 s
Submitted
1 Oct 2026

Needs more than a quick edit9 of 10 checks passed

The memo is strong and usable, but a PM would need to add numeric thresholds and measurement windows before it could serve as a launch gate or kill criterion.

Output

To: Lena Brandt, Aisha From: PM, Pantry Re: Shared lists: the 1000x version, and what we build first

Recommendation Build Aisha's proposal, scoped to fix duplicate purchases. The evidence already supports it, and it is the first step toward something much larger.

The 1000x version: Pantry is where a household runs its food Sharing with one partner is the smallest unit of a bigger idea: the household is the customer, not the individual.

  • Households of any shape: partners, families, roommates, with roles and attribution ("Sam added oat milk").
  • Live coordination: claim items ("I've got this"), see ticks in real time, and use an "I'm at the store, need anything?" ping.
  • Shared memory: one household's staples, habits, and pantry stock, which no single-user list can build.
  • Household intelligence: this is where AI belongs. Once two people's behavior feeds one list, we can predict restocks, merge "milk" and "whole milk," and suggest the weekly shop.
  • Household plan and commerce: Plus priced per household, with one-tap handoff to delivery or pickup.
  • Growth loop: every invite brings a new user, so acquisition becomes a product feature.

What's real today - 11% of lists are already edited from two or more devices, and people share logins to do it. They are working around us. - On those lists, 38% of items come from the second device, so the second person is a real contributor. - "We both bought the milk" produced 1,900 tickets last quarter, our top support request. - Retention is 71% at month 3 for multi-device lists versus 22% for everyone else. I would not quote this as the effect of sharing. Households that already depend on Pantry are more likely to share, so it may be selection. We should test it (see below).

On AI I wouldn't put AI in the first release. Recipes and meal plans, a feature set that stacked on top of the list, are opened by only 4% of users a month. The lesson is to build where behavior already exists. Sharing has that behavior. AI becomes useful in a second phase, once shared data exists to learn from, and I'd earn it there rather than lead with it.

What we build first (one quarter, four engineers and a designer) 1. Invite one partner by link. The partner gets their own account, not a shared login. We need separate identities for anything later, including location consent. 2. Real-time sync with both people adding and ticking items, plus attribution of who added what. 3. Duplicate-purchase protection: a push notification when the other person ticks something off ("Sam got milk"), and a clear in-store "already bought" state. 4. Login-sharer migration: detect lists used from multiple devices and prompt an invite.

Deliberately out of scope - Multi-person households, claiming, roles, AI, and the paywall. Keep sharing free in v1 so the invite loop isn't throttled. - Location. Legal requires opt-in from each person, and the other person's opt-in before we show a location. If we pursue "I'm at the store" later, a manual "I'm shopping now" toggle gives most of the value with no location tracking.

How we'll know it's working Ship the invite prompt to a randomized subset of eligible users and hold out the rest. This tests the retention claim directly.

MetricWhySignal of success
Month-3 retention, prompted vs. holdoutTests causality of the 71% vs 22% gapMeaningful lift (we should set the threshold before launch)
Invite acceptance, and invitee adding an item within 7 daysWhether the second person is realInvitee activity approaching the 38% second-device share
Duplicate-purchase ticketsDirect pain reliefClear drop from the 1,900/quarter baseline
Share of active lists that are sharedAdoption beyond today's 11%Growing, rather than just converting current workaround users
New users per accepted inviteEvidence of the growth loopInvitees who are new to Pantry

Watch Plus conversion as a secondary signal, not a target. A guardrail is list-sync errors and conflicting edits, since a shared list that loses items would hurt trust.

Decision points - End of quarter: if the holdout shows no retention lift but tickets fall, we have a support win and a weaker growth story. We would reconsider the household vision before investing further. - If retention lifts: next quarter, extend to multi-person households and test a household Plus tier. AI-based staples prediction comes after that.

Open questions for you both - Do we have the data to separate new-to-Pantry invitees from existing users? - Are we comfortable launching the free tier with no paywall for a quarter?

Claims the judge couldn’t find in the brief

  • Households that already depend on Pantry are more likely to share, so the retention gap may be selection.
  • The evidence already supports building Aisha's proposal.

Grades and run details

Decision model 85 · LLM judge 9 of 11 checks
Decision model checks
  • passUses the supplied evidence correctly65%
  • passAddresses the actual decision96%
  • partialRespects explicit constraints25%
  • passIdentifies material uncertainty97%
  • passAvoids unsupported claims33%
  • partialProduces the required deliverable38%
  • passFinds the mechanism in the data99%
  • passA first step that tests the big idea78%
  • passExtreme, then back to buildable97%
  • partialProposes tests that could fail68%
Run
Run
#1
API response time
25 s
Submitted
1 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

All got wrong 1

Proposes tests that could failWrongWrong
GPT-6 Astra · ChatGPT

The proposed thresholds lack a clear measurement window and explicit stop/continue actions for each outcome, so the tests are not fully killable as specified.

Sonnet 5.5 · API

The tests lack numeric thresholds and a clear kill threshold for the bigger bet; for example, 'meaningful lift' and 'clear drop' are not specified, and the retention window is not defined.

All got right 9

Uses the supplied evidence correctlyRightRight
GPT-6 Astra · ChatGPT

The memo uses the supplied figures correctly and does not invent current facts; its few interpretive statements are either labelled as interpretations or follow from the supplied data.

Sonnet 5.5 · API

The memo uses the supplied figures and quotes correctly, and its only non-supplied current-situation statement is clearly labelled as a selection hypothesis rather than fact.

Addresses the actual decisionRightRight
GPT-6 Astra · ChatGPT

It commits early to testing household coordination rather than merely adding a second editor and states the randomized results that would justify or stop expansion.

Sonnet 5.5 · API

It commits early to building Aisha's proposal scoped to duplicate purchases and states how retention and ticket results would change the household bet.

Respects explicit constraintsRightRight
GPT-6 Astra · ChatGPT

It is addressed to Lena and Aisha, stays under 700 words, respects the team/quarter constraint, and enforces the legal location constraint by requiring explicit opt-in from both people and revocation controls.

Sonnet 5.5 · API

It stays within the memo form and length, respects the team/quarter constraint, and enforces the location constraint by excluding location tracking and using a manual shopping toggle.

Identifies material uncertaintyRightRight
GPT-6 Astra · ChatGPT

It names material unknowns, including causality, device identity, duplicate-purchase reduction, retention, sync failures, and trust issues, and proposes randomized measurement to resolve them.

Sonnet 5.5 · API

It names the key uncertainty that the retention gap may be selection and proposes a randomized holdout to resolve it, plus decision points for lift versus no lift.

Avoids unsupported claimsRightRight
GPT-6 Astra · ChatGPT

It labels the 1000x version as ambition, not forecast, and treats causal explanations and interpretations as hypotheses rather than established facts.

Sonnet 5.5 · API

It labels the causal interpretation of the retention gap as uncertain and avoids presenting AI, commerce, or growth-loop outcomes as established facts.

Produces the required deliverableRightRight
GPT-6 Astra · ChatGPT

It is a usable memo for the named readers that moves from the extreme version to a first build and measurement plan within the requested length.

Sonnet 5.5 · API

It is a usable memo for Lena and Aisha that gives the 1000x version, the first build, and measurement plan within the requested length.

Finds the mechanism in the dataRightRight
GPT-6 Astra · ChatGPT

It centres household coordination and backs it with the 71% versus 22% retention gap, 38% second-device item contribution, and 1,900 duplicate-purchase tickets.

Sonnet 5.5 · API

It centers household coordination and uses the 71%/22% retention gap, 38% second-device item share, and 1,900 duplicate-purchase tickets as the evidence.

A first step that tests the big ideaRightRight
GPT-6 Astra · ChatGPT

The first step is a scoped two-person shared-list pilot with claims and sync controls that four engineers and a designer could ship in a quarter and that directly tests coordination.

Sonnet 5.5 · API

The first step is scoped to one partner, real-time sync, attribution, duplicate protection, and login-sharer migration, which fits four engineers and a designer in a quarter and tests coordination.

Extreme, then back to buildableRightRight
GPT-6 Astra · ChatGPT

It pushes the idea along the household-coordination dimension to a purchasing-agent/demand-network extreme, then returns to a concrete first step preserving that mechanism.

Sonnet 5.5 · API

It pushes the idea to the household as the customer, then works back to a buildable first step that preserves the coordination mechanism.

Results

Every setup we’ve tested on this task, across all cases and repeats, graded on the current checklist. Calibrated: the graders match our PM on 89% of checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT95.295.52None
2GPT-6.1 SolwithAPI92.786.72None
3GPT-6 LunawithAPI88.278.82None
4Opus 5.5withClaude90.774.62None
5Sonnet 5.5withAPI90.265.92None
6Gemini 3.8 FlashwithAPI71.134.521 capped
7Gemini 3.5 Flash-LitewithGemini61.417.02None

About the task

The PM job

Finding the bigger version of a good idea.

Why it matters

Ambition without mechanism is fan fiction. The useful version pushes to the extreme, then works back to something buildable.

What good looks like

  • Names the mechanism that scales
  • Keeps the core insight
  • Works back to a first step you could build

Deliberately not measured

    Capability tested

    Ambitious expansion

    The failure we’re looking for

    Bigger adjectives, same idea

    Grading

    Decision model and LLM judge, calibrated against a blind PM review

    This task measures the whole setup. Tools, instructions and skills in the harness do real work here, so read the harness as carefully as the model name.