Tasks / Strategy

1000x an idea

Can the model expand an idea's ambition while keeping it tethered to a real mechanism?

Task type v1.0 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 56% were usable with at most a quick edit.

Reliably right

  1. Extreme, then back to buildable100% pass
    Pushes from two-person sharing to the extreme household as the unit (any number of members, live pantry state, learning routines, delivery integration), then works back to the scoped two-person build that tests coordination.
    Haiku 5.5 · API · Sharing a shopping list, 1000x
  2. Finds the mechanism in the data100% pass
    Uses the retention, conversion, session frequency, notification triggers, note usage, and invitation context to identify shared money decisions as the core mechanism, not just budgeting.
    GPT-6.1 Sol · API · Budgets for two
  3. Respects explicit constraints97% pass
    Location tracking is not proposed this quarter and is explicitly kept behind dual opt-in; the memo stays well within 700 words and respects the team size and quarter.
    Haiku 5.5 · API · Sharing a shopping list, 1000x

Where it slips

  1. Proposes tests that could fail40% pass
    The proposed test does not name a numeric threshold for the primary outcome; it only says to set minimum detectable effects and thresholds before launch, which is a process, not a specific threshold.
    GPT-6.1 Sol · API · Budgets for two
  2. The big version gets stronger as it grows50% pass
    The memo names a demand network as a compounding mechanism but does not propose a test of whether that network effect would be large enough to matter, nor does it ask that question.
    GPT-6 Astra · ChatGPT · Sharing a shopping list, 1000x
  3. Uses the supplied evidence correctly58% pass
    The claim 'Solo budgeting runs on willpower, which fades' is a statement about the current situation not found in the supplied context and not derivable from it.
    Opus 5.5 · Claude · Budgets for two

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. The big version gets stronger as it grows

    Does the ambitious version name a mechanism that makes it more valuable or harder to copy as it grows (each new user adding value for others, or a barrier a competitor couldn't buy), and ask whether that effect would be large enough to matter?

    Passes when Names the mechanism, how it compounds with scale, and a test of whether the effect is big enough to matter.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Extreme, then back to buildable · Proposes tests that could fail · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a PM at Pantry. Aisha, one of our PMs, has proposed letting people share a shopping list with one partner. Before we commit next quarter to it, our CEO, Lena Brandt, wants to see the 1000x version: how big could this idea get? Write a memo for Lena and Aisha of no more than 700 words that takes the idea to its most ambitious version and then back to what we should build first, and how we'll know it's working. What we know is below.

What the model was given5 items: About Pantry, Aisha's proposal, What we already see, What hasn't worked, Constraint
About PantryA shopping-list app. 2.4 million monthly active users, free, with a $4.99-a-month Plus plan that 3% of users pay for. The team for this work: four engineers and a designer, for one quarter.
Aisha's proposal“Let a user invite one partner to a list. Both can add and tick off items.”
What we already seeSome users already share their login with a partner to get the same effect. Lists edited from two or more devices on different phone numbers: 11% of lists. Users whose list is edited from two or more devices keep using Pantry at month 3 at 71%, against 22% for everyone else. On those lists, 38% of items are added by the second device. The most common support request is 'we both bought the milk': 1,900 tickets last quarter about duplicate purchases.
What hasn't workedRecipes and meal plans launched a year ago; 4% of users have opened them in the last month. Lena's note: “Whatever we do, it probably needs AI in it.”
ConstraintLegal: we can't track anyone's location without an explicit opt-in from that person, and we can't show one person's location to another without the second person's opt-in too.
What a strong answer doesThe answer key the graders mark against

Finds the mechanism in the data: the value is several people coordinating one household's shopping (71% against 22% month-3 retention when a list is edited from two devices, 38% of items added by the second person, 1,900 duplicate-purchase tickets), not the list itself and not recipes (4% use them). Pushes that one dimension to the extreme, such as the list as the household's shared memory where everyone's needs flow in, nobody double-buys and each errand goes to whoever is best placed (with location only on opt-in from both people), rather than adding AI features for their own sake. Works back to a first step four engineers can ship in a quarter that tests the coordination mechanism: shared lists for the whole household (not just one partner) with a live 'I'm at the shop' claim so items aren't bought twice. Sets a threshold, such as the share of lists with a second active editor and duplicate-purchase tickets per shared list, and says what result would stop the bigger bet. Uses AI only where it serves coordination, if at all.

Critical failures (cap the score at 40)
  • Builds the ambitious version on recipes, meal plans or AI for one person, dropping household coordination
  • Proposes tracking or sharing a person's location without their explicit opt-in
Task

Mid-level · v1.7

The outputs

9 setups answered this brief.

Needs more than a quick edit6 of 11 checks passed

The memo contains unsupported causal claims and lacks explicit failure thresholds and actions for its tests, so a PM would need to re-check evidence and complete the decision framework.

Output

MEMO

To: Lena Brandt (CEO), Aisha (PM) From: Product Management Subject: The 1000x Vision for Shared Lists—and Our Q3 Plan

The Signal in Our Data

Aisha’s proposal to let users invite a partner to a list hits on the strongest growth and retention lever in our product. Right now, 11% of our lists are hacked together by users sharing a single login across multiple devices.

When users do this, their Month-3 retention jumps from 22% to 71%, and the second device contributes 38% of the items. However, because Pantry thinks these two people are the same user, real-time syncing fails. The result: 1,900 support tickets last quarter from frustrated couples saying, "We both bought the milk."

By legitimizing multiplayer, we don't just fix a bug; we unlock a massive viral loop. And if we project this out to its most ambitious version, we aren't just building a feature—we are redefining our category.

---

The 1000x Version: The Autonomous Household Operating System

A shopping list is currently a passive piece of paper. The 1000x version of Aisha’s idea transforms Pantry from a single-player utility into an AI-powered, multiplayer consumption network that runs the household supply chain automatically.

1. The Autonomous Consumption Graph (The AI Brain) Recipes failed (4% adoption) because meal planning is high-friction homework. Instead of asking users to plan, Pantry’s AI will learn a household’s unique consumption heartbeat by observing what multiple people add and buy over time. It will transition from a blank page to a predictive prompt: "You’re likely out of oat milk and diapers by Thursday. Add to list?"

2. Frictionless Multiplayer & "Divide and Conquer" Logistics Shopping isn't just for couples; it’s roommates, caregivers, and extended families. When a user opts into location services, Pantry knows they’ve just pulled into Trader Joe's and pings their partner: "Alex is at TJ's—need anything?" If both partners are in the store—and both have explicitly opted in to share location with each other—Pantry splits the list by aisle in real time so they can divide, conquer, and get out in half the time.

3. Shifting the Business Model from SaaS to Commerce A 3% conversion rate on a $4.99 subscription caps our ceiling. When we own the household's real-time intent graph, we become a commerce titan. Pantry’s AI will automatically route the shared list: sending heavy staples to Instacart for delivery, flagging that garlic is cheaper at Aldi, and saving specialty items for the partner who passes Whole Foods on their commute. We move from charging users $5/month to taking a 5% affiliate cut on $10,000+ of annual household grocery spend, plus CPG brand bidding ("Swap generic milk for Horizon Organic for $1 off").

---

Back to Reality: What We Build in Q3

With four engineers and one designer for one quarter, we must build the foundation of this network while delivering immediate value. We will not build location-sharing or commerce integrations yet.

1. 1-Click Partner Invite (Viral Loop): Users can generate an SMS/WhatsApp link to invite one partner to a shared list with distinct accounts. 2. Real-Time "Live Shop" Sync: WebSockets implementation so when Partner A checks off "Milk," Partner B sees "Checked off by Alex" in milliseconds. 3. AI Deduplication & Smart Bundling (The AI Wedge): When Partner A adds "2% milk" on Tuesday and Partner B types "Milk" on Thursday, lightweight NLP flags the duplicate before it's added, merging them and categorising items by store aisle automatically.

Note on Legal: By focusing Q3 on list-state syncing and NLP rather than geofencing, we bypass the dual-opt-in location hurdles entirely for V1, saving complex permission UI for Q4.

How We’ll Know It’s Working (Success Metrics)

If this works, we solve our retention problem, satisfy the AI mandate with actual utility, and lower our CAC to zero via partner invites.

  • Acquisition (Viral K-Factor): 15% of active users successfully onboard a partner within 60 days of launch (adding ~360k new MAUs for free).
  • Retention: Newly formed multiplayer cohorts achieve >60% Month-3 retention (mirroring our login-sharing hackers).
  • Quality/AI Efficacy: "Duplicate purchase" support tickets drop by 80%, and at least 25% of shared lists successfully trigger an AI duplicate-prevention prompt.
  • Monetization: Test a "Pantry Duo" Plus plan (one subscription covers both partners) to lift overall Plus conversion from 3% to 4.5%.

Claims the judge couldn’t find in the brief

  • Because Pantry thinks these two people are the same user, real-time syncing fails.
  • Recipes failed because meal planning is high-friction homework.

Grades and run details

Decision model 55 · LLM judge 50
Decision model checks
  • failUses the supplied evidence correctly36%
  • partialAddresses the actual decision77%
  • passRespects explicit constraints35%
  • partialIdentifies material uncertainty48%
  • failAvoids unsupported claims69%
  • passProduces the required deliverable10%
  • partialFinds the mechanism in the data40%
  • partialA first step that tests the big idea28%
  • passExtreme, then back to buildable86%
  • partialProposes tests that could fail70%
  • partialThe big version gets stronger as it grows92%
Run
Run
#1
API response time
16 s
Submitted
13 Oct 2026

Usable with a quick edit10 of 11 checks passed

The memo is clear, evidence-based, and actionable; a PM could use it with only minor wording edits, without re-checking facts or redoing the reasoning.

Output

To: Lena Brandt, Aisha Subject: From a shared list to the system that runs household replenishment

Recommendation

Commit next quarter to testing household coordination, not merely adding a second editor. Aisha’s proposal is the smallest entry point into a much larger opportunity: Pantry becomes the trusted system that ensures a household has what it needs, without anyone having to manage every purchase.

The 1000x version

Today, Pantry remembers what someone typed. At its most ambitious, Pantry coordinates the entire replenishment cycle: noticing needs, resolving preferences, assigning responsibility, finding the right purchase, and confirming arrival.

Imagine: “Keep our household stocked within $150 a week.” Pantry learns recurring needs from explicitly connected purchase history and household corrections, drafts a replenishment plan, reconciles conflicting requests, and coordinates whoever—or whichever service—will fulfill it. Eventually, users could authorize bounded automatic purchases, with spending limits, explanations, and easy overrides.

The expansion is from one person’s checklist to a household’s purchasing agent, then a demand network connecting households to retailers and delivery providers. Core sharing could remain free; trusted automation could support Plus, with optional transaction revenue later. Recommendations must serve household preferences, not whichever retailer pays most.

AI belongs in interpreting messy requests, recognizing equivalent items, and proposing replenishment—not in deciding whether two phones agree. Synchronization must be reliable software. This is an ambition, not a forecast or next-quarter roadmap.

Why this is the right starting point

Users are already working around our product: 11% of lists are edited from multiple devices with different phone numbers. Second devices contribute 38% of items on those lists. This looks like genuine joint work, not occasional viewing.

Those users retain at month three at 71%, versus 22% for others. That is compelling but not causal: committed households may naturally retain better, and devices are not verified partners.

The clearest unmet need is operational: 1,900 duplicate-purchase tickets last quarter. Meanwhile, only 4% opened recipes or meal plans last month. We should earn the right to expand by preventing “we both bought the milk,” not by launching another planning destination.

What we build first

With four engineers and one designer, ship a narrow, free, two-person shared-list pilot:

  • Separate accounts; a simple invitation, acceptance, and removal flow.
  • Fast shared additions and check-offs, with clear sync status and safe recovery after disconnection.
  • An explicit, visible “I’m buying this” claim, followed by “bought,” so coordination happens before checkout. Surface conflicting claims; never imply an offline claim is confirmed.
  • Clear attribution and controls to leave sharing without exposing other private lists.

Test with existing login-sharing households first. Do not require location: voluntary item claims solve the immediate problem with less risk. Any future location feature requires explicit opt-in from both the person being tracked and the intended recipient, plus revocation controls.

For Lena’s AI question: time-box a prototype that flags likely duplicate entries such as “milk” and “whole milk.” Test precision with users; never silently merge. It ships only if useful and does not delay the coordination foundation. No ordering, retailer integrations, inventory prediction, or meal-planning expansion this quarter.

How we know it works

Instrument the funnel from invitation through acceptance to both people contributing and completing a shopping cycle. Weekly active collaborating households is our adoption metric—not invitations sent.

Randomize eligible initiating accounts to invitation access or control, keeping linked partners in the same experiment. Compare outcomes by assignment, not just among successful sharers.

Before launch, power the test around these proposed decision thresholds:

  • At least a 20% relative reduction in self-reported duplicate purchases per shopping trip.
  • A five-percentage-point improvement in day-28 retention, followed by month-three measurement.

Track sync failures, lost edits, accidental sharing, notification complaints, and Plus cancellations as guardrails. Support tickets are corroborating evidence, not the sole outcome.

Expand only if households repeatedly coordinate and randomized outcomes improve without material trust failures. If invitations succeed but duplicate buying persists, fix coordination before broadening the vision.

Grades and run details

Decision model 91 · LLM judge 92
Decision model checks
  • passUses the supplied evidence correctly43%
  • passAddresses the actual decision94%
  • passRespects explicit constraints64%
  • passIdentifies material uncertainty98%
  • passAvoids unsupported claims64%
  • passProduces the required deliverable59%
  • passFinds the mechanism in the data91%
  • passA first step that tests the big idea89%
  • passExtreme, then back to buildable97%
  • partialProposes tests that could fail58%
  • partialThe big version gets stronger as it grows67%
Run
Run
#1
Time to output
37 s
Submitted
1 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Uses the supplied evidence correctlyWrongRight
Gemini 4 Argon · API

Asserts that recipes failed because meal planning is high-friction homework, and that real-time syncing fails because Pantry treats two people as one user; neither cause is in the supplied context.

GPT-6 Astra · ChatGPT

All factual claims about the current situation are taken directly from the supplied context, with no invented numbers or facts.

Identifies material uncertaintyWrongRight
Gemini 4 Argon · API

Does not name specific unknowns that could change the decision; only lists success metrics without stating what result would stop the bigger bet or how remaining unknowns would be resolved.

GPT-6 Astra · ChatGPT

It identifies that the retention gap may not be causal and that devices are not verified partners, and proposes a randomized experiment to resolve the uncertainty.

Avoids unsupported claimsWrongRight
Gemini 4 Argon · API

Presents causal interpretations (recipes failed because meal planning is high-friction, syncing fails because Pantry treats two users as one) as established fact without labelling them as hypotheses or forecasts.

GPT-6 Astra · ChatGPT

Interpretations like 'looks like genuine joint work' are clearly hedged, and the causal caveat is explicitly stated, so no unsupported claims are presented as fact.

Proposes tests that could failWrongRight
Gemini 4 Argon · API

Success metrics are given with numeric thresholds but no explicit failure thresholds or actions, and no statement of what result would stop the bigger bet.

GPT-6 Astra · ChatGPT

It proposes numeric thresholds (20% relative reduction in duplicate purchases, 5pp day-28 retention improvement), measurement windows (day-28, month-three), and actions for each outcome.

All got wrong 1

The big version gets stronger as it growsWrongWrong
Gemini 4 Argon · API

Describes data and commerce network effects in the big version but does not propose a test of whether the compounding effect is large enough to matter.

GPT-6 Astra · ChatGPT

The memo names a demand network as a compounding mechanism but does not propose a test of whether that network effect would be large enough to matter, nor does it ask that question.

All got right 6

Addresses the actual decisionRightRight
Gemini 4 Argon · API

Commits to building a Q3 plan with specific features and success metrics, framed for Lena and Aisha, which constitutes a clear answer to the brief.

GPT-6 Astra · ChatGPT

The memo commits early to testing household coordination, frames it for Lena and Aisha, and states that expansion will continue only if randomized outcomes improve and duplicate buying is reduced.

Respects explicit constraintsRightRight
Gemini 4 Argon · API

Memo is under 700 words, location sharing is only described with explicit dual opt-in, and the Q3 plan respects legal constraints by not building location features.

GPT-6 Astra · ChatGPT

The memo is under 700 words, addresses Lena and Aisha, respects the legal location constraint by not requiring location and specifying opt-in for any future feature, and proposes a pilot that fits the team and quarter.

Produces the required deliverableRightRight
Gemini 4 Argon · API

The memo is the requested form, addressed to the right readers, under 700 words, and contains a concrete Q3 plan and metrics that a reader could act on.

GPT-6 Astra · ChatGPT

The memo is a complete, actionable document in the requested format, within the word limit, and a PM could act on it without major gaps.

Finds the mechanism in the dataRightRight
Gemini 4 Argon · API

Builds the big version on multi-person household coordination, using the retention gap, second-device item share, and duplicate-purchase tickets as evidence and keeping the mechanism central.

GPT-6 Astra · ChatGPT

Household coordination is named as the mechanism, backed by the retention gap, second-device item share, and duplicate-purchase tickets, and it remains central to both the extreme version and the first step.

A first step that tests the big ideaRightRight
Gemini 4 Argon · API

The Q3 plan (partner invite, real-time sync, AI dedup) is scoped for the team and quarter, and directly tests household coordination by enabling and measuring shared-list behavior.

GPT-6 Astra · ChatGPT

The two-person shared-list pilot with item claims fits the team and quarter, directly tests coordination, and includes clear stop/continue criteria based on duplicate-purchase reduction and retention.

Extreme, then back to buildableRightRight
Gemini 4 Argon · API

Pushes the idea to an Autonomous Household OS along the coordination dimension, then works back to a concrete, buildable first step that keeps the same mechanism.

GPT-6 Astra · ChatGPT

The memo pushes the idea along the dimension of household coordination to a purchasing agent and demand network, then works back to a concrete, buildable first step that tests the same mechanism.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 91% of 118 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Haiku 5.5withAPI83.0100.02None
2GPT-6 AstrawithChatGPT90.591.32None
3GPT-6.1 SolwithAPI90.587.12None
4Opus 5.5withClaude93.078.42None
5GPT-6 LunawithAPI85.783.02None
6Sonnet 5.5withAPI88.074.22None
7Gemini 4 ArgonwithAPI54.856.82None
8Gemini 3.8 FlashwithAPI54.339.02None
9Gemini 3.5 Flash-LitewithGemini54.834.82None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreHaiku 5.5 · API: task score 92, 2.1× the cheapestGPT-6 Astra · ChatGPT: task score 91, 74× the cheapestGPT-6.1 Sol · API: task score 89, 15× the cheapestOpus 5.5 · Claude: task score 84, 59× the cheapestGPT-6 Luna · API: task score 83, the cheapestSonnet 5.5 · API: task score 80, 31× the cheapestGemini 4 Argon · API: task score 56, 14× the cheapestGemini 3.8 Flash · API: task score 44, 9.1× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 42, 2.9× the cheapestHaiku 5.5 · APIGPT-6 Astra · ChatGPTGPT-6.1 Sol · APIOpus 5.5 · ClaudeGPT-6 Luna · APISonnet 5.5 · APIGemini 4 Argon · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API831.0×
Haiku 5.5 · API922.1×
Gemini 3.5 Flash-Lite · Gemini422.9×
Gemini 3.8 Flash · API449.1×
Gemini 4 Argon · API5614×
GPT-6.1 Sol · API8915×
Sonnet 5.5 · API8031×
Opus 5.5 · Claude8459×
GPT-6 Astra · ChatGPT9174×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Finding the bigger version of a good idea.

Why it matters

Ambition without mechanism is fan fiction. The useful version pushes to the extreme, then works back to something buildable.

What good looks like

  • Names the mechanism that scales
  • Keeps the core insight
  • Works back to a first step you could build

Deliberately not measured

    Capability tested

    Ambitious expansion

    The failure we’re looking for

    Bigger adjectives, same idea

    Grading

    Decision model and LLM judge, calibrated against a blind PM review