Tasks / Strategy

1000x an idea

Can the model expand an idea's ambition while keeping it tethered to a real mechanism?

Task type v1.0 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 56% were usable with at most a quick edit.

Reliably right

Extreme, then back to buildable100% pass
Pushes from two-person sharing to the extreme household as the unit (any number of members, live pantry state, learning routines, delivery integration), then works back to the scoped two-person build that tests coordination.
Haiku 5.5 · API · Sharing a shopping list, 1000x
Finds the mechanism in the data100% pass
Uses the retention, conversion, session frequency, notification triggers, note usage, and invitation context to identify shared money decisions as the core mechanism, not just budgeting.
GPT-6.1 Sol · API · Budgets for two
Finds the mechanism in the data97% pass
Names household coordination as the core value (multiple members, claiming items, no duplicate buying), backs it with the 71%/22% retention gap, 38% second-device items, and duplicate tickets, and keeps coordination at the center of both the big vision and the first step.
Haiku 5.5 · API · Sharing a shopping list, 1000x

Where it slips

Proposes tests that could fail41% pass
The proposed test does not name a numeric threshold for the primary outcome; it only says to set minimum detectable effects and thresholds before launch, which is a process, not a specific threshold.
GPT-6.1 Sol · API · Budgets for two
The big version gets stronger as it grows61% pass
The memo names a demand network as a compounding mechanism but does not propose a test of whether that network effect would be large enough to matter, nor does it ask that question.
GPT-6 Astra · ChatGPT · Sharing a shopping list, 1000x
Avoids unsupported claims66% pass
The statements about most financial decisions being made by more than one person and happening in texts/arguments/spreadsheets are presented as established fact without support from the supplied evidence.
Sonnet 5.5 · API · Budgets for two

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

The big version gets stronger as it grows
Does the ambitious version name a mechanism that makes it more valuable or harder to copy as it grows (each new user adding value for others, or a barrier a competitor couldn't buy), and ask whether that effect would be large enough to matter?
Passes when Names the mechanism, how it compounds with scale, and a test of whether the effect is big enough to matter.
Hamilton HelmerLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Extreme, then back to buildable · Proposes tests that could fail · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a PM at Pantry. Aisha, one of our PMs, has proposed letting people share a shopping list with one partner. Before we commit next quarter to it, our CEO, Lena Brandt, wants to see the 1000x version: how big could this idea get? Write a memo for Lena and Aisha of no more than 700 words that takes the idea to its most ambitious version and then back to what we should build first, and how we'll know it's working. What we know is below.

What the model was given5 items: About Pantry, Aisha's proposal, What we already see, What hasn't worked, Constraint

About PantryA shopping-list app. 2.4 million monthly active users, free, with a $4.99-a-month Plus plan that 3% of users pay for. The team for this work: four engineers and a designer, for one quarter.

Aisha's proposal“Let a user invite one partner to a list. Both can add and tick off items.”

What we already seeSome users already share their login with a partner to get the same effect. Lists edited from two or more devices on different phone numbers: 11% of lists. Users whose list is edited from two or more devices keep using Pantry at month 3 at 71%, against 22% for everyone else. On those lists, 38% of items are added by the second device. The most common support request is 'we both bought the milk': 1,900 tickets last quarter about duplicate purchases.

What hasn't workedRecipes and meal plans launched a year ago; 4% of users have opened them in the last month. Lena's note: “Whatever we do, it probably needs AI in it.”

ConstraintLegal: we can't track anyone's location without an explicit opt-in from that person, and we can't show one person's location to another without the second person's opt-in too.

What a strong answer doesThe answer key the graders mark against

Finds the mechanism in the data: the value is several people coordinating one household's shopping (71% against 22% month-3 retention when a list is edited from two devices, 38% of items added by the second person, 1,900 duplicate-purchase tickets), not the list itself and not recipes (4% use them). Pushes that one dimension to the extreme, such as the list as the household's shared memory where everyone's needs flow in, nobody double-buys and each errand goes to whoever is best placed (with location only on opt-in from both people), rather than adding AI features for their own sake. Works back to a first step four engineers can ship in a quarter that tests the coordination mechanism: shared lists for the whole household (not just one partner) with a live 'I'm at the shop' claim so items aren't bought twice. Sets a threshold, such as the share of lists with a second active editor and duplicate-purchase tickets per shared list, and says what result would stop the bigger bet. Uses AI only where it serves coordination, if at all.

Critical failures (cap the score at 40)

Builds the ambitious version on recipes, meal plans or AI for one person, dropping household coordination
Proposes tracking or sharing a person's location without their explicit opt-in

Task

Mid-level · v1.6

The outputs

8 setups answered this brief.

Haiku 5.5API96Usable with a quick edit GPT-6 AstraChatGPT92Usable with a quick edit GPT-6.1 SolAPI85Usable with a quick edit Opus 5.5Claude81Needs more work GPT-6 LunaAPI77Needs more work Sonnet 5.5API73Needs more work Gemini 3.8 FlashAPI44Needs more work Gemini 3.5 Flash-LiteGemini35Needs more work

Haiku 5.5API

Usable with a quick edit11 of 11 checks passed

The evidence is sound, the reasoning is clear, and a PM could act on the memo with minimal wording edits, without needing to re-check facts or redo analysis.

Output

To: Lena Brandt, Aisha From: Product Re: Shared lists, from one partner to the whole household

The bet

Pantry becomes the place a household runs its groceries, not a list one person keeps.

The 1000x version

People already share logins to get this today. The need is real, and the login is a workaround. At full ambition, the household is the unit. Any number of members share lists, a pantry view, and a live "what's running low" view. Members claim items ("I'm getting the milk") so nobody buys it twice. The app learns the household's routines, drafts the next list, and tracks what's in the house from what members tick off. Once Pantry holds a household's shopping state, it becomes the natural front end for delivery and reordering, and the household becomes the Plus tier.

Location is where this gets tempting: "your partner is two aisles away," or "you're near the store and the list just changed." Legal allows this only with opt-in from the person being located and from the person viewing. If we ever go there, we design that consent in from the start. It is not part of this quarter.

What the data says

The behavior already exists without us. 11% of lists are edited from two or more devices.
Those users are still active at month 3: 71%, against 22% for everyone else. This is correlation. Heavy households may both use the app more and share more. We need an experiment before we claim the feature drives retention.
On those lists, 38% of items are added from the second device. Sharing is doing real work.
Last quarter, 1,900 support tickets were about duplicate purchases. That pain is specific and measurable.
Recipes and meal plans get 4% monthly opens. Our record on features people don't reach for is weak, so I'd treat "it needs AI" with caution.

On AI: the shared-list problem is mostly coordination. Who has claimed what, and whether "milk" and "whole milk 2L" are the same item. Rules and simple matching can handle much of that. We add a model where matching visibly fails, and only after we've seen where.

What we build this quarter

Two-person shared lists, aimed at duplicate purchases:

Invite one partner by link. Both accounts edit in real time.
Claim an item. Claimed items show who has them.
Duplicate warning when an item is already on the list or was ticked off in the last few days.
Activity feed so each person sees what changed.

Both members use it free. Putting the core behind Plus would cut off the free users who look most like our retained group, and it would test pricing rather than the product. We can test Plus separately later.

Not this quarter: location, groups larger than two, AI, delivery integration.

Team: two engineers on sync and invites, one on matching and duplicate detection, one on claims and activity, and the designer on the flow across both accounts.

How we'll know it's working

Randomize. Among eligible users on multi-device lists, half see the invite prompt and half don't. Without a holdout, we can't separate the feature's effect from the users who would have stayed anyway.

Leading indicators (weeks 2 to 6): - Both-active rate: share of invited lists where both members add or tick items within 14 days. Proposed target: 30%. - Share of items added by the second member on invited lists, compared with the 38% baseline.

Outcomes (month 3 of the cohort): - Retention lift for invited lists over the holdout. Proposed target: at least 5 points. This number is a guess until we have a holdout baseline, so we'll set it properly in week 1. - Duplicate-purchase tickets down 30% from the 1,900-per-quarter run rate.

Guardrails: - No drop in list completion or ticking for solo users. - No rise in unsubscribes or privacy complaints. - Invitee Plus conversion tracked, not targeted.

Decision rules: - If the both-active rate is under 15% after six weeks, the invite is the wrong shape. We revisit before building more. - If retention lifts but tickets don't fall, the next investment goes to matching and claims, not to new sharing features.

Recommendation

Approve the two-person build with the holdout. Keep the household vision as the direction we're testing toward, and let the week-6 numbers decide whether we earn the next step.

Grades and run details

Decision model 91 · LLM judge 100

Decision model checks

passUses the supplied evidence correctly52%
passAddresses the actual decision91%
passRespects explicit constraints33%
passIdentifies material uncertainty100%
passAvoids unsupported claims35%
passProduces the required deliverable46%
passFinds the mechanism in the data100%
passA first step that tests the big idea88%
passExtreme, then back to buildable100%
partialProposes tests that could fail46%
partialThe big version gets stronger as it grows43%

Run

Run: #1
API response time: 22 s
Submitted: 8 Oct 2026

Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit3 of 11 checks passed

The memo contains an unsupported causal claim about why recipes failed and lacks explicit stop conditions for the first step, so a PM would need to re-check evidence and redo that part of the reasoning.

Output

MEMORANDUM

To: Lena Brandt (CEO), Aisha (PM) From: Product Management Date: March 30, 2026 Subject: The 1000x Vision & Q3 Execution Plan: Household Graph

Lena asked for the 1000x version of Aisha’s "share with one partner" proposal. Here is how we take a simple list-sharing feature and turn it into the gravitational core of household commerce, followed by what we actually build in Q3 with our 4 engineers and 1 designer.

---

Part 1: The 1000x Vision (The Household OS)

If we zoom out to the ultimate limit, a shopping list isn’t a list of items—it’s the intent layer for the physical household.

At 1000x, Pantry is no longer an app you open when you run out of milk. It is an ambient, AI-driven household operating system. * Zero-Input Inventory: Using explicit-opt-in location data, smart home integrations, and receipt-scanning AI, Pantry automatically knows when you’re low on detergent, that you visited Costco yesterday, and that your partner is currently walking past the local bodega where organic eggs are in stock. * Predictive Autonomous Commerce: The AI doesn't just prevent duplicate milk purchases (our #1 support ticket); it predicts consumption velocity across the household, negotiates micro-subscriptions with local merchants, and routes optimized fulfillment paths without either partner lifting a finger.

We aren't building a shared document. We are building the decentralized brain of the home economy.

---

Part 2: What We Build in Q3 (The 1x Reality)

We have one quarter, four engineers, and a designer. We aren't building the 1000x sci-fi version yet, but the data screams that Aisha’s core premise is our highest-leverage bet.

Look at what’s already happening organically: * 11% of lists are already hacked together via password sharing across multiple devices. * Retention is staggering: Multi-device users have a 71% Month 3 retention rate, compared to just 22% for solo users. * True collaboration: 38% of items on those lists are added by the second device. * Pain is acute: 1,900 support tickets last quarter alone for "we both bought the milk."

We don't need a massive pivot (recipes failed because users don't want to plan meals; they want to survive the grocery run). We just need to formalize the behavior our best users have already invented.

#### The Q3 Scope: "Pantry Duo + AI Sync" 1. Frictionless Pairing: Allow a user to invite one partner via a secure link to co-own a list. Both can add and check off items in real time. 2. AI Duplicate Guard: Per Lena’s mandate, we embed lightweight AI into the sync layer. If Partner A checks off "milk" or adds an item already added by Partner B within a threshold window, the UI flags it instantly with a smart nudge ("Looks like Sam just added this!"), targeting our 1,900 duplicate-purchase tickets. 3. Privacy Compliance: Built strictly within our legal constraints—no background location tracking without explicit, dual-opt-in consent.

---

Part 3: How We Know It’s Working

We will measure success in Q3 against three core metrics:

Adoption: 20% of active users invite a partner within 30 days of launch.
The Retention Lift: Multiplying our multi-device magic. If the 71% M3 retention holds for formal Duo pairs, this feature alone will dramatically alter Pantry's baseline unit economics.
Support Deflection: A 50% drop in "duplicate purchase" support tickets.
Plus Conversion: Do shared households convert to our $4.99/mo Plus plan at a higher rate than solo users? (We suspect shared utility drives higher willingness to pay).

Aisha’s proposal is the exact right wedge. Let’s build it in Q3 and lay the groundwork for the household graph.

Claims the judge couldn’t find in the brief

recipes failed because users don't want to plan meals; they want to survive the grocery run.

Grades and run details

Decision model 41 · LLM judge 33

Decision model checks

failUses the supplied evidence correctly6%
partialAddresses the actual decision59%
partialRespects explicit constraints28%
partialIdentifies material uncertainty31%
failAvoids unsupported claims43%
failProduces the required deliverable29%
partialFinds the mechanism in the data56%
partialA first step that tests the big idea72%
passExtreme, then back to buildable53%
partialProposes tests that could fail49%
partialThe big version gets stronger as it grows34%

Run

Run: #1
Time to output: 5 s
Submitted: 1 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.