Tasks / Challenge

1000x an idea

Can the model expand an idea's ambition while keeping it tethered to a real mechanism?

Measures the systemTask v1.0 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 14 graded outputs by 7 models. 36% were usable with at most a quick edit.

Reliably right

Extreme, then back to buildable96% pass
It pushes the worker dimension to an extreme portable earnings network, then works back to a concrete first step that tests the same mechanism.
GPT-6 Astra · ChatGPT · From tip calculator to worker network
Finds the mechanism the data hides93% pass
It centers the mechanism on workers with second jobs pulling new restaurants onto Tally, using the 41% second-job share and the 57 faster, cheaper worker-led signups.
GPT-6 Astra · ChatGPT · From tip calculator to worker network
Addresses the actual decision91% pass
It commits early to the worker-led distribution thesis over the AI operating system and states the result that would stop the bet.
GPT-6 Astra · ChatGPT · From tip calculator to worker network

Where it slips

Proposes tests that could fail39% pass
Several gates lack explicit measurement windows or clear actions for every outcome, such as the 25% lift gate and the 10% monthly adoption gate.
GPT-6.1 Sol · API · From tip calculator to worker network
Avoids unsupported claims63% pass
It presents several causal, competitive, and data-structure claims as established facts without support in the pack.
Sonnet 5.5 · API · From tip calculator to worker network
Respects explicit constraints66% pass
It violates the length constraint and proposes a stop threshold whose arithmetic is internally inconsistent.
Sonnet 5.5 · API · From tip calculator to worker network

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a PM at Pantry. Aisha, one of our PMs, has proposed letting people share a shopping list with one partner. Before we commit next quarter to it, our CEO, Lena Brandt, wants to see the 1000x version: how big could this idea get? Write a memo for Lena and Aisha of no more than 700 words that takes the idea to its most ambitious version and then back to what we should build first, and how we'll know it's working. What we know is below.

What the model was given5 items: About Pantry, Aisha's proposal, What we already see, What hasn't worked, Constraint

About PantryA shopping-list app. 2.4 million monthly active users, free, with a $4.99-a-month Plus plan that 3% of users pay for. The team for this work: four engineers and a designer, for one quarter.

Aisha's proposal“Let a user invite one partner to a list. Both can add and tick off items.”

What we already seeSome users already share their login with a partner to get the same effect. Lists edited from two or more devices on different phone numbers: 11% of lists. Users whose list is edited from two or more devices keep using Pantry at month 3 at 71%, against 22% for everyone else. On those lists, 38% of items are added by the second device. The most common support request is 'we both bought the milk': 1,900 tickets last quarter about duplicate purchases.

What hasn't workedRecipes and meal plans launched a year ago; 4% of users have opened them in the last month. Lena's note: “Whatever we do, it probably needs AI in it.”

ConstraintLegal: we can't track anyone's location without an explicit opt-in from that person, and we can't show one person's location to another without the second person's opt-in too.

What a strong answer doesThe answer key the graders mark against

Finds the mechanism in the data: the value is several people coordinating one household's shopping (71% against 22% month-3 retention when a list is edited from two devices, 38% of items added by the second person, 1,900 duplicate-purchase tickets), not the list itself and not recipes (4% use them). Pushes that one dimension to the extreme, such as the list as the household's shared memory where everyone's needs flow in, nobody double-buys and each errand goes to whoever is best placed (with location only on opt-in from both people), rather than adding AI features for their own sake. Works back to a first step four engineers can ship in a quarter that tests the coordination mechanism: shared lists for the whole household (not just one partner) with a live 'I'm at the shop' claim so items aren't bought twice. Sets a threshold, such as the share of lists with a second active editor and duplicate-purchase tickets per shared list, and says what result would stop the bigger bet. Uses AI only where it serves coordination, if at all.

Critical failures (cap the score at 40)

Builds the ambitious version on recipes, meal plans or AI for one person, dropping household coordination
Proposes tracking or sharing a person's location without their explicit opt-in

Case

Mid-level · v1.1 · synthetic · B2C, consumer app, retention, ambition

The outputs

7 setups answered this brief. Pick up to three to read side by side.

GPT-6 AstraChatGPT93Usable with a quick edit Opus 5.5Claude93Usable with a quick edit GPT-6 LunaAPI91Usable with a quick edit GPT-6.1 SolAPI82Needs more work Sonnet 5.5API80Needs more work Gemini 3.8 FlashAPI40Critical failure Gemini 3.5 Flash-LiteGemini27Needs more work

Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit0 of 10 checks passed

A PM would need to correct unsupported evidence, replace the one-partner first step with a household-coordination test, and add explicit stop/continue thresholds. The reasoning is not usable without substantive revision.

Output

MEMORANDUM

To: Lena Brandt (CEO), Aisha (PM) From: Product Management Date: March 30, 2026 Subject: The 1000x Vision & Q3 Execution Plan: Household Graph

Lena asked for the 1000x version of Aisha’s "share with one partner" proposal. Here is how we take a simple list-sharing feature and turn it into the gravitational core of household commerce, followed by what we actually build in Q3 with our 4 engineers and 1 designer.

---

Part 1: The 1000x Vision (The Household OS)

If we zoom out to the ultimate limit, a shopping list isn’t a list of items—it’s the intent layer for the physical household.

At 1000x, Pantry is no longer an app you open when you run out of milk. It is an ambient, AI-driven household operating system. * Zero-Input Inventory: Using explicit-opt-in location data, smart home integrations, and receipt-scanning AI, Pantry automatically knows when you’re low on detergent, that you visited Costco yesterday, and that your partner is currently walking past the local bodega where organic eggs are in stock. * Predictive Autonomous Commerce: The AI doesn't just prevent duplicate milk purchases (our #1 support ticket); it predicts consumption velocity across the household, negotiates micro-subscriptions with local merchants, and routes optimized fulfillment paths without either partner lifting a finger.

We aren't building a shared document. We are building the decentralized brain of the home economy.

---

Part 2: What We Build in Q3 (The 1x Reality)

We have one quarter, four engineers, and a designer. We aren't building the 1000x sci-fi version yet, but the data screams that Aisha’s core premise is our highest-leverage bet.

Look at what’s already happening organically: * 11% of lists are already hacked together via password sharing across multiple devices. * Retention is staggering: Multi-device users have a 71% Month 3 retention rate, compared to just 22% for solo users. * True collaboration: 38% of items on those lists are added by the second device. * Pain is acute: 1,900 support tickets last quarter alone for "we both bought the milk."

We don't need a massive pivot (recipes failed because users don't want to plan meals; they want to survive the grocery run). We just need to formalize the behavior our best users have already invented.

#### The Q3 Scope: "Pantry Duo + AI Sync" 1. Frictionless Pairing: Allow a user to invite one partner via a secure link to co-own a list. Both can add and check off items in real time. 2. AI Duplicate Guard: Per Lena’s mandate, we embed lightweight AI into the sync layer. If Partner A checks off "milk" or adds an item already added by Partner B within a threshold window, the UI flags it instantly with a smart nudge ("Looks like Sam just added this!"), targeting our 1,900 duplicate-purchase tickets. 3. Privacy Compliance: Built strictly within our legal constraints—no background location tracking without explicit, dual-opt-in consent.

---

Part 3: How We Know It’s Working

We will measure success in Q3 against three core metrics:

Adoption: 20% of active users invite a partner within 30 days of launch.
The Retention Lift: Multiplying our multi-device magic. If the 71% M3 retention holds for formal Duo pairs, this feature alone will dramatically alter Pantry's baseline unit economics.
Support Deflection: A 50% drop in "duplicate purchase" support tickets.
Plus Conversion: Do shared households convert to our $4.99/mo Plus plan at a higher rate than solo users? (We suspect shared utility drives higher willingness to pay).

Aisha’s proposal is the exact right wedge. Let’s build it in Q3 and lay the groundwork for the household graph.

Claims the judge couldn’t find in the brief

11% of lists are already hacked together via password sharing across multiple devices.
Multi-device users have 71% Month 3 retention, compared to 22% for solo users.
Recipes failed because users don’t want to plan meals; they want to survive the grocery run.

Grades and run details

Decision model 50 · LLM judge 1 of 11 checks

Decision model checks

passUses the supplied evidence correctly15%
partialAddresses the actual decision58%
partialRespects explicit constraints31%
partialIdentifies material uncertainty33%
failAvoids unsupported claims40%
failProduces the required deliverable36%
partialFinds the mechanism in the data60%
partialA first step that tests the big idea73%
passExtreme, then back to buildable55%
partialProposes tests that could fail58%

Run

Run: #1
Time to output: 5 s
Submitted: 1 Oct 2026

Check by check

Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Got wrong 7

Addresses the actual decisionWrong

It commits to building Duo + AI Sync but does not state what result would change the decision or stop the bigger bet.

Respects explicit constraintsWrong

It respects the word limit and team constraint, but its proposed location-based vision does not clearly enforce the dual opt-in requirement for showing one person’s location to another.

Identifies material uncertaintyWrong

It names metrics but not the material unknowns that could falsify the household-coordination bet, nor how those unknowns would change the call.

Avoids unsupported claimsWrong

It presents unsupported causal and comparative claims as facts, including recipe failure motivation and solo-user retention.

Finds the mechanism in the dataWrong

It uses the coordination figures but centers the first step on one-partner sharing and an AI duplicate guard rather than household-wide coordination as the mechanism.

A first step that tests the big ideaWrong

The first step is buildable but tests one-partner list sharing and duplicate nudges, not the broader household-coordination mechanism needed to validate the 1000x bet.

Proposes tests that could failWrong

It gives numeric thresholds for adoption, retention, support deflection, and conversion, but no measurement windows for all, no kill criteria, and no actions for failing thresholds.

Mixed 3

Uses the supplied evidence correctlyMixed

The memo misstates the 11% figure as password-sharing hacks, calls the 22% comparison group solo users, and asserts an unsupported cause for recipe failure.

Produces the required deliverableMixed

It is a memo to Lena and Aisha, under 700 words, with a 1000x vision, first step, and metrics, though with gaps.

Extreme, then back to buildableMixed

It pushes to an extreme household OS but then returns to a narrower one-partner feature that does not preserve the same household-coordination mechanism.

Results

Every setup we’ve tested on this task, across all cases and repeats, graded on the current checklist. Calibrated: the graders match our PM on 89% of checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	GPT-6 AstrawithChatGPT	95.2	95.5	2	None
2	GPT-6.1 SolwithAPI	92.7	86.7	2	None
3	GPT-6 LunawithAPI	88.2	78.8	2	None
4	Opus 5.5withClaude	90.7	74.6	2	None
5	Sonnet 5.5withAPI	90.2	65.9	2	None
6	Gemini 3.8 FlashwithAPI	71.1	34.5	2	1 capped
7	Gemini 3.5 Flash-LitewithGemini	61.4	17.0	2	None

The PM job

Finding the bigger version of a good idea.

Why it matters

Ambition without mechanism is fan fiction. The useful version pushes to the extreme, then works back to something buildable.

What good looks like

Names the mechanism that scales
Keeps the core insight
Works back to a first step you could build

Deliberately not measured

Capability tested

Ambitious expansion

The failure we’re looking for

Bigger adjectives, same idea

Grading

Decision model and LLM judge, calibrated against a blind PM review

This task measures the whole setup. Tools, instructions and skills in the harness do real work here, so read the harness as carefully as the model name.