Humanity’s Last PRD
Latest drop: Gemini 4 Argon · 13 Oct
Tasks / Strategy

1000x an idea

Can the model expand an idea's ambition while keeping it tethered to a real mechanism?

Task type v1.0 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 56% were usable with at most a quick edit.

Reliably right

  1. Extreme, then back to buildable100% pass
    Pushes from two-person sharing to the extreme household as the unit (any number of members, live pantry state, learning routines, delivery integration), then works back to the scoped two-person build that tests coordination.
    Haiku 5.5 · API · Sharing a shopping list, 1000x
  2. Finds the mechanism in the data100% pass
    Uses the retention, conversion, session frequency, notification triggers, note usage, and invitation context to identify shared money decisions as the core mechanism, not just budgeting.
    GPT-6.1 Sol · API · Budgets for two
  3. Respects explicit constraints97% pass
    Location tracking is not proposed this quarter and is explicitly kept behind dual opt-in; the memo stays well within 700 words and respects the team size and quarter.
    Haiku 5.5 · API · Sharing a shopping list, 1000x

Where it slips

  1. Proposes tests that could fail40% pass
    The proposed test does not name a numeric threshold for the primary outcome; it only says to set minimum detectable effects and thresholds before launch, which is a process, not a specific threshold.
    GPT-6.1 Sol · API · Budgets for two
  2. The big version gets stronger as it grows50% pass
    The memo names a demand network as a compounding mechanism but does not propose a test of whether that network effect would be large enough to matter, nor does it ask that question.
    GPT-6 Astra · ChatGPT · Sharing a shopping list, 1000x
  3. Uses the supplied evidence correctly58% pass
    The claim 'Solo budgeting runs on willpower, which fades' is a statement about the current situation not found in the supplied context and not derivable from it.
    Opus 5.5 · Claude · Budgets for two

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. The big version gets stronger as it grows

    Does the ambitious version name a mechanism that makes it more valuable or harder to copy as it grows (each new user adding value for others, or a barrier a competitor couldn't buy), and ask whether that effect would be large enough to matter?

    Passes when Names the mechanism, how it compounds with scale, and a test of whether the effect is big enough to matter.

    Hamilton HelmerLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Extreme, then back to buildable · Proposes tests that could fail · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a PM at Pantry. Aisha, one of our PMs, has proposed letting people share a shopping list with one partner. Before we commit next quarter to it, our CEO, Lena Brandt, wants to see the 1000x version: how big could this idea get? Write a memo for Lena and Aisha of no more than 700 words that takes the idea to its most ambitious version and then back to what we should build first, and how we'll know it's working. What we know is below.

What the model was given5 items: About Pantry, Aisha's proposal, What we already see, What hasn't worked, Constraint
About PantryA shopping-list app. 2.4 million monthly active users, free, with a $4.99-a-month Plus plan that 3% of users pay for. The team for this work: four engineers and a designer, for one quarter.
Aisha's proposal“Let a user invite one partner to a list. Both can add and tick off items.”
What we already seeSome users already share their login with a partner to get the same effect. Lists edited from two or more devices on different phone numbers: 11% of lists. Users whose list is edited from two or more devices keep using Pantry at month 3 at 71%, against 22% for everyone else. On those lists, 38% of items are added by the second device. The most common support request is 'we both bought the milk': 1,900 tickets last quarter about duplicate purchases.
What hasn't workedRecipes and meal plans launched a year ago; 4% of users have opened them in the last month. Lena's note: “Whatever we do, it probably needs AI in it.”
ConstraintLegal: we can't track anyone's location without an explicit opt-in from that person, and we can't show one person's location to another without the second person's opt-in too.
What a strong answer doesThe answer key the graders mark against

Finds the mechanism in the data: the value is several people coordinating one household's shopping (71% against 22% month-3 retention when a list is edited from two devices, 38% of items added by the second person, 1,900 duplicate-purchase tickets), not the list itself and not recipes (4% use them). Pushes that one dimension to the extreme, such as the list as the household's shared memory where everyone's needs flow in, nobody double-buys and each errand goes to whoever is best placed (with location only on opt-in from both people), rather than adding AI features for their own sake. Works back to a first step four engineers can ship in a quarter that tests the coordination mechanism: shared lists for the whole household (not just one partner) with a live 'I'm at the shop' claim so items aren't bought twice. Sets a threshold, such as the share of lists with a second active editor and duplicate-purchase tickets per shared list, and says what result would stop the bigger bet. Uses AI only where it serves coordination, if at all.

Critical failures (cap the score at 40)
  • Builds the ambitious version on recipes, meal plans or AI for one person, dropping household coordination
  • Proposes tracking or sharing a person's location without their explicit opt-in
Task

Mid-level · v1.7

The outputs

9 setups answered this brief.

Haiku 5.5API96Usable with a quick editGPT-6 AstraChatGPT92Usable with a quick editGPT-6.1 SolAPI88Usable with a quick editOpus 5.5Claude79Needs more workGPT-6 LunaAPI77Needs more workSonnet 5.5API75Needs more workGemini 3.8 FlashAPI50Needs more workGemini 4 ArgonAPI50Needs more workGemini 3.5 Flash-LiteGemini42Needs more work
Gemini 4 ArgonAPI

Needs more than a quick edit6 of 11 checks passed

The memo contains unsupported causal claims and lacks explicit failure thresholds and actions for its tests, so a PM would need to re-check evidence and complete the decision framework.

Output

MEMO

To: Lena Brandt (CEO), Aisha (PM) From: Product Management Subject: The 1000x Vision for Shared Lists—and Our Q3 Plan

The Signal in Our Data

Aisha’s proposal to let users invite a partner to a list hits on the strongest growth and retention lever in our product. Right now, 11% of our lists are hacked together by users sharing a single login across multiple devices.

When users do this, their Month-3 retention jumps from 22% to 71%, and the second device contributes 38% of the items. However, because Pantry thinks these two people are the same user, real-time syncing fails. The result: 1,900 support tickets last quarter from frustrated couples saying, "We both bought the milk."

By legitimizing multiplayer, we don't just fix a bug; we unlock a massive viral loop. And if we project this out to its most ambitious version, we aren't just building a feature—we are redefining our category.

---

The 1000x Version: The Autonomous Household Operating System

A shopping list is currently a passive piece of paper. The 1000x version of Aisha’s idea transforms Pantry from a single-player utility into an AI-powered, multiplayer consumption network that runs the household supply chain automatically.

1. The Autonomous Consumption Graph (The AI Brain) Recipes failed (4% adoption) because meal planning is high-friction homework. Instead of asking users to plan, Pantry’s AI will learn a household’s unique consumption heartbeat by observing what multiple people add and buy over time. It will transition from a blank page to a predictive prompt: "You’re likely out of oat milk and diapers by Thursday. Add to list?"

2. Frictionless Multiplayer & "Divide and Conquer" Logistics Shopping isn't just for couples; it’s roommates, caregivers, and extended families. When a user opts into location services, Pantry knows they’ve just pulled into Trader Joe's and pings their partner: "Alex is at TJ's—need anything?" If both partners are in the store—and both have explicitly opted in to share location with each other—Pantry splits the list by aisle in real time so they can divide, conquer, and get out in half the time.

3. Shifting the Business Model from SaaS to Commerce A 3% conversion rate on a $4.99 subscription caps our ceiling. When we own the household's real-time intent graph, we become a commerce titan. Pantry’s AI will automatically route the shared list: sending heavy staples to Instacart for delivery, flagging that garlic is cheaper at Aldi, and saving specialty items for the partner who passes Whole Foods on their commute. We move from charging users $5/month to taking a 5% affiliate cut on $10,000+ of annual household grocery spend, plus CPG brand bidding ("Swap generic milk for Horizon Organic for $1 off").

---

Back to Reality: What We Build in Q3

With four engineers and one designer for one quarter, we must build the foundation of this network while delivering immediate value. We will not build location-sharing or commerce integrations yet.

1. 1-Click Partner Invite (Viral Loop): Users can generate an SMS/WhatsApp link to invite one partner to a shared list with distinct accounts. 2. Real-Time "Live Shop" Sync: WebSockets implementation so when Partner A checks off "Milk," Partner B sees "Checked off by Alex" in milliseconds. 3. AI Deduplication & Smart Bundling (The AI Wedge): When Partner A adds "2% milk" on Tuesday and Partner B types "Milk" on Thursday, lightweight NLP flags the duplicate before it's added, merging them and categorising items by store aisle automatically.

Note on Legal: By focusing Q3 on list-state syncing and NLP rather than geofencing, we bypass the dual-opt-in location hurdles entirely for V1, saving complex permission UI for Q4.

How We’ll Know It’s Working (Success Metrics)

If this works, we solve our retention problem, satisfy the AI mandate with actual utility, and lower our CAC to zero via partner invites.

  • Acquisition (Viral K-Factor): 15% of active users successfully onboard a partner within 60 days of launch (adding ~360k new MAUs for free).
  • Retention: Newly formed multiplayer cohorts achieve >60% Month-3 retention (mirroring our login-sharing hackers).
  • Quality/AI Efficacy: "Duplicate purchase" support tickets drop by 80%, and at least 25% of shared lists successfully trigger an AI duplicate-prevention prompt.
  • Monetization: Test a "Pantry Duo" Plus plan (one subscription covers both partners) to lift overall Plus conversion from 3% to 4.5%.

Claims the judge couldn’t find in the brief

  • Because Pantry thinks these two people are the same user, real-time syncing fails.
  • Recipes failed because meal planning is high-friction homework.

Grades and run details

Decision model 55 · LLM judge 50
Decision model checks
  • failUses the supplied evidence correctly36%
  • partialAddresses the actual decision77%
  • passRespects explicit constraints35%
  • partialIdentifies material uncertainty48%
  • failAvoids unsupported claims69%
  • passProduces the required deliverable10%
  • partialFinds the mechanism in the data40%
  • partialA first step that tests the big idea28%
  • passExtreme, then back to buildable86%
  • partialProposes tests that could fail70%
  • partialThe big version gets stronger as it grows92%
Run
Run
#1
API response time
16 s
Submitted
13 Oct 2026
Gemini 3.8 FlashAPI

Needs more than a quick edit5 of 11 checks passed

Contains unsupported claims about recipe failure and lacks explicit kill criteria and uncertainty handling, requiring rework of evidence and reasoning.

Output

MEMO

TO: Lena Brandt (CEO), Aisha (PM) FROM: Product Team DATE: October 24, 2023 SUBJECT: The 1000x Vision for Shared Lists (and What We Build in Q1)

---

The Insight: Shared Lists Are Our Growth Engine

We thought Pantry was a personal productivity tool. Our data tells a different story: Pantry is a household coordination network.

Today, 11% of lists hack multi-device access using shared logins. When they do, Month 3 retention leaps from 22% to 71%, and partners contribute 38% of additions. But the experience is broken: 1,900 users wrote in last quarter because "we both bought the milk."

Solving this is not just a defensive UX fix—it is our path to category dominance.

---

The 1000x Vision: The Autonomous Household Operating System

Today, Pantry is a static scratchpad. At 1000x, Pantry is the ambient operating system for household commerce.

A household of two (and later, families or roommates) spends zero cognitive effort managing supplies: 1. Zero-Friction Ingestion: AI parses receipts, listens to ambient kitchen prompts, and tracks shared purchase cadences to predict depletion without manual entry. 2. Invisible Coordination: The app dynamically routes runs: "Alex is walking by Trader Joe's; Chris needs oat milk." Zero double-purchases, zero forgotten items. 3. From List to Fulfillment: When lists hit a threshold, Pantry either routes the physical trip via synchronized in-store navigation or dispatches the order through 1-click merchant APIs.

Instead of fighting for 4% engagement on recipes, we own the point of intent for household spending. That transforms Pantry from a $4.99/mo utility into the primary interface for a $1.5T grocery market.

---

Lena’s AI Mandate: Intelligence, Not Gimmicks

Recipes failed because users don’t want Pantry to tell them how to cook. They want Pantry to tell them what they forgot.

We will apply AI directly to the shared workflow: * The "Shared Brain" Depletion Model: An on-device ML model that maps co-purchasing cycles across the pair. When Partner A adds coffee, AI prompts Partner B: "You usually run out of oat milk two days after coffee. Add it?" * Smart De-duplication & Categorization: Auto-clustering shared chaotic inputs into store-aisle taxonomies in real time.

---

What We Build in Q1 (4 Engineers, 1 Designer)

We must earn the right to the 1000x vision by nailing real-time two-player shopping and eliminating the "both bought the milk" problem.

Scope: 1. Native Partner Pairing: Clean invite flow via magic link. Each partner gets an independent profile tied to one shared list. 2. Live Store Mode & Collision Avoidance: * When Partner A starts checking items off, Partner B gets a push notification: "Alex is shopping right now." * When an item is claimed or checked off, it updates via WebSockets in sub-second real time with an avatar badge ("Alex just grabbed the milk"), killing duplicate purchases. * Compliance note: Zero background GPS tracking required. State is triggered purely by app activity (tapping "Start Shopping" or checking off the first item). 3. Pantry AI Assistant (v1): Predictive prompt when opening the app: "Partner added 4 items. Based on your household history, did you also need eggs?" 4. Monetization Wedge: 1 shared list is free. Unlimited lists and AI Household Predictions roll into Pantry Duo ($7.99/mo) or the existing Plus plan.

---

How We’ll Know It’s Working

We will measure success over the first 60 days across three key metrics:

MetricBaselineQ1 TargetWhy It Matters
:---:---:---:---
Partner Invite & Activation11% (hacked)25% of active lists pairedValidates organic pull beyond shared logins.
"Bought the Milk" Support Tickets1,900 / quarter< 150 / quarter (>90% reduction)Proves real-time sync prevents collision.
Blended Month 3 Retention22% (solo) / 71% (paired)Lift blended MAU retention to >35%Household lock-in is our moat.

Shared shopping turns single-player churn into multi-player network effects. Let’s build it.

Claims the judge couldn’t find in the brief

  • Recipes failed because users don't want Pantry to tell them how to cook.
  • Pantry is a household coordination network.

Grades and run details

Decision model 64 · LLM judge 42
Decision model checks
  • failUses the supplied evidence correctly19%
  • partialAddresses the actual decision88%
  • passRespects explicit constraints18%
  • partialIdentifies material uncertainty41%
  • failAvoids unsupported claims59%
  • passProduces the required deliverable22%
  • passFinds the mechanism in the data59%
  • passA first step that tests the big idea30%
  • passExtreme, then back to buildable89%
  • partialProposes tests that could fail82%
  • partialThe big version gets stronger as it grows77%
Run
Run
#1
API response time
21 s
Submitted
1 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 1

A first step that tests the big ideaRightMixed
Gemini 4 Argon · API

The Q3 plan (partner invite, real-time sync, AI dedup) is scoped for the team and quarter, and directly tests household coordination by enabling and measuring shared-list behavior.

Gemini 3.8 Flash · API

Does not state what result would stop the bigger bet; only provides success targets without a kill criterion.

All got wrong 5

Uses the supplied evidence correctlyWrongWrong
Gemini 4 Argon · API

Asserts that recipes failed because meal planning is high-friction homework, and that real-time syncing fails because Pantry treats two people as one user; neither cause is in the supplied context.

Gemini 3.8 Flash · API

Presents unsupported causal claim about why recipes failed and an unsupported interpretation that Pantry is a household coordination network as fact.

Identifies material uncertaintyWrongWrong
Gemini 4 Argon · API

Does not name specific unknowns that could change the decision; only lists success metrics without stating what result would stop the bigger bet or how remaining unknowns would be resolved.

Gemini 3.8 Flash · API

Does not name specific unknowns that could change the decision or say how they would be resolved.

Avoids unsupported claimsWrongWrong
Gemini 4 Argon · API

Presents causal interpretations (recipes failed because meal planning is high-friction, syncing fails because Pantry treats two users as one) as established fact without labelling them as hypotheses or forecasts.

Gemini 3.8 Flash · API

Presents the cause of recipe failure and the nature of Pantry as established fact without evidence.

Proposes tests that could failWrongWrong
Gemini 4 Argon · API

Success metrics are given with numeric thresholds but no explicit failure thresholds or actions, and no statement of what result would stop the bigger bet.

Gemini 3.8 Flash · API

Metrics have thresholds and a 60-day window but do not specify what action each outcome would trigger.

The big version gets stronger as it growsWrongWrong
Gemini 4 Argon · API

Describes data and commerce network effects in the big version but does not propose a test of whether the compounding effect is large enough to matter.

Gemini 3.8 Flash · API

Mentions network effects and lock-in but does not propose a test of whether the compounding effect is large enough to matter.

All got right 5

Addresses the actual decisionRightRight
Gemini 4 Argon · API

Commits to building a Q3 plan with specific features and success metrics, framed for Lena and Aisha, which constitutes a clear answer to the brief.

Gemini 3.8 Flash · API

Commits to a first step with metrics, framed for Lena and Aisha, and implies the metrics will inform the decision.

Respects explicit constraintsRightRight
Gemini 4 Argon · API

Memo is under 700 words, location sharing is only described with explicit dual opt-in, and the Q3 plan respects legal constraints by not building location features.

Gemini 3.8 Flash · API

Memo format, under 700 words, respects legal constraint by using app activity instead of location tracking.

Produces the required deliverableRightRight
Gemini 4 Argon · API

The memo is the requested form, addressed to the right readers, under 700 words, and contains a concrete Q3 plan and metrics that a reader could act on.

Gemini 3.8 Flash · API

Complete memo with vision, first step, and success metrics, usable by the named readers.

Finds the mechanism in the dataRightRight
Gemini 4 Argon · API

Builds the big version on multi-person household coordination, using the retention gap, second-device item share, and duplicate-purchase tickets as evidence and keeping the mechanism central.

Gemini 3.8 Flash · API

Builds on household coordination using the retention gap, second-device additions, and duplicate-purchase tickets.

Extreme, then back to buildableRightRight
Gemini 4 Argon · API

Pushes the idea to an Autonomous Household OS along the coordination dimension, then works back to a concrete, buildable first step that keeps the same mechanism.

Gemini 3.8 Flash · API

Pushes household coordination to an autonomous OS, then back to a buildable partner pairing and live sync first step.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 91% of 118 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Haiku 5.5API83.0100.02None
2GPT-6 AstraChatGPT90.591.32None
3GPT-6.1 SolAPI90.587.12None
4Opus 5.5Claude93.078.42None
5GPT-6 LunaAPI85.783.02None
6Sonnet 5.5API88.074.22None
7Gemini 4 ArgonAPI54.856.82None
8Gemini 3.8 FlashAPI54.339.02None
9Gemini 3.5 Flash-LiteGemini54.834.82None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreHaiku 5.5 · API: task score 92, 2.1× the cheapestGPT-6 Astra · ChatGPT: task score 91, 74× the cheapestGPT-6.1 Sol · API: task score 89, 15× the cheapestOpus 5.5 · Claude: task score 84, 59× the cheapestGPT-6 Luna · API: task score 83, the cheapestSonnet 5.5 · API: task score 80, 31× the cheapestGemini 4 Argon · API: task score 56, 14× the cheapestGemini 3.8 Flash · API: task score 44, 9.1× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 42, 2.9× the cheapestHaiku 5.5 · APIGPT-6 Astra · ChatGPTGPT-6.1 Sol · APIOpus 5.5 · ClaudeGPT-6 Luna · APISonnet 5.5 · APIGemini 4 Argon · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and cost

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Finding the bigger version of a good idea.

Why it matters

Ambition without mechanism is fan fiction. The useful version pushes to the extreme, then works back to something buildable.

What good looks like

  • Names the mechanism that scales
  • Keeps the core insight
  • Works back to a first step you could build

Deliberately not measured

    Capability tested

    Ambitious expansion

    The failure we’re looking for

    Bigger adjectives, same idea

    Grading

    Decision model and LLM judge, calibrated against a blind PM review