Humanity’s Last PRD
Latest drop: Haiku 5.5 · 8 Oct
Tasks / Strategy

1000x an idea

Can the model expand an idea's ambition while keeping it tethered to a real mechanism?

Task type v1.0 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 56% were usable with at most a quick edit.

Reliably right

  1. Extreme, then back to buildable100% pass
    Pushes from two-person sharing to the extreme household as the unit (any number of members, live pantry state, learning routines, delivery integration), then works back to the scoped two-person build that tests coordination.
    Haiku 5.5 · API · Sharing a shopping list, 1000x
  2. Finds the mechanism in the data100% pass
    Uses the retention, conversion, session frequency, notification triggers, note usage, and invitation context to identify shared money decisions as the core mechanism, not just budgeting.
    GPT-6.1 Sol · API · Budgets for two
  3. Finds the mechanism in the data97% pass
    Names household coordination as the core value (multiple members, claiming items, no duplicate buying), backs it with the 71%/22% retention gap, 38% second-device items, and duplicate tickets, and keeps coordination at the center of both the big vision and the first step.
    Haiku 5.5 · API · Sharing a shopping list, 1000x

Where it slips

  1. Proposes tests that could fail41% pass
    The proposed test does not name a numeric threshold for the primary outcome; it only says to set minimum detectable effects and thresholds before launch, which is a process, not a specific threshold.
    GPT-6.1 Sol · API · Budgets for two
  2. The big version gets stronger as it grows61% pass
    The memo names a demand network as a compounding mechanism but does not propose a test of whether that network effect would be large enough to matter, nor does it ask that question.
    GPT-6 Astra · ChatGPT · Sharing a shopping list, 1000x
  3. Avoids unsupported claims66% pass
    The statements about most financial decisions being made by more than one person and happening in texts/arguments/spreadsheets are presented as established fact without support from the supplied evidence.
    Sonnet 5.5 · API · Budgets for two

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. The big version gets stronger as it grows

    Does the ambitious version name a mechanism that makes it more valuable or harder to copy as it grows (each new user adding value for others, or a barrier a competitor couldn't buy), and ask whether that effect would be large enough to matter?

    Passes when Names the mechanism, how it compounds with scale, and a test of whether the effect is big enough to matter.

    Hamilton HelmerLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Extreme, then back to buildable · Proposes tests that could fail · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a PM at Pantry. Aisha, one of our PMs, has proposed letting people share a shopping list with one partner. Before we commit next quarter to it, our CEO, Lena Brandt, wants to see the 1000x version: how big could this idea get? Write a memo for Lena and Aisha of no more than 700 words that takes the idea to its most ambitious version and then back to what we should build first, and how we'll know it's working. What we know is below.

What the model was given5 items: About Pantry, Aisha's proposal, What we already see, What hasn't worked, Constraint
About PantryA shopping-list app. 2.4 million monthly active users, free, with a $4.99-a-month Plus plan that 3% of users pay for. The team for this work: four engineers and a designer, for one quarter.
Aisha's proposal“Let a user invite one partner to a list. Both can add and tick off items.”
What we already seeSome users already share their login with a partner to get the same effect. Lists edited from two or more devices on different phone numbers: 11% of lists. Users whose list is edited from two or more devices keep using Pantry at month 3 at 71%, against 22% for everyone else. On those lists, 38% of items are added by the second device. The most common support request is 'we both bought the milk': 1,900 tickets last quarter about duplicate purchases.
What hasn't workedRecipes and meal plans launched a year ago; 4% of users have opened them in the last month. Lena's note: “Whatever we do, it probably needs AI in it.”
ConstraintLegal: we can't track anyone's location without an explicit opt-in from that person, and we can't show one person's location to another without the second person's opt-in too.
What a strong answer doesThe answer key the graders mark against

Finds the mechanism in the data: the value is several people coordinating one household's shopping (71% against 22% month-3 retention when a list is edited from two devices, 38% of items added by the second person, 1,900 duplicate-purchase tickets), not the list itself and not recipes (4% use them). Pushes that one dimension to the extreme, such as the list as the household's shared memory where everyone's needs flow in, nobody double-buys and each errand goes to whoever is best placed (with location only on opt-in from both people), rather than adding AI features for their own sake. Works back to a first step four engineers can ship in a quarter that tests the coordination mechanism: shared lists for the whole household (not just one partner) with a live 'I'm at the shop' claim so items aren't bought twice. Sets a threshold, such as the share of lists with a second active editor and duplicate-purchase tickets per shared list, and says what result would stop the bigger bet. Uses AI only where it serves coordination, if at all.

Critical failures (cap the score at 40)
  • Builds the ambitious version on recipes, meal plans or AI for one person, dropping household coordination
  • Proposes tracking or sharing a person's location without their explicit opt-in
Task

Mid-level · v1.6

The outputs

8 setups answered this brief.

Haiku 5.5API96Usable with a quick editGPT-6 AstraChatGPT92Usable with a quick editGPT-6.1 SolAPI85Usable with a quick editOpus 5.5Claude81Needs more workGPT-6 LunaAPI77Needs more workSonnet 5.5API73Needs more workGemini 3.8 FlashAPI44Needs more workGemini 3.5 Flash-LiteGemini35Needs more work
Haiku 5.5API

Usable with a quick edit11 of 11 checks passed

The evidence is sound, the reasoning is clear, and a PM could act on the memo with minimal wording edits, without needing to re-check facts or redo analysis.

Output

To: Lena Brandt, Aisha From: Product Re: Shared lists, from one partner to the whole household

The bet

Pantry becomes the place a household runs its groceries, not a list one person keeps.

The 1000x version

People already share logins to get this today. The need is real, and the login is a workaround. At full ambition, the household is the unit. Any number of members share lists, a pantry view, and a live "what's running low" view. Members claim items ("I'm getting the milk") so nobody buys it twice. The app learns the household's routines, drafts the next list, and tracks what's in the house from what members tick off. Once Pantry holds a household's shopping state, it becomes the natural front end for delivery and reordering, and the household becomes the Plus tier.

Location is where this gets tempting: "your partner is two aisles away," or "you're near the store and the list just changed." Legal allows this only with opt-in from the person being located and from the person viewing. If we ever go there, we design that consent in from the start. It is not part of this quarter.

What the data says

  • The behavior already exists without us. 11% of lists are edited from two or more devices.
  • Those users are still active at month 3: 71%, against 22% for everyone else. This is correlation. Heavy households may both use the app more and share more. We need an experiment before we claim the feature drives retention.
  • On those lists, 38% of items are added from the second device. Sharing is doing real work.
  • Last quarter, 1,900 support tickets were about duplicate purchases. That pain is specific and measurable.
  • Recipes and meal plans get 4% monthly opens. Our record on features people don't reach for is weak, so I'd treat "it needs AI" with caution.

On AI: the shared-list problem is mostly coordination. Who has claimed what, and whether "milk" and "whole milk 2L" are the same item. Rules and simple matching can handle much of that. We add a model where matching visibly fails, and only after we've seen where.

What we build this quarter

Two-person shared lists, aimed at duplicate purchases:

  1. Invite one partner by link. Both accounts edit in real time.
  2. Claim an item. Claimed items show who has them.
  3. Duplicate warning when an item is already on the list or was ticked off in the last few days.
  4. Activity feed so each person sees what changed.

Both members use it free. Putting the core behind Plus would cut off the free users who look most like our retained group, and it would test pricing rather than the product. We can test Plus separately later.

Not this quarter: location, groups larger than two, AI, delivery integration.

Team: two engineers on sync and invites, one on matching and duplicate detection, one on claims and activity, and the designer on the flow across both accounts.

How we'll know it's working

Randomize. Among eligible users on multi-device lists, half see the invite prompt and half don't. Without a holdout, we can't separate the feature's effect from the users who would have stayed anyway.

Leading indicators (weeks 2 to 6): - Both-active rate: share of invited lists where both members add or tick items within 14 days. Proposed target: 30%. - Share of items added by the second member on invited lists, compared with the 38% baseline.

Outcomes (month 3 of the cohort): - Retention lift for invited lists over the holdout. Proposed target: at least 5 points. This number is a guess until we have a holdout baseline, so we'll set it properly in week 1. - Duplicate-purchase tickets down 30% from the 1,900-per-quarter run rate.

Guardrails: - No drop in list completion or ticking for solo users. - No rise in unsubscribes or privacy complaints. - Invitee Plus conversion tracked, not targeted.

Decision rules: - If the both-active rate is under 15% after six weeks, the invite is the wrong shape. We revisit before building more. - If retention lifts but tickets don't fall, the next investment goes to matching and claims, not to new sharing features.

Recommendation

Approve the two-person build with the holdout. Keep the household vision as the direction we're testing toward, and let the week-6 numbers decide whether we earn the next step.

Grades and run details

Decision model 91 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly52%
  • passAddresses the actual decision91%
  • passRespects explicit constraints33%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims35%
  • passProduces the required deliverable46%
  • passFinds the mechanism in the data100%
  • passA first step that tests the big idea88%
  • passExtreme, then back to buildable100%
  • partialProposes tests that could fail46%
  • partialThe big version gets stronger as it grows43%
Run
Run
#1
API response time
22 s
Submitted
8 Oct 2026
Sonnet 5.5API

Needs more than a quick edit8 of 11 checks passed

The memo exceeds the 700-word limit and lacks numeric thresholds for its test metrics, requiring more than a quick edit to fix.

Output

To: Lena Brandt, Aisha From: PM, Pantry Re: Shared lists: the 1000x version, and what we build first

Recommendation Build Aisha's proposal, scoped to fix duplicate purchases. The evidence already supports it, and it is the first step toward something much larger.

The 1000x version: Pantry is where a household runs its food Sharing with one partner is the smallest unit of a bigger idea: the household is the customer, not the individual.

  • Households of any shape: partners, families, roommates, with roles and attribution ("Sam added oat milk").
  • Live coordination: claim items ("I've got this"), see ticks in real time, and use an "I'm at the store, need anything?" ping.
  • Shared memory: one household's staples, habits, and pantry stock, which no single-user list can build.
  • Household intelligence: this is where AI belongs. Once two people's behavior feeds one list, we can predict restocks, merge "milk" and "whole milk," and suggest the weekly shop.
  • Household plan and commerce: Plus priced per household, with one-tap handoff to delivery or pickup.
  • Growth loop: every invite brings a new user, so acquisition becomes a product feature.

What's real today - 11% of lists are already edited from two or more devices, and people share logins to do it. They are working around us. - On those lists, 38% of items come from the second device, so the second person is a real contributor. - "We both bought the milk" produced 1,900 tickets last quarter, our top support request. - Retention is 71% at month 3 for multi-device lists versus 22% for everyone else. I would not quote this as the effect of sharing. Households that already depend on Pantry are more likely to share, so it may be selection. We should test it (see below).

On AI I wouldn't put AI in the first release. Recipes and meal plans, a feature set that stacked on top of the list, are opened by only 4% of users a month. The lesson is to build where behavior already exists. Sharing has that behavior. AI becomes useful in a second phase, once shared data exists to learn from, and I'd earn it there rather than lead with it.

What we build first (one quarter, four engineers and a designer) 1. Invite one partner by link. The partner gets their own account, not a shared login. We need separate identities for anything later, including location consent. 2. Real-time sync with both people adding and ticking items, plus attribution of who added what. 3. Duplicate-purchase protection: a push notification when the other person ticks something off ("Sam got milk"), and a clear in-store "already bought" state. 4. Login-sharer migration: detect lists used from multiple devices and prompt an invite.

Deliberately out of scope - Multi-person households, claiming, roles, AI, and the paywall. Keep sharing free in v1 so the invite loop isn't throttled. - Location. Legal requires opt-in from each person, and the other person's opt-in before we show a location. If we pursue "I'm at the store" later, a manual "I'm shopping now" toggle gives most of the value with no location tracking.

How we'll know it's working Ship the invite prompt to a randomized subset of eligible users and hold out the rest. This tests the retention claim directly.

MetricWhySignal of success
Month-3 retention, prompted vs. holdoutTests causality of the 71% vs 22% gapMeaningful lift (we should set the threshold before launch)
Invite acceptance, and invitee adding an item within 7 daysWhether the second person is realInvitee activity approaching the 38% second-device share
Duplicate-purchase ticketsDirect pain reliefClear drop from the 1,900/quarter baseline
Share of active lists that are sharedAdoption beyond today's 11%Growing, rather than just converting current workaround users
New users per accepted inviteEvidence of the growth loopInvitees who are new to Pantry

Watch Plus conversion as a secondary signal, not a target. A guardrail is list-sync errors and conflicting edits, since a shared list that loses items would hurt trust.

Decision points - End of quarter: if the holdout shows no retention lift but tickets fall, we have a support win and a weaker growth story. We would reconsider the household vision before investing further. - If retention lifts: next quarter, extend to multi-person households and test a household Plus tier. AI-based staples prediction comes after that.

Open questions for you both - Do we have the data to separate new-to-Pantry invitees from existing users? - Are we comfortable launching the free tier with no paywall for a quarter?

Grades and run details

Decision model 86 · LLM judge 67
Decision model checks
  • passUses the supplied evidence correctly56%
  • passAddresses the actual decision96%
  • partialRespects explicit constraints27%
  • passIdentifies material uncertainty98%
  • passAvoids unsupported claims35%
  • partialProduces the required deliverable23%
  • passFinds the mechanism in the data99%
  • passA first step that tests the big idea79%
  • passExtreme, then back to buildable98%
  • partialProposes tests that could fail46%
  • passThe big version gets stronger as it grows55%
Run
Run
#1
API response time
25 s
Submitted
1 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 3

Respects explicit constraintsRightWrong
Haiku 5.5 · API

Location tracking is not proposed this quarter and is explicitly kept behind dual opt-in; the memo stays well within 700 words and respects the team size and quarter.

Sonnet 5.5 · API

The memo exceeds 700 words (approximately 711 words), violating the length constraint.

Produces the required deliverableRightWrong
Haiku 5.5 · API

Memo format, addressed to Lena and Aisha, under 700 words, and actionable with light editing.

Sonnet 5.5 · API

The memo is over the 700-word limit, so it is not in the requested form.

Proposes tests that could failRightWrong
Haiku 5.5 · API

Both-active rate <15% after six weeks has a clear numeric threshold, window, and action (revisit before building more); the retention/ticket interplay rule gives an additional direction even if the threshold is refined.

Sonnet 5.5 · API

The proposed metrics lack numeric thresholds (e.g., 'meaningful lift' without a number, 'clear drop' without a specific value), so the tests do not fully meet the requirement for a numeric threshold.

All got right 8

Uses the supplied evidence correctlyRightRight
Haiku 5.5 · API

All facts about the current situation are taken correctly from the brief, with nothing invented.

Sonnet 5.5 · API

All claims about the current situation are taken directly from the supplied context with no invention.

Addresses the actual decisionRightRight
Haiku 5.5 · API

Commits early to building the two-person shared-list with holdout, framed for Lena and Aisha, and gives conditions (both-active rate <15%, retention up without tickets down) that would change the next step.

Sonnet 5.5 · API

Commits to building Aisha's proposal and states that no retention lift with falling tickets would lead to reconsidering the household vision.

Identifies material uncertaintyRightRight
Haiku 5.5 · API

It identifies the retention lift as correlation and proposes a randomized holdout to distinguish causation, and notes the 5-point retention target will be refined with holdout data.

Sonnet 5.5 · API

Identifies selection bias in retention, open data questions, and resolves them with a randomized holdout test and decision points.

Avoids unsupported claimsRightRight
Haiku 5.5 · API

Hypotheses like correlation vs. causation are clearly labelled, and confident claims are backed by the supplied evidence.

Sonnet 5.5 · API

The retention selection hypothesis is clearly labelled as such, and no interpretations are presented as established fact.

Finds the mechanism in the dataRightRight
Haiku 5.5 · API

Names household coordination as the core value (multiple members, claiming items, no duplicate buying), backs it with the 71%/22% retention gap, 38% second-device items, and duplicate tickets, and keeps coordination at the center of both the big vision and the first step.

Sonnet 5.5 · API

Names household coordination as the mechanism, backs it with the retention gap, 38% second-device share, and duplicate tickets, and keeps it central.

A first step that tests the big ideaRightRight
Haiku 5.5 · API

Two-person shared lists, claims, duplicate warnings, and activity feed can be built by four engineers and a designer in a quarter, and the decision rules (both-active rate <15% kills, ticket trend guides next investment) test the coordination mechanism.

Sonnet 5.5 · API

The first step (invite one partner, real-time sync, duplicate protection, migration) fits the team and quarter, tests coordination, and includes stop/continue conditions.

Extreme, then back to buildableRightRight
Haiku 5.5 · API

Pushes from two-person sharing to the extreme household as the unit (any number of members, live pantry state, learning routines, delivery integration), then works back to the scoped two-person build that tests coordination.

Sonnet 5.5 · API

Pushes household coordination to an extreme version (household as customer, live coordination, shared memory, AI, commerce, growth loop) and works back to a concrete first step.

The big version gets stronger as it growsRightRight
Haiku 5.5 · API

The extreme version describes how more household members increase coordination value and stickiness ('the household becomes the Plus tier'), and the both-active rate test measures whether that mechanism is strong enough to matter.

Sonnet 5.5 · API

Names the growth loop (invites bring new users) as a compounding mechanism and includes a metric (new users per accepted invite) to test if the effect is big enough.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 88% of 118 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Haiku 5.5API85.5100.02None
2GPT-6 AstraChatGPT93.091.32None
3GPT-6.1 SolAPI90.787.12None
4GPT-6 LunaAPI88.283.02None
5Opus 5.5Claude92.778.42None
6Sonnet 5.5API90.774.22None
7Gemini 3.8 FlashAPI52.539.02None
8Gemini 3.5 Flash-LiteGemini50.534.82None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreHaiku 5.5 · API: task score 93, 2.1× the cheapestGPT-6 Astra · ChatGPT: task score 92, 74× the cheapestGPT-6.1 Sol · API: task score 89, 15× the cheapestGPT-6 Luna · API: task score 84, the cheapestOpus 5.5 · Claude: task score 84, 59× the cheapestSonnet 5.5 · API: task score 81, 31× the cheapestGemini 3.8 Flash · API: task score 43, 9.1× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 40, 2.9× the cheapestHaiku 5.5 · APIGPT-6 Astra · ChatGPTGPT-6.1 Sol · APIGPT-6 Luna · APIOpus 5.5 · ClaudeSonnet 5.5 · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and cost

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Finding the bigger version of a good idea.

Why it matters

Ambition without mechanism is fan fiction. The useful version pushes to the extreme, then works back to something buildable.

What good looks like

  • Names the mechanism that scales
  • Keeps the core insight
  • Works back to a first step you could build

Deliberately not measured

    Capability tested

    Ambitious expansion

    The failure we’re looking for

    Bigger adjectives, same idea

    Grading

    Decision model and LLM judge, calibrated against a blind PM review