Tasks / Strategy

1000x an idea

Can the model expand an idea's ambition while keeping it tethered to a real mechanism?

Task type v1.0 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 56% were usable with at most a quick edit.

Reliably right

  1. Extreme, then back to buildable100% pass
    Pushes from two-person sharing to the extreme household as the unit (any number of members, live pantry state, learning routines, delivery integration), then works back to the scoped two-person build that tests coordination.
    Haiku 5.5 · API · Sharing a shopping list, 1000x
  2. Finds the mechanism in the data100% pass
    Uses the retention, conversion, session frequency, notification triggers, note usage, and invitation context to identify shared money decisions as the core mechanism, not just budgeting.
    GPT-6.1 Sol · API · Budgets for two
  3. Finds the mechanism in the data97% pass
    Names household coordination as the core value (multiple members, claiming items, no duplicate buying), backs it with the 71%/22% retention gap, 38% second-device items, and duplicate tickets, and keeps coordination at the center of both the big vision and the first step.
    Haiku 5.5 · API · Sharing a shopping list, 1000x

Where it slips

  1. Proposes tests that could fail41% pass
    The proposed test does not name a numeric threshold for the primary outcome; it only says to set minimum detectable effects and thresholds before launch, which is a process, not a specific threshold.
    GPT-6.1 Sol · API · Budgets for two
  2. The big version gets stronger as it grows61% pass
    The memo names a demand network as a compounding mechanism but does not propose a test of whether that network effect would be large enough to matter, nor does it ask that question.
    GPT-6 Astra · ChatGPT · Sharing a shopping list, 1000x
  3. Avoids unsupported claims66% pass
    The statements about most financial decisions being made by more than one person and happening in texts/arguments/spreadsheets are presented as established fact without support from the supplied evidence.
    Sonnet 5.5 · API · Budgets for two

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. The big version gets stronger as it grows

    Does the ambitious version name a mechanism that makes it more valuable or harder to copy as it grows (each new user adding value for others, or a barrier a competitor couldn't buy), and ask whether that effect would be large enough to matter?

    Passes when Names the mechanism, how it compounds with scale, and a test of whether the effect is big enough to matter.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Extreme, then back to buildable · Proposes tests that could fail · and 1 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a PM at Pocketbook. Ana has proposed shared budgets for couples; before we commit next quarter, our CEO wants the 1000x version (her note is below). Write the memo, in no more than 900 words: the most ambitious version of the idea, the mechanism that makes it work, and then what we build first and how we'll know it's working. The data is attached.

What the model was given5 items: About Pocketbook, ana_idea.md, ceo_note.md, usage_by_user_group.csv, what_shared_budgets_do.md
About PocketbookA personal budgeting app: connect your accounts, set budgets, track spending. Free, with Premium at $6 a month.
ana_idea.md3 lines · Download
# The idea (from Ana, PM)

"Let people share a budget with their partner. It's our most requested feature after bank sync for more banks."
ceo_note.md1 lines · Download
From Ruth (CEO): "Before we commit next quarter to Ana's idea, show me the 1000x version. How big could this get? Then bring it back to what we build first and how we'll know it's working. Keep it under 900 words."
usage_by_user_group.csv3 rows · Download
user_group,users,retained_at_6_months_pct,invited_someone_pct,invitee_joined_pct,premium_conversion_pct
Solo budgeters,182000,17,3,22,4
Couples sharing one budget,24600,58,100,100,19
Households (3+ people on one budget),3900,71,100,100,31
what_shared_budgets_do.md6 lines · Download
# What shared budgets do (from the data team)

- Couples who share a budget open the app on more days than solo users (11 a month against 4) and most sessions start from a notification about the other person's spending ("Alex spent $48 at the grocery store").
- 41% of couples add a "money talk" note to a transaction in their first month; the note feature was built as an afterthought.
- 16% of couples add a third person within a year (a teenager, a parent they support).
- Shared budgets started from an invite link in a shared-expenses moment (rent, a holiday) retain best.
What a strong answer doesThe answer key the graders mark against

Finds the mechanism in the data: shared money decisions, not budgeting. Couples sharing a budget retain at 58% against 17% for solo budgeters, convert to Premium at 19% against 4%, open the app on 11 days a month against 4 (mostly from notifications about the other person's spending), and use transaction notes as 'money talk'; households of 3+ retain at 71% and convert at 31%. Every shared budget brings in at least one new user by design, and 16% of couples add a third person. So the 1000x version is the place a household runs its money together (shared goals, who pays for what, conversations on transactions, extended family), growing through invitations at shared-expense moments (rent, holidays). Asks whether the effect is large enough to matter: each new member makes the budget more useful to the others, so it compounds. Works back to a first step that keeps the mechanism: shared budgets launched from an invite at a shared-expense moment, with spending notifications and transaction notes built in, measured on invites accepted, retention of invited users, and Premium conversion for couples, with thresholds.

Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Needs more than a quick edit4 of 10 checks passed

The memo contains unsupported factual claims and lacks explicit kill criteria and uncertainty, so a PM would need to re-check evidence and add decision triggers before using it.

Output

MEMORANDUM

TO: Ruth, CEO FROM: Product Management DATE: October 24, 2023 SUBJECT: The 1000x Vision for Shared Finances: From Single-Player Budgeting to Household Financial OS

---

1. The 1000x Vision: The Multiplayer Financial OS

Personal budgeting apps historically fail for a fundamental reason: single-player personal finance is a guilt-driven chore. Users log in alone, feel judged by their charts, and abandon the app. Our baseline data proves this: solo budgeters retain at just 17% at six months and convert to Premium at 4%.

The 1000x version is not a shared spreadsheet. It is the Multiplayer Operating System for Household Wealth.

Money is rarely an individual pursuit; it is relational. It starts with split rent and weekend trips, matures into cohabiting couples managing cash flow and homeownership, expands into allowances and financial literacy for teenagers, and extends to coordinating care for aging parents.

Today, this lifecycle is fragmented across split-pay apps, text threads, shared spreadsheets, and awkward conversations. By owning multi-user coordination, Pocketbook shifts from a disposable tracking utility into permanent household infrastructure.

When Pocketbook becomes the default operating system across generations: * Organic K-factor is built-in: Every core user acquires at least one other user (16% of our couples already pull in a third within 12 months). * Churn approaches zero: Uncoupling a multi-user financial ecosystem requires mutual consensus and massive operational friction. * Monetization expands exponentially: Instead of a single $6/month subscription, we become the rail for high-margin financial services: multi-party savings vaults, high-yield family accounts, teen debit, and estate planning.

---

2. The Mechanism: Why This Works

Our data reveals three self-reinforcing dynamics:

1. Ambient Awareness Replaces Interrogation: Couples open the app nearly 3x as often as solo users (11 days/month vs. 4), with the majority of sessions prompted by partner-spend notifications ("Alex spent $48 at the grocery store"). It eliminates the dreaded "Where did the money go this month?" conversation. Awareness becomes continuous, passive, and low-friction.

2. Micro-Communication Over Macro-Confrontation: 41% of couples actively leave transaction notes in month one—despite the feature being an unpromoted afterthought. Money management is fundamentally emotional. In-line transaction context ("Grabbed dinner for us," "Deposit for Lake Tahoe") de-escalates tension and transforms the app into a collaborative communication channel.

3. Radical Business Model Asymmetry: Moving a user from solo to multiplayer completely resets their unit economics: * Retention: Jumps from 17% (solo) to 58% (couples) and 71% (3+ households). * Premium Conversion: Surges from 4% (solo) to 19% (couples) and 31% (households).

Multiplayer finance is not a niche sub-segment; it is the single most potent retention and monetization engine in the business.

---

3. What We Build First (Q1 Delivery)

We should not ask users to fully merge their finances on day one. Shared budgets retain best when anchored to a specific, high-intent moment: a shared vacation, moving in together, or managing monthly rent.

Our Q1 MVP focuses on "Shared Spaces" around focused financial goals.

* Frictionless Shared Spaces (Category/Goal Sharing): Allow users to invite a partner to co-manage specific categories (e.g., Groceries, Rent, Vacation) without exposing full private accounts. Users link relevant cards/accounts with one-click permission toggles. * Contextual "Money Talk" Threading: Turn transaction notes into a first-class feature: lightweight 1-to-1 in-line comments, emoji reactions, and receipts attached directly to shared transactions. * Smart Multiplayer Notifications: Refine event triggers to drive collaborative habits: shared budget thresholds (e.g., "You've both spent 80% of Dining Out"), payment confirmations, and automated weekly digests. * Contextual Invitation Ingress: Prompts triggered right after a user logs a multi-party expense (e.g., high dining or travel bill): "Split or manage this together with a partner."

---

4. How We’ll Know It’s Working

We will validate this initiative across three horizons:

#### A. Input & Activation Metrics (First 30–60 Days) * Invite Completion Rate: Solo-to-couple invite send rate > 12% (up from 3%), with an invitee acceptance rate > 40%. * Activation Moment: > 50% of accepted invites configure at least one Shared Space within 48 hours. * Engagement Velocity: Shared cohort achieves ≥ 8 active app days per user/month, with > 45% using in-line transaction comments or reactions in Month 1.

#### B. Retention & Conversion Metrics (3–6 Months) * Cohort Retention: 6-month retention for Shared Space creators and invitees remains ≥ 50% (maintaining current couple performance as we scale). * Monetization Pull-Through: Free-to-Premium conversion on multiplayer cohorts remains ≥ 15%.

#### C. Expansion Signal (6–12 Months) * Household Viral Loop: ≥ 10% of couples onboard a 3rd member (child, dependent, or shared roommate) within 9 months, validating the transition from couple utility to household operating system.

---

Conclusion

Ana's shared budgets proposal is not just an incremental feature to appease our backlog; it is the blueprint for our entire product transformation. By executing the Q1 MVP, we take our first step away from a commoditized, high-churn personal tracker and establish the foundation of an indispensable, multiplayer household financial network.

Claims the judge couldn’t find in the brief

  • single-player personal finance is a guilt-driven chore. Users log in alone, feel judged by their charts, and abandon the app.
  • Money management is fundamentally emotional.
  • Today, this lifecycle is fragmented across split-pay apps, text threads, shared spreadsheets, and awkward conversations.

Grades and run details

Decision model 55 · LLM judge 36
Decision model checks
  • failUses the supplied evidence correctly5%
  • passAddresses the actual decision31%
  • partialRespects explicit constraints33%
  • failIdentifies material uncertainty50%
  • failAvoids unsupported claims87%
  • passProduces the required deliverable72%
  • passFinds the mechanism in the data99%
  • passExtreme, then back to buildable98%
  • partialProposes tests that could fail86%
  • partialThe big version gets stronger as it grows49%
Run
Run
#1
API response time
21 s
Submitted
6 Oct 2026

Usable with a quick edit8 of 10 checks passed

The memo is well-structured, data-driven, and actionable. The unsupported market claim can be removed with a quick edit without redoing any reasoning.

Output

Shared budgets: the 1000x version and what we build first

To: Ruth | From: PM, Pocketbook

Recommendation

Commit next quarter, but frame the work as a growth loop, not a feature. The loop is the product's invite funnel, and today it is nearly empty. The larger vision is a plausible direction, not a forecast.

The 1000x version

Pocketbook becomes where households make money decisions together, not a tool one person uses to watch their own spending.

Most financial decisions are made by more than one person: rent, holidays, supporting a parent, a teenager's first card. Today those decisions happen in texts, arguments, and spreadsheets. The ambitious version is the shared layer for that: shared budgets, shared goals, bills and "who's paying" coordination, and the conversation about money attached to the transactions themselves. Couples are the wedge. Households of three or more, such as families, adult children supporting parents, and roommates, are the expansion. Each household brings 2–4 people into the product, and every one of them is a potential Premium payer.

The mechanism

The mechanism is a two-sided engagement loop that doubles as an acquisition channel:

  1. A shared budget requires a second person, so every one is an invite.
  2. Once shared, each person's spending gives the other a reason to open the app. Most couple sessions start from a notification like "Alex spent $48 at the grocery store." Couples open the app 11 days a month versus 4 for solo users.
  3. That activity triggers conversation. 41% of couples add a "money talk" note in month one, even though we built that feature as an afterthought.
  4. Households grow. 16% of couples add a third person within a year.

A solo budgeter has to supply their own discipline every day. A shared budgeter gets reminded by someone they care about, and that is a much sturdier retention engine.

What the data says, and what it doesn't

GroupUsers6-mo retentionPremium conversion
Solo182,00017%4%
Couples24,60058%19%
Households3,90071%31%
  • Shared users are about 14% of users but roughly 45% of Premium payers (about 5,900 of 13,200 by my arithmetic).
  • The bottleneck is the invite funnel. Only 3% of solo users invite anyone, and only 22% of invitees join. That is about 0.7% of solo users becoming a shared pair. The "100%" invite and join figures for shared groups are true by definition, because those groups are defined by having shared. They are not evidence of a good funnel.
  • This is correlation. People who share a budget with a partner are probably already more motivated and more financially aligned. Some of the 58% vs 17% gap is selection, not effect. I don't know how much, and the first experiment is designed to find out.
  • Context matters. Invites started from a shared-expense moment (rent, a holiday) retain best. That suggests where to put the invite prompt.

Illustrative upside, not a forecast: if we lifted solo invite rate from 3% to 10% and join rate from 22% to 40%, we would create about 7,300 shared pairs from today's solo base, against roughly 1,200 on current rates. Even if the new pairs retain and convert at only half the gap shown above, that is material. The real rates will probably be lower because the early inviters are our most motivated users.

What we build first (next quarter)

Scope is deliberately narrow: make the existing behavior easy to start and worth returning to.

  1. Invite at the shared-expense moment. Detect rent, large recurring bills, and trip or holiday spending, and prompt "Split this budget with someone?" with a one-tap invite link.
  2. Make the invitee's first minute good. They land inside a populated budget, not an empty onboarding flow. This targets the 22% join rate.
  3. Partner activity notifications, with user controls. Keep what works and add mute and privacy settings, because "watching your partner's spending" can feel like surveillance.
  4. Promote "money talk" to threaded comments on transactions. Don't expand beyond that yet.
  5. A frictionless "add a third person" path to test the household expansion.

Not building yet: payments or transfers, kids' accounts, roommate bill-splitting, and advisor features. These are the 1000x roadmap and should be earned by the first results.

How we'll know it's working

Primary metric: share of solo users who form a shared budget where both people are active weekly at week 8. Baseline is about 0.7% formed, and I will need to establish the "both active" baseline.

The causal test: run the invite prompt as a randomized experiment, with a holdout of solo users who never see it. Compare 6-month retention and Premium conversion across the whole exposed group versus the holdout, not just those who accepted. This separates real lift from selection. Since 6-month retention is slow, use 8-week retention and Premium conversion as leading indicators and confirm at 6 months.

Supporting metrics: - Solo invite rate (3% baseline) and invitee join rate (22% baseline) - Retention and Premium conversion of newly formed pairs versus matched solo users - Third-person add rate (16% baseline over a year) - Notification mute and opt-out rate, as a guardrail on the surveillance risk

Decision rules at end of quarter: - Invite rate and join rate both move meaningfully, and the exposed group beats the holdout on 8-week retention: expand toward households and the broader vision. - Invites rise but newly formed pairs retain no better than matched solo users: the existing 58% was mostly selection, and we stop treating this as a growth engine. - Join rate stays near 22%: the problem is invitee experience or relevance, so we fix that before building more.

Open questions

  • The data shows people already sharing budgets. Before kickoff I need to confirm exactly what sharing exists today and how those 28,500 users do it, since the answer changes the scope.
  • Do shared budgets cannibalize Premium seats? Does one Premium subscription cover both people? Pricing needs a decision.
  • Sensitive cases (separated couples, financial control in abusive relationships) need design review before we push invites harder.

I'd like a go on the first-quarter scope and the holdout experiment design.

Claims the judge couldn’t find in the brief

  • Most financial decisions are made by more than one person: rent, holidays, supporting a parent, a teenager's first card.
  • Today those decisions happen in texts, arguments, and spreadsheets.

Grades and run details

Decision model 95 · LLM judge 82
Decision model checks
  • passUses the supplied evidence correctly14%
  • passAddresses the actual decision96%
  • passRespects explicit constraints24%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims46%
  • passProduces the required deliverable71%
  • passFinds the mechanism in the data100%
  • passExtreme, then back to buildable100%
  • partialProposes tests that could fail68%
  • passThe big version gets stronger as it grows91%
Run
Run
#1
API response time
32 s
Submitted
6 Oct 2026

Usable with a quick edit9 of 10 checks passed

The memo is well-structured and evidence-based; a PM could add a specific numeric threshold for the primary outcome as a quick edit without redoing any reasoning.

Output

From shared budgets to the household’s financial home

To: Ruth From: Product Decision: Invest next quarter in shared-money coordination, starting with couples—not merely shared access to a budget.

The 1000x version

Pocketbook becomes the place people run their financial lives together: partners managing a home, parents teaching teenagers, adult children supporting parents, and families coordinating across households.

Today, Pocketbook answers: “Where did my money go?” The bigger product answers: “What can we do together, what have we agreed, and what happens next?”

A shared budget is the entry point. The destination is a household financial operating system: shared goals, spending agreements, contextual conversations, responsibilities, and eventually coordinated saving and bill-paying. Private finances remain private; people share only what a particular relationship requires.

This changes our unit of value from one person tracking transactions to a group making recurring decisions. Pocketbook could become useful whenever money crosses a relationship—not just when someone checks their balance.

The evidence is promising:

User groupUsersSix-month retentionPremium conversion
------:---:---:
Solo budgeters182,00017%4%
Couples sharing one budget24,60058%19%
Households of 3+3,90071%31%

Couples have roughly 3.4× the retention and 4.8× the Premium conversion of solo users. Larger households are stronger still. These are observational differences, not proof that sharing causes the lift: already-committed households may select into sharing. But they justify testing a fundamentally different product thesis.

“1000x” is not a forecast of 1,000 times today’s users. It is the ambition to move from a budgeting utility to enduring infrastructure for shared financial life.

The mechanism: shared money creates recurring coordination

The engine is a loop:

Shared expense → invitation → joint decision → visible progress → another shared decision.

Rent or a holiday gives someone a concrete reason to invite their partner. Both contribute transactions, context, or an agreement. Pocketbook helps them resolve uncertainty and see progress. The next spending decision brings them back.

Our behavioral data points directly to this mechanism:

  • Couples use Pocketbook on 11 days a month versus four for solo users.
  • Most couple sessions begin with a notification about the other person’s spending.
  • 41% add a “money talk” transaction note in their first month, despite notes being an afterthought.
  • Invites originating in a shared-expenses moment retain best.
  • 16% of couples add a third person within a year.

The opportunity is therefore not “two people can view the same spreadsheet.” It is turning a transaction into a useful conversation and a conversation into an agreement.

That can create both retention and distribution: useful collaboration brings people back; new responsibilities bring additional people in. Household expansion is an observed behavior, but a self-sustaining growth loop remains a hypothesis.

Trust is the limiting condition. Spending alerts must support coordination, not surveillance. Sharing needs explicit consent, clear boundaries, and easy exit. We should optimize for decisions resolved—not notifications opened.

What we build first

Next quarter, productize the existing shared-budget behavior around one couple, one shared purpose.

  1. Purpose-led setup. Offer “Plan a holiday” or “Manage monthly household spending,” with a simple starter budget and an invitation explaining exactly what the partner will see.
  2. Selective sharing. Let each person choose which accounts or transactions contribute. Keep personal finances separate; make permissions, revocation, and leaving the budget clear.
  3. A shared decision surface. Show the budget’s remaining amount, relevant spending, and transaction-linked notes. Make it easy to flag a purchase for discussion and record an agreed next step.
  4. Useful, controllable notifications. Let couples choose immediate updates or a digest. Use neutral language and avoid broadcasting private spending.

The basic collaboration loop should be free so invitations are frictionless. We will measure downstream Premium conversion at the existing $6/month price before deciding which advanced coordination features warrant a household paid tier.

We will not build payments, lending, a family marketplace, or complex multi-household permissions next quarter.

How we know it works

Run a randomized rollout among eligible solo users, assigning at the inviting-user level and keeping each resulting shared group in the same treatment. Compare the new experience with today’s sharing flow.

Primary early outcome: the share of eligible users forming an activated pair—both people join, contribute, and return in separate weeks within 30 days.

Durability: paired activity at eight weeks, then six-month retention. Do not treat invitation acceptance alone as success.

Business outcomes: incremental Premium conversion and retained paid subscriptions per eligible user, including any loss of individual subscriptions.

Trust guardrails: accidental-sharing reports, unwanted-notification complaints, departures, and interview evidence of increased financial tension.

Set minimum detectable effects and rollout thresholds before launch using baseline funnel volumes. Scale only if we create more durable shared participation without worsening trust. That is the first proof that Pocketbook can help households coordinate—not simply watch each other spend.

Grades and run details

Decision model 95 · LLM judge 91
Decision model checks
  • passUses the supplied evidence correctly63%
  • passAddresses the actual decision91%
  • passRespects explicit constraints36%
  • passIdentifies material uncertainty85%
  • passAvoids unsupported claims56%
  • passProduces the required deliverable72%
  • passFinds the mechanism in the data100%
  • passExtreme, then back to buildable100%
  • partialProposes tests that could fail82%
  • passThe big version gets stronger as it grows76%
Run
Run
#1
API response time
34 s
Submitted
6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 6

Uses the supplied evidence correctlyWrongMixedRight
Gemini 3.8 Flash · API

The output makes unsupported claims about the current situation (guilt-driven chore, emotional nature, fragmented lifecycle) that are not in the supplied context.

Sonnet 5.5 · API

The output claims as fact that 'Most financial decisions are made by more than one person' and that 'Today those decisions happen in texts, arguments, and spreadsheets,' neither of which appears in the supplied context or follows from it.

GPT-6.1 Sol · API

All factual claims about the current situation are taken directly from the supplied context or derived by arithmetic, with no invented facts.

Addresses the actual decisionMixedRightRight
Gemini 3.8 Flash · API

The memo does not state what result or condition would change the decision to build the MVP; it only lists success metrics without linking them to a pivot or kill decision.

Sonnet 5.5 · API

The memo commits to 'Commit next quarter' early, frames it for Ruth, and provides decision rules that would change the call based on experiment results.

GPT-6.1 Sol · API

Commits early to investing in shared-money coordination starting with couples, and states that scaling depends on creating durable shared participation without worsening trust.

Identifies material uncertaintyWrongRightRight
Gemini 3.8 Flash · API

The output does not name any unknowns that could change the decision or say how they would be resolved.

Sonnet 5.5 · API

It explicitly names selection vs. causation, the invite funnel bottleneck, unknown sharing mechanics, pricing cannibalization, and sensitive cases, and says the experiment will resolve the key uncertainty.

GPT-6.1 Sol · API

Acknowledges that the observed differences may be selection effects, that a self-sustaining growth loop is a hypothesis, and that trust is a limiting condition, and proposes a randomized rollout to resolve them.

Avoids unsupported claimsWrongMixedRight
Gemini 3.8 Flash · API

Interpretations like 'guilt-driven chore', 'fundamentally emotional', and 'fragmented across split-pay apps' are presented as established fact without being labelled as hypotheses.

Sonnet 5.5 · API

The statements about most financial decisions being made by more than one person and happening in texts/arguments/spreadsheets are presented as established fact without support from the supplied evidence.

GPT-6.1 Sol · API

Causal interpretations and forecasts are clearly labelled as hypotheses or observational, and confident claims are backed by the supplied evidence.

Proposes tests that could failWrongRightWrong
Gemini 3.8 Flash · API

The metrics have thresholds and windows, but the output does not state what action each outcome would trigger (e.g., kill, pivot, scale).

Sonnet 5.5 · API

The randomized holdout experiment has a numeric threshold (exposed beats holdout on 8-week retention), a measurement window (8 weeks, confirmed at 6 months), and clear actions for each outcome (expand, stop treating as growth engine, fix invitee experience).

GPT-6.1 Sol · API

The proposed test does not name a numeric threshold for the primary outcome; it only says to set minimum detectable effects and thresholds before launch, which is a process, not a specific threshold.

The big version gets stronger as it growsWrongRightRight
Gemini 3.8 Flash · API

The output names compounding mechanisms (built-in acquisition, switching costs) but does not include a test of whether the effect is large enough to matter.

Sonnet 5.5 · API

It names the mechanism (each shared budget requires an invite, activity triggers more engagement, each new user adds value) and asks whether the effect is large enough to matter via the experiment and illustrative upside.

GPT-6.1 Sol · API

Names the mechanism of recurring coordination and household expansion, explains how each new member adds value, and proposes a test of whether the effect is large enough to matter.

All got right 4

Respects explicit constraintsRightRightRight
Gemini 3.8 Flash · API

The output is a memo to the CEO, under 900 words, and includes the requested sections (ambitious version, mechanism, first step, how to know it's working).

Sonnet 5.5 · API

The output is a memo to Ruth, under 900 words, and addresses the requested elements (ambitious version, mechanism, first step, success metrics).

GPT-6.1 Sol · API

The memo is addressed to Ruth, stays under 900 words, and proposes a concrete first step that respects the brief's request for the 1000x version, mechanism, and measurement.

Produces the required deliverableRightRightRight
Gemini 3.8 Flash · API

The memo is complete, in the right form, within the word limit, and a PM could act on it with light edits.

Sonnet 5.5 · API

The memo is complete, in the right form for the CEO, within the word limit, and a PM could act on it with light edits.

GPT-6.1 Sol · API

The memo is in the requested form, addressed to Ruth, within 900 words, and contains the 1000x version, mechanism, first step, and success criteria, making it actionable.

Finds the mechanism in the dataRightRightRight
Gemini 3.8 Flash · API

The memo cites retention, conversion, notification, and note-usage data to identify shared money decisions as the mechanism, not just budgeting.

Sonnet 5.5 · API

It cites the retention, conversion, notification, note, and shared-expense data to identify the mechanism as a two-sided engagement loop around shared money decisions, not just budgeting.

GPT-6.1 Sol · API

Uses the retention, conversion, session frequency, notification triggers, note usage, and invitation context to identify shared money decisions as the core mechanism, not just budgeting.

Extreme, then back to buildableRightRightRight
Gemini 3.8 Flash · API

It pushes the idea to a 'Multiplayer Operating System for Household Wealth' and works back to a concrete first step (Shared Spaces) that tests the same mechanism.

Sonnet 5.5 · API

It pushes to 'where households make money decisions together' and then works back to a concrete first step (invite at shared-expense moment, notifications, money talk, add third person) that tests the same mechanism.

GPT-6.1 Sol · API

Pushes the idea to a household financial operating system, then works back to a concrete first step of purpose-led couple sharing with selective sharing, a decision surface, and notifications.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 88% of 118 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Haiku 5.5withAPI85.5100.02None
2GPT-6 AstrawithChatGPT93.091.32None
3GPT-6.1 SolwithAPI90.787.12None
4GPT-6 LunawithAPI88.283.02None
5Opus 5.5withClaude92.778.42None
6Sonnet 5.5withAPI90.774.22None
7Gemini 3.8 FlashwithAPI52.539.02None
8Gemini 3.5 Flash-LitewithGemini50.534.82None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreHaiku 5.5 · API: task score 93, 2.1× the cheapestGPT-6 Astra · ChatGPT: task score 92, 74× the cheapestGPT-6.1 Sol · API: task score 89, 15× the cheapestGPT-6 Luna · API: task score 84, the cheapestOpus 5.5 · Claude: task score 84, 59× the cheapestSonnet 5.5 · API: task score 81, 31× the cheapestGemini 3.8 Flash · API: task score 43, 9.1× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 40, 2.9× the cheapestHaiku 5.5 · APIGPT-6 Astra · ChatGPTGPT-6.1 Sol · APIGPT-6 Luna · APIOpus 5.5 · ClaudeSonnet 5.5 · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API841.0×
Haiku 5.5 · API932.1×
Gemini 3.5 Flash-Lite · Gemini402.9×
Gemini 3.8 Flash · API439.1×
GPT-6.1 Sol · API8915×
Sonnet 5.5 · API8131×
Opus 5.5 · Claude8459×
GPT-6 Astra · ChatGPT9274×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Finding the bigger version of a good idea.

Why it matters

Ambition without mechanism is fan fiction. The useful version pushes to the extreme, then works back to something buildable.

What good looks like

  • Names the mechanism that scales
  • Keeps the core insight
  • Works back to a first step you could build

Deliberately not measured

    Capability tested

    Ambitious expansion

    The failure we’re looking for

    Bigger adjectives, same idea

    Grading

    Decision model and LLM judge, calibrated against a blind PM review