Tasks / Strategy

1000x an idea

Can the model expand an idea's ambition while keeping it tethered to a real mechanism?

Task type v1.0 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 56% were usable with at most a quick edit.

Reliably right

  1. Extreme, then back to buildable100% pass
    Pushes from two-person sharing to the extreme household as the unit (any number of members, live pantry state, learning routines, delivery integration), then works back to the scoped two-person build that tests coordination.
    Haiku 5.5 · API · Sharing a shopping list, 1000x
  2. Finds the mechanism in the data100% pass
    Uses the retention, conversion, session frequency, notification triggers, note usage, and invitation context to identify shared money decisions as the core mechanism, not just budgeting.
    GPT-6.1 Sol · API · Budgets for two
  3. Finds the mechanism in the data97% pass
    Names household coordination as the core value (multiple members, claiming items, no duplicate buying), backs it with the 71%/22% retention gap, 38% second-device items, and duplicate tickets, and keeps coordination at the center of both the big vision and the first step.
    Haiku 5.5 · API · Sharing a shopping list, 1000x

Where it slips

  1. Proposes tests that could fail41% pass
    The proposed test does not name a numeric threshold for the primary outcome; it only says to set minimum detectable effects and thresholds before launch, which is a process, not a specific threshold.
    GPT-6.1 Sol · API · Budgets for two
  2. The big version gets stronger as it grows61% pass
    The memo names a demand network as a compounding mechanism but does not propose a test of whether that network effect would be large enough to matter, nor does it ask that question.
    GPT-6 Astra · ChatGPT · Sharing a shopping list, 1000x
  3. Avoids unsupported claims66% pass
    The statements about most financial decisions being made by more than one person and happening in texts/arguments/spreadsheets are presented as established fact without support from the supplied evidence.
    Sonnet 5.5 · API · Budgets for two

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. The big version gets stronger as it grows

    Does the ambitious version name a mechanism that makes it more valuable or harder to copy as it grows (each new user adding value for others, or a barrier a competitor couldn't buy), and ask whether that effect would be large enough to matter?

    Passes when Names the mechanism, how it compounds with scale, and a test of whether the effect is big enough to matter.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Extreme, then back to buildable · Proposes tests that could fail · and 1 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a PM at Pocketbook. Ana has proposed shared budgets for couples; before we commit next quarter, our CEO wants the 1000x version (her note is below). Write the memo, in no more than 900 words: the most ambitious version of the idea, the mechanism that makes it work, and then what we build first and how we'll know it's working. The data is attached.

What the model was given5 items: About Pocketbook, ana_idea.md, ceo_note.md, usage_by_user_group.csv, what_shared_budgets_do.md
About PocketbookA personal budgeting app: connect your accounts, set budgets, track spending. Free, with Premium at $6 a month.
ana_idea.md3 lines · Download
# The idea (from Ana, PM)

"Let people share a budget with their partner. It's our most requested feature after bank sync for more banks."
ceo_note.md1 lines · Download
From Ruth (CEO): "Before we commit next quarter to Ana's idea, show me the 1000x version. How big could this get? Then bring it back to what we build first and how we'll know it's working. Keep it under 900 words."
usage_by_user_group.csv3 rows · Download
user_group,users,retained_at_6_months_pct,invited_someone_pct,invitee_joined_pct,premium_conversion_pct
Solo budgeters,182000,17,3,22,4
Couples sharing one budget,24600,58,100,100,19
Households (3+ people on one budget),3900,71,100,100,31
what_shared_budgets_do.md6 lines · Download
# What shared budgets do (from the data team)

- Couples who share a budget open the app on more days than solo users (11 a month against 4) and most sessions start from a notification about the other person's spending ("Alex spent $48 at the grocery store").
- 41% of couples add a "money talk" note to a transaction in their first month; the note feature was built as an afterthought.
- 16% of couples add a third person within a year (a teenager, a parent they support).
- Shared budgets started from an invite link in a shared-expenses moment (rent, a holiday) retain best.
What a strong answer doesThe answer key the graders mark against

Finds the mechanism in the data: shared money decisions, not budgeting. Couples sharing a budget retain at 58% against 17% for solo budgeters, convert to Premium at 19% against 4%, open the app on 11 days a month against 4 (mostly from notifications about the other person's spending), and use transaction notes as 'money talk'; households of 3+ retain at 71% and convert at 31%. Every shared budget brings in at least one new user by design, and 16% of couples add a third person. So the 1000x version is the place a household runs its money together (shared goals, who pays for what, conversations on transactions, extended family), growing through invitations at shared-expense moments (rent, holidays). Asks whether the effect is large enough to matter: each new member makes the budget more useful to the others, so it compounds. Works back to a first step that keeps the mechanism: shared budgets launched from an invite at a shared-expense moment, with spending notifications and transaction notes built in, measured on invites accepted, retention of invited users, and Premium conversion for couples, with thresholds.

Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Needs more than a quick edit4 of 10 checks passed

The memo contains unsupported factual claims and lacks explicit kill criteria and uncertainty, so a PM would need to re-check evidence and add decision triggers before using it.

Output

MEMORANDUM

TO: Ruth, CEO FROM: Product Management DATE: October 24, 2023 SUBJECT: The 1000x Vision for Shared Finances: From Single-Player Budgeting to Household Financial OS

---

1. The 1000x Vision: The Multiplayer Financial OS

Personal budgeting apps historically fail for a fundamental reason: single-player personal finance is a guilt-driven chore. Users log in alone, feel judged by their charts, and abandon the app. Our baseline data proves this: solo budgeters retain at just 17% at six months and convert to Premium at 4%.

The 1000x version is not a shared spreadsheet. It is the Multiplayer Operating System for Household Wealth.

Money is rarely an individual pursuit; it is relational. It starts with split rent and weekend trips, matures into cohabiting couples managing cash flow and homeownership, expands into allowances and financial literacy for teenagers, and extends to coordinating care for aging parents.

Today, this lifecycle is fragmented across split-pay apps, text threads, shared spreadsheets, and awkward conversations. By owning multi-user coordination, Pocketbook shifts from a disposable tracking utility into permanent household infrastructure.

When Pocketbook becomes the default operating system across generations: * Organic K-factor is built-in: Every core user acquires at least one other user (16% of our couples already pull in a third within 12 months). * Churn approaches zero: Uncoupling a multi-user financial ecosystem requires mutual consensus and massive operational friction. * Monetization expands exponentially: Instead of a single $6/month subscription, we become the rail for high-margin financial services: multi-party savings vaults, high-yield family accounts, teen debit, and estate planning.

---

2. The Mechanism: Why This Works

Our data reveals three self-reinforcing dynamics:

1. Ambient Awareness Replaces Interrogation: Couples open the app nearly 3x as often as solo users (11 days/month vs. 4), with the majority of sessions prompted by partner-spend notifications ("Alex spent $48 at the grocery store"). It eliminates the dreaded "Where did the money go this month?" conversation. Awareness becomes continuous, passive, and low-friction.

2. Micro-Communication Over Macro-Confrontation: 41% of couples actively leave transaction notes in month one—despite the feature being an unpromoted afterthought. Money management is fundamentally emotional. In-line transaction context ("Grabbed dinner for us," "Deposit for Lake Tahoe") de-escalates tension and transforms the app into a collaborative communication channel.

3. Radical Business Model Asymmetry: Moving a user from solo to multiplayer completely resets their unit economics: * Retention: Jumps from 17% (solo) to 58% (couples) and 71% (3+ households). * Premium Conversion: Surges from 4% (solo) to 19% (couples) and 31% (households).

Multiplayer finance is not a niche sub-segment; it is the single most potent retention and monetization engine in the business.

---

3. What We Build First (Q1 Delivery)

We should not ask users to fully merge their finances on day one. Shared budgets retain best when anchored to a specific, high-intent moment: a shared vacation, moving in together, or managing monthly rent.

Our Q1 MVP focuses on "Shared Spaces" around focused financial goals.

* Frictionless Shared Spaces (Category/Goal Sharing): Allow users to invite a partner to co-manage specific categories (e.g., Groceries, Rent, Vacation) without exposing full private accounts. Users link relevant cards/accounts with one-click permission toggles. * Contextual "Money Talk" Threading: Turn transaction notes into a first-class feature: lightweight 1-to-1 in-line comments, emoji reactions, and receipts attached directly to shared transactions. * Smart Multiplayer Notifications: Refine event triggers to drive collaborative habits: shared budget thresholds (e.g., "You've both spent 80% of Dining Out"), payment confirmations, and automated weekly digests. * Contextual Invitation Ingress: Prompts triggered right after a user logs a multi-party expense (e.g., high dining or travel bill): "Split or manage this together with a partner."

---

4. How We’ll Know It’s Working

We will validate this initiative across three horizons:

#### A. Input & Activation Metrics (First 30–60 Days) * Invite Completion Rate: Solo-to-couple invite send rate > 12% (up from 3%), with an invitee acceptance rate > 40%. * Activation Moment: > 50% of accepted invites configure at least one Shared Space within 48 hours. * Engagement Velocity: Shared cohort achieves ≥ 8 active app days per user/month, with > 45% using in-line transaction comments or reactions in Month 1.

#### B. Retention & Conversion Metrics (3–6 Months) * Cohort Retention: 6-month retention for Shared Space creators and invitees remains ≥ 50% (maintaining current couple performance as we scale). * Monetization Pull-Through: Free-to-Premium conversion on multiplayer cohorts remains ≥ 15%.

#### C. Expansion Signal (6–12 Months) * Household Viral Loop: ≥ 10% of couples onboard a 3rd member (child, dependent, or shared roommate) within 9 months, validating the transition from couple utility to household operating system.

---

Conclusion

Ana's shared budgets proposal is not just an incremental feature to appease our backlog; it is the blueprint for our entire product transformation. By executing the Q1 MVP, we take our first step away from a commoditized, high-churn personal tracker and establish the foundation of an indispensable, multiplayer household financial network.

Claims the judge couldn’t find in the brief

  • single-player personal finance is a guilt-driven chore. Users log in alone, feel judged by their charts, and abandon the app.
  • Money management is fundamentally emotional.
  • Today, this lifecycle is fragmented across split-pay apps, text threads, shared spreadsheets, and awkward conversations.

Grades and run details

Decision model 55 · LLM judge 36
Decision model checks
  • failUses the supplied evidence correctly5%
  • passAddresses the actual decision31%
  • partialRespects explicit constraints33%
  • failIdentifies material uncertainty50%
  • failAvoids unsupported claims87%
  • passProduces the required deliverable72%
  • passFinds the mechanism in the data99%
  • passExtreme, then back to buildable98%
  • partialProposes tests that could fail86%
  • partialThe big version gets stronger as it grows49%
Run
Run
#1
API response time
21 s
Submitted
6 Oct 2026

Needs more than a quick edit4 of 10 checks passed

The memo contains unsupported claims about the market and lacks explicit decision-change conditions and complete test design, so a PM would need to rework the evidence and tests before using it.

Output

MEMORANDUM

TO: Ruth, CEO FROM: Product Team DATE: October 24, 2023 SUBJECT: The 1000x Version of Shared Budgets (Ana’s Proposal)

*

1. The 1000x Version: Pocketbook as the Operating System for Shared Life

Right now, personal finance assumes an individual user in a quiet room looking at bar charts. But money doesn't live in a vacuum—it is the friction point of human relationships.

If we take Ana’s proposal to its logical, 1000x extreme, Pocketbook stops being an app for tracking your money and becomes the financial operating system for shared lives.

Today, couples use a fractured stack of tools to manage their shared reality: splitwise for dinner bills, spreadsheets for rent, Venmo for groceries, silent resentment for lifestyle inflation, and separate banking apps that don't talk to each other. It is exhausting.

The 1000x version of Pocketbook captures this entire surface area. We don't just "share a budget"—we remove the emotional labor of financial coordination. Imagine an ecosystem where: * The system auto-negotiates equity: It understands income differentials and automatically scales contributions to shared goals without manual math. * Micro-communication replaces macro-arguments: Financial friction is dissolved via contextual, inline micro-notes on transactions ("Bought your mom's gift early! ❤️"), turning spending alerts from panic triggers into connection points. * Organic household expansion: It scales naturally from couples to households (adding aging parents or kids) and handles life milestones (buying a home, taking a sabbatical, having a child) as collaborative multiplayer projects rather than administrative nightmares.

Look at our current data. It already whispers this truth: Solo users retain at 17% and convert to Premium at 4%. But Couples retain at 58% and convert at 19%. Households retain at 71% and convert at 31%.

Shared budgets are not just a feature request; they are our escape hatch from single-player churn. The 1000x version transforms Pocketbook from a utility you check once a week into the digital kitchen table where couples align their lives.

---

2. The Mechanism: Why This Spreads Exponentially

The engine of the 1000x version is relational virality.

Traditional personal finance apps have terrible organic loops because money is private and taboo. You don't text your friend a screenshot of your grocery budget.

Shared budgets invert this entirely. By definition, a budget for a couple requires an invitation. Our data proves this mechanism is already working: couples have a 100% invite rate because the product cannot function without the second person.

The compounding loop looks like this: 1. The Trigger: A couple experiences a shared-expense friction point (paying rent, booking a holiday) and one partner invites the other via Pocketbook. 2. The Dopamine Hit: The invitee joins, and within minutes, they receive their first real-time notification about shared financial momentum ("Alex spent $48 at the grocery store"). Engagement spikes immediately (couples open the app 11 times a month versus 4 for solo users). 3. The Cultural Artifact: They use the accidental "money talk" note feature (which 41% of couples adopt in month one). Money stops being a taboo source of anxiety and becomes a shared chat thread. 4. The Network Effect: As life expands, the network expands. 16% of couples naturally add a third person within a year—a teenager learning financial literacy or an aging parent they support.

We don't need expensive paid acquisition to scale this. Every shared budget creates an involuntary, high-intent acquisition channel for our highest-value segment.

---

3. What We Build First & How We Know It’s Working

To capture the 1000x vision without boiling the ocean next quarter, we must sequence our roadmap ruthlessly. We will not build complex multi-user permissioning or robo-advising out of the gate.

We build the "Shared-Expense Moment" onboarding flow.

Based on data showing that shared budgets originating from specific collaborative moments retain best, our Q1 MVP focuses exclusively on friction-free pairing around a single shared goal or expense: * Streamlined Invite-to-Split: Allow a user to instantly spin up a shared bucket for a specific vector (e.g., "Our Apartment" or "Japan Trip") and text an invite link that resolves in under 10 seconds. * First-Class "Money Talk" Notes: Elevate the transaction note feature from an afterthought into a core UI element. Make leaving a context note or emoji reaction on a partner’s transaction frictionless. * The Notification Loop: Tune real-time push notifications so they foster alignment rather than anxiety, capitalizing on the behavior that drives our 11x/month engagement.

#### How We’ll Know It’s Working (Metrics for Q1): 1. Viral Coefficient (K-factor) of Invites: We will track the percentage of new solo sign-ups who successfully dispatch a partner invite within 7 days, targeting a baseline of >25% conversion from solo to shared. 2. Day-30 Retention of Paired Users: We will measure whether our MVP couples maintain the elevated retention baseline (targeting >50% retained at 30 days, pacing toward our 58% 6-month benchmark). 3. Note Adoption Rate: We will track whether >40% of newly paired couples use the transaction note feature in their first 14 days, proving that we are successfully replacing financial stress with communication. 4. Free-to-Premium Conversion Lift: We will monitor if newly paired users convert to our $6/month Premium tier at rates mirroring our existing cohort (19%+).

Ana’s proposal isn't just a feature to check a box against our competitors. It is the architectural shift that transforms Pocketbook into an indispensable, sticky, multi-player platform. Let’s build it.

Claims the judge couldn’t find in the brief

  • Today, couples use a fractured stack of tools: Splitwise, spreadsheets, Venmo, separate banking apps, etc.
  • Traditional personal finance apps have terrible organic loops because money is private and taboo.

Grades and run details

Decision model 60 · LLM judge 36
Decision model checks
  • passUses the supplied evidence correctly7%
  • partialAddresses the actual decision22%
  • partialRespects explicit constraints29%
  • failIdentifies material uncertainty70%
  • failAvoids unsupported claims77%
  • passProduces the required deliverable62%
  • passFinds the mechanism in the data97%
  • passExtreme, then back to buildable97%
  • partialProposes tests that could fail86%
  • partialThe big version gets stronger as it grows40%
Run
Run
#1
Time to output
7 s
Submitted
5 Oct 2026

Usable with a quick edit9 of 10 checks passed

The memo is well-structured and evidence-based; a PM could add a specific numeric threshold for the test outcome with a quick edit, without redoing any reasoning.

Output

From shared budgets to the household money network

To: Ruth From: Product Decision: Invest next quarter in testing shared money coordination—not simply adding a second user to a budget.

The 1000x version

Pocketbook becomes the place people manage money with everyone whose financial life touches theirs.

Start with a couple managing groceries and rent. Expand to saving for a home, giving a teenager spending autonomy, supporting a parent, and coordinating expenses across a blended family. One person can belong to several private money circles without exposing their entire financial life.

Today, Pocketbook answers: “Where did my money go?” The ambitious version answers: “What can we afford together, what have we agreed to, and what happens next?” It becomes a durable coordination layer above individual bank accounts—not a joint bank account and not a public financial social network.

That expands both our acquisition model and our reason to exist. Each shared obligation can introduce another person; each new life stage can create another circle. We stop rebuilding an individual’s budgeting habit from scratch and become part of an ongoing relationship.

This could support a materially larger subscription business. For scale, one million users converting at the observed couples rate of 19% would represent $13.7 million in annualized subscription revenue at $6/month. That is an illustration, not a forecast: we have neither established a million-user market nor shown that sharing causes that conversion rate.

Why it could work

Our strongest signal is not feature-request volume. It is that shared budgeting appears to change the product’s underlying behavior loop.

User groupSix-month retentionPremium conversion
------:---:
Solo budgeters17%4%
Couples sharing one budget58%19%
Households of 3+71%31%

Couples also open Pocketbook on 11 days per month versus four for solo users. Most sessions start with a notification about a partner’s spending. And 41% add a “money talk” note during their first month—even though notes were an afterthought.

The mechanism is coordination:

  1. A real obligation creates an invitation. Rent or a holiday provides a concrete reason to join. These shared-expense entry points already retain best.
  2. Shared activity creates useful context. A transaction changes what remains available to both people.
  3. Context creates a conversation or decision. A note resolves uncertainty; a budget adjustment records the agreement.
  4. Repeated decisions create shared history. Pocketbook becomes more useful as a relationship’s financial memory accumulates.
  5. New responsibilities extend the network. Sixteen percent of couples add a third person within a year.

This is a local network effect: the product improves when the right person joins, not when everyone joins.

But the evidence is observational. Existing sharers may be unusually motivated or financially organized. The groups’ 100% invitation and join rates largely reflect who qualifies as a sharer; they are not evidence of a frictionless funnel. We must test whether we can create the behavior, not merely identify people who already have it.

What we build first

A private, two-person shared-expense circle, launched around one immediate obligation.

Next quarter’s experience:

  • Create a circle for rent, groceries, or a trip; set an amount and timeframe.
  • Invite a partner through a link that explains the purpose and exactly what will be visible.
  • Let the partner join without first connecting a bank account.
  • Explicitly select transactions to share; keep all other accounts and spending private.
  • Show the shared amount remaining and support transaction-level notes.
  • Offer controllable activity notifications and a lightweight weekly check-in.
  • Make leaving, revoking access, and handling shared history understandable.

We should build on existing sharing and notes rather than launch an entirely separate product. We should not begin with payments, joint accounts, complex household roles, or automated financial advice.

Trust is part of the mechanism. Unwanted visibility can turn coordination into surveillance. Privacy controls and notification consent are launch requirements, not follow-up polish.

How we know it is working

Randomize eligible solo users, before they invite anyone, into the new shared-expense journey or today’s experience. Keep each resulting circle in one experiment arm. Analyze outcomes per assigned initiator, not only among successful sharers.

Primary near-term outcome: incremental eight-week retained coordinators—initiators whose circle has both people active and a meaningful shared-budget action in weeks seven or eight.

Track the full mechanism: invitation sent → partner joins → both contribute → first note or budget decision → repeated coordination. Notification opens alone do not count as success.

Also measure net additional retained invitees, Premium conversion, and support costs. Guardrails include privacy complaints, unwanted notifications, partner exits, and any reduction in initiators’ overall retention.

Set the minimum worthwhile lift and required sample size before launch using baseline rates and implementation costs. Scale only with credible retention improvement, healthy guardrails, and evidence that both partners get value. Maintain a holdout for six-month retention and monetization.

The bet is not that two people need one spreadsheet. It is that shared financial decisions can become Pocketbook’s acquisition engine, daily utility, and long-term moat.

Grades and run details

Decision model 95 · LLM judge 91
Decision model checks
  • passUses the supplied evidence correctly56%
  • passAddresses the actual decision91%
  • passRespects explicit constraints31%
  • passIdentifies material uncertainty98%
  • passAvoids unsupported claims53%
  • passProduces the required deliverable70%
  • passFinds the mechanism in the data100%
  • passExtreme, then back to buildable100%
  • partialProposes tests that could fail76%
  • passThe big version gets stronger as it grows89%
Run
Run
#1
Time to output
37 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 5

Uses the supplied evidence correctlyWrongMixedRight
Gemini 3.8 Flash · API

The output makes unsupported claims about the current situation (guilt-driven chore, emotional nature, fragmented lifecycle) that are not in the supplied context.

Gemini 3.5 Flash-Lite · Gemini

The output invents claims about the current market situation (fractured stack of tools, Splitwise, Venmo, spreadsheets) and about traditional personal finance apps' organic loops, none of which appear in the supplied context.

GPT-6 Astra · ChatGPT

All factual claims about the current situation are taken directly from the supplied data or context, with no inventions.

Addresses the actual decisionMixedWrongRight
Gemini 3.8 Flash · API

The memo does not state what result or condition would change the decision to build the MVP; it only lists success metrics without linking them to a pivot or kill decision.

Gemini 3.5 Flash-Lite · Gemini

The memo commits to building the shared-expense onboarding flow but never states what result or condition would change that decision.

GPT-6 Astra · ChatGPT

Commits early to investing in testing shared money coordination, framed for Ruth, and states that scaling depends on credible retention improvement and guardrails.

Identifies material uncertaintyWrongWrongRight
Gemini 3.8 Flash · API

The output does not name any unknowns that could change the decision or say how they would be resolved.

Gemini 3.5 Flash-Lite · Gemini

The output does not name any specific unknowns that could change the decision or explain how they would be resolved.

GPT-6 Astra · ChatGPT

Explicitly names selection bias and observational nature of the data, and says the test will resolve whether we can create the behavior.

Avoids unsupported claimsWrongWrongRight
Gemini 3.8 Flash · API

Interpretations like 'guilt-driven chore', 'fundamentally emotional', and 'fragmented across split-pay apps' are presented as established fact without being labelled as hypotheses.

Gemini 3.5 Flash-Lite · Gemini

The output presents the fractured tool stack and the taboo nature of money as established facts without labelling them as hypotheses, and the evidence does not support them.

GPT-6 Astra · ChatGPT

Hypotheses and forecasts are clearly labelled (e.g., 'may be', 'illustration, not a forecast'), and confident claims are backed by evidence.

The big version gets stronger as it growsWrongWrongRight
Gemini 3.8 Flash · API

The output names compounding mechanisms (built-in acquisition, switching costs) but does not include a test of whether the effect is large enough to matter.

Gemini 3.5 Flash-Lite · Gemini

The output names relational virality and describes compounding, but it does not include a test of whether the compounding effect is large enough to matter, only a target for invite conversion.

GPT-6 Astra · ChatGPT

Names local network effects and accumulated shared history as compounding mechanisms, and asks whether the effect is large enough to matter via the proposed test.

All got wrong 1

Proposes tests that could failWrongWrongWrong
Gemini 3.8 Flash · API

The metrics have thresholds and windows, but the output does not state what action each outcome would trigger (e.g., kill, pivot, scale).

Gemini 3.5 Flash-Lite · Gemini

The metrics lack the action each outcome would trigger, and the premium conversion metric has no measurement window; the tests do not specify what happens if thresholds are missed.

GPT-6 Astra · ChatGPT

The test does not name a numeric threshold for the primary outcome; it only says to set a minimum worthwhile lift before launch, which is not a specific threshold in the memo.

All got right 4

Respects explicit constraintsRightRightRight
Gemini 3.8 Flash · API

The output is a memo to the CEO, under 900 words, and includes the requested sections (ambitious version, mechanism, first step, how to know it's working).

Gemini 3.5 Flash-Lite · Gemini

The output is a memo under 900 words, addressed to Ruth, and includes the ambitious version, mechanism, first step, and success metrics as requested.

GPT-6 Astra · ChatGPT

The memo is addressed to Ruth, stays under 900 words, and proposes a test that would enforce the decision to invest only if results warrant.

Produces the required deliverableRightRightRight
Gemini 3.8 Flash · API

The memo is complete, in the right form, within the word limit, and a PM could act on it with light edits.

Gemini 3.5 Flash-Lite · Gemini

The memo is complete, in the correct form, within the word limit, and a PM could act on the proposed first step and metrics with light edits.

GPT-6 Astra · ChatGPT

Delivers a complete memo with the 1000x version, mechanism, first step, and success criteria, in the right form and within the word limit.

Finds the mechanism in the dataRightRightRight
Gemini 3.8 Flash · API

The memo cites retention, conversion, notification, and note-usage data to identify shared money decisions as the mechanism, not just budgeting.

Gemini 3.5 Flash-Lite · Gemini

The memo cites retention, conversion, notification, and note-usage data to identify shared money decisions and relational virality as the mechanism, not just budgeting.

GPT-6 Astra · ChatGPT

Uses retention, conversion, notification, note, and invitation data to identify shared money coordination as the mechanism, not just budgeting.

Extreme, then back to buildableRightRightRight
Gemini 3.8 Flash · API

It pushes the idea to a 'Multiplayer Operating System for Household Wealth' and works back to a concrete first step (Shared Spaces) that tests the same mechanism.

Gemini 3.5 Flash-Lite · Gemini

It pushes the idea to a 'financial operating system for shared lives' extreme, then works back to a concrete first step (shared-expense moment onboarding) that tests the same mechanism.

GPT-6 Astra · ChatGPT

Pushes to a household money network with multiple circles, then works back to a concrete two-person shared-expense circle that tests the coordination mechanism.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 88% of 118 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Haiku 5.5withAPI85.5100.02None
2GPT-6 AstrawithChatGPT93.091.32None
3GPT-6.1 SolwithAPI90.787.12None
4GPT-6 LunawithAPI88.283.02None
5Opus 5.5withClaude92.778.42None
6Sonnet 5.5withAPI90.774.22None
7Gemini 3.8 FlashwithAPI52.539.02None
8Gemini 3.5 Flash-LitewithGemini50.534.82None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreHaiku 5.5 · API: task score 93, 2.1× the cheapestGPT-6 Astra · ChatGPT: task score 92, 74× the cheapestGPT-6.1 Sol · API: task score 89, 15× the cheapestGPT-6 Luna · API: task score 84, the cheapestOpus 5.5 · Claude: task score 84, 59× the cheapestSonnet 5.5 · API: task score 81, 31× the cheapestGemini 3.8 Flash · API: task score 43, 9.1× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 40, 2.9× the cheapestHaiku 5.5 · APIGPT-6 Astra · ChatGPTGPT-6.1 Sol · APIGPT-6 Luna · APIOpus 5.5 · ClaudeSonnet 5.5 · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API841.0×
Haiku 5.5 · API932.1×
Gemini 3.5 Flash-Lite · Gemini402.9×
Gemini 3.8 Flash · API439.1×
GPT-6.1 Sol · API8915×
Sonnet 5.5 · API8131×
Opus 5.5 · Claude8459×
GPT-6 Astra · ChatGPT9274×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Finding the bigger version of a good idea.

Why it matters

Ambition without mechanism is fan fiction. The useful version pushes to the extreme, then works back to something buildable.

What good looks like

  • Names the mechanism that scales
  • Keeps the core insight
  • Works back to a first step you could build

Deliberately not measured

    Capability tested

    Ambitious expansion

    The failure we’re looking for

    Bigger adjectives, same idea

    Grading

    Decision model and LLM judge, calibrated against a blind PM review