Tasks / Strategy

1000x an idea

Can the model expand an idea's ambition while keeping it tethered to a real mechanism?

Task type v1.0 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 56% were usable with at most a quick edit.

Reliably right

  1. Extreme, then back to buildable100% pass
    Pushes from two-person sharing to the extreme household as the unit (any number of members, live pantry state, learning routines, delivery integration), then works back to the scoped two-person build that tests coordination.
    Haiku 5.5 · API · Sharing a shopping list, 1000x
  2. Finds the mechanism in the data100% pass
    Uses the retention, conversion, session frequency, notification triggers, note usage, and invitation context to identify shared money decisions as the core mechanism, not just budgeting.
    GPT-6.1 Sol · API · Budgets for two
  3. Finds the mechanism in the data97% pass
    Names household coordination as the core value (multiple members, claiming items, no duplicate buying), backs it with the 71%/22% retention gap, 38% second-device items, and duplicate tickets, and keeps coordination at the center of both the big vision and the first step.
    Haiku 5.5 · API · Sharing a shopping list, 1000x

Where it slips

  1. Proposes tests that could fail41% pass
    The proposed test does not name a numeric threshold for the primary outcome; it only says to set minimum detectable effects and thresholds before launch, which is a process, not a specific threshold.
    GPT-6.1 Sol · API · Budgets for two
  2. The big version gets stronger as it grows61% pass
    The memo names a demand network as a compounding mechanism but does not propose a test of whether that network effect would be large enough to matter, nor does it ask that question.
    GPT-6 Astra · ChatGPT · Sharing a shopping list, 1000x
  3. Avoids unsupported claims66% pass
    The statements about most financial decisions being made by more than one person and happening in texts/arguments/spreadsheets are presented as established fact without support from the supplied evidence.
    Sonnet 5.5 · API · Budgets for two

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. The big version gets stronger as it grows

    Does the ambitious version name a mechanism that makes it more valuable or harder to copy as it grows (each new user adding value for others, or a barrier a competitor couldn't buy), and ask whether that effect would be large enough to matter?

    Passes when Names the mechanism, how it compounds with scale, and a test of whether the effect is big enough to matter.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Extreme, then back to buildable · Proposes tests that could fail · and 1 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a PM at Pocketbook. Ana has proposed shared budgets for couples; before we commit next quarter, our CEO wants the 1000x version (her note is below). Write the memo, in no more than 900 words: the most ambitious version of the idea, the mechanism that makes it work, and then what we build first and how we'll know it's working. The data is attached.

What the model was given5 items: About Pocketbook, ana_idea.md, ceo_note.md, usage_by_user_group.csv, what_shared_budgets_do.md
About PocketbookA personal budgeting app: connect your accounts, set budgets, track spending. Free, with Premium at $6 a month.
ana_idea.md3 lines · Download
# The idea (from Ana, PM)

"Let people share a budget with their partner. It's our most requested feature after bank sync for more banks."
ceo_note.md1 lines · Download
From Ruth (CEO): "Before we commit next quarter to Ana's idea, show me the 1000x version. How big could this get? Then bring it back to what we build first and how we'll know it's working. Keep it under 900 words."
usage_by_user_group.csv3 rows · Download
user_group,users,retained_at_6_months_pct,invited_someone_pct,invitee_joined_pct,premium_conversion_pct
Solo budgeters,182000,17,3,22,4
Couples sharing one budget,24600,58,100,100,19
Households (3+ people on one budget),3900,71,100,100,31
what_shared_budgets_do.md6 lines · Download
# What shared budgets do (from the data team)

- Couples who share a budget open the app on more days than solo users (11 a month against 4) and most sessions start from a notification about the other person's spending ("Alex spent $48 at the grocery store").
- 41% of couples add a "money talk" note to a transaction in their first month; the note feature was built as an afterthought.
- 16% of couples add a third person within a year (a teenager, a parent they support).
- Shared budgets started from an invite link in a shared-expenses moment (rent, a holiday) retain best.
What a strong answer doesThe answer key the graders mark against

Finds the mechanism in the data: shared money decisions, not budgeting. Couples sharing a budget retain at 58% against 17% for solo budgeters, convert to Premium at 19% against 4%, open the app on 11 days a month against 4 (mostly from notifications about the other person's spending), and use transaction notes as 'money talk'; households of 3+ retain at 71% and convert at 31%. Every shared budget brings in at least one new user by design, and 16% of couples add a third person. So the 1000x version is the place a household runs its money together (shared goals, who pays for what, conversations on transactions, extended family), growing through invitations at shared-expense moments (rent, holidays). Asks whether the effect is large enough to matter: each new member makes the budget more useful to the others, so it compounds. Works back to a first step that keeps the mechanism: shared budgets launched from an invite at a shared-expense moment, with spending notifications and transaction notes built in, measured on invites accepted, retention of invited users, and Premium conversion for couples, with thresholds.

Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Usable with a quick edit9 of 10 checks passed

The 90-day target for 6-month retention creates a minor timeline inconsistency, but it is otherwise complete, evidence is correctly used, and a PM can adjust the metric timing with a quick edit.

Output

Memo: Shared Budgets, the 1000x Case and a First Build

To: Ruth From: PM, Pocketbook Re: Ana's shared-budget proposal

Recommendation

Commit next quarter to shared budgets, scoped as a household product rather than a feature. Start with the invite and the partner-activity loop, and hold the wider household roles until the couple data holds up.

The 1000x version

Today, Pocketbook is a product for one person. Shared budgets make the household the unit. The 1000x version is a household financial operating system: the place a couple, and eventually a family, runs its shared money. That covers bills, spending, savings goals, allowances for teens, and money support for a parent.

The scale argument rests on what the data already shows:

  • Couples are worth far more per user. Couples retain at 58% at six months against 17% for solo users. Their premium conversion is 19% against 4%.
  • Shared users are 14% of our base and about 45% of premium revenue. Our 24,600 couples and 3,900 households produce roughly $35k a month in premium revenue. Our 182,000 solo users produce about $44k.
  • The invite is the bottleneck, not the product. Solo users invite at 3%, but 22% of invitees join. Most people who would share have never been asked.

If couples alone reached 1000x today's count (about 24.6M couples) and kept today's 19% conversion, that would be roughly 4.7M premium users, or about $28M a month. Conversion won't hold at that scale, so treat this as an upper bound. The point is the shape: one shared budget brings in two daily users and a higher-value subscription. Households add a third person, which the data suggests happens for 16% of couples within a year.

The mechanism

Three loops drive it.

  1. Notification pull. Couples open the app on 11 days a month, against 4 for solo users. Most sessions start from a message about a partner's spending ("Alex spent $48 at the grocery store"). Each shared budget creates a second daily user who is pulled back by the other person's activity.
  1. Conversation as the product. 41% of couples add a "money talk" note to a transaction in their first month. The note was an afterthought, but it is the behavior that most resembles what the product is for: couples discussing money inside the app. Making it a first-class feature deepens the habit.
  1. Expansion through real life. Couples who add a third person (a teenager, a parent they support) move from a two-person budget to a household budget. Households retain at 71% and convert at 31%. Each expansion raises value per account.

The invite is the entry point. Shared budgets started from a shared-expense moment, such as rent or a holiday, retain best. The invite should appear at the moment money is actually shared, not as a generic "invite your partner" button.

What we build first

1. An invite anchored to a shared-expense moment. When a user sets up a recurring shared bill (rent, utilities) or tags a transaction as shared, offer to bring in the partner. This targets the 3% invite rate directly.

2. Partner-activity notifications, with controls. The data says these drive sessions. Ship them with clear, per-transaction hiding and easy unsharing from day one.

3. Promote the money-talk note. Make notes on transactions a visible, first-class part of a shared budget.

Deliberately deferred: third-person and household roles, teen allowances, and parent-support flows. We should build these only once the couple version shows retention holding up.

Guardrails. A product where one partner can see the other's spending creates real risk, including financial control in abusive relationships. Before launch we need: - Opt-in visibility, with each person able to hide transactions from the other. - Unsharing that is easy and fully private, with no notice to the partner. - A separate review of defaults and copy with the trust and safety team.

How we'll know it's working

Set baselines now, and measure against cohorts rather than the current group averages.

MetricBaseline90-day target
Invite rate among solo users3%6%
Invitee join rate22%30%
Share of new couples started from a shared-expense momentnot tracked50%+
Couples adding a money-talk note in month one41%50%
New-couple 6-month retention58% (cohort, not yet measured)at or above 50%
Couple premium conversion19%15%+

Caveat on the data. The invited-someone and invitee-joined columns read 100% for the couple and household groups. That is true by definition, since these groups only exist because someone invited a partner. The real comparison is the solo invite rate (3%) against the join rate (22%). The 58% and 71% retention figures may also overstate the effect of shared budgets: couples who stay together may be the ones who stick with the app. We need a randomized holdout, with invited and non-invited users matched on tenure and activity, before we credit the product for that retention.

Kill or rethink criteria. If invites don't lift past 4% within the quarter, or if new-couple retention doesn't separate from the holdout, we stop expanding the feature and keep the notifications and notes as solo-plus-partner improvements.

Grades and run details

Decision model 80 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly53%
  • passAddresses the actual decision99%
  • passRespects explicit constraints31%
  • passIdentifies material uncertainty100%
  • partialAvoids unsupported claims45%
  • passProduces the required deliverable83%
  • passFinds the mechanism in the data100%
  • passExtreme, then back to buildable100%
  • partialProposes tests that could fail59%
  • passThe big version gets stronger as it grows72%
Run
Run
#1
API response time
20 s
Submitted
8 Oct 2026

Usable with a quick edit9 of 10 checks passed

The memo is well-structured and evidence-based; a PM could add a specific numeric threshold for the primary outcome as a quick edit without redoing any reasoning.

Output

From shared budgets to the household’s financial home

To: Ruth From: Product Decision: Invest next quarter in shared-money coordination, starting with couples—not merely shared access to a budget.

The 1000x version

Pocketbook becomes the place people run their financial lives together: partners managing a home, parents teaching teenagers, adult children supporting parents, and families coordinating across households.

Today, Pocketbook answers: “Where did my money go?” The bigger product answers: “What can we do together, what have we agreed, and what happens next?”

A shared budget is the entry point. The destination is a household financial operating system: shared goals, spending agreements, contextual conversations, responsibilities, and eventually coordinated saving and bill-paying. Private finances remain private; people share only what a particular relationship requires.

This changes our unit of value from one person tracking transactions to a group making recurring decisions. Pocketbook could become useful whenever money crosses a relationship—not just when someone checks their balance.

The evidence is promising:

User groupUsersSix-month retentionPremium conversion
------:---:---:
Solo budgeters182,00017%4%
Couples sharing one budget24,60058%19%
Households of 3+3,90071%31%

Couples have roughly 3.4× the retention and 4.8× the Premium conversion of solo users. Larger households are stronger still. These are observational differences, not proof that sharing causes the lift: already-committed households may select into sharing. But they justify testing a fundamentally different product thesis.

“1000x” is not a forecast of 1,000 times today’s users. It is the ambition to move from a budgeting utility to enduring infrastructure for shared financial life.

The mechanism: shared money creates recurring coordination

The engine is a loop:

Shared expense → invitation → joint decision → visible progress → another shared decision.

Rent or a holiday gives someone a concrete reason to invite their partner. Both contribute transactions, context, or an agreement. Pocketbook helps them resolve uncertainty and see progress. The next spending decision brings them back.

Our behavioral data points directly to this mechanism:

  • Couples use Pocketbook on 11 days a month versus four for solo users.
  • Most couple sessions begin with a notification about the other person’s spending.
  • 41% add a “money talk” transaction note in their first month, despite notes being an afterthought.
  • Invites originating in a shared-expenses moment retain best.
  • 16% of couples add a third person within a year.

The opportunity is therefore not “two people can view the same spreadsheet.” It is turning a transaction into a useful conversation and a conversation into an agreement.

That can create both retention and distribution: useful collaboration brings people back; new responsibilities bring additional people in. Household expansion is an observed behavior, but a self-sustaining growth loop remains a hypothesis.

Trust is the limiting condition. Spending alerts must support coordination, not surveillance. Sharing needs explicit consent, clear boundaries, and easy exit. We should optimize for decisions resolved—not notifications opened.

What we build first

Next quarter, productize the existing shared-budget behavior around one couple, one shared purpose.

  1. Purpose-led setup. Offer “Plan a holiday” or “Manage monthly household spending,” with a simple starter budget and an invitation explaining exactly what the partner will see.
  2. Selective sharing. Let each person choose which accounts or transactions contribute. Keep personal finances separate; make permissions, revocation, and leaving the budget clear.
  3. A shared decision surface. Show the budget’s remaining amount, relevant spending, and transaction-linked notes. Make it easy to flag a purchase for discussion and record an agreed next step.
  4. Useful, controllable notifications. Let couples choose immediate updates or a digest. Use neutral language and avoid broadcasting private spending.

The basic collaboration loop should be free so invitations are frictionless. We will measure downstream Premium conversion at the existing $6/month price before deciding which advanced coordination features warrant a household paid tier.

We will not build payments, lending, a family marketplace, or complex multi-household permissions next quarter.

How we know it works

Run a randomized rollout among eligible solo users, assigning at the inviting-user level and keeping each resulting shared group in the same treatment. Compare the new experience with today’s sharing flow.

Primary early outcome: the share of eligible users forming an activated pair—both people join, contribute, and return in separate weeks within 30 days.

Durability: paired activity at eight weeks, then six-month retention. Do not treat invitation acceptance alone as success.

Business outcomes: incremental Premium conversion and retained paid subscriptions per eligible user, including any loss of individual subscriptions.

Trust guardrails: accidental-sharing reports, unwanted-notification complaints, departures, and interview evidence of increased financial tension.

Set minimum detectable effects and rollout thresholds before launch using baseline funnel volumes. Scale only if we create more durable shared participation without worsening trust. That is the first proof that Pocketbook can help households coordinate—not simply watch each other spend.

Grades and run details

Decision model 95 · LLM judge 91
Decision model checks
  • passUses the supplied evidence correctly63%
  • passAddresses the actual decision91%
  • passRespects explicit constraints36%
  • passIdentifies material uncertainty85%
  • passAvoids unsupported claims56%
  • passProduces the required deliverable72%
  • passFinds the mechanism in the data100%
  • passExtreme, then back to buildable100%
  • partialProposes tests that could fail82%
  • passThe big version gets stronger as it grows76%
Run
Run
#1
API response time
34 s
Submitted
6 Oct 2026

Needs more than a quick edit4 of 10 checks passed

The memo contains unsupported claims about the market and lacks explicit decision-change conditions and complete test design, so a PM would need to rework the evidence and tests before using it.

Output

MEMORANDUM

TO: Ruth, CEO FROM: Product Team DATE: October 24, 2023 SUBJECT: The 1000x Version of Shared Budgets (Ana’s Proposal)

*

1. The 1000x Version: Pocketbook as the Operating System for Shared Life

Right now, personal finance assumes an individual user in a quiet room looking at bar charts. But money doesn't live in a vacuum—it is the friction point of human relationships.

If we take Ana’s proposal to its logical, 1000x extreme, Pocketbook stops being an app for tracking your money and becomes the financial operating system for shared lives.

Today, couples use a fractured stack of tools to manage their shared reality: splitwise for dinner bills, spreadsheets for rent, Venmo for groceries, silent resentment for lifestyle inflation, and separate banking apps that don't talk to each other. It is exhausting.

The 1000x version of Pocketbook captures this entire surface area. We don't just "share a budget"—we remove the emotional labor of financial coordination. Imagine an ecosystem where: * The system auto-negotiates equity: It understands income differentials and automatically scales contributions to shared goals without manual math. * Micro-communication replaces macro-arguments: Financial friction is dissolved via contextual, inline micro-notes on transactions ("Bought your mom's gift early! ❤️"), turning spending alerts from panic triggers into connection points. * Organic household expansion: It scales naturally from couples to households (adding aging parents or kids) and handles life milestones (buying a home, taking a sabbatical, having a child) as collaborative multiplayer projects rather than administrative nightmares.

Look at our current data. It already whispers this truth: Solo users retain at 17% and convert to Premium at 4%. But Couples retain at 58% and convert at 19%. Households retain at 71% and convert at 31%.

Shared budgets are not just a feature request; they are our escape hatch from single-player churn. The 1000x version transforms Pocketbook from a utility you check once a week into the digital kitchen table where couples align their lives.

---

2. The Mechanism: Why This Spreads Exponentially

The engine of the 1000x version is relational virality.

Traditional personal finance apps have terrible organic loops because money is private and taboo. You don't text your friend a screenshot of your grocery budget.

Shared budgets invert this entirely. By definition, a budget for a couple requires an invitation. Our data proves this mechanism is already working: couples have a 100% invite rate because the product cannot function without the second person.

The compounding loop looks like this: 1. The Trigger: A couple experiences a shared-expense friction point (paying rent, booking a holiday) and one partner invites the other via Pocketbook. 2. The Dopamine Hit: The invitee joins, and within minutes, they receive their first real-time notification about shared financial momentum ("Alex spent $48 at the grocery store"). Engagement spikes immediately (couples open the app 11 times a month versus 4 for solo users). 3. The Cultural Artifact: They use the accidental "money talk" note feature (which 41% of couples adopt in month one). Money stops being a taboo source of anxiety and becomes a shared chat thread. 4. The Network Effect: As life expands, the network expands. 16% of couples naturally add a third person within a year—a teenager learning financial literacy or an aging parent they support.

We don't need expensive paid acquisition to scale this. Every shared budget creates an involuntary, high-intent acquisition channel for our highest-value segment.

---

3. What We Build First & How We Know It’s Working

To capture the 1000x vision without boiling the ocean next quarter, we must sequence our roadmap ruthlessly. We will not build complex multi-user permissioning or robo-advising out of the gate.

We build the "Shared-Expense Moment" onboarding flow.

Based on data showing that shared budgets originating from specific collaborative moments retain best, our Q1 MVP focuses exclusively on friction-free pairing around a single shared goal or expense: * Streamlined Invite-to-Split: Allow a user to instantly spin up a shared bucket for a specific vector (e.g., "Our Apartment" or "Japan Trip") and text an invite link that resolves in under 10 seconds. * First-Class "Money Talk" Notes: Elevate the transaction note feature from an afterthought into a core UI element. Make leaving a context note or emoji reaction on a partner’s transaction frictionless. * The Notification Loop: Tune real-time push notifications so they foster alignment rather than anxiety, capitalizing on the behavior that drives our 11x/month engagement.

#### How We’ll Know It’s Working (Metrics for Q1): 1. Viral Coefficient (K-factor) of Invites: We will track the percentage of new solo sign-ups who successfully dispatch a partner invite within 7 days, targeting a baseline of >25% conversion from solo to shared. 2. Day-30 Retention of Paired Users: We will measure whether our MVP couples maintain the elevated retention baseline (targeting >50% retained at 30 days, pacing toward our 58% 6-month benchmark). 3. Note Adoption Rate: We will track whether >40% of newly paired couples use the transaction note feature in their first 14 days, proving that we are successfully replacing financial stress with communication. 4. Free-to-Premium Conversion Lift: We will monitor if newly paired users convert to our $6/month Premium tier at rates mirroring our existing cohort (19%+).

Ana’s proposal isn't just a feature to check a box against our competitors. It is the architectural shift that transforms Pocketbook into an indispensable, sticky, multi-player platform. Let’s build it.

Claims the judge couldn’t find in the brief

  • Today, couples use a fractured stack of tools: Splitwise, spreadsheets, Venmo, separate banking apps, etc.
  • Traditional personal finance apps have terrible organic loops because money is private and taboo.

Grades and run details

Decision model 60 · LLM judge 36
Decision model checks
  • passUses the supplied evidence correctly7%
  • partialAddresses the actual decision22%
  • partialRespects explicit constraints29%
  • failIdentifies material uncertainty70%
  • failAvoids unsupported claims77%
  • passProduces the required deliverable62%
  • passFinds the mechanism in the data97%
  • passExtreme, then back to buildable97%
  • partialProposes tests that could fail86%
  • partialThe big version gets stronger as it grows40%
Run
Run
#1
Time to output
7 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 6

Uses the supplied evidence correctlyMixedRightMixed
Haiku 5.5 · API

Every statement about the current situation is taken directly from or derived arithmetically from the supplied context.

GPT-6.1 Sol · API

All factual claims about the current situation are taken directly from the supplied context or derived by arithmetic, with no invented facts.

Gemini 3.5 Flash-Lite · Gemini

The output invents claims about the current market situation (fractured stack of tools, Splitwise, Venmo, spreadsheets) and about traditional personal finance apps' organic loops, none of which appear in the supplied context.

Addresses the actual decisionRightRightWrong
Haiku 5.5 · API

Commits early to building shared budgets as a household product, and gives precise kill criteria that would change the call.

GPT-6.1 Sol · API

Commits early to investing in shared-money coordination starting with couples, and states that scaling depends on creating durable shared participation without worsening trust.

Gemini 3.5 Flash-Lite · Gemini

The memo commits to building the shared-expense onboarding flow but never states what result or condition would change that decision.

Identifies material uncertaintyRightRightWrong
Haiku 5.5 · API

Acknowledges retention may be inflated by selection bias, conversion won't hold at scale, and requires a randomized holdout to resolve.

GPT-6.1 Sol · API

Acknowledges that the observed differences may be selection effects, that a self-sustaining growth loop is a hypothesis, and that trust is a limiting condition, and proposes a randomized rollout to resolve them.

Gemini 3.5 Flash-Lite · Gemini

The output does not name any specific unknowns that could change the decision or explain how they would be resolved.

Avoids unsupported claimsRightRightWrong
Haiku 5.5 · API

Interpretations and forecasts are hedged ('may overstate', 'treat as upper bound'), and no hypotheses are presented as established fact.

GPT-6.1 Sol · API

Causal interpretations and forecasts are clearly labelled as hypotheses or observational, and confident claims are backed by the supplied evidence.

Gemini 3.5 Flash-Lite · Gemini

The output presents the fractured tool stack and the taboo nature of money as established facts without labelling them as hypotheses, and the evidence does not support them.

Proposes tests that could failRightWrongWrong
Haiku 5.5 · API

Kill criteria name a numeric threshold (invites >4%), a window (quarter), and the action (stop expanding); other metrics in the table have 90-day targets.

GPT-6.1 Sol · API

The proposed test does not name a numeric threshold for the primary outcome; it only says to set minimum detectable effects and thresholds before launch, which is a process, not a specific threshold.

Gemini 3.5 Flash-Lite · Gemini

The metrics lack the action each outcome would trigger, and the premium conversion metric has no measurement window; the tests do not specify what happens if thresholds are missed.

The big version gets stronger as it growsRightRightWrong
Haiku 5.5 · API

Describes how each added household member pulls others back via notifications and notes, and tests if the effect is large enough via invite lift and retention thresholds.

GPT-6.1 Sol · API

Names the mechanism of recurring coordination and household expansion, explains how each new member adds value, and proposes a test of whether the effect is large enough to matter.

Gemini 3.5 Flash-Lite · Gemini

The output names relational virality and describes compounding, but it does not include a test of whether the compounding effect is large enough to matter, only a target for invite conversion.

All got right 4

Respects explicit constraintsRightRightRight
Haiku 5.5 · API

Delivers a memo addressed to the CEO, covers all requested sections, and stays well under 900 words.

GPT-6.1 Sol · API

The memo is addressed to Ruth, stays under 900 words, and proposes a concrete first step that respects the brief's request for the 1000x version, mechanism, and measurement.

Gemini 3.5 Flash-Lite · Gemini

The output is a memo under 900 words, addressed to Ruth, and includes the ambitious version, mechanism, first step, and success metrics as requested.

Produces the required deliverableRightRightRight
Haiku 5.5 · API

Memo includes the 1000x vision, mechanism, first build, and success metrics, and is directly usable by the named reader.

GPT-6.1 Sol · API

The memo is in the requested form, addressed to Ruth, within 900 words, and contains the 1000x version, mechanism, first step, and success criteria, making it actionable.

Gemini 3.5 Flash-Lite · Gemini

The memo is complete, in the correct form, within the word limit, and a PM could act on the proposed first step and metrics with light edits.

Finds the mechanism in the dataRightRightRight
Haiku 5.5 · API

Identifies shared money decisions as the mechanism (partner notifications, transaction notes, invites at shared moments) using retention, conversion, and session data.

GPT-6.1 Sol · API

Uses the retention, conversion, session frequency, notification triggers, note usage, and invitation context to identify shared money decisions as the core mechanism, not just budgeting.

Gemini 3.5 Flash-Lite · Gemini

The memo cites retention, conversion, notification, and note-usage data to identify shared money decisions and relational virality as the mechanism, not just budgeting.

Extreme, then back to buildableRightRightRight
Haiku 5.5 · API

Pushes along the dimension of household scope to a household financial OS, then scales back to a first build of invite, notifications, and notes that tests the same mechanism.

GPT-6.1 Sol · API

Pushes the idea to a household financial operating system, then works back to a concrete first step of purpose-led couple sharing with selective sharing, a decision surface, and notifications.

Gemini 3.5 Flash-Lite · Gemini

It pushes the idea to a 'financial operating system for shared lives' extreme, then works back to a concrete first step (shared-expense moment onboarding) that tests the same mechanism.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 88% of 118 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Haiku 5.5withAPI85.5100.02None
2GPT-6 AstrawithChatGPT93.091.32None
3GPT-6.1 SolwithAPI90.787.12None
4GPT-6 LunawithAPI88.283.02None
5Opus 5.5withClaude92.778.42None
6Sonnet 5.5withAPI90.774.22None
7Gemini 3.8 FlashwithAPI52.539.02None
8Gemini 3.5 Flash-LitewithGemini50.534.82None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreHaiku 5.5 · API: task score 93, 2.1× the cheapestGPT-6 Astra · ChatGPT: task score 92, 74× the cheapestGPT-6.1 Sol · API: task score 89, 15× the cheapestGPT-6 Luna · API: task score 84, the cheapestOpus 5.5 · Claude: task score 84, 59× the cheapestSonnet 5.5 · API: task score 81, 31× the cheapestGemini 3.8 Flash · API: task score 43, 9.1× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 40, 2.9× the cheapestHaiku 5.5 · APIGPT-6 Astra · ChatGPTGPT-6.1 Sol · APIGPT-6 Luna · APIOpus 5.5 · ClaudeSonnet 5.5 · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API841.0×
Haiku 5.5 · API932.1×
Gemini 3.5 Flash-Lite · Gemini402.9×
Gemini 3.8 Flash · API439.1×
GPT-6.1 Sol · API8915×
Sonnet 5.5 · API8131×
Opus 5.5 · Claude8459×
GPT-6 Astra · ChatGPT9274×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Finding the bigger version of a good idea.

Why it matters

Ambition without mechanism is fan fiction. The useful version pushes to the extreme, then works back to something buildable.

What good looks like

  • Names the mechanism that scales
  • Keeps the core insight
  • Works back to a first step you could build

Deliberately not measured

    Capability tested

    Ambitious expansion

    The failure we’re looking for

    Bigger adjectives, same idea

    Grading

    Decision model and LLM judge, calibrated against a blind PM review