Tasks / Strategy

1000x an idea

Can the model expand an idea's ambition while keeping it tethered to a real mechanism?

Task type v1.0 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 56% were usable with at most a quick edit.

Reliably right

  1. Extreme, then back to buildable100% pass
    Pushes from two-person sharing to the extreme household as the unit (any number of members, live pantry state, learning routines, delivery integration), then works back to the scoped two-person build that tests coordination.
    Haiku 5.5 · API · Sharing a shopping list, 1000x
  2. Finds the mechanism in the data100% pass
    Uses the retention, conversion, session frequency, notification triggers, note usage, and invitation context to identify shared money decisions as the core mechanism, not just budgeting.
    GPT-6.1 Sol · API · Budgets for two
  3. Finds the mechanism in the data97% pass
    Names household coordination as the core value (multiple members, claiming items, no duplicate buying), backs it with the 71%/22% retention gap, 38% second-device items, and duplicate tickets, and keeps coordination at the center of both the big vision and the first step.
    Haiku 5.5 · API · Sharing a shopping list, 1000x

Where it slips

  1. Proposes tests that could fail41% pass
    The proposed test does not name a numeric threshold for the primary outcome; it only says to set minimum detectable effects and thresholds before launch, which is a process, not a specific threshold.
    GPT-6.1 Sol · API · Budgets for two
  2. The big version gets stronger as it grows61% pass
    The memo names a demand network as a compounding mechanism but does not propose a test of whether that network effect would be large enough to matter, nor does it ask that question.
    GPT-6 Astra · ChatGPT · Sharing a shopping list, 1000x
  3. Avoids unsupported claims66% pass
    The statements about most financial decisions being made by more than one person and happening in texts/arguments/spreadsheets are presented as established fact without support from the supplied evidence.
    Sonnet 5.5 · API · Budgets for two

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. The big version gets stronger as it grows

    Does the ambitious version name a mechanism that makes it more valuable or harder to copy as it grows (each new user adding value for others, or a barrier a competitor couldn't buy), and ask whether that effect would be large enough to matter?

    Passes when Names the mechanism, how it compounds with scale, and a test of whether the effect is big enough to matter.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Extreme, then back to buildable · Proposes tests that could fail · and 1 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a PM at Pocketbook. Ana has proposed shared budgets for couples; before we commit next quarter, our CEO wants the 1000x version (her note is below). Write the memo, in no more than 900 words: the most ambitious version of the idea, the mechanism that makes it work, and then what we build first and how we'll know it's working. The data is attached.

What the model was given5 items: About Pocketbook, ana_idea.md, ceo_note.md, usage_by_user_group.csv, what_shared_budgets_do.md
About PocketbookA personal budgeting app: connect your accounts, set budgets, track spending. Free, with Premium at $6 a month.
ana_idea.md3 lines · Download
# The idea (from Ana, PM)

"Let people share a budget with their partner. It's our most requested feature after bank sync for more banks."
ceo_note.md1 lines · Download
From Ruth (CEO): "Before we commit next quarter to Ana's idea, show me the 1000x version. How big could this get? Then bring it back to what we build first and how we'll know it's working. Keep it under 900 words."
usage_by_user_group.csv3 rows · Download
user_group,users,retained_at_6_months_pct,invited_someone_pct,invitee_joined_pct,premium_conversion_pct
Solo budgeters,182000,17,3,22,4
Couples sharing one budget,24600,58,100,100,19
Households (3+ people on one budget),3900,71,100,100,31
what_shared_budgets_do.md6 lines · Download
# What shared budgets do (from the data team)

- Couples who share a budget open the app on more days than solo users (11 a month against 4) and most sessions start from a notification about the other person's spending ("Alex spent $48 at the grocery store").
- 41% of couples add a "money talk" note to a transaction in their first month; the note feature was built as an afterthought.
- 16% of couples add a third person within a year (a teenager, a parent they support).
- Shared budgets started from an invite link in a shared-expenses moment (rent, a holiday) retain best.
What a strong answer doesThe answer key the graders mark against

Finds the mechanism in the data: shared money decisions, not budgeting. Couples sharing a budget retain at 58% against 17% for solo budgeters, convert to Premium at 19% against 4%, open the app on 11 days a month against 4 (mostly from notifications about the other person's spending), and use transaction notes as 'money talk'; households of 3+ retain at 71% and convert at 31%. Every shared budget brings in at least one new user by design, and 16% of couples add a third person. So the 1000x version is the place a household runs its money together (shared goals, who pays for what, conversations on transactions, extended family), growing through invitations at shared-expense moments (rent, holidays). Asks whether the effect is large enough to matter: each new member makes the budget more useful to the others, so it compounds. Works back to a first step that keeps the mechanism: shared budgets launched from an invite at a shared-expense moment, with spending notifications and transaction notes built in, measured on invites accepted, retention of invited users, and Premium conversion for couples, with thresholds.

Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Usable with a quick edit8 of 10 checks passed

The memo is well-structured, data-driven, and actionable. A quick edit to label the willpower claim as a hypothesis would resolve the only unsupported statement.

Output

Memo: Shared budgets, the 1000x version

To: Ruth | From: Product | Re: Ana's shared-budgets proposal

The short version

The big opportunity isn't a "share a budget" setting. It's turning Pocketbook from a tool one person uses alone into the place a household runs its money and talks about it. Our current users already show this. People who share a budget are 14% of our users. They account for about 36% of users still with us at six months and about 45% of Premium subscribers.

GroupUsersRetained at 6 moPremium
Solo182,00017% (~30,900)4% (~7,300)
Couples24,60058% (~14,300)19% (~4,700)
Households (3+)3,90071% (~2,800)31% (~1,200)

The 1000x version

Every household's money runs through Pocketbook: couples first, then the teenager, the parent you support, the roommates, the trip group. Each group recruits its next member at the moment money is shared. We stop selling budgeting discipline to individuals and become the shared ledger and conversation for the people you split life with. The business model becomes a household plan, not a per-person subscription.

The growth engine is the key difference. Solo budgeting apps grow one user at a time through marketing. A household product grows through its members. Each shared rent payment, holiday or new dependent is a chance to recruit someone.

The mechanism: three loops

1. The other person is the reason to open the app. Couples open Pocketbook 11 days a month, against 4 for solo users. Most of those sessions start from a notification about the partner's spending ("Alex spent $48 at the grocery store"). Solo budgeting runs on willpower, which fades. Shared budgeting runs on another person, who keeps giving you reasons to open the app.

2. The product is the money conversation. 41% of couples add a "money talk" note to a transaction in their first month. They do this even though notes were an afterthought. Couples are using us to talk about money, not just to track it. Talk is the habit that holds a household together.

3. Groups grow, and they start at shared-expense moments. 16% of couples add a third person within a year, and households retain best of all (71%). Shared budgets that start from an invite link during rent or a holiday retain best.

Today the growth loop barely works. 3% of solo users invite someone, and 22% of those invitees join. That's under 1 new member per 100 solo users. This is the biggest lever we have.

What could make this wrong

The numbers above are not proof that sharing causes retention. Couples who found today's sharing are self-selected: committed, organised, probably already engaged. The 100% invite and join rates for shared groups are true by definition, not a result.

The real question is whether a solo user nudged into sharing starts behaving like today's couples (58% retention) or stays like a solo user (17%). The plan below is built to answer that.

We also need to confirm how the 19% and 31% Premium rates are counted, per person or per budget, before designing any household pricing.

What we build first (next quarter)

Couples only. We build three things, aimed at the moments the data says matter:

  1. "Share this" on a transaction or bill. Rent, a trip, a big purchase each generate an invite link that opens straight into a shared budget. This targets the entry point that retains best, instead of a buried settings page.
  2. Partner activity notifications, on by default. Users get simple controls over what their partner sees, for example categories only or purchases over $X. Privacy is a real risk in couples' money; getting the defaults wrong will cause harm and generate support tickets.
  3. Notes upgraded to replies on transactions. This turns the afterthought into a lightweight conversation thread.

Not this quarter: - 3+ person households - Roommate or trip groups - Household pricing

These come only if the couples loop proves causal.

How we'll know it's working

We'll run a holdout: a random slice of solo users who don't see the new invite prompts. That lets us measure impact rather than selection.

Leading indicators (readable within the quarter):

MetricTodayTarget
Solo users who invite someone3%8–10%
Invitees who join22%35%+
New members per 100 solo usersunder 1about 3
Active days per month, new couples (first 30 days)—9+ (vs 11 for today's couples)
New couples adding a note in month one—35%+ (vs 41% today)

Lagging indicators (we'll have early reads; full answer after the quarter): - 3-month retention of new couples against the holdout, as an early read. 6-month retention against the holdout as the verdict. - Premium conversion of new couples against the holdout.

Decision rule: We commit to the household roadmap if both of these hold: - Couples formed through the new flows retain at least twice as well as the holdout at 3 months. - They are active more than 8 days a month.

If they look like solo users, we've learned the uplift is mostly selection. In that case shared budgets stay a solid retention feature for people who already want it, not a growth strategy. That is still worth shipping, but it's a 2x, not a 1000x.

The ask: One quarter for one squad to build the three couples features and the holdout. The household decision goes on the next planning cycle, made on real evidence.

Claims the judge couldn’t find in the brief

  • Solo budgeting runs on willpower, which fades.

Grades and run details

Decision model 90 · LLM judge 82
Decision model checks
  • passUses the supplied evidence correctly30%
  • passAddresses the actual decision94%
  • passRespects explicit constraints41%
  • passIdentifies material uncertainty100%
  • partialAvoids unsupported claims34%
  • passProduces the required deliverable88%
  • passFinds the mechanism in the data100%
  • passExtreme, then back to buildable100%
  • partialProposes tests that could fail36%
  • passThe big version gets stronger as it grows96%
Run
Run
#1
Time to output
46 s
Submitted
5 Oct 2026

Usable with a quick edit9 of 10 checks passed

The 90-day target for 6-month retention creates a minor timeline inconsistency, but it is otherwise complete, evidence is correctly used, and a PM can adjust the metric timing with a quick edit.

Output

Memo: Shared Budgets, the 1000x Case and a First Build

To: Ruth From: PM, Pocketbook Re: Ana's shared-budget proposal

Recommendation

Commit next quarter to shared budgets, scoped as a household product rather than a feature. Start with the invite and the partner-activity loop, and hold the wider household roles until the couple data holds up.

The 1000x version

Today, Pocketbook is a product for one person. Shared budgets make the household the unit. The 1000x version is a household financial operating system: the place a couple, and eventually a family, runs its shared money. That covers bills, spending, savings goals, allowances for teens, and money support for a parent.

The scale argument rests on what the data already shows:

  • Couples are worth far more per user. Couples retain at 58% at six months against 17% for solo users. Their premium conversion is 19% against 4%.
  • Shared users are 14% of our base and about 45% of premium revenue. Our 24,600 couples and 3,900 households produce roughly $35k a month in premium revenue. Our 182,000 solo users produce about $44k.
  • The invite is the bottleneck, not the product. Solo users invite at 3%, but 22% of invitees join. Most people who would share have never been asked.

If couples alone reached 1000x today's count (about 24.6M couples) and kept today's 19% conversion, that would be roughly 4.7M premium users, or about $28M a month. Conversion won't hold at that scale, so treat this as an upper bound. The point is the shape: one shared budget brings in two daily users and a higher-value subscription. Households add a third person, which the data suggests happens for 16% of couples within a year.

The mechanism

Three loops drive it.

  1. Notification pull. Couples open the app on 11 days a month, against 4 for solo users. Most sessions start from a message about a partner's spending ("Alex spent $48 at the grocery store"). Each shared budget creates a second daily user who is pulled back by the other person's activity.
  1. Conversation as the product. 41% of couples add a "money talk" note to a transaction in their first month. The note was an afterthought, but it is the behavior that most resembles what the product is for: couples discussing money inside the app. Making it a first-class feature deepens the habit.
  1. Expansion through real life. Couples who add a third person (a teenager, a parent they support) move from a two-person budget to a household budget. Households retain at 71% and convert at 31%. Each expansion raises value per account.

The invite is the entry point. Shared budgets started from a shared-expense moment, such as rent or a holiday, retain best. The invite should appear at the moment money is actually shared, not as a generic "invite your partner" button.

What we build first

1. An invite anchored to a shared-expense moment. When a user sets up a recurring shared bill (rent, utilities) or tags a transaction as shared, offer to bring in the partner. This targets the 3% invite rate directly.

2. Partner-activity notifications, with controls. The data says these drive sessions. Ship them with clear, per-transaction hiding and easy unsharing from day one.

3. Promote the money-talk note. Make notes on transactions a visible, first-class part of a shared budget.

Deliberately deferred: third-person and household roles, teen allowances, and parent-support flows. We should build these only once the couple version shows retention holding up.

Guardrails. A product where one partner can see the other's spending creates real risk, including financial control in abusive relationships. Before launch we need: - Opt-in visibility, with each person able to hide transactions from the other. - Unsharing that is easy and fully private, with no notice to the partner. - A separate review of defaults and copy with the trust and safety team.

How we'll know it's working

Set baselines now, and measure against cohorts rather than the current group averages.

MetricBaseline90-day target
Invite rate among solo users3%6%
Invitee join rate22%30%
Share of new couples started from a shared-expense momentnot tracked50%+
Couples adding a money-talk note in month one41%50%
New-couple 6-month retention58% (cohort, not yet measured)at or above 50%
Couple premium conversion19%15%+

Caveat on the data. The invited-someone and invitee-joined columns read 100% for the couple and household groups. That is true by definition, since these groups only exist because someone invited a partner. The real comparison is the solo invite rate (3%) against the join rate (22%). The 58% and 71% retention figures may also overstate the effect of shared budgets: couples who stay together may be the ones who stick with the app. We need a randomized holdout, with invited and non-invited users matched on tenure and activity, before we credit the product for that retention.

Kill or rethink criteria. If invites don't lift past 4% within the quarter, or if new-couple retention doesn't separate from the holdout, we stop expanding the feature and keep the notifications and notes as solo-plus-partner improvements.

Grades and run details

Decision model 80 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly53%
  • passAddresses the actual decision99%
  • passRespects explicit constraints31%
  • passIdentifies material uncertainty100%
  • partialAvoids unsupported claims45%
  • passProduces the required deliverable83%
  • passFinds the mechanism in the data100%
  • passExtreme, then back to buildable100%
  • partialProposes tests that could fail59%
  • passThe big version gets stronger as it grows72%
Run
Run
#1
API response time
20 s
Submitted
8 Oct 2026

Usable with a quick edit9 of 10 checks passed

The memo is well-structured and evidence-based; a PM could add a specific numeric threshold for the test outcome with a quick edit, without redoing any reasoning.

Output

From shared budgets to the household money network

To: Ruth From: Product Decision: Invest next quarter in testing shared money coordination—not simply adding a second user to a budget.

The 1000x version

Pocketbook becomes the place people manage money with everyone whose financial life touches theirs.

Start with a couple managing groceries and rent. Expand to saving for a home, giving a teenager spending autonomy, supporting a parent, and coordinating expenses across a blended family. One person can belong to several private money circles without exposing their entire financial life.

Today, Pocketbook answers: “Where did my money go?” The ambitious version answers: “What can we afford together, what have we agreed to, and what happens next?” It becomes a durable coordination layer above individual bank accounts—not a joint bank account and not a public financial social network.

That expands both our acquisition model and our reason to exist. Each shared obligation can introduce another person; each new life stage can create another circle. We stop rebuilding an individual’s budgeting habit from scratch and become part of an ongoing relationship.

This could support a materially larger subscription business. For scale, one million users converting at the observed couples rate of 19% would represent $13.7 million in annualized subscription revenue at $6/month. That is an illustration, not a forecast: we have neither established a million-user market nor shown that sharing causes that conversion rate.

Why it could work

Our strongest signal is not feature-request volume. It is that shared budgeting appears to change the product’s underlying behavior loop.

User groupSix-month retentionPremium conversion
------:---:
Solo budgeters17%4%
Couples sharing one budget58%19%
Households of 3+71%31%

Couples also open Pocketbook on 11 days per month versus four for solo users. Most sessions start with a notification about a partner’s spending. And 41% add a “money talk” note during their first month—even though notes were an afterthought.

The mechanism is coordination:

  1. A real obligation creates an invitation. Rent or a holiday provides a concrete reason to join. These shared-expense entry points already retain best.
  2. Shared activity creates useful context. A transaction changes what remains available to both people.
  3. Context creates a conversation or decision. A note resolves uncertainty; a budget adjustment records the agreement.
  4. Repeated decisions create shared history. Pocketbook becomes more useful as a relationship’s financial memory accumulates.
  5. New responsibilities extend the network. Sixteen percent of couples add a third person within a year.

This is a local network effect: the product improves when the right person joins, not when everyone joins.

But the evidence is observational. Existing sharers may be unusually motivated or financially organized. The groups’ 100% invitation and join rates largely reflect who qualifies as a sharer; they are not evidence of a frictionless funnel. We must test whether we can create the behavior, not merely identify people who already have it.

What we build first

A private, two-person shared-expense circle, launched around one immediate obligation.

Next quarter’s experience:

  • Create a circle for rent, groceries, or a trip; set an amount and timeframe.
  • Invite a partner through a link that explains the purpose and exactly what will be visible.
  • Let the partner join without first connecting a bank account.
  • Explicitly select transactions to share; keep all other accounts and spending private.
  • Show the shared amount remaining and support transaction-level notes.
  • Offer controllable activity notifications and a lightweight weekly check-in.
  • Make leaving, revoking access, and handling shared history understandable.

We should build on existing sharing and notes rather than launch an entirely separate product. We should not begin with payments, joint accounts, complex household roles, or automated financial advice.

Trust is part of the mechanism. Unwanted visibility can turn coordination into surveillance. Privacy controls and notification consent are launch requirements, not follow-up polish.

How we know it is working

Randomize eligible solo users, before they invite anyone, into the new shared-expense journey or today’s experience. Keep each resulting circle in one experiment arm. Analyze outcomes per assigned initiator, not only among successful sharers.

Primary near-term outcome: incremental eight-week retained coordinators—initiators whose circle has both people active and a meaningful shared-budget action in weeks seven or eight.

Track the full mechanism: invitation sent → partner joins → both contribute → first note or budget decision → repeated coordination. Notification opens alone do not count as success.

Also measure net additional retained invitees, Premium conversion, and support costs. Guardrails include privacy complaints, unwanted notifications, partner exits, and any reduction in initiators’ overall retention.

Set the minimum worthwhile lift and required sample size before launch using baseline rates and implementation costs. Scale only with credible retention improvement, healthy guardrails, and evidence that both partners get value. Maintain a holdout for six-month retention and monetization.

The bet is not that two people need one spreadsheet. It is that shared financial decisions can become Pocketbook’s acquisition engine, daily utility, and long-term moat.

Grades and run details

Decision model 95 · LLM judge 91
Decision model checks
  • passUses the supplied evidence correctly56%
  • passAddresses the actual decision91%
  • passRespects explicit constraints31%
  • passIdentifies material uncertainty98%
  • passAvoids unsupported claims53%
  • passProduces the required deliverable70%
  • passFinds the mechanism in the data100%
  • passExtreme, then back to buildable100%
  • partialProposes tests that could fail76%
  • passThe big version gets stronger as it grows89%
Run
Run
#1
Time to output
37 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 3

Uses the supplied evidence correctlyMixedMixedRight
Opus 5.5 · Claude

The claim 'Solo budgeting runs on willpower, which fades' is a statement about the current situation not found in the supplied context and not derivable from it.

Haiku 5.5 · API

Every statement about the current situation is taken directly from or derived arithmetically from the supplied context.

GPT-6 Astra · ChatGPT

All factual claims about the current situation are taken directly from the supplied data or context, with no inventions.

Avoids unsupported claimsWrongRightRight
Opus 5.5 · Claude

Presents 'Solo budgeting runs on willpower, which fades' as an established fact without labelling it as a hypothesis, and the evidence does not support it.

Haiku 5.5 · API

Interpretations and forecasts are hedged ('may overstate', 'treat as upper bound'), and no hypotheses are presented as established fact.

GPT-6 Astra · ChatGPT

Hypotheses and forecasts are clearly labelled (e.g., 'may be', 'illustration, not a forecast'), and confident claims are backed by evidence.

Proposes tests that could failRightRightWrong
Opus 5.5 · Claude

Each test has numeric thresholds (e.g., 8-10% invite rate, 35%+ join, 9+ active days, 2x retention vs holdout), a measurement window (within quarter for leading, 3-6 months for lagging), and a clear action for each outcome.

Haiku 5.5 · API

Kill criteria name a numeric threshold (invites >4%), a window (quarter), and the action (stop expanding); other metrics in the table have 90-day targets.

GPT-6 Astra · ChatGPT

The test does not name a numeric threshold for the primary outcome; it only says to set a minimum worthwhile lift before launch, which is not a specific threshold in the memo.

All got right 7

Addresses the actual decisionRightRightRight
Opus 5.5 · Claude

Commits to building couples features first with a holdout, and sets a clear decision rule for the household roadmap based on retention and activity thresholds.

Haiku 5.5 · API

Commits early to building shared budgets as a household product, and gives precise kill criteria that would change the call.

GPT-6 Astra · ChatGPT

Commits early to investing in testing shared money coordination, framed for Ruth, and states that scaling depends on credible retention improvement and guardrails.

Respects explicit constraintsRightRightRight
Opus 5.5 · Claude

Delivers a memo to Ruth under 900 words that covers the 1000x version, mechanism, first build, and success metrics.

Haiku 5.5 · API

Delivers a memo addressed to the CEO, covers all requested sections, and stays well under 900 words.

GPT-6 Astra · ChatGPT

The memo is addressed to Ruth, stays under 900 words, and proposes a test that would enforce the decision to invest only if results warrant.

Identifies material uncertaintyRightRightRight
Opus 5.5 · Claude

Names selection bias, definitional invite rates, and the open question of whether nudged solo users will behave like existing couples, and proposes a holdout to resolve it.

Haiku 5.5 · API

Acknowledges retention may be inflated by selection bias, conversion won't hold at scale, and requires a randomized holdout to resolve.

GPT-6 Astra · ChatGPT

Explicitly names selection bias and observational nature of the data, and says the test will resolve whether we can create the behavior.

Produces the required deliverableRightRightRight
Opus 5.5 · Claude

The memo is complete, correctly addressed, within the word limit, and actionable by the CEO with at most light edits.

Haiku 5.5 · API

Memo includes the 1000x vision, mechanism, first build, and success metrics, and is directly usable by the named reader.

GPT-6 Astra · ChatGPT

Delivers a complete memo with the 1000x version, mechanism, first step, and success criteria, in the right form and within the word limit.

Finds the mechanism in the dataRightRightRight
Opus 5.5 · Claude

Cites the retention, conversion, and behavior differences (11 vs 4 days, notifications, notes, shared-expense invites) and names shared money decisions and conversation as the mechanism.

Haiku 5.5 · API

Identifies shared money decisions as the mechanism (partner notifications, transaction notes, invites at shared moments) using retention, conversion, and session data.

GPT-6 Astra · ChatGPT

Uses retention, conversion, notification, note, and invitation data to identify shared money coordination as the mechanism, not just budgeting.

Extreme, then back to buildableRightRightRight
Opus 5.5 · Claude

Pushes to a household money platform where every household runs its money through Pocketbook, then works back to a concrete first step of couples-only features that test the same mechanism.

Haiku 5.5 · API

Pushes along the dimension of household scope to a household financial OS, then scales back to a first build of invite, notifications, and notes that tests the same mechanism.

GPT-6 Astra · ChatGPT

Pushes to a household money network with multiple circles, then works back to a concrete two-person shared-expense circle that tests the coordination mechanism.

The big version gets stronger as it growsRightRightRight
Opus 5.5 · Claude

Names growth through members at shared-expense moments as the compounding mechanism, and tests whether the effect is large enough to matter via the holdout and decision rule.

Haiku 5.5 · API

Describes how each added household member pulls others back via notifications and notes, and tests if the effect is large enough via invite lift and retention thresholds.

GPT-6 Astra · ChatGPT

Names local network effects and accumulated shared history as compounding mechanisms, and asks whether the effect is large enough to matter via the proposed test.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 88% of 118 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Haiku 5.5withAPI85.5100.02None
2GPT-6 AstrawithChatGPT93.091.32None
3GPT-6.1 SolwithAPI90.787.12None
4GPT-6 LunawithAPI88.283.02None
5Opus 5.5withClaude92.778.42None
6Sonnet 5.5withAPI90.774.22None
7Gemini 3.8 FlashwithAPI52.539.02None
8Gemini 3.5 Flash-LitewithGemini50.534.82None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreHaiku 5.5 · API: task score 93, 2.1× the cheapestGPT-6 Astra · ChatGPT: task score 92, 74× the cheapestGPT-6.1 Sol · API: task score 89, 15× the cheapestGPT-6 Luna · API: task score 84, the cheapestOpus 5.5 · Claude: task score 84, 59× the cheapestSonnet 5.5 · API: task score 81, 31× the cheapestGemini 3.8 Flash · API: task score 43, 9.1× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 40, 2.9× the cheapestHaiku 5.5 · APIGPT-6 Astra · ChatGPTGPT-6.1 Sol · APIGPT-6 Luna · APIOpus 5.5 · ClaudeSonnet 5.5 · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API841.0×
Haiku 5.5 · API932.1×
Gemini 3.5 Flash-Lite · Gemini402.9×
Gemini 3.8 Flash · API439.1×
GPT-6.1 Sol · API8915×
Sonnet 5.5 · API8131×
Opus 5.5 · Claude8459×
GPT-6 Astra · ChatGPT9274×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Finding the bigger version of a good idea.

Why it matters

Ambition without mechanism is fan fiction. The useful version pushes to the extreme, then works back to something buildable.

What good looks like

  • Names the mechanism that scales
  • Keeps the core insight
  • Works back to a first step you could build

Deliberately not measured

    Capability tested

    Ambitious expansion

    The failure we’re looking for

    Bigger adjectives, same idea

    Grading

    Decision model and LLM judge, calibrated against a blind PM review