Tasks / Strategy

1000x an idea

Can the model expand an idea's ambition while keeping it tethered to a real mechanism?

Task type v1.0 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 56% were usable with at most a quick edit.

Reliably right

Extreme, then back to buildable100% pass
Pushes from two-person sharing to the extreme household as the unit (any number of members, live pantry state, learning routines, delivery integration), then works back to the scoped two-person build that tests coordination.
Haiku 5.5 · API · Sharing a shopping list, 1000x
Finds the mechanism in the data100% pass
Uses the retention, conversion, session frequency, notification triggers, note usage, and invitation context to identify shared money decisions as the core mechanism, not just budgeting.
GPT-6.1 Sol · API · Budgets for two
Finds the mechanism in the data97% pass
Names household coordination as the core value (multiple members, claiming items, no duplicate buying), backs it with the 71%/22% retention gap, 38% second-device items, and duplicate tickets, and keeps coordination at the center of both the big vision and the first step.
Haiku 5.5 · API · Sharing a shopping list, 1000x

Where it slips

Proposes tests that could fail41% pass
The proposed test does not name a numeric threshold for the primary outcome; it only says to set minimum detectable effects and thresholds before launch, which is a process, not a specific threshold.
GPT-6.1 Sol · API · Budgets for two
The big version gets stronger as it grows61% pass
The memo names a demand network as a compounding mechanism but does not propose a test of whether that network effect would be large enough to matter, nor does it ask that question.
GPT-6 Astra · ChatGPT · Sharing a shopping list, 1000x
Avoids unsupported claims66% pass
The statements about most financial decisions being made by more than one person and happening in texts/arguments/spreadsheets are presented as established fact without support from the supplied evidence.
Sonnet 5.5 · API · Budgets for two

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

The big version gets stronger as it grows
Does the ambitious version name a mechanism that makes it more valuable or harder to copy as it grows (each new user adding value for others, or a barrier a competitor couldn't buy), and ask whether that effect would be large enough to matter?
Passes when Names the mechanism, how it compounds with scale, and a test of whether the effect is big enough to matter.
Hamilton HelmerLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Extreme, then back to buildable · Proposes tests that could fail · and 1 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a PM at Pocketbook. Ana has proposed shared budgets for couples; before we commit next quarter, our CEO wants the 1000x version (her note is below). Write the memo, in no more than 900 words: the most ambitious version of the idea, the mechanism that makes it work, and then what we build first and how we'll know it's working. The data is attached.

What the model was given5 items: About Pocketbook, ana_idea.md, ceo_note.md, usage_by_user_group.csv, what_shared_budgets_do.md

About PocketbookA personal budgeting app: connect your accounts, set budgets, track spending. Free, with Premium at $6 a month.

ana_idea.md3 lines · Download

# The idea (from Ana, PM)

"Let people share a budget with their partner. It's our most requested feature after bank sync for more banks."

ceo_note.md1 lines · Download

From Ruth (CEO): "Before we commit next quarter to Ana's idea, show me the 1000x version. How big could this get? Then bring it back to what we build first and how we'll know it's working. Keep it under 900 words."

usage_by_user_group.csv3 rows · Download

user_group,users,retained_at_6_months_pct,invited_someone_pct,invitee_joined_pct,premium_conversion_pct
Solo budgeters,182000,17,3,22,4
Couples sharing one budget,24600,58,100,100,19
Households (3+ people on one budget),3900,71,100,100,31

what_shared_budgets_do.md6 lines · Download

# What shared budgets do (from the data team)

- Couples who share a budget open the app on more days than solo users (11 a month against 4) and most sessions start from a notification about the other person's spending ("Alex spent $48 at the grocery store").
- 41% of couples add a "money talk" note to a transaction in their first month; the note feature was built as an afterthought.
- 16% of couples add a third person within a year (a teenager, a parent they support).
- Shared budgets started from an invite link in a shared-expenses moment (rent, a holiday) retain best.

What a strong answer doesThe answer key the graders mark against

Finds the mechanism in the data: shared money decisions, not budgeting. Couples sharing a budget retain at 58% against 17% for solo budgeters, convert to Premium at 19% against 4%, open the app on 11 days a month against 4 (mostly from notifications about the other person's spending), and use transaction notes as 'money talk'; households of 3+ retain at 71% and convert at 31%. Every shared budget brings in at least one new user by design, and 16% of couples add a third person. So the 1000x version is the place a household runs its money together (shared goals, who pays for what, conversations on transactions, extended family), growing through invitations at shared-expense moments (rent, holidays). Asks whether the effect is large enough to matter: each new member makes the budget more useful to the others, so it compounds. Works back to a first step that keeps the mechanism: shared budgets launched from an invite at a shared-expense moment, with spending notifications and transaction notes built in, measured on invites accepted, retention of invited users, and Premium conversion for couples, with thresholds.

Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

GPT-6.1 SolAPI93Usable with a quick edit GPT-6 AstraChatGPT93Usable with a quick edit GPT-6 LunaAPI91Usable with a quick edit Haiku 5.5API91Usable with a quick edit Sonnet 5.5API89Usable with a quick edit Opus 5.5Claude86Usable with a quick edit Gemini 3.5 Flash-LiteGemini45Needs more work Gemini 3.8 FlashAPI43Needs more work

GPT-6 LunaAPI

Usable with a quick edit9 of 10 checks passed

The memo is well-structured, uses evidence correctly, and only needs a minor edit to add a numeric success threshold for the experiment.

Output

To: Ruth From: Product Subject: Shared budgets: from feature to household money layer

The 1000x version

Don’t just let two people see the same budget. Make Pocketbook the place a household coordinates money: what’s shared, who’s responsible, what changed, and what the group wants to do next.

The product could grow from couples splitting rent or groceries to families managing bills, parents helping teenagers, and adults supporting relatives. Each person would bring in only the accounts and spending they choose to share. The household could set shared budgets and goals, see relevant activity, and discuss a transaction in context—without giving up the ability to keep other finances private.

The bigger opportunity is a financial habit, not a shared spreadsheet. A change in spending prompts a timely heads-up; people can add context or a note; the household can agree on an adjustment and see whether it’s working. Over time, Pocketbook could help households plan around bills, goals, and cash flow. Premium could offer richer household planning and controls.

Why it could work

Shared budgets appear to be associated with much stronger engagement: six-month retention is 58% for couples sharing a budget and 71% for households of three or more, versus 17% for solo budgeters. Premium conversion is 19% and 31%, respectively, versus 4% for solo users. Couples open the app 11 days a month versus four for solo users; most of their sessions start with a notification about the other person’s spending. And 41% add a “money talk” note in their first month—even though notes were built as an afterthought.

That suggests a reinforcing loop: an expense creates a useful moment to coordinate; a notification brings the other person in; context and discussion make the budget more useful; shared progress gives the household a reason to return. An invite sent during a real shared-expense moment—such as rent or a holiday—is associated with the best retention. Some couples also add a third person later, pointing to expansion beyond two-person households.

These are promising signals, not proof that sharing causes better retention or conversion: people who choose to share may already be more engaged. And the 100% invite/join figures for shared-budget groups describe people already in those groups, not a general invite-funnel benchmark.

What we build first

Start with one use case: a couple coordinating a shared expense or budget. Let one person create a shared budget and invite their partner with a link. Each person explicitly chooses which accounts or spending to share. Both can see shared-budget activity, get configurable notifications, and add a note to a transaction. Keep personal spending private by default; make it clear who can see what and allow people to mute notifications or leave the shared budget.

We should make the invite flow work in the moment of need, rather than asking solo users to make a vague “invite your partner” decision. Don’t build full household roles, complex permissions, or automated financial advice yet. First learn whether shared visibility and lightweight conversation produce a lasting joint habit.

How we’ll know it’s working

Track the funnel from shared-budget creation to invite acceptance, both people viewing shared activity, and both taking a meaningful action (for example, reviewing a transaction, adding a note, or changing a budget). Our leading indicator should be jointly active shared budgets: both members take an action in the shared budget in at least three of the first four weeks.

Use a randomized offer or holdout among eligible solo users to test whether offering shared setup improves activation and subsequent retention—not just whether people who opt in retain well. Follow six-month retention and Premium conversion, comparing like-for-like users, while monitoring privacy complaints, notification mutes, and shared-budget exits as guardrails.

If the experiment lifts joint activity and retention without eroding trust, we’ve earned the next step: support more household members and more of the financial decisions they share.

Grades and run details

Decision model 90 · LLM judge 91

Decision model checks

passUses the supplied evidence correctly73%
passAddresses the actual decision58%
passRespects explicit constraints54%
passIdentifies material uncertainty83%
passAvoids unsupported claims56%
passProduces the required deliverable66%
passFinds the mechanism in the data100%
passExtreme, then back to buildable99%
partialProposes tests that could fail82%
partialThe big version gets stronger as it grows35%

Run

Run: #1
API response time: 16 s
Submitted: 6 Oct 2026

Gemini 3.8 FlashAPI

Needs more than a quick edit4 of 10 checks passed

The memo contains unsupported factual claims and lacks explicit kill criteria and uncertainty, so a PM would need to re-check evidence and add decision triggers before using it.

Output

MEMORANDUM

TO: Ruth, CEO FROM: Product Management DATE: October 24, 2023 SUBJECT: The 1000x Vision for Shared Finances: From Single-Player Budgeting to Household Financial OS

---

1. The 1000x Vision: The Multiplayer Financial OS

Personal budgeting apps historically fail for a fundamental reason: single-player personal finance is a guilt-driven chore. Users log in alone, feel judged by their charts, and abandon the app. Our baseline data proves this: solo budgeters retain at just 17% at six months and convert to Premium at 4%.

The 1000x version is not a shared spreadsheet. It is the Multiplayer Operating System for Household Wealth.

Money is rarely an individual pursuit; it is relational. It starts with split rent and weekend trips, matures into cohabiting couples managing cash flow and homeownership, expands into allowances and financial literacy for teenagers, and extends to coordinating care for aging parents.

Today, this lifecycle is fragmented across split-pay apps, text threads, shared spreadsheets, and awkward conversations. By owning multi-user coordination, Pocketbook shifts from a disposable tracking utility into permanent household infrastructure.

When Pocketbook becomes the default operating system across generations: * Organic K-factor is built-in: Every core user acquires at least one other user (16% of our couples already pull in a third within 12 months). * Churn approaches zero: Uncoupling a multi-user financial ecosystem requires mutual consensus and massive operational friction. * Monetization expands exponentially: Instead of a single $6/month subscription, we become the rail for high-margin financial services: multi-party savings vaults, high-yield family accounts, teen debit, and estate planning.

---

2. The Mechanism: Why This Works

Our data reveals three self-reinforcing dynamics:

1. Ambient Awareness Replaces Interrogation: Couples open the app nearly 3x as often as solo users (11 days/month vs. 4), with the majority of sessions prompted by partner-spend notifications ("Alex spent $48 at the grocery store"). It eliminates the dreaded "Where did the money go this month?" conversation. Awareness becomes continuous, passive, and low-friction.

2. Micro-Communication Over Macro-Confrontation: 41% of couples actively leave transaction notes in month one—despite the feature being an unpromoted afterthought. Money management is fundamentally emotional. In-line transaction context ("Grabbed dinner for us," "Deposit for Lake Tahoe") de-escalates tension and transforms the app into a collaborative communication channel.

3. Radical Business Model Asymmetry: Moving a user from solo to multiplayer completely resets their unit economics: * Retention: Jumps from 17% (solo) to 58% (couples) and 71% (3+ households). * Premium Conversion: Surges from 4% (solo) to 19% (couples) and 31% (households).

Multiplayer finance is not a niche sub-segment; it is the single most potent retention and monetization engine in the business.

---

3. What We Build First (Q1 Delivery)

We should not ask users to fully merge their finances on day one. Shared budgets retain best when anchored to a specific, high-intent moment: a shared vacation, moving in together, or managing monthly rent.

Our Q1 MVP focuses on "Shared Spaces" around focused financial goals.

* Frictionless Shared Spaces (Category/Goal Sharing): Allow users to invite a partner to co-manage specific categories (e.g., Groceries, Rent, Vacation) without exposing full private accounts. Users link relevant cards/accounts with one-click permission toggles. * Contextual "Money Talk" Threading: Turn transaction notes into a first-class feature: lightweight 1-to-1 in-line comments, emoji reactions, and receipts attached directly to shared transactions. * Smart Multiplayer Notifications: Refine event triggers to drive collaborative habits: shared budget thresholds (e.g., "You've both spent 80% of Dining Out"), payment confirmations, and automated weekly digests. * Contextual Invitation Ingress: Prompts triggered right after a user logs a multi-party expense (e.g., high dining or travel bill): "Split or manage this together with a partner."

---

4. How We’ll Know It’s Working

We will validate this initiative across three horizons:

#### A. Input & Activation Metrics (First 30–60 Days) * Invite Completion Rate: Solo-to-couple invite send rate > 12% (up from 3%), with an invitee acceptance rate > 40%. * Activation Moment: > 50% of accepted invites configure at least one Shared Space within 48 hours. * Engagement Velocity: Shared cohort achieves ≥ 8 active app days per user/month, with > 45% using in-line transaction comments or reactions in Month 1.

#### B. Retention & Conversion Metrics (3–6 Months) * Cohort Retention: 6-month retention for Shared Space creators and invitees remains ≥ 50% (maintaining current couple performance as we scale). * Monetization Pull-Through: Free-to-Premium conversion on multiplayer cohorts remains ≥ 15%.

#### C. Expansion Signal (6–12 Months) * Household Viral Loop: ≥ 10% of couples onboard a 3rd member (child, dependent, or shared roommate) within 9 months, validating the transition from couple utility to household operating system.

---

Conclusion

Ana's shared budgets proposal is not just an incremental feature to appease our backlog; it is the blueprint for our entire product transformation. By executing the Q1 MVP, we take our first step away from a commoditized, high-churn personal tracker and establish the foundation of an indispensable, multiplayer household financial network.

Claims the judge couldn’t find in the brief

single-player personal finance is a guilt-driven chore. Users log in alone, feel judged by their charts, and abandon the app.
Money management is fundamentally emotional.
Today, this lifecycle is fragmented across split-pay apps, text threads, shared spreadsheets, and awkward conversations.

Grades and run details

Decision model 55 · LLM judge 36

Decision model checks

failUses the supplied evidence correctly5%
passAddresses the actual decision31%
partialRespects explicit constraints33%
failIdentifies material uncertainty50%
failAvoids unsupported claims87%
passProduces the required deliverable72%
passFinds the mechanism in the data99%
passExtreme, then back to buildable98%
partialProposes tests that could fail86%
partialThe big version gets stronger as it grows49%

Run

Run: #1
API response time: 21 s
Submitted: 6 Oct 2026

Haiku 5.5API

Usable with a quick edit9 of 10 checks passed

The 90-day target for 6-month retention creates a minor timeline inconsistency, but it is otherwise complete, evidence is correctly used, and a PM can adjust the metric timing with a quick edit.

Output

Memo: Shared Budgets, the 1000x Case and a First Build

To: Ruth From: PM, Pocketbook Re: Ana's shared-budget proposal

Recommendation

Commit next quarter to shared budgets, scoped as a household product rather than a feature. Start with the invite and the partner-activity loop, and hold the wider household roles until the couple data holds up.

The 1000x version

Today, Pocketbook is a product for one person. Shared budgets make the household the unit. The 1000x version is a household financial operating system: the place a couple, and eventually a family, runs its shared money. That covers bills, spending, savings goals, allowances for teens, and money support for a parent.

The scale argument rests on what the data already shows:

Couples are worth far more per user. Couples retain at 58% at six months against 17% for solo users. Their premium conversion is 19% against 4%.
Shared users are 14% of our base and about 45% of premium revenue. Our 24,600 couples and 3,900 households produce roughly $35k a month in premium revenue. Our 182,000 solo users produce about $44k.
The invite is the bottleneck, not the product. Solo users invite at 3%, but 22% of invitees join. Most people who would share have never been asked.

If couples alone reached 1000x today's count (about 24.6M couples) and kept today's 19% conversion, that would be roughly 4.7M premium users, or about $28M a month. Conversion won't hold at that scale, so treat this as an upper bound. The point is the shape: one shared budget brings in two daily users and a higher-value subscription. Households add a third person, which the data suggests happens for 16% of couples within a year.

The mechanism

Three loops drive it.

Notification pull. Couples open the app on 11 days a month, against 4 for solo users. Most sessions start from a message about a partner's spending ("Alex spent $48 at the grocery store"). Each shared budget creates a second daily user who is pulled back by the other person's activity.

Conversation as the product. 41% of couples add a "money talk" note to a transaction in their first month. The note was an afterthought, but it is the behavior that most resembles what the product is for: couples discussing money inside the app. Making it a first-class feature deepens the habit.

Expansion through real life. Couples who add a third person (a teenager, a parent they support) move from a two-person budget to a household budget. Households retain at 71% and convert at 31%. Each expansion raises value per account.

The invite is the entry point. Shared budgets started from a shared-expense moment, such as rent or a holiday, retain best. The invite should appear at the moment money is actually shared, not as a generic "invite your partner" button.

What we build first

1. An invite anchored to a shared-expense moment. When a user sets up a recurring shared bill (rent, utilities) or tags a transaction as shared, offer to bring in the partner. This targets the 3% invite rate directly.

2. Partner-activity notifications, with controls. The data says these drive sessions. Ship them with clear, per-transaction hiding and easy unsharing from day one.

3. Promote the money-talk note. Make notes on transactions a visible, first-class part of a shared budget.

Deliberately deferred: third-person and household roles, teen allowances, and parent-support flows. We should build these only once the couple version shows retention holding up.

Guardrails. A product where one partner can see the other's spending creates real risk, including financial control in abusive relationships. Before launch we need: - Opt-in visibility, with each person able to hide transactions from the other. - Unsharing that is easy and fully private, with no notice to the partner. - A separate review of defaults and copy with the trust and safety team.

How we'll know it's working

Set baselines now, and measure against cohorts rather than the current group averages.

Metric	Baseline	90-day target
Invite rate among solo users	3%	6%
Invitee join rate	22%	30%
Share of new couples started from a shared-expense moment	not tracked	50%+
Couples adding a money-talk note in month one	41%	50%
New-couple 6-month retention	58% (cohort, not yet measured)	at or above 50%
Couple premium conversion	19%	15%+

Caveat on the data. The invited-someone and invitee-joined columns read 100% for the couple and household groups. That is true by definition, since these groups only exist because someone invited a partner. The real comparison is the solo invite rate (3%) against the join rate (22%). The 58% and 71% retention figures may also overstate the effect of shared budgets: couples who stay together may be the ones who stick with the app. We need a randomized holdout, with invited and non-invited users matched on tenure and activity, before we credit the product for that retention.

Kill or rethink criteria. If invites don't lift past 4% within the quarter, or if new-couple retention doesn't separate from the holdout, we stop expanding the feature and keep the notifications and notes as solo-plus-partner improvements.

Grades and run details

Decision model 80 · LLM judge 100

Decision model checks

failUses the supplied evidence correctly53%
passAddresses the actual decision99%
passRespects explicit constraints31%
passIdentifies material uncertainty100%
partialAvoids unsupported claims45%
passProduces the required deliverable83%
passFinds the mechanism in the data100%
passExtreme, then back to buildable100%
partialProposes tests that could fail59%
passThe big version gets stronger as it grows72%

Run

Run: #1
API response time: 20 s
Submitted: 8 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 6

Uses the supplied evidence correctlyRightWrongMixed

GPT-6 Luna · API

All factual claims about the current situation are directly supported by the supplied data or context.

Gemini 3.8 Flash · API

The output makes unsupported claims about the current situation (guilt-driven chore, emotional nature, fragmented lifecycle) that are not in the supplied context.

Haiku 5.5 · API

Every statement about the current situation is taken directly from or derived arithmetically from the supplied context.

Addresses the actual decisionRightMixedRight

GPT-6 Luna · API

The memo commits to a clear first step and measurement plan, and states that a positive experiment result earns the next step.

Gemini 3.8 Flash · API

The memo does not state what result or condition would change the decision to build the MVP; it only lists success metrics without linking them to a pivot or kill decision.

Haiku 5.5 · API

Commits early to building shared budgets as a household product, and gives precise kill criteria that would change the call.

Identifies material uncertaintyRightWrongRight

GPT-6 Luna · API

It explicitly notes that the data is correlational, not proof of causation, and proposes a randomized experiment to resolve the uncertainty.

Gemini 3.8 Flash · API

The output does not name any unknowns that could change the decision or say how they would be resolved.

Haiku 5.5 · API

Acknowledges retention may be inflated by selection bias, conversion won't hold at scale, and requires a randomized holdout to resolve.

Avoids unsupported claimsRightWrongRight

GPT-6 Luna · API

Hypotheses and interpretations are clearly labeled with words like 'suggests' and 'may', avoiding presenting them as established fact.

Gemini 3.8 Flash · API

Interpretations like 'guilt-driven chore', 'fundamentally emotional', and 'fragmented across split-pay apps' are presented as established fact without being labelled as hypotheses.

Haiku 5.5 · API

Interpretations and forecasts are hedged ('may overstate', 'treat as upper bound'), and no hypotheses are presented as established fact.

Proposes tests that could failWrongWrongRight

GPT-6 Luna · API

The randomized experiment's success condition ('lifts joint activity and retention') lacks a numeric threshold, so it does not fully specify a test that could fail.

Gemini 3.8 Flash · API

The metrics have thresholds and windows, but the output does not state what action each outcome would trigger (e.g., kill, pivot, scale).

Haiku 5.5 · API

Kill criteria name a numeric threshold (invites >4%), a window (quarter), and the action (stop expanding); other metrics in the table have 90-day targets.

The big version gets stronger as it growsRightWrongRight

GPT-6 Luna · API

It names the reinforcing loop mechanism, implies it compounds as more people join, and proposes an experiment to test if the effect is large enough to matter.

Gemini 3.8 Flash · API

The output names compounding mechanisms (built-in acquisition, switching costs) but does not include a test of whether the effect is large enough to matter.

Haiku 5.5 · API

Describes how each added household member pulls others back via notifications and notes, and tests if the effect is large enough via invite lift and retention thresholds.

All got right 4

Respects explicit constraintsRightRightRight

GPT-6 Luna · API

The memo is addressed to Ruth, under 900 words, and includes the 1000x version, mechanism, first build, and success metrics as requested.

Gemini 3.8 Flash · API

The output is a memo to the CEO, under 900 words, and includes the requested sections (ambitious version, mechanism, first step, how to know it's working).

Haiku 5.5 · API

Delivers a memo addressed to the CEO, covers all requested sections, and stays well under 900 words.

Produces the required deliverableRightRightRight

GPT-6 Luna · API

The memo is complete, in the correct form for the CEO, within the word limit, and actionable as written.

Gemini 3.8 Flash · API

The memo is complete, in the right form, within the word limit, and a PM could act on it with light edits.

Haiku 5.5 · API

Memo includes the 1000x vision, mechanism, first build, and success metrics, and is directly usable by the named reader.

Finds the mechanism in the dataRightRightRight

GPT-6 Luna · API

It cites the retention, conversion, notification, and note data to identify a reinforcing loop around shared money decisions, not just budgeting.

Gemini 3.8 Flash · API

The memo cites retention, conversion, notification, and note-usage data to identify shared money decisions as the mechanism, not just budgeting.

Haiku 5.5 · API

Identifies shared money decisions as the mechanism (partner notifications, transaction notes, invites at shared moments) using retention, conversion, and session data.

Extreme, then back to buildableRightRightRight

GPT-6 Luna · API

It pushes to a household money layer and then works back to a concrete first step (couple coordinating a shared expense) that tests the same mechanism.

Gemini 3.8 Flash · API

It pushes the idea to a 'Multiplayer Operating System for Household Wealth' and works back to a concrete first step (Shared Spaces) that tests the same mechanism.

Haiku 5.5 · API

Pushes along the dimension of household scope to a household financial OS, then scales back to a first build of invite, notifications, and notes that tests the same mechanism.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 88% of 118 checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	Haiku 5.5withAPI	85.5	100.0	2	None
2	GPT-6 AstrawithChatGPT	93.0	91.3	2	None
3	GPT-6.1 SolwithAPI	90.7	87.1	2	None
4	GPT-6 LunawithAPI	88.2	83.0	2	None
5	Opus 5.5withClaude	92.7	78.4	2	None
6	Sonnet 5.5withAPI	90.7	74.2	2	None
7	Gemini 3.8 FlashwithAPI	52.5	39.0	2	None
8	Gemini 3.5 Flash-LitewithGemini	50.5	34.8	2	None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Nobody beats these on both score and cost

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
GPT-6 Luna · API	84	1.0×
Haiku 5.5 · API	93	2.1×
Gemini 3.5 Flash-Lite · Gemini	40	2.9×
Gemini 3.8 Flash · API	43	9.1×
GPT-6.1 Sol · API	89	15×
Sonnet 5.5 · API	81	31×
Opus 5.5 · Claude	84	59×
GPT-6 Astra · ChatGPT	92	74×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Finding the bigger version of a good idea.

Why it matters

Ambition without mechanism is fan fiction. The useful version pushes to the extreme, then works back to something buildable.

What good looks like

Names the mechanism that scales
Keeps the core insight
Works back to a first step you could build

Deliberately not measured

Capability tested

Ambitious expansion

The failure we’re looking for

Bigger adjectives, same idea

Grading

Decision model and LLM judge, calibrated against a blind PM review

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

The big version gets stronger as it grows

Plus our standard checks

The tasks

The outputs

Output

The 1000x version

Why it could work

What we build first

How we’ll know it’s working

Grades and run details

Decision model checks

Run

Output

1. The 1000x Vision: The Multiplayer Financial OS

2. The Mechanism: Why This Works

3. What We Build First (Q1 Delivery)

4. How We’ll Know It’s Working

Conclusion

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Run

Output

Memo: Shared Budgets, the 1000x Case and a First Build

Recommendation

The 1000x version

The mechanism

What we build first

How we'll know it's working

Grades and run details

Decision model checks

Run

Check by check

Where they differ 6

All got right 4

Results

What you get for the cost

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured