Tasks / Strategy

Challenge an idea

Can the model find the strongest reason an idea may fail, backed by evidence?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 44% were usable with at most a quick edit.

Reliably right

  1. Finds the load-bearing assumption100% pass
    The critique targets the assumption that agencies' growth is limited by a shortage of new leads, using referral, abandoned-outbound, and capacity evidence.
    GPT-6.1 Sol · API · An AI SDR for small agencies
  2. Uses the interviews faithfully100% pass
    All quotes are accurately taken from the transcripts and correctly attributed to A05, A04, and A08.
    GPT-6.1 Sol · API · An AI SDR for small agencies
  3. Finds the timeline that sinks it100% pass
    Cites 9-month build and concludes Q1 launch is unsupported.
    GPT-6 Astra · ChatGPT · Five billion dollars, on paper

Where it slips

  1. Uses the supplied evidence correctly50% pass
    Claims that local taxes and multi-state operations affect eligibility are not supported by the supplied context, which only mentions state-by-state tax rules.
    GPT-6 Astra · ChatGPT · Five billion dollars, on paper
  2. Avoids unsupported claims56% pass
    Presents the existence of local taxes and multi-state complexity as fact without support from the context.
    GPT-6 Astra · ChatGPT · Five billion dollars, on paper
  3. Finds the need behind the idea71% pass
    It mentions A05's past-client reminder idea but does not state the underlying need or offer a full alternative with its cost and what it displaces.
    GPT-6 Astra · ChatGPT · An AI SDR for small agencies

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Finds the need behind the idea

    Does the critique identify the underlying need the idea is meant to serve, and offer at least one other way to meet that need, at a different cost, saying what would have to be dropped for it?

    Passes when States the need behind the proposal, and gives an alternative way to meet it with its cost and what it displaces.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Tigers, not paper tigers · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the VP Product at Halyard. Our CEO is taking the US launch plan below to the board next week and has asked you to challenge it hard first. Write a memo of no more than 1,000 words: what the plan gets wrong, what would sink it, what's less scary than it looks, and what you'd propose instead. The deck excerpt and our data are attached.

What the model was given4 items: about_halyard.md, ceo_board_deck_excerpt.md, uk_customers_by_size.csv, us_small_business_stats.md
about_halyard.md3 lines · Download
# About Halyard in the UK

Payroll and HR for small businesses. 14,800 customers, $3.9M ARR, five years old. UK payroll rules are built in; US payroll would need state-by-state tax rules (the team estimates 9 months of work for the first 10 states).
ceo_board_deck_excerpt.md5 lines · Download
# US launch (CEO's board deck, excerpt)

- Market: 33 million US small businesses x our UK price of £120 ($150) a year = about $5B. "Even 1% is $50M."
- Plan: launch in the US in Q1, $2.5M budget, hire a US sales team of 6, target $5M ARR in year one.
- Why now: a competitor raised in the US; we need to get there first.
uk_customers_by_size.csv4 rows · Download
company_size,uk_customers,share_of_uk_customers_pct,retained_at_12_months_pct,avg_revenue_usd_per_year
Sole trader,6100,41,18,150
2-9 staff,5300,36,33,150
10-49 staff,2900,20,71,620
50-249 staff,480,3,78,2400
us_small_business_stats.md5 lines · Download
# US small businesses (government statistics)

- 33.2 million small businesses in total.
- About 27.1 million have no employees (sole proprietors).
- About 6.1 million have employees; of those, about 1.1 million have 10 to 249 employees.
What a strong answer doesThe answer key the graders mark against

Shows the $5B market is the wrong number. It multiplies every US small business by the UK price, but in the UK only businesses with 10+ staff stay (71% and 78% at 12 months) and pay more ($620 to $2,400 a year); sole traders and 2-9 businesses mostly leave (18% and 33%). Most US small businesses (27.1M of 33.2M) have no employees and don't need payroll at all; the comparable US market is about 1.1M businesses with 10 to 249 staff, worth roughly $0.7B to $1B at UK revenue per customer, not $5B. The tiger: US payroll needs state-by-state tax rules, about 9 months for the first 10 states, so a Q1 launch with payroll isn't possible and $5M ARR in year one would need thousands of 10+ staff customers from a standing start. Dismisses the 'first mover' worry as less scary: a competitor's raise doesn't make a market winner-takes-all. Proposes instead: target 10+ staff businesses, start with HR in the US while payroll is built for a few high-density states, test demand with a small budget and a clear threshold before hiring a sales team, and fix the UK mix towards 10+ staff companies meanwhile.

Critical failures (cap the score at 40)
  • Endorses the $5B market size
Task

Staff-level · v1.1

The outputs

9 setups answered this brief.

Needs more than a quick edit7 of 10 checks passed

The output relies on unsupported claims about US sales costs, state tax regimes and competitor pricing that a PM would have to fact-check before using it. A quick edit cannot fix missing evidence.

Output

MEMO

TO: CEO FROM: VP of Product SUBJECT: Red-Teaming the US Launch Board Deck

You asked me to challenge the US expansion plan before it goes to the board next week. To be blunt: if you present this slide as-is, the board will tear it apart. The math contradicts our own internal data, the unit economics of the go-to-market plan are upside down, and the timeline ignores our engineering reality.

Here is what the plan gets wrong, what will sink it, what is actually less scary than it looks, and what we should pitch instead.

1. What the Plan Gets Wrong

The "1% of $5B" TAM fallacy: The deck claims a $5B market based on 33.2M US small businesses. However, 27.1M of those are sole proprietors with no employees. They don’t need HR or multi-staff payroll. In the UK, sole traders are 41% of our user base, pay us $150/year, and churn at an astonishing 82% in year one. Even worse, the 2–9 staff segment (another 5M US businesses) only retains at 33% for us.

If you look at our UK data, Halyard actually has product-market fit in the 10–249 employee segment. Although they represent only 23% of our UK customer count, they retain at 71%–78% and generate 75% of our total ARR ($2.95M of our $3.9M). In the US, there are only 1.1 million businesses of this size. Our true, healthy US market isn't 33 million businesses; it’s 1.1 million.

The Year-One Target ($5M ARR): It took us five years to build $3.9M ARR in our home market. Promising the board we will generate $5M of new ARR in a foreign market in 12 months is not credible. At a $150 price point, $5M ARR requires acquiring 33,333 customers in year one—more than double our entire UK customer base. Even if we exclusively target the larger 10–249 staff segment (weighted average ~$870/year), we would need ~5,750 customers in year one. We only have 3,380 of them in the UK after five years.

The "Why Now": "A competitor raised money to go to the US" is a trap, not a thesis. If they are chasing the same $150/year, high-churn micro-businesses with a costly US sales team, let them burn their venture capital doing it.

2. What Will Sink the Plan

The Sales Team Unit Economics: Hiring 6 US sales reps to sell a $150–$620/year product is financial suicide. A fully loaded US B2B sales rep costs $120k–$150k/year. Six reps will burn ~$800k+ in payroll alone. You cannot use human beings to sell a $150/year product; the Customer Acquisition Cost (CAC) will eclipse the Lifetime Value (LTV) by orders of magnitude. To hit $5M ARR at $150 ACV, each of those 6 reps would have to close 22 deals per day, every working day of the year, starting on Day 1.

The Q1 Launch vs. The 9-Month Build: The deck proposes launching in Q1, yet my engineering team estimates 9 months of work just to build the tax rules for the first 10 states. If we hire 6 salespeople in Q1, what are they selling for the first three quarters of the year? We will burn through our $2.5M budget paying a sales team to sit on their hands while waiting for Product to clear regulatory hurdles.

3. What is Less Scary Than It Looks

The "50-State" Regulatory Mountain: Building US payroll sounds terrifying because of the 50-state tax complexity, and 9 months for 10 states sounds slow. It’s actually much less scary than it looks, for two reasons: 1. Concentration: The top 10 US states (CA, TX, NY, FL, etc.) account for over 50% of the US population and GDP. We don’t need 50 states to have a massive market; 10 states gets us half the country. 2. No-Tax States: Nine US states (including massive small-business hubs like Texas, Florida, and Washington) have no state income tax. The payroll rules there are largely federal. We can build for those states in a fraction of the time, giving us a beachhead much faster than 9 months.

Competitor Head-Start: The US market for 10–249 employee businesses is fragmented and notoriously underserved by legacy players (like ADP and Paychex, who have terrible UX) and modern players (like Gusto, which has become very expensive). Being "first" doesn't matter; having the right price-to-value ratio for the 10–249 segment does.

4. What I Propose Instead

Before going to the board for a $2.5M US gamble, we need to align our strategy with our actual strengths. Here is the counter-proposal:

1. Pivot the ICP to 10–249 Employees (In the UK and US): Stop marketing to sole traders. They are a massive distraction destroying our retention metrics. Double down on the 10–249 employee segment where we have 70%+ retention and $620–$2,400 ACVs. 2. Raise Prices for the US: US competitors charge a base fee ($40+/mo) plus $6–$12 per employee per month. For a 20-person company, that’s ~$2,000/year. Our UK price of $620/year is way too cheap for the US market. If we price at even $1,500/year for the 10–49 segment, our US TAM in just the top 10 states is still ~$800M, and the unit economics actually work. 3. Phase the US Launch (Kill the Q1 Sales Team): * Phase 1 (Months 1–6): Spend $500k (or use an embedded payroll API like Check or Gusto Embedded to skip the build) to launch only in Texas and Florida (massive markets, zero state income tax). * Phase 2 (Months 6–9): Use Product-Led Growth (PLG) and digital marketing—not a sales team—to acquire our first 100 US customers (10–249 staff) to prove retention and pricing. * Phase 3 (Month 10+): Once retention is proven and the next 8 states are built, then raise the $2.5M to scale marketing and hire inside sales for the 50–249 segment.

Let’s rewrite the deck to show the board we understand our own data—and that we know how to win the US without lighting $2.5M on fire.

Claims the judge couldn’t find in the brief

  • A fully loaded US B2B sales rep costs $120k–$150k/year.
  • The top 10 US states (CA, TX, NY, FL, etc.) account for over 50% of the US population and GDP.
  • Nine US states (including Texas, Florida, and Washington) have no state income tax, making payroll rules largely federal.
  • US competitors charge a base fee ($40+/mo) plus $6–$12 per employee per month.
  • Gusto has become very expensive.
  • The US market for 10–249 employee businesses is fragmented and underserved by legacy players with terrible UX.

Grades and run details

Decision model 60 · LLM judge 73
Decision model checks
  • failUses the supplied evidence correctly91%
  • passAddresses the actual decision67%
  • passRespects explicit constraints34%
  • partialIdentifies material uncertainty45%
  • failAvoids unsupported claims41%
  • passProduces the required deliverable77%
  • partialRebuilds the market size62%
  • passFinds the timeline that sinks it100%
  • passTigers, not paper tigers34%
  • failFinds the need behind the idea23%
Run
Run
#1
API response time
16 s
Submitted
13 Oct 2026

Usable with a quick edit7 of 10 checks passed

The memo is well-reasoned and actionable; the unsupported local-taxes claim is minor and can be quickly removed or hedged without redoing the analysis.

Output

To: CEO and Board From: VP Product Subject: US expansion: approve validation, not the proposed launch

Recommendation: Do not approve the plan as written. The US may be attractive, but the deck mistakes business counts for demand, imports our weakest UK economics, and commits to distribution before we can reliably deliver the product. Approve a staged validation programme instead, with compliance and commercial gates before releasing the full $2.5M.

What the plan gets wrong

1. The addressable market is not 33 million payroll buyers. Of 33.2 million US small businesses, 27.1 million have no employees. Some owner-only businesses may need payroll, but they are not equivalent to employer businesses. The relevant starting population is approximately 6.1 million employers—and probably the 1.1 million with 10–249 employees, given our UK results.

“Even 1%” is not a strategy. It assumes a reachable segment, competitive differentiation, distribution and retention that we have not demonstrated.

2. Our own data contradicts the proposed price and customer strategy.

UK segmentCustomers12-month retentionAnnual revenue/customer
------:---:---:
Sole trader6,10018%$150
2–9 staff5,30033%$150
10–49 staff2,90071%$620
50–249 staff48078%$2,400

Businesses with 10–249 staff represent approximately 23% of customers but 63% of the annual revenue implied by this table. Their weighted retention is approximately 72%, versus 25% for customers below ten staff. Our strongest evidence favours larger employers, not maximum account volume at $150.

Using our UK customer mix within 10–249 staff gives approximately $873 annual revenue per customer. Applied to 1.1 million US businesses, that suggests a roughly $960M segment revenue pool—not a forecast, and not all serviceable by a ten-state product. US willingness to pay remains untested.

There is also a board-data problem: the table implies $4.66M annual revenue, versus the stated $3.9M ARR. Reconcile definitions, dates and cohorts before presenting either figure as the economic baseline.

3. The sales plan and revenue target do not fit. At $150 annually, $5M ARR requires approximately 33,333 active customers: 463 per salesperson per month across six reps over a full year, before churn. Even allocating the entire $2.5M budget to acquisition would allow only $75 per active customer, leaving nothing for engineering, compliance or support.

At the illustrative $873 blend, we still need approximately 5,730 active customers—80 per rep per month. These are workload checks, not sales forecasts, but neither resembles a credible unproven, sales-led launch. A later launch compresses the selling window further.

4. Competitive fundraising is not customer evidence. A competitor’s financing may raise acquisition costs or accelerate competition. It does not establish that our product is ready or that customers will switch. “Get there first” is especially weak when the competitor is already there.

What would sink it

Payroll reliability and coverage. The team estimates nine months for the first ten states. A Q1 launch is unsupported unless substantial work is already complete or we use a proven partner. Ten states also do not automatically mean ten self-contained markets: employee locations, local taxes and multi-state operations affect eligibility. Incorrect wages, filings or remittances can destroy trust faster than sales can rebuild it.

Scaling the wrong economics. At our smallest-customer retention levels, acquisition fills a leaking bucket. Higher retention among larger employers is promising, but customer retention is not revenue retention or gross margin. We lack US acquisition cost, onboarding effort, support burden and contribution-margin evidence.

Starving the UK business. The proposed budget is approximately 64% of current ARR—not a cash-runway calculation, but a material commitment. We need an explicit runway and UK-roadmap impact assessment, including engineering diversion and downside scenarios.

What is less scary than it looks

We do not need nationwide coverage or a large share of the US to build something valuable. A tightly qualified segment can support a meaningful business.

Nor must we necessarily build every payroll component ourselves. A payroll infrastructure partner could shorten time to market, although coverage, margins, accountability and vendor dependence require diligence.

Finally, weak microbusiness retention does not prove Halyard cannot travel. The larger-employer cohorts provide a credible starting hypothesis. They do not yet prove US product-market fit.

What I would propose instead

1. Release up to $400,000 for a 90-day validation phase; hold the remainder. Reconcile the financial baseline, analyse retention by cohort and acquisition channel, and compare partner versus internal-build economics and timelines.

2. Start with 10–49 employees. Recruit 10–20 design partners with simple, geographically supported payroll needs. Validate switching triggers, differentiation, pricing and onboarding. Keep 50–249 as a subsequent hypothesis: better UK retention may come with substantially greater implementation complexity.

3. Gate live payroll on readiness, not the calendar. Require documented coverage, independent compliance review, successful parallel payroll testing, and clear incident ownership. No unsupported employee locations.

4. Run a 30–50-customer paid pilot before hiring six sellers. Use founder-led sales and limited implementation capacity. Measure acquisition cost, onboarding time, support cost, contribution margin and retention. Seek evidence supporting acquisition payback within 12–18 months; early retention is provisional until cohorts mature.

5. Return for scale funding only after those gates. Replace the $5M year-one promise with a bottom-up forecast based on verified price, conversion, launch timing, capacity and churn. The board should fund the next proven step—not an unsupported percentage of America.

Claims the judge couldn’t find in the brief

  • Employee locations, local taxes and multi-state operations affect eligibility.

Grades and run details

Decision model 85 · LLM judge 82
Decision model checks
  • partialUses the supplied evidence correctly7%
  • passAddresses the actual decision98%
  • passRespects explicit constraints53%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims30%
  • passProduces the required deliverable85%
  • passRebuilds the market size91%
  • passFinds the timeline that sinks it100%
  • passTigers, not paper tigers88%
  • failFinds the need behind the idea10%
Run
Run
#1
Time to output
51 s
Submitted
5 Oct 2026

Needs more than a quick edit8 of 10 checks passed

The TAM calculation incorrectly applies the $150 price to 10+ staff businesses, so a PM would need to redo the market sizing and adjust the argument before using it.

Output

To: CEO Subject: US launch — don’t take this plan to the board as written

Bottom line: The US could be a real opportunity, but this deck turns a large headline market into an unsupported year-one sales forecast. The biggest risk isn’t arriving late; it’s selling payroll before we can run it reliably in the states we promise to support.

What the plan gets wrong

  • The TAM math is arithmetically fine and commercially misleading. The 33.2 million small-business count includes 27.1 million firms with no employees—about 82% of the total. That is not a credible proxy for payroll buyers. There are 6.1 million employer firms, including 1.1 million with 10–249 employees. Even those are not automatically our customers. Applying our UK price to every US small business assumes identical need, willingness to pay, and product fit.
  • The UK data argues against targeting everyone. Retention is 18% for sole traders and 33% for firms with 2–9 staff, versus 71% for 10–49 and 78% for 50–249. Our best evidence points toward employers with more staff—not a mass-market pitch built on the largest, weakest-fit segment.
  • The price and revenue baseline need reconciling. At $150 per year, $5 million ARR means roughly 33,300 customers. That is more than twice our current UK customer base, which took five years to build. Also, the UK segment counts and stated average revenues imply about $4.66 million in annual revenue, versus the reported $3.9 million ARR. These may use different definitions, but we must explain the $760,000 gap before using UK economics to price or forecast the US.
  • “A competitor raised” is not a launch strategy. It may validate investor interest, but it tells us nothing about buyer demand for Halyard, our differentiation, acquisition cost, or which states and customer segments we can serve profitably.

What would sink it

The timeline is incompatible with the product plan. We estimate nine months to build payroll tax rules for the first 10 states; the deck promises a Q1 US launch. A launch without dependable tax calculations, filings, registrations, and support creates compliance exposure and destroys trust. “Ten states” is not the same as nationwide coverage, and the deck does not say what launch actually means.

The target compounds the problem. At the proposed price, we need 33,300 customers in year one—without a validated US product, channel, conversion rate, or sales cycle. Six sales hires do not make that forecast credible. The $2.5 million budget also needs to cover product, tax expertise, support, and go-to-market; the deck gives no cost or hiring ramp to show it can.

What’s less scary than it looks

The market does not need to be 33 million businesses to matter. The 6.1 million employer firms are a meaningful starting universe; at $150 per year, that price would imply a crude ceiling of about $915 million, not a forecast. The 1.1 million firms with 10–249 employees alone imply about $165 million at that same price. Those figures are still not a serviceable-market estimate, but they show we can pursue a focused wedge without pretending every sole proprietor is a prospect.

We also have useful evidence: five years of operating experience, 14,800 UK customers, and better retention among larger employers. That is a foundation for testing the US—not proof that the UK playbook transfers unchanged.

What I’d propose instead

Approve a staged US entry, not a Q1 national launch or a $5 million year-one commitment.

  1. Choose a narrow first customer and geography. Start with employer businesses, prioritizing 10–49 staff; select states using verified prospect demand and implementation feasibility. Defer sole traders and the weakest-retaining segment until we have evidence they can work.
  2. Build and validate before scaling sales. Recruit US design partners, test willingness to pay and workflows, and ship only when payroll accuracy, filing operations, and support are proven in the launch states. Treat the nine-month estimate as a planning constraint, not a promise to cover 10 states by Q1.
  3. Stage the investment. Fund discovery, state selection, and the first production-ready footprint now. Hold most sales hiring until pilots demonstrate successful payroll cycles, repeatable acquisition, and acceptable retention and unit economics. Consider accountant or payroll-partner channels before assuming six direct sellers are the answer.
  4. Return to the board with evidence. Bring a state-by-state scope, reconciled UK revenue data, tested US pricing, customer pipeline, launch-readiness criteria, and a bottom-up forecast tied to conversion and sales capacity.

We should move with urgency—but urgency should accelerate learning, not turn an unvalidated forecast into a launch promise.

Grades and run details

Decision model 85 · LLM judge 82
Decision model checks
  • passUses the supplied evidence correctly38%
  • passAddresses the actual decision88%
  • passRespects explicit constraints80%
  • passIdentifies material uncertainty92%
  • passAvoids unsupported claims43%
  • passProduces the required deliverable80%
  • partialRebuilds the market size92%
  • passFinds the timeline that sinks it100%
  • passTigers, not paper tigers67%
  • failFinds the need behind the idea49%
Run
Run
#1
API response time
29 s
Submitted
6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 3

Uses the supplied evidence correctlyWrongWrongRight
Gemini 4 Argon · API

The output invents several facts not in the supplied context, including US sales rep costs, state tax structures, population concentration, and competitor pricing, without labelling them as assumptions.

GPT-6 Astra · ChatGPT

Claims that local taxes and multi-state operations affect eligibility are not supported by the supplied context, which only mentions state-by-state tax rules.

GPT-6 Luna · API

All claims about the current situation are directly supported by the supplied context or arithmetic from it.

Avoids unsupported claimsWrongMixedRight
Gemini 4 Argon · API

Several claims about US market conditions (sales rep costs, no-tax states, competitor pricing) are presented as established facts without evidence from the supplied context.

GPT-6 Astra · ChatGPT

Presents the existence of local taxes and multi-state complexity as fact without support from the context.

GPT-6 Luna · API

Interpretations and forecasts are presented as such, not as established facts; the revenue gap is arithmetic, not speculation.

Rebuilds the market sizeRightRightWrong
Gemini 4 Argon · API

It correctly uses the US employer counts and UK retention data to argue that only 1.1M businesses are addressable, not 33.2M, effectively rebuilding to a much smaller market.

GPT-6 Astra · ChatGPT

Rebuilds market using employer counts and UK retention, arriving at ~$960M for 1.1M businesses.

GPT-6 Luna · API

It uses the $150 price for 10+ staff businesses instead of the higher UK revenue per customer for those segments, so the market size is understated and not correctly rebuilt.

All mixed 1

Finds the need behind the ideaMixedMixedMixed
Gemini 4 Argon · API

It identifies the underlying need for US growth, proposes a phased, PLG-led alternative with a smaller budget, and explicitly displaces the immediate sales team and broad targeting.

GPT-6 Astra · ChatGPT

Identifies need for US expansion, proposes validation instead of full launch, with cost ($400k) and what is displaced.

GPT-6 Luna · API

It identifies the need to capture the US opportunity and proposes a staged entry as an alternative, with lower upfront cost and what it displaces (national launch, $5M target).

All got right 6

Addresses the actual decisionRightRightRight
Gemini 4 Argon · API

The memo commits early to a clear alternative plan (phased launch, pivot ICP) and states what would need to be proven to scale (retention and pricing).

GPT-6 Astra · ChatGPT

Commits to not approving the plan and proposes a staged validation, with clear conditions for scaling.

GPT-6 Luna · API

The memo commits to a clear call (don't take the plan as written, do a staged entry) early, framed for the CEO, and implies the decision would change based on pilot evidence.

Respects explicit constraintsRightRightRight
Gemini 4 Argon · API

The output is a memo under 1,000 words addressed to the CEO, respecting the requested form and length.

GPT-6 Astra · ChatGPT

Memo format, under 1,000 words, addressed to CEO and board.

GPT-6 Luna · API

The output is a memo to the CEO, under 1,000 words, and addresses all requested elements.

Identifies material uncertaintyRightRightRight
Gemini 4 Argon · API

It identifies that US retention and pricing are unproven and makes the scaling decision contingent on validating them with a small initial cohort.

GPT-6 Astra · ChatGPT

Identifies untested US willingness to pay, acquisition costs, and retention, and proposes validation to resolve them.

GPT-6 Luna · API

It names unknowns (US demand, conversion, unit economics, revenue data gap) and proposes pilots and evidence thresholds to resolve them.

Produces the required deliverableRightRightRight
Gemini 4 Argon · API

The memo is complete, in the correct format, and usable by the CEO to present an alternative board proposal.

GPT-6 Astra · ChatGPT

Complete memo with actionable recommendation, within length, usable with light edits.

GPT-6 Luna · API

The memo is complete, in the right form, within length, and a CEO could act on it with light edits.

Finds the timeline that sinks itRightRightRight
Gemini 4 Argon · API

The memo explicitly names the 9-month state-tax build and uses it to show that a Q1 launch with payroll is impossible and that the year-one ARR target is undermined.

GPT-6 Astra · ChatGPT

Cites 9-month build and concludes Q1 launch is unsupported.

GPT-6 Luna · API

It explicitly names the 9-month build and explains why it makes a Q1 payroll launch impossible and the year-one target unachievable.

Tigers, not paper tigersRightRightRight
Gemini 4 Argon · API

It separates the manageable competitor-head-start worry from the truly dangerous risks (unit economics, TAM fallacy, build timeline) and surfaces the UK mix problem with sole traders.

GPT-6 Astra · ChatGPT

Separates sinkers (payroll reliability, scaling wrong economics) from less scary (competitor raise, nationwide coverage), and surfaces ARR discrepancy.

GPT-6 Luna · API

It ranks the timeline/product risk as the sinker, dismisses the first-mover worry as less scary, and surfaces the UK revenue data discrepancy.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 75% of 75 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6.1 SolwithAPI97.586.42None
2Haiku 5.5withAPI87.590.92None
3GPT-6 LunawithAPI90.490.92None
4GPT-6 AstrawithChatGPT88.379.42None
5Gemini 4 ArgonwithAPI73.882.52None
6Sonnet 5.5withAPI85.474.82None
7Opus 5.5withClaude85.479.421 capped
8Gemini 3.8 FlashwithAPI87.972.721 capped
9Gemini 3.5 Flash-LitewithGemini69.649.721 capped

What you get for the cost

GPT-6 Luna · API scores within a point of GPT-6.1 Sol · API at about a tenth of the cost.

Task score against relative cost for each setup. The table below has the same figures.4050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6.1 Sol · API: task score 90, 11× the cheapestHaiku 5.5 · API: task score 90, the cheapestGPT-6 Luna · API: task score 89, the cheapestGPT-6 Astra · ChatGPT: task score 82, 60× the cheapest (1 of 2 outputs estimated)Gemini 4 Argon · API: task score 77, 13× the cheapestSonnet 5.5 · API: task score 77, 37× the cheapestOpus 5.5 · Claude: task score 60, 59× the cheapest (1 of 2 outputs estimated), 1 critical failureGemini 3.8 Flash · API: task score 50, 9.5× the cheapest, 1 critical failureGemini 3.5 Flash-Lite · Gemini: task score 45, 1.9× the cheapest (1 of 2 outputs estimated), 1 critical failureGPT-6.1 Sol · APIHaiku 5.5 · APIGPT-6 Luna · APIGPT-6 Astra · ChatGPTGemini 4 Argon · APISonnet 5.5 · APIOpus 5.5 · ClaudeGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)At least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API891.0×
Haiku 5.5 · API901.4×
Gemini 3.5 Flash-Lite · Gemini451.9× (partly estimated)
Gemini 3.8 Flash · API509.5×
GPT-6.1 Sol · API9011×
Gemini 4 Argon · API7713×
Sonnet 5.5 · API7737×
Opus 5.5 · Claude6059× (partly estimated)
GPT-6 Astra · ChatGPT8260× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Pressure-testing a proposal before committing a team to it.

Why it matters

The useful critic finds the one assumption everything rests on. Theatrical negativity is easy to generate and useless in a planning meeting.

What good looks like

  • Identifies the load-bearing assumption
  • Separates the risks that could kill it from the ones that only look scary
  • Uses the supplied evidence, not generic risks
  • Proposes the cheapest way to test the assumption

Deliberately not measured

  • Tone
  • Number of objections raised
Capability tested

Evidence-based critique

The failure we’re looking for

Theatrical negativity without evidence

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

Vanilla prompt (core) · With Roast Me skill · Staff level: a company bet with a long evidence pack