Tasks / Strategy

Challenge an idea

Can the model find the strongest reason an idea may fail, backed by evidence?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 44% were usable with at most a quick edit.

Reliably right

  1. Finds the load-bearing assumption100% pass
    The critique targets the assumption that agencies' growth is limited by a shortage of new leads, using referral, abandoned-outbound, and capacity evidence.
    GPT-6.1 Sol · API · An AI SDR for small agencies
  2. Uses the interviews faithfully100% pass
    All quotes are accurately taken from the transcripts and correctly attributed to A05, A04, and A08.
    GPT-6.1 Sol · API · An AI SDR for small agencies
  3. Finds the timeline that sinks it100% pass
    Cites 9-month build and concludes Q1 launch is unsupported.
    GPT-6 Astra · ChatGPT · Five billion dollars, on paper

Where it slips

  1. Uses the supplied evidence correctly50% pass
    Claims that local taxes and multi-state operations affect eligibility are not supported by the supplied context, which only mentions state-by-state tax rules.
    GPT-6 Astra · ChatGPT · Five billion dollars, on paper
  2. Avoids unsupported claims56% pass
    Presents the existence of local taxes and multi-state complexity as fact without support from the context.
    GPT-6 Astra · ChatGPT · Five billion dollars, on paper
  3. Finds the need behind the idea71% pass
    It mentions A05's past-client reminder idea but does not state the underlying need or offer a full alternative with its cost and what it displaces.
    GPT-6 Astra · ChatGPT · An AI SDR for small agencies

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Finds the need behind the idea

    Does the critique identify the underlying need the idea is meant to serve, and offer at least one other way to meet that need, at a different cost, saying what would have to be dropped for it?

    Passes when States the need behind the proposal, and gives an alternative way to meet it with its cost and what it displaces.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Tigers, not paper tigers · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the VP Product at Halyard. Our CEO is taking the US launch plan below to the board next week and has asked you to challenge it hard first. Write a memo of no more than 1,000 words: what the plan gets wrong, what would sink it, what's less scary than it looks, and what you'd propose instead. The deck excerpt and our data are attached.

What the model was given4 items: about_halyard.md, ceo_board_deck_excerpt.md, uk_customers_by_size.csv, us_small_business_stats.md
about_halyard.md3 lines · Download
# About Halyard in the UK

Payroll and HR for small businesses. 14,800 customers, $3.9M ARR, five years old. UK payroll rules are built in; US payroll would need state-by-state tax rules (the team estimates 9 months of work for the first 10 states).
ceo_board_deck_excerpt.md5 lines · Download
# US launch (CEO's board deck, excerpt)

- Market: 33 million US small businesses x our UK price of £120 ($150) a year = about $5B. "Even 1% is $50M."
- Plan: launch in the US in Q1, $2.5M budget, hire a US sales team of 6, target $5M ARR in year one.
- Why now: a competitor raised in the US; we need to get there first.
uk_customers_by_size.csv4 rows · Download
company_size,uk_customers,share_of_uk_customers_pct,retained_at_12_months_pct,avg_revenue_usd_per_year
Sole trader,6100,41,18,150
2-9 staff,5300,36,33,150
10-49 staff,2900,20,71,620
50-249 staff,480,3,78,2400
us_small_business_stats.md5 lines · Download
# US small businesses (government statistics)

- 33.2 million small businesses in total.
- About 27.1 million have no employees (sole proprietors).
- About 6.1 million have employees; of those, about 1.1 million have 10 to 249 employees.
What a strong answer doesThe answer key the graders mark against

Shows the $5B market is the wrong number. It multiplies every US small business by the UK price, but in the UK only businesses with 10+ staff stay (71% and 78% at 12 months) and pay more ($620 to $2,400 a year); sole traders and 2-9 businesses mostly leave (18% and 33%). Most US small businesses (27.1M of 33.2M) have no employees and don't need payroll at all; the comparable US market is about 1.1M businesses with 10 to 249 staff, worth roughly $0.7B to $1B at UK revenue per customer, not $5B. The tiger: US payroll needs state-by-state tax rules, about 9 months for the first 10 states, so a Q1 launch with payroll isn't possible and $5M ARR in year one would need thousands of 10+ staff customers from a standing start. Dismisses the 'first mover' worry as less scary: a competitor's raise doesn't make a market winner-takes-all. Proposes instead: target 10+ staff businesses, start with HR in the US while payroll is built for a few high-density states, test demand with a small budget and a clear threshold before hiring a sales team, and fix the UK mix towards 10+ staff companies meanwhile.

Critical failures (cap the score at 40)
  • Endorses the $5B market size
Task

Staff-level · v1.1

The outputs

9 setups answered this brief.

Needs more than a quick edit7 of 10 checks passed

The output relies on unsupported claims about US sales costs, state tax regimes and competitor pricing that a PM would have to fact-check before using it. A quick edit cannot fix missing evidence.

Output

MEMO

TO: CEO FROM: VP of Product SUBJECT: Red-Teaming the US Launch Board Deck

You asked me to challenge the US expansion plan before it goes to the board next week. To be blunt: if you present this slide as-is, the board will tear it apart. The math contradicts our own internal data, the unit economics of the go-to-market plan are upside down, and the timeline ignores our engineering reality.

Here is what the plan gets wrong, what will sink it, what is actually less scary than it looks, and what we should pitch instead.

1. What the Plan Gets Wrong

The "1% of $5B" TAM fallacy: The deck claims a $5B market based on 33.2M US small businesses. However, 27.1M of those are sole proprietors with no employees. They don’t need HR or multi-staff payroll. In the UK, sole traders are 41% of our user base, pay us $150/year, and churn at an astonishing 82% in year one. Even worse, the 2–9 staff segment (another 5M US businesses) only retains at 33% for us.

If you look at our UK data, Halyard actually has product-market fit in the 10–249 employee segment. Although they represent only 23% of our UK customer count, they retain at 71%–78% and generate 75% of our total ARR ($2.95M of our $3.9M). In the US, there are only 1.1 million businesses of this size. Our true, healthy US market isn't 33 million businesses; it’s 1.1 million.

The Year-One Target ($5M ARR): It took us five years to build $3.9M ARR in our home market. Promising the board we will generate $5M of new ARR in a foreign market in 12 months is not credible. At a $150 price point, $5M ARR requires acquiring 33,333 customers in year one—more than double our entire UK customer base. Even if we exclusively target the larger 10–249 staff segment (weighted average ~$870/year), we would need ~5,750 customers in year one. We only have 3,380 of them in the UK after five years.

The "Why Now": "A competitor raised money to go to the US" is a trap, not a thesis. If they are chasing the same $150/year, high-churn micro-businesses with a costly US sales team, let them burn their venture capital doing it.

2. What Will Sink the Plan

The Sales Team Unit Economics: Hiring 6 US sales reps to sell a $150–$620/year product is financial suicide. A fully loaded US B2B sales rep costs $120k–$150k/year. Six reps will burn ~$800k+ in payroll alone. You cannot use human beings to sell a $150/year product; the Customer Acquisition Cost (CAC) will eclipse the Lifetime Value (LTV) by orders of magnitude. To hit $5M ARR at $150 ACV, each of those 6 reps would have to close 22 deals per day, every working day of the year, starting on Day 1.

The Q1 Launch vs. The 9-Month Build: The deck proposes launching in Q1, yet my engineering team estimates 9 months of work just to build the tax rules for the first 10 states. If we hire 6 salespeople in Q1, what are they selling for the first three quarters of the year? We will burn through our $2.5M budget paying a sales team to sit on their hands while waiting for Product to clear regulatory hurdles.

3. What is Less Scary Than It Looks

The "50-State" Regulatory Mountain: Building US payroll sounds terrifying because of the 50-state tax complexity, and 9 months for 10 states sounds slow. It’s actually much less scary than it looks, for two reasons: 1. Concentration: The top 10 US states (CA, TX, NY, FL, etc.) account for over 50% of the US population and GDP. We don’t need 50 states to have a massive market; 10 states gets us half the country. 2. No-Tax States: Nine US states (including massive small-business hubs like Texas, Florida, and Washington) have no state income tax. The payroll rules there are largely federal. We can build for those states in a fraction of the time, giving us a beachhead much faster than 9 months.

Competitor Head-Start: The US market for 10–249 employee businesses is fragmented and notoriously underserved by legacy players (like ADP and Paychex, who have terrible UX) and modern players (like Gusto, which has become very expensive). Being "first" doesn't matter; having the right price-to-value ratio for the 10–249 segment does.

4. What I Propose Instead

Before going to the board for a $2.5M US gamble, we need to align our strategy with our actual strengths. Here is the counter-proposal:

1. Pivot the ICP to 10–249 Employees (In the UK and US): Stop marketing to sole traders. They are a massive distraction destroying our retention metrics. Double down on the 10–249 employee segment where we have 70%+ retention and $620–$2,400 ACVs. 2. Raise Prices for the US: US competitors charge a base fee ($40+/mo) plus $6–$12 per employee per month. For a 20-person company, that’s ~$2,000/year. Our UK price of $620/year is way too cheap for the US market. If we price at even $1,500/year for the 10–49 segment, our US TAM in just the top 10 states is still ~$800M, and the unit economics actually work. 3. Phase the US Launch (Kill the Q1 Sales Team): * Phase 1 (Months 1–6): Spend $500k (or use an embedded payroll API like Check or Gusto Embedded to skip the build) to launch only in Texas and Florida (massive markets, zero state income tax). * Phase 2 (Months 6–9): Use Product-Led Growth (PLG) and digital marketing—not a sales team—to acquire our first 100 US customers (10–249 staff) to prove retention and pricing. * Phase 3 (Month 10+): Once retention is proven and the next 8 states are built, then raise the $2.5M to scale marketing and hire inside sales for the 50–249 segment.

Let’s rewrite the deck to show the board we understand our own data—and that we know how to win the US without lighting $2.5M on fire.

Claims the judge couldn’t find in the brief

  • A fully loaded US B2B sales rep costs $120k–$150k/year.
  • The top 10 US states (CA, TX, NY, FL, etc.) account for over 50% of the US population and GDP.
  • Nine US states (including Texas, Florida, and Washington) have no state income tax, making payroll rules largely federal.
  • US competitors charge a base fee ($40+/mo) plus $6–$12 per employee per month.
  • Gusto has become very expensive.
  • The US market for 10–249 employee businesses is fragmented and underserved by legacy players with terrible UX.

Grades and run details

Decision model 60 · LLM judge 73
Decision model checks
  • failUses the supplied evidence correctly91%
  • passAddresses the actual decision67%
  • passRespects explicit constraints34%
  • partialIdentifies material uncertainty45%
  • failAvoids unsupported claims41%
  • passProduces the required deliverable77%
  • partialRebuilds the market size62%
  • passFinds the timeline that sinks it100%
  • passTigers, not paper tigers34%
  • failFinds the need behind the idea23%
Run
Run
#1
API response time
16 s
Submitted
13 Oct 2026

Usable with a quick edit7 of 10 checks passed

Minor unsupported claim about crowded market can be edited out; core reasoning and evidence use are solid.

Output

MEMO

To: CEO From: VP Product Re: Challenge to the US launch plan before the board

Bottom line: The market story is wrong, the timeline is not achievable, and the budget can't carry the target. The opportunity is real but much smaller and more specific than $5B. I'd take the board a staged plan with gates, not a Q1 launch.

---

What the plan gets wrong

1. The TAM is built on the wrong customers. 33M × $150 assumes every US small business buys our product at our UK sole-trader price. But 27.1M of those firms have no employees. Payroll is not their problem. Our own data shows sole traders retain at 18% at twelve months and bring in $150 a year. Our customers who retain are the ones with staff.

2. The right pool is about 1.1M firms, and the "1%" math shrinks accordingly. The government figure is 1.1M businesses with 10–249 employees. Our UK revenue is concentrated in the 10–49 and 50–249 segments: about 23% of customers but roughly 60%+ of revenue, with 71–78% retention. Taking a blended ~$620 ARPU, 1% of 1.1M firms is about 11,000 customers and roughly $7M ARR, not $50M. The deck's own logic, applied to the right segment, gives a much smaller prize.

3. The $5M year-one target is not credible. At $620 ARPU, $5M needs about 8,000 paying US customers in twelve months. It took us five years to reach 14,800 UK customers, with a product that is already built. At $150, it needs 33,000, which only works if we sell to the segment that churns most.

4. Q1 is not feasible. The US needs state-by-state payroll tax rules. Our team estimates nine months for the first ten states alone. Q1 launch means either launching with almost no state coverage or launching with a product we can't legally run for most prospects. A firm with employees in an unsupported state cannot onboard at all.

5. "Get there first" is the wrong reason. US payroll is a crowded market with entrenched incumbents, and a competitor's raise means they face the same state-tax build we do. Being early doesn't create a moat in payroll. Being correct on compliance does.

6. The budget doesn't match the plan. Six US quota-carrying reps at a fully loaded cost of roughly $150–180K each is about $1M a year before marketing, tax build, legal, compliance, or US support. That leaves very little of the $2.5M for everything else. (These are my estimates. Please check them against Finance's hiring costs.)

---

What would sink it

  • Compliance liability. Wrong withholding or filing in a state is our error, with penalties landing on our customers. One public failure in an early state would damage referrals across the whole US business.
  • Selling to the wrong segment. If sales pursues sole traders and 2–9 employee firms because the TAM looks big, we'll be acquiring customers at $150 ARPU who churn at 67–82% within a year. CAC payback will never work.
  • Hiring ahead of proof. Six reps with quotas before we know which buyer, which price, and which state set will produce a pipeline we can't convert.
  • Support load. Multi-state questions are hard, and US support can't be staffed from London on a launch timeline.
  • Burn without a gate. If US spend starts before UK growth is secure, a US miss would hit our core business's cash position.

---

What's less scary than it looks

  • The product already works where it matters. Our best UK segments retain at 71–78%. We don't need a new product thesis, just a US version of what works.
  • The tax build is bounded. Nine months for ten states is a real cost, but it's finite. Ten states cover roughly half the US population, so coverage is a sequencing problem. We could also evaluate licensing state tax content from a vendor to cut the timeline.
  • The competitor raise is validation. It shows investors believe in US SMB payroll demand. It also shows we'd be competing in a market where a credible challenger still needs years to build coverage.
  • We don't need a Q1 launch to have a US business. A focused pilot can prove demand within two quarters at a fraction of the cost.

---

What I'd propose instead

I'd ask the board to approve a staged program with explicit gates.

Redefine the market in the deck. The target customer is a US business with 10–249 employees. That's about 1.1M firms and a credible multi-million-dollar opportunity, and we should say so.

Stage 1: Validate (months 0–6, ~$300K). Sign 40–60 design-partner firms with 10–249 employees across two states, one large state and one with simpler rules. Test activation, willingness to pay at $600+ ARPU, and 90-day engagement. Also test a partner channel: accountants and bookkeepers serve our best-fit firms and may be cheaper to reach than direct sales.

Stage 2: Build (months 6–15, ~$800K). Build or license state coverage for the top ten states. Hire one or two sales and customer success people, with the founder leading enterprise-style deals. Gate: pilot cohort retention tracking to 70%+ and CAC payback under 18 months.

Stage 3: Scale (only after gates). Expand sales headcount and marketing against measured unit economics.

Revised target: $1M US ARR by month 24, which is roughly 1,600 customers at $620 ARPU. That is ambitious but defensible, and it's the number the board should hold us to.

The ask: Approve Stages 1 and 2 (about $1.1M) now, with Stage 3 contingent on gate review. Do not approve a $2.5M Q1 launch with a $5M year-one target.

---

I know this is a harder message than the deck. But a board that approves the current plan will be disappointed in twelve months. A board that approves a gated path will get real evidence before committing the big money. I'd rather take the second case to them.

Claims the judge couldn’t find in the brief

  • US payroll is a crowded market with entrenched incumbents.

Grades and run details

Decision model 75 · LLM judge 82
Decision model checks
  • failUses the supplied evidence correctly45%
  • passAddresses the actual decision100%
  • passRespects explicit constraints43%
  • passIdentifies material uncertainty95%
  • partialAvoids unsupported claims56%
  • passProduces the required deliverable91%
  • passRebuilds the market size40%
  • passFinds the timeline that sinks it100%
  • passTigers, not paper tigers58%
  • failFinds the need behind the idea28%
Run
Run
#1
API response time
31 s
Submitted
8 Oct 2026

Needs more than a quick edit7 of 10 checks passed

The output includes unsupported claims about US payroll rules (federal/local) and market stickiness that a PM would need to verify before using, so it cannot be used without re-checking evidence.

Output

MEMO: US launch plan, challenge review To: CEO | From: VP Product

Bottom line: The direction is defensible, but the plan isn't ready for the board. The market is about a fifth the size the deck claims, the Q1 launch date conflicts with our own engineering estimate, and the $5M year-one target is unreachable. I'd take a staged version instead.

---

What the plan gets wrong

1. The $5B market is wrong by roughly 5x. The deck multiplies all 33.2M US small businesses by $150. But 27.1M of them have no employees, so they have no payroll to run. Our own UK data shows how little this segment is worth to us:

  • Sole traders are 41% of UK customers but only about 20% of revenue, and just 18% are still with us at 12 months.
  • Firms with 10+ staff are 23% of customers but about 63% of revenue, with 71–78% retention.

Our proven segment in the US is the 1.1M firms with 10–249 employees. At our UK blended revenue for that group (about $870 per customer), that's roughly $1B. "Even 1%" is about $10M, not $50M. I'd exclude the ~5M micro-employers. In the UK, 2–9 staff firms pay $150 and only 33% stay a year.

2. The timeline contradicts itself. The team estimates 9 months of work for the first 10 states. A Q1 launch means selling a product that doesn't exist, or one that only works in a few states. Ten states is also only about half the US population, by my rough estimate, so a national sales team would be pitching to prospects we can't serve.

3. $5M ARR in year one is out of reach. It exceeds the ARR we built in five years in the UK. At ~$870 per customer it needs about 5,700 customers, roughly 950 per rep from a standing start. With a 9-month build, year-one revenue would be close to zero.

4. "A competitor raised" isn't a reason to launch now. We can't win a spending race with $2.5M against a funded rival. Payroll is also a sticky, slow-switching category, so launching early earns us little. Launching with wrong tax calculations would cost us far more.

What would sink it

  • Compliance failures. A wrong withholding or a missed filing means customers pay penalties and blame us. In the UK, rules are built in and we know them. In the US we'd face federal, state and local rules, each with its own filings. One bad quarter in the first cohort could end our reputation with the segment we need.
  • Targeting the wrong customer. If the sales team chases volume, it will sign small firms that churn at UK micro-segment rates. Acquisition cost would never pay back.
  • Draining the UK business. Nine months of engineering on tax rules comes out of the roadmap for the business that funds everything. UK retention in the smallest segments is already weak, and we'd be neglecting it.
  • Burning the budget before learning anything. Six hires plus a 9-month build could consume most of $2.5M, about 64% of current ARR, before we know whether US customers retain like UK ones. I need to confirm whether the $2.5M includes the engineering cost. If it doesn't, the real ask is larger.
  • Missing the target. A $5M goal that slips will damage board confidence in everything else we present.

What's less scary than it looks

  • The smaller market is still big. About $1B is over 200x our current ARR. We don't need $5B to justify the US.
  • The competitor raise. It validates demand, but funding doesn't fix the state-by-state build, which slows them as well. Nothing here is lost by launching in 12–18 months instead of 3.
  • The 9-month estimate. It's a well-scoped build with a known end date, and a tax-calculation partner or API might shorten it. I'd evaluate build vs. buy before committing.
  • UK retention in micro-segments. It looks alarming, but it's a UK issue we'd avoid importing by targeting 10–249 staff firms, where our retention is strongest.

What I'd propose instead

Phase 0 (now to ~Q3, ~$0.5–0.7M): - Decide build vs. buy for the tax engine. - Pick 3–4 launch states by density of 10–249 staff firms. - Recruit 15–20 design-partner customers. - Hire a US head of sales/GTM and one compliance specialist, not six reps.

Phase 1 (launch in 3–5 states once compliance is proven, ~$1M): - Sell only to 10–249 staff firms with 2 reps and dedicated onboarding. - Set the target at about 300 customers and ~$0.25–0.3M ARR twelve months after launch. That's my estimate, based on UK segment pricing.

Phase 2 (release the remaining ~$1M, expand states and team) only if we hit gates: - 12-month retention of at least 70% in the first cohort. - Zero material filing errors. - Acquisition cost paying back within an agreed window.

This puts about $0.5–0.7M at risk before we learn anything, rather than $2.5M, and it gives the board a credible plan with checkpoints.

Before the board meeting

  1. Reconcile ARR. Our segment table implies about $4.66M (customers × average revenue), but the deck and company profile say $3.9M. A board member will do this multiplication. We need to know whether the gap is discounting, churn timing, or stale data.
  2. Confirm what the $2.5M covers, particularly engineering and compliance.
  3. Restate the market as ~$1B core, with the logic shown.
  4. Check price transfer. UK per-segment pricing may not hold in the US, so the design-partner phase should test willingness to pay.

I'm happy to walk through the numbers before you finalize the deck.

Grades and run details

Decision model 75 · LLM judge 73
Decision model checks
  • failUses the supplied evidence correctly35%
  • passAddresses the actual decision85%
  • passRespects explicit constraints55%
  • passIdentifies material uncertainty100%
  • partialAvoids unsupported claims51%
  • passProduces the required deliverable82%
  • passRebuilds the market size77%
  • passFinds the timeline that sinks it100%
  • passTigers, not paper tigers76%
  • failFinds the need behind the idea35%
Run
Run
#1
API response time
46 s
Submitted
6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

All got wrong 2

Uses the supplied evidence correctlyWrongWrongWrong
Gemini 4 Argon · API

The output invents several facts not in the supplied context, including US sales rep costs, state tax structures, population concentration, and competitor pricing, without labelling them as assumptions.

Haiku 5.5 · API

Claim that US payroll is a crowded market with entrenched incumbents is unsupported by supplied context.

Sonnet 5.5 · API

The output claims US payroll involves 'federal, state and local rules' but the supplied context only mentions state-by-state tax rules, and it presents 'payroll is a sticky, slow-switching category' as fact without support.

Avoids unsupported claimsWrongWrongWrong
Gemini 4 Argon · API

Several claims about US market conditions (sales rep costs, no-tax states, competitor pricing) are presented as established facts without evidence from the supplied context.

Haiku 5.5 · API

States the US payroll market is crowded with entrenched incumbents as a fact without evidence or hedging.

Sonnet 5.5 · API

It presents 'payroll is a sticky, slow-switching category' as an established fact, and 'federal, state and local rules' as a given, neither of which is in the supplied evidence.

All mixed 1

Finds the need behind the ideaMixedMixedMixed
Gemini 4 Argon · API

It identifies the underlying need for US growth, proposes a phased, PLG-led alternative with a smaller budget, and explicitly displaces the immediate sales team and broad targeting.

Haiku 5.5 · API

States the opportunity is real and offers staged validation as an alternative, displacing the full Q1 $2.5M launch.

Sonnet 5.5 · API

It identifies the need for US growth and offers a phased alternative with lower cost and risk, stating what it displaces (the aggressive Q1 launch and $5M target).

All got right 7

Addresses the actual decisionRightRightRight
Gemini 4 Argon · API

The memo commits early to a clear alternative plan (phased launch, pivot ICP) and states what would need to be proven to scale (retention and pricing).

Haiku 5.5 · API

Commits to a staged plan with gates, rejecting Q1 launch, framed for CEO, with conditions for scaling.

Sonnet 5.5 · API

The memo commits to a staged alternative early, frames it for the CEO, and specifies gates that would change the call.

Respects explicit constraintsRightRightRight
Gemini 4 Argon · API

The output is a memo under 1,000 words addressed to the CEO, respecting the requested form and length.

Haiku 5.5 · API

Memo is within 1,000 words and addressed to CEO as requested.

Sonnet 5.5 · API

The output is a memo under 1,000 words addressed to the CEO, respecting the form and length constraint.

Identifies material uncertaintyRightRightRight
Gemini 4 Argon · API

It identifies that US retention and pricing are unproven and makes the scaling decision contingent on validating them with a small initial cohort.

Haiku 5.5 · API

Identifies unknowns (buyer, price, state set, build time) and uses gates to resolve them with measurable triggers.

Sonnet 5.5 · API

It names specific unknowns (budget coverage, ARR discrepancy, US pricing transfer) and says how to resolve them before the board meeting.

Produces the required deliverableRightRightRight
Gemini 4 Argon · API

The memo is complete, in the correct format, and usable by the CEO to present an alternative board proposal.

Haiku 5.5 · API

Memo format, to CEO, actionable, and within the word limit.

Sonnet 5.5 · API

The memo is complete, in the right form for the CEO, within the word limit, and actionable with light edits.

Rebuilds the market sizeRightRightRight
Gemini 4 Argon · API

It correctly uses the US employer counts and UK retention data to argue that only 1.1M businesses are addressable, not 33.2M, effectively rebuilding to a much smaller market.

Haiku 5.5 · API

Uses employer counts and retention data to show relevant market is ~1.1M firms, 1% worth ~$7M ARR, not $50M, with working.

Sonnet 5.5 · API

It rebuilds the market using employer counts and UK retention data, arriving at ~$1B for the 1.1M 10-249 employee businesses, showing the working.

Finds the timeline that sinks itRightRightRight
Gemini 4 Argon · API

The memo explicitly names the 9-month state-tax build and uses it to show that a Q1 launch with payroll is impossible and that the year-one ARR target is undermined.

Haiku 5.5 · API

Names the 9-month build and explains that Q1 launch is infeasible without state coverage.

Sonnet 5.5 · API

It names the 9-month state tax build and explains that a Q1 launch is impossible and the $5M year-one target unreachable.

Tigers, not paper tigersRightRightRight
Gemini 4 Argon · API

It separates the manageable competitor-head-start worry from the truly dangerous risks (unit economics, TAM fallacy, build timeline) and surfaces the UK mix problem with sole traders.

Haiku 5.5 · API

Ranks risks, dismisses competitor raise as less scary, and surfaces TAM churn error as the hidden sink.

Sonnet 5.5 · API

It ranks risks, dismisses the competitor-raise worry as less scary because funding doesn't speed the state build, and surfaces the ARR discrepancy as an unspoken issue.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 75% of 75 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6.1 SolwithAPI97.586.42None
2Haiku 5.5withAPI87.590.92None
3GPT-6 LunawithAPI90.490.92None
4GPT-6 AstrawithChatGPT88.379.42None
5Gemini 4 ArgonwithAPI73.882.52None
6Sonnet 5.5withAPI85.474.82None
7Opus 5.5withClaude85.479.421 capped
8Gemini 3.8 FlashwithAPI87.972.721 capped
9Gemini 3.5 Flash-LitewithGemini69.649.721 capped

What you get for the cost

GPT-6 Luna · API scores within a point of GPT-6.1 Sol · API at about a tenth of the cost.

Task score against relative cost for each setup. The table below has the same figures.4050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6.1 Sol · API: task score 90, 11× the cheapestHaiku 5.5 · API: task score 90, the cheapestGPT-6 Luna · API: task score 89, the cheapestGPT-6 Astra · ChatGPT: task score 82, 60× the cheapest (1 of 2 outputs estimated)Gemini 4 Argon · API: task score 77, 13× the cheapestSonnet 5.5 · API: task score 77, 37× the cheapestOpus 5.5 · Claude: task score 60, 59× the cheapest (1 of 2 outputs estimated), 1 critical failureGemini 3.8 Flash · API: task score 50, 9.5× the cheapest, 1 critical failureGemini 3.5 Flash-Lite · Gemini: task score 45, 1.9× the cheapest (1 of 2 outputs estimated), 1 critical failureGPT-6.1 Sol · APIHaiku 5.5 · APIGPT-6 Luna · APIGPT-6 Astra · ChatGPTGemini 4 Argon · APISonnet 5.5 · APIOpus 5.5 · ClaudeGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)At least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API891.0×
Haiku 5.5 · API901.4×
Gemini 3.5 Flash-Lite · Gemini451.9× (partly estimated)
Gemini 3.8 Flash · API509.5×
GPT-6.1 Sol · API9011×
Gemini 4 Argon · API7713×
Sonnet 5.5 · API7737×
Opus 5.5 · Claude6059× (partly estimated)
GPT-6 Astra · ChatGPT8260× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Pressure-testing a proposal before committing a team to it.

Why it matters

The useful critic finds the one assumption everything rests on. Theatrical negativity is easy to generate and useless in a planning meeting.

What good looks like

  • Identifies the load-bearing assumption
  • Separates the risks that could kill it from the ones that only look scary
  • Uses the supplied evidence, not generic risks
  • Proposes the cheapest way to test the assumption

Deliberately not measured

  • Tone
  • Number of objections raised
Capability tested

Evidence-based critique

The failure we’re looking for

Theatrical negativity without evidence

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

Vanilla prompt (core) · With Roast Me skill · Staff level: a company bet with a long evidence pack