Tasks / Strategy

Challenge an idea

Can the model find the strongest reason an idea may fail, backed by evidence?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 44% were usable with at most a quick edit.

Reliably right

  1. Finds the load-bearing assumption100% pass
    The critique targets the assumption that agencies' growth is limited by a shortage of new leads, using referral, abandoned-outbound, and capacity evidence.
    GPT-6.1 Sol · API · An AI SDR for small agencies
  2. Uses the interviews faithfully100% pass
    All quotes are accurately taken from the transcripts and correctly attributed to A05, A04, and A08.
    GPT-6.1 Sol · API · An AI SDR for small agencies
  3. Finds the timeline that sinks it100% pass
    Cites 9-month build and concludes Q1 launch is unsupported.
    GPT-6 Astra · ChatGPT · Five billion dollars, on paper

Where it slips

  1. Avoids unsupported claims55% pass
    Presents the existence of local taxes and multi-state complexity as fact without support from the context.
    GPT-6 Astra · ChatGPT · Five billion dollars, on paper
  2. Uses the supplied evidence correctly55% pass
    Claims that local taxes and multi-state operations affect eligibility are not supported by the supplied context, which only mentions state-by-state tax rules.
    GPT-6 Astra · ChatGPT · Five billion dollars, on paper
  3. Rebuilds the market size81% pass
    It uses the $150 price for 10+ staff businesses instead of the higher UK revenue per customer for those segments, so the market size is understated and not correctly rebuilt.
    GPT-6 Luna · API · Five billion dollars, on paper

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Finds the need behind the idea

    Does the critique identify the underlying need the idea is meant to serve, and offer at least one other way to meet that need, at a different cost, saying what would have to be dropped for it?

    Passes when States the need behind the proposal, and gives an alternative way to meet it with its cost and what it displaces.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Tigers, not paper tigers · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the VP Product at Halyard. Our CEO is taking the US launch plan below to the board next week and has asked you to challenge it hard first. Write a memo of no more than 1,000 words: what the plan gets wrong, what would sink it, what's less scary than it looks, and what you'd propose instead. The deck excerpt and our data are attached.

What the model was given4 items: about_halyard.md, ceo_board_deck_excerpt.md, uk_customers_by_size.csv, us_small_business_stats.md
about_halyard.md3 lines · Download
# About Halyard in the UK

Payroll and HR for small businesses. 14,800 customers, $3.9M ARR, five years old. UK payroll rules are built in; US payroll would need state-by-state tax rules (the team estimates 9 months of work for the first 10 states).
ceo_board_deck_excerpt.md5 lines · Download
# US launch (CEO's board deck, excerpt)

- Market: 33 million US small businesses x our UK price of £120 ($150) a year = about $5B. "Even 1% is $50M."
- Plan: launch in the US in Q1, $2.5M budget, hire a US sales team of 6, target $5M ARR in year one.
- Why now: a competitor raised in the US; we need to get there first.
uk_customers_by_size.csv4 rows · Download
company_size,uk_customers,share_of_uk_customers_pct,retained_at_12_months_pct,avg_revenue_usd_per_year
Sole trader,6100,41,18,150
2-9 staff,5300,36,33,150
10-49 staff,2900,20,71,620
50-249 staff,480,3,78,2400
us_small_business_stats.md5 lines · Download
# US small businesses (government statistics)

- 33.2 million small businesses in total.
- About 27.1 million have no employees (sole proprietors).
- About 6.1 million have employees; of those, about 1.1 million have 10 to 249 employees.
What a strong answer doesThe answer key the graders mark against

Shows the $5B market is the wrong number. It multiplies every US small business by the UK price, but in the UK only businesses with 10+ staff stay (71% and 78% at 12 months) and pay more ($620 to $2,400 a year); sole traders and 2-9 businesses mostly leave (18% and 33%). Most US small businesses (27.1M of 33.2M) have no employees and don't need payroll at all; the comparable US market is about 1.1M businesses with 10 to 249 staff, worth roughly $0.7B to $1B at UK revenue per customer, not $5B. The tiger: US payroll needs state-by-state tax rules, about 9 months for the first 10 states, so a Q1 launch with payroll isn't possible and $5M ARR in year one would need thousands of 10+ staff customers from a standing start. Dismisses the 'first mover' worry as less scary: a competitor's raise doesn't make a market winner-takes-all. Proposes instead: target 10+ staff businesses, start with HR in the US while payroll is built for a few high-density states, test demand with a small budget and a clear threshold before hiring a sales team, and fix the UK mix towards 10+ staff companies meanwhile.

Critical failures (cap the score at 40)
  • Endorses the $5B market size
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Needs more than a quick edit5 of 10 checks passed

Unsupported claims about acquisition cost and support load would need evidence re-checking, and the missing market-size calculation means the reasoning is incomplete.

Output

MEMORANDUM

TO: CEO FROM: VP Product DATE: October 24, 2023 SUBJECT: Critique and Alternative Proposal for the US Launch Plan

---

You asked me to challenge the US expansion plan hard before you take it to the board next week. I’ve reviewed the deck excerpt against our UK unit economics and US market data.

If we walk into the boardroom with the current plan, a savvy board will pick it apart—and rightly so. Here is my unvarnished assessment of what the plan gets wrong, what will sink us, what is actually noise, and what we should propose instead.

---

1. What the plan gets wrong (The fatal assumptions)

  • The "TAM Fallacy" on Sole Proprietors: The deck multiplies 33 million US small businesses by $150 to claim a $5B market. Government data reveals that 27.1 million of those businesses (82%) have zero employees. In the UK, our data tells a brutal story about sole traders: 41% of our customer base (6,100 accounts) churn within 12 months at an 18% retention rate. They are cheap to acquire, high-churn, high-support-load nightmares who don’t run payroll. Treating the entire 33 million pool as viable is a math error that misleads us on product-market fit.
  • The Compliance Mirage: The deck ignores the engineering reality. US payroll is not a monolith; it is federal, state, and local tax compliance across 50 distinct jurisdictions. Our engineering team estimates 9 months of work just to launch the first 10 states. You cannot launch a payroll product in "the US" in Q1 with a generic software wrapper. Without state tax tables, automated filings, and local garnishments, the product is illegal to use.
  • The Unit Economics Math: At $150/year, acquiring a customer through a dedicated US sales team of 6 with a $2.5M budget is mathematically impossible. A traditional outbound sales team cannot profitably sell a $150 self-serve product. CAC will dwarf LTV on day one.

---

2. What would sink us

  • Regulatory suicide: If we launch prematurely in Q1 to "beat a competitor" without robust state-by-state tax calculation, our first cohort of US customers will face IRS and state tax penalties. In payroll, a software bug doesn't just mean a bad user experience; it means employees don’t get paid or taxes aren't remitted, resulting in immediate churn, devastating brand damage, and potential litigation.
  • Reactive strategy: "A competitor raised in the US; we need to get there first" is a panic motivation, not a product strategy. If we rush a half-baked product into a massive market just to plant a flag, we will burn $2.5M in 12 months, fail to hit the $5M ARR target, and lose credibility with the board.

---

3. What’s less scary than it looks

  • The total market size: We don't need 33 million businesses. Look at our UK data: our sweet spot is companies with 10–49 staff (20% of our base, but 71% retention, generating significantly higher revenue) and 50–249 staff (78% retention, $2,400+ ARR). In the US, there are 1.1 million businesses in the 10–249 employee bracket. That is a massive, highly qualified, high-retention target market. We don't need the sole traders; we should actively avoid them.
  • Geographic concentration: We do not need to launch all 50 states on day one. If we pick 3 to 5 high-density, business-friendly states (e.g., Texas, Florida, Washington—states with no state income tax or straightforward payroll setups), we can drastically reduce engineering scope, bypass the 9-month/10-state bottleneck, and launch a compliant MVP much faster.

---

4. What I propose instead

We should go to the board with a revised, credible strategy that protects our capital and targets the segment we actually know how to monetize.

  1. Shift the ICP (Ideal Customer Profile) away from Sole Traders: Ignore the 27.1 million non-employer businesses. Focus exclusively on US small businesses with 10 to 49 employees. They have the budget, they have payroll complexity, and based on our UK experience, they stick around.
  2. Pivoting the Go-To-Market (GTM): Scrap the plan for a 6-person outbound sales team on a $150 product. At this price point, the US motion must be product-led growth (PLG) with digital acquisition, augmented by a lightweight, inside-sales motion only for accounts scaling toward 50+ employees.
  3. De-risk the Engineering Roadmap: Reallocate the $2.5M budget. Instead of a clumsy nationwide Q1 rush, fund a targeted compliance build for just 3 states to launch a closed beta by Q2. Use the remaining budget for localized digital marketing and product-led onboarding flows.
  4. Refrain the Board Narrative: Do not pitch "capturing 1% of a $5B market." Pitch a disciplined land-grab in a profitable sub-segment: “We are entering the US by targeting the 1.1 million businesses with 10–249 employees, starting with a compliant 3-state footprint, leveraging our proven retention playbook to capture high-value employers at a sustainable CAC.”

If we present this, the board will see operators who understand our unit economics, respect regulatory risk, and have a clear path to sustainable growth. Let’s calibrate the deck before Tuesday.

Claims the judge couldn’t find in the brief

  • They are cheap to acquire, high-churn, high-support-load nightmares

Grades and run details

Decision model 80 · LLM judge 45
Decision model checks
  • partialUses the supplied evidence correctly13%
  • passAddresses the actual decision34%
  • passRespects explicit constraints47%
  • partialIdentifies material uncertainty52%
  • partialAvoids unsupported claims49%
  • passProduces the required deliverable68%
  • passRebuilds the market size57%
  • passFinds the timeline that sinks it100%
  • partialTigers, not paper tigers35%
  • passFinds the need behind the idea26%
Run
Run
#1
Time to output
6 s
Submitted
5 Oct 2026

Usable with a quick edit8 of 10 checks passed

The memo is well-structured and evidence-based; the one unsupported claim about incumbents can be removed with a quick edit without affecting the core reasoning.

Output

MEMO

To: CEO From: VP Product Re: US launch plan: challenge before the board

You asked me to push hard, so I have. I think we should go to the US, but not with this plan. As written, the board will find the holes in it quickly, and the market will find the rest.

---

1. What the plan gets wrong

The market size counts businesses that won't buy payroll. Of the 33.2M US small businesses, 27.1M have no employees. A business with no staff has little use for payroll. Our own UK data points the same way: sole traders are 41% of our customers, but only 18% are still with us after 12 months.

The market that fits our product is the 1.1M US businesses with 10–249 employees. In the UK that segment:

  • makes up 23% of our customers but about 63% of our revenue;
  • keeps 71–78% of customers at 12 months, against 18–33% for smaller firms;
  • pays a blended average of about $873 a year, not $150.

That puts the core US opportunity at about 1.1M × $873 ≈ $1B. The 5M employers with 1–9 staff add up to about $750M more at our current price, but UK retention suggests most of that revenue would leave within a year. The real figure is roughly $1–1.7B, not $5B. That is still huge compared with our $3.9M ARR. We simply don't need the inflated number, and leading with it costs us credibility.

The timeline is impossible. The deck says we launch in Q1. Engineering estimates 9 months to build tax rules for the first 10 states. We cannot sell US payroll before it calculates US payroll taxes. Realistically, the earliest launch is Q4, and only in 10 states.

The $5M year-one target fails on its own arithmetic. It took us five years to reach $3.9M in the UK.

Price assumedCustomers needed for $5MDeals per rep (6 reps)
$150 (deck price)~33,000 (more than double our UK base)~5,500
$873 (mid-market blend)~5,700~950, roughly 4 per working day

At the mid-market price, each rep would need about 950 deals. These are considered purchases, and that volume isn't plausible. It is even less plausible when the product is only live for the last quarter of the year.

The budget is spent in the wrong order. If we hire six salespeople in Q1, we pay them for about nine months while they have nothing to sell. Much of the $2.5M would go on salaries before the product exists.

---

2. What would sink us

  1. Payroll tax errors. Getting someone's pay or taxes wrong is not an ordinary bug. It creates penalties for our customers, refunds for us, and reputational damage in a market where nobody knows us yet. Rushing the tax engine to hit a sales date is the single biggest risk in this plan.
  2. Selling to the wrong customers. If US sales chases volume at $150 among micro-businesses, UK data says we'd lose 67–82% of them within a year. We would be paying to acquire customers who leave, and the board would see revenue growth that isn't real.
  3. Ten states may not cover our best customers. Companies with 10–249 staff are more likely to have employees in several states. If a prospect has people in a state we don't support, we lose the deal. We need data on how many states typical target customers operate in before we finalise the list.
  4. Missing a target we set ourselves. If we commit to $5M and deliver a fraction of it, we lose the board's trust for the next fundraise. That is worse than setting a modest target and beating it.
  5. Neglecting the UK. The UK business funds this expansion. Moving our best people to the US without a plan to protect UK growth puts our cash engine at risk.

---

3. What's less scary than it looks

  • The competitor's raise. The US payroll market is not empty. Well-funded incumbents already exist, so "getting there first" isn't possible for anyone. Another entrant raising money doesn't change our costs, our product gaps or our timing. It is a poor reason to launch before we're ready.
  • The 9-month build. It's real work, but it's bounded. We don't need 50 states on day one. We need the 10 states with the most 10–249-employee businesses. We should also assess licensing a US payroll tax engine instead of building one. That could shorten the timeline and reduce compliance risk, and we should know within a month whether it's viable.
  • The smaller market size. A $1B core market means we need about 0.4% of it to triple the company. That is a stronger story than "1% of $5B," because it rests on segment data and retention we can prove.
  • Our readiness. 71–78% retention in the UK mid-market is real evidence that the product works for these companies. We're not starting from zero. We're moving a proven product into a new market.

---

4. What I'd propose instead

Position: the US launch targets employers with 10–249 staff, priced like our UK mid-market (around $600–2,400 a year, validated with US prospects), not £120.

Phases, with spending released at each milestone:

PhaseTimingWorkSpend
0. DecideMonth 1Build vs. license the tax engine; pick the 10 states by target-business density; interview 30+ US prospects on pricingLow
1. BuildMonths 1–9Tax engine, compliance review, US payroll accuracy testingMost of the early budget goes to engineering and compliance
2. Design partnersMonths 7–1020–30 paying US customers on supported states, with close support1–2 US sales/solutions hires
3. Scale salesMonths 10+Hire toward 6 reps only if design partners hit accuracy and retention barsRest of budget, released by board milestone

Year-one targets, replacing $5M ARR: - payroll runs with no tax-filing errors; - 20–30 design partners, then a few hundred paying customers by year end (roughly $0.2–0.4M ARR at mid-market prices); - at least 70% of US customers staying through early renewal checkpoints, matching the UK mid-market; - evidence on deal size and sales cycle, so a year-two ARR target is grounded in data.

In parallel: keep investing in UK customers with 10–249 staff. They are our highest-value, highest-retention segment, and they fund all of this.

Before the board meeting, two housekeeping items: - Our segment table adds up to about $4.7M in revenue, but we report $3.9M ARR. We should reconcile this, likely churn timing or discounting, before the board asks. - We should remove the "1% of $5B" line entirely.

---

Bottom line: the US is the right ambition, but this plan has the wrong market size, the wrong customer, the wrong timeline and the wrong sequence. If we fix those four things, we go to the board with a plan they can approve and we can deliver.

Claims the judge couldn’t find in the brief

  • Well-funded incumbents already exist.

Grades and run details

Decision model 85 · LLM judge 82
Decision model checks
  • failUses the supplied evidence correctly4%
  • passAddresses the actual decision87%
  • passRespects explicit constraints38%
  • passIdentifies material uncertainty98%
  • partialAvoids unsupported claims64%
  • passProduces the required deliverable87%
  • passRebuilds the market size98%
  • passFinds the timeline that sinks it100%
  • passTigers, not paper tigers82%
  • passFinds the need behind the idea37%
Run
Run
#1
Time to output
46 s
Submitted
5 Oct 2026

Usable with a quick edit8 of 10 checks passed

Minor unsupported claim about crowded market can be edited out; core reasoning and evidence use are solid.

Output

MEMO

To: CEO From: VP Product Re: Challenge to the US launch plan before the board

Bottom line: The market story is wrong, the timeline is not achievable, and the budget can't carry the target. The opportunity is real but much smaller and more specific than $5B. I'd take the board a staged plan with gates, not a Q1 launch.

---

What the plan gets wrong

1. The TAM is built on the wrong customers. 33M × $150 assumes every US small business buys our product at our UK sole-trader price. But 27.1M of those firms have no employees. Payroll is not their problem. Our own data shows sole traders retain at 18% at twelve months and bring in $150 a year. Our customers who retain are the ones with staff.

2. The right pool is about 1.1M firms, and the "1%" math shrinks accordingly. The government figure is 1.1M businesses with 10–249 employees. Our UK revenue is concentrated in the 10–49 and 50–249 segments: about 23% of customers but roughly 60%+ of revenue, with 71–78% retention. Taking a blended ~$620 ARPU, 1% of 1.1M firms is about 11,000 customers and roughly $7M ARR, not $50M. The deck's own logic, applied to the right segment, gives a much smaller prize.

3. The $5M year-one target is not credible. At $620 ARPU, $5M needs about 8,000 paying US customers in twelve months. It took us five years to reach 14,800 UK customers, with a product that is already built. At $150, it needs 33,000, which only works if we sell to the segment that churns most.

4. Q1 is not feasible. The US needs state-by-state payroll tax rules. Our team estimates nine months for the first ten states alone. Q1 launch means either launching with almost no state coverage or launching with a product we can't legally run for most prospects. A firm with employees in an unsupported state cannot onboard at all.

5. "Get there first" is the wrong reason. US payroll is a crowded market with entrenched incumbents, and a competitor's raise means they face the same state-tax build we do. Being early doesn't create a moat in payroll. Being correct on compliance does.

6. The budget doesn't match the plan. Six US quota-carrying reps at a fully loaded cost of roughly $150–180K each is about $1M a year before marketing, tax build, legal, compliance, or US support. That leaves very little of the $2.5M for everything else. (These are my estimates. Please check them against Finance's hiring costs.)

---

What would sink it

  • Compliance liability. Wrong withholding or filing in a state is our error, with penalties landing on our customers. One public failure in an early state would damage referrals across the whole US business.
  • Selling to the wrong segment. If sales pursues sole traders and 2–9 employee firms because the TAM looks big, we'll be acquiring customers at $150 ARPU who churn at 67–82% within a year. CAC payback will never work.
  • Hiring ahead of proof. Six reps with quotas before we know which buyer, which price, and which state set will produce a pipeline we can't convert.
  • Support load. Multi-state questions are hard, and US support can't be staffed from London on a launch timeline.
  • Burn without a gate. If US spend starts before UK growth is secure, a US miss would hit our core business's cash position.

---

What's less scary than it looks

  • The product already works where it matters. Our best UK segments retain at 71–78%. We don't need a new product thesis, just a US version of what works.
  • The tax build is bounded. Nine months for ten states is a real cost, but it's finite. Ten states cover roughly half the US population, so coverage is a sequencing problem. We could also evaluate licensing state tax content from a vendor to cut the timeline.
  • The competitor raise is validation. It shows investors believe in US SMB payroll demand. It also shows we'd be competing in a market where a credible challenger still needs years to build coverage.
  • We don't need a Q1 launch to have a US business. A focused pilot can prove demand within two quarters at a fraction of the cost.

---

What I'd propose instead

I'd ask the board to approve a staged program with explicit gates.

Redefine the market in the deck. The target customer is a US business with 10–249 employees. That's about 1.1M firms and a credible multi-million-dollar opportunity, and we should say so.

Stage 1: Validate (months 0–6, ~$300K). Sign 40–60 design-partner firms with 10–249 employees across two states, one large state and one with simpler rules. Test activation, willingness to pay at $600+ ARPU, and 90-day engagement. Also test a partner channel: accountants and bookkeepers serve our best-fit firms and may be cheaper to reach than direct sales.

Stage 2: Build (months 6–15, ~$800K). Build or license state coverage for the top ten states. Hire one or two sales and customer success people, with the founder leading enterprise-style deals. Gate: pilot cohort retention tracking to 70%+ and CAC payback under 18 months.

Stage 3: Scale (only after gates). Expand sales headcount and marketing against measured unit economics.

Revised target: $1M US ARR by month 24, which is roughly 1,600 customers at $620 ARPU. That is ambitious but defensible, and it's the number the board should hold us to.

The ask: Approve Stages 1 and 2 (about $1.1M) now, with Stage 3 contingent on gate review. Do not approve a $2.5M Q1 launch with a $5M year-one target.

---

I know this is a harder message than the deck. But a board that approves the current plan will be disappointed in twelve months. A board that approves a gated path will get real evidence before committing the big money. I'd rather take the second case to them.

Claims the judge couldn’t find in the brief

  • US payroll is a crowded market with entrenched incumbents.

Grades and run details

Decision model 85 · LLM judge 82
Decision model checks
  • failUses the supplied evidence correctly22%
  • passAddresses the actual decision99%
  • passRespects explicit constraints50%
  • passIdentifies material uncertainty94%
  • partialAvoids unsupported claims66%
  • passProduces the required deliverable90%
  • passRebuilds the market size98%
  • passFinds the timeline that sinks it100%
  • passTigers, not paper tigers57%
  • passFinds the need behind the idea61%
Run
Run
#1
API response time
31 s
Submitted
8 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 3

Identifies material uncertaintyWrongRightRight
Gemini 3.5 Flash-Lite · Gemini

The memo does not name specific unknowns that could change the decision or say how they would be resolved.

Opus 5.5 · Claude

It names unknowns (state coverage needs, build vs. license viability, deal size) and says how to resolve them (prospect interviews, design partner testing, licensing assessment within a month).

Haiku 5.5 · API

Identifies unknowns (buyer, price, state set, build time) and uses gates to resolve them with measurable triggers.

Rebuilds the market sizeMixedRightRight
Gemini 3.5 Flash-Lite · Gemini

It does not rebuild the market to a dollar figure (e.g., under $1B) or show the working; it only states the 1.1M business count without sizing the revenue opportunity.

Opus 5.5 · Claude

It rebuilds the market using employer counts and UK retention data, arriving at about 1.1M businesses and under $1B, showing the working.

Haiku 5.5 · API

Uses employer counts and retention data to show relevant market is ~1.1M firms, 1% worth ~$7M ARR, not $50M, with working.

Finds the need behind the ideaMixedRightRight
Gemini 3.5 Flash-Lite · Gemini

It does not state the underlying need behind the US launch idea (e.g., growth, first-mover advantage) and therefore does not offer an alternative way to meet that need with its cost and trade-offs.

Opus 5.5 · Claude

It states the need (US expansion for growth) and offers an alternative (target 10+ staff, phased build, test demand) with its cost (slower revenue, smaller initial target) and what it displaces (the $5M year-one goal and Q1 launch).

Haiku 5.5 · API

States the opportunity is real and offers staged validation as an alternative, displacing the full Q1 $2.5M launch.

All got wrong 2

Uses the supplied evidence correctlyWrongWrongWrong
Gemini 3.5 Flash-Lite · Gemini

The claim that sole traders are 'cheap to acquire' and 'high-support-load' is not supported by the supplied context.

Opus 5.5 · Claude

The claim that well-funded incumbents already exist is not in the supplied context and is not derivable from it.

Haiku 5.5 · API

Claim that US payroll is a crowded market with entrenched incumbents is unsupported by supplied context.

Avoids unsupported claimsWrongWrongWrong
Gemini 3.5 Flash-Lite · Gemini

It presents 'cheap to acquire' and 'high-support-load' as established facts without labelling them as assumptions or hypotheses.

Opus 5.5 · Claude

The statement 'Well-funded incumbents already exist' is presented as fact without support from the supplied evidence.

Haiku 5.5 · API

States the US payroll market is crowded with entrenched incumbents as a fact without evidence or hedging.

All got right 5

Addresses the actual decisionRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The memo commits to a clear alternative strategy (target 10-49 employee businesses, PLG, 3-state launch) early and unambiguously.

Opus 5.5 · Claude

The memo commits to going to the US with a revised plan, framed for the CEO, and implies conditions (design partner milestones) that would change the call.

Haiku 5.5 · API

Commits to a staged plan with gates, rejecting Q1 launch, framed for CEO, with conditions for scaling.

Respects explicit constraintsRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The output is a memo under 1,000 words, addressed to the CEO, and respects the requested form.

Opus 5.5 · Claude

The output is a memo to the CEO, appears under 1,000 words, and addresses the requested challenge.

Haiku 5.5 · API

Memo is within 1,000 words and addressed to CEO as requested.

Produces the required deliverableRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The memo is complete, in the right form, within length, and usable by the CEO with light edits.

Opus 5.5 · Claude

The memo is complete, in the requested form, for the CEO, and actionable with light edits.

Haiku 5.5 · API

Memo format, to CEO, actionable, and within the word limit.

Finds the timeline that sinks itRightRightRight
Gemini 3.5 Flash-Lite · Gemini

It names the 9-month build and explicitly states that a Q1 payroll launch is impossible.

Opus 5.5 · Claude

It names the 9-month build and concludes a Q1 payroll launch is impossible, with earliest launch Q4.

Haiku 5.5 · API

Names the 9-month build and explains that Q1 launch is infeasible without state coverage.

Tigers, not paper tigersRightRightRight
Gemini 3.5 Flash-Lite · Gemini

It ranks regulatory and reactive strategy as sinkers, dismisses the competitor raise as less scary, and surfaces the need to avoid sole traders.

Opus 5.5 · Claude

It dismisses the competitor's raise as less scary with reasoning, ranks payroll tax errors as the top risk, and surfaces the UK neglect and revenue discrepancy.

Haiku 5.5 · API

Ranks risks, dismisses competitor raise as less scary, and surfaces TAM churn error as the hidden sink.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 76% of 75 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 LunawithAPI100.090.92None
2Haiku 5.5withAPI92.590.92None
3GPT-6.1 SolwithAPI100.086.42None
4GPT-6 AstrawithChatGPT100.079.42None
5Sonnet 5.5withAPI88.374.82None
6Opus 5.5withClaude88.379.421 capped
7Gemini 3.5 Flash-LitewithGemini79.649.721 capped
8Gemini 3.8 FlashwithAPI87.972.721 capped

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.4050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Luna · API: task score 93, the cheapestHaiku 5.5 · API: task score 92, the cheapestGPT-6.1 Sol · API: task score 91, 11× the cheapestGPT-6 Astra · ChatGPT: task score 88, 60× the cheapest (1 of 2 outputs estimated)Sonnet 5.5 · API: task score 78, 37× the cheapestOpus 5.5 · Claude: task score 62, 59× the cheapest (1 of 2 outputs estimated), 1 critical failureGemini 3.5 Flash-Lite · Gemini: task score 50, 1.9× the cheapest (1 of 2 outputs estimated), 1 critical failureGemini 3.8 Flash · API: task score 50, 9.5× the cheapest, 1 critical failureGPT-6 Luna · APIHaiku 5.5 · APIGPT-6.1 Sol · APIGPT-6 Astra · ChatGPTSonnet 5.5 · APIOpus 5.5 · ClaudeGemini 3.5 Flash-Lite · GeminiGemini 3.8 Flash · API
Cost partly estimated (app runs with no usage meter)At least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API931.0×
Haiku 5.5 · API921.4×
Gemini 3.5 Flash-Lite · Gemini501.9× (partly estimated)
Gemini 3.8 Flash · API509.5×
GPT-6.1 Sol · API9111×
Sonnet 5.5 · API7837×
Opus 5.5 · Claude6259× (partly estimated)
GPT-6 Astra · ChatGPT8860× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Pressure-testing a proposal before committing a team to it.

Why it matters

The useful critic finds the one assumption everything rests on. Theatrical negativity is easy to generate and useless in a planning meeting.

What good looks like

  • Identifies the load-bearing assumption
  • Separates the risks that could kill it from the ones that only look scary
  • Uses the supplied evidence, not generic risks
  • Proposes the cheapest way to test the assumption

Deliberately not measured

  • Tone
  • Number of objections raised
Capability tested

Evidence-based critique

The failure we’re looking for

Theatrical negativity without evidence

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

Vanilla prompt (core) · With Roast Me skill · Staff level: a company bet with a long evidence pack