Tasks / Strategy

Challenge an idea

Can the model find the strongest reason an idea may fail, backed by evidence?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 44% were usable with at most a quick edit.

Reliably right

  1. Finds the load-bearing assumption100% pass
    The critique targets the assumption that agencies' growth is limited by a shortage of new leads, using referral, abandoned-outbound, and capacity evidence.
    GPT-6.1 Sol · API · An AI SDR for small agencies
  2. Uses the interviews faithfully100% pass
    All quotes are accurately taken from the transcripts and correctly attributed to A05, A04, and A08.
    GPT-6.1 Sol · API · An AI SDR for small agencies
  3. Finds the timeline that sinks it100% pass
    Cites 9-month build and concludes Q1 launch is unsupported.
    GPT-6 Astra · ChatGPT · Five billion dollars, on paper

Where it slips

  1. Avoids unsupported claims55% pass
    Presents the existence of local taxes and multi-state complexity as fact without support from the context.
    GPT-6 Astra · ChatGPT · Five billion dollars, on paper
  2. Uses the supplied evidence correctly55% pass
    Claims that local taxes and multi-state operations affect eligibility are not supported by the supplied context, which only mentions state-by-state tax rules.
    GPT-6 Astra · ChatGPT · Five billion dollars, on paper
  3. Rebuilds the market size81% pass
    It uses the $150 price for 10+ staff businesses instead of the higher UK revenue per customer for those segments, so the market size is understated and not correctly rebuilt.
    GPT-6 Luna · API · Five billion dollars, on paper

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Finds the need behind the idea

    Does the critique identify the underlying need the idea is meant to serve, and offer at least one other way to meet that need, at a different cost, saying what would have to be dropped for it?

    Passes when States the need behind the proposal, and gives an alternative way to meet it with its cost and what it displaces.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Tigers, not paper tigers · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the VP Product at Halyard. Our CEO is taking the US launch plan below to the board next week and has asked you to challenge it hard first. Write a memo of no more than 1,000 words: what the plan gets wrong, what would sink it, what's less scary than it looks, and what you'd propose instead. The deck excerpt and our data are attached.

What the model was given4 items: about_halyard.md, ceo_board_deck_excerpt.md, uk_customers_by_size.csv, us_small_business_stats.md
about_halyard.md3 lines · Download
# About Halyard in the UK

Payroll and HR for small businesses. 14,800 customers, $3.9M ARR, five years old. UK payroll rules are built in; US payroll would need state-by-state tax rules (the team estimates 9 months of work for the first 10 states).
ceo_board_deck_excerpt.md5 lines · Download
# US launch (CEO's board deck, excerpt)

- Market: 33 million US small businesses x our UK price of £120 ($150) a year = about $5B. "Even 1% is $50M."
- Plan: launch in the US in Q1, $2.5M budget, hire a US sales team of 6, target $5M ARR in year one.
- Why now: a competitor raised in the US; we need to get there first.
uk_customers_by_size.csv4 rows · Download
company_size,uk_customers,share_of_uk_customers_pct,retained_at_12_months_pct,avg_revenue_usd_per_year
Sole trader,6100,41,18,150
2-9 staff,5300,36,33,150
10-49 staff,2900,20,71,620
50-249 staff,480,3,78,2400
us_small_business_stats.md5 lines · Download
# US small businesses (government statistics)

- 33.2 million small businesses in total.
- About 27.1 million have no employees (sole proprietors).
- About 6.1 million have employees; of those, about 1.1 million have 10 to 249 employees.
What a strong answer doesThe answer key the graders mark against

Shows the $5B market is the wrong number. It multiplies every US small business by the UK price, but in the UK only businesses with 10+ staff stay (71% and 78% at 12 months) and pay more ($620 to $2,400 a year); sole traders and 2-9 businesses mostly leave (18% and 33%). Most US small businesses (27.1M of 33.2M) have no employees and don't need payroll at all; the comparable US market is about 1.1M businesses with 10 to 249 staff, worth roughly $0.7B to $1B at UK revenue per customer, not $5B. The tiger: US payroll needs state-by-state tax rules, about 9 months for the first 10 states, so a Q1 launch with payroll isn't possible and $5M ARR in year one would need thousands of 10+ staff customers from a standing start. Dismisses the 'first mover' worry as less scary: a competitor's raise doesn't make a market winner-takes-all. Proposes instead: target 10+ staff businesses, start with HR in the US while payroll is built for a few high-density states, test demand with a small budget and a clear threshold before hiring a sales team, and fix the UK mix towards 10+ staff companies meanwhile.

Critical failures (cap the score at 40)
  • Endorses the $5B market size
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Usable with a quick edit8 of 10 checks passed

Minor unsupported claim about crowded market can be edited out; core reasoning and evidence use are solid.

Output

MEMO

To: CEO From: VP Product Re: Challenge to the US launch plan before the board

Bottom line: The market story is wrong, the timeline is not achievable, and the budget can't carry the target. The opportunity is real but much smaller and more specific than $5B. I'd take the board a staged plan with gates, not a Q1 launch.

---

What the plan gets wrong

1. The TAM is built on the wrong customers. 33M × $150 assumes every US small business buys our product at our UK sole-trader price. But 27.1M of those firms have no employees. Payroll is not their problem. Our own data shows sole traders retain at 18% at twelve months and bring in $150 a year. Our customers who retain are the ones with staff.

2. The right pool is about 1.1M firms, and the "1%" math shrinks accordingly. The government figure is 1.1M businesses with 10–249 employees. Our UK revenue is concentrated in the 10–49 and 50–249 segments: about 23% of customers but roughly 60%+ of revenue, with 71–78% retention. Taking a blended ~$620 ARPU, 1% of 1.1M firms is about 11,000 customers and roughly $7M ARR, not $50M. The deck's own logic, applied to the right segment, gives a much smaller prize.

3. The $5M year-one target is not credible. At $620 ARPU, $5M needs about 8,000 paying US customers in twelve months. It took us five years to reach 14,800 UK customers, with a product that is already built. At $150, it needs 33,000, which only works if we sell to the segment that churns most.

4. Q1 is not feasible. The US needs state-by-state payroll tax rules. Our team estimates nine months for the first ten states alone. Q1 launch means either launching with almost no state coverage or launching with a product we can't legally run for most prospects. A firm with employees in an unsupported state cannot onboard at all.

5. "Get there first" is the wrong reason. US payroll is a crowded market with entrenched incumbents, and a competitor's raise means they face the same state-tax build we do. Being early doesn't create a moat in payroll. Being correct on compliance does.

6. The budget doesn't match the plan. Six US quota-carrying reps at a fully loaded cost of roughly $150–180K each is about $1M a year before marketing, tax build, legal, compliance, or US support. That leaves very little of the $2.5M for everything else. (These are my estimates. Please check them against Finance's hiring costs.)

---

What would sink it

  • Compliance liability. Wrong withholding or filing in a state is our error, with penalties landing on our customers. One public failure in an early state would damage referrals across the whole US business.
  • Selling to the wrong segment. If sales pursues sole traders and 2–9 employee firms because the TAM looks big, we'll be acquiring customers at $150 ARPU who churn at 67–82% within a year. CAC payback will never work.
  • Hiring ahead of proof. Six reps with quotas before we know which buyer, which price, and which state set will produce a pipeline we can't convert.
  • Support load. Multi-state questions are hard, and US support can't be staffed from London on a launch timeline.
  • Burn without a gate. If US spend starts before UK growth is secure, a US miss would hit our core business's cash position.

---

What's less scary than it looks

  • The product already works where it matters. Our best UK segments retain at 71–78%. We don't need a new product thesis, just a US version of what works.
  • The tax build is bounded. Nine months for ten states is a real cost, but it's finite. Ten states cover roughly half the US population, so coverage is a sequencing problem. We could also evaluate licensing state tax content from a vendor to cut the timeline.
  • The competitor raise is validation. It shows investors believe in US SMB payroll demand. It also shows we'd be competing in a market where a credible challenger still needs years to build coverage.
  • We don't need a Q1 launch to have a US business. A focused pilot can prove demand within two quarters at a fraction of the cost.

---

What I'd propose instead

I'd ask the board to approve a staged program with explicit gates.

Redefine the market in the deck. The target customer is a US business with 10–249 employees. That's about 1.1M firms and a credible multi-million-dollar opportunity, and we should say so.

Stage 1: Validate (months 0–6, ~$300K). Sign 40–60 design-partner firms with 10–249 employees across two states, one large state and one with simpler rules. Test activation, willingness to pay at $600+ ARPU, and 90-day engagement. Also test a partner channel: accountants and bookkeepers serve our best-fit firms and may be cheaper to reach than direct sales.

Stage 2: Build (months 6–15, ~$800K). Build or license state coverage for the top ten states. Hire one or two sales and customer success people, with the founder leading enterprise-style deals. Gate: pilot cohort retention tracking to 70%+ and CAC payback under 18 months.

Stage 3: Scale (only after gates). Expand sales headcount and marketing against measured unit economics.

Revised target: $1M US ARR by month 24, which is roughly 1,600 customers at $620 ARPU. That is ambitious but defensible, and it's the number the board should hold us to.

The ask: Approve Stages 1 and 2 (about $1.1M) now, with Stage 3 contingent on gate review. Do not approve a $2.5M Q1 launch with a $5M year-one target.

---

I know this is a harder message than the deck. But a board that approves the current plan will be disappointed in twelve months. A board that approves a gated path will get real evidence before committing the big money. I'd rather take the second case to them.

Claims the judge couldn’t find in the brief

  • US payroll is a crowded market with entrenched incumbents.

Grades and run details

Decision model 85 · LLM judge 82
Decision model checks
  • failUses the supplied evidence correctly22%
  • passAddresses the actual decision99%
  • passRespects explicit constraints50%
  • passIdentifies material uncertainty94%
  • partialAvoids unsupported claims66%
  • passProduces the required deliverable90%
  • passRebuilds the market size98%
  • passFinds the timeline that sinks it100%
  • passTigers, not paper tigers57%
  • passFinds the need behind the idea61%
Run
Run
#1
API response time
31 s
Submitted
8 Oct 2026

Needs more than a quick edit9 of 10 checks passed

The TAM calculation incorrectly applies the $150 price to 10+ staff businesses, so a PM would need to redo the market sizing and adjust the argument before using it.

Output

To: CEO Subject: US launch — don’t take this plan to the board as written

Bottom line: The US could be a real opportunity, but this deck turns a large headline market into an unsupported year-one sales forecast. The biggest risk isn’t arriving late; it’s selling payroll before we can run it reliably in the states we promise to support.

What the plan gets wrong

  • The TAM math is arithmetically fine and commercially misleading. The 33.2 million small-business count includes 27.1 million firms with no employees—about 82% of the total. That is not a credible proxy for payroll buyers. There are 6.1 million employer firms, including 1.1 million with 10–249 employees. Even those are not automatically our customers. Applying our UK price to every US small business assumes identical need, willingness to pay, and product fit.
  • The UK data argues against targeting everyone. Retention is 18% for sole traders and 33% for firms with 2–9 staff, versus 71% for 10–49 and 78% for 50–249. Our best evidence points toward employers with more staff—not a mass-market pitch built on the largest, weakest-fit segment.
  • The price and revenue baseline need reconciling. At $150 per year, $5 million ARR means roughly 33,300 customers. That is more than twice our current UK customer base, which took five years to build. Also, the UK segment counts and stated average revenues imply about $4.66 million in annual revenue, versus the reported $3.9 million ARR. These may use different definitions, but we must explain the $760,000 gap before using UK economics to price or forecast the US.
  • “A competitor raised” is not a launch strategy. It may validate investor interest, but it tells us nothing about buyer demand for Halyard, our differentiation, acquisition cost, or which states and customer segments we can serve profitably.

What would sink it

The timeline is incompatible with the product plan. We estimate nine months to build payroll tax rules for the first 10 states; the deck promises a Q1 US launch. A launch without dependable tax calculations, filings, registrations, and support creates compliance exposure and destroys trust. “Ten states” is not the same as nationwide coverage, and the deck does not say what launch actually means.

The target compounds the problem. At the proposed price, we need 33,300 customers in year one—without a validated US product, channel, conversion rate, or sales cycle. Six sales hires do not make that forecast credible. The $2.5 million budget also needs to cover product, tax expertise, support, and go-to-market; the deck gives no cost or hiring ramp to show it can.

What’s less scary than it looks

The market does not need to be 33 million businesses to matter. The 6.1 million employer firms are a meaningful starting universe; at $150 per year, that price would imply a crude ceiling of about $915 million, not a forecast. The 1.1 million firms with 10–249 employees alone imply about $165 million at that same price. Those figures are still not a serviceable-market estimate, but they show we can pursue a focused wedge without pretending every sole proprietor is a prospect.

We also have useful evidence: five years of operating experience, 14,800 UK customers, and better retention among larger employers. That is a foundation for testing the US—not proof that the UK playbook transfers unchanged.

What I’d propose instead

Approve a staged US entry, not a Q1 national launch or a $5 million year-one commitment.

  1. Choose a narrow first customer and geography. Start with employer businesses, prioritizing 10–49 staff; select states using verified prospect demand and implementation feasibility. Defer sole traders and the weakest-retaining segment until we have evidence they can work.
  2. Build and validate before scaling sales. Recruit US design partners, test willingness to pay and workflows, and ship only when payroll accuracy, filing operations, and support are proven in the launch states. Treat the nine-month estimate as a planning constraint, not a promise to cover 10 states by Q1.
  3. Stage the investment. Fund discovery, state selection, and the first production-ready footprint now. Hold most sales hiring until pilots demonstrate successful payroll cycles, repeatable acquisition, and acceptable retention and unit economics. Consider accountant or payroll-partner channels before assuming six direct sellers are the answer.
  4. Return to the board with evidence. Bring a state-by-state scope, reconciled UK revenue data, tested US pricing, customer pipeline, launch-readiness criteria, and a bottom-up forecast tied to conversion and sales capacity.

We should move with urgency—but urgency should accelerate learning, not turn an unvalidated forecast into a launch promise.

Grades and run details

Decision model 100 · LLM judge 82
Decision model checks
  • passUses the supplied evidence correctly60%
  • passAddresses the actual decision90%
  • passRespects explicit constraints67%
  • passIdentifies material uncertainty89%
  • passAvoids unsupported claims37%
  • passProduces the required deliverable83%
  • passRebuilds the market size60%
  • passFinds the timeline that sinks it100%
  • passTigers, not paper tigers71%
  • passFinds the need behind the idea24%
Run
Run
#1
API response time
29 s
Submitted
6 Oct 2026

Needs more than a quick edit5 of 10 checks passed

The output contains unsupported claims and an incorrect TAM calculation that would require re-checking evidence and redoing the market sizing.

Output

MEMO

TO: CEO FROM: VP Product DATE: October 24, 2023 SUBJECT: Critical Assessment of Proposed US Expansion Plan

*

You asked me to challenge our US launch plan before next week’s board meeting. While expanding to the US is the right ambition eventually, taking this deck to the board in its current form exposes us to serious credibility and execution risks.

Below is an unvarnished breakdown of where the plan breaks down, the existential risks, what shouldn’t scare us, and an alternative path to present to the board.

---

1. What the Plan Gets Wrong

#### The TAM Calculation is a Fiction The deck calculates our TAM as 33M businesses $\times$ $150 = $5B. * 27.1M (82%) of those businesses are non-employers (sole proprietors). They do not run payroll. Selling payroll to someone with no employees is a non-starter. * Our true addressable market of employer SMBs is 6.1M businesses, not 33M. * The “even 1% is $50M” trope will instantly damage our credibility with board members who understand US market demographics.

#### The Sales Math is Mathematically Impossible Targeting $5M ARR in Year 1 at $150/year requires acquiring 33,333 net new customers. * With a team of 6 sales reps, that equates to 5,555 closed deals per rep per year, or ~23 closed deals per rep every single working day. * Even if we targeted our UK mid-tier ($620/year), each rep would need to close 5.5 deals per business day in a brand-new market with zero brand equity. Inside sales reps for SMB SaaS typically close 20–40 deals per month.

#### It Replicates Our Worst UK Cohort In the UK, micro-businesses ($\le$9 staff) represent 77% of our customers, but they suffer from catastrophic churn: * Sole traders: 18% 12-month retention (82% annual churn). * 2–9 staff: 33% 12-month retention (67% annual churn). Building a US go-to-market strategy around high-churn micro-businesses at $150/year guarantees we burn cash filling a leaky bucket.

---

2. What Would Sink Us

#### The 9-Month Compliance Wall vs. Q1 Launch US payroll is not a single product; it is a federation of 50 distinct state tax codes, local jurisdictions, reciprocity rules, and quarterly filing obligations. * Our engineering estimate is 9 months just to support the first 10 states. * Announcing a Q1 launch leaves product and engineering two options: launch a crippled product covering 2–3 states, or miss the launch date entirely. Selling payroll that cannot support a customer hiring an out-of-state remote worker leads immediately to compliance liability, tax penalties, and reputational ruin.

#### Dilution of UK Momentum & Budget Burn We have $3.9M ARR in the UK. A $2.5M capital outlay represents over 60% of our annual revenue. * Splitting engineering to build state tax engines will freeze UK product development for nearly a year. * $2.5M will vanish rapidly once factoring in 6 US base salaries, sales leadership, US legal/tax compliance, localized infrastructure, and paid customer acquisition. We risk running out of cash with a half-built US engine and a stalled UK business.

---

3. What’s Less Scary Than It Looks

#### The Competitor’s US Funding Round A competitor raising capital in the US does not mean they have won. * The US market is dominated by entrenched incumbents (Gusto, Rippling, ADP, Paychex). A UK competitor raising capital will face the exact same compliance, distribution, and CAC headwinds we would. * Raising money to chase micro-SMBs at low price points usually accelerates a competitor's burn, not their moat. We should let them spend heavily to educate the market while we execute deliberately.

#### Winning Larger SMBs (10–249 Staff) Our UK data shows that moving upmarket works: * 10–49 staff: 71% retention, $620 ARR (4.1x ARPU). * 50–249 staff: 78% retention, $2,400 ARR (16x ARPU). * Combined, these two segments represent only 23% of our UK customer count, but they generate over 60% of our revenue and virtually all of our net lifetime value. * In the US, there are 1.1 million businesses with 10–249 employees. We don’t need 33,000 customers; we need a few thousand high-retention accounts at a realistic US market price ($40–$100/month base + $6–$12/employee/month).

---

4. What We Should Propose Instead

Tell the board we are entering the US with a disciplined, phased strategy focused on high-retention economics rather than low-yield volume.

``` PHASE 1 (Months 1–6): UK Consolidation & US Technical Build ├── Target: Accelerate UK upmarket shift (10-249 staff) to self-fund expansion. ├── Product: Build US tax engines for Top 5 employer states (CA, TX, NY, FL, IL). └── GTM: Do NOT hire US reps yet. Evaluate payroll API partners (e.g., Check, Salsa) to eliminate the 9-month internal build entirely.

PHASE 2 (Months 7–12): Controlled Pilot (The 10–99 Staff Segment) ├── Focus: 10–99 employee segment; price at US parity ($1,500–$3,000/year). ├── Team: 2 senior US AE/consultative sellers + 1 US product compliance lead. └── Target: 250 customers, $500k ARR, >80% 12-month retention intent. ```

#### The Revised Board Pitch 1. Realistic TAM: Present the addressable market as 1.1M US businesses with 10–249 employees, representing a $2.5B–$4B TAM at standard US SaaS pricing ($2,000–$3,500 ARR per account). 2. Rationalized Y1 Goal: Target $500k–$750k ARR in Year 1 from a focused pilot in 3–5 key states, proving product-market fit and retention before scaling headcount. 3. Alternative Delivery Mechanism: Propose exploring white-label/embedded payroll infrastructure partners (e.g., Check Technologies) to compress our 9-month build into 60 days, offloading tax filing liabilities. 4. Capital Efficiency: Cap initial US allocation at $1.0M–$1.2M, tying the remaining budget to hitting retention milestones in the pilot cohort.

Let’s position Halyard to the board as a disciplined leadership team that understands unit economics and enterprise value, not one panicking over a competitor's press release.

Claims the judge couldn’t find in the brief

  • Inside sales reps for SMB SaaS typically close 20–40 deals per month.
  • US payroll is not a single product; it is a federation of 50 distinct state tax codes, local jurisdictions, reciprocity rules, and quarterly filing obligations.

Grades and run details

Decision model 80 · LLM judge 45
Decision model checks
  • failUses the supplied evidence correctly56%
  • passAddresses the actual decision56%
  • passRespects explicit constraints24%
  • partialIdentifies material uncertainty29%
  • partialAvoids unsupported claims59%
  • passProduces the required deliverable84%
  • passRebuilds the market size40%
  • passFinds the timeline that sinks it100%
  • passTigers, not paper tigers78%
  • passFinds the need behind the idea57%
Run
Run
#1
API response time
27 s
Submitted
6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 5

Uses the supplied evidence correctlyWrongRightWrong
Haiku 5.5 · API

Claim that US payroll is a crowded market with entrenched incumbents is unsupported by supplied context.

GPT-6 Luna · API

All claims about the current situation are directly supported by the supplied context or arithmetic from it.

Gemini 3.8 Flash · API

The output invents details about US payroll (local jurisdictions, reciprocity rules, quarterly filing obligations) and claims about incumbents and typical sales rep performance not in the supplied context.

Addresses the actual decisionRightRightMixed
Haiku 5.5 · API

Commits to a staged plan with gates, rejecting Q1 launch, framed for CEO, with conditions for scaling.

GPT-6 Luna · API

The memo commits to a clear call (don't take the plan as written, do a staged entry) early, framed for the CEO, and implies the decision would change based on pilot evidence.

Gemini 3.8 Flash · API

The output commits to a phased strategy but does not state what result or condition would change that call.

Identifies material uncertaintyRightRightWrong
Haiku 5.5 · API

Identifies unknowns (buyer, price, state set, build time) and uses gates to resolve them with measurable triggers.

GPT-6 Luna · API

It names unknowns (US demand, conversion, unit economics, revenue data gap) and proposes pilots and evidence thresholds to resolve them.

Gemini 3.8 Flash · API

The output does not explicitly name specific unknowns that could change the decision or how they would be resolved, beyond implying a pilot will test product-market fit.

Avoids unsupported claimsWrongRightWrong
Haiku 5.5 · API

States the US payroll market is crowded with entrenched incumbents as a fact without evidence or hedging.

GPT-6 Luna · API

Interpretations and forecasts are presented as such, not as established facts; the revenue gap is arithmetic, not speculation.

Gemini 3.8 Flash · API

The output presents forecasts (e.g., 'guarantees we burn cash', 'will freeze UK product development') and interpretations as established fact without labelling them as hypotheses.

Rebuilds the market sizeRightMixedMixed
Haiku 5.5 · API

Uses employer counts and retention data to show relevant market is ~1.1M firms, 1% worth ~$7M ARR, not $50M, with working.

GPT-6 Luna · API

It uses the $150 price for 10+ staff businesses instead of the higher UK revenue per customer for those segments, so the market size is understated and not correctly rebuilt.

Gemini 3.8 Flash · API

The output arrives at a $2.5B–$4B TAM using invented US pricing, not the under $1B market size derived from UK revenue per customer as required.

All got right 5

Respects explicit constraintsRightRightRight
Haiku 5.5 · API

Memo is within 1,000 words and addressed to CEO as requested.

GPT-6 Luna · API

The output is a memo to the CEO, under 1,000 words, and addresses all requested elements.

Gemini 3.8 Flash · API

The memo is under 1,000 words, addressed to the CEO, and in memo form.

Produces the required deliverableRightRightRight
Haiku 5.5 · API

Memo format, to CEO, actionable, and within the word limit.

GPT-6 Luna · API

The memo is complete, in the right form, within length, and a CEO could act on it with light edits.

Gemini 3.8 Flash · API

The memo is complete, in the requested form, and usable by the CEO.

Finds the timeline that sinks itRightRightRight
Haiku 5.5 · API

Names the 9-month build and explains that Q1 launch is infeasible without state coverage.

GPT-6 Luna · API

It explicitly names the 9-month build and explains why it makes a Q1 payroll launch impossible and the year-one target unachievable.

Gemini 3.8 Flash · API

The output names the 9-month build and explains it makes a Q1 payroll launch impossible.

Tigers, not paper tigersRightRightRight
Haiku 5.5 · API

Ranks risks, dismisses competitor raise as less scary, and surfaces TAM churn error as the hidden sink.

GPT-6 Luna · API

It ranks the timeline/product risk as the sinker, dismisses the first-mover worry as less scary, and surfaces the UK revenue data discrepancy.

Gemini 3.8 Flash · API

The output triages risks, dismisses the competitor funding worry with reasoning, and surfaces the issue of replicating the worst UK cohort.

Finds the need behind the ideaRightRightRight
Haiku 5.5 · API

States the opportunity is real and offers staged validation as an alternative, displacing the full Q1 $2.5M launch.

GPT-6 Luna · API

It identifies the need to capture the US opportunity and proposes a staged entry as an alternative, with lower upfront cost and what it displaces (national launch, $5M target).

Gemini 3.8 Flash · API

The output states the need to expand to the US and offers a phased alternative with cost and what it displaces.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 76% of 75 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 LunawithAPI100.090.92None
2Haiku 5.5withAPI92.590.92None
3GPT-6.1 SolwithAPI100.086.42None
4GPT-6 AstrawithChatGPT100.079.42None
5Sonnet 5.5withAPI88.374.82None
6Opus 5.5withClaude88.379.421 capped
7Gemini 3.5 Flash-LitewithGemini79.649.721 capped
8Gemini 3.8 FlashwithAPI87.972.721 capped

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.4050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Luna · API: task score 93, the cheapestHaiku 5.5 · API: task score 92, the cheapestGPT-6.1 Sol · API: task score 91, 11× the cheapestGPT-6 Astra · ChatGPT: task score 88, 60× the cheapest (1 of 2 outputs estimated)Sonnet 5.5 · API: task score 78, 37× the cheapestOpus 5.5 · Claude: task score 62, 59× the cheapest (1 of 2 outputs estimated), 1 critical failureGemini 3.5 Flash-Lite · Gemini: task score 50, 1.9× the cheapest (1 of 2 outputs estimated), 1 critical failureGemini 3.8 Flash · API: task score 50, 9.5× the cheapest, 1 critical failureGPT-6 Luna · APIHaiku 5.5 · APIGPT-6.1 Sol · APIGPT-6 Astra · ChatGPTSonnet 5.5 · APIOpus 5.5 · ClaudeGemini 3.5 Flash-Lite · GeminiGemini 3.8 Flash · API
Cost partly estimated (app runs with no usage meter)At least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API931.0×
Haiku 5.5 · API921.4×
Gemini 3.5 Flash-Lite · Gemini501.9× (partly estimated)
Gemini 3.8 Flash · API509.5×
GPT-6.1 Sol · API9111×
Sonnet 5.5 · API7837×
Opus 5.5 · Claude6259× (partly estimated)
GPT-6 Astra · ChatGPT8860× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Pressure-testing a proposal before committing a team to it.

Why it matters

The useful critic finds the one assumption everything rests on. Theatrical negativity is easy to generate and useless in a planning meeting.

What good looks like

  • Identifies the load-bearing assumption
  • Separates the risks that could kill it from the ones that only look scary
  • Uses the supplied evidence, not generic risks
  • Proposes the cheapest way to test the assumption

Deliberately not measured

  • Tone
  • Number of objections raised
Capability tested

Evidence-based critique

The failure we’re looking for

Theatrical negativity without evidence

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

Vanilla prompt (core) · With Roast Me skill · Staff level: a company bet with a long evidence pack