Tasks / Strategy

Challenge an idea

Can the model find the strongest reason an idea may fail, backed by evidence?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 44% were usable with at most a quick edit.

Reliably right

  1. Finds the load-bearing assumption100% pass
    The critique targets the assumption that agencies' growth is limited by a shortage of new leads, using referral, abandoned-outbound, and capacity evidence.
    GPT-6.1 Sol · API · An AI SDR for small agencies
  2. Uses the interviews faithfully100% pass
    All quotes are accurately taken from the transcripts and correctly attributed to A05, A04, and A08.
    GPT-6.1 Sol · API · An AI SDR for small agencies
  3. Finds the timeline that sinks it100% pass
    Cites 9-month build and concludes Q1 launch is unsupported.
    GPT-6 Astra · ChatGPT · Five billion dollars, on paper

Where it slips

  1. Avoids unsupported claims55% pass
    Presents the existence of local taxes and multi-state complexity as fact without support from the context.
    GPT-6 Astra · ChatGPT · Five billion dollars, on paper
  2. Uses the supplied evidence correctly55% pass
    Claims that local taxes and multi-state operations affect eligibility are not supported by the supplied context, which only mentions state-by-state tax rules.
    GPT-6 Astra · ChatGPT · Five billion dollars, on paper
  3. Rebuilds the market size81% pass
    It uses the $150 price for 10+ staff businesses instead of the higher UK revenue per customer for those segments, so the market size is understated and not correctly rebuilt.
    GPT-6 Luna · API · Five billion dollars, on paper

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Finds the need behind the idea

    Does the critique identify the underlying need the idea is meant to serve, and offer at least one other way to meet that need, at a different cost, saying what would have to be dropped for it?

    Passes when States the need behind the proposal, and gives an alternative way to meet it with its cost and what it displaces.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Tigers, not paper tigers · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the VP Product at Halyard. Our CEO is taking the US launch plan below to the board next week and has asked you to challenge it hard first. Write a memo of no more than 1,000 words: what the plan gets wrong, what would sink it, what's less scary than it looks, and what you'd propose instead. The deck excerpt and our data are attached.

What the model was given4 items: about_halyard.md, ceo_board_deck_excerpt.md, uk_customers_by_size.csv, us_small_business_stats.md
about_halyard.md3 lines · Download
# About Halyard in the UK

Payroll and HR for small businesses. 14,800 customers, $3.9M ARR, five years old. UK payroll rules are built in; US payroll would need state-by-state tax rules (the team estimates 9 months of work for the first 10 states).
ceo_board_deck_excerpt.md5 lines · Download
# US launch (CEO's board deck, excerpt)

- Market: 33 million US small businesses x our UK price of £120 ($150) a year = about $5B. "Even 1% is $50M."
- Plan: launch in the US in Q1, $2.5M budget, hire a US sales team of 6, target $5M ARR in year one.
- Why now: a competitor raised in the US; we need to get there first.
uk_customers_by_size.csv4 rows · Download
company_size,uk_customers,share_of_uk_customers_pct,retained_at_12_months_pct,avg_revenue_usd_per_year
Sole trader,6100,41,18,150
2-9 staff,5300,36,33,150
10-49 staff,2900,20,71,620
50-249 staff,480,3,78,2400
us_small_business_stats.md5 lines · Download
# US small businesses (government statistics)

- 33.2 million small businesses in total.
- About 27.1 million have no employees (sole proprietors).
- About 6.1 million have employees; of those, about 1.1 million have 10 to 249 employees.
What a strong answer doesThe answer key the graders mark against

Shows the $5B market is the wrong number. It multiplies every US small business by the UK price, but in the UK only businesses with 10+ staff stay (71% and 78% at 12 months) and pay more ($620 to $2,400 a year); sole traders and 2-9 businesses mostly leave (18% and 33%). Most US small businesses (27.1M of 33.2M) have no employees and don't need payroll at all; the comparable US market is about 1.1M businesses with 10 to 249 staff, worth roughly $0.7B to $1B at UK revenue per customer, not $5B. The tiger: US payroll needs state-by-state tax rules, about 9 months for the first 10 states, so a Q1 launch with payroll isn't possible and $5M ARR in year one would need thousands of 10+ staff customers from a standing start. Dismisses the 'first mover' worry as less scary: a competitor's raise doesn't make a market winner-takes-all. Proposes instead: target 10+ staff businesses, start with HR in the US while payroll is built for a few high-density states, test demand with a small budget and a clear threshold before hiring a sales team, and fix the UK mix towards 10+ staff companies meanwhile.

Critical failures (cap the score at 40)
  • Endorses the $5B market size
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Needs more than a quick edit8 of 10 checks passed

The output includes unsupported claims about US payroll rules (federal/local) and market stickiness that a PM would need to verify before using, so it cannot be used without re-checking evidence.

Output

MEMO: US launch plan, challenge review To: CEO | From: VP Product

Bottom line: The direction is defensible, but the plan isn't ready for the board. The market is about a fifth the size the deck claims, the Q1 launch date conflicts with our own engineering estimate, and the $5M year-one target is unreachable. I'd take a staged version instead.

---

What the plan gets wrong

1. The $5B market is wrong by roughly 5x. The deck multiplies all 33.2M US small businesses by $150. But 27.1M of them have no employees, so they have no payroll to run. Our own UK data shows how little this segment is worth to us:

  • Sole traders are 41% of UK customers but only about 20% of revenue, and just 18% are still with us at 12 months.
  • Firms with 10+ staff are 23% of customers but about 63% of revenue, with 71–78% retention.

Our proven segment in the US is the 1.1M firms with 10–249 employees. At our UK blended revenue for that group (about $870 per customer), that's roughly $1B. "Even 1%" is about $10M, not $50M. I'd exclude the ~5M micro-employers. In the UK, 2–9 staff firms pay $150 and only 33% stay a year.

2. The timeline contradicts itself. The team estimates 9 months of work for the first 10 states. A Q1 launch means selling a product that doesn't exist, or one that only works in a few states. Ten states is also only about half the US population, by my rough estimate, so a national sales team would be pitching to prospects we can't serve.

3. $5M ARR in year one is out of reach. It exceeds the ARR we built in five years in the UK. At ~$870 per customer it needs about 5,700 customers, roughly 950 per rep from a standing start. With a 9-month build, year-one revenue would be close to zero.

4. "A competitor raised" isn't a reason to launch now. We can't win a spending race with $2.5M against a funded rival. Payroll is also a sticky, slow-switching category, so launching early earns us little. Launching with wrong tax calculations would cost us far more.

What would sink it

  • Compliance failures. A wrong withholding or a missed filing means customers pay penalties and blame us. In the UK, rules are built in and we know them. In the US we'd face federal, state and local rules, each with its own filings. One bad quarter in the first cohort could end our reputation with the segment we need.
  • Targeting the wrong customer. If the sales team chases volume, it will sign small firms that churn at UK micro-segment rates. Acquisition cost would never pay back.
  • Draining the UK business. Nine months of engineering on tax rules comes out of the roadmap for the business that funds everything. UK retention in the smallest segments is already weak, and we'd be neglecting it.
  • Burning the budget before learning anything. Six hires plus a 9-month build could consume most of $2.5M, about 64% of current ARR, before we know whether US customers retain like UK ones. I need to confirm whether the $2.5M includes the engineering cost. If it doesn't, the real ask is larger.
  • Missing the target. A $5M goal that slips will damage board confidence in everything else we present.

What's less scary than it looks

  • The smaller market is still big. About $1B is over 200x our current ARR. We don't need $5B to justify the US.
  • The competitor raise. It validates demand, but funding doesn't fix the state-by-state build, which slows them as well. Nothing here is lost by launching in 12–18 months instead of 3.
  • The 9-month estimate. It's a well-scoped build with a known end date, and a tax-calculation partner or API might shorten it. I'd evaluate build vs. buy before committing.
  • UK retention in micro-segments. It looks alarming, but it's a UK issue we'd avoid importing by targeting 10–249 staff firms, where our retention is strongest.

What I'd propose instead

Phase 0 (now to ~Q3, ~$0.5–0.7M): - Decide build vs. buy for the tax engine. - Pick 3–4 launch states by density of 10–249 staff firms. - Recruit 15–20 design-partner customers. - Hire a US head of sales/GTM and one compliance specialist, not six reps.

Phase 1 (launch in 3–5 states once compliance is proven, ~$1M): - Sell only to 10–249 staff firms with 2 reps and dedicated onboarding. - Set the target at about 300 customers and ~$0.25–0.3M ARR twelve months after launch. That's my estimate, based on UK segment pricing.

Phase 2 (release the remaining ~$1M, expand states and team) only if we hit gates: - 12-month retention of at least 70% in the first cohort. - Zero material filing errors. - Acquisition cost paying back within an agreed window.

This puts about $0.5–0.7M at risk before we learn anything, rather than $2.5M, and it gives the board a credible plan with checkpoints.

Before the board meeting

  1. Reconcile ARR. Our segment table implies about $4.66M (customers × average revenue), but the deck and company profile say $3.9M. A board member will do this multiplication. We need to know whether the gap is discounting, churn timing, or stale data.
  2. Confirm what the $2.5M covers, particularly engineering and compliance.
  3. Restate the market as ~$1B core, with the logic shown.
  4. Check price transfer. UK per-segment pricing may not hold in the US, so the design-partner phase should test willingness to pay.

I'm happy to walk through the numbers before you finalize the deck.

Grades and run details

Decision model 85 · LLM judge 73
Decision model checks
  • failUses the supplied evidence correctly23%
  • passAddresses the actual decision81%
  • passRespects explicit constraints59%
  • passIdentifies material uncertainty99%
  • partialAvoids unsupported claims68%
  • passProduces the required deliverable82%
  • passRebuilds the market size99%
  • passFinds the timeline that sinks it100%
  • passTigers, not paper tigers78%
  • passFinds the need behind the idea30%
Run
Run
#1
API response time
46 s
Submitted
6 Oct 2026

Needs more than a quick edit5 of 10 checks passed

The output contains unsupported claims and an incorrect TAM calculation that would require re-checking evidence and redoing the market sizing.

Output

MEMO

TO: CEO FROM: VP Product DATE: October 24, 2023 SUBJECT: Critical Assessment of Proposed US Expansion Plan

*

You asked me to challenge our US launch plan before next week’s board meeting. While expanding to the US is the right ambition eventually, taking this deck to the board in its current form exposes us to serious credibility and execution risks.

Below is an unvarnished breakdown of where the plan breaks down, the existential risks, what shouldn’t scare us, and an alternative path to present to the board.

---

1. What the Plan Gets Wrong

#### The TAM Calculation is a Fiction The deck calculates our TAM as 33M businesses $\times$ $150 = $5B. * 27.1M (82%) of those businesses are non-employers (sole proprietors). They do not run payroll. Selling payroll to someone with no employees is a non-starter. * Our true addressable market of employer SMBs is 6.1M businesses, not 33M. * The “even 1% is $50M” trope will instantly damage our credibility with board members who understand US market demographics.

#### The Sales Math is Mathematically Impossible Targeting $5M ARR in Year 1 at $150/year requires acquiring 33,333 net new customers. * With a team of 6 sales reps, that equates to 5,555 closed deals per rep per year, or ~23 closed deals per rep every single working day. * Even if we targeted our UK mid-tier ($620/year), each rep would need to close 5.5 deals per business day in a brand-new market with zero brand equity. Inside sales reps for SMB SaaS typically close 20–40 deals per month.

#### It Replicates Our Worst UK Cohort In the UK, micro-businesses ($\le$9 staff) represent 77% of our customers, but they suffer from catastrophic churn: * Sole traders: 18% 12-month retention (82% annual churn). * 2–9 staff: 33% 12-month retention (67% annual churn). Building a US go-to-market strategy around high-churn micro-businesses at $150/year guarantees we burn cash filling a leaky bucket.

---

2. What Would Sink Us

#### The 9-Month Compliance Wall vs. Q1 Launch US payroll is not a single product; it is a federation of 50 distinct state tax codes, local jurisdictions, reciprocity rules, and quarterly filing obligations. * Our engineering estimate is 9 months just to support the first 10 states. * Announcing a Q1 launch leaves product and engineering two options: launch a crippled product covering 2–3 states, or miss the launch date entirely. Selling payroll that cannot support a customer hiring an out-of-state remote worker leads immediately to compliance liability, tax penalties, and reputational ruin.

#### Dilution of UK Momentum & Budget Burn We have $3.9M ARR in the UK. A $2.5M capital outlay represents over 60% of our annual revenue. * Splitting engineering to build state tax engines will freeze UK product development for nearly a year. * $2.5M will vanish rapidly once factoring in 6 US base salaries, sales leadership, US legal/tax compliance, localized infrastructure, and paid customer acquisition. We risk running out of cash with a half-built US engine and a stalled UK business.

---

3. What’s Less Scary Than It Looks

#### The Competitor’s US Funding Round A competitor raising capital in the US does not mean they have won. * The US market is dominated by entrenched incumbents (Gusto, Rippling, ADP, Paychex). A UK competitor raising capital will face the exact same compliance, distribution, and CAC headwinds we would. * Raising money to chase micro-SMBs at low price points usually accelerates a competitor's burn, not their moat. We should let them spend heavily to educate the market while we execute deliberately.

#### Winning Larger SMBs (10–249 Staff) Our UK data shows that moving upmarket works: * 10–49 staff: 71% retention, $620 ARR (4.1x ARPU). * 50–249 staff: 78% retention, $2,400 ARR (16x ARPU). * Combined, these two segments represent only 23% of our UK customer count, but they generate over 60% of our revenue and virtually all of our net lifetime value. * In the US, there are 1.1 million businesses with 10–249 employees. We don’t need 33,000 customers; we need a few thousand high-retention accounts at a realistic US market price ($40–$100/month base + $6–$12/employee/month).

---

4. What We Should Propose Instead

Tell the board we are entering the US with a disciplined, phased strategy focused on high-retention economics rather than low-yield volume.

``` PHASE 1 (Months 1–6): UK Consolidation & US Technical Build ├── Target: Accelerate UK upmarket shift (10-249 staff) to self-fund expansion. ├── Product: Build US tax engines for Top 5 employer states (CA, TX, NY, FL, IL). └── GTM: Do NOT hire US reps yet. Evaluate payroll API partners (e.g., Check, Salsa) to eliminate the 9-month internal build entirely.

PHASE 2 (Months 7–12): Controlled Pilot (The 10–99 Staff Segment) ├── Focus: 10–99 employee segment; price at US parity ($1,500–$3,000/year). ├── Team: 2 senior US AE/consultative sellers + 1 US product compliance lead. └── Target: 250 customers, $500k ARR, >80% 12-month retention intent. ```

#### The Revised Board Pitch 1. Realistic TAM: Present the addressable market as 1.1M US businesses with 10–249 employees, representing a $2.5B–$4B TAM at standard US SaaS pricing ($2,000–$3,500 ARR per account). 2. Rationalized Y1 Goal: Target $500k–$750k ARR in Year 1 from a focused pilot in 3–5 key states, proving product-market fit and retention before scaling headcount. 3. Alternative Delivery Mechanism: Propose exploring white-label/embedded payroll infrastructure partners (e.g., Check Technologies) to compress our 9-month build into 60 days, offloading tax filing liabilities. 4. Capital Efficiency: Cap initial US allocation at $1.0M–$1.2M, tying the remaining budget to hitting retention milestones in the pilot cohort.

Let’s position Halyard to the board as a disciplined leadership team that understands unit economics and enterprise value, not one panicking over a competitor's press release.

Claims the judge couldn’t find in the brief

  • Inside sales reps for SMB SaaS typically close 20–40 deals per month.
  • US payroll is not a single product; it is a federation of 50 distinct state tax codes, local jurisdictions, reciprocity rules, and quarterly filing obligations.

Grades and run details

Decision model 80 · LLM judge 45
Decision model checks
  • failUses the supplied evidence correctly56%
  • passAddresses the actual decision56%
  • passRespects explicit constraints24%
  • partialIdentifies material uncertainty29%
  • partialAvoids unsupported claims59%
  • passProduces the required deliverable84%
  • passRebuilds the market size40%
  • passFinds the timeline that sinks it100%
  • passTigers, not paper tigers78%
  • passFinds the need behind the idea57%
Run
Run
#1
API response time
27 s
Submitted
6 Oct 2026

Usable with a quick edit8 of 10 checks passed

The memo is well-reasoned and actionable; the unsupported local-taxes claim is minor and can be quickly removed or hedged without redoing the analysis.

Output

To: CEO and Board From: VP Product Subject: US expansion: approve validation, not the proposed launch

Recommendation: Do not approve the plan as written. The US may be attractive, but the deck mistakes business counts for demand, imports our weakest UK economics, and commits to distribution before we can reliably deliver the product. Approve a staged validation programme instead, with compliance and commercial gates before releasing the full $2.5M.

What the plan gets wrong

1. The addressable market is not 33 million payroll buyers. Of 33.2 million US small businesses, 27.1 million have no employees. Some owner-only businesses may need payroll, but they are not equivalent to employer businesses. The relevant starting population is approximately 6.1 million employers—and probably the 1.1 million with 10–249 employees, given our UK results.

“Even 1%” is not a strategy. It assumes a reachable segment, competitive differentiation, distribution and retention that we have not demonstrated.

2. Our own data contradicts the proposed price and customer strategy.

UK segmentCustomers12-month retentionAnnual revenue/customer
------:---:---:
Sole trader6,10018%$150
2–9 staff5,30033%$150
10–49 staff2,90071%$620
50–249 staff48078%$2,400

Businesses with 10–249 staff represent approximately 23% of customers but 63% of the annual revenue implied by this table. Their weighted retention is approximately 72%, versus 25% for customers below ten staff. Our strongest evidence favours larger employers, not maximum account volume at $150.

Using our UK customer mix within 10–249 staff gives approximately $873 annual revenue per customer. Applied to 1.1 million US businesses, that suggests a roughly $960M segment revenue pool—not a forecast, and not all serviceable by a ten-state product. US willingness to pay remains untested.

There is also a board-data problem: the table implies $4.66M annual revenue, versus the stated $3.9M ARR. Reconcile definitions, dates and cohorts before presenting either figure as the economic baseline.

3. The sales plan and revenue target do not fit. At $150 annually, $5M ARR requires approximately 33,333 active customers: 463 per salesperson per month across six reps over a full year, before churn. Even allocating the entire $2.5M budget to acquisition would allow only $75 per active customer, leaving nothing for engineering, compliance or support.

At the illustrative $873 blend, we still need approximately 5,730 active customers—80 per rep per month. These are workload checks, not sales forecasts, but neither resembles a credible unproven, sales-led launch. A later launch compresses the selling window further.

4. Competitive fundraising is not customer evidence. A competitor’s financing may raise acquisition costs or accelerate competition. It does not establish that our product is ready or that customers will switch. “Get there first” is especially weak when the competitor is already there.

What would sink it

Payroll reliability and coverage. The team estimates nine months for the first ten states. A Q1 launch is unsupported unless substantial work is already complete or we use a proven partner. Ten states also do not automatically mean ten self-contained markets: employee locations, local taxes and multi-state operations affect eligibility. Incorrect wages, filings or remittances can destroy trust faster than sales can rebuild it.

Scaling the wrong economics. At our smallest-customer retention levels, acquisition fills a leaking bucket. Higher retention among larger employers is promising, but customer retention is not revenue retention or gross margin. We lack US acquisition cost, onboarding effort, support burden and contribution-margin evidence.

Starving the UK business. The proposed budget is approximately 64% of current ARR—not a cash-runway calculation, but a material commitment. We need an explicit runway and UK-roadmap impact assessment, including engineering diversion and downside scenarios.

What is less scary than it looks

We do not need nationwide coverage or a large share of the US to build something valuable. A tightly qualified segment can support a meaningful business.

Nor must we necessarily build every payroll component ourselves. A payroll infrastructure partner could shorten time to market, although coverage, margins, accountability and vendor dependence require diligence.

Finally, weak microbusiness retention does not prove Halyard cannot travel. The larger-employer cohorts provide a credible starting hypothesis. They do not yet prove US product-market fit.

What I would propose instead

1. Release up to $400,000 for a 90-day validation phase; hold the remainder. Reconcile the financial baseline, analyse retention by cohort and acquisition channel, and compare partner versus internal-build economics and timelines.

2. Start with 10–49 employees. Recruit 10–20 design partners with simple, geographically supported payroll needs. Validate switching triggers, differentiation, pricing and onboarding. Keep 50–249 as a subsequent hypothesis: better UK retention may come with substantially greater implementation complexity.

3. Gate live payroll on readiness, not the calendar. Require documented coverage, independent compliance review, successful parallel payroll testing, and clear incident ownership. No unsupported employee locations.

4. Run a 30–50-customer paid pilot before hiring six sellers. Use founder-led sales and limited implementation capacity. Measure acquisition cost, onboarding time, support cost, contribution margin and retention. Seek evidence supporting acquisition payback within 12–18 months; early retention is provisional until cohorts mature.

5. Return for scale funding only after those gates. Replace the $5M year-one promise with a bottom-up forecast based on verified price, conversion, launch timing, capacity and churn. The board should fund the next proven step—not an unsupported percentage of America.

Claims the judge couldn’t find in the brief

  • Employee locations, local taxes and multi-state operations affect eligibility.

Grades and run details

Decision model 100 · LLM judge 82
Decision model checks
  • passUses the supplied evidence correctly29%
  • passAddresses the actual decision99%
  • passRespects explicit constraints59%
  • passIdentifies material uncertainty99%
  • passAvoids unsupported claims40%
  • passProduces the required deliverable87%
  • passRebuilds the market size100%
  • passFinds the timeline that sinks it99%
  • passTigers, not paper tigers85%
  • passFinds the need behind the idea44%
Run
Run
#1
Time to output
51 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 5

Uses the supplied evidence correctlyWrongWrongMixed
Sonnet 5.5 · API

The output claims US payroll involves 'federal, state and local rules' but the supplied context only mentions state-by-state tax rules, and it presents 'payroll is a sticky, slow-switching category' as fact without support.

Gemini 3.8 Flash · API

The output invents details about US payroll (local jurisdictions, reciprocity rules, quarterly filing obligations) and claims about incumbents and typical sales rep performance not in the supplied context.

GPT-6 Astra · ChatGPT

Claims that local taxes and multi-state operations affect eligibility are not supported by the supplied context, which only mentions state-by-state tax rules.

Addresses the actual decisionRightMixedRight
Sonnet 5.5 · API

The memo commits to a staged alternative early, frames it for the CEO, and specifies gates that would change the call.

Gemini 3.8 Flash · API

The output commits to a phased strategy but does not state what result or condition would change that call.

GPT-6 Astra · ChatGPT

Commits to not approving the plan and proposes a staged validation, with clear conditions for scaling.

Identifies material uncertaintyRightWrongRight
Sonnet 5.5 · API

It names specific unknowns (budget coverage, ARR discrepancy, US pricing transfer) and says how to resolve them before the board meeting.

Gemini 3.8 Flash · API

The output does not explicitly name specific unknowns that could change the decision or how they would be resolved, beyond implying a pilot will test product-market fit.

GPT-6 Astra · ChatGPT

Identifies untested US willingness to pay, acquisition costs, and retention, and proposes validation to resolve them.

Avoids unsupported claimsWrongWrongMixed
Sonnet 5.5 · API

It presents 'payroll is a sticky, slow-switching category' as an established fact, and 'federal, state and local rules' as a given, neither of which is in the supplied evidence.

Gemini 3.8 Flash · API

The output presents forecasts (e.g., 'guarantees we burn cash', 'will freeze UK product development') and interpretations as established fact without labelling them as hypotheses.

GPT-6 Astra · ChatGPT

Presents the existence of local taxes and multi-state complexity as fact without support from the context.

Rebuilds the market sizeRightMixedRight
Sonnet 5.5 · API

It rebuilds the market using employer counts and UK retention data, arriving at ~$1B for the 1.1M 10-249 employee businesses, showing the working.

Gemini 3.8 Flash · API

The output arrives at a $2.5B–$4B TAM using invented US pricing, not the under $1B market size derived from UK revenue per customer as required.

GPT-6 Astra · ChatGPT

Rebuilds market using employer counts and UK retention, arriving at ~$960M for 1.1M businesses.

All got right 5

Respects explicit constraintsRightRightRight
Sonnet 5.5 · API

The output is a memo under 1,000 words addressed to the CEO, respecting the form and length constraint.

Gemini 3.8 Flash · API

The memo is under 1,000 words, addressed to the CEO, and in memo form.

GPT-6 Astra · ChatGPT

Memo format, under 1,000 words, addressed to CEO and board.

Produces the required deliverableRightRightRight
Sonnet 5.5 · API

The memo is complete, in the right form for the CEO, within the word limit, and actionable with light edits.

Gemini 3.8 Flash · API

The memo is complete, in the requested form, and usable by the CEO.

GPT-6 Astra · ChatGPT

Complete memo with actionable recommendation, within length, usable with light edits.

Finds the timeline that sinks itRightRightRight
Sonnet 5.5 · API

It names the 9-month state tax build and explains that a Q1 launch is impossible and the $5M year-one target unreachable.

Gemini 3.8 Flash · API

The output names the 9-month build and explains it makes a Q1 payroll launch impossible.

GPT-6 Astra · ChatGPT

Cites 9-month build and concludes Q1 launch is unsupported.

Tigers, not paper tigersRightRightRight
Sonnet 5.5 · API

It ranks risks, dismisses the competitor-raise worry as less scary because funding doesn't speed the state build, and surfaces the ARR discrepancy as an unspoken issue.

Gemini 3.8 Flash · API

The output triages risks, dismisses the competitor funding worry with reasoning, and surfaces the issue of replicating the worst UK cohort.

GPT-6 Astra · ChatGPT

Separates sinkers (payroll reliability, scaling wrong economics) from less scary (competitor raise, nationwide coverage), and surfaces ARR discrepancy.

Finds the need behind the ideaRightRightRight
Sonnet 5.5 · API

It identifies the need for US growth and offers a phased alternative with lower cost and risk, stating what it displaces (the aggressive Q1 launch and $5M target).

Gemini 3.8 Flash · API

The output states the need to expand to the US and offers a phased alternative with cost and what it displaces.

GPT-6 Astra · ChatGPT

Identifies need for US expansion, proposes validation instead of full launch, with cost ($400k) and what is displaced.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 76% of 75 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 LunawithAPI100.090.92None
2Haiku 5.5withAPI92.590.92None
3GPT-6.1 SolwithAPI100.086.42None
4GPT-6 AstrawithChatGPT100.079.42None
5Sonnet 5.5withAPI88.374.82None
6Opus 5.5withClaude88.379.421 capped
7Gemini 3.5 Flash-LitewithGemini79.649.721 capped
8Gemini 3.8 FlashwithAPI87.972.721 capped

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.4050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Luna · API: task score 93, the cheapestHaiku 5.5 · API: task score 92, the cheapestGPT-6.1 Sol · API: task score 91, 11× the cheapestGPT-6 Astra · ChatGPT: task score 88, 60× the cheapest (1 of 2 outputs estimated)Sonnet 5.5 · API: task score 78, 37× the cheapestOpus 5.5 · Claude: task score 62, 59× the cheapest (1 of 2 outputs estimated), 1 critical failureGemini 3.5 Flash-Lite · Gemini: task score 50, 1.9× the cheapest (1 of 2 outputs estimated), 1 critical failureGemini 3.8 Flash · API: task score 50, 9.5× the cheapest, 1 critical failureGPT-6 Luna · APIHaiku 5.5 · APIGPT-6.1 Sol · APIGPT-6 Astra · ChatGPTSonnet 5.5 · APIOpus 5.5 · ClaudeGemini 3.5 Flash-Lite · GeminiGemini 3.8 Flash · API
Cost partly estimated (app runs with no usage meter)At least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API931.0×
Haiku 5.5 · API921.4×
Gemini 3.5 Flash-Lite · Gemini501.9× (partly estimated)
Gemini 3.8 Flash · API509.5×
GPT-6.1 Sol · API9111×
Sonnet 5.5 · API7837×
Opus 5.5 · Claude6259× (partly estimated)
GPT-6 Astra · ChatGPT8860× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Pressure-testing a proposal before committing a team to it.

Why it matters

The useful critic finds the one assumption everything rests on. Theatrical negativity is easy to generate and useless in a planning meeting.

What good looks like

  • Identifies the load-bearing assumption
  • Separates the risks that could kill it from the ones that only look scary
  • Uses the supplied evidence, not generic risks
  • Proposes the cheapest way to test the assumption

Deliberately not measured

  • Tone
  • Number of objections raised
Capability tested

Evidence-based critique

The failure we’re looking for

Theatrical negativity without evidence

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

Vanilla prompt (core) · With Roast Me skill · Staff level: a company bet with a long evidence pack