Tasks / Strategy

Challenge an idea

Can the model find the strongest reason an idea may fail, backed by evidence?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 44% were usable with at most a quick edit.

Reliably right

  1. Finds the load-bearing assumption100% pass
    The critique targets the assumption that agencies' growth is limited by a shortage of new leads, using referral, abandoned-outbound, and capacity evidence.
    GPT-6.1 Sol · API · An AI SDR for small agencies
  2. Uses the interviews faithfully100% pass
    All quotes are accurately taken from the transcripts and correctly attributed to A05, A04, and A08.
    GPT-6.1 Sol · API · An AI SDR for small agencies
  3. Finds the timeline that sinks it100% pass
    Cites 9-month build and concludes Q1 launch is unsupported.
    GPT-6 Astra · ChatGPT · Five billion dollars, on paper

Where it slips

  1. Avoids unsupported claims55% pass
    Presents the existence of local taxes and multi-state complexity as fact without support from the context.
    GPT-6 Astra · ChatGPT · Five billion dollars, on paper
  2. Uses the supplied evidence correctly55% pass
    Claims that local taxes and multi-state operations affect eligibility are not supported by the supplied context, which only mentions state-by-state tax rules.
    GPT-6 Astra · ChatGPT · Five billion dollars, on paper
  3. Rebuilds the market size81% pass
    It uses the $150 price for 10+ staff businesses instead of the higher UK revenue per customer for those segments, so the market size is understated and not correctly rebuilt.
    GPT-6 Luna · API · Five billion dollars, on paper

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Finds the need behind the idea

    Does the critique identify the underlying need the idea is meant to serve, and offer at least one other way to meet that need, at a different cost, saying what would have to be dropped for it?

    Passes when States the need behind the proposal, and gives an alternative way to meet it with its cost and what it displaces.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Tigers, not paper tigers · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the VP Product at Halyard. Our CEO is taking the US launch plan below to the board next week and has asked you to challenge it hard first. Write a memo of no more than 1,000 words: what the plan gets wrong, what would sink it, what's less scary than it looks, and what you'd propose instead. The deck excerpt and our data are attached.

What the model was given4 items: about_halyard.md, ceo_board_deck_excerpt.md, uk_customers_by_size.csv, us_small_business_stats.md
about_halyard.md3 lines · Download
# About Halyard in the UK

Payroll and HR for small businesses. 14,800 customers, $3.9M ARR, five years old. UK payroll rules are built in; US payroll would need state-by-state tax rules (the team estimates 9 months of work for the first 10 states).
ceo_board_deck_excerpt.md5 lines · Download
# US launch (CEO's board deck, excerpt)

- Market: 33 million US small businesses x our UK price of £120 ($150) a year = about $5B. "Even 1% is $50M."
- Plan: launch in the US in Q1, $2.5M budget, hire a US sales team of 6, target $5M ARR in year one.
- Why now: a competitor raised in the US; we need to get there first.
uk_customers_by_size.csv4 rows · Download
company_size,uk_customers,share_of_uk_customers_pct,retained_at_12_months_pct,avg_revenue_usd_per_year
Sole trader,6100,41,18,150
2-9 staff,5300,36,33,150
10-49 staff,2900,20,71,620
50-249 staff,480,3,78,2400
us_small_business_stats.md5 lines · Download
# US small businesses (government statistics)

- 33.2 million small businesses in total.
- About 27.1 million have no employees (sole proprietors).
- About 6.1 million have employees; of those, about 1.1 million have 10 to 249 employees.
What a strong answer doesThe answer key the graders mark against

Shows the $5B market is the wrong number. It multiplies every US small business by the UK price, but in the UK only businesses with 10+ staff stay (71% and 78% at 12 months) and pay more ($620 to $2,400 a year); sole traders and 2-9 businesses mostly leave (18% and 33%). Most US small businesses (27.1M of 33.2M) have no employees and don't need payroll at all; the comparable US market is about 1.1M businesses with 10 to 249 staff, worth roughly $0.7B to $1B at UK revenue per customer, not $5B. The tiger: US payroll needs state-by-state tax rules, about 9 months for the first 10 states, so a Q1 launch with payroll isn't possible and $5M ARR in year one would need thousands of 10+ staff customers from a standing start. Dismisses the 'first mover' worry as less scary: a competitor's raise doesn't make a market winner-takes-all. Proposes instead: target 10+ staff businesses, start with HR in the US while payroll is built for a few high-density states, test demand with a small budget and a clear threshold before hiring a sales team, and fix the UK mix towards 10+ staff companies meanwhile.

Critical failures (cap the score at 40)
  • Endorses the $5B market size
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Needs more than a quick edit8 of 10 checks passed

The output includes unsupported claims about US payroll rules (federal/local) and market stickiness that a PM would need to verify before using, so it cannot be used without re-checking evidence.

Output

MEMO: US launch plan, challenge review To: CEO | From: VP Product

Bottom line: The direction is defensible, but the plan isn't ready for the board. The market is about a fifth the size the deck claims, the Q1 launch date conflicts with our own engineering estimate, and the $5M year-one target is unreachable. I'd take a staged version instead.

---

What the plan gets wrong

1. The $5B market is wrong by roughly 5x. The deck multiplies all 33.2M US small businesses by $150. But 27.1M of them have no employees, so they have no payroll to run. Our own UK data shows how little this segment is worth to us:

  • Sole traders are 41% of UK customers but only about 20% of revenue, and just 18% are still with us at 12 months.
  • Firms with 10+ staff are 23% of customers but about 63% of revenue, with 71–78% retention.

Our proven segment in the US is the 1.1M firms with 10–249 employees. At our UK blended revenue for that group (about $870 per customer), that's roughly $1B. "Even 1%" is about $10M, not $50M. I'd exclude the ~5M micro-employers. In the UK, 2–9 staff firms pay $150 and only 33% stay a year.

2. The timeline contradicts itself. The team estimates 9 months of work for the first 10 states. A Q1 launch means selling a product that doesn't exist, or one that only works in a few states. Ten states is also only about half the US population, by my rough estimate, so a national sales team would be pitching to prospects we can't serve.

3. $5M ARR in year one is out of reach. It exceeds the ARR we built in five years in the UK. At ~$870 per customer it needs about 5,700 customers, roughly 950 per rep from a standing start. With a 9-month build, year-one revenue would be close to zero.

4. "A competitor raised" isn't a reason to launch now. We can't win a spending race with $2.5M against a funded rival. Payroll is also a sticky, slow-switching category, so launching early earns us little. Launching with wrong tax calculations would cost us far more.

What would sink it

  • Compliance failures. A wrong withholding or a missed filing means customers pay penalties and blame us. In the UK, rules are built in and we know them. In the US we'd face federal, state and local rules, each with its own filings. One bad quarter in the first cohort could end our reputation with the segment we need.
  • Targeting the wrong customer. If the sales team chases volume, it will sign small firms that churn at UK micro-segment rates. Acquisition cost would never pay back.
  • Draining the UK business. Nine months of engineering on tax rules comes out of the roadmap for the business that funds everything. UK retention in the smallest segments is already weak, and we'd be neglecting it.
  • Burning the budget before learning anything. Six hires plus a 9-month build could consume most of $2.5M, about 64% of current ARR, before we know whether US customers retain like UK ones. I need to confirm whether the $2.5M includes the engineering cost. If it doesn't, the real ask is larger.
  • Missing the target. A $5M goal that slips will damage board confidence in everything else we present.

What's less scary than it looks

  • The smaller market is still big. About $1B is over 200x our current ARR. We don't need $5B to justify the US.
  • The competitor raise. It validates demand, but funding doesn't fix the state-by-state build, which slows them as well. Nothing here is lost by launching in 12–18 months instead of 3.
  • The 9-month estimate. It's a well-scoped build with a known end date, and a tax-calculation partner or API might shorten it. I'd evaluate build vs. buy before committing.
  • UK retention in micro-segments. It looks alarming, but it's a UK issue we'd avoid importing by targeting 10–249 staff firms, where our retention is strongest.

What I'd propose instead

Phase 0 (now to ~Q3, ~$0.5–0.7M): - Decide build vs. buy for the tax engine. - Pick 3–4 launch states by density of 10–249 staff firms. - Recruit 15–20 design-partner customers. - Hire a US head of sales/GTM and one compliance specialist, not six reps.

Phase 1 (launch in 3–5 states once compliance is proven, ~$1M): - Sell only to 10–249 staff firms with 2 reps and dedicated onboarding. - Set the target at about 300 customers and ~$0.25–0.3M ARR twelve months after launch. That's my estimate, based on UK segment pricing.

Phase 2 (release the remaining ~$1M, expand states and team) only if we hit gates: - 12-month retention of at least 70% in the first cohort. - Zero material filing errors. - Acquisition cost paying back within an agreed window.

This puts about $0.5–0.7M at risk before we learn anything, rather than $2.5M, and it gives the board a credible plan with checkpoints.

Before the board meeting

  1. Reconcile ARR. Our segment table implies about $4.66M (customers × average revenue), but the deck and company profile say $3.9M. A board member will do this multiplication. We need to know whether the gap is discounting, churn timing, or stale data.
  2. Confirm what the $2.5M covers, particularly engineering and compliance.
  3. Restate the market as ~$1B core, with the logic shown.
  4. Check price transfer. UK per-segment pricing may not hold in the US, so the design-partner phase should test willingness to pay.

I'm happy to walk through the numbers before you finalize the deck.

Grades and run details

Decision model 85 · LLM judge 73
Decision model checks
  • failUses the supplied evidence correctly23%
  • passAddresses the actual decision81%
  • passRespects explicit constraints59%
  • passIdentifies material uncertainty99%
  • partialAvoids unsupported claims68%
  • passProduces the required deliverable82%
  • passRebuilds the market size99%
  • passFinds the timeline that sinks it100%
  • passTigers, not paper tigers78%
  • passFinds the need behind the idea30%
Run
Run
#1
API response time
46 s
Submitted
6 Oct 2026

Needs more than a quick edit5 of 10 checks passed

Unsupported claims about acquisition cost and support load would need evidence re-checking, and the missing market-size calculation means the reasoning is incomplete.

Output

MEMORANDUM

TO: CEO FROM: VP Product DATE: October 24, 2023 SUBJECT: Critique and Alternative Proposal for the US Launch Plan

---

You asked me to challenge the US expansion plan hard before you take it to the board next week. I’ve reviewed the deck excerpt against our UK unit economics and US market data.

If we walk into the boardroom with the current plan, a savvy board will pick it apart—and rightly so. Here is my unvarnished assessment of what the plan gets wrong, what will sink us, what is actually noise, and what we should propose instead.

---

1. What the plan gets wrong (The fatal assumptions)

  • The "TAM Fallacy" on Sole Proprietors: The deck multiplies 33 million US small businesses by $150 to claim a $5B market. Government data reveals that 27.1 million of those businesses (82%) have zero employees. In the UK, our data tells a brutal story about sole traders: 41% of our customer base (6,100 accounts) churn within 12 months at an 18% retention rate. They are cheap to acquire, high-churn, high-support-load nightmares who don’t run payroll. Treating the entire 33 million pool as viable is a math error that misleads us on product-market fit.
  • The Compliance Mirage: The deck ignores the engineering reality. US payroll is not a monolith; it is federal, state, and local tax compliance across 50 distinct jurisdictions. Our engineering team estimates 9 months of work just to launch the first 10 states. You cannot launch a payroll product in "the US" in Q1 with a generic software wrapper. Without state tax tables, automated filings, and local garnishments, the product is illegal to use.
  • The Unit Economics Math: At $150/year, acquiring a customer through a dedicated US sales team of 6 with a $2.5M budget is mathematically impossible. A traditional outbound sales team cannot profitably sell a $150 self-serve product. CAC will dwarf LTV on day one.

---

2. What would sink us

  • Regulatory suicide: If we launch prematurely in Q1 to "beat a competitor" without robust state-by-state tax calculation, our first cohort of US customers will face IRS and state tax penalties. In payroll, a software bug doesn't just mean a bad user experience; it means employees don’t get paid or taxes aren't remitted, resulting in immediate churn, devastating brand damage, and potential litigation.
  • Reactive strategy: "A competitor raised in the US; we need to get there first" is a panic motivation, not a product strategy. If we rush a half-baked product into a massive market just to plant a flag, we will burn $2.5M in 12 months, fail to hit the $5M ARR target, and lose credibility with the board.

---

3. What’s less scary than it looks

  • The total market size: We don't need 33 million businesses. Look at our UK data: our sweet spot is companies with 10–49 staff (20% of our base, but 71% retention, generating significantly higher revenue) and 50–249 staff (78% retention, $2,400+ ARR). In the US, there are 1.1 million businesses in the 10–249 employee bracket. That is a massive, highly qualified, high-retention target market. We don't need the sole traders; we should actively avoid them.
  • Geographic concentration: We do not need to launch all 50 states on day one. If we pick 3 to 5 high-density, business-friendly states (e.g., Texas, Florida, Washington—states with no state income tax or straightforward payroll setups), we can drastically reduce engineering scope, bypass the 9-month/10-state bottleneck, and launch a compliant MVP much faster.

---

4. What I propose instead

We should go to the board with a revised, credible strategy that protects our capital and targets the segment we actually know how to monetize.

  1. Shift the ICP (Ideal Customer Profile) away from Sole Traders: Ignore the 27.1 million non-employer businesses. Focus exclusively on US small businesses with 10 to 49 employees. They have the budget, they have payroll complexity, and based on our UK experience, they stick around.
  2. Pivoting the Go-To-Market (GTM): Scrap the plan for a 6-person outbound sales team on a $150 product. At this price point, the US motion must be product-led growth (PLG) with digital acquisition, augmented by a lightweight, inside-sales motion only for accounts scaling toward 50+ employees.
  3. De-risk the Engineering Roadmap: Reallocate the $2.5M budget. Instead of a clumsy nationwide Q1 rush, fund a targeted compliance build for just 3 states to launch a closed beta by Q2. Use the remaining budget for localized digital marketing and product-led onboarding flows.
  4. Refrain the Board Narrative: Do not pitch "capturing 1% of a $5B market." Pitch a disciplined land-grab in a profitable sub-segment: “We are entering the US by targeting the 1.1 million businesses with 10–249 employees, starting with a compliant 3-state footprint, leveraging our proven retention playbook to capture high-value employers at a sustainable CAC.”

If we present this, the board will see operators who understand our unit economics, respect regulatory risk, and have a clear path to sustainable growth. Let’s calibrate the deck before Tuesday.

Claims the judge couldn’t find in the brief

  • They are cheap to acquire, high-churn, high-support-load nightmares

Grades and run details

Decision model 80 · LLM judge 45
Decision model checks
  • partialUses the supplied evidence correctly13%
  • passAddresses the actual decision34%
  • passRespects explicit constraints47%
  • partialIdentifies material uncertainty52%
  • partialAvoids unsupported claims49%
  • passProduces the required deliverable68%
  • passRebuilds the market size57%
  • passFinds the timeline that sinks it100%
  • partialTigers, not paper tigers35%
  • passFinds the need behind the idea26%
Run
Run
#1
Time to output
6 s
Submitted
5 Oct 2026

Needs more than a quick edit9 of 10 checks passed

The TAM calculation incorrectly applies the $150 price to 10+ staff businesses, so a PM would need to redo the market sizing and adjust the argument before using it.

Output

To: CEO Subject: US launch — don’t take this plan to the board as written

Bottom line: The US could be a real opportunity, but this deck turns a large headline market into an unsupported year-one sales forecast. The biggest risk isn’t arriving late; it’s selling payroll before we can run it reliably in the states we promise to support.

What the plan gets wrong

  • The TAM math is arithmetically fine and commercially misleading. The 33.2 million small-business count includes 27.1 million firms with no employees—about 82% of the total. That is not a credible proxy for payroll buyers. There are 6.1 million employer firms, including 1.1 million with 10–249 employees. Even those are not automatically our customers. Applying our UK price to every US small business assumes identical need, willingness to pay, and product fit.
  • The UK data argues against targeting everyone. Retention is 18% for sole traders and 33% for firms with 2–9 staff, versus 71% for 10–49 and 78% for 50–249. Our best evidence points toward employers with more staff—not a mass-market pitch built on the largest, weakest-fit segment.
  • The price and revenue baseline need reconciling. At $150 per year, $5 million ARR means roughly 33,300 customers. That is more than twice our current UK customer base, which took five years to build. Also, the UK segment counts and stated average revenues imply about $4.66 million in annual revenue, versus the reported $3.9 million ARR. These may use different definitions, but we must explain the $760,000 gap before using UK economics to price or forecast the US.
  • “A competitor raised” is not a launch strategy. It may validate investor interest, but it tells us nothing about buyer demand for Halyard, our differentiation, acquisition cost, or which states and customer segments we can serve profitably.

What would sink it

The timeline is incompatible with the product plan. We estimate nine months to build payroll tax rules for the first 10 states; the deck promises a Q1 US launch. A launch without dependable tax calculations, filings, registrations, and support creates compliance exposure and destroys trust. “Ten states” is not the same as nationwide coverage, and the deck does not say what launch actually means.

The target compounds the problem. At the proposed price, we need 33,300 customers in year one—without a validated US product, channel, conversion rate, or sales cycle. Six sales hires do not make that forecast credible. The $2.5 million budget also needs to cover product, tax expertise, support, and go-to-market; the deck gives no cost or hiring ramp to show it can.

What’s less scary than it looks

The market does not need to be 33 million businesses to matter. The 6.1 million employer firms are a meaningful starting universe; at $150 per year, that price would imply a crude ceiling of about $915 million, not a forecast. The 1.1 million firms with 10–249 employees alone imply about $165 million at that same price. Those figures are still not a serviceable-market estimate, but they show we can pursue a focused wedge without pretending every sole proprietor is a prospect.

We also have useful evidence: five years of operating experience, 14,800 UK customers, and better retention among larger employers. That is a foundation for testing the US—not proof that the UK playbook transfers unchanged.

What I’d propose instead

Approve a staged US entry, not a Q1 national launch or a $5 million year-one commitment.

  1. Choose a narrow first customer and geography. Start with employer businesses, prioritizing 10–49 staff; select states using verified prospect demand and implementation feasibility. Defer sole traders and the weakest-retaining segment until we have evidence they can work.
  2. Build and validate before scaling sales. Recruit US design partners, test willingness to pay and workflows, and ship only when payroll accuracy, filing operations, and support are proven in the launch states. Treat the nine-month estimate as a planning constraint, not a promise to cover 10 states by Q1.
  3. Stage the investment. Fund discovery, state selection, and the first production-ready footprint now. Hold most sales hiring until pilots demonstrate successful payroll cycles, repeatable acquisition, and acceptable retention and unit economics. Consider accountant or payroll-partner channels before assuming six direct sellers are the answer.
  4. Return to the board with evidence. Bring a state-by-state scope, reconciled UK revenue data, tested US pricing, customer pipeline, launch-readiness criteria, and a bottom-up forecast tied to conversion and sales capacity.

We should move with urgency—but urgency should accelerate learning, not turn an unvalidated forecast into a launch promise.

Grades and run details

Decision model 100 · LLM judge 82
Decision model checks
  • passUses the supplied evidence correctly60%
  • passAddresses the actual decision90%
  • passRespects explicit constraints67%
  • passIdentifies material uncertainty89%
  • passAvoids unsupported claims37%
  • passProduces the required deliverable83%
  • passRebuilds the market size60%
  • passFinds the timeline that sinks it100%
  • passTigers, not paper tigers71%
  • passFinds the need behind the idea24%
Run
Run
#1
API response time
29 s
Submitted
6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 5

Uses the supplied evidence correctlyWrongWrongRight
Sonnet 5.5 · API

The output claims US payroll involves 'federal, state and local rules' but the supplied context only mentions state-by-state tax rules, and it presents 'payroll is a sticky, slow-switching category' as fact without support.

Gemini 3.5 Flash-Lite · Gemini

The claim that sole traders are 'cheap to acquire' and 'high-support-load' is not supported by the supplied context.

GPT-6 Luna · API

All claims about the current situation are directly supported by the supplied context or arithmetic from it.

Identifies material uncertaintyRightWrongRight
Sonnet 5.5 · API

It names specific unknowns (budget coverage, ARR discrepancy, US pricing transfer) and says how to resolve them before the board meeting.

Gemini 3.5 Flash-Lite · Gemini

The memo does not name specific unknowns that could change the decision or say how they would be resolved.

GPT-6 Luna · API

It names unknowns (US demand, conversion, unit economics, revenue data gap) and proposes pilots and evidence thresholds to resolve them.

Avoids unsupported claimsWrongWrongRight
Sonnet 5.5 · API

It presents 'payroll is a sticky, slow-switching category' as an established fact, and 'federal, state and local rules' as a given, neither of which is in the supplied evidence.

Gemini 3.5 Flash-Lite · Gemini

It presents 'cheap to acquire' and 'high-support-load' as established facts without labelling them as assumptions or hypotheses.

GPT-6 Luna · API

Interpretations and forecasts are presented as such, not as established facts; the revenue gap is arithmetic, not speculation.

Rebuilds the market sizeRightMixedMixed
Sonnet 5.5 · API

It rebuilds the market using employer counts and UK retention data, arriving at ~$1B for the 1.1M 10-249 employee businesses, showing the working.

Gemini 3.5 Flash-Lite · Gemini

It does not rebuild the market to a dollar figure (e.g., under $1B) or show the working; it only states the 1.1M business count without sizing the revenue opportunity.

GPT-6 Luna · API

It uses the $150 price for 10+ staff businesses instead of the higher UK revenue per customer for those segments, so the market size is understated and not correctly rebuilt.

Finds the need behind the ideaRightMixedRight
Sonnet 5.5 · API

It identifies the need for US growth and offers a phased alternative with lower cost and risk, stating what it displaces (the aggressive Q1 launch and $5M target).

Gemini 3.5 Flash-Lite · Gemini

It does not state the underlying need behind the US launch idea (e.g., growth, first-mover advantage) and therefore does not offer an alternative way to meet that need with its cost and trade-offs.

GPT-6 Luna · API

It identifies the need to capture the US opportunity and proposes a staged entry as an alternative, with lower upfront cost and what it displaces (national launch, $5M target).

All got right 5

Addresses the actual decisionRightRightRight
Sonnet 5.5 · API

The memo commits to a staged alternative early, frames it for the CEO, and specifies gates that would change the call.

Gemini 3.5 Flash-Lite · Gemini

The memo commits to a clear alternative strategy (target 10-49 employee businesses, PLG, 3-state launch) early and unambiguously.

GPT-6 Luna · API

The memo commits to a clear call (don't take the plan as written, do a staged entry) early, framed for the CEO, and implies the decision would change based on pilot evidence.

Respects explicit constraintsRightRightRight
Sonnet 5.5 · API

The output is a memo under 1,000 words addressed to the CEO, respecting the form and length constraint.

Gemini 3.5 Flash-Lite · Gemini

The output is a memo under 1,000 words, addressed to the CEO, and respects the requested form.

GPT-6 Luna · API

The output is a memo to the CEO, under 1,000 words, and addresses all requested elements.

Produces the required deliverableRightRightRight
Sonnet 5.5 · API

The memo is complete, in the right form for the CEO, within the word limit, and actionable with light edits.

Gemini 3.5 Flash-Lite · Gemini

The memo is complete, in the right form, within length, and usable by the CEO with light edits.

GPT-6 Luna · API

The memo is complete, in the right form, within length, and a CEO could act on it with light edits.

Finds the timeline that sinks itRightRightRight
Sonnet 5.5 · API

It names the 9-month state tax build and explains that a Q1 launch is impossible and the $5M year-one target unreachable.

Gemini 3.5 Flash-Lite · Gemini

It names the 9-month build and explicitly states that a Q1 payroll launch is impossible.

GPT-6 Luna · API

It explicitly names the 9-month build and explains why it makes a Q1 payroll launch impossible and the year-one target unachievable.

Tigers, not paper tigersRightRightRight
Sonnet 5.5 · API

It ranks risks, dismisses the competitor-raise worry as less scary because funding doesn't speed the state build, and surfaces the ARR discrepancy as an unspoken issue.

Gemini 3.5 Flash-Lite · Gemini

It ranks regulatory and reactive strategy as sinkers, dismisses the competitor raise as less scary, and surfaces the need to avoid sole traders.

GPT-6 Luna · API

It ranks the timeline/product risk as the sinker, dismisses the first-mover worry as less scary, and surfaces the UK revenue data discrepancy.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 76% of 75 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 LunawithAPI100.090.92None
2Haiku 5.5withAPI92.590.92None
3GPT-6.1 SolwithAPI100.086.42None
4GPT-6 AstrawithChatGPT100.079.42None
5Sonnet 5.5withAPI88.374.82None
6Opus 5.5withClaude88.379.421 capped
7Gemini 3.5 Flash-LitewithGemini79.649.721 capped
8Gemini 3.8 FlashwithAPI87.972.721 capped

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.4050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Luna · API: task score 93, the cheapestHaiku 5.5 · API: task score 92, the cheapestGPT-6.1 Sol · API: task score 91, 11× the cheapestGPT-6 Astra · ChatGPT: task score 88, 60× the cheapest (1 of 2 outputs estimated)Sonnet 5.5 · API: task score 78, 37× the cheapestOpus 5.5 · Claude: task score 62, 59× the cheapest (1 of 2 outputs estimated), 1 critical failureGemini 3.5 Flash-Lite · Gemini: task score 50, 1.9× the cheapest (1 of 2 outputs estimated), 1 critical failureGemini 3.8 Flash · API: task score 50, 9.5× the cheapest, 1 critical failureGPT-6 Luna · APIHaiku 5.5 · APIGPT-6.1 Sol · APIGPT-6 Astra · ChatGPTSonnet 5.5 · APIOpus 5.5 · ClaudeGemini 3.5 Flash-Lite · GeminiGemini 3.8 Flash · API
Cost partly estimated (app runs with no usage meter)At least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API931.0×
Haiku 5.5 · API921.4×
Gemini 3.5 Flash-Lite · Gemini501.9× (partly estimated)
Gemini 3.8 Flash · API509.5×
GPT-6.1 Sol · API9111×
Sonnet 5.5 · API7837×
Opus 5.5 · Claude6259× (partly estimated)
GPT-6 Astra · ChatGPT8860× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Pressure-testing a proposal before committing a team to it.

Why it matters

The useful critic finds the one assumption everything rests on. Theatrical negativity is easy to generate and useless in a planning meeting.

What good looks like

  • Identifies the load-bearing assumption
  • Separates the risks that could kill it from the ones that only look scary
  • Uses the supplied evidence, not generic risks
  • Proposes the cheapest way to test the assumption

Deliberately not measured

  • Tone
  • Number of objections raised
Capability tested

Evidence-based critique

The failure we’re looking for

Theatrical negativity without evidence

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

Vanilla prompt (core) · With Roast Me skill · Staff level: a company bet with a long evidence pack