Tasks / Strategy

Challenge an idea

Can the model find the strongest reason an idea may fail, backed by evidence?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 44% were usable with at most a quick edit.

Reliably right

  1. Finds the load-bearing assumption100% pass
    The critique targets the assumption that agencies' growth is limited by a shortage of new leads, using referral, abandoned-outbound, and capacity evidence.
    GPT-6.1 Sol · API · An AI SDR for small agencies
  2. Uses the interviews faithfully100% pass
    All quotes are accurately taken from the transcripts and correctly attributed to A05, A04, and A08.
    GPT-6.1 Sol · API · An AI SDR for small agencies
  3. Finds the timeline that sinks it100% pass
    Cites 9-month build and concludes Q1 launch is unsupported.
    GPT-6 Astra · ChatGPT · Five billion dollars, on paper

Where it slips

  1. Avoids unsupported claims55% pass
    Presents the existence of local taxes and multi-state complexity as fact without support from the context.
    GPT-6 Astra · ChatGPT · Five billion dollars, on paper
  2. Uses the supplied evidence correctly55% pass
    Claims that local taxes and multi-state operations affect eligibility are not supported by the supplied context, which only mentions state-by-state tax rules.
    GPT-6 Astra · ChatGPT · Five billion dollars, on paper
  3. Rebuilds the market size81% pass
    It uses the $150 price for 10+ staff businesses instead of the higher UK revenue per customer for those segments, so the market size is understated and not correctly rebuilt.
    GPT-6 Luna · API · Five billion dollars, on paper

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Finds the need behind the idea

    Does the critique identify the underlying need the idea is meant to serve, and offer at least one other way to meet that need, at a different cost, saying what would have to be dropped for it?

    Passes when States the need behind the proposal, and gives an alternative way to meet it with its cost and what it displaces.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Tigers, not paper tigers · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the VP Product at Halyard. Our CEO is taking the US launch plan below to the board next week and has asked you to challenge it hard first. Write a memo of no more than 1,000 words: what the plan gets wrong, what would sink it, what's less scary than it looks, and what you'd propose instead. The deck excerpt and our data are attached.

What the model was given4 items: about_halyard.md, ceo_board_deck_excerpt.md, uk_customers_by_size.csv, us_small_business_stats.md
about_halyard.md3 lines · Download
# About Halyard in the UK

Payroll and HR for small businesses. 14,800 customers, $3.9M ARR, five years old. UK payroll rules are built in; US payroll would need state-by-state tax rules (the team estimates 9 months of work for the first 10 states).
ceo_board_deck_excerpt.md5 lines · Download
# US launch (CEO's board deck, excerpt)

- Market: 33 million US small businesses x our UK price of £120 ($150) a year = about $5B. "Even 1% is $50M."
- Plan: launch in the US in Q1, $2.5M budget, hire a US sales team of 6, target $5M ARR in year one.
- Why now: a competitor raised in the US; we need to get there first.
uk_customers_by_size.csv4 rows · Download
company_size,uk_customers,share_of_uk_customers_pct,retained_at_12_months_pct,avg_revenue_usd_per_year
Sole trader,6100,41,18,150
2-9 staff,5300,36,33,150
10-49 staff,2900,20,71,620
50-249 staff,480,3,78,2400
us_small_business_stats.md5 lines · Download
# US small businesses (government statistics)

- 33.2 million small businesses in total.
- About 27.1 million have no employees (sole proprietors).
- About 6.1 million have employees; of those, about 1.1 million have 10 to 249 employees.
What a strong answer doesThe answer key the graders mark against

Shows the $5B market is the wrong number. It multiplies every US small business by the UK price, but in the UK only businesses with 10+ staff stay (71% and 78% at 12 months) and pay more ($620 to $2,400 a year); sole traders and 2-9 businesses mostly leave (18% and 33%). Most US small businesses (27.1M of 33.2M) have no employees and don't need payroll at all; the comparable US market is about 1.1M businesses with 10 to 249 staff, worth roughly $0.7B to $1B at UK revenue per customer, not $5B. The tiger: US payroll needs state-by-state tax rules, about 9 months for the first 10 states, so a Q1 launch with payroll isn't possible and $5M ARR in year one would need thousands of 10+ staff customers from a standing start. Dismisses the 'first mover' worry as less scary: a competitor's raise doesn't make a market winner-takes-all. Proposes instead: target 10+ staff businesses, start with HR in the US while payroll is built for a few high-density states, test demand with a small budget and a clear threshold before hiring a sales team, and fix the UK mix towards 10+ staff companies meanwhile.

Critical failures (cap the score at 40)
  • Endorses the $5B market size
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Needs more than a quick edit5 of 10 checks passed

Unsupported claims about acquisition cost and support load would need evidence re-checking, and the missing market-size calculation means the reasoning is incomplete.

Output

MEMORANDUM

TO: CEO FROM: VP Product DATE: October 24, 2023 SUBJECT: Critique and Alternative Proposal for the US Launch Plan

---

You asked me to challenge the US expansion plan hard before you take it to the board next week. I’ve reviewed the deck excerpt against our UK unit economics and US market data.

If we walk into the boardroom with the current plan, a savvy board will pick it apart—and rightly so. Here is my unvarnished assessment of what the plan gets wrong, what will sink us, what is actually noise, and what we should propose instead.

---

1. What the plan gets wrong (The fatal assumptions)

  • The "TAM Fallacy" on Sole Proprietors: The deck multiplies 33 million US small businesses by $150 to claim a $5B market. Government data reveals that 27.1 million of those businesses (82%) have zero employees. In the UK, our data tells a brutal story about sole traders: 41% of our customer base (6,100 accounts) churn within 12 months at an 18% retention rate. They are cheap to acquire, high-churn, high-support-load nightmares who don’t run payroll. Treating the entire 33 million pool as viable is a math error that misleads us on product-market fit.
  • The Compliance Mirage: The deck ignores the engineering reality. US payroll is not a monolith; it is federal, state, and local tax compliance across 50 distinct jurisdictions. Our engineering team estimates 9 months of work just to launch the first 10 states. You cannot launch a payroll product in "the US" in Q1 with a generic software wrapper. Without state tax tables, automated filings, and local garnishments, the product is illegal to use.
  • The Unit Economics Math: At $150/year, acquiring a customer through a dedicated US sales team of 6 with a $2.5M budget is mathematically impossible. A traditional outbound sales team cannot profitably sell a $150 self-serve product. CAC will dwarf LTV on day one.

---

2. What would sink us

  • Regulatory suicide: If we launch prematurely in Q1 to "beat a competitor" without robust state-by-state tax calculation, our first cohort of US customers will face IRS and state tax penalties. In payroll, a software bug doesn't just mean a bad user experience; it means employees don’t get paid or taxes aren't remitted, resulting in immediate churn, devastating brand damage, and potential litigation.
  • Reactive strategy: "A competitor raised in the US; we need to get there first" is a panic motivation, not a product strategy. If we rush a half-baked product into a massive market just to plant a flag, we will burn $2.5M in 12 months, fail to hit the $5M ARR target, and lose credibility with the board.

---

3. What’s less scary than it looks

  • The total market size: We don't need 33 million businesses. Look at our UK data: our sweet spot is companies with 10–49 staff (20% of our base, but 71% retention, generating significantly higher revenue) and 50–249 staff (78% retention, $2,400+ ARR). In the US, there are 1.1 million businesses in the 10–249 employee bracket. That is a massive, highly qualified, high-retention target market. We don't need the sole traders; we should actively avoid them.
  • Geographic concentration: We do not need to launch all 50 states on day one. If we pick 3 to 5 high-density, business-friendly states (e.g., Texas, Florida, Washington—states with no state income tax or straightforward payroll setups), we can drastically reduce engineering scope, bypass the 9-month/10-state bottleneck, and launch a compliant MVP much faster.

---

4. What I propose instead

We should go to the board with a revised, credible strategy that protects our capital and targets the segment we actually know how to monetize.

  1. Shift the ICP (Ideal Customer Profile) away from Sole Traders: Ignore the 27.1 million non-employer businesses. Focus exclusively on US small businesses with 10 to 49 employees. They have the budget, they have payroll complexity, and based on our UK experience, they stick around.
  2. Pivoting the Go-To-Market (GTM): Scrap the plan for a 6-person outbound sales team on a $150 product. At this price point, the US motion must be product-led growth (PLG) with digital acquisition, augmented by a lightweight, inside-sales motion only for accounts scaling toward 50+ employees.
  3. De-risk the Engineering Roadmap: Reallocate the $2.5M budget. Instead of a clumsy nationwide Q1 rush, fund a targeted compliance build for just 3 states to launch a closed beta by Q2. Use the remaining budget for localized digital marketing and product-led onboarding flows.
  4. Refrain the Board Narrative: Do not pitch "capturing 1% of a $5B market." Pitch a disciplined land-grab in a profitable sub-segment: “We are entering the US by targeting the 1.1 million businesses with 10–249 employees, starting with a compliant 3-state footprint, leveraging our proven retention playbook to capture high-value employers at a sustainable CAC.”

If we present this, the board will see operators who understand our unit economics, respect regulatory risk, and have a clear path to sustainable growth. Let’s calibrate the deck before Tuesday.

Claims the judge couldn’t find in the brief

  • They are cheap to acquire, high-churn, high-support-load nightmares

Grades and run details

Decision model 80 · LLM judge 45
Decision model checks
  • partialUses the supplied evidence correctly13%
  • passAddresses the actual decision34%
  • passRespects explicit constraints47%
  • partialIdentifies material uncertainty52%
  • partialAvoids unsupported claims49%
  • passProduces the required deliverable68%
  • passRebuilds the market size57%
  • passFinds the timeline that sinks it100%
  • partialTigers, not paper tigers35%
  • passFinds the need behind the idea26%
Run
Run
#1
Time to output
6 s
Submitted
5 Oct 2026

Usable with a quick edit8 of 10 checks passed

The memo is well-reasoned and actionable; the unsupported local-taxes claim is minor and can be quickly removed or hedged without redoing the analysis.

Output

To: CEO and Board From: VP Product Subject: US expansion: approve validation, not the proposed launch

Recommendation: Do not approve the plan as written. The US may be attractive, but the deck mistakes business counts for demand, imports our weakest UK economics, and commits to distribution before we can reliably deliver the product. Approve a staged validation programme instead, with compliance and commercial gates before releasing the full $2.5M.

What the plan gets wrong

1. The addressable market is not 33 million payroll buyers. Of 33.2 million US small businesses, 27.1 million have no employees. Some owner-only businesses may need payroll, but they are not equivalent to employer businesses. The relevant starting population is approximately 6.1 million employers—and probably the 1.1 million with 10–249 employees, given our UK results.

“Even 1%” is not a strategy. It assumes a reachable segment, competitive differentiation, distribution and retention that we have not demonstrated.

2. Our own data contradicts the proposed price and customer strategy.

UK segmentCustomers12-month retentionAnnual revenue/customer
------:---:---:
Sole trader6,10018%$150
2–9 staff5,30033%$150
10–49 staff2,90071%$620
50–249 staff48078%$2,400

Businesses with 10–249 staff represent approximately 23% of customers but 63% of the annual revenue implied by this table. Their weighted retention is approximately 72%, versus 25% for customers below ten staff. Our strongest evidence favours larger employers, not maximum account volume at $150.

Using our UK customer mix within 10–249 staff gives approximately $873 annual revenue per customer. Applied to 1.1 million US businesses, that suggests a roughly $960M segment revenue pool—not a forecast, and not all serviceable by a ten-state product. US willingness to pay remains untested.

There is also a board-data problem: the table implies $4.66M annual revenue, versus the stated $3.9M ARR. Reconcile definitions, dates and cohorts before presenting either figure as the economic baseline.

3. The sales plan and revenue target do not fit. At $150 annually, $5M ARR requires approximately 33,333 active customers: 463 per salesperson per month across six reps over a full year, before churn. Even allocating the entire $2.5M budget to acquisition would allow only $75 per active customer, leaving nothing for engineering, compliance or support.

At the illustrative $873 blend, we still need approximately 5,730 active customers—80 per rep per month. These are workload checks, not sales forecasts, but neither resembles a credible unproven, sales-led launch. A later launch compresses the selling window further.

4. Competitive fundraising is not customer evidence. A competitor’s financing may raise acquisition costs or accelerate competition. It does not establish that our product is ready or that customers will switch. “Get there first” is especially weak when the competitor is already there.

What would sink it

Payroll reliability and coverage. The team estimates nine months for the first ten states. A Q1 launch is unsupported unless substantial work is already complete or we use a proven partner. Ten states also do not automatically mean ten self-contained markets: employee locations, local taxes and multi-state operations affect eligibility. Incorrect wages, filings or remittances can destroy trust faster than sales can rebuild it.

Scaling the wrong economics. At our smallest-customer retention levels, acquisition fills a leaking bucket. Higher retention among larger employers is promising, but customer retention is not revenue retention or gross margin. We lack US acquisition cost, onboarding effort, support burden and contribution-margin evidence.

Starving the UK business. The proposed budget is approximately 64% of current ARR—not a cash-runway calculation, but a material commitment. We need an explicit runway and UK-roadmap impact assessment, including engineering diversion and downside scenarios.

What is less scary than it looks

We do not need nationwide coverage or a large share of the US to build something valuable. A tightly qualified segment can support a meaningful business.

Nor must we necessarily build every payroll component ourselves. A payroll infrastructure partner could shorten time to market, although coverage, margins, accountability and vendor dependence require diligence.

Finally, weak microbusiness retention does not prove Halyard cannot travel. The larger-employer cohorts provide a credible starting hypothesis. They do not yet prove US product-market fit.

What I would propose instead

1. Release up to $400,000 for a 90-day validation phase; hold the remainder. Reconcile the financial baseline, analyse retention by cohort and acquisition channel, and compare partner versus internal-build economics and timelines.

2. Start with 10–49 employees. Recruit 10–20 design partners with simple, geographically supported payroll needs. Validate switching triggers, differentiation, pricing and onboarding. Keep 50–249 as a subsequent hypothesis: better UK retention may come with substantially greater implementation complexity.

3. Gate live payroll on readiness, not the calendar. Require documented coverage, independent compliance review, successful parallel payroll testing, and clear incident ownership. No unsupported employee locations.

4. Run a 30–50-customer paid pilot before hiring six sellers. Use founder-led sales and limited implementation capacity. Measure acquisition cost, onboarding time, support cost, contribution margin and retention. Seek evidence supporting acquisition payback within 12–18 months; early retention is provisional until cohorts mature.

5. Return for scale funding only after those gates. Replace the $5M year-one promise with a bottom-up forecast based on verified price, conversion, launch timing, capacity and churn. The board should fund the next proven step—not an unsupported percentage of America.

Claims the judge couldn’t find in the brief

  • Employee locations, local taxes and multi-state operations affect eligibility.

Grades and run details

Decision model 100 · LLM judge 82
Decision model checks
  • passUses the supplied evidence correctly29%
  • passAddresses the actual decision99%
  • passRespects explicit constraints59%
  • passIdentifies material uncertainty99%
  • passAvoids unsupported claims40%
  • passProduces the required deliverable87%
  • passRebuilds the market size100%
  • passFinds the timeline that sinks it99%
  • passTigers, not paper tigers85%
  • passFinds the need behind the idea44%
Run
Run
#1
Time to output
51 s
Submitted
5 Oct 2026

Needs more than a quick edit8 of 10 checks passed

The 76% revenue share error would require rechecking the data and correcting the figure before use. A product manager could not use it as-is without fixing that factual mistake.

Output

To: CEO and Board From: VP Product Subject: US expansion: approve validation, not the proposed launch

Recommendation: Do not approve the current Q1 launch, six-person sales team, or $5M year-one ARR commitment. The US opportunity deserves investigation, but this plan mistakes a large business population for an addressable payroll market, ignores our strongest customer evidence, and commits to distribution before establishing product readiness.

What the plan gets wrong

1. The market calculation counts the wrong customers at the wrong price.

Of 33.2 million US small businesses, 27.1 million have no employees. They are not a credible default market for an employer payroll product. Some may have relevant needs, but we must establish those rather than count every sole proprietor.

Our UK results argue against prioritizing this population:

SegmentCustomersAnnual revenue/customer12-month retention
------:---:---:
Sole trader6,100$15018%
2–9 staff5,300$15033%
10–49 staff2,900$62071%
50–249 staff480$2,40078%

Customers with 10–249 staff represent roughly 23% of customers but 76% of annual revenue implied by this data. The relevant starting market is the approximately 1.1 million US employers in that size range, narrowed further by geography, payroll complexity, and product fit.

At our UK segment prices, that population represents $682 million–$2.64 billion in hypothetical annual revenue—not a validated TAM. We lack its size mix, US willingness to pay, and the share we could actually serve. “Even 1%” supplies no acquisition strategy.

2. The year-one target has no operating bridge.

At $150 annually, $5 million ARR requires about 33,333 active customers—more than twice our entire UK customer base after five years. At $620, it requires approximately 8,065; at $2,400, 2,083.

With six sellers, the $620 scenario requires each to acquire about 1,344 customers annually before allowing for ramp time or churn. There is no evidence here for the lead volume, conversion, onboarding capacity, pricing, or retention necessary.

The $2.5 million budget is an allocation, not an acquisition model. We need separate estimates for compliance engineering, operations, support, acquisition, and contingency.

3. “Why now” is competitive anxiety, not customer evidence.

A competitor’s fundraising neither proves demand for Halyard nor makes our product ready. Being first matters less than paying employees correctly and retaining employers.

What would sink it

Compliance failure is the existential risk. Payroll mistakes create customer harm, remediation costs, and reputational damage. The team estimates nine months merely to build rules for the first ten states. Unless work is already sufficiently advanced, that contradicts Q1 readiness. Rule implementation also does not establish filing, remittance, testing, exception handling, or operational readiness.

A state-limited launch must constrain employee locations and payroll complexity—not merely customer headquarters. Multi-state employers can breach that boundary immediately.

Retention could make growth economically hollow. Even our best UK segments lose 22–29% of customers by month twelve. We should investigate whether this reflects business closures, product failures, poor fit, or measurement issues. We cannot assume better US performance while adding regulatory and operational complexity.

Premature hiring could consume the budget before learning occurs. Six sellers cannot compensate for an unready product or an unidentified buyer. Low-price segments are particularly unlikely to support conventional sales-led acquisition.

What is less scary than it looks

We do not need nationwide coverage or millions of customers to establish a worthwhile business. A tightly bounded employer segment offers ample room for a meaningful foothold.

Nor must we automatically build the entire US payroll stack ourselves. A payroll infrastructure partner could reduce time and technical risk. It would not remove our obligations to customers, integration risk, or the need to understand liability and unit economics.

Finally, our UK data contains a genuine positive signal: larger employers pay substantially more and retain substantially better. That is a credible hypothesis to test—not proof that it transfers to the US.

What I propose instead

Approve a staged US validation program, with a first tranche capped at $300,000 and further spending conditional on evidence.

  1. Validate the buyer over 8–12 weeks. Start with 10–49-person employers, where we have the strongest combination of customer volume, revenue, and retention. Interview buyers and test paid commitments at US-specific prices. In parallel, diagnose UK churn and compare build-versus-partner economics.
  1. Choose one narrowly serviceable footprint. Select geography based on customer demand and compliance feasibility. Explicitly exclude unsupported employee locations and payroll cases. Do not announce a live-payroll date before readiness is independently reviewed.
  1. Run a paid, founder-led pilot with 20–30 employers. Hire US payroll/compliance expertise before a sales team. Measure payroll accuracy and timeliness, onboarding effort, support costs, willingness to pay, and early retention. Early results will not substitute for twelve-month retention evidence.
  1. Release expansion funding only against gates. Require no unresolved critical payroll defects, demonstrated operational controls, positive contribution economics, and a credible acquisition model targeting under twelve-month CAC payback. Hire sellers only after that motion is repeatable.

Replace $5 million ARR with a conditional first-year objective of 100–200 paying employers after readiness, roughly $62,000–$124,000 annualized at the UK 10–49 price benchmark. That is a learning milestone, not a forecast.

The board should fund the next decision, not an unsupported promise of scale.

Claims the judge couldn’t find in the brief

  • 10-249 staff represent 76% of annual revenue.

Grades and run details

Decision model 100 · LLM judge 73
Decision model checks
  • passUses the supplied evidence correctly50%
  • passAddresses the actual decision99%
  • passRespects explicit constraints55%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims38%
  • passProduces the required deliverable84%
  • passRebuilds the market size82%
  • passFinds the timeline that sinks it100%
  • passTigers, not paper tigers78%
  • passFinds the need behind the idea57%
Run
Run
#1
API response time
51 s
Submitted
6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 5

Uses the supplied evidence correctlyWrongMixedMixed
Gemini 3.5 Flash-Lite · Gemini

The claim that sole traders are 'cheap to acquire' and 'high-support-load' is not supported by the supplied context.

GPT-6 Astra · ChatGPT

Claims that local taxes and multi-state operations affect eligibility are not supported by the supplied context, which only mentions state-by-state tax rules.

GPT-6.1 Sol · API

The memo incorrectly states that 10-249 staff represent 76% of annual revenue; the correct figure from the data is 63%.

Identifies material uncertaintyWrongRightRight
Gemini 3.5 Flash-Lite · Gemini

The memo does not name specific unknowns that could change the decision or say how they would be resolved.

GPT-6 Astra · ChatGPT

Identifies untested US willingness to pay, acquisition costs, and retention, and proposes validation to resolve them.

GPT-6.1 Sol · API

It names unknowns like US willingness to pay, size mix, and retention causes, and ties further funding to evidence gates.

Avoids unsupported claimsWrongMixedMixed
Gemini 3.5 Flash-Lite · Gemini

It presents 'cheap to acquire' and 'high-support-load' as established facts without labelling them as assumptions or hypotheses.

GPT-6 Astra · ChatGPT

Presents the existence of local taxes and multi-state complexity as fact without support from the context.

GPT-6.1 Sol · API

The 76% revenue share is presented as a fact from the data but is incorrect, making it an unsupported claim.

Rebuilds the market sizeMixedRightRight
Gemini 3.5 Flash-Lite · Gemini

It does not rebuild the market to a dollar figure (e.g., under $1B) or show the working; it only states the 1.1M business count without sizing the revenue opportunity.

GPT-6 Astra · ChatGPT

Rebuilds market using employer counts and UK retention, arriving at ~$960M for 1.1M businesses.

GPT-6.1 Sol · API

It rebuilds the market using employer counts and UK retention, arriving at 1.1M businesses and $682M-$2.64B, showing the working.

Finds the need behind the ideaMixedRightRight
Gemini 3.5 Flash-Lite · Gemini

It does not state the underlying need behind the US launch idea (e.g., growth, first-mover advantage) and therefore does not offer an alternative way to meet that need with its cost and trade-offs.

GPT-6 Astra · ChatGPT

Identifies need for US expansion, proposes validation instead of full launch, with cost ($400k) and what is displaced.

GPT-6.1 Sol · API

It identifies the need for a viable US entry, proposes a staged validation alternative, and states what it displaces (the $5M ARR target, Q1 launch, sales team) and its cost ($300k).

All got right 5

Addresses the actual decisionRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The memo commits to a clear alternative strategy (target 10-49 employee businesses, PLG, 3-state launch) early and unambiguously.

GPT-6 Astra · ChatGPT

Commits to not approving the plan and proposes a staged validation, with clear conditions for scaling.

GPT-6.1 Sol · API

The memo clearly recommends not approving the launch and proposes a staged validation program with conditions, framed for the CEO and board.

Respects explicit constraintsRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The output is a memo under 1,000 words, addressed to the CEO, and respects the requested form.

GPT-6 Astra · ChatGPT

Memo format, under 1,000 words, addressed to CEO and board.

GPT-6.1 Sol · API

The output is a memo under 1,000 words, addresses the CEO and board, and respects the requested form.

Produces the required deliverableRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The memo is complete, in the right form, within length, and usable by the CEO with light edits.

GPT-6 Astra · ChatGPT

Complete memo with actionable recommendation, within length, usable with light edits.

GPT-6.1 Sol · API

The memo is complete, within word limit, and directly usable by the CEO and board with a clear recommendation and plan.

Finds the timeline that sinks itRightRightRight
Gemini 3.5 Flash-Lite · Gemini

It names the 9-month build and explicitly states that a Q1 payroll launch is impossible.

GPT-6 Astra · ChatGPT

Cites 9-month build and concludes Q1 launch is unsupported.

GPT-6.1 Sol · API

It names the 9-month build and states it contradicts Q1 readiness, directly linking it to the year-one target.

Tigers, not paper tigersRightRightRight
Gemini 3.5 Flash-Lite · Gemini

It ranks regulatory and reactive strategy as sinkers, dismisses the competitor raise as less scary, and surfaces the need to avoid sole traders.

GPT-6 Astra · ChatGPT

Separates sinkers (payroll reliability, scaling wrong economics) from less scary (competitor raise, nationwide coverage), and surfaces ARR discrepancy.

GPT-6.1 Sol · API

It ranks compliance failure as existential, dismisses the competitor fear as less scary, and surfaces the unspoken need for product readiness.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 76% of 75 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 LunawithAPI100.090.92None
2Haiku 5.5withAPI92.590.92None
3GPT-6.1 SolwithAPI100.086.42None
4GPT-6 AstrawithChatGPT100.079.42None
5Sonnet 5.5withAPI88.374.82None
6Opus 5.5withClaude88.379.421 capped
7Gemini 3.5 Flash-LitewithGemini79.649.721 capped
8Gemini 3.8 FlashwithAPI87.972.721 capped

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.4050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Luna · API: task score 93, the cheapestHaiku 5.5 · API: task score 92, the cheapestGPT-6.1 Sol · API: task score 91, 11× the cheapestGPT-6 Astra · ChatGPT: task score 88, 60× the cheapest (1 of 2 outputs estimated)Sonnet 5.5 · API: task score 78, 37× the cheapestOpus 5.5 · Claude: task score 62, 59× the cheapest (1 of 2 outputs estimated), 1 critical failureGemini 3.5 Flash-Lite · Gemini: task score 50, 1.9× the cheapest (1 of 2 outputs estimated), 1 critical failureGemini 3.8 Flash · API: task score 50, 9.5× the cheapest, 1 critical failureGPT-6 Luna · APIHaiku 5.5 · APIGPT-6.1 Sol · APIGPT-6 Astra · ChatGPTSonnet 5.5 · APIOpus 5.5 · ClaudeGemini 3.5 Flash-Lite · GeminiGemini 3.8 Flash · API
Cost partly estimated (app runs with no usage meter)At least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API931.0×
Haiku 5.5 · API921.4×
Gemini 3.5 Flash-Lite · Gemini501.9× (partly estimated)
Gemini 3.8 Flash · API509.5×
GPT-6.1 Sol · API9111×
Sonnet 5.5 · API7837×
Opus 5.5 · Claude6259× (partly estimated)
GPT-6 Astra · ChatGPT8860× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Pressure-testing a proposal before committing a team to it.

Why it matters

The useful critic finds the one assumption everything rests on. Theatrical negativity is easy to generate and useless in a planning meeting.

What good looks like

  • Identifies the load-bearing assumption
  • Separates the risks that could kill it from the ones that only look scary
  • Uses the supplied evidence, not generic risks
  • Proposes the cheapest way to test the assumption

Deliberately not measured

  • Tone
  • Number of objections raised
Capability tested

Evidence-based critique

The failure we’re looking for

Theatrical negativity without evidence

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

Vanilla prompt (core) · With Roast Me skill · Staff level: a company bet with a long evidence pack