Tasks / Strategy

Challenge an idea

Can the model find the strongest reason an idea may fail, backed by evidence?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 44% were usable with at most a quick edit.

Reliably right

  1. Finds the load-bearing assumption100% pass
    The critique targets the assumption that agencies' growth is limited by a shortage of new leads, using referral, abandoned-outbound, and capacity evidence.
    GPT-6.1 Sol · API · An AI SDR for small agencies
  2. Uses the interviews faithfully100% pass
    All quotes are accurately taken from the transcripts and correctly attributed to A05, A04, and A08.
    GPT-6.1 Sol · API · An AI SDR for small agencies
  3. Finds the timeline that sinks it100% pass
    Cites 9-month build and concludes Q1 launch is unsupported.
    GPT-6 Astra · ChatGPT · Five billion dollars, on paper

Where it slips

  1. Avoids unsupported claims55% pass
    Presents the existence of local taxes and multi-state complexity as fact without support from the context.
    GPT-6 Astra · ChatGPT · Five billion dollars, on paper
  2. Uses the supplied evidence correctly55% pass
    Claims that local taxes and multi-state operations affect eligibility are not supported by the supplied context, which only mentions state-by-state tax rules.
    GPT-6 Astra · ChatGPT · Five billion dollars, on paper
  3. Rebuilds the market size81% pass
    It uses the $150 price for 10+ staff businesses instead of the higher UK revenue per customer for those segments, so the market size is understated and not correctly rebuilt.
    GPT-6 Luna · API · Five billion dollars, on paper

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Finds the need behind the idea

    Does the critique identify the underlying need the idea is meant to serve, and offer at least one other way to meet that need, at a different cost, saying what would have to be dropped for it?

    Passes when States the need behind the proposal, and gives an alternative way to meet it with its cost and what it displaces.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Tigers, not paper tigers · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the VP Product at Halyard. Our CEO is taking the US launch plan below to the board next week and has asked you to challenge it hard first. Write a memo of no more than 1,000 words: what the plan gets wrong, what would sink it, what's less scary than it looks, and what you'd propose instead. The deck excerpt and our data are attached.

What the model was given4 items: about_halyard.md, ceo_board_deck_excerpt.md, uk_customers_by_size.csv, us_small_business_stats.md
about_halyard.md3 lines · Download
# About Halyard in the UK

Payroll and HR for small businesses. 14,800 customers, $3.9M ARR, five years old. UK payroll rules are built in; US payroll would need state-by-state tax rules (the team estimates 9 months of work for the first 10 states).
ceo_board_deck_excerpt.md5 lines · Download
# US launch (CEO's board deck, excerpt)

- Market: 33 million US small businesses x our UK price of £120 ($150) a year = about $5B. "Even 1% is $50M."
- Plan: launch in the US in Q1, $2.5M budget, hire a US sales team of 6, target $5M ARR in year one.
- Why now: a competitor raised in the US; we need to get there first.
uk_customers_by_size.csv4 rows · Download
company_size,uk_customers,share_of_uk_customers_pct,retained_at_12_months_pct,avg_revenue_usd_per_year
Sole trader,6100,41,18,150
2-9 staff,5300,36,33,150
10-49 staff,2900,20,71,620
50-249 staff,480,3,78,2400
us_small_business_stats.md5 lines · Download
# US small businesses (government statistics)

- 33.2 million small businesses in total.
- About 27.1 million have no employees (sole proprietors).
- About 6.1 million have employees; of those, about 1.1 million have 10 to 249 employees.
What a strong answer doesThe answer key the graders mark against

Shows the $5B market is the wrong number. It multiplies every US small business by the UK price, but in the UK only businesses with 10+ staff stay (71% and 78% at 12 months) and pay more ($620 to $2,400 a year); sole traders and 2-9 businesses mostly leave (18% and 33%). Most US small businesses (27.1M of 33.2M) have no employees and don't need payroll at all; the comparable US market is about 1.1M businesses with 10 to 249 staff, worth roughly $0.7B to $1B at UK revenue per customer, not $5B. The tiger: US payroll needs state-by-state tax rules, about 9 months for the first 10 states, so a Q1 launch with payroll isn't possible and $5M ARR in year one would need thousands of 10+ staff customers from a standing start. Dismisses the 'first mover' worry as less scary: a competitor's raise doesn't make a market winner-takes-all. Proposes instead: target 10+ staff businesses, start with HR in the US while payroll is built for a few high-density states, test demand with a small budget and a clear threshold before hiring a sales team, and fix the UK mix towards 10+ staff companies meanwhile.

Critical failures (cap the score at 40)
  • Endorses the $5B market size
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Needs more than a quick edit8 of 10 checks passed

The output includes unsupported claims about US payroll rules (federal/local) and market stickiness that a PM would need to verify before using, so it cannot be used without re-checking evidence.

Output

MEMO: US launch plan, challenge review To: CEO | From: VP Product

Bottom line: The direction is defensible, but the plan isn't ready for the board. The market is about a fifth the size the deck claims, the Q1 launch date conflicts with our own engineering estimate, and the $5M year-one target is unreachable. I'd take a staged version instead.

---

What the plan gets wrong

1. The $5B market is wrong by roughly 5x. The deck multiplies all 33.2M US small businesses by $150. But 27.1M of them have no employees, so they have no payroll to run. Our own UK data shows how little this segment is worth to us:

  • Sole traders are 41% of UK customers but only about 20% of revenue, and just 18% are still with us at 12 months.
  • Firms with 10+ staff are 23% of customers but about 63% of revenue, with 71–78% retention.

Our proven segment in the US is the 1.1M firms with 10–249 employees. At our UK blended revenue for that group (about $870 per customer), that's roughly $1B. "Even 1%" is about $10M, not $50M. I'd exclude the ~5M micro-employers. In the UK, 2–9 staff firms pay $150 and only 33% stay a year.

2. The timeline contradicts itself. The team estimates 9 months of work for the first 10 states. A Q1 launch means selling a product that doesn't exist, or one that only works in a few states. Ten states is also only about half the US population, by my rough estimate, so a national sales team would be pitching to prospects we can't serve.

3. $5M ARR in year one is out of reach. It exceeds the ARR we built in five years in the UK. At ~$870 per customer it needs about 5,700 customers, roughly 950 per rep from a standing start. With a 9-month build, year-one revenue would be close to zero.

4. "A competitor raised" isn't a reason to launch now. We can't win a spending race with $2.5M against a funded rival. Payroll is also a sticky, slow-switching category, so launching early earns us little. Launching with wrong tax calculations would cost us far more.

What would sink it

  • Compliance failures. A wrong withholding or a missed filing means customers pay penalties and blame us. In the UK, rules are built in and we know them. In the US we'd face federal, state and local rules, each with its own filings. One bad quarter in the first cohort could end our reputation with the segment we need.
  • Targeting the wrong customer. If the sales team chases volume, it will sign small firms that churn at UK micro-segment rates. Acquisition cost would never pay back.
  • Draining the UK business. Nine months of engineering on tax rules comes out of the roadmap for the business that funds everything. UK retention in the smallest segments is already weak, and we'd be neglecting it.
  • Burning the budget before learning anything. Six hires plus a 9-month build could consume most of $2.5M, about 64% of current ARR, before we know whether US customers retain like UK ones. I need to confirm whether the $2.5M includes the engineering cost. If it doesn't, the real ask is larger.
  • Missing the target. A $5M goal that slips will damage board confidence in everything else we present.

What's less scary than it looks

  • The smaller market is still big. About $1B is over 200x our current ARR. We don't need $5B to justify the US.
  • The competitor raise. It validates demand, but funding doesn't fix the state-by-state build, which slows them as well. Nothing here is lost by launching in 12–18 months instead of 3.
  • The 9-month estimate. It's a well-scoped build with a known end date, and a tax-calculation partner or API might shorten it. I'd evaluate build vs. buy before committing.
  • UK retention in micro-segments. It looks alarming, but it's a UK issue we'd avoid importing by targeting 10–249 staff firms, where our retention is strongest.

What I'd propose instead

Phase 0 (now to ~Q3, ~$0.5–0.7M): - Decide build vs. buy for the tax engine. - Pick 3–4 launch states by density of 10–249 staff firms. - Recruit 15–20 design-partner customers. - Hire a US head of sales/GTM and one compliance specialist, not six reps.

Phase 1 (launch in 3–5 states once compliance is proven, ~$1M): - Sell only to 10–249 staff firms with 2 reps and dedicated onboarding. - Set the target at about 300 customers and ~$0.25–0.3M ARR twelve months after launch. That's my estimate, based on UK segment pricing.

Phase 2 (release the remaining ~$1M, expand states and team) only if we hit gates: - 12-month retention of at least 70% in the first cohort. - Zero material filing errors. - Acquisition cost paying back within an agreed window.

This puts about $0.5–0.7M at risk before we learn anything, rather than $2.5M, and it gives the board a credible plan with checkpoints.

Before the board meeting

  1. Reconcile ARR. Our segment table implies about $4.66M (customers × average revenue), but the deck and company profile say $3.9M. A board member will do this multiplication. We need to know whether the gap is discounting, churn timing, or stale data.
  2. Confirm what the $2.5M covers, particularly engineering and compliance.
  3. Restate the market as ~$1B core, with the logic shown.
  4. Check price transfer. UK per-segment pricing may not hold in the US, so the design-partner phase should test willingness to pay.

I'm happy to walk through the numbers before you finalize the deck.

Grades and run details

Decision model 85 · LLM judge 73
Decision model checks
  • failUses the supplied evidence correctly23%
  • passAddresses the actual decision81%
  • passRespects explicit constraints59%
  • passIdentifies material uncertainty99%
  • partialAvoids unsupported claims68%
  • passProduces the required deliverable82%
  • passRebuilds the market size99%
  • passFinds the timeline that sinks it100%
  • passTigers, not paper tigers78%
  • passFinds the need behind the idea30%
Run
Run
#1
API response time
46 s
Submitted
6 Oct 2026

Needs more than a quick edit8 of 10 checks passed

The 76% revenue share error would require rechecking the data and correcting the figure before use. A product manager could not use it as-is without fixing that factual mistake.

Output

To: CEO and Board From: VP Product Subject: US expansion: approve validation, not the proposed launch

Recommendation: Do not approve the current Q1 launch, six-person sales team, or $5M year-one ARR commitment. The US opportunity deserves investigation, but this plan mistakes a large business population for an addressable payroll market, ignores our strongest customer evidence, and commits to distribution before establishing product readiness.

What the plan gets wrong

1. The market calculation counts the wrong customers at the wrong price.

Of 33.2 million US small businesses, 27.1 million have no employees. They are not a credible default market for an employer payroll product. Some may have relevant needs, but we must establish those rather than count every sole proprietor.

Our UK results argue against prioritizing this population:

SegmentCustomersAnnual revenue/customer12-month retention
------:---:---:
Sole trader6,100$15018%
2–9 staff5,300$15033%
10–49 staff2,900$62071%
50–249 staff480$2,40078%

Customers with 10–249 staff represent roughly 23% of customers but 76% of annual revenue implied by this data. The relevant starting market is the approximately 1.1 million US employers in that size range, narrowed further by geography, payroll complexity, and product fit.

At our UK segment prices, that population represents $682 million–$2.64 billion in hypothetical annual revenue—not a validated TAM. We lack its size mix, US willingness to pay, and the share we could actually serve. “Even 1%” supplies no acquisition strategy.

2. The year-one target has no operating bridge.

At $150 annually, $5 million ARR requires about 33,333 active customers—more than twice our entire UK customer base after five years. At $620, it requires approximately 8,065; at $2,400, 2,083.

With six sellers, the $620 scenario requires each to acquire about 1,344 customers annually before allowing for ramp time or churn. There is no evidence here for the lead volume, conversion, onboarding capacity, pricing, or retention necessary.

The $2.5 million budget is an allocation, not an acquisition model. We need separate estimates for compliance engineering, operations, support, acquisition, and contingency.

3. “Why now” is competitive anxiety, not customer evidence.

A competitor’s fundraising neither proves demand for Halyard nor makes our product ready. Being first matters less than paying employees correctly and retaining employers.

What would sink it

Compliance failure is the existential risk. Payroll mistakes create customer harm, remediation costs, and reputational damage. The team estimates nine months merely to build rules for the first ten states. Unless work is already sufficiently advanced, that contradicts Q1 readiness. Rule implementation also does not establish filing, remittance, testing, exception handling, or operational readiness.

A state-limited launch must constrain employee locations and payroll complexity—not merely customer headquarters. Multi-state employers can breach that boundary immediately.

Retention could make growth economically hollow. Even our best UK segments lose 22–29% of customers by month twelve. We should investigate whether this reflects business closures, product failures, poor fit, or measurement issues. We cannot assume better US performance while adding regulatory and operational complexity.

Premature hiring could consume the budget before learning occurs. Six sellers cannot compensate for an unready product or an unidentified buyer. Low-price segments are particularly unlikely to support conventional sales-led acquisition.

What is less scary than it looks

We do not need nationwide coverage or millions of customers to establish a worthwhile business. A tightly bounded employer segment offers ample room for a meaningful foothold.

Nor must we automatically build the entire US payroll stack ourselves. A payroll infrastructure partner could reduce time and technical risk. It would not remove our obligations to customers, integration risk, or the need to understand liability and unit economics.

Finally, our UK data contains a genuine positive signal: larger employers pay substantially more and retain substantially better. That is a credible hypothesis to test—not proof that it transfers to the US.

What I propose instead

Approve a staged US validation program, with a first tranche capped at $300,000 and further spending conditional on evidence.

  1. Validate the buyer over 8–12 weeks. Start with 10–49-person employers, where we have the strongest combination of customer volume, revenue, and retention. Interview buyers and test paid commitments at US-specific prices. In parallel, diagnose UK churn and compare build-versus-partner economics.
  1. Choose one narrowly serviceable footprint. Select geography based on customer demand and compliance feasibility. Explicitly exclude unsupported employee locations and payroll cases. Do not announce a live-payroll date before readiness is independently reviewed.
  1. Run a paid, founder-led pilot with 20–30 employers. Hire US payroll/compliance expertise before a sales team. Measure payroll accuracy and timeliness, onboarding effort, support costs, willingness to pay, and early retention. Early results will not substitute for twelve-month retention evidence.
  1. Release expansion funding only against gates. Require no unresolved critical payroll defects, demonstrated operational controls, positive contribution economics, and a credible acquisition model targeting under twelve-month CAC payback. Hire sellers only after that motion is repeatable.

Replace $5 million ARR with a conditional first-year objective of 100–200 paying employers after readiness, roughly $62,000–$124,000 annualized at the UK 10–49 price benchmark. That is a learning milestone, not a forecast.

The board should fund the next decision, not an unsupported promise of scale.

Claims the judge couldn’t find in the brief

  • 10-249 staff represent 76% of annual revenue.

Grades and run details

Decision model 100 · LLM judge 73
Decision model checks
  • passUses the supplied evidence correctly50%
  • passAddresses the actual decision99%
  • passRespects explicit constraints55%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims38%
  • passProduces the required deliverable84%
  • passRebuilds the market size82%
  • passFinds the timeline that sinks it100%
  • passTigers, not paper tigers78%
  • passFinds the need behind the idea57%
Run
Run
#1
API response time
51 s
Submitted
6 Oct 2026

Usable with a quick edit8 of 10 checks passed

The memo is well-structured and evidence-based; the one unsupported claim about incumbents can be removed with a quick edit without affecting the core reasoning.

Output

MEMO

To: CEO From: VP Product Re: US launch plan: challenge before the board

You asked me to push hard, so I have. I think we should go to the US, but not with this plan. As written, the board will find the holes in it quickly, and the market will find the rest.

---

1. What the plan gets wrong

The market size counts businesses that won't buy payroll. Of the 33.2M US small businesses, 27.1M have no employees. A business with no staff has little use for payroll. Our own UK data points the same way: sole traders are 41% of our customers, but only 18% are still with us after 12 months.

The market that fits our product is the 1.1M US businesses with 10–249 employees. In the UK that segment:

  • makes up 23% of our customers but about 63% of our revenue;
  • keeps 71–78% of customers at 12 months, against 18–33% for smaller firms;
  • pays a blended average of about $873 a year, not $150.

That puts the core US opportunity at about 1.1M × $873 ≈ $1B. The 5M employers with 1–9 staff add up to about $750M more at our current price, but UK retention suggests most of that revenue would leave within a year. The real figure is roughly $1–1.7B, not $5B. That is still huge compared with our $3.9M ARR. We simply don't need the inflated number, and leading with it costs us credibility.

The timeline is impossible. The deck says we launch in Q1. Engineering estimates 9 months to build tax rules for the first 10 states. We cannot sell US payroll before it calculates US payroll taxes. Realistically, the earliest launch is Q4, and only in 10 states.

The $5M year-one target fails on its own arithmetic. It took us five years to reach $3.9M in the UK.

Price assumedCustomers needed for $5MDeals per rep (6 reps)
$150 (deck price)~33,000 (more than double our UK base)~5,500
$873 (mid-market blend)~5,700~950, roughly 4 per working day

At the mid-market price, each rep would need about 950 deals. These are considered purchases, and that volume isn't plausible. It is even less plausible when the product is only live for the last quarter of the year.

The budget is spent in the wrong order. If we hire six salespeople in Q1, we pay them for about nine months while they have nothing to sell. Much of the $2.5M would go on salaries before the product exists.

---

2. What would sink us

  1. Payroll tax errors. Getting someone's pay or taxes wrong is not an ordinary bug. It creates penalties for our customers, refunds for us, and reputational damage in a market where nobody knows us yet. Rushing the tax engine to hit a sales date is the single biggest risk in this plan.
  2. Selling to the wrong customers. If US sales chases volume at $150 among micro-businesses, UK data says we'd lose 67–82% of them within a year. We would be paying to acquire customers who leave, and the board would see revenue growth that isn't real.
  3. Ten states may not cover our best customers. Companies with 10–249 staff are more likely to have employees in several states. If a prospect has people in a state we don't support, we lose the deal. We need data on how many states typical target customers operate in before we finalise the list.
  4. Missing a target we set ourselves. If we commit to $5M and deliver a fraction of it, we lose the board's trust for the next fundraise. That is worse than setting a modest target and beating it.
  5. Neglecting the UK. The UK business funds this expansion. Moving our best people to the US without a plan to protect UK growth puts our cash engine at risk.

---

3. What's less scary than it looks

  • The competitor's raise. The US payroll market is not empty. Well-funded incumbents already exist, so "getting there first" isn't possible for anyone. Another entrant raising money doesn't change our costs, our product gaps or our timing. It is a poor reason to launch before we're ready.
  • The 9-month build. It's real work, but it's bounded. We don't need 50 states on day one. We need the 10 states with the most 10–249-employee businesses. We should also assess licensing a US payroll tax engine instead of building one. That could shorten the timeline and reduce compliance risk, and we should know within a month whether it's viable.
  • The smaller market size. A $1B core market means we need about 0.4% of it to triple the company. That is a stronger story than "1% of $5B," because it rests on segment data and retention we can prove.
  • Our readiness. 71–78% retention in the UK mid-market is real evidence that the product works for these companies. We're not starting from zero. We're moving a proven product into a new market.

---

4. What I'd propose instead

Position: the US launch targets employers with 10–249 staff, priced like our UK mid-market (around $600–2,400 a year, validated with US prospects), not £120.

Phases, with spending released at each milestone:

PhaseTimingWorkSpend
0. DecideMonth 1Build vs. license the tax engine; pick the 10 states by target-business density; interview 30+ US prospects on pricingLow
1. BuildMonths 1–9Tax engine, compliance review, US payroll accuracy testingMost of the early budget goes to engineering and compliance
2. Design partnersMonths 7–1020–30 paying US customers on supported states, with close support1–2 US sales/solutions hires
3. Scale salesMonths 10+Hire toward 6 reps only if design partners hit accuracy and retention barsRest of budget, released by board milestone

Year-one targets, replacing $5M ARR: - payroll runs with no tax-filing errors; - 20–30 design partners, then a few hundred paying customers by year end (roughly $0.2–0.4M ARR at mid-market prices); - at least 70% of US customers staying through early renewal checkpoints, matching the UK mid-market; - evidence on deal size and sales cycle, so a year-two ARR target is grounded in data.

In parallel: keep investing in UK customers with 10–249 staff. They are our highest-value, highest-retention segment, and they fund all of this.

Before the board meeting, two housekeeping items: - Our segment table adds up to about $4.7M in revenue, but we report $3.9M ARR. We should reconcile this, likely churn timing or discounting, before the board asks. - We should remove the "1% of $5B" line entirely.

---

Bottom line: the US is the right ambition, but this plan has the wrong market size, the wrong customer, the wrong timeline and the wrong sequence. If we fix those four things, we go to the board with a plan they can approve and we can deliver.

Claims the judge couldn’t find in the brief

  • Well-funded incumbents already exist.

Grades and run details

Decision model 85 · LLM judge 82
Decision model checks
  • failUses the supplied evidence correctly4%
  • passAddresses the actual decision87%
  • passRespects explicit constraints38%
  • passIdentifies material uncertainty98%
  • partialAvoids unsupported claims64%
  • passProduces the required deliverable87%
  • passRebuilds the market size98%
  • passFinds the timeline that sinks it100%
  • passTigers, not paper tigers82%
  • passFinds the need behind the idea37%
Run
Run
#1
Time to output
46 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 2

Uses the supplied evidence correctlyWrongMixedWrong
Sonnet 5.5 · API

The output claims US payroll involves 'federal, state and local rules' but the supplied context only mentions state-by-state tax rules, and it presents 'payroll is a sticky, slow-switching category' as fact without support.

GPT-6.1 Sol · API

The memo incorrectly states that 10-249 staff represent 76% of annual revenue; the correct figure from the data is 63%.

Opus 5.5 · Claude

The claim that well-funded incumbents already exist is not in the supplied context and is not derivable from it.

Avoids unsupported claimsWrongMixedWrong
Sonnet 5.5 · API

It presents 'payroll is a sticky, slow-switching category' as an established fact, and 'federal, state and local rules' as a given, neither of which is in the supplied evidence.

GPT-6.1 Sol · API

The 76% revenue share is presented as a fact from the data but is incorrect, making it an unsupported claim.

Opus 5.5 · Claude

The statement 'Well-funded incumbents already exist' is presented as fact without support from the supplied evidence.

All got right 8

Addresses the actual decisionRightRightRight
Sonnet 5.5 · API

The memo commits to a staged alternative early, frames it for the CEO, and specifies gates that would change the call.

GPT-6.1 Sol · API

The memo clearly recommends not approving the launch and proposes a staged validation program with conditions, framed for the CEO and board.

Opus 5.5 · Claude

The memo commits to going to the US with a revised plan, framed for the CEO, and implies conditions (design partner milestones) that would change the call.

Respects explicit constraintsRightRightRight
Sonnet 5.5 · API

The output is a memo under 1,000 words addressed to the CEO, respecting the form and length constraint.

GPT-6.1 Sol · API

The output is a memo under 1,000 words, addresses the CEO and board, and respects the requested form.

Opus 5.5 · Claude

The output is a memo to the CEO, appears under 1,000 words, and addresses the requested challenge.

Identifies material uncertaintyRightRightRight
Sonnet 5.5 · API

It names specific unknowns (budget coverage, ARR discrepancy, US pricing transfer) and says how to resolve them before the board meeting.

GPT-6.1 Sol · API

It names unknowns like US willingness to pay, size mix, and retention causes, and ties further funding to evidence gates.

Opus 5.5 · Claude

It names unknowns (state coverage needs, build vs. license viability, deal size) and says how to resolve them (prospect interviews, design partner testing, licensing assessment within a month).

Produces the required deliverableRightRightRight
Sonnet 5.5 · API

The memo is complete, in the right form for the CEO, within the word limit, and actionable with light edits.

GPT-6.1 Sol · API

The memo is complete, within word limit, and directly usable by the CEO and board with a clear recommendation and plan.

Opus 5.5 · Claude

The memo is complete, in the requested form, for the CEO, and actionable with light edits.

Rebuilds the market sizeRightRightRight
Sonnet 5.5 · API

It rebuilds the market using employer counts and UK retention data, arriving at ~$1B for the 1.1M 10-249 employee businesses, showing the working.

GPT-6.1 Sol · API

It rebuilds the market using employer counts and UK retention, arriving at 1.1M businesses and $682M-$2.64B, showing the working.

Opus 5.5 · Claude

It rebuilds the market using employer counts and UK retention data, arriving at about 1.1M businesses and under $1B, showing the working.

Finds the timeline that sinks itRightRightRight
Sonnet 5.5 · API

It names the 9-month state tax build and explains that a Q1 launch is impossible and the $5M year-one target unreachable.

GPT-6.1 Sol · API

It names the 9-month build and states it contradicts Q1 readiness, directly linking it to the year-one target.

Opus 5.5 · Claude

It names the 9-month build and concludes a Q1 payroll launch is impossible, with earliest launch Q4.

Tigers, not paper tigersRightRightRight
Sonnet 5.5 · API

It ranks risks, dismisses the competitor-raise worry as less scary because funding doesn't speed the state build, and surfaces the ARR discrepancy as an unspoken issue.

GPT-6.1 Sol · API

It ranks compliance failure as existential, dismisses the competitor fear as less scary, and surfaces the unspoken need for product readiness.

Opus 5.5 · Claude

It dismisses the competitor's raise as less scary with reasoning, ranks payroll tax errors as the top risk, and surfaces the UK neglect and revenue discrepancy.

Finds the need behind the ideaRightRightRight
Sonnet 5.5 · API

It identifies the need for US growth and offers a phased alternative with lower cost and risk, stating what it displaces (the aggressive Q1 launch and $5M target).

GPT-6.1 Sol · API

It identifies the need for a viable US entry, proposes a staged validation alternative, and states what it displaces (the $5M ARR target, Q1 launch, sales team) and its cost ($300k).

Opus 5.5 · Claude

It states the need (US expansion for growth) and offers an alternative (target 10+ staff, phased build, test demand) with its cost (slower revenue, smaller initial target) and what it displaces (the $5M year-one goal and Q1 launch).

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 76% of 75 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 LunawithAPI100.090.92None
2Haiku 5.5withAPI92.590.92None
3GPT-6.1 SolwithAPI100.086.42None
4GPT-6 AstrawithChatGPT100.079.42None
5Sonnet 5.5withAPI88.374.82None
6Opus 5.5withClaude88.379.421 capped
7Gemini 3.5 Flash-LitewithGemini79.649.721 capped
8Gemini 3.8 FlashwithAPI87.972.721 capped

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.4050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Luna · API: task score 93, the cheapestHaiku 5.5 · API: task score 92, the cheapestGPT-6.1 Sol · API: task score 91, 11× the cheapestGPT-6 Astra · ChatGPT: task score 88, 60× the cheapest (1 of 2 outputs estimated)Sonnet 5.5 · API: task score 78, 37× the cheapestOpus 5.5 · Claude: task score 62, 59× the cheapest (1 of 2 outputs estimated), 1 critical failureGemini 3.5 Flash-Lite · Gemini: task score 50, 1.9× the cheapest (1 of 2 outputs estimated), 1 critical failureGemini 3.8 Flash · API: task score 50, 9.5× the cheapest, 1 critical failureGPT-6 Luna · APIHaiku 5.5 · APIGPT-6.1 Sol · APIGPT-6 Astra · ChatGPTSonnet 5.5 · APIOpus 5.5 · ClaudeGemini 3.5 Flash-Lite · GeminiGemini 3.8 Flash · API
Cost partly estimated (app runs with no usage meter)At least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API931.0×
Haiku 5.5 · API921.4×
Gemini 3.5 Flash-Lite · Gemini501.9× (partly estimated)
Gemini 3.8 Flash · API509.5×
GPT-6.1 Sol · API9111×
Sonnet 5.5 · API7837×
Opus 5.5 · Claude6259× (partly estimated)
GPT-6 Astra · ChatGPT8860× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Pressure-testing a proposal before committing a team to it.

Why it matters

The useful critic finds the one assumption everything rests on. Theatrical negativity is easy to generate and useless in a planning meeting.

What good looks like

  • Identifies the load-bearing assumption
  • Separates the risks that could kill it from the ones that only look scary
  • Uses the supplied evidence, not generic risks
  • Proposes the cheapest way to test the assumption

Deliberately not measured

  • Tone
  • Number of objections raised
Capability tested

Evidence-based critique

The failure we’re looking for

Theatrical negativity without evidence

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

Vanilla prompt (core) · With Roast Me skill · Staff level: a company bet with a long evidence pack