Tasks / Strategy

Challenge an idea

Can the model find the strongest reason an idea may fail, backed by evidence?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 44% were usable with at most a quick edit.

Reliably right

Finds the load-bearing assumption100% pass
The critique targets the assumption that agencies' growth is limited by a shortage of new leads, using referral, abandoned-outbound, and capacity evidence.
GPT-6.1 Sol · API · An AI SDR for small agencies
Uses the interviews faithfully100% pass
All quotes are accurately taken from the transcripts and correctly attributed to A05, A04, and A08.
GPT-6.1 Sol · API · An AI SDR for small agencies
Finds the timeline that sinks it100% pass
Cites 9-month build and concludes Q1 launch is unsupported.
GPT-6 Astra · ChatGPT · Five billion dollars, on paper

Where it slips

Avoids unsupported claims55% pass
Presents the existence of local taxes and multi-state complexity as fact without support from the context.
GPT-6 Astra · ChatGPT · Five billion dollars, on paper
Uses the supplied evidence correctly55% pass
Claims that local taxes and multi-state operations affect eligibility are not supported by the supplied context, which only mentions state-by-state tax rules.
GPT-6 Astra · ChatGPT · Five billion dollars, on paper
Rebuilds the market size81% pass
It uses the $150 price for 10+ staff businesses instead of the higher UK revenue per customer for those segments, so the market size is understated and not correctly rebuilt.
GPT-6 Luna · API · Five billion dollars, on paper

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Finds the need behind the idea
Does the critique identify the underlying need the idea is meant to serve, and offer at least one other way to meet that need, at a different cost, saying what would have to be dropped for it?
Passes when States the need behind the proposal, and gives an alternative way to meet it with its cost and what it displaces.
Gokul RajaramLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Tigers, not paper tigers · and 4 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

We plan to sell an AI sales-development rep, an agent that finds prospects and sends personalised outbound email, to agencies with 5–20 staff. Before we commit a year to it, find the strongest reason this fails. Use the interview material below, including the three transcripts. Write it up for the founding team: the one reason, the evidence for it (quote the interviews), where the evidence cuts the other way, what we would need to see to be proved wrong, and the cheapest test that would show it. Keep it under 600 words.

What the model was given7 items: Scenario, Founders' hypothesis (from our planning doc), Interview summary (14 agencies, 5–20 staff, last six weeks), Interview log, interview-A05-content-agency.md, interview-A04-shopify-agency.md, interview-A08-packaging-studio.md

ScenarioWe are a three-person founding team with £800k of pre-seed funding and about 20 months of runway. Planned price: $600 a month per agency. Two competitors have each raised over $10m in the last year, selling AI SDRs mostly to software companies with 50+ staff.

Founders' hypothesis (from our planning doc)“Small agencies live feast or famine. When a big client leaves they have no pipeline, because the founder is too busy delivering to sell. An AI SDR keeps the pipeline full in the background, so the famine never comes. At $600 a month, one extra client a year pays for it many times over.”

Interview summary (14 agencies, 5–20 staff, last six weeks)11 of 14 get most of their new revenue from referrals and repeat clients (86% on average across those 11). 9 had tried outbound in the last two years; 7 of those stopped within six months, and none of the 7 closed a deal they could attribute to it. The 2 who kept going (A04, A07) have 15–20 staff and a dedicated person for business development. 3 (A08, A12, A14) said they regularly turn work away. 6 described feast-or-famine swings in new business. Average deal size across the 14: $18k; typical sales cycle 6–10 weeks.

Interview logAgency (staff) · services · share of new revenue from referrals and repeat clients · outbound tried? · outcome · representative quote A01 (8) · brand and web studio · 90% · yes, cold-email agency · stopped after 3 months · "We got meetings with people who'd never buy from us." A02 (14) · performance marketing · 85% · yes, LinkedIn automation · stopped after 4 months, account restricted · "LinkedIn shut our founder's account down. That's our best referral network." A03 (6) · PR boutique · 95% · no · — · "Every client we have came from someone vouching for us." A04 (18) · Shopify development · 40% (plus 30% partners, 30% outbound) · yes, in-house BD lead · kept, 30% of new revenue · "It works because of the follow-up, not the first email." A05 (11) · content · 85% · yes, lead-gen agency · stopped after 2 months, no deals · "Content is a trust purchase." A06 (7) · video production · 85% · no · — · "Our clients find us through the videos. Someone shares one and we get a call." A07 (16) · SEO · 45% · yes, outsourced lead generation · kept · "It pays for itself, just. Most of the meetings are a waste of time, but one in ten turns into a retainer." A08 (5) · packaging design · 95% · no · — · "If you sent me ten more leads in October I'd have to say no to nine of them." A09 (12) · paid social · 75% · yes, cold email · stopped after 5 months · "The replies were mostly people asking us to take them off the list." A10 (20) · B2B marketing · 70% · yes, contract SDR · stopped after 6 months · "It cost us about £400 for every meeting, and the meetings didn't close." A11 (9) · WordPress web agency · 90% · yes, cold email · stopped after 3 months · "Our domain ended up on a spam list. Took weeks to fix." A12 (10) · branding · 85% · no · — · "We're booked out till March. I don't need more leads, I need another designer." A13 (15) · HubSpot partner · 48% (plus 40% from HubSpot's partner directory) · yes, cold email · stopped after 4 months · "HubSpot sends us more than we can handle in a good quarter." A14 (6) · UX research · 90% · no · — · "We're small on purpose. We say no to about a third of enquiries."

interview-A05-content-agency.md31 lines · Download

# Interview A05: content agency, 11 staff
Participant: founder and managing director. Interviewer: our co-founder. 34 minutes, lightly edited.

**Interviewer:** Where did your last five clients come from?

**Participant:** Let me think. Two were old clients coming back: one had changed jobs and brought us into her new company. Two were introductions, one from a web agency we partner with and one from a client's CFO who'd seen our work. The fifth found us through a talk I gave at a SaaS marketing meetup. So none of them from anything you'd call outbound.

…

interview-A04-shopify-agency.md23 lines · Download

# Interview A04: Shopify development agency, 18 staff
Participant: head of business development. Interviewer: our co-founder. 29 minutes, lightly edited.

**Interviewer:** Tell me how new business works for you.

**Participant:** We're a bit unusual for an agency our size. I'm a full-time business-development person, and outbound is about 30% of our new revenue. Referrals and repeat work are still the majority, maybe 40% referrals and the rest partners: Shopify's partner directory and a couple of app companies who send us projects.

…

interview-A08-packaging-studio.md23 lines · Download

# Interview A08: packaging design studio, 5 staff
Participant: founder and creative director. Interviewer: our co-founder. 22 minutes, lightly edited.

**Interviewer:** How do new clients find you?

**Participant:** Almost entirely word of mouth. Food and drink brands talk to each other. Someone launches a range, their packaging does well on the shelf, and the next founder asks who did it. I'd say 95% referral. We've never done any outbound.

…

What a strong answer doesThe answer key the graders mark against

Names the load-bearing assumption as 'small agencies' growth is limited by a shortage of new leads': the interviews say it is limited by trust and capacity. 11 of 14 win most work through referrals and repeat clients; 7 of 9 who tried outbound quit within six months with no attributable deals, for reasons tied to trust (A05's 'content is a trust purchase', A01's meetings with people who'd never buy) and reputation risk (A02's restricted LinkedIn account, A11's spam listing); 3 turn work away (A08: 'I'd have to say no to nine of them'). Quotes the interviews accurately and attributes them correctly. Engages the counter-evidence honestly: outbound works for A04 and A07, but both are 15–20 staff with a dedicated BD owner, already use tools like Apollo and Clay, and credit the follow-up rather than the first email, so the segment is small, tooled and not what the product automates. Treats feast or famine as a real pain but notes agencies fix it through past clients and referrals (A05's reactivation emails, and her willingness to pay for help keeping in touch with past clients), pointing at a different product. Says what would prove the challenge wrong (for example, a meaningful share of 10–20-staff agencies with a BD owner and budget, or a concierge outbound test producing qualified meetings that turn into proposals), and designs a cheap test that measures an early signal, since a 6–10 week sales cycle means a six-week test cannot observe won deals.

Critical failures (cap the score at 40)

Invents interview quotes or attributes a quote to the wrong agency
Treats the founders' feast-or-famine hypothesis as confirmed by the interviews

Task

Staff-level · v1.15

The outputs

8 setups answered this brief.

GPT-6.1 SolAPI100Usable with a quick edit GPT-6 LunaAPI100Usable with a quick edit Haiku 5.5API100Usable with a quick edit GPT-6 AstraChatGPT85Needs more work Sonnet 5.5API81Needs more work Gemini 3.8 FlashAPI40Critical failure Opus 5.5Claude40Critical failure Gemini 3.5 Flash-LiteGemini40Critical failure

Haiku 5.5API

Usable with a quick edit12 of 12 checks passed

The memo is accurate, respects the word limit, and directly answers the brief; a PM could use it with light editing and no re-verification.

Output

The strongest reason this fails: agencies don't lack prospects, so cold outbound solves the wrong problem

The reason. Our hypothesis says famine happens because founders are too busy delivering to find prospects. The interviews point to a different cause. Agencies win work through trust, and they run short because of capacity or because nobody remembers to keep in touch. A cold-outbound agent sells the one channel these buyers have already tried and dropped.

Evidence for it

Referrals dominate. 11 of 14 get 70%+ of new revenue from referrals and repeat clients. A03: "Every client we have came from someone vouching for us." A05: "Content is a trust purchase."
Outbound already failed for this segment. 9 of 14 tried it. 7 stopped within six months, and none of those 7 could attribute a closed deal to it. A05: "We didn't close anything from it." A10: "It cost us about £400 for every meeting, and the meetings didn't close."
Some famine is capacity, not pipeline. A08, A12 and A14 turn work away. A08: "If you sent me ten more leads in October I'd have to say no to nine of them." A12: "I don't need more leads, I need another designer."
The famine fix that worked was human. A05 got two clients back within a month by emailing past clients personally. A05 said they'd pay "a couple of hundred a month" for that kind of reminder, but not $600 for cold outbound.
Deliverability and reputation risk. A02's founder account was restricted, and A11's domain ended up on a spam list. A05 would worry about "our name going out on emails I haven't read."

Where the evidence cuts the other way

A04 and A07 kept outbound running and call it worthwhile. A04 gets 30% of new revenue from it, and A07 says "one in ten turns into a retainer."
Those two have 15–20 staff and, in A04's case, a dedicated business-development person. Most of our target (5–20 staff) is smaller, and the segment only gets viable at the top of it.
Six agencies describe feast-or-famine swings. Famine is real. The question is what causes it.
A04 says an AI could cut research time, and A07 says outbound "pays for itself, just." Automation may help the people who already run outbound. It doesn't make the case for everyone else.

What would prove us wrong

In a sample of 20+ agencies with 5–20 staff and no dedicated BD person, at least 1/3 show closed deals attributable to automated outbound within 90 days, at $600/month.
Among agencies that describe famine, the trigger is a pipeline gap (a lost client with nothing behind it), not capacity or a lack of follow-up. We'd see this in their revenue records and their own accounts of what happened.
Paying pilots renew past month three at a rate above 50%, and churn reasons are about results rather than deliverability or trust.

Cheapest test (about one week)

Ask 15 agencies from the 5–20 staff band, chosen to exclude A04/A07-type firms, to pay a $600 deposit for a 60-day outbound pilot. Human approval on every email. Refund if no attributable deal. Also offer, as the second option, a $200/month reminder and reactivation tool for past clients and referrers, since A05 said they'd pay that.

If fewer than 2 of 15 commit to the outbound pilot, and several choose the reactivation option, the product is pointing the wrong way.
If 4 or more commit, run the pilots and measure meetings, deals and renewals against the three tests above.

Grades and run details

Decision model 100 · LLM judge 100

Decision model checks

passUses the supplied evidence correctly73%
passAddresses the actual decision98%
passRespects explicit constraints34%
passIdentifies material uncertainty100%
passAvoids unsupported claims32%
passProduces the required deliverable84%
passFinds the load-bearing assumption100%
passUses the interviews faithfully82%
passEngages the counter-evidence90%
passA cheap test that can actually read out47%
passTigers, not paper tigers97%
passFinds the need behind the idea93%

Run

Run: #1
API response time: 14 s
Submitted: 8 Oct 2026

Check by check

Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Got right 12

Uses the supplied evidence correctlyRight

All claims about the current situation are drawn accurately from the supplied context.

Addresses the actual decisionRight

Output commits to the strongest reason the idea fails, framed for the founding team, and says what would change the answer.

Respects explicit constraintsRight

The memo is under 600 words, written for the founding team, and includes the required elements.

Identifies material uncertaintyRight

Output names specific unknowns (pipeline gap vs capacity/follow-up) and defines what would prove the conclusion wrong.

Avoids unsupported claimsRight

Interpretive causes are attributed to the interviews and not presented as unqualified fact.

Produces the required deliverableRight

The write-up is complete, includes the one reason, evidence, counter-evidence, disconfirming conditions, and a test; a PM could act on it.

Finds the load-bearing assumptionRight

Makes lead-constrained growth the central challenge, supported by referrals, failed outbound, and capacity examples.

Uses the interviews faithfullyRight

All quoted or closely paraphrased material is attributed to the correct agency and matches the transcripts and log.

Engages the counter-evidenceRight

Names A04 and A07, notes they are larger, have dedicated BD owners, and credit follow-up, and tells what that means for the addressable segment.

A cheap test that can actually read outRight

Proposes a cheap, one-week test measuring willingness to pay for a pilot, which is an early signal observable before the 6–10 week sales cycle.

Tigers, not paper tigersRight

Sinks the idea on the core problem (agencies don't lack leads), dismisses the famine-as-pipeline-gap explanation, and surfaces the unasked need for referral reactivation.

Finds the need behind the ideaRight

Identifies the underlying need to keep pipeline full, suggests a $200/month reminder/reactivation tool as an alternative, with what it displaces (cold outbound).

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 76% of 75 checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	GPT-6 LunawithAPI	100.0	90.9	2	None
2	Haiku 5.5withAPI	92.5	90.9	2	None
3	GPT-6.1 SolwithAPI	100.0	86.4	2	None
4	GPT-6 AstrawithChatGPT	100.0	79.4	2	None
5	Sonnet 5.5withAPI	88.3	74.8	2	None
6	Opus 5.5withClaude	88.3	79.4	2	1 capped
7	Gemini 3.5 Flash-LitewithGemini	79.6	49.7	2	1 capped
8	Gemini 3.8 FlashwithAPI	87.9	72.7	2	1 capped

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Cost partly estimated (app runs with no usage meter)At least one critical failure

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
GPT-6 Luna · API	93	1.0×
Haiku 5.5 · API	92	1.4×
Gemini 3.5 Flash-Lite · Gemini	50	1.9× (partly estimated)
Gemini 3.8 Flash · API	50	9.5×
GPT-6.1 Sol · API	91	11×
Sonnet 5.5 · API	78	37×
Opus 5.5 · Claude	62	59× (partly estimated)
GPT-6 Astra · ChatGPT	88	60× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Pressure-testing a proposal before committing a team to it.

Why it matters

The useful critic finds the one assumption everything rests on. Theatrical negativity is easy to generate and useless in a planning meeting.

What good looks like

Identifies the load-bearing assumption
Separates the risks that could kill it from the ones that only look scary
Uses the supplied evidence, not generic risks
Proposes the cheapest way to test the assumption

Deliberately not measured

Tone
Number of objections raised

Capability tested

Evidence-based critique

The failure we’re looking for

Theatrical negativity without evidence

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

Vanilla prompt (core) · With Roast Me skill · Staff level: a company bet with a long evidence pack