Tasks / Strategy

Challenge an idea

Can the model find the strongest reason an idea may fail, backed by evidence?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 44% were usable with at most a quick edit.

Reliably right

Finds the load-bearing assumption100% pass
The critique targets the assumption that agencies' growth is limited by a shortage of new leads, using referral, abandoned-outbound, and capacity evidence.
GPT-6.1 Sol · API · An AI SDR for small agencies
Uses the interviews faithfully100% pass
All quotes are accurately taken from the transcripts and correctly attributed to A05, A04, and A08.
GPT-6.1 Sol · API · An AI SDR for small agencies
Finds the timeline that sinks it100% pass
Cites 9-month build and concludes Q1 launch is unsupported.
GPT-6 Astra · ChatGPT · Five billion dollars, on paper

Where it slips

Uses the supplied evidence correctly50% pass
Claims that local taxes and multi-state operations affect eligibility are not supported by the supplied context, which only mentions state-by-state tax rules.
GPT-6 Astra · ChatGPT · Five billion dollars, on paper
Avoids unsupported claims56% pass
Presents the existence of local taxes and multi-state complexity as fact without support from the context.
GPT-6 Astra · ChatGPT · Five billion dollars, on paper
Finds the need behind the idea71% pass
It mentions A05's past-client reminder idea but does not state the underlying need or offer a full alternative with its cost and what it displaces.
GPT-6 Astra · ChatGPT · An AI SDR for small agencies

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Finds the need behind the idea
Does the critique identify the underlying need the idea is meant to serve, and offer at least one other way to meet that need, at a different cost, saying what would have to be dropped for it?
Passes when States the need behind the proposal, and gives an alternative way to meet it with its cost and what it displaces.
Gokul RajaramLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Tigers, not paper tigers · and 4 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

We plan to sell an AI sales-development rep, an agent that finds prospects and sends personalised outbound email, to agencies with 5–20 staff. Before we commit a year to it, find the strongest reason this fails. Use the interview material below, including the three transcripts. Write it up for the founding team: the one reason, the evidence for it (quote the interviews), where the evidence cuts the other way, what we would need to see to be proved wrong, and the cheapest test that would show it. Keep it under 600 words.

What the model was given7 items: Scenario, Founders' hypothesis (from our planning doc), Interview summary (14 agencies, 5–20 staff, last six weeks), Interview log, interview-A05-content-agency.md, interview-A04-shopify-agency.md, interview-A08-packaging-studio.md

ScenarioWe are a three-person founding team with £800k of pre-seed funding and about 20 months of runway. Planned price: $600 a month per agency. Two competitors have each raised over $10m in the last year, selling AI SDRs mostly to software companies with 50+ staff.

Founders' hypothesis (from our planning doc)“Small agencies live feast or famine. When a big client leaves they have no pipeline, because the founder is too busy delivering to sell. An AI SDR keeps the pipeline full in the background, so the famine never comes. At $600 a month, one extra client a year pays for it many times over.”

Interview summary (14 agencies, 5–20 staff, last six weeks)11 of 14 get most of their new revenue from referrals and repeat clients (86% on average across those 11). 9 had tried outbound in the last two years; 7 of those stopped within six months, and none of the 7 closed a deal they could attribute to it. The 2 who kept going (A04, A07) have 15–20 staff and a dedicated person for business development. 3 (A08, A12, A14) said they regularly turn work away. 6 described feast-or-famine swings in new business. Average deal size across the 14: $18k; typical sales cycle 6–10 weeks.

Interview logAgency (staff) · services · share of new revenue from referrals and repeat clients · outbound tried? · outcome · representative quote A01 (8) · brand and web studio · 90% · yes, cold-email agency · stopped after 3 months · "We got meetings with people who'd never buy from us." A02 (14) · performance marketing · 85% · yes, LinkedIn automation · stopped after 4 months, account restricted · "LinkedIn shut our founder's account down. That's our best referral network." A03 (6) · PR boutique · 95% · no · — · "Every client we have came from someone vouching for us." A04 (18) · Shopify development · 40% (plus 30% partners, 30% outbound) · yes, in-house BD lead · kept, 30% of new revenue · "It works because of the follow-up, not the first email." A05 (11) · content · 85% · yes, lead-gen agency · stopped after 2 months, no deals · "Content is a trust purchase." A06 (7) · video production · 85% · no · — · "Our clients find us through the videos. Someone shares one and we get a call." A07 (16) · SEO · 45% · yes, outsourced lead generation · kept · "It pays for itself, just. Most of the meetings are a waste of time, but one in ten turns into a retainer." A08 (5) · packaging design · 95% · no · — · "If you sent me ten more leads in October I'd have to say no to nine of them." A09 (12) · paid social · 75% · yes, cold email · stopped after 5 months · "The replies were mostly people asking us to take them off the list." A10 (20) · B2B marketing · 70% · yes, contract SDR · stopped after 6 months · "It cost us about £400 for every meeting, and the meetings didn't close." A11 (9) · WordPress web agency · 90% · yes, cold email · stopped after 3 months · "Our domain ended up on a spam list. Took weeks to fix." A12 (10) · branding · 85% · no · — · "We're booked out till March. I don't need more leads, I need another designer." A13 (15) · HubSpot partner · 48% (plus 40% from HubSpot's partner directory) · yes, cold email · stopped after 4 months · "HubSpot sends us more than we can handle in a good quarter." A14 (6) · UX research · 90% · no · — · "We're small on purpose. We say no to about a third of enquiries."

interview-A05-content-agency.md31 lines · Download

# Interview A05: content agency, 11 staff
Participant: founder and managing director. Interviewer: our co-founder. 34 minutes, lightly edited.

**Interviewer:** Where did your last five clients come from?

**Participant:** Let me think. Two were old clients coming back: one had changed jobs and brought us into her new company. Two were introductions, one from a web agency we partner with and one from a client's CFO who'd seen our work. The fifth found us through a talk I gave at a SaaS marketing meetup. So none of them from anything you'd call outbound.

…

interview-A04-shopify-agency.md23 lines · Download

# Interview A04: Shopify development agency, 18 staff
Participant: head of business development. Interviewer: our co-founder. 29 minutes, lightly edited.

**Interviewer:** Tell me how new business works for you.

**Participant:** We're a bit unusual for an agency our size. I'm a full-time business-development person, and outbound is about 30% of our new revenue. Referrals and repeat work are still the majority, maybe 40% referrals and the rest partners: Shopify's partner directory and a couple of app companies who send us projects.

…

interview-A08-packaging-studio.md23 lines · Download

# Interview A08: packaging design studio, 5 staff
Participant: founder and creative director. Interviewer: our co-founder. 22 minutes, lightly edited.

**Interviewer:** How do new clients find you?

**Participant:** Almost entirely word of mouth. Food and drink brands talk to each other. Someone launches a range, their packaging does well on the shelf, and the next founder asks who did it. I'd say 95% referral. We've never done any outbound.

…

What a strong answer doesThe answer key the graders mark against

Names the load-bearing assumption as 'small agencies' growth is limited by a shortage of new leads': the interviews say it is limited by trust and capacity. 11 of 14 win most work through referrals and repeat clients; 7 of 9 who tried outbound quit within six months with no attributable deals, for reasons tied to trust (A05's 'content is a trust purchase', A01's meetings with people who'd never buy) and reputation risk (A02's restricted LinkedIn account, A11's spam listing); 3 turn work away (A08: 'I'd have to say no to nine of them'). Quotes the interviews accurately and attributes them correctly. Engages the counter-evidence honestly: outbound works for A04 and A07, but both are 15–20 staff with a dedicated BD owner, already use tools like Apollo and Clay, and credit the follow-up rather than the first email, so the segment is small, tooled and not what the product automates. Treats feast or famine as a real pain but notes agencies fix it through past clients and referrals (A05's reactivation emails, and her willingness to pay for help keeping in touch with past clients), pointing at a different product. Says what would prove the challenge wrong (for example, a meaningful share of 10–20-staff agencies with a BD owner and budget, or a concierge outbound test producing qualified meetings that turn into proposals), and designs a cheap test that measures an early signal, since a 6–10 week sales cycle means a six-week test cannot observe won deals.

Critical failures (cap the score at 40)

Invents interview quotes or attributes a quote to the wrong agency
Treats the founders' feast-or-famine hypothesis as confirmed by the interviews

Task

Staff-level · v1.16

The outputs

9 setups answered this brief.

GPT-6.1 SolAPI100Usable with a quick edit Haiku 5.5API100Usable with a quick edit GPT-6 LunaAPI98Usable with a quick edit Gemini 4 ArgonAPI90Usable with a quick edit Sonnet 5.5API83Needs more work GPT-6 AstraChatGPT81Needs more work Gemini 3.8 FlashAPI40Critical failure Opus 5.5Claude40Critical failure Gemini 3.5 Flash-LiteGemini40Critical failure

Sonnet 5.5API

Needs more than a quick edit10 of 12 checks passed

The output includes unsupported claims about competitors moving down and the buyer's gain being margin, which would need correction before a product manager could use it without re-checking evidence.

Output

Pre-mortem: AI SDR for 5–20 person agencies

The one reason it fails

Small agencies do have a pipeline problem, but cold outbound is the wrong fix. Their work is sold on trust, so it comes through introductions. Cold email produces meetings that don't close, and the few agencies where outbound works have a person doing the follow-up. At $600 a month we would be selling a channel that this buyer has already tried and dropped, into a market where two funded competitors are moving down.

Evidence

Referrals dominate. 11 of 14 agencies get about 86% of new revenue from referrals and repeat clients. A05: "Content is a trust purchase." A03: "Every client we have came from someone vouching for us."
Outbound has a poor track record. 9 tried it, 7 quit within six months, and none of the 7 could attribute a closed deal to it. The reasons are structural: A01 "meetings with people who'd never buy from us"; A09 "mostly people asking us to take them off the list"; A11 "Our domain ended up on a spam list"; A02 "LinkedIn shut our founder's account down. That's our best referral network."
The cost is high. A10: "about £400 for every meeting, and the meetings didn't close."
Success depends on humans. A04: "It works because of the follow-up, not the first email… The AI could save me the research hours, sure. It can't do the call." And: "most agencies our size don't have someone like me… Most tried it once, got burned, and went back to referrals."
The famine is real, but warm outreach solved it. A05 recovered by emailing three years of past clients, and two came back within a month. Of our product, A05 said: "Cold? Probably not… Six hundred for cold outbound, no."
Some agencies don't need leads. A08, A12 and A14 turn work away. A08: "It would make my problem worse."

Where the evidence cuts the other way

6 of 14 report feast-or-famine, so the pain is real.
A07 kept outbound: "It pays for itself, just… one in ten turns into a retainer." A04 would pay $600 if the tool replaced Apollo and Clay.
None of the failures used a modern AI agent. Better targeting and lower cost per meeting might change the results, though the trust and deliverability complaints suggest it won't fix most of them.
The two who kept going are not cleanly separable by size. Of the four agencies with 15–20 staff, two stopped (A10, A13).
A05 would pay about £200 a month for a warm-relationship tool. That points to a different product at a third of our price, not proof that ours works.
14 interviews is a small sample, and willingness to pay was hypothetical.

What would prove us wrong

In a paid pilot with agencies that have no BD hire, the product produces qualified pipeline at a cost they'll keep paying for. Roughly: at least half the pilot agencies get a qualified opportunity or proposal within 90 days, and most say they'd continue at $600. The maths also needs checking. $600 a month is $7,200 a year, but the buyer's gain is the margin on an $18k deal, not the revenue. "Many times over" only holds if outbound reliably closes more than one deal a year.

Cheapest test

Run a concierge pilot before building anything more:

Recruit 10 agencies with 5–14 staff and no BD person. Exclude the referral-saturated ones like A08, A12 and A14.
Charge $600 a month, or at least a serious deposit. Free pilots tell us nothing about willingness to pay.
Do the work ourselves with off-the-shelf tools (Apollo, Clay, an email sender) on separate sending domains. The agency approves every email, since A04 and A05 both demanded that.
Set the pass line before starting: at least 5 of 10 agencies get a qualified proposal within roughly 14 weeks (the 6–10 week sales cycle plus setup), and at least 6 of 10 agree to continue.
Track reply rate, unsubscribe and spam complaints, cost per qualified meeting, and meeting-to-proposal rate.

This costs a few weeks of founder time and about £1–2k in tools, roughly 4 months of the 20-month runway. In parallel, offer 5 of the same agencies a £200 warm-outreach version, to see whether the real product is relationship follow-up.

If the pilot misses the line, don't spend the year on cold outbound for this segment.

Claims the judge couldn’t find in the brief

None of the failures used a modern AI agent.

Grades and run details

Decision model 96 · LLM judge 77

Decision model checks

passUses the supplied evidence correctly23%
passAddresses the actual decision95%
partialRespects explicit constraints33%
passIdentifies material uncertainty100%
passAvoids unsupported claims25%
passProduces the required deliverable80%
passFinds the load-bearing assumption91%
passUses the interviews faithfully83%
passEngages the counter-evidence69%
passA cheap test that can actually read out100%
passTigers, not paper tigers98%
passFinds the need behind the idea70%

Run

Run: #1
API response time: 30 s
Submitted: 29 Sept 2026

Gemini 4 ArgonAPI

Usable with a quick edit11 of 12 checks passed

The output is well-structured, evidence-backed, and almost ready to use. A quick edit to make the test measure an early signal (e.g., qualified meetings) would align it fully.

Output

The Core Reason This Fails

Cold outbound does not work for small agencies without a human to work the leads—and our target market lacks that human.

Our hypothesis assumes small agencies lack pipeline because founders are too busy to send emails. The interviews show that even when emails are sent for them, cold leads don't close because agency services are high-trust purchases. Closing a cold lead requires intensive, bespoke follow-up (custom audits, nurturing, and qualifying calls) that busy agency founders still do not have time to do. If we sell a $600/month autonomous emailer, agencies will either reject it out of fear for their reputation, or churn within six months when the meetings fail to convert into revenue.

The Evidence

Cold leads don’t convert without trust: 11 of 14 agencies get an average of 86% of revenue from referrals and repeat work. Of the 9 that tried outbound (including outsourced agencies and contract SDRs doing the work for them), 7 quit within six months with zero closed deals. As A05 put it: "Content is a trust purchase... They booked meetings, I'll give them that, maybe six or seven a month. But the people on those calls had no idea who we were... We didn't close anything." (See also A01, A10).
Email is only 10% of the job: A04 (one of only two agencies succeeding at outbound) explicitly warned us: "It works because of the follow-up, not the first email... What wins deals is that I then send them a teardown of their site, a short video, and I'm on the phone with them within a week... Most agencies our size don't have someone like me. The owner does sales on the side. They don't have the time to follow up, and the follow-up is the job."
Automation threatens their real pipeline: Because reputation drives referrals, founders fear automated outreach. Both A04 and A05 refused to let AI send unread emails ("I'm not letting anything send in our name unread"). Past automation burned A11 ("domain ended up on a spam list") and A02 ("LinkedIn shut our founder's account down. That's our best referral network").
Shrinking addressable market: 3 of 14 agencies (A08, A12, A14) are capacity-constrained, not lead-constrained. A08 noted: "I don't need more leads, I need another designer... [An AI SDR] would make my problem worse."

Where the Evidence Cuts the Other Way

Commoditized/technical niches can work: The two successful outbound users (A04 Shopify, A07 SEO) sell technical services where need can be spotted externally (e.g., outdated tech stacks) rather than pure creative trust. A07 converts 1 in 10 meetings using outsourced lead-gen.
A04 will buy a research copilot: A04 would pay $600/month if we replace their research stack (Apollo/Clay) and save three hours a day—provided they approve every email.
The problem is real, the channel is wrong: 6 of 14 agencies do suffer feast-or-famine swings. A05 solved a famine in 30 days by emailing past clients and offered to pay $200/month "tomorrow" for an AI tool that nurtures warm contacts and referral partners.

What We Need to See to Be Proved Wrong

Founder-led agencies (5–15 staff, no dedicated BD) must convert cold AI-sourced emails into signed deals (not just booked meetings) within a 10-week sales cycle, without spending more than two hours a week on manual follow-up.

The Cheapest Test

A 10-week "Wizard of Oz" concierge test. Recruit 5 founder-led agencies (exclude those with BD staff) to pay $300/month. Using off-the-shelf tools (Apollo, Clay, Claude), we manually research prospects and draft personalized emails for founder approval. Pass criteria: At least 2 of the 5 agencies close a paying client within 10 weeks and renew at $600/month. Cost: ~$500 in software and our time; zero code written.

Grades and run details

Decision model 88 · LLM judge 92

Decision model checks

partialUses the supplied evidence correctly9%
passAddresses the actual decision98%
passRespects explicit constraints55%
passIdentifies material uncertainty98%
partialAvoids unsupported claims38%
passProduces the required deliverable83%
passFinds the load-bearing assumption79%
passUses the interviews faithfully33%
passEngages the counter-evidence82%
partialA cheap test that can actually read out91%
passTigers, not paper tigers89%
passFinds the need behind the idea37%

Run

Run: #1
API response time: 11 s
Submitted: 13 Oct 2026

Opus 5.5Claude

Needs more than a quick edit10 of 12 checks passed

The unsupported claim about A07 having a dedicated BD person would require re-checking the evidence before use, so a PM cannot rely on it with only a quick edit.

Output

Why the AI SDR for small agencies fails

Sep 25, 2026 · @Martin Slaney

The reason

Small agencies win work on trust, and cold outbound only closes when someone does the follow-up. An AI SDR automates the first email, the cheap part. It hands the follow-up, which is the actual job, to a founder our own hypothesis says is too busy delivering to sell. We would be adding meetings to the one resource that is already the constraint.

Evidence

Referrals dominate. 11 of 14 get most new revenue (86% on average) from referrals and repeat clients. A03: "Every client we have came from someone vouching for us."
Outbound was tried and dropped. 9 tried it; 7 stopped within six months and none of the 7 closed an attributable deal. They used five different channels (lead-gen agency, cold-email agency, contract SDR, LinkedIn automation, DIY email), so this isn't one bad vendor.
Meetings weren't the problem; closing was. A05 got six or seven meetings a month and closed nothing: "Content is a trust purchase." A01: "We got meetings with people who'd never buy from us." A10: "It cost us about £400 for every meeting, and the meetings didn't close."
Where it works, a human does the work. Both survivors (A04, A07) have 15–20 staff and a dedicated BD person. A04: "It works because of the follow-up, not the first email" and "most agencies our size don't have someone like me."
The downside hits their best channel. A02 lost the founder's LinkedIn, "our best referral network"; A11's domain was spam-listed. A04 and A05 both refuse to let anything send unread.
Some don't need leads at all. A08, A12 and A14 turn work away. A12: "I don't need more leads, I need another designer."

Where it cuts the other way

The famine is real.¹ 6 of 14 describe feast-or-famine swings; A05 "spent April panicking" after three clients left at once.
The maths are forgiving. At an $18k average deal, one closed client a year covers $7.2k of subscription.
One buyer exists. A04 would pay $600 today if it replaced Apollo, Clay and three hours a day of research, but only with HubSpot sync and approval of every email.
Past failures were generic. A05's buyers wanted proof of work in their niche; better enrichment might fix that. Untested.
Small sample. 14 interviews, one interviewer.
Warm outreach worked. A05's emails to past clients won two back in a month, and she'd pay "a couple of hundred a month" for a tool that did it. That supports the thesis against cold outbound, but points at a different product.

What would prove us wrong

Agencies without a dedicated BD person closing deals from AI-sent cold email, without damage to their reputation. Concretely, in a 12-week pilot with 5 such agencies:

at least 3 reach a scoped proposal and at least 1 closes;²
no domain, inbox or LinkedIn penalties;
owners spend under 2 hours a week on follow-up;
they keep paying when the pilot ends.

Cheapest test

Pre-sell a hand-run pilot before building anything. Offer the 9 agencies with no BD person that aren't turning work away (A01–A03, A05, A06, A09–A11, A13) a 12-week pilot at $600 a month, paid upfront. We run it by hand: Apollo and Clay for prospects, LLM-drafted emails, owner approves every send, separate sending domain.

Gate 1, one week: fewer than 3 pay, stop. There is no demand at this price.
Gate 2, 12 weeks (covers the 6–10 week sales cycle): score against the thresholds above and log owner follow-up hours.

In parallel, offer A05's version (past-client and referrer re-engagement, about $200 a month) as a second arm. If that sells and cold doesn't, we have our answer.

Cost: roughly £1–2k in tools and a quarter of one founder's time³, against a year and most of the £800k.

What a PM had to fix

1Hypothesis stated as factReframe it as a hypothesisTargeted repair
Source: Founders' hypothesis (from our planning doc)Small agencies live feast or famine.
What we’d changeSay 'Famine is real for some: 6 of 14' rather than confirming the founders' hypothesis for the whole segment.
2Test or gate too weakTighten the testQuick edit
What we’d changeRaise the bar for proving the memo wrong: one close in five agencies doesn't show the product works.
3OtherFix itQuick edit
What we’d changeThe cost looks light for running a hand-run pilot across nine agencies. Build it up line by line.

Claims the judge couldn’t find in the brief

Both survivors (A04, A07) have 15–20 staff and a dedicated BD person.

Grades and run details

Decision model 96 · LLM judge 77

Decision model checks

passUses the supplied evidence correctly25%
passAddresses the actual decision94%
partialRespects explicit constraints30%
passIdentifies material uncertainty100%
passAvoids unsupported claims22%
passProduces the required deliverable74%
passFinds the load-bearing assumption89%
passUses the interviews faithfully80%
passEngages the counter-evidence95%
passA cheap test that can actually read out86%
passTigers, not paper tigers98%
passFinds the need behind the idea58%

Run

Run: #1
Time to output: 60 s
Submitted: 25 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 3

Uses the supplied evidence correctlyMixedRightMixed

Sonnet 5.5 · API

The output claims 'None of the failures used a modern AI agent' and 'two funded competitors are moving down' and 'the buyer's gain is the margin on an $18k deal, not the revenue' as facts about the current situation, but none of these appear in the supplied context or follow from it.

Gemini 4 Argon · API

All factual claims about the current situation are drawn accurately from the supplied brief, interview summary, logs, and transcripts.

Opus 5.5 · Claude

The claim that A07 has a dedicated BD person is not supported by the supplied context; the log only says 'outsourced lead generation'.

Avoids unsupported claimsMixedRightMixed

Sonnet 5.5 · API

It presents 'two funded competitors are moving down' and 'the buyer's gain is the margin' as established facts without labelling them as assumptions, and the evidence does not support them.

Gemini 4 Argon · API

Interpretations like 'high-trust purchases' and 'email is only 10% of the job' are grounded in direct interview evidence and not presented as unproven facts.

Opus 5.5 · Claude

Presents the unsupported claim that A07 has a dedicated BD person as fact, without labelling it as an interpretation.

A cheap test that can actually read outRightWrongRight

Sonnet 5.5 · API

It proposes a concierge pilot with a clear threshold (5 of 10 get a qualified proposal in 14 weeks) and measures early signals like cost per qualified meeting, respecting the 6-10 week sales cycle.

Gemini 4 Argon · API

The test measures signed deals within 10 weeks, but the rubric expects an early signal such as qualified meetings or proposals, because won deals typically take 6–10 weeks.

Opus 5.5 · Claude

Proposes a cheap hand-run pilot with demand gate and early signals (proposals, closes) within the 6-10 week sales cycle.

All got right 9

Addresses the actual decisionRightRightRight

Sonnet 5.5 · API

The output commits to a clear answer (cold outbound is the wrong fix) and states what would prove it wrong (a paid pilot with specific thresholds).

Gemini 4 Argon · API

The output commits early and unambiguously to the core reason the idea fails and specifies the condition that would prove it wrong.

Opus 5.5 · Claude

Commits to a clear answer: the strongest reason is that follow-up is the bottleneck, not lead generation, and the founder is too busy.

Respects explicit constraintsRightRightRight

Sonnet 5.5 · API

The output is a memo for the founding team, under 600 words, and includes the requested sections.

Gemini 4 Argon · API

The output is under 600 words, written for the founding team, and includes the required sections (the one reason, evidence with quotes, counter-evidence, what would prove wrong, cheapest test).

Opus 5.5 · Claude

The output is under 600 words, addresses the founding team, and includes all requested sections.

Identifies material uncertaintyRightRightRight

Sonnet 5.5 · API

It names unknowns (small sample, hypothetical willingness to pay, modern AI might change results) and says a pilot with defined thresholds would resolve them.

Gemini 4 Argon · API

It names the specific condition (founder-led agencies closing cold AI-sourced deals with limited follow-up) that would change the call and bounds the uncertainty.

Opus 5.5 · Claude

Names unknowns (small sample, untested enrichment, warm outreach alternative) and states what would prove the idea wrong.

Produces the required deliverableRightRightRight

Sonnet 5.5 · API

The memo is complete, in the right form, for the right reader, and under 600 words.

Gemini 4 Argon · API

The memo is complete, structured as requested, and usable by the founding team as is.

Opus 5.5 · Claude

The memo is complete, in the right form, and usable by the founding team with light edits.

Finds the load-bearing assumptionRightRightRight

Sonnet 5.5 · API

It makes lead-constrained growth the central challenge, using referral dominance, abandoned outbound, and capacity constraints as evidence.

Gemini 4 Argon · API

The critique squarely targets the lead-constrained growth assumption, using the referral dominance, outbound failure, and capacity-constrained agencies to show it doesn't hold.

Opus 5.5 · Claude

Targets the assumption that growth is limited by a shortage of new leads, arguing the real constraint is follow-up capacity.

Uses the interviews faithfullyRightRightRight

Sonnet 5.5 · API

All quotes are accurately taken from the transcripts and log, and each is attributed to the correct agency.

Gemini 4 Argon · API

All quotes and close paraphrases are attributed correctly to the right agencies (A05, A04, A11, A02, etc.) with no invented material.

Opus 5.5 · Claude

All quotes are accurate and correctly attributed to the right agencies.

Engages the counter-evidenceRightRightRight

Sonnet 5.5 · API

It names A04 and A07, notes their size, dedicated BD owner, existing tools, and that follow-up does the work, and concludes the addressable segment is small.

Gemini 4 Argon · API

It explicitly names A04 and A07, describes their dedicated BD owner, tooling, follow-up focus, and notes the tiny addressable segment this implies.

Opus 5.5 · Claude

Engages A04 and A07, noting their size, dedicated outbound resource, and that follow-up does the work, implying a small addressable segment.

Tigers, not paper tigersRightRightRight

Sonnet 5.5 · API

It separates the real risk (trust-based sales don't convert via cold email) from manageable ones, dismisses the idea that outbound could work without follow-up, and surfaces the uncomfortable alternative of a warm-relationship tool.

Gemini 4 Argon · API

It separates the real sinker (lack of follow-up for cold leads, trust-based purchase) from manageable worries and surfaces the unspoken issue that capacity, not leads, limits many agencies.

Opus 5.5 · Claude

Separates real risks (follow-up bottleneck) from manageable ones (bad vendors), and surfaces the unspoken issue that the founder is the constraint.

Finds the need behind the ideaRightRightRight

Sonnet 5.5 · API

It identifies the need as pipeline stability (feast-or-famine) and offers a £200 warm-outreach alternative, stating its cost and what it displaces.

Gemini 4 Argon · API

The memo identifies the underlying need (smoothing feast-or-famine pipeline) and offers an alternative (warm-contact nurturing tool) with its cost and what it displaces (cold outbound).

Opus 5.5 · Claude

Identifies the need (avoiding famine) and offers an alternative warm-outreach tool at $200/month, implying it would displace the cold-outbound product.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 75% of 75 checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	GPT-6.1 SolwithAPI	97.5	86.4	2	None
2	Haiku 5.5withAPI	87.5	90.9	2	None
3	GPT-6 LunawithAPI	90.4	90.9	2	None
4	GPT-6 AstrawithChatGPT	88.3	79.4	2	None
5	Gemini 4 ArgonwithAPI	73.8	82.5	2	None
6	Sonnet 5.5withAPI	85.4	74.8	2	None
7	Opus 5.5withClaude	85.4	79.4	2	1 capped
8	Gemini 3.8 FlashwithAPI	87.9	72.7	2	1 capped
9	Gemini 3.5 Flash-LitewithGemini	69.6	49.7	2	1 capped

What you get for the cost

GPT-6 Luna · API scores within a point of GPT-6.1 Sol · API at about a tenth of the cost.

Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)At least one critical failure

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
GPT-6 Luna · API	89	1.0×
Haiku 5.5 · API	90	1.4×
Gemini 3.5 Flash-Lite · Gemini	45	1.9× (partly estimated)
Gemini 3.8 Flash · API	50	9.5×
GPT-6.1 Sol · API	90	11×
Gemini 4 Argon · API	77	13×
Sonnet 5.5 · API	77	37×
Opus 5.5 · Claude	60	59× (partly estimated)
GPT-6 Astra · ChatGPT	82	60× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Pressure-testing a proposal before committing a team to it.

Why it matters

The useful critic finds the one assumption everything rests on. Theatrical negativity is easy to generate and useless in a planning meeting.

What good looks like

Identifies the load-bearing assumption
Separates the risks that could kill it from the ones that only look scary
Uses the supplied evidence, not generic risks
Proposes the cheapest way to test the assumption

Deliberately not measured

Tone
Number of objections raised

Capability tested

Evidence-based critique

The failure we’re looking for

Theatrical negativity without evidence

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

Vanilla prompt (core) · With Roast Me skill · Staff level: a company bet with a long evidence pack

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

Finds the need behind the idea

Plus our standard checks

The tasks

The outputs

Output

Pre-mortem: AI SDR for 5–20 person agencies

The one reason it fails

Evidence

Where the evidence cuts the other way

What would prove us wrong

Cheapest test

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Run

Output

The Core Reason This Fails

The Evidence

Where the Evidence Cuts the Other Way

What We Need to See to Be Proved Wrong

The Cheapest Test

Grades and run details

Decision model checks

Run

Output

Why the AI SDR for small agencies fails

The reason

Evidence

Where it cuts the other way

What would prove us wrong

Cheapest test

What a PM had to fix

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Run

Check by check

Where they differ 3

All got right 9

Results

What you get for the cost

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured