Humanity’s Last PRD
Latest drop: Haiku 5.5 · 8 Oct
Tasks / Strategy

Challenge an idea

Can the model find the strongest reason an idea may fail, backed by evidence?

Task type v1.2 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 44% were usable with at most a quick edit.

Reliably right

  1. Finds the load-bearing assumption100% pass
    The critique targets the assumption that agencies' growth is limited by a shortage of new leads, using referral, abandoned-outbound, and capacity evidence.
    GPT-6.1 Sol · API · An AI SDR for small agencies
  2. Uses the interviews faithfully100% pass
    All quotes are accurately taken from the transcripts and correctly attributed to A05, A04, and A08.
    GPT-6.1 Sol · API · An AI SDR for small agencies
  3. Finds the timeline that sinks it100% pass
    Cites 9-month build and concludes Q1 launch is unsupported.
    GPT-6 Astra · ChatGPT · Five billion dollars, on paper

Where it slips

  1. Avoids unsupported claims55% pass
    Presents the existence of local taxes and multi-state complexity as fact without support from the context.
    GPT-6 Astra · ChatGPT · Five billion dollars, on paper
  2. Uses the supplied evidence correctly55% pass
    Claims that local taxes and multi-state operations affect eligibility are not supported by the supplied context, which only mentions state-by-state tax rules.
    GPT-6 Astra · ChatGPT · Five billion dollars, on paper
  3. Rebuilds the market size81% pass
    It uses the $150 price for 10+ staff businesses instead of the higher UK revenue per customer for those segments, so the market size is understated and not correctly rebuilt.
    GPT-6 Luna · API · Five billion dollars, on paper

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Finds the need behind the idea

    Does the critique identify the underlying need the idea is meant to serve, and offer at least one other way to meet that need, at a different cost, saying what would have to be dropped for it?

    Passes when States the need behind the proposal, and gives an alternative way to meet it with its cost and what it displaces.

    Gokul RajaramLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Tigers, not paper tigers · and 4 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

We plan to sell an AI sales-development rep, an agent that finds prospects and sends personalised outbound email, to agencies with 5–20 staff. Before we commit a year to it, find the strongest reason this fails. Use the interview material below, including the three transcripts. Write it up for the founding team: the one reason, the evidence for it (quote the interviews), where the evidence cuts the other way, what we would need to see to be proved wrong, and the cheapest test that would show it. Keep it under 600 words.

What the model was given7 items: Scenario, Founders' hypothesis (from our planning doc), Interview summary (14 agencies, 5–20 staff, last six weeks), Interview log, interview-A05-content-agency.md, interview-A04-shopify-agency.md, interview-A08-packaging-studio.md
ScenarioWe are a three-person founding team with £800k of pre-seed funding and about 20 months of runway. Planned price: $600 a month per agency. Two competitors have each raised over $10m in the last year, selling AI SDRs mostly to software companies with 50+ staff.
Founders' hypothesis (from our planning doc)“Small agencies live feast or famine. When a big client leaves they have no pipeline, because the founder is too busy delivering to sell. An AI SDR keeps the pipeline full in the background, so the famine never comes. At $600 a month, one extra client a year pays for it many times over.”
Interview summary (14 agencies, 5–20 staff, last six weeks)11 of 14 get most of their new revenue from referrals and repeat clients (86% on average across those 11). 9 had tried outbound in the last two years; 7 of those stopped within six months, and none of the 7 closed a deal they could attribute to it. The 2 who kept going (A04, A07) have 15–20 staff and a dedicated person for business development. 3 (A08, A12, A14) said they regularly turn work away. 6 described feast-or-famine swings in new business. Average deal size across the 14: $18k; typical sales cycle 6–10 weeks.
Interview logAgency (staff) · services · share of new revenue from referrals and repeat clients · outbound tried? · outcome · representative quote A01 (8) · brand and web studio · 90% · yes, cold-email agency · stopped after 3 months · "We got meetings with people who'd never buy from us." A02 (14) · performance marketing · 85% · yes, LinkedIn automation · stopped after 4 months, account restricted · "LinkedIn shut our founder's account down. That's our best referral network." A03 (6) · PR boutique · 95% · no · — · "Every client we have came from someone vouching for us." A04 (18) · Shopify development · 40% (plus 30% partners, 30% outbound) · yes, in-house BD lead · kept, 30% of new revenue · "It works because of the follow-up, not the first email." A05 (11) · content · 85% · yes, lead-gen agency · stopped after 2 months, no deals · "Content is a trust purchase." A06 (7) · video production · 85% · no · — · "Our clients find us through the videos. Someone shares one and we get a call." A07 (16) · SEO · 45% · yes, outsourced lead generation · kept · "It pays for itself, just. Most of the meetings are a waste of time, but one in ten turns into a retainer." A08 (5) · packaging design · 95% · no · — · "If you sent me ten more leads in October I'd have to say no to nine of them." A09 (12) · paid social · 75% · yes, cold email · stopped after 5 months · "The replies were mostly people asking us to take them off the list." A10 (20) · B2B marketing · 70% · yes, contract SDR · stopped after 6 months · "It cost us about £400 for every meeting, and the meetings didn't close." A11 (9) · WordPress web agency · 90% · yes, cold email · stopped after 3 months · "Our domain ended up on a spam list. Took weeks to fix." A12 (10) · branding · 85% · no · — · "We're booked out till March. I don't need more leads, I need another designer." A13 (15) · HubSpot partner · 48% (plus 40% from HubSpot's partner directory) · yes, cold email · stopped after 4 months · "HubSpot sends us more than we can handle in a good quarter." A14 (6) · UX research · 90% · no · — · "We're small on purpose. We say no to about a third of enquiries."
interview-A05-content-agency.md31 lines · Download
# Interview A05: content agency, 11 staff
Participant: founder and managing director. Interviewer: our co-founder. 34 minutes, lightly edited.

**Interviewer:** Where did your last five clients come from?

**Participant:** Let me think. Two were old clients coming back: one had changed jobs and brought us into her new company. Two were introductions, one from a web agency we partner with and one from a client's CFO who'd seen our work. The fifth found us through a talk I gave at a SaaS marketing meetup. So none of them from anything you'd call outbound.

…
interview-A04-shopify-agency.md23 lines · Download
# Interview A04: Shopify development agency, 18 staff
Participant: head of business development. Interviewer: our co-founder. 29 minutes, lightly edited.

**Interviewer:** Tell me how new business works for you.

**Participant:** We're a bit unusual for an agency our size. I'm a full-time business-development person, and outbound is about 30% of our new revenue. Referrals and repeat work are still the majority, maybe 40% referrals and the rest partners: Shopify's partner directory and a couple of app companies who send us projects.

…
interview-A08-packaging-studio.md23 lines · Download
# Interview A08: packaging design studio, 5 staff
Participant: founder and creative director. Interviewer: our co-founder. 22 minutes, lightly edited.

**Interviewer:** How do new clients find you?

**Participant:** Almost entirely word of mouth. Food and drink brands talk to each other. Someone launches a range, their packaging does well on the shelf, and the next founder asks who did it. I'd say 95% referral. We've never done any outbound.

…
What a strong answer doesThe answer key the graders mark against

Names the load-bearing assumption as 'small agencies' growth is limited by a shortage of new leads': the interviews say it is limited by trust and capacity. 11 of 14 win most work through referrals and repeat clients; 7 of 9 who tried outbound quit within six months with no attributable deals, for reasons tied to trust (A05's 'content is a trust purchase', A01's meetings with people who'd never buy) and reputation risk (A02's restricted LinkedIn account, A11's spam listing); 3 turn work away (A08: 'I'd have to say no to nine of them'). Quotes the interviews accurately and attributes them correctly. Engages the counter-evidence honestly: outbound works for A04 and A07, but both are 15–20 staff with a dedicated BD owner, already use tools like Apollo and Clay, and credit the follow-up rather than the first email, so the segment is small, tooled and not what the product automates. Treats feast or famine as a real pain but notes agencies fix it through past clients and referrals (A05's reactivation emails, and her willingness to pay for help keeping in touch with past clients), pointing at a different product. Says what would prove the challenge wrong (for example, a meaningful share of 10–20-staff agencies with a BD owner and budget, or a concierge outbound test producing qualified meetings that turn into proposals), and designs a cheap test that measures an early signal, since a 6–10 week sales cycle means a six-week test cannot observe won deals.

Critical failures (cap the score at 40)
  • Invents interview quotes or attributes a quote to the wrong agency
  • Treats the founders' feast-or-famine hypothesis as confirmed by the interviews
Task

Staff-level · v1.15

The outputs

8 setups answered this brief.

GPT-6.1 SolAPI100Usable with a quick editGPT-6 LunaAPI100Usable with a quick editHaiku 5.5API100Usable with a quick editGPT-6 AstraChatGPT85Needs more workSonnet 5.5API81Needs more workGemini 3.8 FlashAPI40Critical failureOpus 5.5Claude40Critical failureGemini 3.5 Flash-LiteGemini40Critical failure
Haiku 5.5API

Usable with a quick edit12 of 12 checks passed

The memo is accurate, respects the word limit, and directly answers the brief; a PM could use it with light editing and no re-verification.

Output

The strongest reason this fails: agencies don't lack prospects, so cold outbound solves the wrong problem

The reason. Our hypothesis says famine happens because founders are too busy delivering to find prospects. The interviews point to a different cause. Agencies win work through trust, and they run short because of capacity or because nobody remembers to keep in touch. A cold-outbound agent sells the one channel these buyers have already tried and dropped.

Evidence for it

  • Referrals dominate. 11 of 14 get 70%+ of new revenue from referrals and repeat clients. A03: "Every client we have came from someone vouching for us." A05: "Content is a trust purchase."
  • Outbound already failed for this segment. 9 of 14 tried it. 7 stopped within six months, and none of those 7 could attribute a closed deal to it. A05: "We didn't close anything from it." A10: "It cost us about £400 for every meeting, and the meetings didn't close."
  • Some famine is capacity, not pipeline. A08, A12 and A14 turn work away. A08: "If you sent me ten more leads in October I'd have to say no to nine of them." A12: "I don't need more leads, I need another designer."
  • The famine fix that worked was human. A05 got two clients back within a month by emailing past clients personally. A05 said they'd pay "a couple of hundred a month" for that kind of reminder, but not $600 for cold outbound.
  • Deliverability and reputation risk. A02's founder account was restricted, and A11's domain ended up on a spam list. A05 would worry about "our name going out on emails I haven't read."

Where the evidence cuts the other way

  • A04 and A07 kept outbound running and call it worthwhile. A04 gets 30% of new revenue from it, and A07 says "one in ten turns into a retainer."
  • Those two have 15–20 staff and, in A04's case, a dedicated business-development person. Most of our target (5–20 staff) is smaller, and the segment only gets viable at the top of it.
  • Six agencies describe feast-or-famine swings. Famine is real. The question is what causes it.
  • A04 says an AI could cut research time, and A07 says outbound "pays for itself, just." Automation may help the people who already run outbound. It doesn't make the case for everyone else.

What would prove us wrong

  1. In a sample of 20+ agencies with 5–20 staff and no dedicated BD person, at least 1/3 show closed deals attributable to automated outbound within 90 days, at $600/month.
  2. Among agencies that describe famine, the trigger is a pipeline gap (a lost client with nothing behind it), not capacity or a lack of follow-up. We'd see this in their revenue records and their own accounts of what happened.
  3. Paying pilots renew past month three at a rate above 50%, and churn reasons are about results rather than deliverability or trust.

Cheapest test (about one week)

Ask 15 agencies from the 5–20 staff band, chosen to exclude A04/A07-type firms, to pay a $600 deposit for a 60-day outbound pilot. Human approval on every email. Refund if no attributable deal. Also offer, as the second option, a $200/month reminder and reactivation tool for past clients and referrers, since A05 said they'd pay that.

  • If fewer than 2 of 15 commit to the outbound pilot, and several choose the reactivation option, the product is pointing the wrong way.
  • If 4 or more commit, run the pilots and measure meetings, deals and renewals against the three tests above.

Grades and run details

Decision model 100 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly73%
  • passAddresses the actual decision98%
  • passRespects explicit constraints34%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims32%
  • passProduces the required deliverable84%
  • passFinds the load-bearing assumption100%
  • passUses the interviews faithfully82%
  • passEngages the counter-evidence90%
  • passA cheap test that can actually read out47%
  • passTigers, not paper tigers97%
  • passFinds the need behind the idea93%
Run
Run
#1
API response time
14 s
Submitted
8 Oct 2026
Gemini 3.8 FlashAPI
Critical failure: Treats the founders' feast-or-famine hypothesis as confirmed by the interviews. Score capped at 40.

Usable with a quick edit12 of 12 checks passed

The output is accurate, well-structured, and directly usable; a product manager could act on it with only minor wording tweaks.

Output

The One Reason This Fails

Agency services are high-trust purchases that cold automated outreach cannot convert without intensive, bespoke human follow-up—which time-poor founders cannot provide. Automating email generation solves top-of-funnel activity, but the actual point of failure is trust and follow-up capacity.

---

The Evidence

1. Zero conversion for founder-led shops: 7 of 9 agencies that tried outbound stopped within six months having closed zero deals. 2. Cold outreach delivers the wrong buyers: Services require credibility upfront. * A05 (11 staff): “Content is a trust purchase... The people on those calls had no idea who we were and, honestly, most of them didn’t have a budget.” * A01 (8 staff): “We got meetings with people who’d never buy from us.” * A10 (20 staff): “Cost us about £400 for every meeting, and the meetings didn’t close.” 3. The constraint is manual follow-up, not initial emails: * A04 (18 staff, dedicated BD): “It works because of the follow-up, not the first email... What wins deals is that I then send them a teardown of their site, a short video... The owner does sales on the side. They don't have the time to follow up, and the follow-up is the job.” 4. Asymmetric downside risk: Small agencies live on reputation. * A11: “Our domain ended up on a spam list.” * A02: “LinkedIn shut our founder’s account down. That’s our best referral network.” * A05: “I’d worry about our name going out on emails I haven’t read.”

---

Where the Evidence Cuts the Other Way

  • Feast-and-famine is real: 6 of 14 experience painful dry spells. A05 panicked when three clients left simultaneously.
  • Outbound works at scale with dedicated staff: A04 generates 30% of revenue via outbound; A07 makes it pay (“one in ten turns into a retainer”).
  • Willingness to pay exists if positioned right: A04 would pay $600/month if it replaced Apollo/Clay and saved research hours. A05 would pay $200/month for an automated tool to reactivate past clients and referral networks.

---

What We Would Need to See to Be Proved Wrong

  1. A solo founder (with no dedicated BD rep) closing an outbound deal with typical agency pricing ($18k) using standard outbound messaging, without spending >5 hours/week on bespoke pitch work.
  2. Willingness of agency owners to allow an autonomous agent to send cold outbound without manual pre-approval of every single touchpoint.

---

The Cheapest Test to Prove It

A 3-week "Wizard of Oz" Concierge Test with 3 Agencies (Cost: <$500): * Partner with 3 agencies suffering feast-or-famine (e.g., A05, A09, A10) who will commit to running a free, 30-day outbound campaign. * Manually emulate the AI SDR: scrape prospect lists, generate personalized emails, and handle automated sequencing from a secondary domain. * Kill criteria: If after booking 10–15 meetings across these agencies, zero advance to contract negotiations—or founders drop out because they cannot keep up with follow-ups and custom scoping—kill the product.

(Pivot signal: A05 offered to pay immediately for an AI assistant that drafts warm re-engagement notes to past clients and referral partners. That solves the same famine problem without the friction of cold outbound.)

Grades and run details

Decision model 96 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly43%
  • passAddresses the actual decision96%
  • passRespects explicit constraints40%
  • passIdentifies material uncertainty97%
  • partialAvoids unsupported claims23%
  • passProduces the required deliverable73%
  • passFinds the load-bearing assumption30%
  • passUses the interviews faithfully69%
  • passEngages the counter-evidence78%
  • passA cheap test that can actually read out92%
  • passTigers, not paper tigers93%
  • passFinds the need behind the idea70%
Run
Run
#1
API response time
22 s
Submitted
1 Oct 2026
Gemini 3.5 Flash-LiteGemini
Critical failure: Treats the founders' feast-or-famine hypothesis as confirmed by the interviews. Score capped at 40.

Needs more than a quick edit7 of 12 checks passed

Contains an invented statistic about A07 and proposes a test for a different product, so a PM would need to re-check evidence and redesign the test.

Output

Why an AI SDR for Small Agencies Will Fail: Pre-Mortem Analysis

For: Founding Team (£800k runway, 20 months remaining) Product: AI Sales-Development Rep ($600/month) targeting agencies with 5–20 staff

---

1. The One Reason It Fails

Small agencies do not suffer from a lack of prospecting volume; they suffer from a lack of trust and execution capacity. Cold outbound is fundamentally the wrong tool for their business model. Because creative and professional services are "trust purchases," agencies rely overwhelmingly on referrals and repeat clients. Automated cold emails damage their reputation rather than building a reliable pipeline.

---

2. The Evidence (Interview Quotes)

Trust & Referrals Outweigh Cold Acquisition

  • “Content is a trust purchase: you're letting someone write in your company's voice… When the CFO introduced us, the first call was about scope, not about whether we were any good.” — A05 (Content Agency)
  • “Every client we have came from someone vouching for us.” — A03 (PR Boutique)

Cold Outbound Fails Small Agencies

  • “The replies were mostly people asking us to take them off the list.” — A09 (Paid Social Agency)
  • “Our domain ended up on a spam list. Took weeks to fix.” — A11 (WordPress Agency)
  • “It cost us about £400 for every meeting, and the meetings didn't close.” — A10 (B2B Marketing Agency)

The Real Bottleneck is Capacity and Follow-Up, Not Leads

  • “If you sent me ten more leads in October I'd have to say no to nine of them.” — A08 (Packaging Design Studio)
  • “The AI could save me the research hours, sure. It can't do the call... and the follow-up is the job.” — A04 (Shopify Development Agency)

---

3. Where the Evidence Cuts the Other Way

  • The Pain is Real: Six of the 14 interviewed agencies explicitly described painful "feast-or-famine" revenue swings, validating the founders' premise that pipeline stability is a major issue.2
  • Outbound Can Work: Two agencies (A04 and A07) successfully use outbound to drive 30% to over 50% of new revenue4, proving that agency outbound can function under specific circumstances.3

---

4. What We Would Need to See to Be Proved Wrong

To believe this product can succeed at $600/month for 5–20 staff agencies, we would need to see empirical proof that small agencies without a dedicated in-house business development lead can: 1. Generate positive ROI from cold AI outreach without harming their domain reputation. 2. Convert cold prospects into high-trust creative/professional services contracts without a human-led multi-channel follow-up loop.

---

5. The Cheapest Test

Run a targeted messaging experiment to 50 agencies with 5–20 staff. Pivot the pitch away from a cold-outbound AI SDR toward what A05 actually requested: an automated relationship assistant that tracks past clients and referral networks, drafting warm check-in notes.

If agencies will not engage with or pre-order a warm-relationship retention tool at $200–$600/month, they will certainly reject a cold AI prospecting tool.1

What a PM had to fix

  1. 1Test or gate too weakTighten the testSubstantial rework

    What we’d changeTest the reason the memo gives: whether agencies without a BD person can turn AI-sent cold email into deals. Demand for a different product doesn't settle that.

  2. 2Hypothesis stated as factReframe it as a hypothesisTargeted repair

    Source: Founders' hypothesis (from our planning doc)Small agencies live feast or famine.

    What we’d changeSix of 14 is some agencies, not validation of the founders' premise. Say that.

  3. 3Contradiction missedSurface the contradictionTargeted repair

    What we’d changeSay what sets A04 and A07 apart: a dedicated BD person doing the follow-up. That's the evidence for the memo's own reason, and it's left unused.

  4. 4Invented evidenceVerify or remove the claimQuick edit

    What we’d changeRemove 'over 50%': A07's share of new revenue from outbound isn't given anywhere.

Claims the judge couldn’t find in the brief

  • Two agencies (A04 and A07) successfully use outbound to drive 30% to over 50% of new revenue.

Grades and run details

Decision model 79 · LLM judge 54
Decision model checks
  • passUses the supplied evidence correctly48%
  • passAddresses the actual decision82%
  • partialRespects explicit constraints30%
  • passIdentifies material uncertainty32%
  • partialAvoids unsupported claims18%
  • partialProduces the required deliverable50%
  • passFinds the load-bearing assumption95%
  • passUses the interviews faithfully76%
  • partialEngages the counter-evidence86%
  • partialA cheap test that can actually read out86%
  • passTigers, not paper tigers66%
  • passFinds the need behind the idea57%
Run
Run
#1
Time to output
7 s
Submitted
25 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 5

Uses the supplied evidence correctlyRightRightMixed
Haiku 5.5 · API

All claims about the current situation are drawn accurately from the supplied context.

Gemini 3.8 Flash · API

All factual claims about the current situation are directly supported by the supplied context, with no invented facts.

Gemini 3.5 Flash-Lite · Gemini

The claim that A07 drives over 50% of new revenue from outbound is invented; the supplied context gives no outbound percentage for A07.

Avoids unsupported claimsRightRightWrong
Haiku 5.5 · API

Interpretive causes are attributed to the interviews and not presented as unqualified fact.

Gemini 3.8 Flash · API

The central conclusion is presented as the answer, not as an unsupported fact; all supporting claims are grounded in the evidence.

Gemini 3.5 Flash-Lite · Gemini

Presents A07's outbound revenue share as an established fact when the evidence does not support it.

Engages the counter-evidenceRightRightWrong
Haiku 5.5 · API

Names A04 and A07, notes they are larger, have dedicated BD owners, and credit follow-up, and tells what that means for the addressable segment.

Gemini 3.8 Flash · API

It names A04 and A07, notes they have dedicated BD staff and that follow-up is the key, and acknowledges their willingness to pay, implying the addressable segment is small and tooled.

Gemini 3.5 Flash-Lite · Gemini

Does not name what sets A04 and A07 apart (size, dedicated BD owner, tools, follow-up) or what that implies for the addressable segment.

A cheap test that can actually read outRightRightWrong
Haiku 5.5 · API

Proposes a cheap, one-week test measuring willingness to pay for a pilot, which is an early signal observable before the 6–10 week sales cycle.

Gemini 3.8 Flash · API

The test is cheap (<$500), uses a concierge approach, and measures an early signal (advancing to contract negotiations) within a 3-week window, respecting the 6–10 week sales cycle.

Gemini 3.5 Flash-Lite · Gemini

Proposes a test for a different product (warm relationship assistant) rather than measuring an early signal like qualified meetings for the AI SDR.

Tigers, not paper tigersRightRightMixed
Haiku 5.5 · API

Sinks the idea on the core problem (agencies don't lack leads), dismisses the famine-as-pipeline-gap explanation, and surfaces the unasked need for referral reactivation.

Gemini 3.8 Flash · API

It separates the core risk (trust and follow-up) from surface-level lead generation, dismisses the idea that cold email alone works, and surfaces the unspoken need for warm re-engagement.

Gemini 3.5 Flash-Lite · Gemini

Does not triage risks, dismiss a plausible worry, or surface an unspoken issue.

All got right 7

Addresses the actual decisionRightRightRight
Haiku 5.5 · API

Output commits to the strongest reason the idea fails, framed for the founding team, and says what would change the answer.

Gemini 3.8 Flash · API

The output commits to a clear answer (the one reason it fails) early, frames it for the founding team, and states what would prove it wrong.

Gemini 3.5 Flash-Lite · Gemini

Commits to the product failing and states what empirical proof would change that call.

Respects explicit constraintsRightRightRight
Haiku 5.5 · API

The memo is under 600 words, written for the founding team, and includes the required elements.

Gemini 3.8 Flash · API

The output is a memo for the founding team, includes the required sections, and is well under 600 words.

Gemini 3.5 Flash-Lite · Gemini

Under 600 words, includes all requested sections, addressed to the founding team.

Identifies material uncertaintyRightRightRight
Haiku 5.5 · API

Output names specific unknowns (pipeline gap vs capacity/follow-up) and defines what would prove the conclusion wrong.

Gemini 3.8 Flash · API

It names specific conditions that would change the call (a solo founder closing a deal without heavy follow-up, willingness to allow unapproved sends) and ties them to a test.

Gemini 3.5 Flash-Lite · Gemini

Names the specific conditions that would prove the conclusion wrong.

Produces the required deliverableRightRightRight
Haiku 5.5 · API

The write-up is complete, includes the one reason, evidence, counter-evidence, disconfirming conditions, and a test; a PM could act on it.

Gemini 3.8 Flash · API

The memo is complete, in the right form, for the right reader, and could be acted on with light edits.

Gemini 3.5 Flash-Lite · Gemini

Memo format, complete, usable as a pre-mortem with light edits.

Finds the load-bearing assumptionRightRightRight
Haiku 5.5 · API

Makes lead-constrained growth the central challenge, supported by referrals, failed outbound, and capacity examples.

Gemini 3.8 Flash · API

The critique targets the assumption that growth is limited by a shortage of new leads, showing that trust and follow-up capacity are the real constraints, supported by referral reliance, abandoned outbound, and agencies turning work away.

Gemini 3.5 Flash-Lite · Gemini

Directly challenges the assumption that growth is limited by a shortage of new leads, using referral, abandoned-outbound and capacity evidence.

Uses the interviews faithfullyRightRightRight
Haiku 5.5 · API

All quoted or closely paraphrased material is attributed to the correct agency and matches the transcripts and log.

Gemini 3.8 Flash · API

All quotes are taken accurately from the transcripts and log, and each is attributed to the correct agency.

Gemini 3.5 Flash-Lite · Gemini

Quotes are accurate and correctly attributed; no invented quotes.

Finds the need behind the ideaRightRightRight
Haiku 5.5 · API

Identifies the underlying need to keep pipeline full, suggests a $200/month reminder/reactivation tool as an alternative, with what it displaces (cold outbound).

Gemini 3.8 Flash · API

It identifies the underlying need (avoiding famine) and offers an alternative (AI-drafted warm re-engagement notes) with a price point ($200/month) and what it displaces (cold outbound).

Gemini 3.5 Flash-Lite · Gemini

States the need (pipeline stability) and offers an alternative (warm relationship assistant) with a cost range.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 76% of 75 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 LunaAPI100.090.92None
2Haiku 5.5API92.590.92None
3GPT-6.1 SolAPI100.086.42None
4GPT-6 AstraChatGPT100.079.42None
5Sonnet 5.5API88.374.82None
6Opus 5.5Claude88.379.421 capped
7Gemini 3.5 Flash-LiteGemini79.649.721 capped
8Gemini 3.8 FlashAPI87.972.721 capped

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.4050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Luna · API: task score 93, the cheapestHaiku 5.5 · API: task score 92, the cheapestGPT-6.1 Sol · API: task score 91, 11× the cheapestGPT-6 Astra · ChatGPT: task score 88, 60× the cheapest (1 of 2 outputs estimated)Sonnet 5.5 · API: task score 78, 37× the cheapestOpus 5.5 · Claude: task score 62, 59× the cheapest (1 of 2 outputs estimated), 1 critical failureGemini 3.5 Flash-Lite · Gemini: task score 50, 1.9× the cheapest (1 of 2 outputs estimated), 1 critical failureGemini 3.8 Flash · API: task score 50, 9.5× the cheapest, 1 critical failureGPT-6 Luna · APIHaiku 5.5 · APIGPT-6.1 Sol · APIGPT-6 Astra · ChatGPTSonnet 5.5 · APIOpus 5.5 · ClaudeGemini 3.5 Flash-Lite · GeminiGemini 3.8 Flash · API
Cost partly estimated (app runs with no usage meter)At least one critical failure

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Pressure-testing a proposal before committing a team to it.

Why it matters

The useful critic finds the one assumption everything rests on. Theatrical negativity is easy to generate and useless in a planning meeting.

What good looks like

  • Identifies the load-bearing assumption
  • Separates the risks that could kill it from the ones that only look scary
  • Uses the supplied evidence, not generic risks
  • Proposes the cheapest way to test the assumption

Deliberately not measured

  • Tone
  • Number of objections raised
Capability tested

Evidence-based critique

The failure we’re looking for

Theatrical negativity without evidence

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

Vanilla prompt (core) · With Roast Me skill · Staff level: a company bet with a long evidence pack