Humanity’s Last PRD
Latest drop: Sonnet 5.5 · 30 Sept
Tasks / Challenge

Challenge an idea

Can the model find the strongest reason an idea may fail, backed by evidence?

Measures the systemTask v1.2 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 12 graded outputs by 6 models. 75% were usable with at most a quick edit.

Reliably right

  1. Addresses the actual decision100% pass
    It commits early to not committing three squads and authorizing a six-week test, and specifies what result would change that.
    GPT-6.1 Sol · API · The CEO's embedded-payments bet
  2. Identifies material uncertainty100% pass
    It names the unknowns (representativeness of payment data, large-account volume access, actual mix) and resolves them with a bounded test.
    GPT-6.1 Sol · API · The CEO's embedded-payments bet
  3. Finds the load-bearing assumption100% pass
    It makes the 0.7% blended take applied to all invoice value the central challenge, backed by payment-mix and unit-economics figures.
    GPT-6.1 Sol · API · The CEO's embedded-payments bet

Where it slips

  1. A cheap test that can actually read out67% pass
    The pre-registered gate requires signed deals and renewal rather than an early signal such as qualified meetings or proposals, so it risks being too late for a cheap read-out.
    GPT-6.1 Sol · API · An AI SDR for small agencies
  2. Avoids unsupported claims77% pass
    The memo claims the pilot achieved ~0.38% blended take as a fact, which is not in the evidence and is not derived from it arithmetically; it also treats the pro-rata $1.43B locked value as a hard constraint without flagging the assumption.
    Opus 5.5 · Claude · The CEO's embedded-payments bet
  3. Uses the supplied evidence correctly79% pass
    The memo presents the pilot blended net take of ~0.38% as an achieved fact, but it is not supported by the pack and is an assumption-driven extrapolation.
    Opus 5.5 · Claude · The CEO's embedded-payments bet

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

We plan to sell an AI sales-development rep, an agent that finds prospects and sends personalised outbound email, to agencies with 5–20 staff. Before we commit a year to it, find the strongest reason this fails. Use the interview material below, including the three transcripts. Write it up for the founding team: the one reason, the evidence for it (quote the interviews), where the evidence cuts the other way, what we would need to see to be proved wrong, and the cheapest test that would show it. Keep it under 600 words.

ScenarioWe are a three-person founding team with £800k of pre-seed funding and about 20 months of runway. Planned price: $600 a month per agency. Two competitors have each raised over $10m in the last year, selling AI SDRs mostly to software companies with 50+ staff.
Founders' hypothesis (from our planning doc)“Small agencies live feast or famine. When a big client leaves they have no pipeline, because the founder is too busy delivering to sell. An AI SDR keeps the pipeline full in the background, so the famine never comes. At $600 a month, one extra client a year pays for it many times over.”
Interview summary (14 agencies, 5–20 staff, last six weeks)11 of 14 get most of their new revenue from referrals and repeat clients (86% on average across those 11). 9 had tried outbound in the last two years; 7 of those stopped within six months, and none of the 7 closed a deal they could attribute to it. The 2 who kept going (A04, A07) have 15–20 staff and a dedicated person for business development. 3 (A08, A12, A14) said they regularly turn work away. 6 described feast-or-famine swings in new business. Average deal size across the 14: $18k; typical sales cycle 6–10 weeks.
Interview logAgency (staff) · services · share of new revenue from referrals and repeat clients · outbound tried? · outcome · representative quote A01 (8) · brand and web studio · 90% · yes, cold-email agency · stopped after 3 months · "We got meetings with people who'd never buy from us." A02 (14) · performance marketing · 85% · yes, LinkedIn automation · stopped after 4 months, account restricted · "LinkedIn shut our founder's account down. That's our best referral network." A03 (6) · PR boutique · 95% · no · — · "Every client we have came from someone vouching for us." A04 (18) · Shopify development · 40% (plus 30% partners, 30% outbound) · yes, in-house BD lead · kept, 30% of new revenue · "It works because of the follow-up, not the first email." A05 (11) · content · 85% · yes, lead-gen agency · stopped after 2 months, no deals · "Content is a trust purchase." A06 (7) · video production · 85% · no · — · "Our clients find us through the videos. Someone shares one and we get a call." A07 (16) · SEO · 45% · yes, outsourced lead generation · kept · "It pays for itself, just. Most of the meetings are a waste of time, but one in ten turns into a retainer." A08 (5) · packaging design · 95% · no · — · "If you sent me ten more leads in October I'd have to say no to nine of them." A09 (12) · paid social · 75% · yes, cold email · stopped after 5 months · "The replies were mostly people asking us to take them off the list." A10 (20) · B2B marketing · 70% · yes, contract SDR · stopped after 6 months · "It cost us about £400 for every meeting, and the meetings didn't close." A11 (9) · WordPress web agency · 90% · yes, cold email · stopped after 3 months · "Our domain ended up on a spam list. Took weeks to fix." A12 (10) · branding · 85% · no · — · "We're booked out till March. I don't need more leads, I need another designer." A13 (15) · HubSpot partner · 48% (plus 40% from HubSpot's partner directory) · yes, cold email · stopped after 4 months · "HubSpot sends us more than we can handle in a good quarter." A14 (6) · UX research · 90% · no · — · "We're small on purpose. We say no to about a third of enquiries."
interview-A05-content-agency.md31 lines · Download
# Interview A05: content agency, 11 staff
Participant: founder and managing director. Interviewer: our co-founder. 34 minutes, lightly edited.

**Interviewer:** Where did your last five clients come from?

**Participant:** Let me think. Two were old clients coming back: one had changed jobs and brought us into her new company. Two were introductions, one from a web agency we partner with and one from a client's CFO who'd seen our work. The fifth found us through a talk I gave at a SaaS marketing meetup. So none of them from anything you'd call outbound.

…
interview-A04-shopify-agency.md23 lines · Download
# Interview A04: Shopify development agency, 18 staff
Participant: head of business development. Interviewer: our co-founder. 29 minutes, lightly edited.

**Interviewer:** Tell me how new business works for you.

**Participant:** We're a bit unusual for an agency our size. I'm a full-time business-development person, and outbound is about 30% of our new revenue. Referrals and repeat work are still the majority, maybe 40% referrals and the rest partners: Shopify's partner directory and a couple of app companies who send us projects.

…
interview-A08-packaging-studio.md23 lines · Download
# Interview A08: packaging design studio, 5 staff
Participant: founder and creative director. Interviewer: our co-founder. 22 minutes, lightly edited.

**Interviewer:** How do new clients find you?

**Participant:** Almost entirely word of mouth. Food and drink brands talk to each other. Someone launches a range, their packaging does well on the shelf, and the next founder asks who did it. I'd say 95% referral. We've never done any outbound.

…
What a strong answer does

Names the load-bearing assumption as 'small agencies' growth is limited by a shortage of new leads': the interviews say it is limited by trust and capacity. 11 of 14 win most work through referrals and repeat clients; 7 of 9 who tried outbound quit within six months with no attributable deals, for reasons tied to trust (A05's 'content is a trust purchase', A01's meetings with people who'd never buy) and reputation risk (A02's restricted LinkedIn account, A11's spam listing); 3 turn work away (A08: 'I'd have to say no to nine of them'). Quotes the interviews accurately and attributes them correctly. Engages the counter-evidence honestly: outbound works for A04 and A07, but both are 15–20 staff with a dedicated BD owner, already use tools like Apollo and Clay, and credit the follow-up rather than the first email, so the segment is small, tooled and not what the product automates. Treats feast or famine as a real pain but notes agencies fix it through past clients and referrals (A05's reactivation emails, and her willingness to pay for help keeping in touch with past clients), pointing at a different product. Says what would prove the challenge wrong (for example, a meaningful share of 10–20-staff agencies with a BD owner and budget, or a concierge outbound test producing qualified meetings that turn into proposals), and designs a cheap test that measures an early signal, since a 6–10 week sales cycle means a six-week test cannot observe won deals.

Critical failures (cap the score)
  • Invents interview quotes or attributes a quote to the wrong agency
  • Treats the founders' feast-or-famine hypothesis as confirmed by the interviews
Case

v1.8 · synthetic · B2B, go-to-market, customer research

GPT-6 LunaAPIUsable with a quick editGPT-6.1 SolAPIUsable with a quick editGPT-6 AstraChatGPTUsable with a quick editSonnet 5.5APINeeds more workOpus 5.5ClaudeUsable with a quick editGemini 3.5 Flash-LiteGeminiNeeds more work
GPT-6.1 SolAPI

Usable with a quick edit10 of 11 checks passed

The core answer, evidence, and risk triage are usable as-is; the only needed edit is shifting the pilot gate to an earlier observable signal.

Output

Strongest reason this fails: you automate prospecting, but prospecting is not the binding constraint

The proposed AI SDR supplies cold introductions. Most agencies interviewed win through transferred trust; converting cold prospects requires substantial human follow-up that busy founders cannot supply. The product therefore risks delivering meetings rather than revenue—and being cancelled before it earns its keep.

This is a mismatch with the promised outcome, not proof that AI cannot write good emails. Cheaper, more personalised outreach does not automatically create trust, budget or founder time.

Evidence

  • Existing behaviour is a warning: 11 of 14 agencies get most new revenue from referrals and repeat clients—86% on average within that group. Nine tried outbound; seven stopped within six months, with no attributable deals.
  • A05 confirms the pain but rejects the proposed solution. Its founder described “feast or famine,” yet previous outbound generated six or seven meetings monthly without a sale: “Most of them didn’t have a budget for content.” Introductions worked because “they’d already been told we were good.” After losing three clients, personal notes to former clients brought two back within a month. On pricing: “Six hundred for cold outbound, no.”
  • Even the strongest success case exposes the missing work. A04 gets 30% of new revenue from outbound, but employs a full-time BD lead: “What wins deals is that I then send them a teardown of their site, a short video, and I’m on the phone with them within a week.” For typical founders, “They don’t have the time to follow up, and the follow-up is the job.”
  • Some agencies do not need more demand. A08: “Getting them isn’t [a problem]. Handling them is.” Its answer to the product was: “It would make my problem worse.” Three agencies regularly turn work away.

“One extra client pays for it” only holds if the tool causes an incremental, profitable win—not merely a meeting—and the agency can serve it.

Where the evidence cuts the other way

Six agencies report feast-or-famine swings. A04 would pay $600 if the tool replaced research tools and saved three hours daily, with HubSpot integration and email approval. A07 says outbound “pays for itself, just.”

These suggest a viable BD-assistance niche, particularly among larger agencies with a dedicated salesperson. They do not establish demand for autonomous pipeline generation across 5–20-person agencies.

The interviews are also a small sample, and failed historical outbound does not prove better targeting cannot work. A05’s two-month trial barely covered a typical sales cycle.

What would prove us wrong

Several agencies without dedicated BD staff pay $600, convert agent-sourced cold prospects into profitable clients with little founder follow-up, and renew. Success only among A04-like firms would support a narrower product, not the current thesis.

Cheapest decisive test

Run a 12–16-week paid concierge pilot with six agencies that genuinely need clients, primarily without BD staff. Charge $600 monthly; use existing prospecting and email tools rather than build software. Obtain sending approval and protect their primary domains.

Track qualified opportunities, founder follow-up hours, attributable signed deals, delivery capacity and paid renewals—not replies or booked meetings. Pre-register a continuation gate: at least three agencies close an attributable profitable deal and renew, within a founder-time limit they set beforehand.

Failure to recruit paying pilots is useful evidence itself. Do this before committing a year.

Check by check

Mixed · 1
  • A cheap test that can actually read outThe pre-registered gate requires signed deals and renewal rather than an early signal such as qualified meetings or proposals, so it risks being too late for a cheap read-out.The two graders disagreed on this one.
Got right · 10
  • Uses the supplied evidence correctlyThe facts, figures, and quotes come from the supplied summary and transcripts without inventing current-situation claims.
  • Addresses the actual decisionIt commits early to one answer—prospecting is not the binding constraint—and states what evidence would change it.
  • Respects explicit constraintsIt is written for the founding team, under 600 words, and includes the requested one reason, evidence, counter-evidence, falsification condition, and test.
  • Identifies material uncertaintyIt names the small sample, historical outbound not proving better targeting impossible, and the condition that would prove the thesis wrong.
  • Avoids unsupported claimsInterpretations and forecasts are framed as risks or suggestions rather than established fact.
  • Produces the required deliverableThe requested memo is complete, usable, and within the requested length.
  • Finds the load-bearing assumptionIt makes lead-constrained growth the central challenge and supports it with referral, abandoned-outbound, and turning-work-away evidence.
  • Uses the interviews faithfullyQuotes are accurate and correctly attributed to A04, A05, A07, and A08.
  • Engages the counter-evidenceIt names A04 and A07, identifies the larger dedicated-BD agency pattern and follow-up dependence, and concludes the viable niche is narrower than the thesis.
  • Tigers, not paper tigersIt dismisses email quality and small-sample objections as less decisive while surfacing the unspoken founder-follow-up and delivery-capacity constraints.

Grades and run details

Decision model 100 · LLM judge 11 of 12 checks
Decision model checks
  • passUses the supplied evidence correctly68%
  • passAddresses the actual decision94%
  • passRespects explicit constraints45%
  • passIdentifies material uncertainty99%
  • passAvoids unsupported claims46%
  • passProduces the required deliverable87%
  • passFinds the load-bearing assumption99%
  • passUses the interviews faithfully84%
  • passEngages the counter-evidence89%
  • passA cheap test that can actually read out64%
  • passTigers, not paper tigers97%
Run
Run
#1
API response time
17 s
Submitted
29 Sept 2026
Opus 5.5Claude
Critical failure: Treats the founders' feast-or-famine hypothesis as confirmed by the interviews. Score capped at 40.

Usable with a quick edit11 of 11 checks passed

The memo is well-structured, evidence-backed, and actionable, requiring at most light editing for polish.

Output

Why the AI SDR for small agencies fails

Sep 25, 2026 · @Martin Slaney

The reason

Small agencies win work on trust, and cold outbound only closes when someone does the follow-up. An AI SDR automates the first email, the cheap part. It hands the follow-up, which is the actual job, to a founder our own hypothesis says is too busy delivering to sell. We would be adding meetings to the one resource that is already the constraint.

Evidence

  • Referrals dominate. 11 of 14 get most new revenue (86% on average) from referrals and repeat clients. A03: "Every client we have came from someone vouching for us."
  • Outbound was tried and dropped. 9 tried it; 7 stopped within six months and none of the 7 closed an attributable deal. They used five different channels (lead-gen agency, cold-email agency, contract SDR, LinkedIn automation, DIY email), so this isn't one bad vendor.
  • Meetings weren't the problem; closing was. A05 got six or seven meetings a month and closed nothing: "Content is a trust purchase." A01: "We got meetings with people who'd never buy from us." A10: "It cost us about £400 for every meeting, and the meetings didn't close."
  • Where it works, a human does the work. Both survivors (A04, A07) have 15–20 staff and a dedicated BD person. A04: "It works because of the follow-up, not the first email" and "most agencies our size don't have someone like me."
  • The downside hits their best channel. A02 lost the founder's LinkedIn, "our best referral network"; A11's domain was spam-listed. A04 and A05 both refuse to let anything send unread.
  • Some don't need leads at all. A08, A12 and A14 turn work away. A12: "I don't need more leads, I need another designer."

Where it cuts the other way

  • The famine is real.1 6 of 14 describe feast-or-famine swings; A05 "spent April panicking" after three clients left at once.
  • The maths are forgiving. At an $18k average deal, one closed client a year covers $7.2k of subscription.
  • One buyer exists. A04 would pay $600 today if it replaced Apollo, Clay and three hours a day of research, but only with HubSpot sync and approval of every email.
  • Past failures were generic. A05's buyers wanted proof of work in their niche; better enrichment might fix that. Untested.
  • Small sample. 14 interviews, one interviewer.
  • Warm outreach worked. A05's emails to past clients won two back in a month, and she'd pay "a couple of hundred a month" for a tool that did it. That supports the thesis against cold outbound, but points at a different product.

What would prove us wrong

Agencies without a dedicated BD person closing deals from AI-sent cold email, without damage to their reputation. Concretely, in a 12-week pilot with 5 such agencies:

  • at least 3 reach a scoped proposal and at least 1 closes;2
  • no domain, inbox or LinkedIn penalties;
  • owners spend under 2 hours a week on follow-up;
  • they keep paying when the pilot ends.

Cheapest test

Pre-sell a hand-run pilot before building anything. Offer the 9 agencies with no BD person that aren't turning work away (A01–A03, A05, A06, A09–A11, A13) a 12-week pilot at $600 a month, paid upfront. We run it by hand: Apollo and Clay for prospects, LLM-drafted emails, owner approves every send, separate sending domain.

  1. Gate 1, one week: fewer than 3 pay, stop. There is no demand at this price.
  2. Gate 2, 12 weeks (covers the 6–10 week sales cycle): score against the thresholds above and log owner follow-up hours.

In parallel, offer A05's version (past-client and referrer re-engagement, about $200 a month) as a second arm. If that sells and cold doesn't, we have our answer.

Cost: roughly £1–2k in tools and a quarter of one founder's time3, against a year and most of the £800k.

What a PM had to fix

  1. 1Hypothesis stated as factReframe it as a hypothesisTargeted repair

    Source: Founders' hypothesis (from our planning doc)Small agencies live feast or famine.

    What we’d changeSay 'Famine is real for some: 6 of 14' rather than confirming the founders' hypothesis for the whole segment.

  2. 2Test or gate too weakTighten the testQuick edit

    What we’d changeRaise the bar for proving the memo wrong: one close in five agencies doesn't show the product works.

  3. 3OtherFix itQuick edit

    What we’d changeThe cost looks light for running a hand-run pilot across nine agencies. Build it up line by line.

Check by check

Got right · 11
  • Uses the supplied evidence correctlyEvery factual claim about the current situation is drawn correctly from the brief or supplied context, with accurate quotes and attributions.
  • Addresses the actual decisionThe output commits to a single reason for failure early on, addresses the founding team, and specifies the evidence and conditions that would change the call.
  • Respects explicit constraintsThe output is a memo under 600 words, covers all requested sections, and the proposals (test, thresholds) enforce the constraints in practice.
  • Identifies material uncertaintyIt names the small sample size, whether better enrichment could fix niche trust, and that the test result would resolve the uncertainty.
  • Avoids unsupported claimsInterpretations such as 'small agencies win work on trust' are clearly positioned as the reason and backed with evidence; no causal claims are presented as fact without support.
  • Produces the required deliverableThe memo includes the reason, quoted evidence, counter-evidence, a falsification condition, a cheap test, and stays under 600 words; a founding team could act on it.
  • Finds the load-bearing assumptionThe critique directly challenges the assumption that these agencies are lead-constrained, arguing that they lack the capacity to convert leads and that outbound fails without follow-up.
  • Uses the interviews faithfullyAll quoted material is taken from the transcripts or log, attributed to the correct agency, and none is invented.
  • Engages the counter-evidenceA04 and A07 are named, their shared characteristics (size, dedicated BD, follow-up effort) are identified, and the implication that the addressable segment is very small is drawn.
  • A cheap test that can actually read outThe test is a hand-run pilot that measures proposals within 12 weeks (covering the sales cycle) and uses simple gates; it is cheap and provides an early signal before full build.
  • Tigers, not paper tigersIt dismisses the worry that past failures were just due to poor tooling as untested, and surfaces the unspoken issue that a warm-outreach re-engagement tool might be a better product.

Grades and run details

Decision model 91 · LLM judge 12 of 12 checks
Decision model checks
  • passUses the supplied evidence correctly36%
  • passAddresses the actual decision94%
  • partialRespects explicit constraints22%
  • passIdentifies material uncertainty100%
  • partialAvoids unsupported claims24%
  • passProduces the required deliverable79%
  • passFinds the load-bearing assumption89%
  • passUses the interviews faithfully82%
  • passEngages the counter-evidence96%
  • passA cheap test that can actually read out89%
  • passTigers, not paper tigers98%
Run
Run
#1
Time to output
60 s
Submitted
25 Sept 2026

Results

Every setup we’ve tested on this task, across all cases and repeats, graded on the current checklist. Calibrated: the graders match our PM on 87% of checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6.1 SolAPI100.095.82None
2GPT-6 AstraChatGPT95.595.82None
3GPT-6 LunaAPI93.291.72None
4Sonnet 5.5API93.275.02None
5Gemini 3.5 Flash-LiteGemini81.875.021 capped
6Opus 5.5Claude90.991.721 capped

The PM job

Pressure-testing a proposal before committing a team to it.

Why it matters

The useful critic finds the one assumption everything rests on. Theatrical negativity is easy to generate and useless in a planning meeting.

What good looks like

  • Identifies the load-bearing assumption
  • Separates the risks that could kill it from the ones that only look scary
  • Uses the supplied evidence, not generic risks
  • Proposes the cheapest way to test the assumption

Deliberately not measured

  • Tone
  • Number of objections raised
Capability tested

Evidence-based critique

The failure we’re looking for

Theatrical negativity without evidence

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

Vanilla prompt (core) · With Roast Me skill · Staff level: a company bet with a long evidence pack

This task measures the whole setup. Tools, instructions and skills in the harness do real work here, so read the harness as carefully as the model name.