Humanity’s Last PRD
Latest drop: Gemini 3.8 Flash · 1 Oct
Tasks / Challenge

Challenge an idea

Can the model find the strongest reason an idea may fail, backed by evidence?

Measures the systemTask v1.2 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 14 graded outputs by 7 models. 64% were usable with at most a quick edit.

Reliably right

  1. Addresses the actual decision100% pass
    It commits early to not committing three squads and authorizing a six-week test, and specifies what result would change that.
    GPT-6.1 Sol · API · The CEO's embedded-payments bet
  2. Identifies material uncertainty100% pass
    It names the unknowns (representativeness of payment data, large-account volume access, actual mix) and resolves them with a bounded test.
    GPT-6.1 Sol · API · The CEO's embedded-payments bet
  3. Uses the interviews faithfully100% pass
    All quotes are accurate and correctly attributed to A05, A08, A04, A07, and A12.
    GPT-6 Luna · API · An AI SDR for small agencies

Where it slips

  1. A cheap test that can actually read out64% pass
    The pre-registered gate requires signed deals and renewal rather than an early signal such as qualified meetings or proposals, so it risks being too late for a cheap read-out.
    GPT-6.1 Sol · API · An AI SDR for small agencies
  2. Re-estimates the revenue correctly71% pass
    It shows working but lands on a central $2.3M and $1M–$6M range, not roughly the $5–10M range required by the grading rubric, and its $9.1M pilot upper bound is not used as the main re-estimate.
    Sonnet 5.5 · API · The CEO's embedded-payments bet
  3. Avoids unsupported claims71% pass
    The memo claims the pilot achieved ~0.38% blended take as a fact, which is not in the evidence and is not derived from it arithmetically; it also treats the pro-rata $1.43B locked value as a hard constraint without flagging the assumption.
    Opus 5.5 · Claude · The CEO's embedded-payments bet

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

We plan to sell an AI sales-development rep, an agent that finds prospects and sends personalised outbound email, to agencies with 5–20 staff. Before we commit a year to it, find the strongest reason this fails. Use the interview material below, including the three transcripts. Write it up for the founding team: the one reason, the evidence for it (quote the interviews), where the evidence cuts the other way, what we would need to see to be proved wrong, and the cheapest test that would show it. Keep it under 600 words.

What the model was given7 items: Scenario, Founders' hypothesis (from our planning doc), Interview summary (14 agencies, 5–20 staff, last six weeks), Interview log, interview-A05-content-agency.md, interview-A04-shopify-agency.md, interview-A08-packaging-studio.md
ScenarioWe are a three-person founding team with £800k of pre-seed funding and about 20 months of runway. Planned price: $600 a month per agency. Two competitors have each raised over $10m in the last year, selling AI SDRs mostly to software companies with 50+ staff.
Founders' hypothesis (from our planning doc)“Small agencies live feast or famine. When a big client leaves they have no pipeline, because the founder is too busy delivering to sell. An AI SDR keeps the pipeline full in the background, so the famine never comes. At $600 a month, one extra client a year pays for it many times over.”
Interview summary (14 agencies, 5–20 staff, last six weeks)11 of 14 get most of their new revenue from referrals and repeat clients (86% on average across those 11). 9 had tried outbound in the last two years; 7 of those stopped within six months, and none of the 7 closed a deal they could attribute to it. The 2 who kept going (A04, A07) have 15–20 staff and a dedicated person for business development. 3 (A08, A12, A14) said they regularly turn work away. 6 described feast-or-famine swings in new business. Average deal size across the 14: $18k; typical sales cycle 6–10 weeks.
Interview logAgency (staff) · services · share of new revenue from referrals and repeat clients · outbound tried? · outcome · representative quote A01 (8) · brand and web studio · 90% · yes, cold-email agency · stopped after 3 months · "We got meetings with people who'd never buy from us." A02 (14) · performance marketing · 85% · yes, LinkedIn automation · stopped after 4 months, account restricted · "LinkedIn shut our founder's account down. That's our best referral network." A03 (6) · PR boutique · 95% · no · — · "Every client we have came from someone vouching for us." A04 (18) · Shopify development · 40% (plus 30% partners, 30% outbound) · yes, in-house BD lead · kept, 30% of new revenue · "It works because of the follow-up, not the first email." A05 (11) · content · 85% · yes, lead-gen agency · stopped after 2 months, no deals · "Content is a trust purchase." A06 (7) · video production · 85% · no · — · "Our clients find us through the videos. Someone shares one and we get a call." A07 (16) · SEO · 45% · yes, outsourced lead generation · kept · "It pays for itself, just. Most of the meetings are a waste of time, but one in ten turns into a retainer." A08 (5) · packaging design · 95% · no · — · "If you sent me ten more leads in October I'd have to say no to nine of them." A09 (12) · paid social · 75% · yes, cold email · stopped after 5 months · "The replies were mostly people asking us to take them off the list." A10 (20) · B2B marketing · 70% · yes, contract SDR · stopped after 6 months · "It cost us about £400 for every meeting, and the meetings didn't close." A11 (9) · WordPress web agency · 90% · yes, cold email · stopped after 3 months · "Our domain ended up on a spam list. Took weeks to fix." A12 (10) · branding · 85% · no · — · "We're booked out till March. I don't need more leads, I need another designer." A13 (15) · HubSpot partner · 48% (plus 40% from HubSpot's partner directory) · yes, cold email · stopped after 4 months · "HubSpot sends us more than we can handle in a good quarter." A14 (6) · UX research · 90% · no · — · "We're small on purpose. We say no to about a third of enquiries."
interview-A05-content-agency.md31 lines · Download
# Interview A05: content agency, 11 staff
Participant: founder and managing director. Interviewer: our co-founder. 34 minutes, lightly edited.

**Interviewer:** Where did your last five clients come from?

**Participant:** Let me think. Two were old clients coming back: one had changed jobs and brought us into her new company. Two were introductions, one from a web agency we partner with and one from a client's CFO who'd seen our work. The fifth found us through a talk I gave at a SaaS marketing meetup. So none of them from anything you'd call outbound.

…
interview-A04-shopify-agency.md23 lines · Download
# Interview A04: Shopify development agency, 18 staff
Participant: head of business development. Interviewer: our co-founder. 29 minutes, lightly edited.

**Interviewer:** Tell me how new business works for you.

**Participant:** We're a bit unusual for an agency our size. I'm a full-time business-development person, and outbound is about 30% of our new revenue. Referrals and repeat work are still the majority, maybe 40% referrals and the rest partners: Shopify's partner directory and a couple of app companies who send us projects.

…
interview-A08-packaging-studio.md23 lines · Download
# Interview A08: packaging design studio, 5 staff
Participant: founder and creative director. Interviewer: our co-founder. 22 minutes, lightly edited.

**Interviewer:** How do new clients find you?

**Participant:** Almost entirely word of mouth. Food and drink brands talk to each other. Someone launches a range, their packaging does well on the shelf, and the next founder asks who did it. I'd say 95% referral. We've never done any outbound.

…
What a strong answer doesThe answer key the graders mark against

Names the load-bearing assumption as 'small agencies' growth is limited by a shortage of new leads': the interviews say it is limited by trust and capacity. 11 of 14 win most work through referrals and repeat clients; 7 of 9 who tried outbound quit within six months with no attributable deals, for reasons tied to trust (A05's 'content is a trust purchase', A01's meetings with people who'd never buy) and reputation risk (A02's restricted LinkedIn account, A11's spam listing); 3 turn work away (A08: 'I'd have to say no to nine of them'). Quotes the interviews accurately and attributes them correctly. Engages the counter-evidence honestly: outbound works for A04 and A07, but both are 15–20 staff with a dedicated BD owner, already use tools like Apollo and Clay, and credit the follow-up rather than the first email, so the segment is small, tooled and not what the product automates. Treats feast or famine as a real pain but notes agencies fix it through past clients and referrals (A05's reactivation emails, and her willingness to pay for help keeping in touch with past clients), pointing at a different product. Says what would prove the challenge wrong (for example, a meaningful share of 10–20-staff agencies with a BD owner and budget, or a concierge outbound test producing qualified meetings that turn into proposals), and designs a cheap test that measures an early signal, since a 6–10 week sales cycle means a six-week test cannot observe won deals.

Critical failures (cap the score at 40)
  • Invents interview quotes or attributes a quote to the wrong agency
  • Treats the founders' feast-or-famine hypothesis as confirmed by the interviews
Case

Staff-level · v1.10 · synthetic · B2B, go-to-market, customer research

The outputs

7 setups answered this brief. Pick up to three to read side by side.

GPT-6 LunaAPI100Usable with a quick editGPT-6.1 SolAPI96Usable with a quick editGPT-6 AstraChatGPT94Usable with a quick editSonnet 5.5API71Needs more workGemini 3.8 FlashAPI56Needs more workOpus 5.5Claude40Critical failureGemini 3.5 Flash-LiteGemini40Critical failure
GPT-6 AstraChatGPT

Usable with a quick edit10 of 11 checks passed

The memo is structurally complete and well‑reasoned; a quick edit to shift the test gate to an early signal (e.g., qualified proposals) would make it fully usable without rechecking evidence or redoing the reasoning.

Output

To: Founding team Subject: We risk automating a channel our customers cannot turn into revenue

Do not commit a year to this product yet. The strongest reason it fails is that finding prospects and sending personalised emails does not remove the constraint on small agencies’ sales: the human work of establishing trust and converting interest. Our promise assumes an agent can keep revenue flowing while the founder stays busy delivering. The interviews suggest we would generate more work for that same founder.

Eleven of 14 agencies get most new revenue from referrals and repeat clients, averaging 86% across those eleven. Of nine that tried outbound, seven stopped within six months without an attributable deal. This is more than unfamiliarity with the channel: most have bought or attempted it and abandoned it.

A05 shows the mechanism. Its lead-generation agency booked six or seven meetings a month, but prospects lacked budgets or wanted evidence of relevant experience. With a referred prospect, “the first call was about scope, not about whether we were any good”. Personalisation may earn attention; these interviews do not establish that it supplies the credibility needed to buy.

The successful case makes the constraint clearer. A04 generates 30% of new revenue through outbound, but has a dedicated BD lead who follows emails with site teardowns, videos and calls: “They don’t have the time to follow up, and the follow-up is the job.” Automating the first step leaves that job with the founder.

Some agencies do not need additional demand at all. A08 says, “Getting them isn’t. Handling them is,” and needs a senior designer. Three agencies regularly turn work away. Feast or famine is not a universal unmet need.

What cuts against this conclusion

Six agencies do report feast-or-famine swings. A04 and A07 prove outbound can work, although both have dedicated business-development capacity. A04 would pay $600 if we replaced research tools, saved time, integrated with HubSpot and required email approval. That supports a narrower research assistant proposition.

Past failures also do not prove our product would fail: targeting and execution may improve, and A05’s two-month trial barely covered the typical 6–10-week sales cycle. Fourteen interviews cannot establish market prevalence.

But “one extra client pays for it” skips the disputed step: winning that client. The $7,200 annual subscription is 40% of the $18,000 average deal’s revenue, before delivery costs and selling time.

What would prove us wrong, and the cheapest test

Run a paid concierge pilot before building. Recruit five agencies with spare delivery capacity, a stated growth goal and no dedicated BD person. Charge the planned $600 monthly price. Using existing tools, manually deliver prospect research and personalised drafts, with approval before sending. Founders must handle follow-up; we must not secretly supply the missing salesperson.

Run for 12 weeks to cover the stated sales cycle.1 Agree a founder-time ceiling upfront, record all selling time and track qualified opportunities through to attributable signed deals, contribution margin and renewal.

A proposed gate: at least three of five win business whose expected contribution covers subscription and selling costs, remain within the time ceiling and renew at full price. Meetings alone do not pass.

If only agencies with BD staff succeed, narrow the customer and promise. A05’s request for past-client reminders suggests another test, but “I’d pay for that tomorrow” is not payment. Do not turn one appealing interview into the next year-long commitment.

What a PM had to fix

  1. 1Test or gate too weakTighten the testQuick edit

    What we’d changeSigning and renewing inside 12 weeks is a bar that quietly favours the memo's own conclusion. Extend the window or add a leading indicator such as qualified proposals, and include the risk to agencies' LinkedIn channel.

Grades and run details

Decision model 95 · LLM judge 11 of 12 checks
Decision model checks
  • passUses the supplied evidence correctly67%
  • passAddresses the actual decision99%
  • passRespects explicit constraints39%
  • passIdentifies material uncertainty99%
  • passAvoids unsupported claims20%
  • passProduces the required deliverable87%
  • passFinds the load-bearing assumption99%
  • passUses the interviews faithfully79%
  • passEngages the counter-evidence79%
  • partialA cheap test that can actually read out33%
  • passTigers, not paper tigers98%
Run
Run
#1
Time to output
72 s
Submitted
25 Sept 2026
GPT-6.1 SolAPI

Usable with a quick edit10 of 11 checks passed

The core answer, evidence, and risk triage are usable as-is; the only needed edit is shifting the pilot gate to an earlier observable signal.

Output

Strongest reason this fails: you automate prospecting, but prospecting is not the binding constraint

The proposed AI SDR supplies cold introductions. Most agencies interviewed win through transferred trust; converting cold prospects requires substantial human follow-up that busy founders cannot supply. The product therefore risks delivering meetings rather than revenue—and being cancelled before it earns its keep.

This is a mismatch with the promised outcome, not proof that AI cannot write good emails. Cheaper, more personalised outreach does not automatically create trust, budget or founder time.

Evidence

  • Existing behaviour is a warning: 11 of 14 agencies get most new revenue from referrals and repeat clients—86% on average within that group. Nine tried outbound; seven stopped within six months, with no attributable deals.
  • A05 confirms the pain but rejects the proposed solution. Its founder described “feast or famine,” yet previous outbound generated six or seven meetings monthly without a sale: “Most of them didn’t have a budget for content.” Introductions worked because “they’d already been told we were good.” After losing three clients, personal notes to former clients brought two back within a month. On pricing: “Six hundred for cold outbound, no.”
  • Even the strongest success case exposes the missing work. A04 gets 30% of new revenue from outbound, but employs a full-time BD lead: “What wins deals is that I then send them a teardown of their site, a short video, and I’m on the phone with them within a week.” For typical founders, “They don’t have the time to follow up, and the follow-up is the job.”
  • Some agencies do not need more demand. A08: “Getting them isn’t [a problem]. Handling them is.” Its answer to the product was: “It would make my problem worse.” Three agencies regularly turn work away.

“One extra client pays for it” only holds if the tool causes an incremental, profitable win—not merely a meeting—and the agency can serve it.

Where the evidence cuts the other way

Six agencies report feast-or-famine swings. A04 would pay $600 if the tool replaced research tools and saved three hours daily, with HubSpot integration and email approval. A07 says outbound “pays for itself, just.”

These suggest a viable BD-assistance niche, particularly among larger agencies with a dedicated salesperson. They do not establish demand for autonomous pipeline generation across 5–20-person agencies.

The interviews are also a small sample, and failed historical outbound does not prove better targeting cannot work. A05’s two-month trial barely covered a typical sales cycle.

What would prove us wrong

Several agencies without dedicated BD staff pay $600, convert agent-sourced cold prospects into profitable clients with little founder follow-up, and renew. Success only among A04-like firms would support a narrower product, not the current thesis.

Cheapest decisive test

Run a 12–16-week paid concierge pilot with six agencies that genuinely need clients, primarily without BD staff. Charge $600 monthly; use existing prospecting and email tools rather than build software. Obtain sending approval and protect their primary domains.

Track qualified opportunities, founder follow-up hours, attributable signed deals, delivery capacity and paid renewals—not replies or booked meetings. Pre-register a continuation gate: at least three agencies close an attributable profitable deal and renew, within a founder-time limit they set beforehand.

Failure to recruit paying pilots is useful evidence itself. Do this before committing a year.

Grades and run details

Decision model 100 · LLM judge 11 of 12 checks
Decision model checks
  • passUses the supplied evidence correctly79%
  • passAddresses the actual decision97%
  • passRespects explicit constraints41%
  • passIdentifies material uncertainty99%
  • passAvoids unsupported claims46%
  • passProduces the required deliverable88%
  • passFinds the load-bearing assumption100%
  • passUses the interviews faithfully84%
  • passEngages the counter-evidence89%
  • passA cheap test that can actually read out68%
  • passTigers, not paper tigers97%
Run
Run
#1
API response time
17 s
Submitted
29 Sept 2026
Gemini 3.8 FlashAPI

Needs more than a quick edit3 of 11 checks passed

A PM would need to re-check evidence and repair the test before acting. The core argument is useful but not reliable as written.

Output

The One Reason This Fails

Agency services are high-trust purchases that cold automated outreach cannot convert without intensive, bespoke human follow-up—which time-poor founders cannot provide. Automating email generation solves top-of-funnel activity, but the actual point of failure is trust and follow-up capacity.

---

The Evidence

1. Zero conversion for founder-led shops: 7 of 9 agencies that tried outbound stopped within six months having closed zero deals. 2. Cold outreach delivers the wrong buyers: Services require credibility upfront. * A05 (11 staff): “Content is a trust purchase... The people on those calls had no idea who we were and, honestly, most of them didn’t have a budget.” * A01 (8 staff): “We got meetings with people who’d never buy from us.” * A10 (20 staff): “Cost us about £400 for every meeting, and the meetings didn’t close.” 3. The constraint is manual follow-up, not initial emails: * A04 (18 staff, dedicated BD): “It works because of the follow-up, not the first email... What wins deals is that I then send them a teardown of their site, a short video... The owner does sales on the side. They don't have the time to follow up, and the follow-up is the job.” 4. Asymmetric downside risk: Small agencies live on reputation. * A11: “Our domain ended up on a spam list.” * A02: “LinkedIn shut our founder’s account down. That’s our best referral network.” * A05: “I’d worry about our name going out on emails I haven’t read.”

---

Where the Evidence Cuts the Other Way

  • Feast-and-famine is real: 6 of 14 experience painful dry spells. A05 panicked when three clients left simultaneously.
  • Outbound works at scale with dedicated staff: A04 generates 30% of revenue via outbound; A07 makes it pay (“one in ten turns into a retainer”).
  • Willingness to pay exists if positioned right: A04 would pay $600/month if it replaced Apollo/Clay and saved research hours. A05 would pay $200/month for an automated tool to reactivate past clients and referral networks.

---

What We Would Need to See to Be Proved Wrong

  1. A solo founder (with no dedicated BD rep) closing an outbound deal with typical agency pricing ($18k) using standard outbound messaging, without spending >5 hours/week on bespoke pitch work.
  2. Willingness of agency owners to allow an autonomous agent to send cold outbound without manual pre-approval of every single touchpoint.

---

The Cheapest Test to Prove It

A 3-week "Wizard of Oz" Concierge Test with 3 Agencies (Cost: <$500): * Partner with 3 agencies suffering feast-or-famine (e.g., A05, A09, A10) who will commit to running a free, 30-day outbound campaign. * Manually emulate the AI SDR: scrape prospect lists, generate personalized emails, and handle automated sequencing from a secondary domain. * Kill criteria: If after booking 10–15 meetings across these agencies, zero advance to contract negotiations—or founders drop out because they cannot keep up with follow-ups and custom scoping—kill the product.

(Pivot signal: A05 offered to pay immediately for an AI assistant that drafts warm re-engagement notes to past clients and referral partners. That solves the same famine problem without the friction of cold outbound.)

Claims the judge couldn’t find in the brief

  • 7 of 9 agencies that tried outbound stopped within six months having closed zero deals.
  • 6 of 14 agencies experience painful dry spells.
  • A04 generates 30% of revenue via outbound.
  • A05, A09, and A10 are agencies suffering feast-or-famine.

Grades and run details

Decision model 95 · LLM judge 3 of 12 checks
Decision model checks
  • passUses the supplied evidence correctly44%
  • passAddresses the actual decision96%
  • passRespects explicit constraints36%
  • passIdentifies material uncertainty97%
  • partialAvoids unsupported claims23%
  • passProduces the required deliverable71%
  • passFinds the load-bearing assumption26%
  • passUses the interviews faithfully74%
  • passEngages the counter-evidence79%
  • passA cheap test that can actually read out94%
  • passTigers, not paper tigers94%
Run
Run
#1
API response time
22 s
Submitted
1 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 8

Uses the supplied evidence correctlyRightRightMixed
GPT-6 Astra · ChatGPT

All factual claims about the current situation are drawn accurately from the brief and supplied context.

GPT-6.1 Sol · API

The facts, figures, and quotes come from the supplied summary and transcripts without inventing current-situation claims.

Gemini 3.8 Flash · API

It misstates or overreaches several current-situation facts, including 'zero deals', 'painful dry spells', A04's 30% of revenue, and A09/A10 as feast-or-famine agencies.

Respects explicit constraintsRightRightMixed
GPT-6 Astra · ChatGPT

The memo is under 600 words, addresses the founding team, and includes the required one reason, evidence, counter-evidence, proof-wrong condition, and cheapest test.

GPT-6.1 Sol · API

It is written for the founding team, under 600 words, and includes the requested one reason, evidence, counter-evidence, falsification condition, and test.

Gemini 3.8 Flash · API

The required sections and length are present, but the proposed test is internally inconsistent (3-week test vs 30-day campaign) and would not reliably enforce the brief's cheap-test requirement.

Avoids unsupported claimsRightRightWrong
GPT-6 Astra · ChatGPT

Interpretations are clearly labelled as hypotheses or suggestions, not presented as established facts.

GPT-6.1 Sol · API

Interpretations and forecasts are framed as risks or suggestions rather than established fact.

Gemini 3.8 Flash · API

It presents unsupported or overconfident claims as fact, especially the feast-or-famine status of A09/A10 and the zero-deal characterisation.

Produces the required deliverableRightRightMixed
GPT-6 Astra · ChatGPT

The memo is a complete, usable document in the requested form, length, and reader, requiring only light editing.

GPT-6.1 Sol · API

The requested memo is complete, usable, and within the requested length.

Gemini 3.8 Flash · API

Although structured as requested, it is not usable without correcting evidence errors and the contradictory test design.

Finds the load-bearing assumptionRightRightMixed
GPT-6 Astra · ChatGPT

The critique centers on the assumption that small agencies' growth is limited by a shortage of new leads, showing trust and capacity as the real constraints.

GPT-6.1 Sol · API

It makes lead-constrained growth the central challenge and supports it with referral, abandoned-outbound, and turning-work-away evidence.

Gemini 3.8 Flash · API

It focuses on trust and follow-up capacity but does not make lead-constrained growth the central challenge or use the turning-work-away evidence that directly refutes the founders' assumption.

Engages the counter-evidenceRightRightMixed
GPT-6 Astra · ChatGPT

A04 and A07 are named, their dedicated BD capacity and follow-up are highlighted, and the implication of a narrow addressable segment is stated.

GPT-6.1 Sol · API

It names A04 and A07, identifies the larger dedicated-BD agency pattern and follow-up dependence, and concludes the viable niche is narrower than the thesis.

Gemini 3.8 Flash · API

It names A04 and A07 but does not fully explain what sets them apart or the implication for the addressable segment, such as A07's dedicated BD owner and existing tooling.

A cheap test that can actually read outWrongMixedMixed
GPT-6 Astra · ChatGPT

The proposed test gate relies on signed deals (won business), but a 6–10 week sales cycle means the 12-week test likely cannot observe this late signal; the criteria require measuring an early signal like qualified meetings or proposals.

GPT-6.1 Sol · API

The pre-registered gate requires signed deals and renewal rather than an early signal such as qualified meetings or proposals, so it risks being too late for a cheap read-out.

Gemini 3.8 Flash · API

The test is cheap in spirit but inconsistent and not clearly observable within the 6–10 week sales cycle, using contract negotiations after a 30-day campaign.

Tigers, not paper tigersRightRightMixed
GPT-6 Astra · ChatGPT

It dismisses the worry that a small sample invalidates the finding by noting that 14 interviews cannot establish prevalence, and surfaces the unspoken issue that outbound adds follow‑up work the founder cannot do.

GPT-6.1 Sol · API

It dismisses email quality and small-sample objections as less decisive while surfacing the unspoken founder-follow-up and delivery-capacity constraints.

Gemini 3.8 Flash · API

It does not triage risks or dismiss a plausible worry; it leaves capacity/turning-work-away and the small addressable segment underdeveloped.

All got right 3

Addresses the actual decisionRightRightRight
GPT-6 Astra · ChatGPT

The output gives a clear 'do not commit a year' answer early, framed for the founding team, and specifies what would change it.

GPT-6.1 Sol · API

It commits early to one answer—prospecting is not the binding constraint—and states what evidence would change it.

Gemini 3.8 Flash · API

It commits early to one failure reason for the founding team and states conditions that would prove the challenge wrong.

Identifies material uncertaintyRightRightRight
GPT-6 Astra · ChatGPT

It acknowledges that past failures do not prove the product will fail, the small sample limits prevalence estimates, and states what results would change the call.

GPT-6.1 Sol · API

It names the small sample, historical outbound not proving better targeting impossible, and the condition that would prove the thesis wrong.

Gemini 3.8 Flash · API

It names material unknowns—solo-founder conversion and autonomy tolerance—and gives a test and kill criteria to resolve them.

Uses the interviews faithfullyRightRightRight
GPT-6 Astra · ChatGPT

Quotes and paraphrases from A05, A04, A08 are accurate and correctly attributed, with no invented material.

GPT-6.1 Sol · API

Quotes are accurate and correctly attributed to A04, A05, A07, and A08.

Gemini 3.8 Flash · API

The quoted interview material is attributed to the right agencies and is accurate apart from minor ellipses and wording.

Results

Every setup we’ve tested on this task, across all cases and repeats, graded on the current checklist. Calibrated: the graders match our PM on 88% of checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6.1 SolAPI100.095.82None
2GPT-6 AstraChatGPT93.295.82None
3GPT-6 LunaAPI97.791.72None
4Sonnet 5.5API90.975.02None
5Gemini 3.5 Flash-LiteGemini88.675.021 capped
6Opus 5.5Claude93.291.721 capped
7Gemini 3.8 FlashAPI93.237.521 capped

The PM job

Pressure-testing a proposal before committing a team to it.

Why it matters

The useful critic finds the one assumption everything rests on. Theatrical negativity is easy to generate and useless in a planning meeting.

What good looks like

  • Identifies the load-bearing assumption
  • Separates the risks that could kill it from the ones that only look scary
  • Uses the supplied evidence, not generic risks
  • Proposes the cheapest way to test the assumption

Deliberately not measured

  • Tone
  • Number of objections raised
Capability tested

Evidence-based critique

The failure we’re looking for

Theatrical negativity without evidence

Grading

Decision model and LLM judge, calibrated against a blind PM review

Variants

Vanilla prompt (core) · With Roast Me skill · Staff level: a company bet with a long evidence pack

This task measures the whole setup. Tools, instructions and skills in the harness do real work here, so read the harness as carefully as the model name.