Tasks / Challenge

Challenge an idea

Can the model find the strongest reason an idea may fail, backed by evidence?

Measures the systemTask v1.0 · 2 casesDifficulty

The PM job

Pressure-testing a proposal before committing a team to it.

Why it matters

The useful critic finds the one assumption everything rests on. Theatrical negativity is easy to generate and useless in a planning meeting.

What good looks like

Identifies the load-bearing assumption
Uses the supplied evidence, not generic risks
Proposes the cheapest way to test the assumption

Deliberately not measured

Tone
Number of objections raised

Capability tested

Evidence-based critique

The failure we’re looking for

Theatrical negativity without evidence

Grading

Decision model, LLM judge and blind PM review

Variants

Vanilla prompt (core) · With Roast Me skill

This task measures the whole setup. Tools, instructions and skills in the harness do real work here, so read the harness as carefully as the model name.

Results

Every setup we’ve tested on this task, across all cases and repeats.

#	Model · Harness	Task score	Decision model	LLM judge	PM review	Runs	Critical failures	Cost / run	Latency

Case viewer

Read the brief, then put up to three outputs side by side. The outputs are the point; the scores just tell you where to look.

The brief

Our product team proposes adding a community forum to our budgeting app, described below. Challenge the proposal: write the strongest case against building it as proposed, for the product lead. Keep it under 400 words.

The proposalPenny is a UK budgeting app with 400,000 monthly active users. The proposal is a public forum inside the app with four categories (Budgeting, Saving, Debt, App help), user profiles with a chosen username, and upvotes. The goal is to raise 90-day retention from 31% to 36%. Estimated cost: one squad for a quarter, plus a full-time community moderator.

EvidenceSurvey (2,100 users): 71% agree “money is a private matter”; 12% say they would post about their own finances. Support: “How do I…” questions are 46% of our 18,000 monthly support tickets, and ten topics account for 60% of those. A competitor launched a similar forum last year; it has about 1,900 posts in total.

What a strong answer does

The privacy norm undermines participation, so the forum will not produce the retention it assumes; the real, evidenced need is self-serve help content for the top 'how do I' topics.

Case

v1.2 · synthetic · consumer, fintech