Tasks / Product Craft

Customer research call guide

Can the model write a guide that uncovers behaviour rather than opinions?

Task type v1.0 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 38% were usable with at most a quick edit.

Reliably right

Asks what people did, not what they think92% pass
Core questions ask about specific recent actions (last boxes, skipped weeks, what they did instead) rather than opinions.
Opus 5.5 · Claude · Too expensive, by default
Tests both theories fairly91% pass
Both theories get questions that could disprove them (e.g., no action on robotics, happiness after slow go-live), and open-ended timeline questions leave room for a third cause.
Sonnet 5.5 · API · Why are warehouses leaving? Two execs, two theories
Different questions for user and signer91% pass
The guide has distinct tracks: daily user questions focus on day-to-day experience and workarounds, signer questions on the decision process and alternatives.
Sonnet 5.5 · API · Why are warehouses leaving? Two execs, two theories

Where it slips

Marks what to cut if the call runs over28% pass
The guide has timed sections but does not mark which questions are must-ask or what to cut if time runs short.
Opus 5.5 · Claude · Too expensive, by default
Doesn't trust the default reason41% pass
The plan names the pre-selected default as a bias but does not use the behaviour columns to show that most 'too expensive' cancellers had stopped cooking; it only picks two contrasting individuals without the aggregate pattern.
GPT-6 Astra · ChatGPT · Too expensive, by default
Uses the supplied evidence correctly64% pass
The claim 'Customers will say what helps their negotiation' is an unsupported behavioural statement not derived from the context.
Haiku 5.5 · API · Why are warehouses leaving? Two execs, two theories

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Designed to falsify
For each learning goal, does the guide include a question whose honest answer could show the team's hypothesis is wrong?
Passes when Every learning goal has at least one disconfirming, non-leading question, such as asking about the last time they had the problem and what they did instead.
Judd AntinLenny’s Podcast
Asks what people did, not what they think
Are the core questions about specific things the person actually did recently, rather than opinions, predictions or hypotheticals?
Passes when The core of the guide asks for recent, specific instances (the last time, what happened, what they did) and follows up on them; opinions and predictions are absent or clearly secondary.
Bob MoestaLenny’s Podcast
Doesn't lead the witness
Are the questions free of leading wording, of pitching the idea, and of hints at the answer the team hopes for?
Passes when Questions are neutral and open, and the idea is not described (or only at the very end, after the problem questions).
Product and growth leadersOn Lenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Marks what to cut if the call runs over · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for retention at Sprig. The execs each have a theory about why subscribers cancel (below). Plan next week's six churn interviews: who to recruit from the cancellation list and why, and the guide the interviewers will use, in no more than 900 words. Two of the interviewers are new to it. The files are attached.

What the model was given5 items: About Sprig, exec_hypotheses.md, cancel_form.md, cancellations_last_quarter.csv, research_constraints.md

About SprigA meal-kit subscription: recipes and ingredients delivered weekly. Subscribers can skip a week or cancel any time.

exec_hypotheses.md7 lines · Download

# What the execs think (for the research)

- CEO: "It's price. Competitors discount harder."
- Head of Food: "Our recipes are too repetitive after a few months."
- Head of Ops: "Late deliveries."

Retained subscribers (for comparison): 81% cooked at least 3 recipes a week in their first month; 34% of cancellers did.

cancel_form.md3 lines · Download

# The cancel form

A single-choice list, in this order: "Too expensive" (selected by default), "Recipes got repetitive", "Moving / travelling", "Delivery problems", "Other". Free text is optional.

cancellations_last_quarter.csv300 rows · Download

cancellation,cancel_reason_selected,weeks_subscribed,deliveries_skipped_last_6_weeks,recipes_cooked_last_4_weeks,free_text
S-8000,Recipes got repetitive,47,3,0,kept piling up in the fridge
S-8001,Recipes got repetitive,16,3,2,too much food for two
S-8002,Moving / travelling,33,2,5,cheaper at the supermarket
S-8003,Too expensive (pre-selected),29,3,2,kept piling up in the fridge
S-8004,Too expensive (pre-selected),11,5,0,
S-8005,Too expensive (pre-selected),20,5,0,too much food for two
…

research_constraints.md4 lines · Download

# Research constraints

- Six calls, 30 minutes each, next week. The researcher can recruit from the cancellation list (consent to contact is on file for all of them).
- Two of the PMs running calls are new to interviewing.

What a strong answer doesThe answer key the graders mark against

Reads the cancellation data before trusting it: 'Too expensive' is the top reason (156 of 300), but it's the pre-selected first option, and 118 of those 156 had skipped two or more deliveries and cooked two or fewer recipes in the last four weeks: they'd stopped using the product before they left. Overall 204 of 300 skipped two or more deliveries. So the guide investigates what happened to the cooking habit, and price as one hypothesis to test rather than the answer. Recruits deliberately from the list across the patterns: people who stopped cooking and picked 'too expensive', people who kept cooking and still picked 'too expensive' (the real price cases), and people who said recipes got repetitive. The guide asks about specific recent weeks (the last box they skipped, what happened to the food, what they ate instead), each exec hypothesis has a question whose honest answer could disprove it, questions don't lead or mention price first, the guide fits 30 minutes with timings and marks what to cut, and gives the new interviewers prompts for digging deeper. Notes that the default on the cancel form should be removed.

Critical failures (cap the score at 40)

Builds the guide around confirming that price is the reason

Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Opus 5.5Claude94Usable with a quick edit GPT-6 AstraChatGPT85Usable with a quick edit Sonnet 5.5API79Needs more work GPT-6.1 SolAPI77Needs more work GPT-6 LunaAPI71Usable with a quick edit Haiku 5.5API69Needs more work Gemini 3.8 FlashAPI56Needs more work Gemini 3.5 Flash-LiteGemini44Needs more work

GPT-6 LunaAPI

Usable with a quick edit8 of 12 checks passed

The plan is evidence-based and actionable; a PM could use it with a quick edit to mark cuttable sections for time management.

Output

Recruit these six

Recruit all six, with the listed cancellation-form reason treated as a clue—not a confirmed cause. “Too expensive” was pre-selected, so it is especially unreliable as a standalone explanation. These interviews are for exploring different churn paths, not estimating how common each one is.

S-8004 — 11 weeks subscribed; five skips and no recipes cooked in the last four weeks; selected “Too expensive.” A low-engagement case to explore whether price was the trigger, or whether skipped deliveries and unused food came first.
S-8008 — 18 weeks; no skips and six recipes cooked; selected “Too expensive,” with “cheaper at the supermarket” in free text. An active subscriber whose account is more consistent with a price comparison worth probing.
S-8006 — 29 weeks; no skips and eight recipes cooked; selected “Too expensive,” but wrote “too much food for two.” Tests whether perceived cost may be tied to household fit or waste, even for a frequent cook.
S-8013 — 28 weeks; two skips and three recipes cooked; selected “Recipes got repetitive,” but wrote “didn’t have time to cook.” Explore how recipe experience, time, and skipped weeks related to one another.
S-8025 — 50 weeks; one skip and no recipes cooked; selected “Recipes got repetitive,” but wrote “too much food for two.” A long-tenure contrast to S-8013: investigate whether repetition was an experienced problem or a label for a broader mismatch.
S-8028 — three weeks; two skips and nine recipes cooked; selected “Delivery problems,” with “kept piling up in the fridge.” A distinct early-tenure, high-cooking case to investigate what happened with delivery and whether delivery was connected to the food backlog.

30-minute interview guide

Before the calls: Use the same guide for all six. Don’t tell participants which executive theory you’re testing. One PM moderates and the other takes notes; rotate roles so each new interviewer leads three calls. Record exact examples and the participant’s wording, not just interpretations. Don’t defend Sprig, pitch a save offer, or rush to fill silences.

0–2 min | Welcome > “Thanks for speaking with me. I’d like to understand your experience with Sprig and what led you to cancel. There are no right answers, and we’re not trying to persuade you to come back. Is it okay if I take notes?”

2–8 min | Let them tell the story > “Could you walk me through how Sprig fit into your routine, from when you joined through when you decided to cancel?”

Follow up neutrally: “What happened next?” “Can you give me a specific example?” “When did you first notice that?”

8–15 min | Reconstruct recent behavior > “Think about your last few weeks as a subscriber. What did a typical week look like?”

Ask what arrived, what they cooked, skipped, or didn’t use, and what was happening in their household or schedule. If relevant: “Can you tell me about the last delivery you remember?” Don’t assume a skip, delivery issue, or uneaten food had a particular cause.

15–23 min | Explore the cancellation decision > “When did cancelling first feel like the right choice? What was happening then?”

> “What were the main things that went into that decision?”

After the open account, ask about specifics that have come up naturally: price and alternatives; recipe choice or repetition; delivery timing or condition; cooking time; household size; food left over; and how often they skipped. Ask for concrete examples and how each factor affected the decision.

23–27 min | Check their form response > “The cancellation form shows [selected reason]. How well does that capture what was going on for you? Was anything more important?”

If they selected “Too expensive,” note that it was pre-selected; don’t treat the selection as confirmation. Ask what they compared Sprig with and what “expensive” meant in practice, without suggesting an answer.

27–30 min | Close > “If one thing had been different, might you have made a different choice—or was cancelling still the right decision?” > “What haven’t I asked that would help me understand?”

Thank them. After each call, the PMs should separately note the apparent trigger, contributing factors, and unresolved questions before comparing notes.

Grades and run details

Decision model 71 · LLM judge 69

Decision model checks

passUses the supplied evidence correctly63%
partialAddresses the actual decision25%
partialRespects explicit constraints58%
partialIdentifies material uncertainty84%
passAvoids unsupported claims85%
partialProduces the required deliverable85%
partialDoesn't trust the default reason88%
passRecruits from the data90%
partialDesigned to falsify43%
passAsks what people did, not what they think52%
passDoesn't lead the witness69%
partialMarks what to cut if the call runs over40%

Run

Run: #1
API response time: 26 s
Submitted: 6 Oct 2026

Haiku 5.5API

Needs more than a quick edit8 of 12 checks passed

The free-text claim is misleading, and the guide lacks cut marks for time management, so a PM would need to recheck the free-text analysis and add cut instructions.

Output

Churn interviews: recruiting plan and guide

What the data says (and doesn't)

Across 300 cancellations last quarter:

Reason selected	Count	Share
Too expensive (pre-selected default)	156	52%
Recipes got repetitive	52	17%
Moving / travelling	41	14%
Delivery problems	29	10%
Other	22	7%

The price number is inflated. "Too expensive" is pre-selected, so we can't tell deliberate choices from people who clicked Next. Treat it as a hypothesis to test, not a finding.
Free text points away from all three exec theories. Where people wrote something, the recurring themes were over-supply ("too much food for two," "kept piling up in the fridge") and time ("didn't have time to cook"). Neither is in the exec hypotheses.
Delivery cancellers were engaged. All 29 cooked at least 3 recipes in the last 4 weeks, and many cooked 8 or more. Late deliveries may be hurting people who want to cook.
The retention benchmark isn't testable here. The 34% figure is first-month cooking. The export only has the last 4 weeks, so we can't compare the two. Early-tenure cancellers are the closest we can get, so we'll include one.

Who to recruit (6 calls)

Recruit from the list by reason, tenure, and cooking level so each call tests a different explanation. Keep 2 backups for each slot.

#	Account	Why this person
1	S-8007 (Too expensive, 37 wks, 4 skips, 0 recipes last 4 wks)	Price claim from a long-tenure subscriber who stopped cooking. Tests whether price is the real reason or cover for disengagement.
2	S-8056 (Too expensive, 29 wks, 12 recipes last 4 wks, "cheaper at the supermarket")	Engaged cook who cites price. Tests the CEO's theory where it's most plausible.
3	S-8186 (Repetitive, 54 wks, 2 recipes)	Tests the Head of Food's theory with a long-tenure subscriber.
4	S-8115 (Delivery problems, 17 wks, 10 recipes)	Tests the Head of Ops's theory with an engaged cook.
5	S-8171 (Too expensive, 6 wks, 0 recipes, "kept piling up in the fridge")	Early-tenure, non-cooking cancel. Closest to the first-month question. Explores whether the box was used at all.
6	S-8253 (Too expensive, 32 wks, 5 skips, 1 recipe, "too much food for two")	Surfaces the over-supply theme no exec raised.

Backups: for slot 5, S-8153 (2 wks tenure, 8 recipes, delivery). For slot 6, S-8055 (Other, 5 skips, "too much food for two").

Interview guide (30 minutes)

Rules for all interviewers - Don't mention the cancel form's default or any exec theory. - Don't ask leading questions ("Was it the price?"). Ask what happened, then probe. - Let silences run. Don't fill them with suggestions. - Ask for specific examples: "Tell me about the last time…" - Don't defend Sprig or argue with the subscriber.

1. Opening (2 min) "Thanks for making time. We're trying to understand what happened with your subscription so we can improve it. There are no right or wrong answers, and nothing you say will affect your account. Is it okay if I take notes and record for accuracy?"

2. Their Sprig story (8 min) - "Take me back to when you first signed up. What did you hope Sprig would be for you?" - "How did a typical week go once the boxes started arriving?" - "What was happening in your life when you decided to cancel?" - "Walk me through the moment you decided. What happened first?"

3. The cancel form (4 min) - "On the cancel screen, you chose a reason. Can you walk me through how you picked it?" - If they say "too expensive": "What made that the best fit? Was there anything else you considered?" - If they chose something else: "What made that the closest match?" - "Was there a reason you wanted to share that the options didn't capture?"

4. Cooking and the box (6 min) - "In your first month, how many of the recipes did you actually cook?" - "What got in the way of cooking the others?" - "What happened to ingredients you didn't use? Walk me through a recent week." - "How did the portion sizes work for you?" - "How did the recipes feel after a few weeks?"

5. Price, delivery, and alternatives (5 min) - "How did the cost compare to what you'd spend otherwise?" - "You mentioned [free text, if any]. Tell me more about that." - "Have you had any issues with deliveries? What happened, and how did it affect your week?" - "Did you pause or skip boxes before cancelling? What was going on then?"

6. Counterfactual (3 min) - "If Sprig had changed one thing, would you have stayed? What would it have been?" - "What would have had to be different for you to skip only for a month instead of cancelling?"

7. Close (2 min) "Is there anything I didn't ask that you think we should know?" Thank them and confirm how the findings will be used.

Notes for the two new interviewers

Pair each new interviewer with an experienced one for the first call, with the experienced person leading.
Practise the "walk me through" and silence-holding moves beforehand. Most of the signal comes from follow-up probes.
If a subscriber gets upset, acknowledge it briefly, offer to end early, and move to the close.
Write down direct quotes, not summaries, especially about the cancel form and the portions.

After the calls

Code each interview against the three exec theories and two emerging themes (over-supply, time to cook). Report what each subscriber did and said, not just their stated reason. Flag any case where the stated reason and the story don't match, since that's the pre-selection problem showing up.

Claims the judge couldn’t find in the brief

Free text points away from all three exec theories, with recurring themes over-supply and time to cook.

Grades and run details

Decision model 83 · LLM judge 62

Decision model checks

passUses the supplied evidence correctly41%
passAddresses the actual decision37%
partialRespects explicit constraints31%
passIdentifies material uncertainty54%
passAvoids unsupported claims34%
passProduces the required deliverable38%
partialDoesn't trust the default reason33%
passRecruits from the data100%
passDesigned to falsify62%
partialAsks what people did, not what they think35%
passDoesn't lead the witness67%
partialMarks what to cut if the call runs over52%

Run

Run: #1
API response time: 46 s
Submitted: 8 Oct 2026

Gemini 3.8 FlashAPI

Needs more than a quick edit7 of 12 checks passed

The output invents a 'discount cliff' not present in the data and lacks must-ask/cut markings, so a PM would need to verify evidence and add cutting instructions.

Output

Strategic Recruitment Plan

The cancellation form pre-selects "Too expensive", severely skewing quantitative data toward price. Furthermore, while 81% of retained users cook ≥3 recipes/week in month 1, only 34% of churned users do—suggesting retention is fundamentally won or lost on habit formation, pace, and food waste guilt (evident in recurring free-text notes like "kept piling up").

To evaluate exec hypotheses (Price, Recipe Repetition, Delivery) against these underlying behavioral dynamics, we will recruit six subscribers across distinct behavioral archetypes:

ID	Weeks Subscribed	Skips (Last 6 Wks)	Cooked (Last 4 Wks)	Form Reason	Free Text	Rationale & Hypothesis Tested
:---	:---	:---	:---	:---	:---	:---
S-8056	29	1	12	Too expensive	Cheaper at supermarket	CEO (Price): High cooking engagement, low skips. Tests whether high-value cooks leave for discounted competitors vs. grocery shopping.
S-8196	57	2	1	Repetitive	[None]	Head of Food (Repetition): Long tenure; actively changed radio button to "Repetitive"; cooking volume collapsed. Tests recipe fatigue threshold.
S-8072	23	1	8	Delivery problems	[None]	Head of Ops (Delivery): High cook volume; deliberately chose "Delivery problems". Tests if logistical failures break established habits.
S-8016	14	4	0	Too expensive	Kept piling up in fridge	The "Food Guilt" Trap: High skips, zero cooking. Tests if "Too expensive" is actually guilt over uneaten food and subscription velocity.
S-8006	29	0	8	Too expensive	Too much food for two	Portion/Household Mismatch: Highly engaged, zero skips, but churned. Tests unit economics and portion sizing vs. true price sensitivity.
S-8120	4	0	10	Too expensive	Cheaper at supermarket	Early Onboarding Drop-off: Active cook (10 recipes) who dropped at Week 4 discount cliff. Tests initial value proposition and promo-hopper behavior.

Backup recruits: S-8044 (Repetitive), S-8115 (Delivery), S-8038 (Price/High engagement).

---

Interviewer Guide (30 Minutes)

> Interviewer Note (for new PMs): > 1. Never mention their cancel form response. It was pre-selected and biases answers. > 2. Focus on stories, not opinions. Don't ask "Was it too expensive?" Ask "Walk me through the last box you unpacked." > 3. Embrace 3 seconds of silence. Let the participant fill pauses.

#### 1. Welcome & Rapport (3 mins) * "Thanks for taking the time! I’m [Name], a product manager at Sprig. We’re working to make our boxes better, and we learn the most from people who decided Sprig wasn't right for them." * "There are no right or wrong answers, and you won’t hurt my feelings. I’m recording this for notes—is that okay?"

#### 2. The Early Days & Habit Formation (7 mins) Goal: Understand the baseline expectation and initial activation (testing month-1 cooking habit). * "Think back to when you first signed up for Sprig. What was going on in your life that made you want meal kits?" * "Walk me through your first couple of weeks. How often were you cooking the meals?" * "Did you cook with someone else? How did the portion sizes fit your household?"

#### 3. The Turning Point & Kitchen Reality (12 mins) Goal: Neutral deep dive to evaluate price, delivery, repetition, and food waste without leading. * "Think about the last box you received before canceling. Walk me through what happened from the moment it arrived to when you decided to cancel." * If Ops signal: "Did it arrive when and how you expected? How did that affect your week?" * If Food signal: "How did picking your meals feel toward the end compared to the beginning? Did you feel excited by the menu?" * If Fridge/Time signal: "Did you ever have ingredients left over when the next box arrived? How did you handle that?" * "I noticed you skipped a few deliveries [or never skipped]. How did skipping fit into your routine?" * "Tell me about the specific moment you went to the website to hit 'Cancel'. What tipped you over the edge that day?"

#### 4. The Alternative & True Cost (5 mins) Goal: Test CEO hypothesis (competitors vs. grocery vs. takeout). * "Now that you aren't using Sprig, what are you doing for dinner on busy weeknights?" * "Did you try or consider any other meal kits or services? Why or why not?" * "When you think about the total cost of Sprig, what felt like the biggest driver of whether it was 'worth it'?"

#### 5. Wrap-Up (3 mins) * "If you had a magic wand and could change one thing about Sprig so that it fit your life perfectly, what would it be?" * "Thank you so much for your candor. We’ll send your $30 gift card within 24 hours."

---

Post-Interview Debrief Checklist (Immediately after call)

Capture within 10 minutes in Slack/Notion: 1. Primary root cause: (e.g., Recipe boredom, Food waste guilt, Box delivery timing, Absolute price, Household mismatch). 2. True competitor: (e.g., Trader Joe’s, HelloFresh, Takeout, Cooking from pantry). 3. Exec hypothesis validation: Did this support CEO, Food, or Ops? (Yes/No/Partial + 1-sentence quote).

Claims the judge couldn’t find in the brief

S-8120 cooked 10 recipes in 4 weeks and dropped at Week 4 discount cliff.

Grades and run details

Decision model 63 · LLM judge 54

Decision model checks

failUses the supplied evidence correctly3%
passAddresses the actual decision33%
partialRespects explicit constraints48%
partialIdentifies material uncertainty53%
partialAvoids unsupported claims40%
partialProduces the required deliverable35%
partialDoesn't trust the default reason70%
passRecruits from the data99%
passDesigned to falsify48%
passAsks what people did, not what they think39%
partialDoesn't lead the witness28%
partialMarks what to cut if the call runs over60%

Run

Run: #1
API response time: 34 s
Submitted: 6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 6

Uses the supplied evidence correctlyRightMixedWrong

GPT-6 Luna · API

All factual claims about the current situation are directly supported by the supplied context.

Haiku 5.5 · API

The claim that free text points away from all three exec theories is incorrect; free text includes price-related comments like ‘cheaper at the supermarket’ that support the CEO’s theory.

Gemini 3.8 Flash · API

The claim about a 'Week 4 discount cliff' for S-8120 is invented; no discount information exists in the provided data.

Addresses the actual decisionRightRightMixed

GPT-6 Luna · API

The output commits to a clear plan: recruit these six specific subscribers and use the provided guide.

Haiku 5.5 · API

The output commits to a clear recruiting plan and interview guide, naming specific accounts and a full guide.

Gemini 3.8 Flash · API

The output does not state what result or condition would change the recruitment choice.

Identifies material uncertaintyWrongRightWrong

GPT-6 Luna · API

The output does not name specific unknowns that could change the recruitment plan or say how they would be resolved.

Haiku 5.5 · API

It identifies the pre-selection bias, notes the data gap for first-month cooking, and flags that the price number is inflated.

Gemini 3.8 Flash · API

The output does not name specific unknowns that could change the decision or say how they would be resolved.

Avoids unsupported claimsRightMixedWrong

GPT-6 Luna · API

Interpretations are clearly presented as exploration goals, not as established facts.

Haiku 5.5 · API

The statement that free text points away from all three exec theories is unsupported by the data, because some free text supports the price hypothesis.

Gemini 3.8 Flash · API

The 'Week 4 discount cliff' is an unsupported interpretation presented as fact, and no caveat is given.

Doesn't trust the default reasonWrongWrongRight

GPT-6 Luna · API

It names the pre-selected default as a bias but does not show from the file that most 'too expensive' cancellers had stopped cooking (e.g., the 118 of 156 pattern).

Haiku 5.5 · API

It names the pre-selected default but does not use the behaviour columns to demonstrate that most ‘too expensive’ cancellers had stopped cooking; it only states the number is inflated.

Gemini 3.8 Flash · API

It explicitly names the pre-selected bias and uses behavioural data to show many 'too expensive' cancellers had stopped cooking.

Recruits from the dataMixedRightRight

GPT-6 Luna · API

It picks six individuals with reasons but does not define recruitment groups (e.g., stopped cooking, still cooking but price) with how many from each.

Haiku 5.5 · API

It picks six interviewees that cover the needed behaviour patterns: stopped cooking, still cooking but price, repetitive recipes, delivery, early-tenure, and over-supply.

Gemini 3.8 Flash · API

Six distinct behavioural archetypes are drawn from the data, each linked to an exec hypothesis.

All got wrong 1

Marks what to cut if the call runs overWrongWrongWrong

GPT-6 Luna · API

The guide has timed sections that add up to 30 minutes but does not mark which questions are must-ask or what to cut if time runs short.

Haiku 5.5 · API

The guide has timed sections that add to 30 minutes but does not mark which questions to cut if time runs over.

Gemini 3.8 Flash · API

Timings add up to 30 minutes but the guide does not mark which questions are must-ask or what to cut if time runs over.

All got right 5

Respects explicit constraintsRightRightRight

GPT-6 Luna · API

The output is under 900 words, addresses the two new interviewers with role rotation and guidance, and delivers the requested plan.

Haiku 5.5 · API

The output is a plan plus guide within ~800 words, addresses new interviewers, and respects the six-call, 30-minute constraint.

Gemini 3.8 Flash · API

The output respects the 900-word limit, provides a recruitment plan and interview guide, and includes a note for new interviewers.

Produces the required deliverableRightRightRight

GPT-6 Luna · API

The output is a complete, usable plan with recruitment choices and a timed interview guide.

Haiku 5.5 · API

It delivers a complete recruiting plan and interview guide that a PM could use with light edits.

Gemini 3.8 Flash · API

It delivers a ready-to-use recruitment table and a timed interview guide for the retention PM.

Designed to falsifyRightRightRight

GPT-6 Luna · API

The guide asks open, behaviour-based questions that allow any executive hypothesis to be contradicted by honest answers.

Haiku 5.5 · API

The guide includes open questions (e.g., on cost, recipes, deliveries) whose honest answers could disprove each exec hypothesis.

Gemini 3.8 Flash · API

Each hypothesis has a question whose honest answer could disprove it, e.g. asking about what they do for dinner now instead.

Asks what people did, not what they thinkRightRightRight

GPT-6 Luna · API

Core questions ask about specific recent actions (last delivery, typical week, when they first considered cancelling) rather than opinions or hypotheticals.

Haiku 5.5 · API

Core questions ask about specific recent weeks, what happened to ingredients, and walk-throughs of events rather than opinions.

Gemini 3.8 Flash · API

Core questions ask about specific recent events like the last box received and the moment of cancellation.

Doesn't lead the witnessRightRightRight

GPT-6 Luna · API

Questions are neutral and open; the guide explicitly warns against leading or defending Sprig.

Haiku 5.5 · API

Questions are neutral (e.g., ‘Walk me through the moment you decided’) and the guide explicitly bans leading wording.

Gemini 3.8 Flash · API

The guide instructs interviewers not to mention the cancel form and questions are neutrally worded.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 98% of 42 checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	Sonnet 5.5withAPI	86.1	88.5	2	None
2	GPT-6.1 SolwithAPI	87.8	84.9	2	None
3	Opus 5.5withClaude	92.1	74.7	2	None
4	GPT-6 AstrawithChatGPT	89.7	74.5	2	None
5	Haiku 5.5withAPI	87.8	70.1	2	None
6	GPT-6 LunawithAPI	81.6	73.9	2	None
7	Gemini 3.8 FlashwithAPI	69.7	51.9	2	None
8	Gemini 3.5 Flash-LitewithGemini	60.1	26.1	2	1 capped

What you get for the cost

GPT-6.1 Sol · API scores within a point of Sonnet 5.5 · API at a third of the cost.

Best valueNobody beats these on both score and costAt least one critical failure

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
GPT-6 Luna · API	77	1.0×
Haiku 5.5 · API	77	2.3×
Gemini 3.5 Flash-Lite · Gemini	41	3.0×
Gemini 3.8 Flash · API	58	8.3×
GPT-6.1 Sol · API	85	13×
Sonnet 5.5 · API	86	43×
GPT-6 Astra · ChatGPT	81	66×
Opus 5.5 · Claude	82	71×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Preparing for customer calls.

Why it matters

Leading questions produce the answers you wanted. A good guide is designed to prove you wrong.

What good looks like

Past-behaviour questions
No leading prompts
Clear learning goals
Could disprove the team's hypothesis

Deliberately not measured

Capability tested

Question design

The failure we’re looking for

Hypothetical, leading questions

Grading

Decision model and LLM judge, calibrated against a blind PM review

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

Designed to falsify

Asks what people did, not what they think

Doesn't lead the witness

Plus our standard checks

The tasks

The outputs

Output

Recruit these six

30-minute interview guide

Grades and run details

Decision model checks

Run

Output

Churn interviews: recruiting plan and guide

What the data says (and doesn't)

Who to recruit (6 calls)

Interview guide (30 minutes)

Notes for the two new interviewers

After the calls

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Run

Output

Strategic Recruitment Plan

Interviewer Guide (30 Minutes)

Post-Interview Debrief Checklist (Immediately after call)

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Run

Check by check

Where they differ 6

All got wrong 1

All got right 5

Results

What you get for the cost

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured