Tasks / Product Craft

Customer research call guide

Can the model write a guide that uncovers behaviour rather than opinions?

Task type v1.0 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 38% were usable with at most a quick edit.

Reliably right

Asks what people did, not what they think92% pass
Core questions ask about specific recent actions (last boxes, skipped weeks, what they did instead) rather than opinions.
Opus 5.5 · Claude · Too expensive, by default
Tests both theories fairly91% pass
Both theories get questions that could disprove them (e.g., no action on robotics, happiness after slow go-live), and open-ended timeline questions leave room for a third cause.
Sonnet 5.5 · API · Why are warehouses leaving? Two execs, two theories
Different questions for user and signer91% pass
The guide has distinct tracks: daily user questions focus on day-to-day experience and workarounds, signer questions on the decision process and alternatives.
Sonnet 5.5 · API · Why are warehouses leaving? Two execs, two theories

Where it slips

Marks what to cut if the call runs over28% pass
The guide has timed sections but does not mark which questions are must-ask or what to cut if time runs short.
Opus 5.5 · Claude · Too expensive, by default
Doesn't trust the default reason41% pass
The plan names the pre-selected default as a bias but does not use the behaviour columns to show that most 'too expensive' cancellers had stopped cooking; it only picks two contrasting individuals without the aggregate pattern.
GPT-6 Astra · ChatGPT · Too expensive, by default
Uses the supplied evidence correctly64% pass
The claim 'Customers will say what helps their negotiation' is an unsupported behavioural statement not derived from the context.
Haiku 5.5 · API · Why are warehouses leaving? Two execs, two theories

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Designed to falsify
For each learning goal, does the guide include a question whose honest answer could show the team's hypothesis is wrong?
Passes when Every learning goal has at least one disconfirming, non-leading question, such as asking about the last time they had the problem and what they did instead.
Judd AntinLenny’s Podcast
Asks what people did, not what they think
Are the core questions about specific things the person actually did recently, rather than opinions, predictions or hypotheticals?
Passes when The core of the guide asks for recent, specific instances (the last time, what happened, what they did) and follows up on them; opinions and predictions are absent or clearly secondary.
Bob MoestaLenny’s Podcast
Doesn't lead the witness
Are the questions free of leading wording, of pitching the idea, and of hints at the answer the team hopes for?
Passes when Questions are neutral and open, and the idea is not described (or only at the very end, after the problem questions).
Product and growth leadersOn Lenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Marks what to cut if the call runs over · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for retention at Sprig. The execs each have a theory about why subscribers cancel (below). Plan next week's six churn interviews: who to recruit from the cancellation list and why, and the guide the interviewers will use, in no more than 900 words. Two of the interviewers are new to it. The files are attached.

What the model was given5 items: About Sprig, exec_hypotheses.md, cancel_form.md, cancellations_last_quarter.csv, research_constraints.md

About SprigA meal-kit subscription: recipes and ingredients delivered weekly. Subscribers can skip a week or cancel any time.

exec_hypotheses.md7 lines · Download

# What the execs think (for the research)

- CEO: "It's price. Competitors discount harder."
- Head of Food: "Our recipes are too repetitive after a few months."
- Head of Ops: "Late deliveries."

Retained subscribers (for comparison): 81% cooked at least 3 recipes a week in their first month; 34% of cancellers did.

cancel_form.md3 lines · Download

# The cancel form

A single-choice list, in this order: "Too expensive" (selected by default), "Recipes got repetitive", "Moving / travelling", "Delivery problems", "Other". Free text is optional.

cancellations_last_quarter.csv300 rows · Download

cancellation,cancel_reason_selected,weeks_subscribed,deliveries_skipped_last_6_weeks,recipes_cooked_last_4_weeks,free_text
S-8000,Recipes got repetitive,47,3,0,kept piling up in the fridge
S-8001,Recipes got repetitive,16,3,2,too much food for two
S-8002,Moving / travelling,33,2,5,cheaper at the supermarket
S-8003,Too expensive (pre-selected),29,3,2,kept piling up in the fridge
S-8004,Too expensive (pre-selected),11,5,0,
S-8005,Too expensive (pre-selected),20,5,0,too much food for two
…

research_constraints.md4 lines · Download

# Research constraints

- Six calls, 30 minutes each, next week. The researcher can recruit from the cancellation list (consent to contact is on file for all of them).
- Two of the PMs running calls are new to interviewing.

What a strong answer doesThe answer key the graders mark against

Reads the cancellation data before trusting it: 'Too expensive' is the top reason (156 of 300), but it's the pre-selected first option, and 118 of those 156 had skipped two or more deliveries and cooked two or fewer recipes in the last four weeks: they'd stopped using the product before they left. Overall 204 of 300 skipped two or more deliveries. So the guide investigates what happened to the cooking habit, and price as one hypothesis to test rather than the answer. Recruits deliberately from the list across the patterns: people who stopped cooking and picked 'too expensive', people who kept cooking and still picked 'too expensive' (the real price cases), and people who said recipes got repetitive. The guide asks about specific recent weeks (the last box they skipped, what happened to the food, what they ate instead), each exec hypothesis has a question whose honest answer could disprove it, questions don't lead or mention price first, the guide fits 30 minutes with timings and marks what to cut, and gives the new interviewers prompts for digging deeper. Notes that the default on the cancel form should be removed.

Critical failures (cap the score at 40)

Builds the guide around confirming that price is the reason

Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Opus 5.5Claude94Usable with a quick edit GPT-6 AstraChatGPT85Usable with a quick edit Sonnet 5.5API79Needs more work GPT-6.1 SolAPI77Needs more work GPT-6 LunaAPI71Usable with a quick edit Haiku 5.5API69Needs more work Gemini 3.8 FlashAPI56Needs more work Gemini 3.5 Flash-LiteGemini44Needs more work

Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit4 of 12 checks passed

The output contains a factual error (12 recipes a week) and leading questions that would require rework before use. A PM would need to re-check the data and rewrite several questions.

Output

Sprig Retention: Churn Interview Plan (Next Week)

To stress-test our executive hypotheses (CEO: Price/Competitors; Head of Food: Repetitive recipes; Head of Ops: Late deliveries) and address the alarming baseline metric—only 34% of cancellers cooked $\ge$3 recipes/week in month one, versus 81% of retained users—we have selected a targeted cross-section of 6 users from last quarter’s cancellation dataset.

---

Part 1: Recruitment Plan (Who & Why)

We need a mix of profiles to isolate the real drivers of churn while accommodating our two interviewers who are new to the process (we will pair them with experienced leads or give them highly structured transcripts).

1. S-8028 (Delivery Problems / High Engagement) * Profile: Subscribed 3 weeks, skipped 2, cooked 9 recipes (heavy initial usage). Reason: Delivery problems. Free text: "kept piling up in the fridge". * Why: Tests Head of Ops' delivery hypothesis against actual usage. With 9 recipes cooked, they were engaged until logistics broke down. Good for new interviewers (clear timeline). 2. S-8056 (Price / Power User / Churn after 29 weeks) * Profile: Subscribed 29 weeks, skipped 1, cooked 12 recipes in the last 4 weeks. Reason: Too expensive (default). Free text: "cheaper at the supermarket". * Why: Tests CEO's price hypothesis. Crucially, they were cooking 12 recipes a week right up until cancellation—meaning value was high, but price sensitivity or a competitor offer triggered the exit. 3. S-8035 (Recipes / Long-term Churn) * Profile: Subscribed 56 weeks, skipped 1, cooked 2 recipes. Reason: Recipes got repetitive. Free text: "didn't have time to cook". * Why: Directly tests Head of Food’s hypothesis on long-term fatigue. Shows how a loyal user of over a year slowly disengaged. 4. S-8005 (Price / Early Drop-off / Zero Engagement) * Profile: Subscribed 20 weeks, skipped 5, cooked 0 recipes. Reason: Too expensive. Free text: "too much food for two". * Why: Represents the bulk of cancellers who fail activation (0 recipes cooked). Unpacks whether "Too expensive" is a proxy for value failure (didn't use it, so it felt like a waste). 5. S-8115 (Delivery / Mid-term / High Usage) * Profile: Subscribed 17 weeks, skipped 2, cooked 10 recipes. Reason: Delivery problems. Free text: "cheaper at the supermarket". * Why: Combines delivery issues with secondary price/supermarket comparison. Great for testing if operational friction causes users to re-evaluate overall cost. 6. S-8214 (Recipes / Mid-term / Moderate Usage) * Profile: Subscribed 31 weeks, skipped 3, cooked 3 recipes. Reason: Recipes got repetitive. Free text: "too much food for two". * Why: Explores the intersection of recipe fatigue and household sizing ("too much food for two"), providing actionable product feedback for the Head of Food.

---

Part 2: 30-Minute Interview Guide

Designed for clarity, supporting our two new interviewers with clear guardrails against leading questions.

Phase 1: Introduction & Context (5 mins)

Interviewer Script: "Hi [Name], thanks for taking 30 minutes. I’m with Sprig. We’re not here to win you back or sell you anything—I genuinely want to understand your experience so we can fix things for current subscribers. There are no right or wrong answers."
Warm-up: "Take me back to when you first signed up. What were you hoping Sprig would solve for you?"

Phase 2: The Journey & Engagement (10 mins)

(Focusing on the gap between our 81% retained benchmark and their early experience) * "Looking back at your first month, how easy or hard was it to get through [X] recipes a week?" * Probe (for New Interviewers): Listen for signals matching the data (e.g., "I see you skipped a few weeks around month two..."). Ask: "What caused you to skip those weeks?" * "When you thought about skipping or pausing, what usually drove that decision? (e.g., schedule, food piling up, getting bored)?"

Phase 3: Stress-Testing Executive Hypotheses (10 mins)

Testing Delivery (Ops): "Did delivery timing or box condition ever impact your cooking plans? Can you share a specific instance?"
Testing Recipes & Portions (Food): "You mentioned [repetitive recipes / too much food for two]. At what point did the menu start feeling stale? Were there specific meals you started dreading?"
Testing Price & Alternatives (CEO): "When you selected 'Too expensive' / noted supermarket prices, what were you comparing Sprig to? Did a competitor offer trigger the switch, or did the value equation shift over time?"

Phase 4: The Final Decision & Wrap-Up (5 mins)

"Take me to the exact week you decided to cancel. What was the final straw?"
"If you could wave a magic wand and change one thing about Sprig the day before you cancelled, what would it have been?"
Close: "Thank you so much for your candor. This is immensely helpful for our team."

Claims the judge couldn’t find in the brief

S-8056 was cooking 12 recipes a week right up until cancellation.

Grades and run details

Decision model 63 · LLM judge 31

Decision model checks

passUses the supplied evidence correctly36%
partialAddresses the actual decision24%
partialRespects explicit constraints68%
partialIdentifies material uncertainty76%
passAvoids unsupported claims24%
partialProduces the required deliverable80%
partialDoesn't trust the default reason88%
passRecruits from the data72%
partialDesigned to falsify57%
partialAsks what people did, not what they think82%
partialDoesn't lead the witness46%
partialMarks what to cut if the call runs over57%

Run

Run: #1
Time to output: 6 s
Submitted: 5 Oct 2026

Gemini 3.8 FlashAPI

Needs more than a quick edit7 of 12 checks passed

The output invents a 'discount cliff' not present in the data and lacks must-ask/cut markings, so a PM would need to verify evidence and add cutting instructions.

Output

Strategic Recruitment Plan

The cancellation form pre-selects "Too expensive", severely skewing quantitative data toward price. Furthermore, while 81% of retained users cook ≥3 recipes/week in month 1, only 34% of churned users do—suggesting retention is fundamentally won or lost on habit formation, pace, and food waste guilt (evident in recurring free-text notes like "kept piling up").

To evaluate exec hypotheses (Price, Recipe Repetition, Delivery) against these underlying behavioral dynamics, we will recruit six subscribers across distinct behavioral archetypes:

ID	Weeks Subscribed	Skips (Last 6 Wks)	Cooked (Last 4 Wks)	Form Reason	Free Text	Rationale & Hypothesis Tested
:---	:---	:---	:---	:---	:---	:---
S-8056	29	1	12	Too expensive	Cheaper at supermarket	CEO (Price): High cooking engagement, low skips. Tests whether high-value cooks leave for discounted competitors vs. grocery shopping.
S-8196	57	2	1	Repetitive	[None]	Head of Food (Repetition): Long tenure; actively changed radio button to "Repetitive"; cooking volume collapsed. Tests recipe fatigue threshold.
S-8072	23	1	8	Delivery problems	[None]	Head of Ops (Delivery): High cook volume; deliberately chose "Delivery problems". Tests if logistical failures break established habits.
S-8016	14	4	0	Too expensive	Kept piling up in fridge	The "Food Guilt" Trap: High skips, zero cooking. Tests if "Too expensive" is actually guilt over uneaten food and subscription velocity.
S-8006	29	0	8	Too expensive	Too much food for two	Portion/Household Mismatch: Highly engaged, zero skips, but churned. Tests unit economics and portion sizing vs. true price sensitivity.
S-8120	4	0	10	Too expensive	Cheaper at supermarket	Early Onboarding Drop-off: Active cook (10 recipes) who dropped at Week 4 discount cliff. Tests initial value proposition and promo-hopper behavior.

Backup recruits: S-8044 (Repetitive), S-8115 (Delivery), S-8038 (Price/High engagement).

---

Interviewer Guide (30 Minutes)

> Interviewer Note (for new PMs): > 1. Never mention their cancel form response. It was pre-selected and biases answers. > 2. Focus on stories, not opinions. Don't ask "Was it too expensive?" Ask "Walk me through the last box you unpacked." > 3. Embrace 3 seconds of silence. Let the participant fill pauses.

#### 1. Welcome & Rapport (3 mins) * "Thanks for taking the time! I’m [Name], a product manager at Sprig. We’re working to make our boxes better, and we learn the most from people who decided Sprig wasn't right for them." * "There are no right or wrong answers, and you won’t hurt my feelings. I’m recording this for notes—is that okay?"

#### 2. The Early Days & Habit Formation (7 mins) Goal: Understand the baseline expectation and initial activation (testing month-1 cooking habit). * "Think back to when you first signed up for Sprig. What was going on in your life that made you want meal kits?" * "Walk me through your first couple of weeks. How often were you cooking the meals?" * "Did you cook with someone else? How did the portion sizes fit your household?"

#### 3. The Turning Point & Kitchen Reality (12 mins) Goal: Neutral deep dive to evaluate price, delivery, repetition, and food waste without leading. * "Think about the last box you received before canceling. Walk me through what happened from the moment it arrived to when you decided to cancel." * If Ops signal: "Did it arrive when and how you expected? How did that affect your week?" * If Food signal: "How did picking your meals feel toward the end compared to the beginning? Did you feel excited by the menu?" * If Fridge/Time signal: "Did you ever have ingredients left over when the next box arrived? How did you handle that?" * "I noticed you skipped a few deliveries [or never skipped]. How did skipping fit into your routine?" * "Tell me about the specific moment you went to the website to hit 'Cancel'. What tipped you over the edge that day?"

#### 4. The Alternative & True Cost (5 mins) Goal: Test CEO hypothesis (competitors vs. grocery vs. takeout). * "Now that you aren't using Sprig, what are you doing for dinner on busy weeknights?" * "Did you try or consider any other meal kits or services? Why or why not?" * "When you think about the total cost of Sprig, what felt like the biggest driver of whether it was 'worth it'?"

#### 5. Wrap-Up (3 mins) * "If you had a magic wand and could change one thing about Sprig so that it fit your life perfectly, what would it be?" * "Thank you so much for your candor. We’ll send your $30 gift card within 24 hours."

---

Post-Interview Debrief Checklist (Immediately after call)

Capture within 10 minutes in Slack/Notion: 1. Primary root cause: (e.g., Recipe boredom, Food waste guilt, Box delivery timing, Absolute price, Household mismatch). 2. True competitor: (e.g., Trader Joe’s, HelloFresh, Takeout, Cooking from pantry). 3. Exec hypothesis validation: Did this support CEO, Food, or Ops? (Yes/No/Partial + 1-sentence quote).

Claims the judge couldn’t find in the brief

S-8120 cooked 10 recipes in 4 weeks and dropped at Week 4 discount cliff.

Grades and run details

Decision model 63 · LLM judge 54

Decision model checks

failUses the supplied evidence correctly3%
passAddresses the actual decision33%
partialRespects explicit constraints48%
partialIdentifies material uncertainty53%
partialAvoids unsupported claims40%
partialProduces the required deliverable35%
partialDoesn't trust the default reason70%
passRecruits from the data99%
passDesigned to falsify48%
passAsks what people did, not what they think39%
partialDoesn't lead the witness28%
partialMarks what to cut if the call runs over60%

Run

Run: #1
API response time: 34 s
Submitted: 6 Oct 2026

Sonnet 5.5API

Needs more than a quick edit10 of 12 checks passed

The output fails to show that most 'Too expensive' cancellers had stopped cooking, a key insight from the data, and does not mark what to cut if the call runs over. A PM would need to re-analyze the data and add cut instructions.

Output

Churn interviews: recruiting plan and guide

What the data tells us

The form can't be trusted. "Too expensive" is pre-selected and listed first, and roughly half of cancellers have it. Many of those cooked heavily or wrote free text about something else. Treat the selected reason as a lead, not an answer.
Cancellers look like two groups, whatever they ticked.
Fading: 3-5 skipped deliveries and 0-2 recipes cooked in 4 weeks. Their free text is mostly "piling up in the fridge", "too much food for two" and "no time", which is about quantity and routine. That fits none of the three exec theories directly.
Still cooking: 0-1 skips and 6-12 recipes. Many of them ticked price, and almost all the delivery complainers sit here.
Caveat: the retained-vs-churned stat (first-month cooking) isn't the same measure as the CSV (last 4 weeks), so I can't tell yet whether the fading group never built the habit or lost it. The calls should find out.

Six calls can't size anything. The aim is to find out what's really behind each pattern, so we can fix the form and then measure.

Who to recruit

#	Subscriber	Why
1	S-8005 (price, 20 wks, 5 skips, 0 cooked, "too much food for two")	Fading group with a default-looking price tick. Tests whether the real issue is portion size or cadence.
2	S-8056 (price, 29 wks, 1 skip, 12 cooked, "cheaper at the supermarket")	Heavy user with a price complaint. This is the best test of whether the CEO's theory holds.
3	S-8141 (price, 15 wks, 0 skips, 12 cooked, no text)	Heavy user, price tick, no explanation. Likely clicked the default. What was the real reason?
4	S-8133 (repetitive, 45 wks, 1 skip, 3 cooked, no text)	Long-tenure subscriber with no competing explanation, so a fair test of the Head of Food's theory.
5	S-8097 (delivery, 2 wks, 10 cooked)	Cooked almost every recipe, then left in week 2. Tests whether delivery or the first-box experience is the issue.
6	S-8062 (moving/travelling, 36 wks, 0 skips, text: "kept piling up")	Tests whether travel was the cause or a polite exit, and whether a pause would have kept them.

Recruit a backup for each slot with a similar profile: S-8008, S-8090, S-8196, S-8153, S-8091 and S-8025. The Head of Ops' theory gets one slot because delivery complainers are few and look different from the rest.

Running the calls

Pair each new interviewer with an experienced one. One leads and one takes notes, and they swap roles on the next call.
New interviewers should lead calls 2 and 3, which are the least delicate.
Spend 10 minutes after each call writing the cancellation story in three lines and noting which exec theory it supports, contradicts or ignores.

Interview guide (30 min)

Rules for new interviewers 1. Ask open questions and then stay quiet. Silence gets more than prompting. 2. Don't mention price, recipes or delivery first. Let them raise it. 3. Ask for specific events ("the last box"), not opinions ("what do you generally think?"). 4. Don't defend Sprig, don't pitch, and don't ask "would you come back if…?" People answer hypotheticals unreliably. 5. Follow up with "Tell me more", "What happened next?" and "Can you give me an example?"

1. Open (0-3 min) "Thanks for talking to us. We're trying to learn, not win you back, and nothing you say is wrong. Can we record for note-taking?"

2. Context (3-8 min) - "Who did you cook for? Tell me about a typical week of eating before Sprig." - "What made you sign up? What did you hope it would do for you?"

3. Timeline (8-20 min), the core of the call - "Take me through your first few weeks." (Probe: how many recipes did you cook, and what did it feel like?) - "Tell me about the last box you received. What happened to it, from the doorstep onwards?" - "When did you first think about cancelling? What was going on that week?" - "What did you do for meals instead?" - "Did you think about skipping a week instead? What happened with that?"

4. Only if not raised unprompted (20-26 min) - Cost: "How did the cost sit with you, and compared with what?" - Food: "How did you feel about the recipes over time? Any you made again or avoided?" - Delivery: "What was getting the box like?"

5. The form (26-29 min) Show what they selected. "You chose '[reason]'. What was going through your mind? Does it capture it?" For #2, #3 and #6, add: "Is there anything else that mattered more?"

6. Close (29-30 min) - "If you could change one thing about Sprig, what would it be?" - "Is there anything I should have asked and didn't?"

Tailored probes - #1: "Tell me about the week you had too much food. What did you do with it?" - #2: "Where did you do the comparison, and what exactly was cheaper?" - #3: "You were cooking a lot. What changed?" - #4: "Describe a week when you didn't feel like any of the recipes." - #5: "Walk me through your first delivery." - #6: "Could you have paused? Did you know that?"

After the calls

Review all six together and note which exec theories were supported, contradicted or absent. Separately, remove the default selection from the cancel form now, so the next batch of data is cleaner.

Grades and run details

Decision model 88 · LLM judge 77

Decision model checks

passUses the supplied evidence correctly17%
passAddresses the actual decision57%
passRespects explicit constraints24%
passIdentifies material uncertainty45%
partialAvoids unsupported claims24%
passProduces the required deliverable57%
partialDoesn't trust the default reason30%
passRecruits from the data99%
passDesigned to falsify70%
passAsks what people did, not what they think83%
passDoesn't lead the witness66%
partialMarks what to cut if the call runs over39%

Run

Run: #1
API response time: 51 s
Submitted: 6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 8

Uses the supplied evidence correctlyMixedWrongRight

Gemini 3.5 Flash-Lite · Gemini

The output claims S-8056 was cooking 12 recipes a week, but the data shows 12 recipes over 4 weeks, not per week.

Gemini 3.8 Flash · API

The claim about a 'Week 4 discount cliff' for S-8120 is invented; no discount information exists in the provided data.

Sonnet 5.5 · API

All factual claims about the current situation are directly supported by the supplied context or derived from the data without invention.

Addresses the actual decisionRightMixedRight

Gemini 3.5 Flash-Lite · Gemini

The output commits to a specific recruitment plan and interview guide, which is the requested answer.

Gemini 3.8 Flash · API

The output does not state what result or condition would change the recruitment choice.

Sonnet 5.5 · API

The output commits to a clear plan (six specific recruits and a full interview guide) and states the aim and next steps after the calls.

Identifies material uncertaintyWrongWrongRight

Gemini 3.5 Flash-Lite · Gemini

The output does not name unknowns that could change the plan, such as the unreliability of the cancel form default or how interviews might surface unexpected reasons.

Gemini 3.8 Flash · API

The output does not name specific unknowns that could change the decision or say how they would be resolved.

Sonnet 5.5 · API

It identifies the key unknown about whether the fading group never built the habit or lost it, and notes the calls should find out.

Avoids unsupported claimsMixedWrongRight

Gemini 3.5 Flash-Lite · Gemini

It presents interpretations like 'value was high, but price sensitivity triggered the exit' and 'engaged until logistics broke down' as facts without labelling them as hypotheses.

Gemini 3.8 Flash · API

The 'Week 4 discount cliff' is an unsupported interpretation presented as fact, and no caveat is given.

Sonnet 5.5 · API

Interpretations like the two-group pattern are presented as observations from the data, not as proven causes, and are appropriately hedged.

Doesn't trust the default reasonWrongRightWrong

Gemini 3.5 Flash-Lite · Gemini

The plan does not mention that 'Too expensive' is the pre-selected default or use the behaviour columns to show most of those cancellers had stopped cooking.

Gemini 3.8 Flash · API

It explicitly names the pre-selected bias and uses behavioural data to show many 'too expensive' cancellers had stopped cooking.

Sonnet 5.5 · API

It names the pre-selected default as a bias but does not show from the file that most 'Too expensive' cancellers had stopped cooking; it instead says many cooked heavily or wrote other text, missing the key pattern.

Recruits from the dataMixedRightRight

Gemini 3.5 Flash-Lite · Gemini

It picks six individuals but does not define recruitment groups (e.g., stopped cooking, still cooking but price) with counts from the data.

Gemini 3.8 Flash · API

Six distinct behavioural archetypes are drawn from the data, each linked to an exec hypothesis.

Sonnet 5.5 · API

It defines behavioral groups (fading, still cooking, repetitive, delivery) and selects six interviewees from the list to test each exec hypothesis, with clear reasons.

Designed to falsifyWrongRightRight

Gemini 3.5 Flash-Lite · Gemini

The recipe question ('At what point did the menu start feeling stale?') assumes fatigue and is leading, so it cannot cleanly disprove the hypothesis.

Gemini 3.8 Flash · API

Each hypothesis has a question whose honest answer could disprove it, e.g. asking about what they do for dinner now instead.

Sonnet 5.5 · API

Each exec hypothesis has a non-leading question that could disprove it (e.g., 'Where did you do the comparison, and what exactly was cheaper?' for price).

Doesn't lead the witnessWrongRightRight

Gemini 3.5 Flash-Lite · Gemini

Questions like 'At what point did the menu start feeling stale?' and 'Were there specific meals you started dreading?' lead the participant toward the fatigue hypothesis.

Gemini 3.8 Flash · API

The guide instructs interviewers not to mention the cancel form and questions are neutrally worded.

Sonnet 5.5 · API

Questions are open and neutral, and the guide instructs interviewers not to mention price, recipes, or delivery first.

All got wrong 1

Marks what to cut if the call runs overWrongWrongWrong

Gemini 3.5 Flash-Lite · Gemini

Sections are timed to 30 minutes but the guide does not mark must-ask questions or indicate what to cut if time runs short.

Gemini 3.8 Flash · API

Timings add up to 30 minutes but the guide does not mark which questions are must-ask or what to cut if time runs over.

Sonnet 5.5 · API

The guide has timed sections adding to 30 minutes but does not mark which questions are must-ask or what to cut if the call runs over.

All got right 3

Respects explicit constraintsRightRightRight

Gemini 3.5 Flash-Lite · Gemini

The plan is under 900 words, addresses the two new interviewers, and provides the requested recruitment and guide.

Gemini 3.8 Flash · API

The output respects the 900-word limit, provides a recruitment plan and interview guide, and includes a note for new interviewers.

Sonnet 5.5 · API

The output is under 900 words, addresses the two new interviewers with pairing and rules, and provides both the recruitment plan and the guide as requested.

Produces the required deliverableRightRightRight

Gemini 3.5 Flash-Lite · Gemini

The recruitment list and 30-minute guide are present, complete, and within the word limit.

Gemini 3.8 Flash · API

It delivers a ready-to-use recruitment table and a timed interview guide for the retention PM.

Sonnet 5.5 · API

The output is a complete, usable plan with recruitment table and timed interview guide, within the word limit.

Asks what people did, not what they thinkRightRightRight

Gemini 3.5 Flash-Lite · Gemini

Core questions ask for specific recent instances (e.g., 'Take me to the exact week you decided to cancel', 'Can you share a specific instance?').

Gemini 3.8 Flash · API

Core questions ask about specific recent events like the last box received and the moment of cancellation.

Sonnet 5.5 · API

Core questions ask about specific recent actions (last box, first few weeks, what they did for meals instead) rather than opinions or hypotheticals.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 98% of 42 checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	Sonnet 5.5withAPI	86.1	88.5	2	None
2	GPT-6.1 SolwithAPI	87.8	84.9	2	None
3	Opus 5.5withClaude	92.1	74.7	2	None
4	GPT-6 AstrawithChatGPT	89.7	74.5	2	None
5	Haiku 5.5withAPI	87.8	70.1	2	None
6	GPT-6 LunawithAPI	81.6	73.9	2	None
7	Gemini 3.8 FlashwithAPI	69.7	51.9	2	None
8	Gemini 3.5 Flash-LitewithGemini	60.1	26.1	2	1 capped

What you get for the cost

GPT-6.1 Sol · API scores within a point of Sonnet 5.5 · API at a third of the cost.

Best valueNobody beats these on both score and costAt least one critical failure

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
GPT-6 Luna · API	77	1.0×
Haiku 5.5 · API	77	2.3×
Gemini 3.5 Flash-Lite · Gemini	41	3.0×
Gemini 3.8 Flash · API	58	8.3×
GPT-6.1 Sol · API	85	13×
Sonnet 5.5 · API	86	43×
GPT-6 Astra · ChatGPT	81	66×
Opus 5.5 · Claude	82	71×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Preparing for customer calls.

Why it matters

Leading questions produce the answers you wanted. A good guide is designed to prove you wrong.

What good looks like

Past-behaviour questions
No leading prompts
Clear learning goals
Could disprove the team's hypothesis

Deliberately not measured

Capability tested

Question design

The failure we’re looking for

Hypothetical, leading questions

Grading

Decision model and LLM judge, calibrated against a blind PM review

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

Designed to falsify

Asks what people did, not what they think

Doesn't lead the witness

Plus our standard checks

The tasks

The outputs

Output

Sprig Retention: Churn Interview Plan (Next Week)

Part 1: Recruitment Plan (Who & Why)

Part 2: 30-Minute Interview Guide

Phase 1: Introduction & Context (5 mins)

Phase 2: The Journey & Engagement (10 mins)

Phase 3: Stress-Testing Executive Hypotheses (10 mins)

Phase 4: The Final Decision & Wrap-Up (5 mins)

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Run

Output

Strategic Recruitment Plan

Interviewer Guide (30 Minutes)

Post-Interview Debrief Checklist (Immediately after call)

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Run

Output

Churn interviews: recruiting plan and guide

What the data tells us

Who to recruit

Running the calls

Interview guide (30 min)

After the calls

Grades and run details

Decision model checks

Run

Check by check

Where they differ 8

All got wrong 1

All got right 3

Results

What you get for the cost

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured