Tasks / Product Craft

Customer research call guide

Can the model write a guide that uncovers behaviour rather than opinions?

Task type v1.0 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 38% were usable with at most a quick edit.

Reliably right

  1. Asks what people did, not what they think92% pass
    Core questions ask about specific recent actions (last boxes, skipped weeks, what they did instead) rather than opinions.
    Opus 5.5 · Claude · Too expensive, by default
  2. Tests both theories fairly91% pass
    Both theories get questions that could disprove them (e.g., no action on robotics, happiness after slow go-live), and open-ended timeline questions leave room for a third cause.
    Sonnet 5.5 · API · Why are warehouses leaving? Two execs, two theories
  3. Different questions for user and signer91% pass
    The guide has distinct tracks: daily user questions focus on day-to-day experience and workarounds, signer questions on the decision process and alternatives.
    Sonnet 5.5 · API · Why are warehouses leaving? Two execs, two theories

Where it slips

  1. Marks what to cut if the call runs over28% pass
    The guide has timed sections but does not mark which questions are must-ask or what to cut if time runs short.
    Opus 5.5 · Claude · Too expensive, by default
  2. Doesn't trust the default reason41% pass
    The plan names the pre-selected default as a bias but does not use the behaviour columns to show that most 'too expensive' cancellers had stopped cooking; it only picks two contrasting individuals without the aggregate pattern.
    GPT-6 Astra · ChatGPT · Too expensive, by default
  3. Uses the supplied evidence correctly64% pass
    The claim 'Customers will say what helps their negotiation' is an unsupported behavioural statement not derived from the context.
    Haiku 5.5 · API · Why are warehouses leaving? Two execs, two theories

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Designed to falsify

    For each learning goal, does the guide include a question whose honest answer could show the team's hypothesis is wrong?

    Passes when Every learning goal has at least one disconfirming, non-leading question, such as asking about the last time they had the problem and what they did instead.

  2. Asks what people did, not what they think

    Are the core questions about specific things the person actually did recently, rather than opinions, predictions or hypotheticals?

    Passes when The core of the guide asks for recent, specific instances (the last time, what happened, what they did) and follows up on them; opinions and predictions are absent or clearly secondary.

  3. Doesn't lead the witness

    Are the questions free of leading wording, of pitching the idea, and of hints at the answer the team hopes for?

    Passes when Questions are neutral and open, and the idea is not described (or only at the very end, after the problem questions).

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Marks what to cut if the call runs over · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for retention at Sprig. The execs each have a theory about why subscribers cancel (below). Plan next week's six churn interviews: who to recruit from the cancellation list and why, and the guide the interviewers will use, in no more than 900 words. Two of the interviewers are new to it. The files are attached.

What the model was given5 items: About Sprig, exec_hypotheses.md, cancel_form.md, cancellations_last_quarter.csv, research_constraints.md
About SprigA meal-kit subscription: recipes and ingredients delivered weekly. Subscribers can skip a week or cancel any time.
exec_hypotheses.md7 lines · Download
# What the execs think (for the research)

- CEO: "It's price. Competitors discount harder."
- Head of Food: "Our recipes are too repetitive after a few months."
- Head of Ops: "Late deliveries."

Retained subscribers (for comparison): 81% cooked at least 3 recipes a week in their first month; 34% of cancellers did.
cancel_form.md3 lines · Download
# The cancel form

A single-choice list, in this order: "Too expensive" (selected by default), "Recipes got repetitive", "Moving / travelling", "Delivery problems", "Other". Free text is optional.
cancellations_last_quarter.csv300 rows · Download
cancellation,cancel_reason_selected,weeks_subscribed,deliveries_skipped_last_6_weeks,recipes_cooked_last_4_weeks,free_text
S-8000,Recipes got repetitive,47,3,0,kept piling up in the fridge
S-8001,Recipes got repetitive,16,3,2,too much food for two
S-8002,Moving / travelling,33,2,5,cheaper at the supermarket
S-8003,Too expensive (pre-selected),29,3,2,kept piling up in the fridge
S-8004,Too expensive (pre-selected),11,5,0,
S-8005,Too expensive (pre-selected),20,5,0,too much food for two
…
research_constraints.md4 lines · Download
# Research constraints

- Six calls, 30 minutes each, next week. The researcher can recruit from the cancellation list (consent to contact is on file for all of them).
- Two of the PMs running calls are new to interviewing.
What a strong answer doesThe answer key the graders mark against

Reads the cancellation data before trusting it: 'Too expensive' is the top reason (156 of 300), but it's the pre-selected first option, and 118 of those 156 had skipped two or more deliveries and cooked two or fewer recipes in the last four weeks: they'd stopped using the product before they left. Overall 204 of 300 skipped two or more deliveries. So the guide investigates what happened to the cooking habit, and price as one hypothesis to test rather than the answer. Recruits deliberately from the list across the patterns: people who stopped cooking and picked 'too expensive', people who kept cooking and still picked 'too expensive' (the real price cases), and people who said recipes got repetitive. The guide asks about specific recent weeks (the last box they skipped, what happened to the food, what they ate instead), each exec hypothesis has a question whose honest answer could disprove it, questions don't lead or mention price first, the guide fits 30 minutes with timings and marks what to cut, and gives the new interviewers prompts for digging deeper. Notes that the default on the cancel form should be removed.

Critical failures (cap the score at 40)
  • Builds the guide around confirming that price is the reason
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Usable with a quick edit10 of 12 checks passed

The guide is well-structured and evidence-based; a PM could add a note on what to cut if time runs short and the aggregate data point about 'too expensive' cancellers in a quick edit.

Output

Aim and guardrails

Understand the sequence leading to cancellation, not ask subscribers to vote on executives’ theories. Test price, repetition, and delivery alongside an alternative: subscribers never established—or lost—a workable cooking routine, making unused food feel expensive.

The cancellation form is weak evidence: price is first and preselected, only one answer is allowed, and comments sometimes conflict with selections. The first-month cooking gap—81% of retained subscribers versus 34% of cancellers cooking at least three recipes weekly—is an important association, not proof of causation. Last-four-week cooking counts cannot identify first-month activation.

Recruit these six

Use purposive contrasts, not a representative sample. Counts below mean weeks subscribed / skips in last six weeks / recipes cooked in last four weeks.

Primary recruitEvidenceWhy this interview; backup
S-800329 / 3 / 2; price selected; “kept piling up in the fridge”Distinguish affordability from paying for unused food; reconstruct whether disengagement preceded price concerns. Backup: S-8093.
S-805629 / 1 / 12; price selected; “cheaper at the supermarket”Same tenure as S-8003, much higher recent cooking. Strong contrast for genuine price pressure despite use; supermarket comparison is not evidence of competitor discounting. Backup: S-8096.
S-800047 / 3 / 0; repetition selected; “kept piling up in the fridge”Test whether menu fatigue caused declining use, or whether routine/quantity problems came first. Backup: S-8209.
S-807223 / 1 / 8; delivery problems selected; no commentExplore an operational explanation among someone still cooking relatively often. Establish whether the problem was lateness, something else, and consequential enough to trigger cancellation. Backup: S-8179.
S-80246 / 5 / 1; other selected; “kept piling up in the fridge”Short-tenure, high-skipping case outside executive labels: explore onboarding, quantity, expectations, and routine formation. Backup: S-8131.
S-803733 / 0 / 5; moving/travelling selected; no commentContrast with frequent skippers; examine life circumstances, whether the label fits, and why cancel rather than skip. Backup: S-8215.

Contact primaries immediately; approach backups if unavailable or declining. Offer the same fixed incentive regardless of feedback. Invite them to discuss “your experience using and stopping Sprig,” without naming hypotheses. Confirm they managed the subscription or cooking; do not screen for agreement with a theory.

Before calls, verify metric definitions, subscription dates, and available delivery history. Skipping reduces opportunities to cook; raw cooking counts are not cooking rates. Schedule three calls Tuesday, three Thursday, with debrief time.

Shared 30-minute interview guide

0–2 minutes: introduction

“We’re learning how Sprig fitted into your life and what led you to stop. This isn’t a sales call; honest criticism is useful. You can skip questions or stop. May we record for research notes?”

2–6: expectations and early use

  • “What was happening in your life when you signed up?”
  • “What did you hope Sprig would replace or make easier?”
  • “Walk me through an early delivery: who cooked, what got cooked, and what happened to the rest?”
  • “How typical was that during your first month?”

6–15: reconstruct the change

  • “Walk me through your last few weeks with Sprig.”
  • “Tell me about the last box you received—from arrival to what happened to each meal.”
  • “When did you first consider stopping? What happened just before that?”
  • “Was there a time it worked better? What changed?”
  • “Tell me about a week you skipped, if any. What drove that decision?”

Anchor memories to deliveries or calendar events; accept “I don’t remember.”

15–20: cancellation and alternatives

  • “Tell me about the day you cancelled. Why then?”
  • “What had you already tried?”
  • “What options did you consider, including skipping?”
  • “What have you actually done for meals since?”

20–27: test explanations, only after the unaided story

Ask everyone: - “How did the cost fit your budget and what you got from it?” Probe actual alternatives, prices or discounts used—not hypothetical willingness to pay. - “How did recipe choice change over time?” Request specific examples and timing. - “How did deliveries fit your plans?” If problems arose: promised versus actual arrival, consequences, frequency, and attempted resolution. - Then show their selected reason/comment: “How well does this capture what happened? What’s missing?”

27–30: confirm

Summarize the sequence, separating ongoing frustrations from the final trigger. “What have I misunderstood? What else mattered?”

Support and synthesis

Pair each novice with an experienced observer; rehearse Monday. Use silence and “Tell me about a specific time,” not suggested answers. Never defend Sprig or pitch fixes.

After each call, record timeline, supporting examples, contradictions, and unknowns separately from interpretation. Friday, compare cases across all explanations, explicitly noting disconfirming evidence. Report mechanisms and next validation steps—not churn percentages or causal conclusions from six interviews.

Grades and run details

Decision model 83 · LLM judge 85
Decision model checks
  • passUses the supplied evidence correctly29%
  • passAddresses the actual decision47%
  • partialRespects explicit constraints38%
  • partialIdentifies material uncertainty43%
  • passAvoids unsupported claims75%
  • passProduces the required deliverable57%
  • partialDoesn't trust the default reason47%
  • passRecruits from the data99%
  • passDesigned to falsify73%
  • passAsks what people did, not what they think89%
  • passDoesn't lead the witness72%
  • partialMarks what to cut if the call runs over55%
Run
Run
#1
Time to output
33 s
Submitted
5 Oct 2026

Usable with a quick edit11 of 12 checks passed

The only missing element is marking what to cut if the call runs over, which a PM can add in a quick edit; otherwise the plan is complete and evidence-based.

Output

Churn Interviews: Recruit Plan and Guide

What the data already tells us

  • The cancel form inflates "Too expensive." It is pre-selected, so we can't tell real price objections from people clicking through. Most cancellers left it on that answer.
  • Most cancellers had stopped cooking before they left. Many cooked 0–2 recipes in their last four weeks and skipped several deliveries. Their free text points to fit, not price: "kept piling up in the fridge," "too much food for two," "didn't have time to cook." The comparison stat agrees: 81% of retained subscribers cooked 3+ recipes a week in month one, versus 34% of cancellers. That's a correlation, but it suggests a fourth hypothesis none of the execs named: the kit never became a habit.
  • "Delivery problems" cancellers look different. Every one of them cooked 3+ recipes in their last four weeks, and several left within weeks 2–3. These are engaged customers we lost to an ops failure.
  • Free text often contradicts the selected reason. For example, "Moving / travelling" sits next to "cheaper at the supermarket."

Six calls can't tell us how many people leave for each reason; the data does that. The calls are for understanding how people decide to leave, so we give each hypothesis a fair, clean case.

Who to recruit

#PrimaryBackupProfileWhy
1S-8056S-8096Pre-selected price; cooked 12 (backup 9) in last 4 wks; "cheaper at the supermarket"CEO's hypothesis at its strongest: an engaged user who says price. Was it really competitors' discounts?
2S-8004S-8183Pre-selected price; 5 skips, 0 cooked, no free textWhat the default hides. Was price the reason or just the first option?
3S-8005S-8016Pre-selected price; 4–5 skips, 0 cooked; "too much food for two" / "piling up"Portion and waste: the theory none of the execs named.
4S-8036S-8282Pre-selected price; 5 skips, 0 cooked; "didn't have time to cook"Time and habit. When did cooking stop, and why?
5S-8133S-8057Chose "Recipes got repetitive"; 45–53 weeks; still cooking a littleHead of Food's hypothesis: a long-tenure user who picked this reason actively.
6S-8097S-8072Chose "Delivery problems"; cooked 8–10 in last 4 wksHead of Ops' hypothesis: an engaged user lost to a delivery failure.

Contact backups only if the primary doesn't respond within 48 hours. Assign the experienced interviewers to calls 1 and 2, where people most easily give a stock answer like "it was the price." New interviewers take 3–6 and shadow one call first if the schedule allows.

Interview guide (30 min)

Rules for interviewers (read before every call)

  1. Ask about what happened, not opinions. Prefer "Tell me about the last box you got" to "Do you think the recipes are repetitive?"
  2. Don't mention their cancel reason or our hypotheses until section 4. If you say "price" first, you'll hear "price."
  3. Don't defend, explain, or offer anything. No discounts, no "actually you can skip weeks." If they ask, note it and say someone can follow up.
  4. Allow silence. Count to five before your next question. The best answers come after the pause.
  5. Use neutral probes: "Tell me more." "What happened next?" "Can you give me an example?" "Why was that?"
  6. Follow their story, not the script. The script makes sure you cover everything; you don't need to ask it in order.
  7. Write down exact quotes, especially words they use about money, time, food, and delivery.

1. Intro (2 min)

"Thanks for making time. I'm [name] from Sprig. We're trying to understand what Sprig was like to use, including the bad parts. There are no wrong answers, and I'm not here to win you back. Is it OK if I take notes or record?"

2. Their life with Sprig (8 min)

  • "Take me back to when you signed up. What was going on? What were you hoping it would do for you?"
  • "Walk me through a typical week with a box. When did it arrive? Who cooked? What happened to the food?"
  • "How did that change over time?"
  • Probe: "When did that start?" "What was different then?"

3. The decision to leave (10 min)

  • "Tell me about the last few boxes before you cancelled."
  • "When did you first think about stopping? What happened that day or week?"
  • "Before cancelling, did you skip any weeks? What was going on then?"
  • Probe for leftovers and waste: "What happened to meals you didn't cook?"
  • "What did you do instead for meals? What does that cost and take?"
  • "Was there one moment that tipped it, or did it build up?"

4. Checking the reasons (6 min)

Ask these only now. - "On the cancel form you chose [reason]. Is that how you'd describe it to a friend?" - If the reason was pre-selected, say: "That option was already ticked by default. Was it your actual reason?" - Ask about each of the following, one at a time, unless they've already covered it. Get a specific example for each. - "How did the price feel by the end? Compared to what?" - "How did you feel about the recipe choices?" - "Any issues with deliveries? What happened?" - "If you could change one thing, would you have stayed? What would it be?"

5. Wrap (4 min)

  • "Is there anything I should have asked but didn't?"
  • Thank them and explain any incentive.

After each call (15 min, same day)

Fill in the shared sheet with: - Their main reason, in their words. - When cooking dropped off: week, and the trigger. - Supporting quotes. - Whether the form reason matched their story.

After all six calls, I'll write up how each person's decision unfolded, mapped against the four hypotheses. I'll be clear that six interviews show how people decide, not how many decide each way. The write-up will also recommend changing the cancel form so no option is pre-selected; that one change will make the form data usable.

Grades and run details

Decision model 96 · LLM judge 92
Decision model checks
  • passUses the supplied evidence correctly23%
  • passAddresses the actual decision53%
  • passRespects explicit constraints30%
  • passIdentifies material uncertainty29%
  • passAvoids unsupported claims31%
  • passProduces the required deliverable68%
  • passDoesn't trust the default reason59%
  • passRecruits from the data100%
  • passDesigned to falsify61%
  • passAsks what people did, not what they think76%
  • passDoesn't lead the witness70%
  • partialMarks what to cut if the call runs over52%
Run
Run
#1
Time to output
49 s
Submitted
5 Oct 2026

Needs more than a quick edit8 of 12 checks passed

The free-text claim is misleading, and the guide lacks cut marks for time management, so a PM would need to recheck the free-text analysis and add cut instructions.

Output

Churn interviews: recruiting plan and guide

What the data says (and doesn't)

Across 300 cancellations last quarter:

Reason selectedCountShare
Too expensive (pre-selected default)15652%
Recipes got repetitive5217%
Moving / travelling4114%
Delivery problems2910%
Other227%
  • The price number is inflated. "Too expensive" is pre-selected, so we can't tell deliberate choices from people who clicked Next. Treat it as a hypothesis to test, not a finding.
  • Free text points away from all three exec theories. Where people wrote something, the recurring themes were over-supply ("too much food for two," "kept piling up in the fridge") and time ("didn't have time to cook"). Neither is in the exec hypotheses.
  • Delivery cancellers were engaged. All 29 cooked at least 3 recipes in the last 4 weeks, and many cooked 8 or more. Late deliveries may be hurting people who want to cook.
  • The retention benchmark isn't testable here. The 34% figure is first-month cooking. The export only has the last 4 weeks, so we can't compare the two. Early-tenure cancellers are the closest we can get, so we'll include one.

Who to recruit (6 calls)

Recruit from the list by reason, tenure, and cooking level so each call tests a different explanation. Keep 2 backups for each slot.

#AccountWhy this person
1S-8007 (Too expensive, 37 wks, 4 skips, 0 recipes last 4 wks)Price claim from a long-tenure subscriber who stopped cooking. Tests whether price is the real reason or cover for disengagement.
2S-8056 (Too expensive, 29 wks, 12 recipes last 4 wks, "cheaper at the supermarket")Engaged cook who cites price. Tests the CEO's theory where it's most plausible.
3S-8186 (Repetitive, 54 wks, 2 recipes)Tests the Head of Food's theory with a long-tenure subscriber.
4S-8115 (Delivery problems, 17 wks, 10 recipes)Tests the Head of Ops's theory with an engaged cook.
5S-8171 (Too expensive, 6 wks, 0 recipes, "kept piling up in the fridge")Early-tenure, non-cooking cancel. Closest to the first-month question. Explores whether the box was used at all.
6S-8253 (Too expensive, 32 wks, 5 skips, 1 recipe, "too much food for two")Surfaces the over-supply theme no exec raised.

Backups: for slot 5, S-8153 (2 wks tenure, 8 recipes, delivery). For slot 6, S-8055 (Other, 5 skips, "too much food for two").

Interview guide (30 minutes)

Rules for all interviewers - Don't mention the cancel form's default or any exec theory. - Don't ask leading questions ("Was it the price?"). Ask what happened, then probe. - Let silences run. Don't fill them with suggestions. - Ask for specific examples: "Tell me about the last time…" - Don't defend Sprig or argue with the subscriber.

1. Opening (2 min) "Thanks for making time. We're trying to understand what happened with your subscription so we can improve it. There are no right or wrong answers, and nothing you say will affect your account. Is it okay if I take notes and record for accuracy?"

2. Their Sprig story (8 min) - "Take me back to when you first signed up. What did you hope Sprig would be for you?" - "How did a typical week go once the boxes started arriving?" - "What was happening in your life when you decided to cancel?" - "Walk me through the moment you decided. What happened first?"

3. The cancel form (4 min) - "On the cancel screen, you chose a reason. Can you walk me through how you picked it?" - If they say "too expensive": "What made that the best fit? Was there anything else you considered?" - If they chose something else: "What made that the closest match?" - "Was there a reason you wanted to share that the options didn't capture?"

4. Cooking and the box (6 min) - "In your first month, how many of the recipes did you actually cook?" - "What got in the way of cooking the others?" - "What happened to ingredients you didn't use? Walk me through a recent week." - "How did the portion sizes work for you?" - "How did the recipes feel after a few weeks?"

5. Price, delivery, and alternatives (5 min) - "How did the cost compare to what you'd spend otherwise?" - "You mentioned [free text, if any]. Tell me more about that." - "Have you had any issues with deliveries? What happened, and how did it affect your week?" - "Did you pause or skip boxes before cancelling? What was going on then?"

6. Counterfactual (3 min) - "If Sprig had changed one thing, would you have stayed? What would it have been?" - "What would have had to be different for you to skip only for a month instead of cancelling?"

7. Close (2 min) "Is there anything I didn't ask that you think we should know?" Thank them and confirm how the findings will be used.

Notes for the two new interviewers

  • Pair each new interviewer with an experienced one for the first call, with the experienced person leading.
  • Practise the "walk me through" and silence-holding moves beforehand. Most of the signal comes from follow-up probes.
  • If a subscriber gets upset, acknowledge it briefly, offer to end early, and move to the close.
  • Write down direct quotes, not summaries, especially about the cancel form and the portions.

After the calls

Code each interview against the three exec theories and two emerging themes (over-supply, time to cook). Report what each subscriber did and said, not just their stated reason. Flag any case where the stated reason and the story don't match, since that's the pre-selection problem showing up.

Claims the judge couldn’t find in the brief

  • Free text points away from all three exec theories, with recurring themes over-supply and time to cook.

Grades and run details

Decision model 83 · LLM judge 62
Decision model checks
  • passUses the supplied evidence correctly41%
  • passAddresses the actual decision37%
  • partialRespects explicit constraints31%
  • passIdentifies material uncertainty54%
  • passAvoids unsupported claims34%
  • passProduces the required deliverable38%
  • partialDoesn't trust the default reason33%
  • passRecruits from the data100%
  • passDesigned to falsify62%
  • partialAsks what people did, not what they think35%
  • passDoesn't lead the witness67%
  • partialMarks what to cut if the call runs over52%
Run
Run
#1
API response time
46 s
Submitted
8 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 3

Uses the supplied evidence correctlyRightRightMixed
GPT-6 Astra · ChatGPT

All factual claims about the current situation are taken directly from the supplied context or derived by simple arithmetic from it.

Opus 5.5 · Claude

All factual claims about the current situation are directly from the supplied context or derived from it by arithmetic.

Haiku 5.5 · API

The claim that free text points away from all three exec theories is incorrect; free text includes price-related comments like ‘cheaper at the supermarket’ that support the CEO’s theory.

Avoids unsupported claimsRightRightMixed
GPT-6 Astra · ChatGPT

Interpretations are clearly labelled as such (e.g., 'important association, not proof of causation'), and no cause is presented as established fact.

Opus 5.5 · Claude

Interpretations like 'engaged customers lost to ops failure' are presented as patterns to test, not as established fact.

Haiku 5.5 · API

The statement that free text points away from all three exec theories is unsupported by the data, because some free text supports the price hypothesis.

Doesn't trust the default reasonWrongRightWrong
GPT-6 Astra · ChatGPT

The plan names the pre-selected default as a bias but does not use the behaviour columns to show that most 'too expensive' cancellers had stopped cooking; it only picks two contrasting individuals without the aggregate pattern.

Opus 5.5 · Claude

It explicitly names the pre-selected default as a bias and shows that most 'too expensive' cancellers had stopped cooking.

Haiku 5.5 · API

It names the pre-selected default but does not use the behaviour columns to demonstrate that most ‘too expensive’ cancellers had stopped cooking; it only states the number is inflated.

All got wrong 1

Marks what to cut if the call runs overWrongWrongWrong
GPT-6 Astra · ChatGPT

The guide includes timed sections that sum to 30 minutes but does not mark which questions to cut if the call runs over, as required.

Opus 5.5 · Claude

The guide has timed sections but does not mark which questions are must-ask or what to cut if time runs short.

Haiku 5.5 · API

The guide has timed sections that add to 30 minutes but does not mark which questions to cut if time runs over.

All got right 8

Addresses the actual decisionRightRightRight
GPT-6 Astra · ChatGPT

The output commits to a clear recruitment plan and interview guide, and says to approach backups if primaries are unavailable.

Opus 5.5 · Claude

The output commits to a clear recruitment plan and interview guide, framed for the PM, and states the purpose of the calls.

Haiku 5.5 · API

The output commits to a clear recruiting plan and interview guide, naming specific accounts and a full guide.

Respects explicit constraintsRightRightRight
GPT-6 Astra · ChatGPT

The plan is under 900 words, addresses the two new interviewers with pairing and rehearsal, and delivers the requested recruit list and guide.

Opus 5.5 · Claude

The output is under 900 words, addresses the two new interviewers, and provides a complete plan and guide.

Haiku 5.5 · API

The output is a plan plus guide within ~800 words, addresses new interviewers, and respects the six-call, 30-minute constraint.

Identifies material uncertaintyRightRightRight
GPT-6 Astra · ChatGPT

The output labels the cooking gap as an association, notes the cancel form is weak evidence, and instructs to report mechanisms and next validation steps rather than causal conclusions.

Opus 5.5 · Claude

It names the pre-selected default bias, the habit hypothesis, and the limitation of six interviews for quantifying reasons.

Haiku 5.5 · API

It identifies the pre-selection bias, notes the data gap for first-month cooking, and flags that the price number is inflated.

Produces the required deliverableRightRightRight
GPT-6 Astra · ChatGPT

The output provides a complete recruitment plan with reasons and a timed interview guide, usable by the retention PM and interviewers.

Opus 5.5 · Claude

The output is a complete recruitment plan and interview guide, usable by a PM with light edits.

Haiku 5.5 · API

It delivers a complete recruiting plan and interview guide that a PM could use with light edits.

Recruits from the dataRightRightRight
GPT-6 Astra · ChatGPT

The six interviewees are chosen by behaviour pattern (low-cooking price, high-cooking price, repetition, delivery, short-tenure other, moving with no skips) to test each hypothesis.

Opus 5.5 · Claude

Recruitment groups are defined by behavior patterns (engaged price, default-hiders, waste, time, recipes, delivery) with specific IDs.

Haiku 5.5 · API

It picks six interviewees that cover the needed behaviour patterns: stopped cooking, still cooking but price, repetitive recipes, delivery, early-tenure, and over-supply.

Designed to falsifyRightRightRight
GPT-6 Astra · ChatGPT

Each executive hypothesis has a disconfirming question (e.g., 'How did the cost fit your budget and what you got from it?' could show price wasn't the issue).

Opus 5.5 · Claude

Each hypothesis has a non-leading question (e.g., 'How did the price feel by the end?') that could yield disconfirming answers.

Haiku 5.5 · API

The guide includes open questions (e.g., on cost, recipes, deliveries) whose honest answers could disprove each exec hypothesis.

Asks what people did, not what they thinkRightRightRight
GPT-6 Astra · ChatGPT

Core questions ask for specific recent instances (last box, week skipped, day of cancellation) and follow up on actions, not opinions or hypotheticals.

Opus 5.5 · Claude

Core questions ask about specific recent actions (last boxes, skipped weeks, what they did instead) rather than opinions.

Haiku 5.5 · API

Core questions ask about specific recent weeks, what happened to ingredients, and walk-throughs of events rather than opinions.

Doesn't lead the witnessRightRightRight
GPT-6 Astra · ChatGPT

Questions are open and neutral; the guide explicitly instructs interviewers not to defend Sprig or suggest answers.

Opus 5.5 · Claude

Questions are neutral and open; the guide instructs not to mention the cancel reason or hypotheses until section 4.

Haiku 5.5 · API

Questions are neutral (e.g., ‘Walk me through the moment you decided’) and the guide explicitly bans leading wording.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 98% of 42 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5withAPI86.188.52None
2GPT-6.1 SolwithAPI87.884.92None
3Opus 5.5withClaude92.174.72None
4GPT-6 AstrawithChatGPT89.774.52None
5Haiku 5.5withAPI87.870.12None
6GPT-6 LunawithAPI81.673.92None
7Gemini 3.8 FlashwithAPI69.751.92None
8Gemini 3.5 Flash-LitewithGemini60.126.121 capped

What you get for the cost

GPT-6.1 Sol · API scores within a point of Sonnet 5.5 · API at a third of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreSonnet 5.5 · API: task score 86, 43× the cheapestGPT-6.1 Sol · API: task score 85, 13× the cheapestOpus 5.5 · Claude: task score 82, 71× the cheapestGPT-6 Astra · ChatGPT: task score 81, 66× the cheapestHaiku 5.5 · API: task score 77, 2.3× the cheapestGPT-6 Luna · API: task score 77, the cheapestGemini 3.8 Flash · API: task score 58, 8.3× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 41, 3.0× the cheapest, 1 critical failureSonnet 5.5 · APIGPT-6.1 Sol · APIOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTHaiku 5.5 · APIGPT-6 Luna · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costAt least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API771.0×
Haiku 5.5 · API772.3×
Gemini 3.5 Flash-Lite · Gemini413.0×
Gemini 3.8 Flash · API588.3×
GPT-6.1 Sol · API8513×
Sonnet 5.5 · API8643×
GPT-6 Astra · ChatGPT8166×
Opus 5.5 · Claude8271×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Preparing for customer calls.

Why it matters

Leading questions produce the answers you wanted. A good guide is designed to prove you wrong.

What good looks like

  • Past-behaviour questions
  • No leading prompts
  • Clear learning goals
  • Could disprove the team's hypothesis

Deliberately not measured

    Capability tested

    Question design

    The failure we’re looking for

    Hypothetical, leading questions

    Grading

    Decision model and LLM judge, calibrated against a blind PM review