Tasks / Product Craft

Customer research call guide

Can the model write a guide that uncovers behaviour rather than opinions?

Task type v1.0 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 38% were usable with at most a quick edit.

Reliably right

  1. Asks what people did, not what they think92% pass
    Core questions ask about specific recent actions (last boxes, skipped weeks, what they did instead) rather than opinions.
    Opus 5.5 · Claude · Too expensive, by default
  2. Tests both theories fairly91% pass
    Both theories get questions that could disprove them (e.g., no action on robotics, happiness after slow go-live), and open-ended timeline questions leave room for a third cause.
    Sonnet 5.5 · API · Why are warehouses leaving? Two execs, two theories
  3. Different questions for user and signer91% pass
    The guide has distinct tracks: daily user questions focus on day-to-day experience and workarounds, signer questions on the decision process and alternatives.
    Sonnet 5.5 · API · Why are warehouses leaving? Two execs, two theories

Where it slips

  1. Marks what to cut if the call runs over28% pass
    The guide has timed sections but does not mark which questions are must-ask or what to cut if time runs short.
    Opus 5.5 · Claude · Too expensive, by default
  2. Doesn't trust the default reason41% pass
    The plan names the pre-selected default as a bias but does not use the behaviour columns to show that most 'too expensive' cancellers had stopped cooking; it only picks two contrasting individuals without the aggregate pattern.
    GPT-6 Astra · ChatGPT · Too expensive, by default
  3. Uses the supplied evidence correctly64% pass
    The claim 'Customers will say what helps their negotiation' is an unsupported behavioural statement not derived from the context.
    Haiku 5.5 · API · Why are warehouses leaving? Two execs, two theories

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Designed to falsify

    For each learning goal, does the guide include a question whose honest answer could show the team's hypothesis is wrong?

    Passes when Every learning goal has at least one disconfirming, non-leading question, such as asking about the last time they had the problem and what they did instead.

  2. Asks what people did, not what they think

    Are the core questions about specific things the person actually did recently, rather than opinions, predictions or hypotheticals?

    Passes when The core of the guide asks for recent, specific instances (the last time, what happened, what they did) and follows up on them; opinions and predictions are absent or clearly secondary.

  3. Doesn't lead the witness

    Are the questions free of leading wording, of pitching the idea, and of hints at the answer the team hopes for?

    Passes when Questions are neutral and open, and the idea is not described (or only at the very end, after the problem questions).

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Marks what to cut if the call runs over · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for retention at Sprig. The execs each have a theory about why subscribers cancel (below). Plan next week's six churn interviews: who to recruit from the cancellation list and why, and the guide the interviewers will use, in no more than 900 words. Two of the interviewers are new to it. The files are attached.

What the model was given5 items: About Sprig, exec_hypotheses.md, cancel_form.md, cancellations_last_quarter.csv, research_constraints.md
About SprigA meal-kit subscription: recipes and ingredients delivered weekly. Subscribers can skip a week or cancel any time.
exec_hypotheses.md7 lines · Download
# What the execs think (for the research)

- CEO: "It's price. Competitors discount harder."
- Head of Food: "Our recipes are too repetitive after a few months."
- Head of Ops: "Late deliveries."

Retained subscribers (for comparison): 81% cooked at least 3 recipes a week in their first month; 34% of cancellers did.
cancel_form.md3 lines · Download
# The cancel form

A single-choice list, in this order: "Too expensive" (selected by default), "Recipes got repetitive", "Moving / travelling", "Delivery problems", "Other". Free text is optional.
cancellations_last_quarter.csv300 rows · Download
cancellation,cancel_reason_selected,weeks_subscribed,deliveries_skipped_last_6_weeks,recipes_cooked_last_4_weeks,free_text
S-8000,Recipes got repetitive,47,3,0,kept piling up in the fridge
S-8001,Recipes got repetitive,16,3,2,too much food for two
S-8002,Moving / travelling,33,2,5,cheaper at the supermarket
S-8003,Too expensive (pre-selected),29,3,2,kept piling up in the fridge
S-8004,Too expensive (pre-selected),11,5,0,
S-8005,Too expensive (pre-selected),20,5,0,too much food for two
…
research_constraints.md4 lines · Download
# Research constraints

- Six calls, 30 minutes each, next week. The researcher can recruit from the cancellation list (consent to contact is on file for all of them).
- Two of the PMs running calls are new to interviewing.
What a strong answer doesThe answer key the graders mark against

Reads the cancellation data before trusting it: 'Too expensive' is the top reason (156 of 300), but it's the pre-selected first option, and 118 of those 156 had skipped two or more deliveries and cooked two or fewer recipes in the last four weeks: they'd stopped using the product before they left. Overall 204 of 300 skipped two or more deliveries. So the guide investigates what happened to the cooking habit, and price as one hypothesis to test rather than the answer. Recruits deliberately from the list across the patterns: people who stopped cooking and picked 'too expensive', people who kept cooking and still picked 'too expensive' (the real price cases), and people who said recipes got repetitive. The guide asks about specific recent weeks (the last box they skipped, what happened to the food, what they ate instead), each exec hypothesis has a question whose honest answer could disprove it, questions don't lead or mention price first, the guide fits 30 minutes with timings and marks what to cut, and gives the new interviewers prompts for digging deeper. Notes that the default on the cancel form should be removed.

Critical failures (cap the score at 40)
  • Builds the guide around confirming that price is the reason
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Needs more than a quick edit10 of 12 checks passed

The output fails to show that most 'Too expensive' cancellers had stopped cooking, a key insight from the data, and does not mark what to cut if the call runs over. A PM would need to re-analyze the data and add cut instructions.

Output

Churn interviews: recruiting plan and guide

What the data tells us

  • The form can't be trusted. "Too expensive" is pre-selected and listed first, and roughly half of cancellers have it. Many of those cooked heavily or wrote free text about something else. Treat the selected reason as a lead, not an answer.
  • Cancellers look like two groups, whatever they ticked.
  • Fading: 3-5 skipped deliveries and 0-2 recipes cooked in 4 weeks. Their free text is mostly "piling up in the fridge", "too much food for two" and "no time", which is about quantity and routine. That fits none of the three exec theories directly.
  • Still cooking: 0-1 skips and 6-12 recipes. Many of them ticked price, and almost all the delivery complainers sit here.
  • Caveat: the retained-vs-churned stat (first-month cooking) isn't the same measure as the CSV (last 4 weeks), so I can't tell yet whether the fading group never built the habit or lost it. The calls should find out.

Six calls can't size anything. The aim is to find out what's really behind each pattern, so we can fix the form and then measure.

Who to recruit

#SubscriberWhy
1S-8005 (price, 20 wks, 5 skips, 0 cooked, "too much food for two")Fading group with a default-looking price tick. Tests whether the real issue is portion size or cadence.
2S-8056 (price, 29 wks, 1 skip, 12 cooked, "cheaper at the supermarket")Heavy user with a price complaint. This is the best test of whether the CEO's theory holds.
3S-8141 (price, 15 wks, 0 skips, 12 cooked, no text)Heavy user, price tick, no explanation. Likely clicked the default. What was the real reason?
4S-8133 (repetitive, 45 wks, 1 skip, 3 cooked, no text)Long-tenure subscriber with no competing explanation, so a fair test of the Head of Food's theory.
5S-8097 (delivery, 2 wks, 10 cooked)Cooked almost every recipe, then left in week 2. Tests whether delivery or the first-box experience is the issue.
6S-8062 (moving/travelling, 36 wks, 0 skips, text: "kept piling up")Tests whether travel was the cause or a polite exit, and whether a pause would have kept them.

Recruit a backup for each slot with a similar profile: S-8008, S-8090, S-8196, S-8153, S-8091 and S-8025. The Head of Ops' theory gets one slot because delivery complainers are few and look different from the rest.

Running the calls

  • Pair each new interviewer with an experienced one. One leads and one takes notes, and they swap roles on the next call.
  • New interviewers should lead calls 2 and 3, which are the least delicate.
  • Spend 10 minutes after each call writing the cancellation story in three lines and noting which exec theory it supports, contradicts or ignores.

Interview guide (30 min)

Rules for new interviewers 1. Ask open questions and then stay quiet. Silence gets more than prompting. 2. Don't mention price, recipes or delivery first. Let them raise it. 3. Ask for specific events ("the last box"), not opinions ("what do you generally think?"). 4. Don't defend Sprig, don't pitch, and don't ask "would you come back if…?" People answer hypotheticals unreliably. 5. Follow up with "Tell me more", "What happened next?" and "Can you give me an example?"

1. Open (0-3 min) "Thanks for talking to us. We're trying to learn, not win you back, and nothing you say is wrong. Can we record for note-taking?"

2. Context (3-8 min) - "Who did you cook for? Tell me about a typical week of eating before Sprig." - "What made you sign up? What did you hope it would do for you?"

3. Timeline (8-20 min), the core of the call - "Take me through your first few weeks." (Probe: how many recipes did you cook, and what did it feel like?) - "Tell me about the last box you received. What happened to it, from the doorstep onwards?" - "When did you first think about cancelling? What was going on that week?" - "What did you do for meals instead?" - "Did you think about skipping a week instead? What happened with that?"

4. Only if not raised unprompted (20-26 min) - Cost: "How did the cost sit with you, and compared with what?" - Food: "How did you feel about the recipes over time? Any you made again or avoided?" - Delivery: "What was getting the box like?"

5. The form (26-29 min) Show what they selected. "You chose '[reason]'. What was going through your mind? Does it capture it?" For #2, #3 and #6, add: "Is there anything else that mattered more?"

6. Close (29-30 min) - "If you could change one thing about Sprig, what would it be?" - "Is there anything I should have asked and didn't?"

Tailored probes - #1: "Tell me about the week you had too much food. What did you do with it?" - #2: "Where did you do the comparison, and what exactly was cheaper?" - #3: "You were cooking a lot. What changed?" - #4: "Describe a week when you didn't feel like any of the recipes." - #5: "Walk me through your first delivery." - #6: "Could you have paused? Did you know that?"

After the calls

Review all six together and note which exec theories were supported, contradicted or absent. Separately, remove the default selection from the cancel form now, so the next batch of data is cleaner.

Grades and run details

Decision model 88 · LLM judge 77
Decision model checks
  • passUses the supplied evidence correctly17%
  • passAddresses the actual decision57%
  • passRespects explicit constraints24%
  • passIdentifies material uncertainty45%
  • partialAvoids unsupported claims24%
  • passProduces the required deliverable57%
  • partialDoesn't trust the default reason30%
  • passRecruits from the data99%
  • passDesigned to falsify70%
  • passAsks what people did, not what they think83%
  • passDoesn't lead the witness66%
  • partialMarks what to cut if the call runs over39%
Run
Run
#1
API response time
51 s
Submitted
6 Oct 2026

Usable with a quick edit10 of 12 checks passed

The guide is well-structured and evidence-based; a PM could add a note on what to cut if time runs short and the aggregate data point about 'too expensive' cancellers in a quick edit.

Output

Aim and guardrails

Understand the sequence leading to cancellation, not ask subscribers to vote on executives’ theories. Test price, repetition, and delivery alongside an alternative: subscribers never established—or lost—a workable cooking routine, making unused food feel expensive.

The cancellation form is weak evidence: price is first and preselected, only one answer is allowed, and comments sometimes conflict with selections. The first-month cooking gap—81% of retained subscribers versus 34% of cancellers cooking at least three recipes weekly—is an important association, not proof of causation. Last-four-week cooking counts cannot identify first-month activation.

Recruit these six

Use purposive contrasts, not a representative sample. Counts below mean weeks subscribed / skips in last six weeks / recipes cooked in last four weeks.

Primary recruitEvidenceWhy this interview; backup
S-800329 / 3 / 2; price selected; “kept piling up in the fridge”Distinguish affordability from paying for unused food; reconstruct whether disengagement preceded price concerns. Backup: S-8093.
S-805629 / 1 / 12; price selected; “cheaper at the supermarket”Same tenure as S-8003, much higher recent cooking. Strong contrast for genuine price pressure despite use; supermarket comparison is not evidence of competitor discounting. Backup: S-8096.
S-800047 / 3 / 0; repetition selected; “kept piling up in the fridge”Test whether menu fatigue caused declining use, or whether routine/quantity problems came first. Backup: S-8209.
S-807223 / 1 / 8; delivery problems selected; no commentExplore an operational explanation among someone still cooking relatively often. Establish whether the problem was lateness, something else, and consequential enough to trigger cancellation. Backup: S-8179.
S-80246 / 5 / 1; other selected; “kept piling up in the fridge”Short-tenure, high-skipping case outside executive labels: explore onboarding, quantity, expectations, and routine formation. Backup: S-8131.
S-803733 / 0 / 5; moving/travelling selected; no commentContrast with frequent skippers; examine life circumstances, whether the label fits, and why cancel rather than skip. Backup: S-8215.

Contact primaries immediately; approach backups if unavailable or declining. Offer the same fixed incentive regardless of feedback. Invite them to discuss “your experience using and stopping Sprig,” without naming hypotheses. Confirm they managed the subscription or cooking; do not screen for agreement with a theory.

Before calls, verify metric definitions, subscription dates, and available delivery history. Skipping reduces opportunities to cook; raw cooking counts are not cooking rates. Schedule three calls Tuesday, three Thursday, with debrief time.

Shared 30-minute interview guide

0–2 minutes: introduction

“We’re learning how Sprig fitted into your life and what led you to stop. This isn’t a sales call; honest criticism is useful. You can skip questions or stop. May we record for research notes?”

2–6: expectations and early use

  • “What was happening in your life when you signed up?”
  • “What did you hope Sprig would replace or make easier?”
  • “Walk me through an early delivery: who cooked, what got cooked, and what happened to the rest?”
  • “How typical was that during your first month?”

6–15: reconstruct the change

  • “Walk me through your last few weeks with Sprig.”
  • “Tell me about the last box you received—from arrival to what happened to each meal.”
  • “When did you first consider stopping? What happened just before that?”
  • “Was there a time it worked better? What changed?”
  • “Tell me about a week you skipped, if any. What drove that decision?”

Anchor memories to deliveries or calendar events; accept “I don’t remember.”

15–20: cancellation and alternatives

  • “Tell me about the day you cancelled. Why then?”
  • “What had you already tried?”
  • “What options did you consider, including skipping?”
  • “What have you actually done for meals since?”

20–27: test explanations, only after the unaided story

Ask everyone: - “How did the cost fit your budget and what you got from it?” Probe actual alternatives, prices or discounts used—not hypothetical willingness to pay. - “How did recipe choice change over time?” Request specific examples and timing. - “How did deliveries fit your plans?” If problems arose: promised versus actual arrival, consequences, frequency, and attempted resolution. - Then show their selected reason/comment: “How well does this capture what happened? What’s missing?”

27–30: confirm

Summarize the sequence, separating ongoing frustrations from the final trigger. “What have I misunderstood? What else mattered?”

Support and synthesis

Pair each novice with an experienced observer; rehearse Monday. Use silence and “Tell me about a specific time,” not suggested answers. Never defend Sprig or pitch fixes.

After each call, record timeline, supporting examples, contradictions, and unknowns separately from interpretation. Friday, compare cases across all explanations, explicitly noting disconfirming evidence. Report mechanisms and next validation steps—not churn percentages or causal conclusions from six interviews.

Grades and run details

Decision model 83 · LLM judge 85
Decision model checks
  • passUses the supplied evidence correctly29%
  • passAddresses the actual decision47%
  • partialRespects explicit constraints38%
  • partialIdentifies material uncertainty43%
  • passAvoids unsupported claims75%
  • passProduces the required deliverable57%
  • partialDoesn't trust the default reason47%
  • passRecruits from the data99%
  • passDesigned to falsify73%
  • passAsks what people did, not what they think89%
  • passDoesn't lead the witness72%
  • partialMarks what to cut if the call runs over55%
Run
Run
#1
Time to output
33 s
Submitted
5 Oct 2026

Needs more than a quick edit10 of 12 checks passed

The output fails to demonstrate the default-option bias with data from the file and does not mark cuttable questions for the 30-minute call, so a PM would need to add those before use.

Output

Aim and guardrails

Use six interviews to identify how cancellation happens, not vote on which executive is right. Explore price, repetition and delivery alongside an alternative: subscribers stop cooking the meals, food accumulates, and the subscription loses value.

The cancellation form is weak evidence: “Too expensive” is pre-selected, only one reason is allowed, and comments often suggest overlapping problems. The first-month cooking difference—81% of retained subscribers versus 34% of cancellers cooked at least three recipes weekly—is a useful lead, not proof that cooking prevents churn. Last-four-week cooking counts cannot establish first-month habits.

Recruit these six

Purposefully sample contrasting behaviors and mismatches between selected reasons and comments. Counts below are weeks subscribed / skips in last six weeks / recipes cooked in last four weeks.

RecruitEvidenceWhy this person
S-805629 / 1 / 12; default price; “cheaper at the supermarket”Relatively active cook with an explicit price comparison. Explore affordability, competing offers and value even when meals are used.
S-800329 / 3 / 2; default price; “kept piling up in the fridge”Same tenure as S-8056, very different usage. Distinguish unaffordable price from paying for unused food; investigate whether disengagement preceded the price concern.
S-800047 / 3 / 0; repetition; “kept piling up in the fridge”Long-tenure repetition claim with no recent cooking. Look for specific recipe fatigue and when it started, versus meal backlog or changed routines.
S-807223 / 1 / 8; delivery problems; no commentContinued cooking despite a delivery complaint. Reconstruct a concrete delivery incident, its consequences and whether lateness actually drove cancellation.
S-80246 / 5 / 1; other; “kept piling up in the fridge”Early disengagement outside the executive categories. Explore onboarding, cooking effort, portions and whether Sprig ever fitted the household routine.
S-825044 / 1 / 6; moving/travelling; “too much food for two”Test a situational explanation against household/quantity mismatch; understand why cancellation rather than skipping made sense.

Backups, in corresponding order: S-8096, S-8122, S-8209, S-8179, S-8259, S-8151. They preserve the main behavioral contrast, though comments differ.

Invite neutrally: “We’re learning about your experience using and stopping Sprig; this is not a sales call.” Book six 30-minute calls plus replacement availability. Do not screen for agreement with an executive theory. Confirm dates and household context; validate unusual counts rather than treating administrative data as ground truth.

Shared 30-minute interview guide

0–2 minutes: Introduction

“Thanks for joining. We want to understand what happened in your experience, including anything that worked or didn’t. There are no right answers, and we won’t try to persuade you to return. May we record for research notes? You can decline any question.”

2–6 minutes: Expectations and first month

  • “What was happening in your life when you signed up?”
  • “What did you hope Sprig would replace or make easier?”
  • “Walk me through a typical week in your first month: receiving, choosing and cooking meals.”
  • “What happened to meals you didn’t cook?”

Establish actual routine before asking whether it changed.

6–16 minutes: Reconstruct the cancellation timeline

  • “Take me through your last few weeks with Sprig.”
  • “When did you first consider stopping? What happened just before that?”
  • “Tell me about the last box you received: what arrived, what you cooked, and what happened to the rest.”
  • “Tell me about a week you skipped. What led to that?”
  • “What happened on the day you cancelled? Why then rather than earlier?”

Probe for dates, examples, actions and consequences. Separate gradual dissatisfaction from the final trigger. Ask what they tried, including changes to orders, skipping or contacting support.

16–21 minutes: Replacement and trade-offs

  • “How are you handling dinners now? Tell me about last week.”
  • “What alternatives did you actually consider or use?”
  • “What did those cost, and what did you get for that?”
  • “What made cancelling preferable to continuing or skipping?”

Distinguish actual competitor switching from general perceptions of cheaper options.

21–27 minutes: Cover gaps, then reconcile the form

Only after the unprompted account: - Price: “How did the cost fit your budget? Did anything change?” - Recipes: “How did recipe choices change over time? Any specific examples?” - Delivery: “How did deliveries fit your plans? Describe any problem and what followed.” - Usage: “What made cooking the meals easier or harder?”

Then: “The form recorded [reason]. How well did that capture your situation? Were other reasons missing?” Clarify whether they actively chose it.

27–30 minutes: Close

Summarize the sequence and ask: “What have I misunderstood? What mattered most that we haven’t discussed?”

Support and synthesis

Run a rehearsal for both new interviewers; pair each with an experienced observer. Use silence and neutral probes—“What happened next?”—not “Was it too expensive?” Never defend Sprig.

Debrief immediately using one template: expectations, usage trajectory, incidents, underlying constraints, cancellation trigger, replacement, supporting quotes and uncertainty. Compare confirming and contradicting cases. Report plausible mechanisms and follow-up tests—not prevalence estimates or a causal verdict from six calls.

Grades and run details

Decision model 83 · LLM judge 77
Decision model checks
  • passUses the supplied evidence correctly56%
  • passAddresses the actual decision35%
  • partialRespects explicit constraints26%
  • partialIdentifies material uncertainty37%
  • passAvoids unsupported claims82%
  • passProduces the required deliverable47%
  • partialDoesn't trust the default reason66%
  • passRecruits from the data98%
  • passDesigned to falsify71%
  • passAsks what people did, not what they think92%
  • passDoesn't lead the witness75%
  • partialMarks what to cut if the call runs over45%
Run
Run
#1
API response time
47 s
Submitted
6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

All got wrong 2

Doesn't trust the default reasonWrongWrongWrong
Sonnet 5.5 · API

It names the pre-selected default as a bias but does not show from the file that most 'Too expensive' cancellers had stopped cooking; it instead says many cooked heavily or wrote other text, missing the key pattern.

GPT-6 Astra · ChatGPT

The plan names the pre-selected default as a bias but does not use the behaviour columns to show that most 'too expensive' cancellers had stopped cooking; it only picks two contrasting individuals without the aggregate pattern.

GPT-6.1 Sol · API

It names the pre-selected default as a bias but does not use the behaviour columns to show that most 'too expensive' cancellers had stopped cooking (e.g., the 118 of 156 pattern).

Marks what to cut if the call runs overWrongWrongWrong
Sonnet 5.5 · API

The guide has timed sections adding to 30 minutes but does not mark which questions are must-ask or what to cut if the call runs over.

GPT-6 Astra · ChatGPT

The guide includes timed sections that sum to 30 minutes but does not mark which questions to cut if the call runs over, as required.

GPT-6.1 Sol · API

The guide has timed sections that sum to 30 minutes but does not mark which questions are must-ask or what to cut if time runs short.

All got right 10

Uses the supplied evidence correctlyRightRightRight
Sonnet 5.5 · API

All factual claims about the current situation are directly supported by the supplied context or derived from the data without invention.

GPT-6 Astra · ChatGPT

All factual claims about the current situation are taken directly from the supplied context or derived by simple arithmetic from it.

GPT-6.1 Sol · API

All factual claims about the current situation are taken directly from the supplied context or follow from the data.

Addresses the actual decisionRightRightRight
Sonnet 5.5 · API

The output commits to a clear plan (six specific recruits and a full interview guide) and states the aim and next steps after the calls.

GPT-6 Astra · ChatGPT

The output commits to a clear recruitment plan and interview guide, and says to approach backups if primaries are unavailable.

GPT-6.1 Sol · API

The output commits to a clear recruitment plan and interview guide, which is the requested deliverable.

Respects explicit constraintsRightRightRight
Sonnet 5.5 · API

The output is under 900 words, addresses the two new interviewers with pairing and rules, and provides both the recruitment plan and the guide as requested.

GPT-6 Astra · ChatGPT

The plan is under 900 words, addresses the two new interviewers with pairing and rehearsal, and delivers the requested recruit list and guide.

GPT-6.1 Sol · API

The output is under 900 words, addresses the two new interviewers with rehearsal and pairing, and follows the requested form.

Identifies material uncertaintyRightRightRight
Sonnet 5.5 · API

It identifies the key unknown about whether the fading group never built the habit or lost it, and notes the calls should find out.

GPT-6 Astra · ChatGPT

The output labels the cooking gap as an association, notes the cancel form is weak evidence, and instructs to report mechanisms and next validation steps rather than causal conclusions.

GPT-6.1 Sol · API

It acknowledges the form's weakness, the first-month data's limitations, and instructs to report uncertainty and not draw causal conclusions from six calls.

Avoids unsupported claimsRightRightRight
Sonnet 5.5 · API

Interpretations like the two-group pattern are presented as observations from the data, not as proven causes, and are appropriately hedged.

GPT-6 Astra · ChatGPT

Interpretations are clearly labelled as such (e.g., 'important association, not proof of causation'), and no cause is presented as established fact.

GPT-6.1 Sol · API

Interpretations like 'comments often suggest overlapping problems' are grounded in the data and not presented as absolute facts.

Produces the required deliverableRightRightRight
Sonnet 5.5 · API

The output is a complete, usable plan with recruitment table and timed interview guide, within the word limit.

GPT-6 Astra · ChatGPT

The output provides a complete recruitment plan with reasons and a timed interview guide, usable by the retention PM and interviewers.

GPT-6.1 Sol · API

The output provides a complete, usable recruitment plan and interview guide within the word limit.

Recruits from the dataRightRightRight
Sonnet 5.5 · API

It defines behavioral groups (fading, still cooking, repetitive, delivery) and selects six interviewees from the list to test each exec hypothesis, with clear reasons.

GPT-6 Astra · ChatGPT

The six interviewees are chosen by behaviour pattern (low-cooking price, high-cooking price, repetition, delivery, short-tenure other, moving with no skips) to test each hypothesis.

GPT-6.1 Sol · API

The six recruits are chosen by contrasting behaviour patterns (active cook/price, disengaged/price, repetition, delivery, early disengagement, moving) that test each executive hypothesis.

Designed to falsifyRightRightRight
Sonnet 5.5 · API

Each exec hypothesis has a non-leading question that could disprove it (e.g., 'Where did you do the comparison, and what exactly was cheaper?' for price).

GPT-6 Astra · ChatGPT

Each executive hypothesis has a disconfirming question (e.g., 'How did the cost fit your budget and what you got from it?' could show price wasn't the issue).

GPT-6.1 Sol · API

The guide includes questions like 'How did the cost fit your budget?' and 'Describe any problem and what followed' that could yield answers disproving each hypothesis.

Asks what people did, not what they thinkRightRightRight
Sonnet 5.5 · API

Core questions ask about specific recent actions (last box, first few weeks, what they did for meals instead) rather than opinions or hypotheticals.

GPT-6 Astra · ChatGPT

Core questions ask for specific recent instances (last box, week skipped, day of cancellation) and follow up on actions, not opinions or hypotheticals.

GPT-6.1 Sol · API

Core questions ask for specific recent instances (last box, last skipped week, last week's dinners) rather than opinions or hypotheticals.

Doesn't lead the witnessRightRightRight
Sonnet 5.5 · API

Questions are open and neutral, and the guide instructs interviewers not to mention price, recipes, or delivery first.

GPT-6 Astra · ChatGPT

Questions are open and neutral; the guide explicitly instructs interviewers not to defend Sprig or suggest answers.

GPT-6.1 Sol · API

Questions are open and neutral, and the guide explicitly warns against leading probes like 'Was it too expensive?'.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 98% of 42 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5withAPI86.188.52None
2GPT-6.1 SolwithAPI87.884.92None
3Opus 5.5withClaude92.174.72None
4GPT-6 AstrawithChatGPT89.774.52None
5Haiku 5.5withAPI87.870.12None
6GPT-6 LunawithAPI81.673.92None
7Gemini 3.8 FlashwithAPI69.751.92None
8Gemini 3.5 Flash-LitewithGemini60.126.121 capped

What you get for the cost

GPT-6.1 Sol · API scores within a point of Sonnet 5.5 · API at a third of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreSonnet 5.5 · API: task score 86, 43× the cheapestGPT-6.1 Sol · API: task score 85, 13× the cheapestOpus 5.5 · Claude: task score 82, 71× the cheapestGPT-6 Astra · ChatGPT: task score 81, 66× the cheapestHaiku 5.5 · API: task score 77, 2.3× the cheapestGPT-6 Luna · API: task score 77, the cheapestGemini 3.8 Flash · API: task score 58, 8.3× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 41, 3.0× the cheapest, 1 critical failureSonnet 5.5 · APIGPT-6.1 Sol · APIOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTHaiku 5.5 · APIGPT-6 Luna · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costAt least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API771.0×
Haiku 5.5 · API772.3×
Gemini 3.5 Flash-Lite · Gemini413.0×
Gemini 3.8 Flash · API588.3×
GPT-6.1 Sol · API8513×
Sonnet 5.5 · API8643×
GPT-6 Astra · ChatGPT8166×
Opus 5.5 · Claude8271×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Preparing for customer calls.

Why it matters

Leading questions produce the answers you wanted. A good guide is designed to prove you wrong.

What good looks like

  • Past-behaviour questions
  • No leading prompts
  • Clear learning goals
  • Could disprove the team's hypothesis

Deliberately not measured

    Capability tested

    Question design

    The failure we’re looking for

    Hypothetical, leading questions

    Grading

    Decision model and LLM judge, calibrated against a blind PM review