Tasks / Product Craft

Customer research call guide

Can the model write a guide that uncovers behaviour rather than opinions?

Task type v1.0 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 38% were usable with at most a quick edit.

Reliably right

  1. Asks what people did, not what they think92% pass
    Core questions ask about specific recent actions (last boxes, skipped weeks, what they did instead) rather than opinions.
    Opus 5.5 · Claude · Too expensive, by default
  2. Tests both theories fairly91% pass
    Both theories get questions that could disprove them (e.g., no action on robotics, happiness after slow go-live), and open-ended timeline questions leave room for a third cause.
    Sonnet 5.5 · API · Why are warehouses leaving? Two execs, two theories
  3. Different questions for user and signer91% pass
    The guide has distinct tracks: daily user questions focus on day-to-day experience and workarounds, signer questions on the decision process and alternatives.
    Sonnet 5.5 · API · Why are warehouses leaving? Two execs, two theories

Where it slips

  1. Marks what to cut if the call runs over28% pass
    The guide has timed sections but does not mark which questions are must-ask or what to cut if time runs short.
    Opus 5.5 · Claude · Too expensive, by default
  2. Doesn't trust the default reason41% pass
    The plan names the pre-selected default as a bias but does not use the behaviour columns to show that most 'too expensive' cancellers had stopped cooking; it only picks two contrasting individuals without the aggregate pattern.
    GPT-6 Astra · ChatGPT · Too expensive, by default
  3. Uses the supplied evidence correctly64% pass
    The claim 'Customers will say what helps their negotiation' is an unsupported behavioural statement not derived from the context.
    Haiku 5.5 · API · Why are warehouses leaving? Two execs, two theories

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Designed to falsify

    For each learning goal, does the guide include a question whose honest answer could show the team's hypothesis is wrong?

    Passes when Every learning goal has at least one disconfirming, non-leading question, such as asking about the last time they had the problem and what they did instead.

  2. Asks what people did, not what they think

    Are the core questions about specific things the person actually did recently, rather than opinions, predictions or hypotheticals?

    Passes when The core of the guide asks for recent, specific instances (the last time, what happened, what they did) and follows up on them; opinions and predictions are absent or clearly secondary.

  3. Doesn't lead the witness

    Are the questions free of leading wording, of pitching the idea, and of hints at the answer the team hopes for?

    Passes when Questions are neutral and open, and the idea is not described (or only at the very end, after the problem questions).

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Marks what to cut if the call runs over · and 3 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a Staff PM at Crate. In four weeks, the exec team decides which of two bets gets our two squads next half, and both bets rest on a theory of why customers are leaving. We have three weeks for 12 customer calls of 45 minutes. Write the guide for them. It should include: the learning goals; the questions, with timings, for the two kinds of people we'll talk to; guidance for whoever runs the calls; any changes you'd make to the plan below; and what we'd need to hear to back each bet, or neither. Keep it under 1,500 words. The exec team will read it before the calls start.

What the model was given8 items: About Crate, The decision, CEO, Ana Ferreira, CPO, Marcus Hill, Exit survey (41 churned customers, last 12 months), Implementation and CRM data, The call plan, Notes from Customer Success and Legal
About CrateWarehouse management software for mid-size third-party logistics companies (3PLs). 340 customers, $22M ARR. Gross logo churn rose from 11% to 18% over the last year.
The decisionBet A: integrations with warehouse robotics (autonomous carts, pick-assist robots). Bet B: rebuild implementation so customers go live faster. Two squads for the next half go to one of them.
CEO, Ana Ferreira“We're losing warehouses because they're automating and we don't talk to their robots. Every lost customer I've spoken to mentions it.”
CPO, Marcus Hill“We lose them in the first year because implementation takes forever. They're gone before they ever see the value.”
Exit survey (41 churned customers, last 12 months)A single-choice question, 'Why are you leaving?'. Missing features or integrations: 46%. Too hard to implement: 22%. Price: 17%. Other: 15% (free-text box, left blank by all but two). 'Missing features or integrations' was the first option in the list.
Implementation and CRM dataMedian time from signing to go-live: 94 days; the contract promises 45. Customers who went live after 90 days churned at 2.4× the rate of the rest in their first year. 3 churned customers moved to a competitor with robotics integrations; robotics was cited in 9 lost new-logo deals. We don't know when in the year churned customers decided to leave.
The call plan12 calls: 6 with customers who churned in the last 9 months, 6 with at-risk customers (health score red). On each account we want to talk to the warehouse operations manager who used Crate every day and, separately, the VP of Operations or COO who signed the contract. Ana wants to run 4 of the calls with former customers she knows personally.
Notes from Customer Success and Legal5 of the 6 at-risk accounts are in renewal talks, and Customer Success is offering them discounts. Interviewers must not discuss pricing, discounts or what's on our roadmap. Every call must start by asking permission to record.
What a strong answer doesThe answer key the graders mark against

A guide that tests both theories fairly and leaves room for a third: the goals ask what actually drove each decision to leave or doubt, when it happened, and who made it, with questions that could disprove robotics (did they automate, did robots come up before they decided, what did they switch to and why) and implementation (how long go-live took, what happened in those months, whether value arrived) alike. It treats the exit survey as weak evidence (single choice, integrations listed first, 'Other' left blank) and the 2.4× figure as a correlation to probe, not a cause. The core is a timeline of the account: the story from signing to the moment they decided to leave or started doubting, with sequential probes. It has separate tracks for the daily user (what went wrong day to day) and the signer (how the decision was made, what alternatives they weighed), fits 45 minutes with timings, and opens with consent to record. It guards the calls: no pitching, pricing, discounts or roadmap promises, with a script for when an at-risk customer asks, and it addresses Ana's bias: she has a public view and personal relationships, so she should observe or take notes rather than lead, or at minimum use the same script and have her calls read separately. It ends with decision signals: what pattern across the 12 would back Bet A, what would back Bet B, and what would back neither (price, a business change, a missing capability that isn't robotics).

Critical failures (cap the score at 40)
  • Includes questions that promise roadmap items or discuss pricing or discounts with customers in renewal talks
  • Asks customers to choose between robotics and implementation as the reason they left
Task

Staff-level · v1.1

The outputs

8 setups answered this brief.

Usable with a quick edit10 of 13 checks passed

A PM could quickly trim 3 minutes from the timings and add a note on what to cut if over time, and caveat the behavioural claim; the structure and reasoning are otherwise sound.

Output

Customer Interview Guide: Why Are Customers Leaving?

Owner: Staff PM | Calls: 12 × 45 min, over three weeks | Readout: before the exec decision in four weeks

1. Learning goals

  1. Timing: When did the customer first consider leaving, and what was happening then (signing, go-live, first 90 days, renewal)? This tells us whether churn is decided early or late.
  2. Reasons, in their order: What were the reasons for leaving or staying, ranked in the customer's own words, before we prompt anything?
  3. Automation: Are they automating their floor, which systems do they need Crate to connect to, and was that gap a blocker or a workaround?
  4. Implementation: Where did time go between signing and daily use, and who was waiting on whom?
  5. Alternatives: What did they evaluate instead, and what did that option do better?

2. Changes to the plan (please read before the calls start)

  1. Ana should not run calls with accounts she knows. Personal relationships make candid criticism less likely, and Ana is the sponsor of Bet A. She can provide introductions and read the notes afterward. Neutral interviewers run those four calls on accounts she doesn't manage.
  2. The plan implies 24 conversations, not 12. Each account has two contacts. I propose 12 × 45-minute calls with the operations manager on every account, plus a 20-minute executive call (VP/COO) on as many accounts as we can reach, aiming for six.
  3. Rebalance the sample. Churned accounts are 9 of the 12 calls, and the at-risk sample is weighted toward accounts in renewal talks (see point 4). Within the churned group, make sure we have at least three go-lives over 90 days and three under 90, and at least three accounts running automation and three that aren't. Without this spread, we can't tell robotics from implementation.
  4. Renewal talks distort at-risk answers. Five of six at-risk accounts are negotiating renewals with discounts on the table. Customers will say what helps their negotiation, and we can't fully control for that. Treat at-risk answers as evidence of current pain, not reasons for churn. If possible, swap two at-risk accounts for churned accounts from months 9–12 of the exit survey window.
  5. Pull data this week, not after the calls. Get churn-notice dates, go-live dates, and integration usage for all 41 churned accounts. This lets us test the timing question quantitatively.
  6. Don't treat the exit survey as proof. Missing features was the first option, the "Other" box was almost empty, and the question forced one answer. The 46% doesn't tell us it's robotics. Use these calls to check it.
  7. Agree the decision thresholds now (section 5), before we hear anything.

3. Questions and timings

Opening (3 min), verbatim: > "Before we start, may I record this? It's for internal research only, and I'll share unattributed summaries. You can stop at any point. We're not selling anything today, and I can't discuss pricing, offers, or product plans. I want to understand what happened for you."

A. Context (5 min) - Walk me through your warehouse operation and your role in it. - What does Crate do for you day to day, or what did it do before you stopped using it?

B. The timeline (12 min) Most important block. Get dates. - Walk me through the last time you were seriously considering leaving [or: seriously worried about the relationship]. When did that start? - What was happening at that point? Who was involved? - When did it become a real decision? What triggered it? - Probe for dates: "Roughly which month was that? What was going on at go-live?"

C. Reasons, unprompted (8 min) Don't mention robotics yet. - Looking back, what were all the reasons, in order of importance? - What made the difference between staying and leaving? - If we'd fixed one thing, would you have stayed? - What did you look at instead? What did it do that we didn't?

D. Automation (7 min) Only after block C. - What has changed on your floor over the past two years? Any carts, pick-assist robots, or other automation? - Which systems does Crate need to connect with? How did your team work around the gap? - Was that a blocker, or a nice-to-have? - Did anyone raise it in a renewal or evaluation discussion?

E. Implementation (8 min) - How long did it take from signing to your team using Crate daily? - Where did the time go? Who was waiting on whom? - When did your team first feel it was working?

F. Close (5 min) - If you were us, what would you fix first? - Is there anything I should have asked?

Executive variant (20 min): Spend the time on blocks A, B, and C, plus: how the decision was made and who approved it; what they evaluated; how automation fits into their operating plan for the next 12 months; and what they told their team about Crate.

At-risk variant: Replace "what made the difference between staying and leaving?" with "what's working well today and what's getting in the way?" Do not ask about renewal, pricing, or whether they plan to stay.

4. Guidance for interviewers

  • Two people per call: one leads, one takes notes. No one from Sales or Customer Success on the call.
  • Don't name robotics, the 46% figure, or our hypotheses. Don't ask leading questions ("Isn't it frustrating that…?").
  • Ask about specific past events, not hypotheticals ("Would you use…?"). Ask "What else?" until they run out.
  • Let silence work. Don't fill it.
  • Don't promise anything. If asked about features or fixes: "I'll pass that on, but I can't say what we'll build."
  • Pricing or discounts: Don't engage. Say "I can't discuss that today. I'll make sure your account team knows you raised it." Record their words verbatim and tell Customer Success after the call. CS owns the follow-up.
  • Debrief within two hours. Fill in one template: go-live and churn dates, reasons in the customer's order, whether robotics came up unprompted or after a prompt, implementation duration, and up to three verbatim quotes.

5. What we'd need to hear

Thresholds are directional. Twelve calls can't prove anything, but they can show whether a theory holds up.

Back Bet A (robotics integrations) if: - At least 5 of 12 name a specific automation system that Crate can't connect to as a primary reason for leaving or evaluating alternatives. This should come up in block C or D, not only after prompting. - At least 3 say the integration would have changed the outcome. - The decision to leave happened late in the contract, not during implementation. - Churned accounts that ran automation churned for reasons that match the exit survey.

Back Bet B (faster implementation) if: - At least 5 of 12 say the decision started during go-live or in the first 90 days. - They name specific bottlenecks: data migration, configuration, waiting on our team, or connecting to their existing systems. - At least 3 say faster go-live would have kept them. - In the pulled data, go-live time separates churned accounts from retained ones more sharply than automation does.

Back neither if: - Reasons are spread across price, service, reporting, or other issues, with no dominant theme. - Robotics comes up, but few customers have automation projects planned in the next 12 months. - Implementation was slow, but the decision to leave was made at renewal for unrelated reasons. - The real driver is weak executive sponsorship or a lack of internal champions, which neither bet addresses.

If neither bet is backed, the readout should say so directly and name the pattern that did emerge. Don't force a choice between A and B if the evidence points elsewhere. If the evidence supports both, the readout should say which has stronger support and what would change that conclusion.

Claims the judge couldn’t find in the brief

  • Customers will say what helps their negotiation.

Grades and run details

Decision model 92 · LLM judge 79
Decision model checks
  • passUses the supplied evidence correctly1%
  • passAddresses the actual decision91%
  • partialRespects explicit constraints23%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims64%
  • passProduces the required deliverable80%
  • passTests both theories fairly98%
  • passDifferent questions for user and signer66%
  • passProtects the calls and the accounts95%
  • passDesigned to falsify82%
  • passAsks what people did, not what they think79%
  • passDoesn't lead the witness76%
  • partialMarks what to cut if the call runs over37%
Run
Run
#1
API response time
45 s
Submitted
8 Oct 2026

Usable with a quick edit12 of 13 checks passed

The guide is thorough, respects all constraints, and provides actionable scripts and decision criteria. A PM could use it with only minor wording tweaks.

Output

Crate churn calls: interview guide

Purpose: Decide between Bet A (robotics integrations) and Bet B (faster implementation) by learning why customers actually leave. Neither theory is proven yet.

Why we're not already sure

  • Exit survey: "Missing features or integrations" bundles two things, isn't robotics-specific, was listed first, and has only 41 responses. People also tend to name the most recent or most socially easy reason rather than the first cause.
  • Robotics: 3 of roughly 60 churned accounts went to a robotics competitor. The 9 lost deals are real but are new-logo losses, not churn.
  • Implementation: The 2.4× churn gap after 90+ days is a correlation. Complex sites may be both slow to launch and likely to leave. We also don't know when customers decided to leave.
  • Ana's and Marcus's evidence: Each is anecdotal and selected. Ana hears from customers who will talk to the CEO.
  • The two theories may be linked. Robotics-heavy sites may be the slow implementations.

Learning goals

  1. When and why each account first considered leaving, relative to go-live.
  2. Whether automation was a cause, a rationalization, or irrelevant. The test is whether they took action (bought robots, evaluated replacements), not whether they mention it.
  3. Whether slow go-live caused the exit, and whether the delay was Crate's doing or the customer's (their data, staffing, customizations).
  4. What else drives exits: price, support, core product gaps, or events outside our control, such as a 3PL losing a shipper contract or being acquired.
  5. Whether the two causes overlap.

Changes to the plan

  • Call math. Two roles per account means 12 calls covers only 6 accounts. I'd use 8 accounts: 4 churned accounts with both roles (8 calls), and 4 at-risk accounts with the operations manager only (4 calls). At-risk executives are the people in renewal negotiations, which is where contamination risk is highest. Comparing what the signer and the daily user each say is itself valuable.
  • Recruit on criteria, not convenience. Among churned accounts, aim for 2 that left within 12 months of signing and 2 that left later. Include go-lives both over and under 90 days, at least 2 sites with robots or carts, and at least 1 that chose "price" in the survey. Pull this from CRM and Customer Success notes this week.
  • Ana doesn't run calls. Customers who know the CEO will be polite, and she has a stated hypothesis. Her relationships are valuable for introductions, and at most 1–2 of her contacts should be in the churned set. Someone with no stake in either bet runs the calls, with a second person taking notes. Ana and Marcus get recordings and summaries afterward.
  • At-risk accounts need Customer Success coordination. CS confirms no discount conversation is happening that week. The invitation states that this is research, separate from renewal. If a customer tries to negotiate, redirect and tell CS afterward. Weight at-risk answers less, since they describe intentions, not decisions.
  • Pre-register. Before call one, Ana and Marcus each read the last section and write what would change their mind.
  • Run a parallel data pull. Get tenure at churn, go-live days, and robotics on site for all roughly 60 churned accounts. If the calls and the data disagree, the data wins.

Guidance for whoever runs the calls

  • Consent first. The first thing you say after your name is a request to record. If they decline, take notes only and continue.
  • Off-limits: pricing, discounts, and roadmap. If they raise price, listen and ask what they compared us with and what they got, but no numbers or offers. If asked "will you build X?", say: "I can't speak to plans. I'm here to understand your experience." If they ask for a commercial conversation, point them to their account manager.
  • Do not say "robotics" or "implementation" until the prompted section. Record whether each topic came up unprompted or only when asked. These carry very different weight.
  • Alternate the order of the two prompted topics from call to call.
  • Ask for stories and dates, not opinions: "Tell me about the last time," "What happened next?" Favor actions ("Who did you call? What did you buy?") over stated reasons.
  • Don't defend Crate or fix problems. Answer criticism with "Say more about that."
  • Use silence. Wait a few seconds after an answer; the real reason often comes next.
  • Debrief within 24 hours on a one-page sheet: decision date, tenure, go-live days, robots on site, trigger event, unprompted causes, prompted causes, strongest quote, and anything that contradicts our hypotheses.

Guide: Operations manager (daily user), 45 min

TimeSectionQuestions
0:00–0:03OpenAsk permission to record. "I'm [name] from Crate's product team. I'm here to learn, not sell, and nothing you say affects your account. I can't discuss pricing or plans."
0:03–0:08ContextTell me about your site and your role. What does a normal day look like? What equipment and systems run alongside Crate?
0:08–0:20Start-up storyTake me back to when you started with Crate. What happened between signing and running real orders? What got delayed, and why? Who was involved on each side? When did it first feel like it was working?
0:20–0:32Doubt timelineWhen did you first wonder if Crate was the right system? Where were you and what happened? What did you do next, and who did you talk to? (Churned: what was the last straw? At-risk: have you looked at alternatives, and what prompted that?)
0:32–0:39Prompted topicsAutomation: Do you use, or plan to use, robots, autonomous carts, or pick-assist? What's in place, and when did it arrive? How did it work with Crate, and what did you do about it? Setup: Looking back, how did the time to get live affect your team or your view of Crate? (Skip anything already covered in depth.)
0:39–0:43CounterfactualWhat one thing would have kept you? (At-risk: what would need to change for you to be confident staying?) Has anything you've seen elsewhere worked better?
0:43–0:45CloseWhat haven't I asked that I should have? Who else should I talk to? Thank them.

Guide: VP Operations / COO (signer), 45 min

TimeSectionQuestions
0:00–0:03OpenSame as above. Ask permission to record first.
0:03–0:08ContextTell me about your business and your customers. How has the past year changed things for you?
0:08–0:18BuyingWhy did you choose Crate? What did you hope would be different after 6–12 months? What were you told about go-live, and what happened? How did you know whether it was working?
0:18–0:30DecisionWalk me through the decision to leave (or to reconsider). When did it first come up, and who raised it? What did you look at? Which alternatives did you consider? Who made the final call, and when? What would have changed the outcome?
0:30–0:38Prompted topicsAutomation: What's your automation plan over the next 2–3 years? What's happened so far? How did it figure in the decision? Setup: How did the time to go live figure in your thinking, if at all? (Skip anything already covered in depth.)
0:38–0:43Business contextWhat else changed around then: shipper contracts won or lost, sites opened or closed, leadership, budget, acquisitions? If you ranked everything that mattered, what's first? What do you use now, and why?
0:43–0:45CloseAnything I missed? Thank them.

What we'd need to hear

Eight accounts can't produce percentages. We're looking for a pattern strong enough to act on.

Back Bet A (robotics) if, in at least 3 of the 4 churned accounts: - Automation comes up unprompted, or with only light prompting, as a reason. - They acted on it: robots bought, piloted, or contracted before they decided to leave. - They were live and broadly content with Crate beforehand. - They evaluated or moved to something that connects to their robots, and say they'd likely have stayed if we had. - The same story appears in at least 2 of the 4 at-risk accounts.

Back Bet B (implementation) if, in at least 3 of the 4 churned accounts: - Their first serious doubt came during or shortly after a late go-live, typically within the first year. - They describe never reaching value, or losing their internal champion during the delay. - The delay was mainly Crate's doing (process, handoffs, configuration), not their own data or staffing. - They would say faster go-live would probably have kept them.

Neither bet if: - Exits are mainly driven by events such as a lost shipper contract, site closure, acquisition, or price. - Robotics is mentioned but no one acted on it. - Go-live was slow but they were happy afterward. - Reasons scatter with no pattern. - Core product gaps, such as reporting or inventory accuracy, show up repeatedly. That would point to a third bet we haven't framed.

Both: If the same accounts show robotics-heavy sites with long go-lives because integrations took months, that is a third reading. It favors Bet A if the cause is missing integrations, and Bet B if it's our process. Decide by asking what the customer says was the blocker.

Split results: If churned accounts split evenly, or contradict the data pull, say so. Recommend a targeted follow-up, such as segmenting churn by tenure, rather than forcing a choice.

Grades and run details

Decision model 85 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly13%
  • passAddresses the actual decision94%
  • partialRespects explicit constraints19%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims71%
  • passProduces the required deliverable91%
  • passTests both theories fairly100%
  • passDifferent questions for user and signer100%
  • passProtects the calls and the accounts96%
  • passDesigned to falsify82%
  • passAsks what people did, not what they think80%
  • passDoesn't lead the witness42%
  • partialMarks what to cut if the call runs over60%
Run
Run
#1
API response time
1.6 min
Submitted
30 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 2

Avoids unsupported claimsMixedRight
Haiku 5.5 · API

The statement 'Customers will say what helps their negotiation' is presented as fact without caveat or support.

Sonnet 5.5 · API

Hypotheses and interpretations are clearly labelled as such, and no cause or forecast is presented as established fact.

Marks what to cut if the call runs overWrongRight
Haiku 5.5 · API

The timed sections sum to 48 minutes, exceeding the 45-minute limit, and the guide does not mark what to cut if time runs short.

Sonnet 5.5 · API

Both 45-minute guides have timed sections that sum correctly, a realistic number of questions, and instructions to skip covered topics if time is short.

All mixed 1

Uses the supplied evidence correctlyMixedMixed
Haiku 5.5 · API

The claim 'Customers will say what helps their negotiation' is an unsupported behavioural statement not derived from the context.

Sonnet 5.5 · API

All factual claims about the current situation are directly supported by the brief or derived correctly from its numbers.

All got right 10

Addresses the actual decisionRightRight
Haiku 5.5 · API

Section 5 clearly lays out what patterns would back Bet A, Bet B, or neither, satisfying the requirement for decision signals.

Sonnet 5.5 · API

The output provides clear, actionable criteria for what would back each bet or neither after the calls, exactly as the brief requests.

Respects explicit constraintsRightRight
Haiku 5.5 · API

The output is under 1,500 words, includes all requested elements, and enforces pricing/roadmap silence with a script.

Sonnet 5.5 · API

The guide respects all constraints: it stays under 1,500 words, includes all requested sections, and provides concrete scripts and plans to enforce the off-limits topics and consent.

Identifies material uncertaintyRightRight
Haiku 5.5 · API

It explicitly treats the exit survey as weak evidence, notes the small sample cannot prove anything, and says what to do if neither bet is backed.

Sonnet 5.5 · API

The output explicitly names the key unknowns (weak survey, correlation vs causation, unknown decision timing) and says how the calls and a parallel data pull would resolve them.

Produces the required deliverableRightRight
Haiku 5.5 · API

It delivers a complete interview guide in the required form for the exec team, within the word limit.

Sonnet 5.5 · API

The guide is complete with learning goals, timed questions for two roles, interviewer guidance, plan changes, and decision signals, all under 1,500 words and ready for the exec team.

Tests both theories fairlyRightRight
Haiku 5.5 · API

Open questions on reasons come first; automation and implementation are probed separately with neutral questions, giving both theories equal chance to fail.

Sonnet 5.5 · API

Both theories get questions that could disprove them (e.g., no action on robotics, happiness after slow go-live), and open-ended timeline questions leave room for a third cause.

Different questions for user and signerRightRight
Haiku 5.5 · API

Distinct tracks exist for the operations manager (main 45-min guide) and the executive (20-min variant focusing on decision process).

Sonnet 5.5 · API

The guide has distinct tracks: daily user questions focus on day-to-day experience and workarounds, signer questions on the decision process and alternatives.

Protects the calls and the accountsRightRight
Haiku 5.5 · API

Consent is requested at the start, a script deals with pricing/roadmap inquiries, and the CEO is instructed not to run calls with accounts she knows.

Sonnet 5.5 · API

It includes a clear off-limits rule with scripts for pricing/roadmap, starts with consent, and addresses the CEO's bias by having her not lead calls and limiting her contacts.

Designed to falsifyRightRight
Haiku 5.5 · API

Every learning goal includes open, non-leading questions that can yield answers disproving the team's hypotheses (e.g., asking if automation was a blocker or nice-to-have).

Sonnet 5.5 · API

Every learning goal has a disconfirming question, such as asking what they did about robots or whether they ever reached value after a slow go-live.

Asks what people did, not what they thinkRightRight
Haiku 5.5 · API

Core questions ask about the last time they considered leaving, what happened, and what they did, focusing on specific past events.

Sonnet 5.5 · API

Core questions ask for specific recent actions and events ('What happened next?', 'Who did you call?'), not opinions or predictions.

Doesn't lead the witnessRightRight
Haiku 5.5 · API

Questions are neutral (e.g., 'What has changed on your floor?', 'Was that a blocker, or a nice-to-have?') and avoid pitching or hinting at desired answers.

Sonnet 5.5 · API

Questions are neutral and open-ended; the prompted topics are introduced without pitching or hinting at the desired answer.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 98% of 42 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5withAPI86.188.52None
2GPT-6.1 SolwithAPI87.884.92None
3Opus 5.5withClaude92.174.72None
4GPT-6 AstrawithChatGPT89.774.52None
5Haiku 5.5withAPI87.870.12None
6GPT-6 LunawithAPI81.673.92None
7Gemini 3.8 FlashwithAPI69.751.92None
8Gemini 3.5 Flash-LitewithGemini60.126.121 capped

What you get for the cost

GPT-6.1 Sol · API scores within a point of Sonnet 5.5 · API at a third of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreSonnet 5.5 · API: task score 86, 43× the cheapestGPT-6.1 Sol · API: task score 85, 13× the cheapestOpus 5.5 · Claude: task score 82, 71× the cheapestGPT-6 Astra · ChatGPT: task score 81, 66× the cheapestHaiku 5.5 · API: task score 77, 2.3× the cheapestGPT-6 Luna · API: task score 77, the cheapestGemini 3.8 Flash · API: task score 58, 8.3× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 41, 3.0× the cheapest, 1 critical failureSonnet 5.5 · APIGPT-6.1 Sol · APIOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTHaiku 5.5 · APIGPT-6 Luna · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costAt least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API771.0×
Haiku 5.5 · API772.3×
Gemini 3.5 Flash-Lite · Gemini413.0×
Gemini 3.8 Flash · API588.3×
GPT-6.1 Sol · API8513×
Sonnet 5.5 · API8643×
GPT-6 Astra · ChatGPT8166×
Opus 5.5 · Claude8271×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Preparing for customer calls.

Why it matters

Leading questions produce the answers you wanted. A good guide is designed to prove you wrong.

What good looks like

  • Past-behaviour questions
  • No leading prompts
  • Clear learning goals
  • Could disprove the team's hypothesis

Deliberately not measured

    Capability tested

    Question design

    The failure we’re looking for

    Hypothetical, leading questions

    Grading

    Decision model and LLM judge, calibrated against a blind PM review