Tasks / Product Craft

Customer research call guide

Can the model write a guide that uncovers behaviour rather than opinions?

Task type v1.0 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 38% were usable with at most a quick edit.

Reliably right

  1. Asks what people did, not what they think92% pass
    Core questions ask about specific recent actions (last boxes, skipped weeks, what they did instead) rather than opinions.
    Opus 5.5 · Claude · Too expensive, by default
  2. Tests both theories fairly91% pass
    Both theories get questions that could disprove them (e.g., no action on robotics, happiness after slow go-live), and open-ended timeline questions leave room for a third cause.
    Sonnet 5.5 · API · Why are warehouses leaving? Two execs, two theories
  3. Different questions for user and signer91% pass
    The guide has distinct tracks: daily user questions focus on day-to-day experience and workarounds, signer questions on the decision process and alternatives.
    Sonnet 5.5 · API · Why are warehouses leaving? Two execs, two theories

Where it slips

  1. Marks what to cut if the call runs over28% pass
    The guide has timed sections but does not mark which questions are must-ask or what to cut if time runs short.
    Opus 5.5 · Claude · Too expensive, by default
  2. Doesn't trust the default reason41% pass
    The plan names the pre-selected default as a bias but does not use the behaviour columns to show that most 'too expensive' cancellers had stopped cooking; it only picks two contrasting individuals without the aggregate pattern.
    GPT-6 Astra · ChatGPT · Too expensive, by default
  3. Uses the supplied evidence correctly64% pass
    The claim 'Customers will say what helps their negotiation' is an unsupported behavioural statement not derived from the context.
    Haiku 5.5 · API · Why are warehouses leaving? Two execs, two theories

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Designed to falsify

    For each learning goal, does the guide include a question whose honest answer could show the team's hypothesis is wrong?

    Passes when Every learning goal has at least one disconfirming, non-leading question, such as asking about the last time they had the problem and what they did instead.

  2. Asks what people did, not what they think

    Are the core questions about specific things the person actually did recently, rather than opinions, predictions or hypotheticals?

    Passes when The core of the guide asks for recent, specific instances (the last time, what happened, what they did) and follows up on them; opinions and predictions are absent or clearly secondary.

  3. Doesn't lead the witness

    Are the questions free of leading wording, of pitching the idea, and of hints at the answer the team hopes for?

    Passes when Questions are neutral and open, and the idea is not described (or only at the very end, after the problem questions).

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Marks what to cut if the call runs over · and 3 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a Staff PM at Crate. In four weeks, the exec team decides which of two bets gets our two squads next half, and both bets rest on a theory of why customers are leaving. We have three weeks for 12 customer calls of 45 minutes. Write the guide for them. It should include: the learning goals; the questions, with timings, for the two kinds of people we'll talk to; guidance for whoever runs the calls; any changes you'd make to the plan below; and what we'd need to hear to back each bet, or neither. Keep it under 1,500 words. The exec team will read it before the calls start.

What the model was given8 items: About Crate, The decision, CEO, Ana Ferreira, CPO, Marcus Hill, Exit survey (41 churned customers, last 12 months), Implementation and CRM data, The call plan, Notes from Customer Success and Legal
About CrateWarehouse management software for mid-size third-party logistics companies (3PLs). 340 customers, $22M ARR. Gross logo churn rose from 11% to 18% over the last year.
The decisionBet A: integrations with warehouse robotics (autonomous carts, pick-assist robots). Bet B: rebuild implementation so customers go live faster. Two squads for the next half go to one of them.
CEO, Ana Ferreira“We're losing warehouses because they're automating and we don't talk to their robots. Every lost customer I've spoken to mentions it.”
CPO, Marcus Hill“We lose them in the first year because implementation takes forever. They're gone before they ever see the value.”
Exit survey (41 churned customers, last 12 months)A single-choice question, 'Why are you leaving?'. Missing features or integrations: 46%. Too hard to implement: 22%. Price: 17%. Other: 15% (free-text box, left blank by all but two). 'Missing features or integrations' was the first option in the list.
Implementation and CRM dataMedian time from signing to go-live: 94 days; the contract promises 45. Customers who went live after 90 days churned at 2.4× the rate of the rest in their first year. 3 churned customers moved to a competitor with robotics integrations; robotics was cited in 9 lost new-logo deals. We don't know when in the year churned customers decided to leave.
The call plan12 calls: 6 with customers who churned in the last 9 months, 6 with at-risk customers (health score red). On each account we want to talk to the warehouse operations manager who used Crate every day and, separately, the VP of Operations or COO who signed the contract. Ana wants to run 4 of the calls with former customers she knows personally.
Notes from Customer Success and Legal5 of the 6 at-risk accounts are in renewal talks, and Customer Success is offering them discounts. Interviewers must not discuss pricing, discounts or what's on our roadmap. Every call must start by asking permission to record.
What a strong answer doesThe answer key the graders mark against

A guide that tests both theories fairly and leaves room for a third: the goals ask what actually drove each decision to leave or doubt, when it happened, and who made it, with questions that could disprove robotics (did they automate, did robots come up before they decided, what did they switch to and why) and implementation (how long go-live took, what happened in those months, whether value arrived) alike. It treats the exit survey as weak evidence (single choice, integrations listed first, 'Other' left blank) and the 2.4× figure as a correlation to probe, not a cause. The core is a timeline of the account: the story from signing to the moment they decided to leave or started doubting, with sequential probes. It has separate tracks for the daily user (what went wrong day to day) and the signer (how the decision was made, what alternatives they weighed), fits 45 minutes with timings, and opens with consent to record. It guards the calls: no pitching, pricing, discounts or roadmap promises, with a script for when an at-risk customer asks, and it addresses Ana's bias: she has a public view and personal relationships, so she should observe or take notes rather than lead, or at minimum use the same script and have her calls read separately. It ends with decision signals: what pattern across the 12 would back Bet A, what would back Bet B, and what would back neither (price, a business change, a missing capability that isn't robotics).

Critical failures (cap the score at 40)
  • Includes questions that promise roadmap items or discuss pricing or discounts with customers in renewal talks
  • Asks customers to choose between robotics and implementation as the reason they left
Task

Staff-level · v1.1

The outputs

8 setups answered this brief.

Usable with a quick edit10 of 13 checks passed

A PM could quickly trim 3 minutes from the timings and add a note on what to cut if over time, and caveat the behavioural claim; the structure and reasoning are otherwise sound.

Output

Customer Interview Guide: Why Are Customers Leaving?

Owner: Staff PM | Calls: 12 × 45 min, over three weeks | Readout: before the exec decision in four weeks

1. Learning goals

  1. Timing: When did the customer first consider leaving, and what was happening then (signing, go-live, first 90 days, renewal)? This tells us whether churn is decided early or late.
  2. Reasons, in their order: What were the reasons for leaving or staying, ranked in the customer's own words, before we prompt anything?
  3. Automation: Are they automating their floor, which systems do they need Crate to connect to, and was that gap a blocker or a workaround?
  4. Implementation: Where did time go between signing and daily use, and who was waiting on whom?
  5. Alternatives: What did they evaluate instead, and what did that option do better?

2. Changes to the plan (please read before the calls start)

  1. Ana should not run calls with accounts she knows. Personal relationships make candid criticism less likely, and Ana is the sponsor of Bet A. She can provide introductions and read the notes afterward. Neutral interviewers run those four calls on accounts she doesn't manage.
  2. The plan implies 24 conversations, not 12. Each account has two contacts. I propose 12 × 45-minute calls with the operations manager on every account, plus a 20-minute executive call (VP/COO) on as many accounts as we can reach, aiming for six.
  3. Rebalance the sample. Churned accounts are 9 of the 12 calls, and the at-risk sample is weighted toward accounts in renewal talks (see point 4). Within the churned group, make sure we have at least three go-lives over 90 days and three under 90, and at least three accounts running automation and three that aren't. Without this spread, we can't tell robotics from implementation.
  4. Renewal talks distort at-risk answers. Five of six at-risk accounts are negotiating renewals with discounts on the table. Customers will say what helps their negotiation, and we can't fully control for that. Treat at-risk answers as evidence of current pain, not reasons for churn. If possible, swap two at-risk accounts for churned accounts from months 9–12 of the exit survey window.
  5. Pull data this week, not after the calls. Get churn-notice dates, go-live dates, and integration usage for all 41 churned accounts. This lets us test the timing question quantitatively.
  6. Don't treat the exit survey as proof. Missing features was the first option, the "Other" box was almost empty, and the question forced one answer. The 46% doesn't tell us it's robotics. Use these calls to check it.
  7. Agree the decision thresholds now (section 5), before we hear anything.

3. Questions and timings

Opening (3 min), verbatim: > "Before we start, may I record this? It's for internal research only, and I'll share unattributed summaries. You can stop at any point. We're not selling anything today, and I can't discuss pricing, offers, or product plans. I want to understand what happened for you."

A. Context (5 min) - Walk me through your warehouse operation and your role in it. - What does Crate do for you day to day, or what did it do before you stopped using it?

B. The timeline (12 min) Most important block. Get dates. - Walk me through the last time you were seriously considering leaving [or: seriously worried about the relationship]. When did that start? - What was happening at that point? Who was involved? - When did it become a real decision? What triggered it? - Probe for dates: "Roughly which month was that? What was going on at go-live?"

C. Reasons, unprompted (8 min) Don't mention robotics yet. - Looking back, what were all the reasons, in order of importance? - What made the difference between staying and leaving? - If we'd fixed one thing, would you have stayed? - What did you look at instead? What did it do that we didn't?

D. Automation (7 min) Only after block C. - What has changed on your floor over the past two years? Any carts, pick-assist robots, or other automation? - Which systems does Crate need to connect with? How did your team work around the gap? - Was that a blocker, or a nice-to-have? - Did anyone raise it in a renewal or evaluation discussion?

E. Implementation (8 min) - How long did it take from signing to your team using Crate daily? - Where did the time go? Who was waiting on whom? - When did your team first feel it was working?

F. Close (5 min) - If you were us, what would you fix first? - Is there anything I should have asked?

Executive variant (20 min): Spend the time on blocks A, B, and C, plus: how the decision was made and who approved it; what they evaluated; how automation fits into their operating plan for the next 12 months; and what they told their team about Crate.

At-risk variant: Replace "what made the difference between staying and leaving?" with "what's working well today and what's getting in the way?" Do not ask about renewal, pricing, or whether they plan to stay.

4. Guidance for interviewers

  • Two people per call: one leads, one takes notes. No one from Sales or Customer Success on the call.
  • Don't name robotics, the 46% figure, or our hypotheses. Don't ask leading questions ("Isn't it frustrating that…?").
  • Ask about specific past events, not hypotheticals ("Would you use…?"). Ask "What else?" until they run out.
  • Let silence work. Don't fill it.
  • Don't promise anything. If asked about features or fixes: "I'll pass that on, but I can't say what we'll build."
  • Pricing or discounts: Don't engage. Say "I can't discuss that today. I'll make sure your account team knows you raised it." Record their words verbatim and tell Customer Success after the call. CS owns the follow-up.
  • Debrief within two hours. Fill in one template: go-live and churn dates, reasons in the customer's order, whether robotics came up unprompted or after a prompt, implementation duration, and up to three verbatim quotes.

5. What we'd need to hear

Thresholds are directional. Twelve calls can't prove anything, but they can show whether a theory holds up.

Back Bet A (robotics integrations) if: - At least 5 of 12 name a specific automation system that Crate can't connect to as a primary reason for leaving or evaluating alternatives. This should come up in block C or D, not only after prompting. - At least 3 say the integration would have changed the outcome. - The decision to leave happened late in the contract, not during implementation. - Churned accounts that ran automation churned for reasons that match the exit survey.

Back Bet B (faster implementation) if: - At least 5 of 12 say the decision started during go-live or in the first 90 days. - They name specific bottlenecks: data migration, configuration, waiting on our team, or connecting to their existing systems. - At least 3 say faster go-live would have kept them. - In the pulled data, go-live time separates churned accounts from retained ones more sharply than automation does.

Back neither if: - Reasons are spread across price, service, reporting, or other issues, with no dominant theme. - Robotics comes up, but few customers have automation projects planned in the next 12 months. - Implementation was slow, but the decision to leave was made at renewal for unrelated reasons. - The real driver is weak executive sponsorship or a lack of internal champions, which neither bet addresses.

If neither bet is backed, the readout should say so directly and name the pattern that did emerge. Don't force a choice between A and B if the evidence points elsewhere. If the evidence supports both, the readout should say which has stronger support and what would change that conclusion.

Claims the judge couldn’t find in the brief

  • Customers will say what helps their negotiation.

Grades and run details

Decision model 92 · LLM judge 79
Decision model checks
  • passUses the supplied evidence correctly1%
  • passAddresses the actual decision91%
  • partialRespects explicit constraints23%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims64%
  • passProduces the required deliverable80%
  • passTests both theories fairly98%
  • passDifferent questions for user and signer66%
  • passProtects the calls and the accounts95%
  • passDesigned to falsify82%
  • passAsks what people did, not what they think79%
  • passDoesn't lead the witness76%
  • partialMarks what to cut if the call runs over37%
Run
Run
#1
API response time
45 s
Submitted
8 Oct 2026
Critical failure: Includes questions that promise roadmap items or discuss pricing or discounts with customers in renewal talks. Score capped at 40.

Needs more than a quick edit3 of 13 checks passed

A PM would need to fix the prohibited either/or question, add a pricing/roadmap handling script, remove unsupported claims, and clarify the revised call coverage before using it. The reasoning and evidence also need rechecking.

Output

Customer Research Guide: H2 Strategic Bets (Robotics vs. Implementation)

To: Executive Team From: Staff Product Manager, Crate Date: Today Subject: 3-Week Customer Research Plan for H2 Resource Allocation (12 Calls)

---

Executive Summary & Plan Adjustments

In four weeks, the executive team will decide where to allocate both engineering squads for the next half. Ana believes we are losing customers because we lack warehouse robotics integrations (Bet A). Marcus believes we are losing them because our implementation cycle is broken and slow (Bet B).

The exit survey heavily favors Ana’s view (46% cite missing features/integrations), but UX research 101 teaches us that exit surveys are structurally flawed. Listing "missing features" first creates primacy bias, and the survey failed to capture temporal dynamics—specifically, when the decision to leave was made. Meanwhile, our internal data tells a damning story: our median time to go-live is 94 days (vs. 45 promised), and customers crossing 90 days churn at $2.4\times$ the rate.

We have 3 weeks to conduct 12 interviews (45 minutes each) to uncover the truth.

Critical Plan Adjustments (Read Before We Start)

  1. Split the Interviewees by Persona, Not Account: The original plan called for interviewing both the Ops Manager and the VP/COO on the same accounts. In 45 minutes, splitting time between a daily user and a high-level signer dilutes the insight. Instead, we will interview Warehouse Ops Managers to understand day-to-day friction (Implementation, UI, operational bottlenecks) and VPs of Ops / COOs to understand strategic drivers (ROI, contract expectations, robotics, and when the churn decision was actually made).
  2. Revise Ana’s Participation: Ana wants to run 4 calls with former customers she knows personally. We must advise against this. Founder-led interviews with churned customers carry massive politeness bias; customers will pull punches about implementation failures or invent reasons to let Ana down gently. Ana should observe these calls anonymously or listen to the recordings, but an unbiased PM or User Researcher must conduct them.
  3. Guardrails for At-Risk Accounts: Five of our six at-risk accounts are in active renewal talks with CS discounts on the table. Interviewers are strictly prohibited from discussing pricing, commercial concessions, or our product roadmap. We are strictly here to diagnose reality.

---

Learning Goals

  1. The Churn Timeline: Determine the exact month the customer mentally decided to leave. Did they decide during a painful 90+ day onboarding (supporting Bet B), or later when evaluating modernizing warehouse automation (supporting Bet A)?
  2. The "Missing Features" Audit: Deconstruct the 46% exit survey metric. When respondents checked "missing features/integrations," were they missing critical WMS functionality (e.g., wave planning, inventory accuracy) or specifically robotics middleware?
  3. Implementation Reality Check: Measure the operational impact of our 94-day median go-live time. Did the delay create cash-flow/trust erosion that doomed the account before day one, regardless of subsequent features?
  4. Robotics Urgency: For customers who mention robotics, are they actively deploying autonomous carts today, or is it a 3-year roadmap item they use as a stick to beat us with during negotiations?

---

Interview Guide: Two Personas

Persona 1: The Daily User (Warehouse Operations Manager)

Focus: Implementation friction, daily utility, operational roadblocks.

  • 00:00–03:00 | Intro & Permission:
  • "Thanks for taking the time. I’m [Name] from Crate. We’re doing research to figure out where we failed our customers and how we can improve. I am strictly here to learn about your experience—we won't be talking about pricing, renewals, or selling you anything today. Do I have your permission to record this call for internal notes?"
  • 03:00–12:00 | Day-to-Day Workflow & Reality:
  • "Take me back to your first 30 days using Crate. What did the day-to-day onboarding look like?"
  • "How long did it actually take for your floor team to feel comfortable using Crate without holding their breath?"
  • "What was the most painful workaround your team had to invent during your first three months?"
  • 12:00–25:00 | The Implementation Experience (Bet B Probe):
  • "Our data shows it takes about 90+ days for most warehouses to go live on Crate. What was your experience? How did those delays affect your team's stress levels and your relationship with our implementation team?"
  • "Looking back, did the slow start permanently damage your trust in the software, or did you bounce back once things were running?"
  • 25:00–35:00 | Equipment & Floor Integration (Bet A Probe):
  • "What automated hardware or mobile carts are running on your floor today? How do they talk to Crate, or do they?"
  • "When a pick-assist cart or AGV moves through your warehouse, where does the process break down between the robot and Crate?"
  • 35:00–42:00 | The Breaking Point:
  • "Think about the moment you realized you wanted to leave Crate (or started looking at alternatives). What happened on the warehouse floor that week?"
  • 42:00–45:00 | Wrap-Up:
  • "If you could wave a magic wand and change one thing about your onboarding or your day-one experience with Crate, what would it be?"

---

Persona 2: The Decision Maker (VP of Operations / COO)

Focus: Strategic expectations, ROI timelines, contract promises, and robotics strategy.

  • 00:00–03:00 | Intro & Permission:
  • "Thanks for joining. I’m [Name] from Crate. We’re reviewing why customers leave us to make radical changes to our roadmap and processes. I'm not here to pitch you or talk commercial terms—just to understand your true experience. May I record this call for note-taking purposes?"
  • 03:00–10:00 | The Purchase & Promise:
  • "When you signed with Crate, what was the primary business problem you needed solved in your first 6 months?"
  • "What timeline were you given for go-live during the sales cycle, and how did reality compare?"
  • 10:00–22:00 | Implementation & Time-to-Value (Bet B Probe):
  • "For a mid-size 3PL, cash flow and client onboarding speed are everything. How did Crate's onboarding timeline impact your ability to onboard your own end-clients?"
  • "At what point in your first year did you officially consider the implementation 'failed' or deeply troubled?"
  • 22:00–34:00 | The Robotics Mandate (Bet A Probe):
  • "Let’s talk about warehouse automation. How central are autonomous carts or robotics to your strategy today versus three years from now? Are you actively running them?"
  • "When you evaluated staying vs. leaving Crate, how heavy was the absence of robotics integrations in that final calculus? Was it the straw that broke the camel's back, or a nice-to-have?"
  • 34:00–42:00 | Uncovering the Timeline:
  • "Looking back at the lifecycle of your account, when did you mentally check out and decide you wouldn't renew? Was it during implementation, or months later?"
  • 42:00–45:00 | Wrap-Up:
  • "If Crate could fix only one thing—either our onboarding speed or our hardware/robotics connectivity—which one would have saved your account?"

---

Decision Criteria: What We Need to Hear

To make our final recommendation to Ana and Marcus, we will evaluate the 12 calls against the following thresholds:

Backing Bet A (Robotics Integrations)

We back Bet A only if: 1. VPs/COOs explicitly state that the lack of robotics was the primary, active catalyst for churning (not just a grievance listed in a post-hoc survey). 2. Churned customers are actively utilizing or piloting autonomous hardware today that could not function with Crate, resulting in operational deadlocks. 3. The exit survey's 46% "missing features" bucket is proven via interviews to be overwhelmingly about hardware/robotics connectivity rather than core WMS gaps (e.g., billing, inventory tracking).

Backing Bet B (Implementation Rebuild)

We back Bet B only if: 1. Interviews reveal that the 90+ day onboarding process created an insurmountable trust deficit; customers felt abandoned and never recovered faith in the product, rendering subsequent features irrelevant. 2. Daily users report that the complexity of setup and configuration was the primary driver of operational frustration, overshadowing missing integrations. 3. Churned customers admit they mentally checked out during or immediately after the botched implementation window, long before robotics ever entered the conversation.

Backing Neither (The Pivot Option)

We back Neither if: * The qualitative data shows a split narrative where both bets are symptoms of a deeper root cause—such as core WMS unreliability, pricing/value misalignment (17% in exit survey), or poor account management—meaning that building robots or speeding up a broken implementation will simply churn customers faster down a different path.

Claims the judge couldn’t find in the brief

  • For a mid-size 3PL, cash flow and client onboarding speed are everything.

Grades and run details

Decision model 58 · LLM judge 21
Decision model checks
  • partialUses the supplied evidence correctly23%
  • passAddresses the actual decision93%
  • partialRespects explicit constraints21%
  • passIdentifies material uncertainty97%
  • partialAvoids unsupported claims46%
  • partialProduces the required deliverable26%
  • partialTests both theories fairly47%
  • passDifferent questions for user and signer98%
  • partialProtects the calls and the accounts84%
  • partialDesigned to falsify40%
  • partialAsks what people did, not what they think52%
  • failDoesn't lead the witness30%
  • partialMarks what to cut if the call runs over45%
Run
Run
#1
Time to output
11 s
Submitted
1 Oct 2026

Needs more than a quick edit11 of 13 checks passed

The guide lacks separate question tracks for the daily user and the decision-maker, and it doesn't prioritize questions for time management. A PM would need to add role-specific questions and mark must-asks, which is more than a quick edit.

Output

Crate customer research guide: why are warehouses leaving?

Purpose: In three weeks, learn whether churn is more plausibly driven by missing robotics integrations (Bet A), slow implementation (Bet B), or something else. These calls are for understanding customers’ experiences—not selling, negotiating renewals, or validating either executive’s theory.

Learning goals

For each account, establish:

  • What happened, and when? Separate implementation, go-live, first value, automation needs, and the point when leaving became likely or decided.
  • What mattered most? Distinguish a decisive problem from a feature request, contributing frustration, or reason offered after the fact.
  • What evidence supports the explanation? Look for actual workflow impact, workarounds, alternatives evaluated, and actions taken—not just opinions or hypotheticals.
  • What might change the outcome? Explore credible counterfactuals without promising anything.
  • What else explains churn? Surface causes beyond A and B.

Recommended plan changes

The proposed 12 calls cannot cover both the daily user and contract signer on all 12 accounts: that would require 24 interviews. Keep the 12-call cap, but interview two roles separately at each of six accounts: three churned accounts and three red-health accounts, with one warehouse operations manager and one VP of Operations/COO per account. This gives paired perspectives and preserves both customer situations, but is a small, directional sample—not a prevalence estimate. Select accounts for varied implementation times and automation situations where possible; don’t choose only accounts with known robotics issues.

Do not have Ana conduct four calls with former customers she knows personally. Her involvement risks courtesy bias and leading the conversation. Use an independent interviewer; ideally don’t include personal contacts in this small core sample. If one is included, disclose the relationship, and have Ana neither attend nor receive attributable notes.

Because five at-risk accounts are in renewal talks and receiving discount offers, use an interviewer outside the account/renewal team. Tell participants their answers won’t affect service or renewal discussions. Don’t share interview content with the account team in a way that could be used in negotiation. Follow Legal’s recording requirement below.

45-minute guide: churned customers

TimeQuestions
0–3Start by asking: “May I record this conversation?” If no, take notes instead. Explain the purpose, that there are no sales or renewal implications, and that we won’t discuss pricing, discounts, or roadmap.
3–7“What was your role in choosing and using Crate?” “What did your warehouse need the system to help you do?”
7–18“Take me through your experience with Crate, from signing through the decision to leave.” “When did you first consider switching? What happened just before that?” “Who was involved, and when was the decision effectively made?”
18–27“How did implementation and go-live go?” “Where, if anywhere, did it take longer or require more effort than expected?” “What was the impact on your operation?” “When did you first get value from Crate?”
27–35“Did automation or robotics play a role in your warehouse during this period?” If yes: “What equipment or workflow?” “How did it work with Crate?” “What did you do when it didn’t?” “When did this become a problem relative to the decision to leave?”
35–41“What alternatives did you consider, and what drove the choice?” “What, if anything, could realistically have changed your decision?” “What was the most important reason—not just one frustration?”
41–45“What haven’t I asked that matters?” “Is there anything we should correct about our understanding?” Thank them.

45-minute guide: red-health customers

Use the same opening and close. Do not imply that Crate can fix an issue or influence renewal terms.

TimeQuestions
0–3Start by asking: “May I record this conversation?” If no, take notes. Explain that this is research, not a sales or renewal conversation; participation won’t affect service or renewal discussions.
3–7“What is your role in using or overseeing Crate?” “What does a successful warehouse operation look like for you?”
7–17“How has your experience with Crate unfolded so far?” “When have things gone better or worse than expected?” “What is working well today?”
17–27“Where are you in implementation and go-live?” “What has taken the most effort or time?” “What value, if any, have you realized so far?” “What is the operational impact of anything that remains difficult?”
27–35“How, if at all, does automation or robotics fit into your operation?” If relevant: “What equipment or workflow?” “How does it work with Crate today?” “What workarounds do you use?” “When did this become important?”
35–41“Have you considered alternatives to Crate? What prompted that, if so?” “What would need to be true for Crate to work well for your operation?” “What are the most important unresolved issues?”
41–45“What haven’t I asked that matters?” “Is there anything we should correct?” Thank them.

Guidance for interviewers

  • Ask the questions as written, but follow the participant’s timeline. Use neutral probes: “What happened next?” “Can you give me an example?” “How do you know?”
  • Don’t offer Ana’s or Marcus’s theories, mention the exit-survey results, or ask leading questions such as “Would robots have saved you?” Ask about events before asking for interpretations.
  • Separate when an issue occurred from when the participant says it mattered. Probe for specific actions and evidence: evaluations, workarounds, escalations, or a switch decision.
  • Don’t discuss pricing, discounts, renewal negotiations, or roadmap. If asked about roadmap, say you can’t speak to future plans. Don’t make promises or relay an interview answer as a renewal signal.
  • Interview roles separately. Record role and account context in notes, but distinguish direct experience from second-hand views. Afterward, write a short timeline, leading cause(s), evidence, and uncertainties for each account.
  • Treat recordings and notes as confidential; use only with consent and Legal-approved handling. Don’t present anecdotes as representative of all customers.

What would support each bet?

Support Bet A (robotics integrations): In multiple independent accounts, a specific robotics workflow or integration gap caused meaningful operational friction, and that issue arose before the decision to leave or seriously consider leaving. Stronger evidence: an actual robot deployment or concrete plan, a costly workaround, and a competitor selected or evaluated for this capability. A feature mention alone—or robotics cited in a new-logo deal—doesn’t establish that it drove existing-customer churn.

Support Bet B (faster implementation): In multiple accounts, implementation delays or effort materially postponed go-live or first value, caused operational consequences, and preceded the decision to leave or consider leaving. Stronger evidence: specific preventable blockers, an unmet expectation, and a credible account that earlier go-live would have changed the outcome. The 94-day median, 45-day promise, and 2.4× association make this worth testing, but do not prove causation.

Support neither / remain uncertain: The call evidence points more strongly to another cause; A or B appears only after the decision; accounts describe issues without meaningful consequences; or evidence is mixed and no cause consistently precedes churn. Don’t force a winner from 12 interviews. Compare patterns across accounts, not individual interview counts, and triangulate with CRM timelines and other churn evidence. The exit survey’s single-choice format, first-listed feature option, and sparse free text limit what it can establish.

Grades and run details

Decision model 92 · LLM judge 79
Decision model checks
  • passUses the supplied evidence correctly68%
  • passAddresses the actual decision92%
  • passRespects explicit constraints27%
  • passIdentifies material uncertainty94%
  • passAvoids unsupported claims89%
  • passProduces the required deliverable75%
  • passTests both theories fairly99%
  • partialDifferent questions for user and signer71%
  • passProtects the calls and the accounts76%
  • passDesigned to falsify77%
  • passAsks what people did, not what they think68%
  • passDoesn't lead the witness65%
  • partialMarks what to cut if the call runs over50%
Run
Run
#1
API response time
29 s
Submitted
30 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 10

Uses the supplied evidence correctlyMixedWrongRight
Haiku 5.5 · API

The claim 'Customers will say what helps their negotiation' is an unsupported behavioural statement not derived from the context.

Gemini 3.5 Flash-Lite · Gemini

It uses most supplied numbers correctly, but invents or overstates current-situation claims such as “cash flow and client onboarding speed are everything” and “90+ days for most warehouses” when the context only gives a 94-day median.

GPT-6 Luna · API

Every statement about the current situation is taken directly from the supplied context, with no invented facts.

Respects explicit constraintsRightWrongRight
Haiku 5.5 · API

The output is under 1,500 words, includes all requested elements, and enforces pricing/roadmap silence with a script.

Gemini 3.5 Flash-Lite · Gemini

It is under 1,500 words and includes the required sections, but it does not enforce the no-pricing/no-roadmap rule in practice because it lacks a script for when customers raise those topics and even mentions “roadmap” in the executive intro.

GPT-6 Luna · API

The guide respects all constraints: it includes learning goals, timed questions for both customer types, interviewer guidance, plan changes, and decision signals; it stays under 1,500 words; it enforces no pricing/roadmap discussion and recording consent.

Avoids unsupported claimsMixedWrongRight
Haiku 5.5 · API

The statement 'Customers will say what helps their negotiation' is presented as fact without caveat or support.

Gemini 3.5 Flash-Lite · Gemini

It presents several interpretations as established facts, including that the exit survey “heavily favors” Ana’s view, that founder-led interviews carry “massive” politeness bias, and that cash flow/onboarding speed are “everything” for mid-size 3PLs.

GPT-6 Luna · API

Interpretations like the exit survey's limitations and the 2.4× association are clearly labeled as not proving causation, not presented as established fact.

Produces the required deliverableRightWrongRight
Haiku 5.5 · API

It delivers a complete interview guide in the required form for the exec team, within the word limit.

Gemini 3.5 Flash-Lite · Gemini

The guide is mostly present, but it is not fully usable because it changes the plan to split personas without specifying how the 12 calls and 6 accounts will be covered, and it omits a required handling script for pricing/roadmap questions.

GPT-6 Luna · API

The output is a complete call guide with all requested sections, written for the exec team, and well under 1,500 words.

Tests both theories fairlyRightWrongRight
Haiku 5.5 · API

Open questions on reasons come first; automation and implementation are probed separately with neutral questions, giving both theories equal chance to fail.

Gemini 3.5 Flash-Lite · Gemini

It gives both theories probes, but the daily-user track is weighted toward implementation, the signer track asks about robotics only after implementation, and the wrap-up asks which bet would have saved the account, so it does not test both theories fairly.

GPT-6 Luna · API

Both the robotics and implementation theories get questions that could disprove them, open timeline questions come first, and the guide explicitly leaves room for a third cause.

Different questions for user and signerRightRightWrong
Haiku 5.5 · API

Distinct tracks exist for the operations manager (main 45-min guide) and the executive (20-min variant focusing on decision process).

Gemini 3.5 Flash-Lite · Gemini

It has distinct tracks: the daily user is asked about onboarding, workarounds, floor equipment, and the breaking point, while the signer is asked about purchase promises, time-to-value, robotics strategy, and renewal decision timing.

GPT-6 Luna · API

The guide does not provide distinct question tracks for the daily warehouse operations manager and the executive who signed; it only differentiates churned vs at-risk customers, with the same questions for both roles within each track.

Protects the calls and the accountsRightWrongRight
Haiku 5.5 · API

Consent is requested at the start, a script deals with pricing/roadmap inquiries, and the CEO is instructed not to run calls with accounts she knows.

Gemini 3.5 Flash-Lite · Gemini

It includes consent to record and a no-pricing/no-roadmap rule, but it provides no script for handling pricing, discount, or roadmap questions and the executive intro mentions roadmap changes.

GPT-6 Luna · API

It gives a clear rule and script for pricing/roadmap questions, starts with recording consent, and addresses Ana's bias by recommending she not lead calls and, if a contact is included, that she not attend or receive attributable notes.

Designed to falsifyRightWrongRight
Haiku 5.5 · API

Every learning goal includes open, non-leading questions that can yield answers disproving the team's hypotheses (e.g., asking if automation was a blocker or nice-to-have).

Gemini 3.5 Flash-Lite · Gemini

Not every learning goal has a neutral disconfirming question; for example, the implementation goal asks whether delays “permanently damaged trust,” and the robotics goal asks whether robotics was “the straw that broke the camel's back,” rather than first eliciting the decision story.

GPT-6 Luna · API

Every learning goal has questions whose honest answers could disprove the hypothesis, such as asking when automation became a problem relative to the decision to leave.

Asks what people did, not what they thinkRightWrongRight
Haiku 5.5 · API

Core questions ask about the last time they considered leaving, what happened, and what they did, focusing on specific past events.

Gemini 3.5 Flash-Lite · Gemini

Some questions ask about recent actions, but the core includes many opinions, hypotheticals, and leading prompts such as “magic wand,” “how heavy was the absence of robotics,” and “which one would have saved your account.”

GPT-6 Luna · API

Core questions ask for specific past events and actions (e.g., 'When did you first consider switching? What happened just before that?'), with opinions and predictions clearly secondary.

Doesn't lead the witnessRightWrongRight
Haiku 5.5 · API

Questions are neutral (e.g., 'What has changed on your floor?', 'Was that a blocker, or a nice-to-have?') and avoid pitching or hinting at desired answers.

Gemini 3.5 Flash-Lite · Gemini

Several questions are leading or biased, including “Our data shows it takes about 90+ days,” “did the slow start permanently damage your trust,” “cash flow and client onboarding speed are everything,” and the final either/or bet question.

GPT-6 Luna · API

Questions are neutral and open; the guide avoids mentioning either executive's theory until after the participant's own timeline, and prohibits leading probes.

All got wrong 1

Marks what to cut if the call runs overWrongWrongWrong
Haiku 5.5 · API

The timed sections sum to 48 minutes, exceeding the 45-minute limit, and the guide does not mark what to cut if time runs short.

Gemini 3.5 Flash-Lite · Gemini

Timings add to 45 minutes, but the number of questions is too high for 45 minutes, must-ask questions are not marked, and there is no guidance on what to cut if time runs short.

GPT-6 Luna · API

Although timed sections add up to 45 minutes, the guide does not mark must-ask questions or say what to cut if time runs short, so an interviewer could run out of time without covering the most critical probes.

All got right 2

Addresses the actual decisionRightRightRight
Haiku 5.5 · API

Section 5 clearly lays out what patterns would back Bet A, Bet B, or neither, satisfying the requirement for decision signals.

Gemini 3.5 Flash-Lite · Gemini

It does not choose a bet now, but the supplied evidence cannot support a final call; it clearly says the research will settle the decision and gives explicit thresholds for Bet A, Bet B, or neither.

GPT-6 Luna · API

The guide clearly states what evidence from the calls would back Bet A, Bet B, or neither, which is the decision framework the exec team needs.

Identifies material uncertaintyRightRightRight
Haiku 5.5 · API

It explicitly treats the exit survey as weak evidence, notes the small sample cannot prove anything, and says what to do if neither bet is backed.

Gemini 3.5 Flash-Lite · Gemini

It names the key unknowns—when customers decided to leave, whether “missing features” means robotics or core WMS, and whether slow implementation caused trust loss—and says the calls and decision thresholds would resolve them.

GPT-6 Luna · API

It names key unknowns (exit survey weakness, correlation vs causation, timing of churn decision) and says how the calls will resolve them through patterns and timelines.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 98% of 42 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5withAPI86.188.52None
2GPT-6.1 SolwithAPI87.884.92None
3Opus 5.5withClaude92.174.72None
4GPT-6 AstrawithChatGPT89.774.52None
5Haiku 5.5withAPI87.870.12None
6GPT-6 LunawithAPI81.673.92None
7Gemini 3.8 FlashwithAPI69.751.92None
8Gemini 3.5 Flash-LitewithGemini60.126.121 capped

What you get for the cost

GPT-6.1 Sol · API scores within a point of Sonnet 5.5 · API at a third of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreSonnet 5.5 · API: task score 86, 43× the cheapestGPT-6.1 Sol · API: task score 85, 13× the cheapestOpus 5.5 · Claude: task score 82, 71× the cheapestGPT-6 Astra · ChatGPT: task score 81, 66× the cheapestHaiku 5.5 · API: task score 77, 2.3× the cheapestGPT-6 Luna · API: task score 77, the cheapestGemini 3.8 Flash · API: task score 58, 8.3× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 41, 3.0× the cheapest, 1 critical failureSonnet 5.5 · APIGPT-6.1 Sol · APIOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTHaiku 5.5 · APIGPT-6 Luna · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costAt least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API771.0×
Haiku 5.5 · API772.3×
Gemini 3.5 Flash-Lite · Gemini413.0×
Gemini 3.8 Flash · API588.3×
GPT-6.1 Sol · API8513×
Sonnet 5.5 · API8643×
GPT-6 Astra · ChatGPT8166×
Opus 5.5 · Claude8271×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Preparing for customer calls.

Why it matters

Leading questions produce the answers you wanted. A good guide is designed to prove you wrong.

What good looks like

  • Past-behaviour questions
  • No leading prompts
  • Clear learning goals
  • Could disprove the team's hypothesis

Deliberately not measured

    Capability tested

    Question design

    The failure we’re looking for

    Hypothetical, leading questions

    Grading

    Decision model and LLM judge, calibrated against a blind PM review