Tasks / Discover

Customer research call guide

Can the model write a guide that uncovers behaviour rather than opinions?

Measures the modelTask v1.0 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 9 graded outputs by 5 models. 50% were usable with at most a quick edit.

Reliably right

  1. Identifies material uncertainty100% pass
    Names specific unknowns (cost of lateness, ease of chasing, automation fit, willingness to pay) and says mixed findings mean targeted follow-up.
    GPT-6.1 Sol · API · Would freelancers pay to stop chasing invoices?
  2. Designed to falsify100% pass
    Every learning goal has disconfirming questions, e.g., 'What did you do next?', 'When do you avoid sending another reminder?'
    GPT-6.1 Sol · API · Would freelancers pay to stop chasing invoices?
  3. Asks what people did, not what they think100% pass
    Core questions ask for the most recent overdue invoice and specific actions taken, not opinions or predictions.
    GPT-6.1 Sol · API · Would freelancers pay to stop chasing invoices?

Where it slips

  1. Fits the call33% pass
    Timings add up to 30 minutes but no must-ask questions are marked and no guidance on what to cut if time runs short.
    GPT-6.1 Sol · API · Would freelancers pay to stop chasing invoices?
  2. Respects explicit constraints81% pass
    The guide is clearly over 900 words, violating the explicit length constraint.
    Opus 5.5 · Claude · Would freelancers pay to stop chasing invoices?
  3. Different questions for user and signer81% pass
    The guide does not provide distinct question tracks for the daily warehouse operations manager and the executive who signed; it only differentiates churned vs at-risk customers, with the same questions for both roles within each track.
    GPT-6 Luna · API · Why are warehouses leaving? Two execs, two theories

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a Staff PM at Crate. In four weeks, the exec team decides which of two bets gets our two squads next half, and both bets rest on a theory of why customers are leaving. We have three weeks for 12 customer calls of 45 minutes. Write the guide for them. It should include: the learning goals; the questions, with timings, for the two kinds of people we'll talk to; guidance for whoever runs the calls; any changes you'd make to the plan below; and what we'd need to hear to back each bet, or neither. Keep it under 1,500 words. The exec team will read it before the calls start.

About CrateWarehouse management software for mid-size third-party logistics companies (3PLs). 340 customers, $22M ARR. Gross logo churn rose from 11% to 18% over the last year.
The decisionBet A: integrations with warehouse robotics (autonomous carts, pick-assist robots). Bet B: rebuild implementation so customers go live faster. Two squads for the next half go to one of them.
CEO, Ana Ferreira“We're losing warehouses because they're automating and we don't talk to their robots. Every lost customer I've spoken to mentions it.”
CPO, Marcus Hill“We lose them in the first year because implementation takes forever. They're gone before they ever see the value.”
Exit survey (41 churned customers, last 12 months)A single-choice question, 'Why are you leaving?'. Missing features or integrations: 46%. Too hard to implement: 22%. Price: 17%. Other: 15% (free-text box, left blank by all but two). 'Missing features or integrations' was the first option in the list.
Implementation and CRM dataMedian time from signing to go-live: 94 days; the contract promises 45. Customers who went live after 90 days churned at 2.4× the rate of the rest in their first year. 3 churned customers moved to a competitor with robotics integrations; robotics was cited in 9 lost new-logo deals. We don't know when in the year churned customers decided to leave.
The call plan12 calls: 6 with customers who churned in the last 9 months, 6 with at-risk customers (health score red). On each account we want to talk to the warehouse operations manager who used Crate every day and, separately, the VP of Operations or COO who signed the contract. Ana wants to run 4 of the calls with former customers she knows personally.
Notes from Customer Success and Legal5 of the 6 at-risk accounts are in renewal talks, and Customer Success is offering them discounts. Interviewers must not discuss pricing, discounts or what's on our roadmap. Every call must start by asking permission to record.
What a strong answer does

A guide that tests both theories fairly and leaves room for a third: the goals ask what actually drove each decision to leave or doubt, when it happened, and who made it, with questions that could disprove robotics (did they automate, did robots come up before they decided, what did they switch to and why) and implementation (how long go-live took, what happened in those months, whether value arrived) alike. It treats the exit survey as weak evidence (single choice, integrations listed first, 'Other' left blank) and the 2.4× figure as a correlation to probe, not a cause. The core is a timeline of the account: the story from signing to the moment they decided to leave or started doubting, with sequential probes. It has separate tracks for the daily user (what went wrong day to day) and the signer (how the decision was made, what alternatives they weighed), fits 45 minutes with timings, and opens with consent to record. It guards the calls: no pitching, pricing, discounts or roadmap promises, with a script for when an at-risk customer asks, and it addresses Ana's bias: she has a public view and personal relationships, so she should observe or take notes rather than lead, or at minimum use the same script and have her calls read separately. It ends with decision signals: what pattern across the 12 would back Bet A, what would back Bet B, and what would back neither (price, a business change, a missing capability that isn't robotics).

Critical failures (cap the score)
  • Includes questions that promise roadmap items or discuss pricing or discounts with customers in renewal talks
  • Asks customers to choose between robotics and implementation as the reason they left
Case

v1.0 · synthetic · B2B SaaS, logistics, churn, exec bias, staff-level

11 of 13 checks passed

Output

Churn Discovery Calls: Discussion Guide

For: exec team (read before calls start), interviewers, note-takers Timeline: week 0 for data pull and recruiting, weeks 1–3 for calls, week 4 for synthesis and recommendation

Why we're doing this

Both bets rest on a theory we haven't tested. The data we have can't settle it:

  • The exit survey doesn't separate the theories. "Missing features or integrations" (46%) was listed first, it was a single choice, and it combines robotics with every other feature gap. The free-text box was almost always left blank.
  • The robotics evidence is thin for churn. 3 churned customers moved to a competitor with robotics integrations. That is 3 of roughly 60 churned accounts. The 9 lost new-logo deals are about acquisition, not churn.
  • The implementation evidence is a correlation. Customers who went live after 90 days churned at 2.4× the rate. But complex customers may both take longer to go live and churn more for other reasons.
  • We don't know when or why the leave decision formed. That is the question the calls exist to answer.

Twelve calls can show us how customers decide to leave. They can't show us how many leave for each reason. The CRM pull below fills that gap, and the calls and the data should be read together.

1. Learning goals

  1. When did the leave decision form, and what triggered it? In the first months or later? During implementation, after go-live, or around an automation project?
  2. Does implementation delay cause churn, or just come with it? If delay mattered, was the cause on our side (fixable by Bet B) or on theirs (data readiness, their staffing)?
  3. Is automation a real driver? Are customers actually automating? Did Crate's lack of robot integration block them, or did it come up afterwards as a reason?
  4. Do signers and daily users tell the same story? Executives may cite strategy (automation). Operations managers may cite experience (implementation, usability).
  5. Are the reasons something else entirely? For example: price, support, acquisition, consolidation, or a non-robotics feature gap.

2. Changes to the plan

1. Pull CRM data in week 0, before the first call. For every account that churned in the last 12 months, pull: - tenure at churn - days from signing to go-live - whether they run or were deploying automation (ask CS and account managers) - where they went

Also answer one question: did the rise from 11% to 18% churn come from first-year customers? If it did, that supports Bet B. If the increase is among long-tenured accounts, that points toward Bet A. This may be the most decisive single fact we get, and it costs a day.

2. Replace at-risk accounts that are in renewal talks. Five of the six are negotiating discounts. They have a reason to exaggerate complaints, and a call with them could easily drift into pricing. Use red-health accounts that are not in active renewal. If there aren't enough of those, add churned accounts instead.

3. Fix the call count. The plan says 12 calls, but two people on each of 12 accounts is 24 calls. Within the 12, I propose 8 accounts:

GroupAccountsWho we interviewCalls
Churned, paired4Signer and ops manager, separately8
Churned, signer only2Signer2
At-risk, not in renewal2Signer2
  • Split the churned accounts so that 3 left in their first year and 3 left later.
  • Include at least one account that moved to a competitor with robotics integrations.
  • Signers get priority because they made the decision.
  • The paired accounts let us compare what the signer says with what the daily user saw.
  • If we can fund 24 calls, pair every account.

4. Ana should not interview her own contacts. Anyone will soften or reshape their story when the CEO asks, especially a CEO they know and whose view they may already have heard. Her contacts also aren't a random sample of the people who left.

My suggestion: - Her contacts can enter the pool if they fit the sampling groups, but a neutral interviewer runs those calls. - Ana and Marcus both listen to the recordings. - If Ana wants to speak with former customers personally, those calls are valuable for the relationship. We run them in addition to the 12 and don't count them as research.

The same rule applies to Marcus: neither exec interviews.

5. Offer a thank-you gift to former customers, such as a gift card. It must not be a credit or discount on Crate.

3. Discussion guide: VP Ops / COO (signer), 45 minutes

0–3 min · Opening - Ask permission to record. If they say no, don't record and take notes only. - Say: "This isn't a sales call. I can't discuss pricing or our product plans. I'm here to understand your experience, good and bad."

3–8 min · Context - "Tell me about your business and how it's changed over the last two years." - "What were your biggest operational priorities this past year?"

8–22 min · The decision story (the core of the call) - Churned: "Take me back to the first moment you started to wonder whether Crate was right for you. What was going on?" - At-risk: "Tell me about the last time you seriously questioned renewing." - Build a timeline with follow-ups: - "When was that, relative to signing?" - "What happened next?" - "Who else was involved?" - "What alternatives did you look at?" - "When was the decision final?" - "What finally tipped it?"

22–30 min · Expectations and early months - "When you signed, what did you expect Crate to do for you, and by when?" - "How did going live compare to what you expected?" - "When, if ever, did you first see the value you signed up for?"

30–37 min · Operational direction - "What investments in the warehouse have you made or planned: equipment, systems, staffing?" - "What did you need your WMS to do as part of those?" - If they mention automation: "How did Crate fit, or not? What did you do about it?"

37–42 min · Prompted check - Hand over a card of possible reasons: implementation time, robotics/automation integration, other missing features, ease of use, support, price, business change. - Rotate the order on every call. The survey showed the first option gets picked more. - Ask: "Pick the top three for you." Then: "Which one, if it had been different, would have changed the outcome?"

42–45 min · Close - "What should I have asked?" - "Who else should we talk to?" - Thank them.

4. Discussion guide: Warehouse operations manager, 45 minutes

0–3 min · Opening - Same as the signer call: recording permission and ground rules.

3–10 min · Daily work - "Walk me through a typical day using Crate." - "What other tools or spreadsheets did you use alongside it?"

10–22 min · Implementation story - "Tell me what happened from signing to go-live." - "What took longest?" - "What were you waiting on, and who was it waiting on?" - "When did it start working the way you needed?" - Get specific episodes, not a verdict. Always ask whether a delay came from us, from them, or from a third party.

22–32 min · Gaps and workarounds - "Tell me about the last time Crate got in your way." - "What equipment or automation runs on your floor? How does information move between it and Crate?" - "What manual steps did that create?"

32–37 min · The decision - "Were you part of the decision to leave or renew? What would you have told leadership?"

37–42 min · Prompted check - Same card as the signer call, with a new random order.

42–45 min · Close - Same as the signer call.

5. Guidance for interviewers

Stay neutral - Don't say "robotics," "automation," or "implementation" before the prompted check, unless the interviewee says it first. What people raise unprompted is our strongest evidence. - Ask about specific past events ("the last time…", "walk me through…"), not opinions or hypotheticals. Treat "we'd have stayed if you'd had X" as weak evidence. - Use their words, not ours. Don't correct them or defend Crate. - In paired accounts, use different interviewers where possible. Never tell the second person what the first one said.

Stay within the rules from Legal and CS - Pricing or discounts: "That's outside what I can cover today. I'll let your account team know you'd like to discuss it." Then tell CS. - Roadmap: "I can't speak to product plans, but I'll make sure what you've said reaches the people who make them." Never promise a feature. - If there's no recording consent, don't record. The note-taker writes verbatim quotes where possible.

Process - A note-taker joins every call. - Within 24 hours, fill in the debrief template: - primary reason, in their words - the decision timeline, with the month the doubt started - robotics raised unprompted? (Y/N) - implementation raised unprompted? (Y/N) - was the delay caused by us, them, or both? - top three from the card - the best quote - Hold a 30-minute team review of the debriefs every Friday.

6. What would back each bet

Each account gets classified by its primary cause, meaning the thing that started or tipped the decision. When signer and ops manager disagree, record both views; don't average them.

Back Bet A (robotics) if: - In at least 3 of the 6 churned accounts, automation was raised unprompted and came before the leave decision. - The customer had a live or funded automation project that Crate blocked or made harder. - Ops managers describe concrete workarounds between robots and Crate. - The CRM pull shows churn concentrated in longer-tenured accounts, with meaningfully more automating churners than the 3 we already know about. - Warning sign: robotics only comes up at the card, or appears in the timeline after the decision was already made. That suggests it's a justification, not a cause.

Back Bet B (implementation) if: - In at least 3 of the 6 churned accounts, doubt formed during or soon after a slow implementation, often phrased as "never saw the value" or "lost confidence." - Ops managers put the main delays on our side: our process, staffing, data migration, or configuration. Those are what a rebuild would fix. - The CRM pull shows the rise in churn comes mostly from first-year customers. - Warning sign: delays were mostly on the customer's side (their data wasn't ready, they didn't have staff). A faster process on our end would then help less than the 2.4× figure suggests.

Back neither if: - Primary causes are spread across several reasons with no clear lead, such as price, support, consolidation or acquisition, or a feature gap unrelated to robotics. - Or signers and ops managers consistently tell different stories that neither bet addresses.

In that case, I'd recommend a smaller fix aimed at whichever cause leads, plus a better-designed exit survey, rather than committing two squads on a weak signal.

If both show up: check whether automating customers are also the ones with the slowest implementations. If they are, those customers' problem is really integration work during setup, and the recommendation should address that directly instead of forcing a choice between A and B.

In week 4, I'll bring the exec team a one-page summary that combines the call findings with the CRM figures, a recommendation, and our confidence in it.

Check by check

Mixed · 2
  • Respects explicit constraintsThe two graders disagreed on this one.
  • Fits the callThe two graders disagreed on this one.
Got right · 11
  • Uses the supplied evidence correctly
  • Addresses the actual decision
  • Identifies material uncertainty
  • Avoids unsupported claims
  • Produces the required deliverable
  • Tests both theories fairly
  • Different questions for user and signer
  • Protects the calls and the accounts
  • Designed to falsify
  • Asks what people did, not what they think
  • Doesn't lead the witness

Grades and run details

Decision model 92
Decision model checks
  • passUses the supplied evidence correctly8%
  • passAddresses the actual decision20%
  • partialRespects explicit constraints25%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims71%
  • passProduces the required deliverable81%
  • passTests both theories fairly99%
  • passDifferent questions for user and signer100%
  • passProtects the calls and the accounts99%
  • passDesigned to falsify76%
  • passAsks what people did, not what they think92%
  • passDoesn't lead the witness43%
  • partialFits the call32%
Run
Run
#1
Time to output
1.7 min
Submitted
30 Sept 2026

Usable with a quick edit11 of 13 checks passed

The guide is thorough and evidence-based; a PM could add a note on which questions to prioritize if time is tight and use it immediately.

Output

Crate churn research: executive pre-read and interview guide

Decision: Allocate two squads next half to robotics integrations (A), implementation rebuild (B), or neither if the evidence does not justify either.

Research stance: Test both explanations, not confirm either executive’s account. These interviews can reveal mechanisms and decision timelines; they cannot estimate their prevalence across 340 customers or prove causality.

1. Learning goals

  1. Reconstruct the departure process: When did concerns emerge, when did leaving become likely, and when was the decision effectively made?
  2. Identify consequential problems: Separate reasons volunteered after leaving from problems that materially changed warehouse outcomes and the buying decision.
  3. Test Bet A: Was a specific robotics incompatibility a binding obstacle? Would addressing it have plausibly retained the account?
  4. Test Bet B: Did implementation delay prevent value, damage trust, or trigger departure? What caused the delay, and could Crate control it?
  5. Find alternatives and interactions: Price, service, reliability, business changes, or other missing capabilities may dominate. Robotics work may itself complicate implementation.
  6. Connect learning to an investable scope: What could two squads deliver next half that would address the demonstrated retention mechanism?

What the existing evidence does—and does not—say

  • The exit survey’s broad integration category, single-choice format, first-position placement, and sparse free text do not establish robotics as the cause.
  • The 2.4× churn association makes implementation worth investigating, but complexity, customer readiness, or other factors could drive both delay and churn.
  • Three competitor moves and nine lost deals indicate a robotics opportunity; acquisition losses are not evidence of a retention mechanism.
  • Unknown decision dates make chronology essential.

2. Changes to the call plan

Resolve the arithmetic: Twelve separate interviews cannot cover twelve accounts with two roles each. Use six accounts × two separate role interviews = twelve 45-minute calls.

Recommended mix: Four churned accounts and two at-risk accounts. This reduces reliance on renewal negotiations while retaining a prospective view. Findings will be depth-oriented, not representative.

Select accounts from the full CRM population, not executive contacts or survey answers alone:

  • Across churned accounts, include slow and faster implementations, first-year and later departures, and robotics-exposed and non-exposed customers.
  • Include at least one confirmed move to a robotics-enabled competitor and one slow implementation without an apparent robotics need.
  • For at-risk accounts, prefer one with active automation plans and one with implementation/value-realization problems. Include the non-renewal account if suitable.
  • Seek contrasting retained accounts through existing operational data, even though the call budget does not cover them.

Recruit both the daily warehouse operations manager and the contract-signing VP Operations/COO. If the original signer has left, recruit someone directly involved in the departure or renewal decision and document that substitution.

Ana should not lead calls with her personal contacts. Use a neutral interviewer without renewal responsibility. Ana can help recruit; observing requires explicit customer consent. Her existing conversations are useful leads, not independently verified findings.

CS should introduce research as separate from renewal discussions, then step out. Participation must not affect commercial treatment.

3. Interview guides: 45 minutes each

Use the relevant role column. In every section, distinguish direct experience from hearsay. Ask broad questions before naming either bet.

TimeDaily warehouse operations managerVP Operations / COO
0–3 min: permission and framingFirst ask: “May we record this conversation for internal research?” If declined, continue with notes. Then explain purpose and confidentiality limits.Same opening.
3–7 min: context and expected value“What did your warehouse handle, and what was your role with Crate? What was supposed to improve?”“What led you to buy Crate? What outcomes and deadlines mattered? How would you judge success?”
7–17 min: chronological reconstruction“Walk me from signing through onboarding, first operational use, and the point when problems became serious.” “Describe a particular shift or incident.” Establish dates, workarounds, operational impact, and who knew.“Walk me from purchase to the first concern, evaluating alternatives, and the decision.” “When did staying stop being the default?” Establish dates, decision-makers, triggers, and alternatives.
17–27 min: implementation and value“What had to happen before you could use Crate successfully? Where did work stall? Who owned each step?” “When did you first achieve useful results?” Probe migration, configuration, training, integrations, staffing, and rework after an open answer.“What go-live date did you expect, and what happened? What consequences did that have?” “What other factors accompanied the delay?” “If the same product had gone live in 45 days, what would still have put the relationship at risk?” Ask why.
27–36 min: workflow, automation, and unmet needs“Which workflows did Crate support poorly? Show or describe a recent example.” Then: “Were robots in use or planned? Which systems, what workflow, and what connection was required?” “What happened without it?”“What capabilities influenced staying or switching?” Then probe automation: vendor, deployment date, committed budget, required integration, and alternatives considered. “If that connection had existed, what else would have needed to change for you to stay?”
36–42 min: decision and competing explanations“Which problem mattered most in daily operations? What did you escalate, to whom, and when?” “What worked well?” “What have we missed?”“Which issues were necessary to the decision, and which were secondary?” “What would have had to be different for you to stay?” Ask about business changes, service, reliability, and other alternatives without forcing a category.
42–45 min: verify and closeSummarize the timeline and mechanism: “What have I misunderstood?” Request relevant artifacts and permission for a brief clarification follow-up.Same; verify whether operational problems actually influenced the commercial decision.

Status-specific wording

  • Churned: Ask what the replacement actually delivered, whether it went live, and whether the cited problem improved—not merely what the competitor promised.
  • At-risk: Ask what has already happened, current unresolved consequences, and what would cause escalation or departure. Do not imply a departure decision exists. Distinguish funded plans from aspirations.

4. Guidance for interviewers

  • Begin with recording permission; record only after consent. Explain that this is research, not a support, roadmap, or renewal discussion.
  • Do not discuss prices, discounts, or roadmap commitments. If raised, acknowledge without probing commercial terms: “I can’t discuss that here; your account team handles commercial conversations.” Record volunteered price concerns as an alternative explanation.
  • Avoid “Did missing robotics make you leave?” and “Would faster onboarding have saved you?” Start with events; use counterfactuals only afterward.
  • Ask for concrete examples, dates, artifacts, and consequences. “It was painful” is not yet a mechanism.
  • Do not defend Crate, disclose executive hypotheses, or offer fixes.
  • Interview roles separately before reconciling accounts. Treat disagreement as evidence, not an error to erase.
  • Use a note-taker and common template: timeline, problem, consequence, decision link, alternative explanation, evidence source, and confidence. Count accounts, not two interviewees as independent votes.

5. Evidence required for the decision

Back A: robotics integrations

Look for a repeated chain across independent accounts:

Committed robotics deployment → specific missing connection → material operational/business constraint → documented role in departure or credible near-term risk.

Stronger evidence includes decision-maker confirmation, pre-decision correspondence, and a replacement that solved the problem. Match the implicated vendors/workflows to current customer exposure. Before funding, establish that a feasible integration scope could address meaningful retention risk next half.

Mentions of “automation,” hypothetical plans, or competitor marketing are insufficient.

Back B: implementation rebuild

Look for:

Crate-controllable implementation bottleneck → delayed usable value or broken operational commitments → loss of confidence/avoidable cost → departure decision before value realization.

Seek repeated bottlenecks, role-level corroboration, and contrasts with faster implementations. Validate the mechanism against the full cohort, accounting where possible for complexity and customer readiness. Identify changes two squads could deliver; do not assume a wholesale rebuild is necessary.

Back neither—or declare the result inconclusive

Neither is justified if dominant causes lie elsewhere, robotics demand is narrow or speculative, delays are largely outside Crate’s control, or either intervention arrives too late to affect retention. If interviews remain contradictory or unsupported, report insufficient evidence, not “neither causes churn.”

Compare both bets on reachable ARR at risk, evidence strength, controllability, delivery feasibility, and time to impact—not mention counts.

6. Three-week execution

  • Week 1: Audit cohorts and timelines, recruit, pilot one role pair.
  • Week 2: Complete remaining interviews; synthesize after each pair without changing standards to favor an emerging winner.
  • Week 3: Verify discrepancies, triangulate with records, assess engineering scope, and prepare the decision memo with supporting evidence, counterexamples, uncertainties, and a recommendation.

Do not commit two squads merely because one story sounds more persuasive.

Check by check

Got wrong · 1
  • Fits the callThe guide includes timed sections that add up to 45 minutes but does not say what to cut if time runs short, nor does it mark must-ask questions.
Mixed · 1
  • Addresses the actual decisionThe output recommends more research before deciding, states the supplied evidence cannot yet support a call, and specifies exactly what evidence would back each bet or neither.The two graders disagreed on this one.
Got right · 11
  • Uses the supplied evidence correctlyAll factual statements about the current situation are taken directly from the supplied context without invention.
  • Respects explicit constraintsThe guide is under 1,500 words, includes all requested sections (learning goals, timed questions for two roles, guidance, plan changes, decision signals), and is addressed to the exec team.
  • Identifies material uncertaintyIt names key unknowns (decision dates, whether robotics was a binding obstacle, whether delays were Crate-controllable) and says how the interview evidence would resolve them.
  • Avoids unsupported claimsInterpretations are clearly labelled as such (e.g., the exit survey does not establish robotics as the cause), and no cause is presented as established fact.
  • Produces the required deliverableThe output is a complete guide with learning goals, timed questions for both roles, interviewer guidance, plan changes, and decision signals; an exec team could act on it.
  • Tests both theories fairlyQuestions could disprove robotics (e.g., 'Were robots in use? What happened without it?') and implementation (e.g., 'Where did work stall? If go-live had been 45 days, what would still have put the relationship at risk?'), and open questions leave room for a third cause.
  • Different questions for user and signerThe table provides distinct tracks: daily user gets questions about specific shifts, workarounds, and escalations; the signer gets questions about the purchase decision, alternatives, and go-live expectations.
  • Protects the calls and the accountsIt forbids discussing pricing/discounts/roadmap, provides a script to deflect, starts with recording consent, and has a concrete plan for Ana (she does not lead, observes with consent, her prior conversations are treated as leads).
  • Designed to falsifyEach learning goal has disconfirming questions (e.g., 'What happened without the robot connection?' for Bet A, 'If the same product had gone live in 45 days, what would still have put the relationship at risk?' for Bet B).
  • Asks what people did, not what they thinkCore questions ask for a chronological walk-through, specific incidents, dates, and what they did, not opinions or predictions.
  • Doesn't lead the witnessQuestions are open and neutral; robotics and implementation are not named until after broad problem questions, and no answer is hinted at.

Grades and run details

Decision model 88 · LLM judge 13 of 14 checks
Decision model checks
  • passUses the supplied evidence correctly62%
  • failAddresses the actual decision55%
  • passRespects explicit constraints20%
  • passIdentifies material uncertainty99%
  • passAvoids unsupported claims95%
  • passProduces the required deliverable86%
  • passTests both theories fairly100%
  • passDifferent questions for user and signer100%
  • passProtects the calls and the accounts95%
  • passDesigned to falsify64%
  • passAsks what people did, not what they think71%
  • passDoesn't lead the witness58%
  • partialFits the call68%
Run
Run
#1
API response time
80 s
Submitted
30 Sept 2026

Usable with a quick edit11 of 13 checks passed

The guide is thorough, respects all constraints, and provides actionable scripts and decision criteria. A PM could use it with only minor wording tweaks.

Output

Crate churn calls: interview guide

Purpose: Decide between Bet A (robotics integrations) and Bet B (faster implementation) by learning why customers actually leave. Neither theory is proven yet.

Why we're not already sure

  • Exit survey: "Missing features or integrations" bundles two things, isn't robotics-specific, was listed first, and has only 41 responses. People also tend to name the most recent or most socially easy reason rather than the first cause.
  • Robotics: 3 of roughly 60 churned accounts went to a robotics competitor. The 9 lost deals are real but are new-logo losses, not churn.
  • Implementation: The 2.4× churn gap after 90+ days is a correlation. Complex sites may be both slow to launch and likely to leave. We also don't know when customers decided to leave.
  • Ana's and Marcus's evidence: Each is anecdotal and selected. Ana hears from customers who will talk to the CEO.
  • The two theories may be linked. Robotics-heavy sites may be the slow implementations.

Learning goals

  1. When and why each account first considered leaving, relative to go-live.
  2. Whether automation was a cause, a rationalization, or irrelevant. The test is whether they took action (bought robots, evaluated replacements), not whether they mention it.
  3. Whether slow go-live caused the exit, and whether the delay was Crate's doing or the customer's (their data, staffing, customizations).
  4. What else drives exits: price, support, core product gaps, or events outside our control, such as a 3PL losing a shipper contract or being acquired.
  5. Whether the two causes overlap.

Changes to the plan

  • Call math. Two roles per account means 12 calls covers only 6 accounts. I'd use 8 accounts: 4 churned accounts with both roles (8 calls), and 4 at-risk accounts with the operations manager only (4 calls). At-risk executives are the people in renewal negotiations, which is where contamination risk is highest. Comparing what the signer and the daily user each say is itself valuable.
  • Recruit on criteria, not convenience. Among churned accounts, aim for 2 that left within 12 months of signing and 2 that left later. Include go-lives both over and under 90 days, at least 2 sites with robots or carts, and at least 1 that chose "price" in the survey. Pull this from CRM and Customer Success notes this week.
  • Ana doesn't run calls. Customers who know the CEO will be polite, and she has a stated hypothesis. Her relationships are valuable for introductions, and at most 1–2 of her contacts should be in the churned set. Someone with no stake in either bet runs the calls, with a second person taking notes. Ana and Marcus get recordings and summaries afterward.
  • At-risk accounts need Customer Success coordination. CS confirms no discount conversation is happening that week. The invitation states that this is research, separate from renewal. If a customer tries to negotiate, redirect and tell CS afterward. Weight at-risk answers less, since they describe intentions, not decisions.
  • Pre-register. Before call one, Ana and Marcus each read the last section and write what would change their mind.
  • Run a parallel data pull. Get tenure at churn, go-live days, and robotics on site for all roughly 60 churned accounts. If the calls and the data disagree, the data wins.

Guidance for whoever runs the calls

  • Consent first. The first thing you say after your name is a request to record. If they decline, take notes only and continue.
  • Off-limits: pricing, discounts, and roadmap. If they raise price, listen and ask what they compared us with and what they got, but no numbers or offers. If asked "will you build X?", say: "I can't speak to plans. I'm here to understand your experience." If they ask for a commercial conversation, point them to their account manager.
  • Do not say "robotics" or "implementation" until the prompted section. Record whether each topic came up unprompted or only when asked. These carry very different weight.
  • Alternate the order of the two prompted topics from call to call.
  • Ask for stories and dates, not opinions: "Tell me about the last time," "What happened next?" Favor actions ("Who did you call? What did you buy?") over stated reasons.
  • Don't defend Crate or fix problems. Answer criticism with "Say more about that."
  • Use silence. Wait a few seconds after an answer; the real reason often comes next.
  • Debrief within 24 hours on a one-page sheet: decision date, tenure, go-live days, robots on site, trigger event, unprompted causes, prompted causes, strongest quote, and anything that contradicts our hypotheses.

Guide: Operations manager (daily user), 45 min

TimeSectionQuestions
0:00–0:03OpenAsk permission to record. "I'm [name] from Crate's product team. I'm here to learn, not sell, and nothing you say affects your account. I can't discuss pricing or plans."
0:03–0:08ContextTell me about your site and your role. What does a normal day look like? What equipment and systems run alongside Crate?
0:08–0:20Start-up storyTake me back to when you started with Crate. What happened between signing and running real orders? What got delayed, and why? Who was involved on each side? When did it first feel like it was working?
0:20–0:32Doubt timelineWhen did you first wonder if Crate was the right system? Where were you and what happened? What did you do next, and who did you talk to? (Churned: what was the last straw? At-risk: have you looked at alternatives, and what prompted that?)
0:32–0:39Prompted topicsAutomation: Do you use, or plan to use, robots, autonomous carts, or pick-assist? What's in place, and when did it arrive? How did it work with Crate, and what did you do about it? Setup: Looking back, how did the time to get live affect your team or your view of Crate? (Skip anything already covered in depth.)
0:39–0:43CounterfactualWhat one thing would have kept you? (At-risk: what would need to change for you to be confident staying?) Has anything you've seen elsewhere worked better?
0:43–0:45CloseWhat haven't I asked that I should have? Who else should I talk to? Thank them.

Guide: VP Operations / COO (signer), 45 min

TimeSectionQuestions
0:00–0:03OpenSame as above. Ask permission to record first.
0:03–0:08ContextTell me about your business and your customers. How has the past year changed things for you?
0:08–0:18BuyingWhy did you choose Crate? What did you hope would be different after 6–12 months? What were you told about go-live, and what happened? How did you know whether it was working?
0:18–0:30DecisionWalk me through the decision to leave (or to reconsider). When did it first come up, and who raised it? What did you look at? Which alternatives did you consider? Who made the final call, and when? What would have changed the outcome?
0:30–0:38Prompted topicsAutomation: What's your automation plan over the next 2–3 years? What's happened so far? How did it figure in the decision? Setup: How did the time to go live figure in your thinking, if at all? (Skip anything already covered in depth.)
0:38–0:43Business contextWhat else changed around then: shipper contracts won or lost, sites opened or closed, leadership, budget, acquisitions? If you ranked everything that mattered, what's first? What do you use now, and why?
0:43–0:45CloseAnything I missed? Thank them.

What we'd need to hear

Eight accounts can't produce percentages. We're looking for a pattern strong enough to act on.

Back Bet A (robotics) if, in at least 3 of the 4 churned accounts: - Automation comes up unprompted, or with only light prompting, as a reason. - They acted on it: robots bought, piloted, or contracted before they decided to leave. - They were live and broadly content with Crate beforehand. - They evaluated or moved to something that connects to their robots, and say they'd likely have stayed if we had. - The same story appears in at least 2 of the 4 at-risk accounts.

Back Bet B (implementation) if, in at least 3 of the 4 churned accounts: - Their first serious doubt came during or shortly after a late go-live, typically within the first year. - They describe never reaching value, or losing their internal champion during the delay. - The delay was mainly Crate's doing (process, handoffs, configuration), not their own data or staffing. - They would say faster go-live would probably have kept them.

Neither bet if: - Exits are mainly driven by events such as a lost shipper contract, site closure, acquisition, or price. - Robotics is mentioned but no one acted on it. - Go-live was slow but they were happy afterward. - Reasons scatter with no pattern. - Core product gaps, such as reporting or inventory accuracy, show up repeatedly. That would point to a third bet we haven't framed.

Both: If the same accounts show robotics-heavy sites with long go-lives because integrations took months, that is a third reading. It favors Bet A if the cause is missing integrations, and Bet B if it's our process. Decide by asking what the customer says was the blocker.

Split results: If churned accounts split evenly, or contradict the data pull, say so. Recommend a targeted follow-up, such as segmenting churn by tenure, rather than forcing a choice.

Check by check

Mixed · 2
  • Uses the supplied evidence correctlyAll factual claims about the current situation are directly supported by the brief or derived correctly from its numbers.The two graders disagreed on this one.
  • Addresses the actual decisionThe output provides clear, actionable criteria for what would back each bet or neither after the calls, exactly as the brief requests.The two graders disagreed on this one.
Got right · 11
  • Respects explicit constraintsThe guide respects all constraints: it stays under 1,500 words, includes all requested sections, and provides concrete scripts and plans to enforce the off-limits topics and consent.
  • Identifies material uncertaintyThe output explicitly names the key unknowns (weak survey, correlation vs causation, unknown decision timing) and says how the calls and a parallel data pull would resolve them.
  • Avoids unsupported claimsHypotheses and interpretations are clearly labelled as such, and no cause or forecast is presented as established fact.
  • Produces the required deliverableThe guide is complete with learning goals, timed questions for two roles, interviewer guidance, plan changes, and decision signals, all under 1,500 words and ready for the exec team.
  • Tests both theories fairlyBoth theories get questions that could disprove them (e.g., no action on robotics, happiness after slow go-live), and open-ended timeline questions leave room for a third cause.
  • Different questions for user and signerThe guide has distinct tracks: daily user questions focus on day-to-day experience and workarounds, signer questions on the decision process and alternatives.
  • Protects the calls and the accountsIt includes a clear off-limits rule with scripts for pricing/roadmap, starts with consent, and addresses the CEO's bias by having her not lead calls and limiting her contacts.
  • Designed to falsifyEvery learning goal has a disconfirming question, such as asking what they did about robots or whether they ever reached value after a slow go-live.
  • Asks what people did, not what they thinkCore questions ask for specific recent actions and events ('What happened next?', 'Who did you call?'), not opinions or predictions.
  • Doesn't lead the witnessQuestions are neutral and open-ended; the prompted topics are introduced without pitching or hinting at the desired answer.
  • Fits the callBoth 45-minute guides have timed sections that sum correctly, a realistic number of questions, and instructions to skip covered topics if time is short.

Grades and run details

Decision model 77 · LLM judge 14 of 14 checks
Decision model checks
  • failUses the supplied evidence correctly16%
  • failAddresses the actual decision16%
  • partialRespects explicit constraints15%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims75%
  • passProduces the required deliverable91%
  • passTests both theories fairly100%
  • passDifferent questions for user and signer100%
  • passProtects the calls and the accounts97%
  • passDesigned to falsify85%
  • passAsks what people did, not what they think82%
  • passDoesn't lead the witness49%
  • partialFits the call33%
Run
Run
#1
API response time
1.6 min
Submitted
30 Sept 2026

Results

Every setup we’ve tested on this task, across all cases and repeats, graded on the current checklist. Provisional The checklist is still being calibrated against our PM.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT95.892.31None
2GPT-6.1 SolwithAPI92.192.62None
3Opus 5.5withClaude94.169.22None
4Sonnet 5.5withAPI86.484.62None
5GPT-6 LunawithAPI90.273.921 capped

About the task

The PM job

Preparing for customer calls.

Why it matters

Leading questions produce the answers you wanted. A good guide is designed to prove you wrong.

What good looks like

  • Past-behaviour questions
  • No leading prompts
  • Clear learning goals
  • Could disprove the team's hypothesis

Deliberately not measured

    Capability tested

    Question design

    The failure we’re looking for

    Hypothetical, leading questions

    Grading

    Decision model and LLM judge, calibrated against a blind PM review