Needs more than a quick edit10 of 13 checks passed
The guide lacks separate question tracks for the daily user and the decision-maker, and it doesn't prioritize questions for time management. A PM would need to add role-specific questions and mark must-asks, which is more than a quick edit.
Output
Crate customer research guide: why are warehouses leaving?
Purpose: In three weeks, learn whether churn is more plausibly driven by missing robotics integrations (Bet A), slow implementation (Bet B), or something else. These calls are for understanding customers’ experiences—not selling, negotiating renewals, or validating either executive’s theory.
Learning goals
For each account, establish:
- What happened, and when? Separate implementation, go-live, first value, automation needs, and the point when leaving became likely or decided.
- What mattered most? Distinguish a decisive problem from a feature request, contributing frustration, or reason offered after the fact.
- What evidence supports the explanation? Look for actual workflow impact, workarounds, alternatives evaluated, and actions taken—not just opinions or hypotheticals.
- What might change the outcome? Explore credible counterfactuals without promising anything.
- What else explains churn? Surface causes beyond A and B.
Recommended plan changes
The proposed 12 calls cannot cover both the daily user and contract signer on all 12 accounts: that would require 24 interviews. Keep the 12-call cap, but interview two roles separately at each of six accounts: three churned accounts and three red-health accounts, with one warehouse operations manager and one VP of Operations/COO per account. This gives paired perspectives and preserves both customer situations, but is a small, directional sample—not a prevalence estimate. Select accounts for varied implementation times and automation situations where possible; don’t choose only accounts with known robotics issues.
Do not have Ana conduct four calls with former customers she knows personally. Her involvement risks courtesy bias and leading the conversation. Use an independent interviewer; ideally don’t include personal contacts in this small core sample. If one is included, disclose the relationship, and have Ana neither attend nor receive attributable notes.
Because five at-risk accounts are in renewal talks and receiving discount offers, use an interviewer outside the account/renewal team. Tell participants their answers won’t affect service or renewal discussions. Don’t share interview content with the account team in a way that could be used in negotiation. Follow Legal’s recording requirement below.
45-minute guide: churned customers
| Time | Questions |
|---|---|
| 0–3 | Start by asking: “May I record this conversation?” If no, take notes instead. Explain the purpose, that there are no sales or renewal implications, and that we won’t discuss pricing, discounts, or roadmap. |
| 3–7 | “What was your role in choosing and using Crate?” “What did your warehouse need the system to help you do?” |
| 7–18 | “Take me through your experience with Crate, from signing through the decision to leave.” “When did you first consider switching? What happened just before that?” “Who was involved, and when was the decision effectively made?” |
| 18–27 | “How did implementation and go-live go?” “Where, if anywhere, did it take longer or require more effort than expected?” “What was the impact on your operation?” “When did you first get value from Crate?” |
| 27–35 | “Did automation or robotics play a role in your warehouse during this period?” If yes: “What equipment or workflow?” “How did it work with Crate?” “What did you do when it didn’t?” “When did this become a problem relative to the decision to leave?” |
| 35–41 | “What alternatives did you consider, and what drove the choice?” “What, if anything, could realistically have changed your decision?” “What was the most important reason—not just one frustration?” |
| 41–45 | “What haven’t I asked that matters?” “Is there anything we should correct about our understanding?” Thank them. |
45-minute guide: red-health customers
Use the same opening and close. Do not imply that Crate can fix an issue or influence renewal terms.
| Time | Questions |
|---|---|
| 0–3 | Start by asking: “May I record this conversation?” If no, take notes. Explain that this is research, not a sales or renewal conversation; participation won’t affect service or renewal discussions. |
| 3–7 | “What is your role in using or overseeing Crate?” “What does a successful warehouse operation look like for you?” |
| 7–17 | “How has your experience with Crate unfolded so far?” “When have things gone better or worse than expected?” “What is working well today?” |
| 17–27 | “Where are you in implementation and go-live?” “What has taken the most effort or time?” “What value, if any, have you realized so far?” “What is the operational impact of anything that remains difficult?” |
| 27–35 | “How, if at all, does automation or robotics fit into your operation?” If relevant: “What equipment or workflow?” “How does it work with Crate today?” “What workarounds do you use?” “When did this become important?” |
| 35–41 | “Have you considered alternatives to Crate? What prompted that, if so?” “What would need to be true for Crate to work well for your operation?” “What are the most important unresolved issues?” |
| 41–45 | “What haven’t I asked that matters?” “Is there anything we should correct?” Thank them. |
Guidance for interviewers
- Ask the questions as written, but follow the participant’s timeline. Use neutral probes: “What happened next?” “Can you give me an example?” “How do you know?”
- Don’t offer Ana’s or Marcus’s theories, mention the exit-survey results, or ask leading questions such as “Would robots have saved you?” Ask about events before asking for interpretations.
- Separate when an issue occurred from when the participant says it mattered. Probe for specific actions and evidence: evaluations, workarounds, escalations, or a switch decision.
- Don’t discuss pricing, discounts, renewal negotiations, or roadmap. If asked about roadmap, say you can’t speak to future plans. Don’t make promises or relay an interview answer as a renewal signal.
- Interview roles separately. Record role and account context in notes, but distinguish direct experience from second-hand views. Afterward, write a short timeline, leading cause(s), evidence, and uncertainties for each account.
- Treat recordings and notes as confidential; use only with consent and Legal-approved handling. Don’t present anecdotes as representative of all customers.
What would support each bet?
Support Bet A (robotics integrations): In multiple independent accounts, a specific robotics workflow or integration gap caused meaningful operational friction, and that issue arose before the decision to leave or seriously consider leaving. Stronger evidence: an actual robot deployment or concrete plan, a costly workaround, and a competitor selected or evaluated for this capability. A feature mention alone—or robotics cited in a new-logo deal—doesn’t establish that it drove existing-customer churn.
Support Bet B (faster implementation): In multiple accounts, implementation delays or effort materially postponed go-live or first value, caused operational consequences, and preceded the decision to leave or consider leaving. Stronger evidence: specific preventable blockers, an unmet expectation, and a credible account that earlier go-live would have changed the outcome. The 94-day median, 45-day promise, and 2.4× association make this worth testing, but do not prove causation.
Support neither / remain uncertain: The call evidence points more strongly to another cause; A or B appears only after the decision; accounts describe issues without meaningful consequences; or evidence is mixed and no cause consistently precedes churn. Don’t force a winner from 12 interviews. Compare patterns across accounts, not individual interview counts, and triangulate with CRM timelines and other churn evidence. The exit survey’s single-choice format, first-listed feature option, and sparse free text limit what it can establish.
Check by check
Got wrong · 2
- Different questions for user and signerThe guide does not provide distinct question tracks for the daily warehouse operations manager and the executive who signed; it only differentiates churned vs at-risk customers, with the same questions for both roles within each track.
- Fits the callAlthough timed sections add up to 45 minutes, the guide does not mark must-ask questions or say what to cut if time runs short, so an interviewer could run out of time without covering the most critical probes.
Mixed · 1
- Addresses the actual decisionThe guide clearly states what evidence from the calls would back Bet A, Bet B, or neither, which is the decision framework the exec team needs.The two graders disagreed on this one.
Got right · 10
- Uses the supplied evidence correctlyEvery statement about the current situation is taken directly from the supplied context, with no invented facts.
- Respects explicit constraintsThe guide respects all constraints: it includes learning goals, timed questions for both customer types, interviewer guidance, plan changes, and decision signals; it stays under 1,500 words; it enforces no pricing/roadmap discussion and recording consent.
- Identifies material uncertaintyIt names key unknowns (exit survey weakness, correlation vs causation, timing of churn decision) and says how the calls will resolve them through patterns and timelines.
- Avoids unsupported claimsInterpretations like the exit survey's limitations and the 2.4× association are clearly labeled as not proving causation, not presented as established fact.
- Produces the required deliverableThe output is a complete call guide with all requested sections, written for the exec team, and well under 1,500 words.
- Tests both theories fairlyBoth the robotics and implementation theories get questions that could disprove them, open timeline questions come first, and the guide explicitly leaves room for a third cause.
- Protects the calls and the accountsIt gives a clear rule and script for pricing/roadmap questions, starts with recording consent, and addresses Ana's bias by recommending she not lead calls and, if a contact is included, that she not attend or receive attributable notes.
- Designed to falsifyEvery learning goal has questions whose honest answers could disprove the hypothesis, such as asking when automation became a problem relative to the decision to leave.
- Asks what people did, not what they thinkCore questions ask for specific past events and actions (e.g., 'When did you first consider switching? What happened just before that?'), with opinions and predictions clearly secondary.
- Doesn't lead the witnessQuestions are neutral and open; the guide avoids mentioning either executive's theory until after the participant's own timeline, and prohibits leading probes.
Grades and run details
Decision model 85 · LLM judge 11 of 14 checks
Decision model checks
- passUses the supplied evidence correctly81%
- failAddresses the actual decision50%
- passRespects explicit constraints26%
- passIdentifies material uncertainty95%
- passAvoids unsupported claims91%
- passProduces the required deliverable73%
- passTests both theories fairly99%
- partialDifferent questions for user and signer74%
- passProtects the calls and the accounts79%
- passDesigned to falsify75%
- passAsks what people did, not what they think70%
- passDoesn't lead the witness55%
- partialFits the call29%
Run
- Run
- #1
- API response time
- 29 s
- Submitted
- 30 Sept 2026