Usable with a quick edit11 of 13 checks passed
The guide is thorough, respects all constraints, and provides actionable scripts and decision criteria. A PM could use it with only minor wording tweaks.
Output
Crate churn calls: interview guide
Purpose: Decide between Bet A (robotics integrations) and Bet B (faster implementation) by learning why customers actually leave. Neither theory is proven yet.
Why we're not already sure
- Exit survey: "Missing features or integrations" bundles two things, isn't robotics-specific, was listed first, and has only 41 responses. People also tend to name the most recent or most socially easy reason rather than the first cause.
- Robotics: 3 of roughly 60 churned accounts went to a robotics competitor. The 9 lost deals are real but are new-logo losses, not churn.
- Implementation: The 2.4× churn gap after 90+ days is a correlation. Complex sites may be both slow to launch and likely to leave. We also don't know when customers decided to leave.
- Ana's and Marcus's evidence: Each is anecdotal and selected. Ana hears from customers who will talk to the CEO.
- The two theories may be linked. Robotics-heavy sites may be the slow implementations.
Learning goals
- When and why each account first considered leaving, relative to go-live.
- Whether automation was a cause, a rationalization, or irrelevant. The test is whether they took action (bought robots, evaluated replacements), not whether they mention it.
- Whether slow go-live caused the exit, and whether the delay was Crate's doing or the customer's (their data, staffing, customizations).
- What else drives exits: price, support, core product gaps, or events outside our control, such as a 3PL losing a shipper contract or being acquired.
- Whether the two causes overlap.
Changes to the plan
- Call math. Two roles per account means 12 calls covers only 6 accounts. I'd use 8 accounts: 4 churned accounts with both roles (8 calls), and 4 at-risk accounts with the operations manager only (4 calls). At-risk executives are the people in renewal negotiations, which is where contamination risk is highest. Comparing what the signer and the daily user each say is itself valuable.
- Recruit on criteria, not convenience. Among churned accounts, aim for 2 that left within 12 months of signing and 2 that left later. Include go-lives both over and under 90 days, at least 2 sites with robots or carts, and at least 1 that chose "price" in the survey. Pull this from CRM and Customer Success notes this week.
- Ana doesn't run calls. Customers who know the CEO will be polite, and she has a stated hypothesis. Her relationships are valuable for introductions, and at most 1–2 of her contacts should be in the churned set. Someone with no stake in either bet runs the calls, with a second person taking notes. Ana and Marcus get recordings and summaries afterward.
- At-risk accounts need Customer Success coordination. CS confirms no discount conversation is happening that week. The invitation states that this is research, separate from renewal. If a customer tries to negotiate, redirect and tell CS afterward. Weight at-risk answers less, since they describe intentions, not decisions.
- Pre-register. Before call one, Ana and Marcus each read the last section and write what would change their mind.
- Run a parallel data pull. Get tenure at churn, go-live days, and robotics on site for all roughly 60 churned accounts. If the calls and the data disagree, the data wins.
Guidance for whoever runs the calls
- Consent first. The first thing you say after your name is a request to record. If they decline, take notes only and continue.
- Off-limits: pricing, discounts, and roadmap. If they raise price, listen and ask what they compared us with and what they got, but no numbers or offers. If asked "will you build X?", say: "I can't speak to plans. I'm here to understand your experience." If they ask for a commercial conversation, point them to their account manager.
- Do not say "robotics" or "implementation" until the prompted section. Record whether each topic came up unprompted or only when asked. These carry very different weight.
- Alternate the order of the two prompted topics from call to call.
- Ask for stories and dates, not opinions: "Tell me about the last time," "What happened next?" Favor actions ("Who did you call? What did you buy?") over stated reasons.
- Don't defend Crate or fix problems. Answer criticism with "Say more about that."
- Use silence. Wait a few seconds after an answer; the real reason often comes next.
- Debrief within 24 hours on a one-page sheet: decision date, tenure, go-live days, robots on site, trigger event, unprompted causes, prompted causes, strongest quote, and anything that contradicts our hypotheses.
Guide: Operations manager (daily user), 45 min
| Time | Section | Questions |
|---|---|---|
| 0:00–0:03 | Open | Ask permission to record. "I'm [name] from Crate's product team. I'm here to learn, not sell, and nothing you say affects your account. I can't discuss pricing or plans." |
| 0:03–0:08 | Context | Tell me about your site and your role. What does a normal day look like? What equipment and systems run alongside Crate? |
| 0:08–0:20 | Start-up story | Take me back to when you started with Crate. What happened between signing and running real orders? What got delayed, and why? Who was involved on each side? When did it first feel like it was working? |
| 0:20–0:32 | Doubt timeline | When did you first wonder if Crate was the right system? Where were you and what happened? What did you do next, and who did you talk to? (Churned: what was the last straw? At-risk: have you looked at alternatives, and what prompted that?) |
| 0:32–0:39 | Prompted topics | Automation: Do you use, or plan to use, robots, autonomous carts, or pick-assist? What's in place, and when did it arrive? How did it work with Crate, and what did you do about it? Setup: Looking back, how did the time to get live affect your team or your view of Crate? (Skip anything already covered in depth.) |
| 0:39–0:43 | Counterfactual | What one thing would have kept you? (At-risk: what would need to change for you to be confident staying?) Has anything you've seen elsewhere worked better? |
| 0:43–0:45 | Close | What haven't I asked that I should have? Who else should I talk to? Thank them. |
Guide: VP Operations / COO (signer), 45 min
| Time | Section | Questions |
|---|---|---|
| 0:00–0:03 | Open | Same as above. Ask permission to record first. |
| 0:03–0:08 | Context | Tell me about your business and your customers. How has the past year changed things for you? |
| 0:08–0:18 | Buying | Why did you choose Crate? What did you hope would be different after 6–12 months? What were you told about go-live, and what happened? How did you know whether it was working? |
| 0:18–0:30 | Decision | Walk me through the decision to leave (or to reconsider). When did it first come up, and who raised it? What did you look at? Which alternatives did you consider? Who made the final call, and when? What would have changed the outcome? |
| 0:30–0:38 | Prompted topics | Automation: What's your automation plan over the next 2–3 years? What's happened so far? How did it figure in the decision? Setup: How did the time to go live figure in your thinking, if at all? (Skip anything already covered in depth.) |
| 0:38–0:43 | Business context | What else changed around then: shipper contracts won or lost, sites opened or closed, leadership, budget, acquisitions? If you ranked everything that mattered, what's first? What do you use now, and why? |
| 0:43–0:45 | Close | Anything I missed? Thank them. |
What we'd need to hear
Eight accounts can't produce percentages. We're looking for a pattern strong enough to act on.
Back Bet A (robotics) if, in at least 3 of the 4 churned accounts: - Automation comes up unprompted, or with only light prompting, as a reason. - They acted on it: robots bought, piloted, or contracted before they decided to leave. - They were live and broadly content with Crate beforehand. - They evaluated or moved to something that connects to their robots, and say they'd likely have stayed if we had. - The same story appears in at least 2 of the 4 at-risk accounts.
Back Bet B (implementation) if, in at least 3 of the 4 churned accounts: - Their first serious doubt came during or shortly after a late go-live, typically within the first year. - They describe never reaching value, or losing their internal champion during the delay. - The delay was mainly Crate's doing (process, handoffs, configuration), not their own data or staffing. - They would say faster go-live would probably have kept them.
Neither bet if: - Exits are mainly driven by events such as a lost shipper contract, site closure, acquisition, or price. - Robotics is mentioned but no one acted on it. - Go-live was slow but they were happy afterward. - Reasons scatter with no pattern. - Core product gaps, such as reporting or inventory accuracy, show up repeatedly. That would point to a third bet we haven't framed.
Both: If the same accounts show robotics-heavy sites with long go-lives because integrations took months, that is a third reading. It favors Bet A if the cause is missing integrations, and Bet B if it's our process. Decide by asking what the customer says was the blocker.
Split results: If churned accounts split evenly, or contradict the data pull, say so. Recommend a targeted follow-up, such as segmenting churn by tenure, rather than forcing a choice.
Check by check
Mixed · 2
- Uses the supplied evidence correctlyAll factual claims about the current situation are directly supported by the brief or derived correctly from its numbers.The two graders disagreed on this one.
- Addresses the actual decisionThe output provides clear, actionable criteria for what would back each bet or neither after the calls, exactly as the brief requests.The two graders disagreed on this one.
Got right · 11
- Respects explicit constraintsThe guide respects all constraints: it stays under 1,500 words, includes all requested sections, and provides concrete scripts and plans to enforce the off-limits topics and consent.
- Identifies material uncertaintyThe output explicitly names the key unknowns (weak survey, correlation vs causation, unknown decision timing) and says how the calls and a parallel data pull would resolve them.
- Avoids unsupported claimsHypotheses and interpretations are clearly labelled as such, and no cause or forecast is presented as established fact.
- Produces the required deliverableThe guide is complete with learning goals, timed questions for two roles, interviewer guidance, plan changes, and decision signals, all under 1,500 words and ready for the exec team.
- Tests both theories fairlyBoth theories get questions that could disprove them (e.g., no action on robotics, happiness after slow go-live), and open-ended timeline questions leave room for a third cause.
- Different questions for user and signerThe guide has distinct tracks: daily user questions focus on day-to-day experience and workarounds, signer questions on the decision process and alternatives.
- Protects the calls and the accountsIt includes a clear off-limits rule with scripts for pricing/roadmap, starts with consent, and addresses the CEO's bias by having her not lead calls and limiting her contacts.
- Designed to falsifyEvery learning goal has a disconfirming question, such as asking what they did about robots or whether they ever reached value after a slow go-live.
- Asks what people did, not what they thinkCore questions ask for specific recent actions and events ('What happened next?', 'Who did you call?'), not opinions or predictions.
- Doesn't lead the witnessQuestions are neutral and open-ended; the prompted topics are introduced without pitching or hinting at the desired answer.
- Fits the callBoth 45-minute guides have timed sections that sum correctly, a realistic number of questions, and instructions to skip covered topics if time is short.
Grades and run details
Decision model 77 · LLM judge 14 of 14 checks
Decision model checks
- failUses the supplied evidence correctly16%
- failAddresses the actual decision16%
- partialRespects explicit constraints15%
- passIdentifies material uncertainty100%
- passAvoids unsupported claims75%
- passProduces the required deliverable91%
- passTests both theories fairly100%
- passDifferent questions for user and signer100%
- passProtects the calls and the accounts97%
- passDesigned to falsify85%
- passAsks what people did, not what they think82%
- passDoesn't lead the witness49%
- partialFits the call33%
Run
- Run
- #1
- API response time
- 1.6 min
- Submitted
- 30 Sept 2026