Tasks / Discover

Customer research call guide

Can the model write a guide that uncovers behaviour rather than opinions?

Measures the modelTask v1.0 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 14 graded outputs by 7 models. 36% were usable with at most a quick edit.

Reliably right

  1. Identifies material uncertainty96% pass
    Names specific unknowns (cost of lateness, ease of chasing, automation fit, willingness to pay) and says mixed findings mean targeted follow-up.
    GPT-6.1 Sol · API · Would freelancers pay to stop chasing invoices?
  2. Coaches first-time interviewers96% pass
    Provides specific, actionable instructions: alternate roles, practice call, avoid pitching, allow silence, separate observations from interpretations.
    GPT-6.1 Sol · API · Would freelancers pay to stop chasing invoices?
  3. Addresses the actual decision93% pass
    Provides clear go/stop conditions tied to specific call outcomes, framed for the team.
    GPT-6.1 Sol · API · Would freelancers pay to stop chasing invoices?

Where it slips

  1. Marks what to cut if the call runs over29% pass
    Timings add up to 30 minutes but no must-ask questions are marked and no guidance on what to cut if time runs short.
    GPT-6.1 Sol · API · Would freelancers pay to stop chasing invoices?
  2. Respects explicit constraints61% pass
    The guide is clearly over 900 words, violating the explicit length constraint.
    Opus 5.5 · Claude · Would freelancers pay to stop chasing invoices?
  3. Uses the supplied evidence correctly70% pass
    It uses most supplied numbers correctly, but invents or overstates current-situation details such as an 'Autochase support ticket' population and treats recruiting-plan inferences as facts.
    Sonnet 5.5 · API · Would freelancers pay to stop chasing invoices?

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a PM at Paydeck. Next week, Mei (a designer) and Tomas (an engineer) will run 8 customer calls to test whether we should build Autochase next quarter. Neither has run a customer interview before. Write the call guide they'll use: the learning goals, the questions with rough timings for a 30-minute call, notes for first-time interviewers, who we should talk to, and what we'd need to hear to go ahead or to stop. Keep it under 900 words. Sam, the PM who owns Autochase, has drafted some questions and a recruiting plan. They're below with everything else we know.

What the model was given6 items: About Paydeck, The idea, Sam's hypothesis, What we know, Sam's draft questions, Sam's recruiting plan
About PaydeckInvoicing software for freelancers and small agencies (1 to 10 people). 48,000 accounts, 6,100 of them paying $12 a month for Pro.
The ideaAutochase: automatic, escalating reminders for overdue invoices, sold as a $10-a-month add-on. There is no prototype yet.
Sam's hypothesis“Late payments cost our users real money, and chasing them is painful enough that they'd pay $10 a month to automate it.”
What we know34% of invoices sent through Paydeck are paid after their due date; the median is 16 days late. The manual 'Send reminder' button was used by 12% of accounts in the last 90 days. 40 support tickets last quarter asked for automatic reminders.
Sam's draft questions1. How frustrating do you find chasing late payments? 2. Would an automatic reminder feature be useful to you? 3. How much would you pay for it: $5, $10 or $15 a month? 4. Do you think your clients would be annoyed by automated reminders? 5. What other features would you like us to build?
Sam's recruiting planEmail the 300 accounts that used the 'Send reminder' button most last month, and book the first 8 who reply.
What a strong answer doesThe answer key the graders mark against

A guide that splits Sam's hypothesis into goals that can each fail: do late payments actually cost users time or money, how do they deal with them today, would they pay to fix it, and does chasing harm client relationships. The core asks for the story of the last overdue invoice (what happened, what they did, how long it took, what it cost) and what they've already tried or paid for, instead of opinions or predictions. It replaces Sam's leading and hypothetical questions: no 'how frustrating', no 'would it be useful', no price menu; payment is tested through past spending or a real commitment (say, a paid pilot sign-up) at the end, after the problem questions and without pitching Autochase first. It fits 30 minutes with timings, coaches first-timers (ask for specifics, follow up with 'what happened next', stay quiet, don't pitch or defend), and fixes the recruiting plan: the heaviest reminder users will over-state demand, so it adds accounts with overdue invoices that never send reminders, and says who to exclude or balance. It ends with clear go and stop signals tied to what people did, such as most calls describing recent chasing that took real time plus a workaround they've tried or paid for, against most saying one nudge does the job.

Critical failures (cap the score at 40)
  • Relies on hypothetical questions (would you use it, how much would you pay) as the main evidence for the decision
  • Describes or pitches Autochase before asking how the person handles late payments today
Case

Mid-level · v1.1 · synthetic · B2B SaaS, freelancers, willingness to pay, first-time interviewers

The outputs

7 setups answered this brief. Pick up to three to read side by side.

Critical failure: Relies on hypothetical questions (would you use it, how much would you pay) as the main evidence for the decision. Score capped at 40.

Needs more than a quick edit3 of 12 checks passed

A PM would need to fix the recruiting plan, remove the pitch and hypothetical price test, and replace the go/stop criteria with behavior-based evidence. The current guide would likely produce biased demand signals.

Output

Welcome to user research week! Mei and Tomas, running these calls is one of the highest-leverage things you can do this quarter. Don't worry about being first-timers; your fresh eyes are an asset. Here is your battle-tested guide for next week.

---

1. Learning Goals

We need to answer two core questions before committing engineering time to Autochase: 1. Pain: Does chasing late invoices cause enough operational friction and cash flow anxiety that users want it off their plates completely? 2. Value: Is automation worth $10/month, or do users prefer control (e.g., hitting a manual button)?

---

2. Who We Should Talk To

Per Sam’s plan: We are emailing the 300 Pro accounts that used the manual 'Send reminder' button most frequently last month and booking the first 8 who reply. * Why: These users already feel the pain. If they won't pay for automation, nobody will.

---

3. Notes for First-Time Interviewers (Read This Twice!)

  • Do not pitch Autochase early. If you pitch it, they will be polite and say yes. Sell the problem, not our solution.
  • Past behavior beats future promises. When people say "I would pay $10 for that," they are often lying to be nice. Ask about what they currently do and what they've paid for in the past.
  • Embrace silence. After you ask a question, stop talking. Let them fill the silence. The best insights live in the second minute of their answer.
  • One person leads, one takes notes. Mei, take the lead on questions today; Tomas, focus on verbatim quotes and body language (and vice versa for call #2).

---

4. The 30-Minute Call Guide

#### Phase 1: Context & Warm-up (5 mins) Goal: Build rapport and understand their business. * "Tell me a bit about your business—what do you do, and who are your typical clients?" * "Walk me through how invoicing and getting paid works in your workflow today."

#### Phase 2: Past Behavior & Pain (10 mins) Goal: Uncover the true emotional and financial cost of late payments. * [Replace Sam's Q1] "Take me back to the last time an invoice went significantly past its due date. What happened?" * "How did you handle it? Walk me through the exact steps you took to chase them." * "How does chasing clients for money impact your relationship with them, or your peace of mind?" * Data check: 34% of our invoices are paid late (median 16 days). Ask: "Does that match your reality?"

#### Phase 3: Current Workarounds & Solution Testing (10 mins) Goal: See how they solve this now, then introduce Autochase. * "You’ve used our manual 'Send reminder' button recently. How is that working for you? What do you wish it did that it doesn't?" * The Reveal: "We’re exploring a feature called Autochase that automatically sends escalating reminders on a schedule you set until the invoice is paid. Based on what you've told me about your workflow, how would something like that fit into your business?" * [Replace Sam's Q4] "Some freelancers worry automated reminders might annoy their clients. How do you feel about that risk for your clients?"

#### Phase 4: Pricing & Wrap-up (5 mins) Goal: Test the $10/mo price point and wrap up. * [Replace Sam's Q2/Q3] "We're looking at pricing this as a $10-a-month add-on. When you think about the time it saves or the awkwardness it avoids, does $10/mo feel like a bargain, too expensive, or about right?" * "Before we wrap up, is there anything else about getting paid by clients that drives you crazy that we didn't touch on?" * Thank them warmly and end the call.

---

5. Go / No-Go Criteria

After 8 calls, Mei, Tomas, and I will sync. We move forward to build Autochase next quarter only if: 1. At least 6 out of 8 interviewers describe late payments as a top-3 operational headache (not just a minor annoyance). 2. Users unprompted mention spending hours a month on this, or losing sleep over cash flow. 3. At least 5 out of 8 react to the $10/month price point as reasonable or cheap relative to the value.

We stop and pivot if: * Users say, "I actually like reviewing and sending reminders manually because I customize the tone per client." (This kills the automation thesis). * Users think $10/month is wildly overpriced for a reminder tool.

Claims the judge couldn’t find in the brief

  • We are emailing the 300 Pro accounts that used the manual 'Send reminder' button most frequently last month and booking the first 8 who reply.
  • These users already feel the pain.
  • If they won't pay for automation, nobody will.
  • When people say 'I would pay $10 for that,' they are often lying to be nice.
  • Autochase automatically sends escalating reminders on a schedule the user sets until the invoice is paid.
  • Autochase would save time and avoid awkwardness.
  • Late payments cause cash-flow anxiety and operational friction for users.
  • Running these calls is one of the highest-leverage things the team can do this quarter.
  • The guide is battle-tested.
  • Users may prefer control, such as hitting a manual button.
  • At least 6 out of 8 interviewers will describe late payments as a top-3 operational headache if the feature is worth building.
  • At least 5 out of 8 users will react to the $10/month price point as reasonable or cheap if the feature is worth building.

Grades and run details

Decision model 50 · LLM judge 4 of 13 checks
Decision model checks
  • passUses the supplied evidence correctly14%
  • passAddresses the actual decision60%
  • failRespects explicit constraints80%
  • passIdentifies material uncertainty28%
  • partialAvoids unsupported claims35%
  • partialProduces the required deliverable75%
  • failTalks to the right people100%
  • partialCoaches first-time interviewers61%
  • partialDesigned to falsify44%
  • partialAsks what people did, not what they think79%
  • failDoesn't lead the witness56%
  • partialMarks what to cut if the call runs over57%
Run
Run
#1
Time to output
5 s
Submitted
1 Oct 2026

Needs more than a quick edit9 of 12 checks passed

The guide is strong and mostly usable, but it needs evidence fixes and a must-ask/cut plan before a PM could run it without re-checking assumptions.

Output

Autochase Call Guide: 8 calls, 30 minutes each

What we're trying to learn

Sam's hypothesis has three parts, and each can fail separately: 1. Cost: Late payment hurts in money or time, not just annoyance. (34% of invoices are late, but that doesn't mean it hurts.) 2. Behavior: People already try to fix it. Only 12% of accounts used the manual Send reminder button, so we need to know why the other 88% didn't. 3. Willingness: They'd let software message their clients, and pay $10 a month on top of $12 for Pro.

Interviews can't settle price. They can tell us whether it's worth testing.

Who to talk to

Sam's plan (the 300 heaviest button users, first 8 to reply) would only find people who already like reminders and respond fast to email. Recruit against quotas instead:

  • 2 heavy button users (Sam's list is fine for these)
  • 3 accounts with several late invoices in the last 90 days who never used the button
  • 1 who filed an Autochase support ticket
  • 2 agencies (3+ people) and 2 solo freelancers. These overlap with the groups above.
  • Mix of Pro and free accounts, with at least 2 free.

Screen for at least one invoice paid late in the last 90 days. Pick people to fill the quotas, not first come first served. Offer a $50 gift card.

Call structure

0:00–0:03 Intro. "We're learning how people get paid, and we're not selling anything. There are no right answers, and critical feedback helps most. Okay to record?"

0:03–0:05 Context. What do you do, how many clients, how many invoices a month?

0:05–0:15 The last late invoice (the core of the call) 1. Tell me about the last time a client paid you late. Probe: how late, how much, what happened first, what did you do, how did it feel? 2. Walk me through what you do between sending an invoice and getting paid. Probe: what tool, how often, who does it? 3. What did that lateness cost you? Probe: hours, cash flow, bills or contractors you had to delay.

0:15–0:21 Current behavior 4. (Button users) What made you use Send reminder? (Non-users) What do you do instead, or why not just let it go? 5. Is there a client you would never chase? Why? 6. Have you tried anything else, such as late fees, deposits, shorter terms or other tools? What happened?

0:21–0:27 Concept. Read this, don't demo it: "Suppose Paydeck automatically emailed your client at set points after the due date, getting firmer each time." 7. What's your first reaction? What worries you? 8. What would have to be true for you to switch it on? Probe: tone, control, per-client exceptions, seeing the emails first.

0:27–0:30 Money and wrap 9. What do you pay for today that helps you get paid or run your finances? 10. "We'd likely charge extra for this. Would you want early access when it's ready?" Note the exact response. Hesitation counts as data. 11. Anything I should have asked? Anyone else we should talk to?

Why Sam's questions changed

  • "How frustrating is chasing?" and "Would it be useful?" invite polite yes answers. Past-behavior questions don't.
  • "$5, $10 or $15?" anchors people on our options and measures nothing real. Questions 9–10 are better evidence.
  • "Would your clients be annoyed?" asks for a guess about someone else. Question 5 asks what they've actually avoided.
  • "What other features?" is off-topic. Drop it.

Notes for first-time interviewers

  • Split roles. One person asks and the other takes verbatim notes. Swap halfway through the 8 calls.
  • Ask about the past, not the future. "What did you do last time?" beats "Would you ever...?"
  • Don't pitch or rescue. Don't name Autochase until minute 21. If they struggle to answer, wait, then ask "Can you say more?" Silence is fine.
  • Follow the story. Ask "why?" and "what happened next?" more than you ask the next scripted question. The script is a backstop.
  • Discount compliments. "That sounds great" is worth little. Specific past actions, money spent and workarounds built are worth a lot.
  • Stay neutral if they criticize the product or the idea. Don't defend it.
  • Debrief within 30 minutes of each call. Write down three surprises and the best quotes. Do a practice run with a colleague first.

What we'd need to hear

Count only the people who had a late invoice.

Go (build, or run a paid pre-sale test): - At least 5 of 8 describe a specific recent late payment that cost real money, stress or more than about an hour a month. - At least 5 already chase manually or with workarounds, including people who skip the button, and can explain why the button doesn't work for them. - At least 4 would turn on automation given controls we could build, and at least 3 say yes to early access for a paid add-on.

Stop or rethink: - Most shrug lateness off, or say it comes from a few clients they'd never automate against. - The real problem is something else, such as clients not opening invoices, wanting deposits, or needing a payment link, and reminders wouldn't fix it. - Most won't let software contact their clients, or wouldn't pay anything for it.

Mixed results (for example, it only matters for agencies, or only with tone controls): narrow the target segment and run a second round of calls.

Even a clear "go" from 8 calls only justifies a cheap next test, such as a fake-door button or a pre-sale. It doesn't justify committing the quarter.

Claims the judge couldn’t find in the brief

  • There is an 'Autochase support ticket' population to recruit from.
  • Sam's plan would only find people who already like reminders and respond fast to email.
  • We can identify accounts with several late invoices in the last 90 days who never used the button.

Grades and run details

Decision model 96 · LLM judge 9 of 13 checks
Decision model checks
  • passUses the supplied evidence correctly26%
  • passAddresses the actual decision96%
  • passRespects explicit constraints38%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims68%
  • passProduces the required deliverable96%
  • passTalks to the right people99%
  • passCoaches first-time interviewers100%
  • passDesigned to falsify96%
  • passAsks what people did, not what they think77%
  • passDoesn't lead the witness61%
  • partialMarks what to cut if the call runs over61%
Run
Run
#1
API response time
32 s
Submitted
30 Sept 2026

Usable with a quick edit11 of 12 checks passed

The guide is complete and usable; a quick edit to add a note about cutting the check-understanding section if time is short would fix the minor omission.

Output

Autochase: 30-minute customer call guide

Interviewers: Mei leads; Tomas takes notes and asks brief follow-ups at the end. Keep roles consistent across the eight calls.

Decision: Should Paydeck prioritize automatic, escalating overdue-invoice reminders as a $10/month add-on next quarter?

1. Learning goals

Test whether: - Late payments create recurring, material costs—not just irritation. - Chasing takes enough time or money that customers actively try to reduce it. - Reminders address the cause of delays, rather than disputes, approval processes, or clients lacking funds. - Customers would trust automation, and what control they require. - There is credible demand at $10/month, beyond polite enthusiasm.

Starting evidence is suggestive, not conclusive: 34% of invoices are late, with a median delay of 16 days. That does not tell us how many accounts experience serious pain. Reminder usage (12% of accounts) may miss chasing outside Paydeck. The 40 support requests show interest, not willingness to pay.

2. Who to recruit

Do not book the first eight respondents from the 300 heaviest reminder users. That would favor highly engaged users already chasing through Paydeck.

Recruit purposively: - 3 accounts: recurring overdue invoices; frequent manual reminders. - 3 accounts: recurring overdue invoices; little or no manual reminder use. - 2 accounts: mostly on-time invoices, as a contrast group.

Across those groups, aim for six Pro and two free accounts, with both freelancers and small agencies. Include no more than two support-ticket requesters. Speak to whoever actually handles collections; establish whether they also approve software spending.

Randomly invite accounts within each group, then fill group slots rather than accepting the first eight overall. Replace no-shows within the same group. Use a neutral invitation: “Help us understand how you manage invoices and payments.” If offering an incentive, keep it fixed and unrelated to feedback.

3. Call script and timings

0–3 minutes: Welcome and permission

“Thanks for helping. We’re learning how people manage payments—not testing you or selling anything. Honest criticism is useful. You can skip anything confidential.”

Ask permission before recording. Otherwise, take notes.

“What’s your role, who handles overdue invoices, and roughly how many invoices do you send each month?”

3–12 minutes: Reconstruct a real incident

“Tell me about the most recent invoice that wasn’t paid by its due date.”

Follow the timeline: - “When was it due, and when did you notice?” - “What did you do next? Then what happened?” - “Who else was involved?” - “What reason did the client give for the delay?” - “Was it paid? When?”

“Could you walk us through a redacted reminder or your tracking process?” Do not request client-identifying information.

If they have no recent late invoice: “Walk us through how your last invoice got paid. What helps payments arrive on time?”

12–19 minutes: Frequency, consequences, alternatives

“In the last three months, about how often has this happened?”

“For that example, roughly how much time did you spend chasing? What, if anything, did the delay cost you?”

“What happens if you don’t follow up?”

“How do you decide when and how firmly to chase? Have you ever chosen not to?”

“What have you tried to improve this? What did it cost, and why did you keep or stop using it?”

If relevant: “Do you use Paydeck’s reminder button? Why or why not?”

19–23 minutes: Automation fit and risks

“Which parts of that process need your judgment?”

“Tell me about a follow-up that went badly—or worked particularly well.”

“What situations would make sending another reminder inappropriate?”

Probe only after open answers: payments already made, disputed invoices, important relationships, tone, timing, approvals.

23–28 minutes: Concept and price

Only now introduce the idea:

“We’re considering Autochase: automatic reminders that escalate while an invoice remains overdue. There’s no prototype, and details aren’t decided.”

“For the invoice we discussed, where would this help, fail, or create extra work?”

“What would you need to control before trusting it?”

“The proposed price is $10/month, on top of your current plan. How would you decide whether to buy it? What would it replace? Who approves that spending?”

Ask interested participants whether they will join a follow-up pilot discussion. Record this as interest, not a purchase.

28–30 minutes: Check understanding

Summarize their situation and ask what you misunderstood. Tomas asks outstanding clarifications. Thank them.

4. First-time interviewer notes

  • Ask one question at a time; allow silence.
  • Prefer “Tell me about the last time” to predictions.
  • Don’t defend Autochase, suggest answers, or praise positive reactions.
  • Separate quotes, observed behavior, estimates, and interpretation.
  • Debrief immediately: pain, frequency, cost, workaround, automation fit, price evidence, contradictions.

5. Decision rules

Agree these directional thresholds before interviewing:

Advance: At least four of six overdue-account participants demonstrate recurring, consequential pain and reminder-suitable workflows; at least three Pro participants show credible $10 value grounded in current costs or spending and accept a concrete follow-up. Proceed to a scoped pilot and real pricing test—not a full-build commitment from interviews alone.

Stop or rethink: Pain is mostly minor, delays are not reminder-solvable, relationship risks outweigh savings, or interest disappears at $10.

Mixed evidence: Narrow the target segment and recruit more. Eight purposive interviews cannot estimate market demand.

Grades and run details

Decision model 96 · LLM judge 12 of 13 checks
Decision model checks
  • passUses the supplied evidence correctly52%
  • passAddresses the actual decision95%
  • passRespects explicit constraints48%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims91%
  • passProduces the required deliverable93%
  • passTalks to the right people100%
  • passCoaches first-time interviewers90%
  • passDesigned to falsify87%
  • passAsks what people did, not what they think89%
  • passDoesn't lead the witness81%
  • partialMarks what to cut if the call runs over58%
Run
Run
#1
Time to output
38 s
Submitted
30 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 8

Uses the supplied evidence correctlyMixedMixedRight
Gemini 3.5 Flash-Lite · Gemini

It misstates the recruiting plan as Pro accounts and presents several unlabelled assumptions about users' pain, honesty, and preferences as current facts.

Sonnet 5.5 · API

It uses most supplied numbers correctly, but invents or overstates current-situation details such as an 'Autochase support ticket' population and treats recruiting-plan inferences as facts.

GPT-6 Astra · ChatGPT

All factual claims about the current situation are taken directly from the brief or supplied context.

Respects explicit constraintsMixedRightRight
Gemini 3.5 Flash-Lite · Gemini

It is a call guide for Mei and Tomas, includes the requested sections, has 30-minute timings, and appears under 900 words.

Sonnet 5.5 · API

It is a call guide under 900 words with learning goals, timed questions, interviewer notes, recruiting, and go/stop signals.

GPT-6 Astra · ChatGPT

The output is under 900 words, includes all requested sections, and respects the form and reader.

Identifies material uncertaintyMixedRightRight
Gemini 3.5 Flash-Lite · Gemini

It does not name the key uncertainty that heavy reminder users may overstate demand, nor does it bound the unknowns with behavior-based tests.

Sonnet 5.5 · API

It names material unknowns—cost, behavior, willingness, client-contact concerns—and says how calls plus a cheap test would resolve them.

GPT-6 Astra · ChatGPT

It names specific unknowns (e.g., whether pain is minor, delays not reminder-solvable) and says how the interviews will resolve them.

Avoids unsupported claimsWrongMixedRight
Gemini 3.5 Flash-Lite · Gemini

It presents unsupported claims such as 'these users already feel the pain', 'if they won't pay nobody will', and that people saying they would pay are often lying.

Sonnet 5.5 · API

It presents some interpretations as established, especially that Sam's list would only find reminder-likers/fast email responders and that Autochase tickets exist.

GPT-6 Astra · ChatGPT

Interpretations are clearly labelled as suggestive, and no unsupported claims are presented as established fact.

Talks to the right peopleWrongRightRight
Gemini 3.5 Flash-Lite · Gemini

It keeps Sam's plan of only the heaviest reminder users and does not explain that this overstates demand or add non-chasing overdue-invoice users.

Sonnet 5.5 · API

It explains the heaviest-button-user bias and gives a quota mix including non-button users with late invoices, ticket requesters, agencies, solos, and free accounts.

GPT-6 Astra · ChatGPT

It explicitly avoids only the heaviest reminder users, explains why, and balances with users who have overdue invoices but don't chase in Paydeck.

Designed to falsifyWrongRightRight
Gemini 3.5 Flash-Lite · Gemini

The main value test is a hypothetical price reaction after pitching Autochase, so the guide cannot cleanly falsify willingness to pay.

Sonnet 5.5 · API

Each learning goal has disconfirming questions about recent late payments, actual chasing behavior, client-contact limits, and past spending/early-access response.

GPT-6 Astra · ChatGPT

Every learning goal has disconfirming questions, such as reconstructing a real late invoice and asking what happens if they don't follow up.

Asks what people did, not what they thinkWrongRightRight
Gemini 3.5 Flash-Lite · Gemini

Although it asks about the last overdue invoice, the core decision evidence includes pitching the solution and asking price reactions, not only recent behavior.

Sonnet 5.5 · API

The core asks for the last late invoice, what happened, what they did, what it cost, and what they already tried.

GPT-6 Astra · ChatGPT

Core questions ask for the most recent overdue invoice and what they did, with opinions and predictions kept secondary and at the end.

Doesn't lead the witnessWrongRightRight
Gemini 3.5 Flash-Lite · Gemini

It describes and pitches Autochase before pricing and asks leading questions such as whether $10 feels like a bargain and whether automation fits their workflow.

Sonnet 5.5 · API

It removes Sam's leading/hypothetical questions, keeps the concept late, and avoids pitching before current-behavior questions.

GPT-6 Astra · ChatGPT

Questions are neutral and open; Autochase is introduced only after the problem exploration, with no pitching beforehand.

All got wrong 1

Marks what to cut if the call runs overWrongWrongWrong
Gemini 3.5 Flash-Lite · Gemini

The timings add to 30 minutes, but the guide has too many questions and no must-ask or cut-if-short instructions, making it unrealistic for first-time interviewers.

Sonnet 5.5 · API

Timings add to 30 minutes, but it does not mark must-ask questions or say what to cut if time runs short.

GPT-6 Astra · ChatGPT

The guide does not mark must-ask questions or say what to cut if time runs short, as required by the criterion.

All got right 3

Addresses the actual decisionRightRightRight
Gemini 3.5 Flash-Lite · Gemini

It gives a clear conditional go/no-go rule for whether to build Autochase next quarter, though the rule itself is weak.

Sonnet 5.5 · API

It commits to a clear call: 8 calls can justify only a cheap next test, not committing the quarter, and gives go/stop conditions.

GPT-6 Astra · ChatGPT

The guide commits to clear go/stop decision rules with thresholds and says what would change the answer.

Produces the required deliverableRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The requested call guide is present and usable as a draft, with learning goals, questions, timings, interviewer notes, recruiting, and go/stop criteria.

Sonnet 5.5 · API

Mei and Tomas could run the calls from it with light edits.

GPT-6 Astra · ChatGPT

The guide is a complete, usable call guide with learning goals, timed questions, interviewer notes, recruiting plan, and decision rules.

Coaches first-time interviewersRightRightRight
Gemini 3.5 Flash-Lite · Gemini

It gives practical first-timer coaching: don't pitch early, ask about past behavior, embrace silence, and split lead/notes roles.

Sonnet 5.5 · API

It gives specific first-timer instructions: role split, verbatim notes, silence, follow-ups, no pitching/defending, debriefs, and a practice run.

GPT-6 Astra · ChatGPT

It gives specific, usable instructions: ask one question at a time, allow silence, prefer past-behaviour questions, don't defend or pitch, and separate notes.

Results

Every setup we’ve tested on this task, across all cases and repeats, graded on the current checklist. Calibrated: the graders match our PM on 98% of checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6.1 SolwithAPI94.192.62None
2Sonnet 5.5withAPI90.284.62None
3GPT-6 AstrawithChatGPT96.078.32None
4Opus 5.5withClaude92.163.22None
5Gemini 3.8 FlashwithAPI80.167.32None
6GPT-6 LunawithAPI92.073.921 capped
7Gemini 3.5 Flash-LitewithGemini53.826.122 capped

About the task

The PM job

Preparing for customer calls.

Why it matters

Leading questions produce the answers you wanted. A good guide is designed to prove you wrong.

What good looks like

  • Past-behaviour questions
  • No leading prompts
  • Clear learning goals
  • Could disprove the team's hypothesis

Deliberately not measured

    Capability tested

    Question design

    The failure we’re looking for

    Hypothetical, leading questions

    Grading

    Decision model and LLM judge, calibrated against a blind PM review