Tasks / Discover

Extract discovery insights

Can the model separate evidence, themes and hypotheses without inventing consensus?

Measures the modelTask v1.1 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 12 graded outputs by 6 models. 58% were usable with at most a quick edit.

Reliably right

  1. Respects explicit constraints100% pass
    It is a findings summary for the product team and is under the 600-word limit.
    GPT-6 Astra · ChatGPT · Eight calls with finance teams
  2. Produces the required deliverable100% pass
    The summary is complete, usable, and would let a PM act with only light edits.
    GPT-6 Astra · ChatGPT · Eight calls with finance teams
  3. Keeps dissent visible100% pass
    The SaaS and charity close-is-fine views and the manufacturing CFO's switching regret are kept visible.
    GPT-6 Astra · ChatGPT · Eight calls with finance teams

Where it slips

  1. Says how many sources support each finding52% pass
    Findings reference call numbers but do not say how many of the eight support each theme, and it uses an unsupported universal.
    Gemini 3.5 Flash-Lite · Gemini · Eight calls with finance teams
  2. Avoids unsupported claims56% pass
    The bottom line asserts 'Adoption will depend on handling messy data and reducing switching effort' as an established forecast, though the evidence only suggests it.
    GPT-6.1 Sol · API · Eight calls with finance teams
  3. Uses the supplied evidence correctly65% pass
    The output states the calls happened in August 2026, but the supplied context only says August without a year; this date is an invented fact not supported by the brief.
    Opus 5.5 · Claude · Eight calls with finance teams

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

Synthesise the eight discovery calls below with finance leads about month-end close. Write a findings summary for the product team, who are deciding whether to build a reconciliation product: what we learned, and how confident we can be in it. Keep it under 600 words.

ScenarioWe make spend-management software for mid-sized companies and are exploring a reconciliation product. Our PM ran these 40-minute calls in August. Below are the relevant excerpts from each, lightly edited.
Call 1: Financial controller, logistics company (180 staff)“Close takes us eight working days and at least three of those are reconciliation. The bank feed drops transactions, so we tick and bash against statements in Excel.” Asked what she'd pay to fix it: “If it gave me two days back, it pays for itself. But I've heard that before.”
Call 2: Head of finance, SaaS company (95 staff)“Honestly close is fine. We moved to a proper ERP two years ago, it's five days and nobody's complaining.” Later: “The only annoying bit is chasing people for receipts, not reconciling.”
Call 3: Finance manager, retail chain (400 staff)“Card transactions are the nightmare. Hundreds of store cards, the statements come in as PDFs, someone re-keys them. We found £14k of duplicates last quarter.” She reconciles card spend in a shared spreadsheet with four people editing it.
Call 4: CFO, manufacturing company (250 staff)“We switched reconciliation tools last year and I regret it. The migration took four months, we lost all our matching rules, and half the team quietly went back to spreadsheets.” Asked if he'd switch again: “Not unless someone else does the migration for us. Never again otherwise.”
Call 5: Accountant, agency group (120 staff)“Intercompany is where it breaks. Three entities, money moving between them every week, and the balances never agree at month end. I spend a day and a half on it.”
Call 6: Finance director, charity (70 staff)“Close is fine for us. We're small, two of us, it takes four days and we know every transaction.” When asked about tools: “We'd not pay for anything new this year; budgets are frozen.”
Call 7: Controller, hospitality group (300 staff)“Payment processor payouts are the worst: they net off fees and refunds, so nothing matches the bank line. We reconcile by hand and it's always the last thing holding up close.”
Call 8: Financial controller, construction firm (210 staff)“Reconciliation is about 40% of my close. It's not hard, it's just volume, matching the same things every month.” On automation: “I'd want to see it match our messy data before I believed it. Every demo uses clean data.”
What a strong answer does

Themes with honest counts (five of eight describe reconciliation pain, from different sources), the two who say close is fine and the CFO's switching regret kept visible, and hypotheses about willingness to pay and switching cost labelled as such.

Critical failures (cap the score)
  • Invents a quote
Case

v1.7 · anonymised real · B2B, finance

Usable with a quick edit8 of 9 checks passed

The output is evidence-backed and actionable as-is. A PM would only need to add an explicit build trigger, which is a light edit.

Output

Month-end close discovery: findings summary

Bottom line

Reconciliation is a real, recurring pain for mid-sized finance teams, but these eight calls don't yet show which product to build or whether anyone will pay for it. I'd treat this as grounds for a further, more targeted round of research, not a build decision.

What we learned

1. Five of eight leads have a real reconciliation problem (calls 1, 3, 5, 7, 8). Two said close is fine (calls 2 and 6), and one has a tool and regrets it (call 4).

2. They describe different problems, not one. - Bank feeds dropping transactions, matched by hand in Excel (call 1) - Card statements arriving as PDFs and re-keyed, with £14k of duplicates in a quarter (call 3) - Intercompany balances that never agree, costing about 1.5 days (call 5) - Payment processor payouts that net off fees and refunds, and are always the last thing holding up close (call 7) - Sheer matching volume, about 40% of one controller's close (call 8)

No problem appears in more than one call, so we can't yet say which is most common or most valuable.

3. The time cost is meaningful but self-reported. Estimates ranged from 1.5 days to 3 of 8 days. These are recollections, not measurements.

4. Those who are fine tend to be smaller or better equipped. The two "close is fine" leads are the smallest companies in the sample (70 and 95 staff), and one already has a modern ERP. This is suggestive at best with eight calls. The pain sits mostly in the 180–400 staff range.

5. Adoption barriers are as prominent as the pain. - Call 4's failed migration (four months, lost matching rules, team back on spreadsheets) is a warning about switching costs. He would only switch again if someone else did the migration. - Call 8 won't believe automation works until he sees it handle messy data: "Every demo uses clean data." - Call 1 is sceptical: "I've heard that before."

6. Willingness to pay is essentially untested. Only call 1 gave a conditional answer ("if it gave me two days back, it pays for itself"), and she hedged it. Call 6 has a budget freeze. No one expressed urgency or asked about pricing.

7. An adjacent finding: call 2's only complaint was chasing receipts, which sits closer to our existing spend-management product than reconciliation does.

How confident can we be?

ClaimConfidence
Reconciliation is a meaningful time sink for some mid-sized teamsModerate
Pain varies by reconciliation typeModerate to high (consistent across calls)
Size of the time savingLow (self-reported, varied definitions)
Willingness to payLow (one hedged signal)
Which segment or use case to targetLow
Smaller or ERP-equipped firms are a weaker fitLow (three data points)

Other limitations: - Eight calls is a small sample, and we don't know how participants were recruited, so existing customers or those already interested may be overrepresented. - The excerpts are edited and one interviewer ran all the calls. - Hypothetical questions ("would you pay?") overstate real intent.

Suggested next steps

  1. Pick a wedge. Card reconciliation (call 3) is closest to our current product. Follow up with calls 3, 5 and 7 to size how often each problem occurs and what it costs.
  2. Test with real data. Ask call 8 and similar leads for anonymised, messy samples to see whether matching is feasible.
  3. Design for migration. Call 4 suggests onboarding and preserving matching rules may decide adoption. Treat this as a product requirement, not an afterthought.
  4. Test price and urgency. Use concrete pricing or a paid pilot, not hypotheticals.
  5. Widen the sample. Include firms of 150–500 staff and a mix of ERP and non-ERP setups, and record how each participant was recruited.

Check by check

Mixed · 1
  • Addresses the actual decisionIt recommends more research but does not state what specific result or condition would trigger a build decision.The two graders disagreed on this one.
Got right · 8
  • Uses the supplied evidence correctlyAll factual statements trace to the supplied calls, with no invented current-situation facts.
  • Respects explicit constraintsThe output is a findings summary for the product team and is under the 600-word limit.
  • Identifies material uncertaintyIt names unknown WTP, segment, and time savings, and says how to resolve them in next steps.
  • Avoids unsupported claimsHypotheses and limitations are labelled as suggestive or untested, not as established fact.
  • Produces the required deliverableThe summary is complete, readable, and directly usable by the product team.
  • Keeps dissent visibleThe two close-is-fine calls and the CFO's switching regret are kept visible throughout.
  • Weights behaviour over opinionIt flags self-reported recollections and hypothetical pricing as weaker than observed behaviours and workarounds.
  • Says how many sources support each findingFindings consistently include source counts ('five of eight', 'two said', 'call 4').

Grades and run details

Decision model 100 · LLM judge 9 of 10 checks
Decision model checks
  • passUses the supplied evidence correctly60%
  • passAddresses the actual decision82%
  • passRespects explicit constraints44%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims35%
  • passProduces the required deliverable72%
  • passKeeps dissent visible99%
  • passWeights behaviour over opinion97%
  • passSays how many sources support each finding61%
Run
Run
#1
API response time
18 s
Submitted
29 Sept 2026

Usable with a quick edit7 of 9 checks passed

The only error is a stray year in the title; the rest of the reasoning, evidence, and structure is solid and usable immediately after deleting '2026'.

Output

Reconciliation discovery: findings from 8 finance-lead calls (August 2026)3

Bottom line: Most of the finance leads we spoke to have real reconciliation pain. But it shows up as several different problems, not one, and the biggest barrier to adoption is distrust and switching cost. We have almost no evidence on willingness to pay. This justifies more targeted discovery. It doesn't yet justify a decision to build.

What we learned

  1. The pain is common but not universal. Six of eight say reconciliation is a significant part of close (Calls 1, 3, 4, 5, 7, 8).1 The two who said close is fine (2, 6) run the smallest companies, at 95 and 70 staff, and one of them has a modern ERP. That hints at a size threshold around 100+ staff, but it rests on two data points.

2. Where people gave numbers, the cost is large. - About 3 of 8 close days (1) - 1.5 days a month (5) - About 40% of close (8) - £14k of duplicates found in one quarter (3)

All of these are self-reported.

3. "Reconciliation" is at least five separate problems. Each call with pain named a different source: - Bank feed dropping transactions (1) - Card statements arriving as PDFs and re-keyed into a shared spreadsheet (3) - Intercompany balances that don't agree across entities (5) - Payment-processor payouts that net off fees and refunds, so nothing matches the bank line (7) - Sheer volume of repetitive matching (8)

No two calls named the same source. One "reconciliation product" would have to handle different matching logic for different segments. Card spend (3) sits closest to what we already do and could be a natural starting point, but it came up in only one call.

4. Switching cost and skepticism are the main barrier.2 - Call 4 switched reconciliation tools last year. Migration took four months, they lost their matching rules, and half the team drifted back to spreadsheets. He won't switch again unless someone else does the migration. - Call 8 won't believe a tool works until it handles their messy data, because demos always use clean data. - Call 1 has "heard that before."

That's three of eight, including people with heavy pain. Done-for-you migration and proof on the customer's own data look like requirements, not differentiators.

  1. One adjacent signal. Call 2's only complaint was chasing people for receipts. That's our existing spend-management territory, not reconciliation.

How confident can we be?

Moderate confidence: - Reconciliation takes a meaningful share of close time for mid-sized finance teams. - The underlying causes vary widely between companies. - Buyers are wary of switching tools and of vendor demos.

Low or no confidence: - Willingness to pay. Only one person answered the question, and only conditionally. One has a budget freeze. - Which problem to build for. Each pain source appears exactly once. - Size threshold. It rests on two calls. - Prevalence. We don't know how these eight were recruited, whether they're customers, or how representative they are.

Method limitations: - Eight calls, one interviewer, one month. - We're working from excerpts chosen as "relevant," so the full transcripts may shift the emphasis. - The pay question was open-ended and hypothetical.

What would raise confidence before a build decision

  • Recruit 8–10 more finance leads at companies with 150–400 staff. Screen them by pain source so we learn which problem repeats.
  • Test the card-spend starting point with existing customers. Ask how they reconcile card spend today. We may already hold the data needed to solve it.
  • Test willingness to pay concretely. Use price anchors or offer a paid pilot rather than asking an open question.
  • Run a matching test on one prospect's real, messy data. Call 8 set that bar, and it's the test that would convince the skeptics.

What a PM had to fix

  1. 1Numbers wrongRedo the arithmeticTargeted repair

    Source: Call 4: CFO, manufacturing company (250 staff)We switched reconciliation tools last year and I regret it.

    What we’d changeRecount without Call 4, which is about a failed tool switch, not close time. It's five of eight.

  2. 2Hypothesis stated as factReframe it as a hypothesisQuick edit

    What we’d changePresent it as a concern three people raised. Eight calls can't rank the barriers.

  3. 3Constraint missedRestore the constraintQuick edit

    Source: BriefKeep it under 600 words.

    What we’d changeTrim it: it runs past the 600-word limit. And drop the year, which the brief doesn't give.

Check by check

Got wrong · 2
  • Uses the supplied evidence correctlyThe output states the calls happened in August 2026, but the supplied context only says August without a year; this date is an invented fact not supported by the brief.
  • Avoids unsupported claimsThe date 'August 2026' is presented as a confident fact without any basis in the supplied context.
Got right · 7
  • Addresses the actual decisionThe output clearly recommends that the evidence doesn't yet justify building and that more targeted discovery is needed, and states what would change that call.
  • Respects explicit constraintsThe output is a findings summary under 600 words for the product team, as requested.
  • Identifies material uncertaintyIt names unknown willingness to pay, problem to build for, size threshold, and recruitment bias, and how to resolve them.
  • Produces the required deliverableThe deliverable is a complete findings summary that a product team could act on with minimal edits.
  • Keeps dissent visibleCalls 2 and 6 who said close is fine are explicitly mentioned and kept in view.
  • Weights behaviour over opinionThe output flags self-reported numbers and hypothetical pay answers, and weights the actual migration experience (an observed behaviour) heavily.
  • Says how many sources support each findingEvery finding is sized by number of calls (e.g., 'six of eight', 'three of eight', 'only one call').

Claims the judge couldn’t find in the brief

  • The discovery calls took place in August 2026.

Grades and run details

Decision model 83 · LLM judge 8 of 10 checks
Decision model checks
  • failUses the supplied evidence correctly22%
  • passAddresses the actual decision95%
  • passRespects explicit constraints22%
  • passIdentifies material uncertainty100%
  • partialAvoids unsupported claims18%
  • passProduces the required deliverable45%
  • passKeeps dissent visible49%
  • passWeights behaviour over opinion87%
  • passSays how many sources support each finding27%
Run
Run
#1
Time to output
35 s
Submitted
24 Sept 2026

Results

Every setup we’ve tested on this task, across all cases and repeats, graded on the current checklist. Calibrated: the graders match our PM on 85% of checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT97.295.02None
2GPT-6.1 SolwithAPI94.490.02None
3GPT-6 LunawithAPI88.980.02None
4Sonnet 5.5withAPI94.465.02None
5Opus 5.5withClaude75.070.02None
6Gemini 3.5 Flash-LitewithGemini52.830.02None

About the task

The PM job

Turning a stack of call transcripts into what we actually learned.

Why it matters

Synthesis is where teams fool themselves. A model that smooths away dissent or turns one loud customer into a trend produces confident, wrong roadmaps.

What good looks like

  • Quotes evidence for each theme and counts sources honestly
  • Weights what customers did above what they say they'd do
  • Keeps important dissent visible
  • Labels hypotheses as hypotheses
  • Says what the research cannot tell us

Deliberately not measured

  • Transcript clean-up
  • Persona illustration
Capability tested

Faithful synthesis of qualitative research

The failure we’re looking for

Invents customer consensus or loses important dissent

Grading

Decision model and LLM judge, calibrated against a blind PM review