Tasks / Discover

Extract discovery insights

Can the model separate evidence, themes and hypotheses without inventing consensus?

Measures the modelTask v1.1 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 12 graded outputs by 6 models. 58% were usable with at most a quick edit.

Reliably right

  1. Respects explicit constraints100% pass
    It is a findings summary for the product team and is under the 600-word limit.
    GPT-6 Astra · ChatGPT · Eight calls with finance teams
  2. Produces the required deliverable100% pass
    The summary is complete, usable, and would let a PM act with only light edits.
    GPT-6 Astra · ChatGPT · Eight calls with finance teams
  3. Keeps dissent visible100% pass
    The SaaS and charity close-is-fine views and the manufacturing CFO's switching regret are kept visible.
    GPT-6 Astra · ChatGPT · Eight calls with finance teams

Where it slips

  1. Says how many sources support each finding52% pass
    Findings reference call numbers but do not say how many of the eight support each theme, and it uses an unsupported universal.
    Gemini 3.5 Flash-Lite · Gemini · Eight calls with finance teams
  2. Avoids unsupported claims56% pass
    The bottom line asserts 'Adoption will depend on handling messy data and reducing switching effort' as an established forecast, though the evidence only suggests it.
    GPT-6.1 Sol · API · Eight calls with finance teams
  3. Uses the supplied evidence correctly65% pass
    The output states the calls happened in August 2026, but the supplied context only says August without a year; this date is an invented fact not supported by the brief.
    Opus 5.5 · Claude · Eight calls with finance teams

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

Synthesise the eight discovery calls below with finance leads about month-end close. Write a findings summary for the product team, who are deciding whether to build a reconciliation product: what we learned, and how confident we can be in it. Keep it under 600 words.

ScenarioWe make spend-management software for mid-sized companies and are exploring a reconciliation product. Our PM ran these 40-minute calls in August. Below are the relevant excerpts from each, lightly edited.
Call 1: Financial controller, logistics company (180 staff)“Close takes us eight working days and at least three of those are reconciliation. The bank feed drops transactions, so we tick and bash against statements in Excel.” Asked what she'd pay to fix it: “If it gave me two days back, it pays for itself. But I've heard that before.”
Call 2: Head of finance, SaaS company (95 staff)“Honestly close is fine. We moved to a proper ERP two years ago, it's five days and nobody's complaining.” Later: “The only annoying bit is chasing people for receipts, not reconciling.”
Call 3: Finance manager, retail chain (400 staff)“Card transactions are the nightmare. Hundreds of store cards, the statements come in as PDFs, someone re-keys them. We found £14k of duplicates last quarter.” She reconciles card spend in a shared spreadsheet with four people editing it.
Call 4: CFO, manufacturing company (250 staff)“We switched reconciliation tools last year and I regret it. The migration took four months, we lost all our matching rules, and half the team quietly went back to spreadsheets.” Asked if he'd switch again: “Not unless someone else does the migration for us. Never again otherwise.”
Call 5: Accountant, agency group (120 staff)“Intercompany is where it breaks. Three entities, money moving between them every week, and the balances never agree at month end. I spend a day and a half on it.”
Call 6: Finance director, charity (70 staff)“Close is fine for us. We're small, two of us, it takes four days and we know every transaction.” When asked about tools: “We'd not pay for anything new this year; budgets are frozen.”
Call 7: Controller, hospitality group (300 staff)“Payment processor payouts are the worst: they net off fees and refunds, so nothing matches the bank line. We reconcile by hand and it's always the last thing holding up close.”
Call 8: Financial controller, construction firm (210 staff)“Reconciliation is about 40% of my close. It's not hard, it's just volume, matching the same things every month.” On automation: “I'd want to see it match our messy data before I believed it. Every demo uses clean data.”
What a strong answer does

Themes with honest counts (five of eight describe reconciliation pain, from different sources), the two who say close is fine and the CFO's switching regret kept visible, and hypotheses about willingness to pay and switching cost labelled as such.

Critical failures (cap the score)
  • Invents a quote
Case

v1.7 · anonymised real · B2B, finance

Usable with a quick edit9 of 9 checks passed

The output is evidence-accurate, well-structured, and actionable with no need to re-check facts or redo reasoning.

Output

Findings summary

The pain is real, but it is not universal—and it is concentrated in specific reconciliation workflows. Five of the eight interviewees described reconciliation as a meaningful close burden. Their problems were concrete: dropped bank-feed transactions and manual matching (Call 1); high-volume card statements re-keyed from PDFs (Call 3); intercompany balances (Call 5); processor payouts obscured by fees and refunds (Call 7); and repetitive matching at scale (Call 8). One reported finding £14k in duplicate transactions. These examples suggest the opportunity may be less “speed up month-end close” and more “handle particular messy, high-volume matching jobs reliably.”

Company context seems to matter.1 The clearest pain came from organisations with volume, multiple entities, many cards, or complex payment flows. Two interviewees said close was fine: one had a well-established ERP and mainly struggled with receipts; another was small, knew every transaction, and had frozen budgets. A reconciliation product is therefore unlikely to be equally valuable to every mid-sized company.

Trust and implementation are central adoption barriers. One finance leader regretted a tool switch after a four-month migration, lost matching rules, and a return to spreadsheets. Another would not trust automation until it worked on messy real data. A product that matches well in a demo may still fail if it cannot preserve existing rules, handle exceptions, and prove accuracy on customers’ own data.

Willingness to pay is unproven. One controller said saving two days could justify the cost, but explicitly expressed scepticism based on past promises. Another prospect would not buy this year. These calls reveal value hypotheses, not validated demand or pricing.

Implications for the product team

Explore a narrow, workflow-led wedge rather than assuming a broad reconciliation platform is needed. Card-spend reconciliation may fit our existing product particularly well; processor payouts and intercompany are other distinct opportunities, but may require different data and capabilities. Test which segment has both acute pain and access to usable data. Prioritise messy-data performance, explainable matches and exceptions, and low-friction migration or coexistence with spreadsheets.

Confidence

  • Moderate confidence that reconciliation is a significant pain for some complex, high-volume finance teams: several independent, specific examples support this.
  • Low-to-moderate confidence on which workflow or customer segment is the best product wedge: each use case appears in few calls.
  • Low confidence on market-wide prevalence, willingness to pay, achievable time savings, or product feasibility: eight qualitative calls cannot establish these, and no prototype or pricing was tested.

Treat this as a reason to run targeted workflow and data-validation research—not yet as evidence to commit to a broad build.

What a PM had to fix

  1. 1Invented evidenceVerify or remove the claimTargeted repair

    What we’d changeCompany context matters.

Check by check

Got right · 9
  • Uses the supplied evidence correctlyAll factual claims about the current situation trace directly to specific calls in the supplied context with no invented facts.
  • Addresses the actual decisionThe output commits to not committing to a broad build now and running targeted validation, and it says what to test (segment with acute pain and usable data).
  • Respects explicit constraintsThe output is under 600 words, addressed to the product team, and in the requested findings-summary form.
  • Identifies material uncertaintyIt names the key unknowns (willingness to pay, time savings, workflow/segment, prevalence, feasibility) and says targeted workflow and data-validation research would resolve them.
  • Avoids unsupported claimsHypotheses and interpretations are hedged with 'suggest', 'may', 'seems', 'unlikely', and 'unproven'; no unsupported confident causal claims are presented as fact.
  • Produces the required deliverableThe findings summary is complete, actionable, and usable for the product team without major gaps.
  • Keeps dissent visibleThe dissenting interviews (Calls 2 and 6, and the switch-regret in Call 4) are explicitly included.
  • Weights behaviour over opinionIt highlights actual behaviour (Excel tick-and-bash, PDF re-keying, shared spreadsheet, migration, manual processor reconciliation) and flags willingness to pay as unproven stated intent.
  • Says how many sources support each findingFindings are sized with 'five of eight', 'two interviewees', 'one finance leader', and 'another' rather than generalising to everyone.

Grades and run details

Decision model 100 · LLM judge 10 of 10 checks
Decision model checks
  • passUses the supplied evidence correctly65%
  • passAddresses the actual decision46%
  • passRespects explicit constraints45%
  • passIdentifies material uncertainty98%
  • passAvoids unsupported claims39%
  • passProduces the required deliverable85%
  • passKeeps dissent visible90%
  • passWeights behaviour over opinion94%
  • passSays how many sources support each finding77%
Run
Run
#1
API response time
11 s
Submitted
29 Sept 2026

Needs more than a quick edit3 of 9 checks passed

The output contains a false universal claim and lacks source counts per theme. A PM would need to re-check the call evidence before using it.

Output

Product Findings: Month-End Close & Reconciliation Opportunities

We spoke with eight finance leads at mid-sized companies (70–400 staff) to evaluate appetite for a reconciliation product. Here is what we learned, followed by our confidence assessment.

---

What We Learned

#### 1. Pain is Acute, but Fragmented by Niche Reconciliation is universally recognized as a bottleneck1, but the root cause varies heavily by business model: * Retail, Hospitality & Construction (High Volume/Messy Data): Companies dealing with multiple store cards (Call 3), complex payment processor payouts (Call 7), or high-volume line items (Call 8) spend up to 40% of their close cycle manually matching data in shared spreadsheets or re-keying PDF statements.3 * Agencies & Multi-Entity (Intercompany): Multi-entity structures suffer from recurring intercompany balancing discrepancies that defy easy automation4 (Call 5). * Logistics (Basic Bank Feeds): Basic bank feed drops force manual tick-and-bash routines in Excel (Call 1).

#### 2. Adoption Barriers and Skepticism Run High Even where pain is severe, market entry is fraught with buyer resistance: * Migration PTSD: Switching costs are massive. One CFO noted a four-month migration failure that destroyed custom matching rules and drove staff back to spreadsheets, refusing to switch again unless migration is fully managed (Call 4). * Deep Trust Deficit: Finance leads have been burned by over-promised automation and pristine demo environments. They demand proof on their own messy data before believing claims (Calls 1 & 8). * Budget Freezes & Apathy: Smaller teams (70–95 staff) with clean ERP setups or frozen budgets report no pressing need for new point solutions (Calls 2 & 6).

---

Confidence Assessment: Low-Moderate (Conditional)

  • Problem Validation (High Confidence): We are highly confident that reconciliation is tedious, manual, and a major driver of close delays in high-volume, multi-card, or multi-entity businesses.
  • Commercial Viability (Low-Moderate Confidence): Despite the pain, we cannot rely on a self-serve, software-only motion.2 High switching friction, deep skepticism toward AI/automation demos, and strict migration/implementation requirements mean buyers will churn or bounce without hands-on onboarding.

#### Recommendation Do not build a generic, self-serve reconciliation tool. If we proceed, the product must include a heavy-touch onboarding/migration service to overcome migration PTSD, alongside a "sandbox" feature allowing prospects to test matching rules on their actual messy data before purchasing.

What a PM had to fix

  1. 1Invented evidenceVerify or remove the claimSubstantial rework

    Source: Call 2: Head of finance, SaaS company (95 staff)Honestly close is fine.

    What we’d changeSay most, not all: two of eight say close is fine. Keep that dissent visible rather than folding it into 'apathy'.

  2. 2Hypothesis stated as factReframe it as a hypothesisTargeted repair

    What we’d changeScale the conclusion back to what eight calls show: adoption concerns from a few leads, not proof a self-serve product can't work. Give a confidence level per claim.

  3. 3Numbers wrongRedo the arithmeticTargeted repair

    Source: Call 8: Financial controller, construction firm (210 staff)Reconciliation is about 40% of my close.

    What we’d changeOnly Call 8 gives the 40% figure. Don't spread it across retail and hospitality.

  4. 4Hypothesis stated as factReframe it as a hypothesisQuick edit

    Source: Call 5: Accountant, agency group (120 staff)Intercompany is where it breaks.

    What we’d changeNothing in Call 5 says it defies automation. Report the day and a half it takes instead.

Check by check

Got wrong · 6
  • Uses the supplied evidence correctlyIt asserts reconciliation is 'universally recognized as a bottleneck' even though Calls 2 and 6 describe close as fine, and it overstates Call 1/8 as 'burned by over-promised automation'.
  • Addresses the actual decisionThe call is only made at the end and does not state what result or condition would change it.
  • Identifies material uncertaintyIt labels confidence low-moderate but does not name specific unknowns or how they would be resolved.
  • Avoids unsupported claimsIt presents 'universally' and 'burned by over-promised automation' as established fact rather than as hypotheses.
  • Weights behaviour over opinionIt does not label themes by evidence type or flag stated preferences, such as Call 4's refusal to switch, as weaker than observed workarounds.
  • Says how many sources support each findingFindings reference call numbers but do not say how many of the eight support each theme, and it uses an unsupported universal.
Got right · 3
  • Respects explicit constraintsIt delivers a findings summary under 600 words for the product team, with no apparent violation of stated constraints.
  • Produces the required deliverableThe required summary, confidence assessment and recommendation are present and readable, though some claims need correction.
  • Keeps dissent visibleIt keeps Call 2, Call 6, and Call 4's switching regret visible.

Claims the judge couldn’t find in the brief

  • Reconciliation is universally recognized as a bottleneck.
  • Finance leads have been burned by over-promised automation and pristine demo environments, and demand proof on their own messy data (Calls 1 & 8).

Grades and run details

Decision model 56 · LLM judge 3 of 10 checks
Decision model checks
  • failUses the supplied evidence correctly29%
  • partialAddresses the actual decision36%
  • passRespects explicit constraints24%
  • partialIdentifies material uncertainty43%
  • failAvoids unsupported claims65%
  • passProduces the required deliverable58%
  • passKeeps dissent visible85%
  • partialWeights behaviour over opinion21%
  • partialSays how many sources support each finding46%
Artefacts
Run
Run
#1
Time to output
7 s
Submitted
24 Sept 2026

Results

Every setup we’ve tested on this task, across all cases and repeats, graded on the current checklist. Calibrated: the graders match our PM on 85% of checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT97.295.02None
2GPT-6.1 SolwithAPI94.490.02None
3GPT-6 LunawithAPI88.980.02None
4Sonnet 5.5withAPI94.465.02None
5Opus 5.5withClaude75.070.02None
6Gemini 3.5 Flash-LitewithGemini52.830.02None

About the task

The PM job

Turning a stack of call transcripts into what we actually learned.

Why it matters

Synthesis is where teams fool themselves. A model that smooths away dissent or turns one loud customer into a trend produces confident, wrong roadmaps.

What good looks like

  • Quotes evidence for each theme and counts sources honestly
  • Weights what customers did above what they say they'd do
  • Keeps important dissent visible
  • Labels hypotheses as hypotheses
  • Says what the research cannot tell us

Deliberately not measured

  • Transcript clean-up
  • Persona illustration
Capability tested

Faithful synthesis of qualitative research

The failure we’re looking for

Invents customer consensus or loses important dissent

Grading

Decision model and LLM judge, calibrated against a blind PM review