Tasks / Product Craft

Extract discovery insights

Can the model separate evidence, themes and hypotheses without inventing consensus?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 50% were usable with at most a quick edit.

Reliably right

  1. Frames findings for the decision100% pass
    Each main finding is framed as an opportunity (delivery-to-invoice pain, concentrated supplier need) and linked directly to what to build next quarter.
    GPT-6 Astra · ChatGPT · The loudest ticket
  2. Corrects the draft's claim100% pass
    It corrects the draft's headline by counting interviews (8 describe delivery/invoice pain, 2 mention supplier expansion) and showing the marketplace claim is unsupported.
    GPT-6 Astra · ChatGPT · The loudest ticket
  3. Keeps dissent visible100% pass

Where it slips

  1. Says how many sources support each finding52% pass
    The finding on smaller issues sizes Ordering as 31 tickets, but the actual count is 30, so the finding is not accurately sized.
    Sonnet 5.5 · API · The loudest ticket
  2. Uses the supplied evidence correctly59% pass
    The output states Ordering has 31 tickets, but the CSV contains 30 Ordering tickets; this factual claim is not supported by the supplied data.
    GPT-6.1 Sol · API · The loudest ticket
  3. Avoids unsupported claims63% pass
    The claim 'Ordering (31 tickets)' is presented as an established fact, but the evidence shows 30 Ordering tickets, so it is an unsupported confident claim.
    Sonnet 5.5 · API · The loudest ticket

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Weights behaviour over opinion

    Does the synthesis distinguish evidence of what customers actually did (specific past instances, observed behaviour, workarounds) from stated preferences, predictions and feature requests, and weight them accordingly?

    Passes when Themes are labelled or weighted by evidence type, and at least one theme resting mainly on stated intent or requests is flagged as weaker.

  2. Frames findings for the decision

    Does the synthesis turn its findings into opportunities tied to the decision or outcome the team faces, saying what each one means for that decision?

    Passes when Each main finding is framed as an opportunity (a need or pain to address) and linked to the decision at hand, with what it implies.

  3. Reports what cuts against the plan

    Does the synthesis report the evidence that cuts against the team's current belief or plan, rather than only what supports it?

    Passes when Names at least one finding that challenges the team's hypothesis or plan, and says what it means for it.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Says how many sources support each finding · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for Larder's ordering product. A colleague has drafted the synthesis below for our CPO, recommending a supplier marketplace. Before it goes, rewrite the synthesis from the evidence, in no more than 800 words: the main findings, how strong the evidence for each is, and what they mean for what we build next quarter. The interviews and the ticket export are attached.

What the model was given4 items: About Larder, draft_synthesis.md, interview_excerpts.md, support_tickets_last_quarter.csv
About LarderOrdering software for restaurants: order from your suppliers in one place, track deliveries and invoices. 1,140 accounts, most of them independent restaurants.
draft_synthesis.md6 lines · Download
# Draft synthesis (PM, for the CPO)

Headline: customers want a supplier marketplace.
- "Add more suppliers" is our top support category by far.
- Most customers we spoke to want a marketplace (see Calder Group and the pizzeria).
- Recommendation: build a supplier marketplace next quarter.
interview_excerpts.md12 lines · Download
# Interview excerpts (restaurant owners and managers, September)

1. Bistro, 2 sites: "Last Tuesday the fish order came short again. I only found out when the invoice came, a week later. I spent Sunday night matching delivery notes to invoices with a highlighter."
2. Café group, 5 sites: "We switched produce supplier twice this year, but finding suppliers isn't the problem. Knowing what actually arrived is."
3. Pizzeria, 1 site: "Honestly I'd love more suppliers in the app. More choice, better prices."
4. Gastropub: "My head chef signs the delivery note without checking. Then I pay for things we never got. Last month that was about £600."
5. Hotel restaurant: "Our finance team chases credit notes for short deliveries every week. It takes days."
…
support_tickets_last_quarter.csv200 rows · Download
ticket,account,account_sites,category,summary
LT-5000,Calder Group,14,Add more suppliers,Can you onboard our meat supplier
LT-5001,Account 1886,3,Ordering,Change delivery day
LT-5002,Account 2004,1,Delivery didn't match order,Short delivery not shown in app
LT-5003,Calder Group,14,Add more suppliers,Add dairy supplier please
LT-5004,Calder Group,14,Add more suppliers,Add dairy supplier please
LT-5005,Account 1858,2,Add more suppliers,Any plans for more suppliers?
…
What a strong answer doesThe answer key the graders mark against

Shows that the marketplace headline doesn't hold. 'Add more suppliers' is the biggest ticket category (75 of 200), but 58 of those come from one account, Calder Group; across other accounts, delivery and invoice problems ('delivery didn't match order' and 'invoice reconciliation', 63 tickets) are far more common than supplier requests (17). In the interviews, 7 of 10 describe a specific recent instance of the same pain (short deliveries, invoices that don't match, credit notes, re-typing paper checks), while the marketplace mentions are wishes (the pizzeria 'would love' it, the tapas bar 'I suppose') or come from Calder. Weights what people did over what they say they'd want, counts how many sources support each finding, and recommends focusing next quarter on checking deliveries and reconciling invoices, with the marketplace treated as one large customer's request to handle separately.

Critical failures (cap the score at 40)
  • Recommends building the supplier marketplace next quarter as the main finding
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Usable with a quick edit9 of 12 checks passed

The output is clear, well-structured, and only needs a minor correction to the Ordering ticket count (30, not 31) to be fully accurate and usable.

Output

Synthesis for the CPO: what to build next quarter

Headline: The evidence doesn't support a marketplace. It points to the gap between what we order, what arrives and what we're invoiced. I recommend building delivery checking and reconciliation next quarter.

Corrections to the earlier draft

  • "Add more suppliers is our top category by far" is an artefact of one account. Of 200 tickets, 75 are in that category, and 58 of those come from Calder Group, which repeats four requests: Brakes, a meat supplier, a dairy supplier and a wine merchant. The other 17 come from 17 different accounts. Without Calder, it is our smallest category (17 of 142 tickets, 12%).
  • "Most customers want a marketplace" is wrong. Only 2 of 10 interviewees asked for more suppliers (Calder and the pizzeria). The tapas bar was lukewarm ("I suppose") and went straight to paperwork. The café group said finding suppliers "isn't the problem."

Findings and strength of evidence

1. Customers can't easily verify what was delivered against what they ordered and were invoiced for. Strength: strong. - Interviews: 7 of 10 describe this unprompted, and an eighth (tapas bar) says "paperwork" without detail. Examples are short fish orders found a week later, a chef signing notes unchecked (~£600 in a month), finance chasing credit notes, "my word against theirs," clipboard re-typing, and invoices that never match. - Tickets: "Delivery didn't match order" (34) and "Invoice reconciliation" (29) total 63 tickets, 44% of non-Calder volume, from about 62 distinct accounts. The ticket summaries include "short delivery not shown in app," "invoice includes items not delivered," "credit note missing" and "export doesn't show credits." - Interviews and tickets are independent sources and agree. The combined grouping is my judgement, but delivery tickets alone (34) are still twice the non-Calder supplier requests.

2. Generic demand for more suppliers is small and vague. Strength: moderate. - The 17 non-Calder tickets are "any plans for more suppliers?" (8) and "more produce suppliers" (9). They show mild interest, not a pain point. One interviewee (the pizzeria) wants more choice and better prices.

3. Calder Group has a real, specific need. Strength: strong for Calder, weak as a market signal. - Calder is one account with 14 sites, which says it would move all ordering to us with a marketplace. This is a stated, conditional intent. Its tickets name four suppliers, which is an onboarding request, not evidence for a marketplace. We don't yet know Calder's revenue or how many similar multi-site groups we have.

4. Smaller issues: ordering, logins and tablets. Strength: moderate, low priority. - Ordering (31 tickets) and login/access (31 tickets) are mostly routine. "Can't log in on tablet" matters because delivery checking will likely happen on a tablet at the back door.

What this means for next quarter

Build: a receiving and reconciliation flow. - Staff record what actually arrived at the door: shorts, substitutions and weights. - The app flags mismatches against the order and the invoice. - It produces a timestamped record the owner can send to the supplier as a credit-note claim. - Credits flow into the accountant export.

This addresses the main complaints: finding out a week late, the "my word against theirs" disputes, and double entry. It also helps the people who file tickets about short deliveries and missing credit notes.

Don't build: a marketplace. It's a large investment backed by one account and a few vague requests.

Handle Calder separately. Onboard their four named suppliers through our existing process if feasible. Then ask Calder whether that satisfies their "move everything to you" condition, and get their revenue.

Fix before building: tablet login reliability, since the new flow depends on it.

Caveats

  • The sample is 10 interviews from September, and I don't know how they were recruited. Eight of ten raising the same pain is encouraging but not a measured prevalence.
  • Tickets show what people contact support about, not everything that hurts. Customers with delivery problems may just pay the invoice, as the steakhouse does.
  • Ticket data is one quarter, and the `account_sites` values are inconsistent for a few accounts (for example 1418 and 1625), so I haven't segmented by size.
  • The £600 loss is one anecdote. Before committing scope, we should size the cost of short deliveries and unclaimed credits across a sample of accounts.

What would change my view

If we find a meaningful segment of multi-site groups like Calder, with material revenue that is blocked on supplier coverage, supplier breadth becomes a segment-specific bet. It would then be worth testing through targeted onboarding before any full marketplace.

Claims the judge couldn’t find in the brief

  • Ordering (31 tickets) and login/access (31 tickets) are mostly routine.

Grades and run details

Decision model 100 · LLM judge 77
Decision model checks
  • passUses the supplied evidence correctly36%
  • passAddresses the actual decision100%
  • passRespects explicit constraints53%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims56%
  • passProduces the required deliverable95%
  • passSees one account behind the top ticket100%
  • passCorrects the draft's claim99%
  • passWeights behaviour over opinion100%
  • passSays how many sources support each finding87%
  • passFrames findings for the decision99%
  • passReports what cuts against the plan100%
Run
Run
#1
API response time
47 s
Submitted
6 Oct 2026

Needs more than a quick edit9 of 12 checks passed

Factual errors in ticket counts (ordering 31 vs 30, credits export 8 vs 10) and word count exceedance mean a PM would need to re-check evidence and edit for length.

Output

Synthesis: what customers need from Larder next (for the CPO)

Headline: The broadest and best-evidenced problem is that customers can't easily tell what actually arrived, and it costs them money and hours every week. Demand for "more suppliers" is real, but nearly all of it comes from one account. That account wants its own four suppliers connected, which is not the same as a marketplace. Next quarter we should build delivery check-in and discrepancy tracking, and handle Calder Group's request as a bounded onboarding project.

Finding 1: Delivery and invoice mismatches are the most widespread pain

Evidence: strong. It is consistent across both sources, comes from many accounts, and customers describe concrete costs.

  • Tickets: "Delivery didn't match order" (34) and "Invoice reconciliation" (29) describe the same problem from two ends: ordered, received and invoiced don't line up. Together they account for 63 of 200 tickets (32%) from about 62 different accounts. By number of customers affected, this is our largest problem.
  • Interviews: 8 of 10 describe it. Examples:
  • About £600 lost in a month on undelivered items (gastropub).
  • A finance team spending days every week chasing credit notes (hotel).
  • Sunday nights reconciling invoices with a highlighter (bistro).
  • Disputes with suppliers that can't be won without proof (bakery).
  • Checking deliveries on a clipboard, then re-typing everything (noodle bar).
  • Paying mismatched invoices because there's no time to check (steakhouse).

Finding 2: "Add more suppliers" is the top ticket category, but it is essentially one customer

Evidence: strong that the demand is concentrated; weak that there is broad demand for a marketplace.

- Calder Group's tickets: 75 tickets (37.5%) fall in this category, but 58 come from Calder Group alone. They repeatedly ask for the same four suppliers: - Brakes (15 tickets) - Their meat supplier (15) - Their dairy supplier (14) - Their wine merchant (14) They want suppliers they already use connected to Larder. They are not asking to discover new suppliers. - Everyone else: The remaining 17 tickets come from 17 accounts and are mostly generic ("Any plans for more suppliers?", "Would like more produce suppliers"). - Interviews: Only 2 of 10 clearly want this (the pizzeria and Calder). The tapas bar is lukewarm ("I suppose") and says paperwork is the real problem. The café group says outright that "finding suppliers isn't the problem."

The earlier claim that "most customers we spoke to want a marketplace" is not supported.

Finding 3: Calder is a real commercial opportunity of a specific kind

Evidence: moderate. This is one account's clear statement, untested.

  • Calder has 14 sites and says it would move all its ordering to us if its suppliers were on the platform.
  • We don't yet know how much of its ordering runs through Larder today or what the account is worth. We need that before sizing the effort.
  • Their need is met by onboarding four named suppliers, not by building a marketplace.

Other signals

  • Login and access (31 tickets) and ordering questions (31) look routine.
  • One subset matters for the recommendation below: 10 tickets are "can't log in on tablet," and the tablet is where delivery check-in would happen.

Limits of this evidence

  • Small interview sample. There are 10 interviews and we don't know how participants were chosen.
  • One quarter of tickets. Ticket volume measures who contacts support, not who has the problem. Customers who silently pay wrong invoices, like the steakhouse, won't file tickets, so mismatch pain is probably under-counted.
  • Data quality. Support staff assign the categories, and a few accounts show inconsistent site counts across tickets.
  • No usage or revenue data yet. We can't weigh Calder against the long tail of smaller accounts.

What this means for next quarter

1. Build delivery check-in and discrepancy tracking. This is our core promise ("track deliveries and invoices") and is where the broad pain sits. Scope: - Check a delivery against the order at the door, on phone or tablet, replacing the clipboard and the re-typing. - Record shorts, substitutions and short weights, with a photo and timestamp to use as evidence in disputes. - Flag invoice lines that don't match what was received. - Raise credit notes and track them until they arrive. - Show credits in the accountant export (8 tickets ask for this). - Fix tablet login as part of the work.

Success measures: - Share of deliveries checked in. - £ of discrepancies recorded and credited. - Fewer tickets in the two mismatch categories. - Time spent on reconciliation, re-checked with these interviewees.

2. Treat Calder as an account project, not a platform bet. - Work with sales to confirm the account's value and how much of its ordering would move. - Then scope onboarding of the four requested suppliers. This is bounded work that doesn't require a marketplace.

3. Don't build a marketplace next quarter. - If we want to test demand, do it cheaply. Follow up with the 17 non-Calder accounts or run a fake-door test in the app. - Revisit with that data.

Claims the judge couldn’t find in the brief

  • Login and access tickets: 31; ordering questions: 31.
  • Show credits in the accountant export (8 tickets ask for this).

Grades and run details

Decision model 100 · LLM judge 69
Decision model checks
  • passUses the supplied evidence correctly39%
  • passAddresses the actual decision95%
  • passRespects explicit constraints52%
  • passIdentifies material uncertainty92%
  • passAvoids unsupported claims47%
  • passProduces the required deliverable95%
  • passSees one account behind the top ticket100%
  • passCorrects the draft's claim98%
  • passWeights behaviour over opinion100%
  • passSays how many sources support each finding72%
  • passFrames findings for the decision99%
  • passReports what cuts against the plan100%
Run
Run
#1
Time to output
72 s
Submitted
5 Oct 2026

Needs more than a quick edit8 of 12 checks passed

The output contains incorrect ticket and interview counts that would require re-checking the source data. A PM cannot use it without verifying and correcting those numbers.

Output

Product Synthesis: Customer Research & Support Ticket Analysis

To: Chief Product Officer From: Product Manager, Ordering Date: October 2023 Subject: Evidence-based review of next quarter’s roadmap (revising the supplier marketplace proposal)

---

Executive Summary

Our draft synthesis concluded that customers want a supplier marketplace and recommended building one next quarter. A rigorous re-examination of our customer interviews and last quarter's support tickets reveals this was a misreading of the data.

While "Add more suppliers" appears frequently in our ticket logs, over 45% of those tickets originate from a single multi-site account (Calder Group). Across our broader base of 1,140 mostly independent accounts, the core operational pain points are invoice reconciliation, short deliveries, and tracking actuals against orders.

Building a marketplace addresses a growth/acquisition lever heavily championed by enterprise accounts, but it does not solve the acute, daily operational friction draining time and money for our core user base. We should refocus next quarter’s build on delivery discrepancy tracking and invoice matching.

---

Main Findings & Evidence Strength

#### Finding 1: The "Supplier Marketplace" demand is heavily skewed by a single enterprise account. * The Claim: "Add more suppliers" is our top support category by far, proving a marketplace is our most urgent need. * The Evidence: A raw count of last quarter’s tickets (LT-5000 to LT-5199) shows roughly 45 tickets categorized under "Add more suppliers." However, a closer look at the account data reveals that 22 of those tickets come from Calder Group (a 14-site group). Furthermore, in customer interviews, while one pizzeria and Calder Group explicitly asked for a marketplace, other multi-site and independent operators stated otherwise: "Finding suppliers isn't the problem. Knowing what actually arrived is" (Café group, 5 sites); "What kills me is the paperwork" (Tapas bar). * Evidence Strength: Moderate to Weak. The support volume for suppliers is inflated by one outlier account. Independent operators view suppliers as readily available; their bottleneck is post-order administration.

#### Finding 2: Invoice reconciliation and short deliveries are massive, recurring operational drains. * The Claim: Customers spend significant uncompensated time matching invoices, handling missing credit notes, and absorbing losses on unfulfilled goods. * The Evidence: * Interviews: 7 out of 10 interviewed operators independently volunteered painful anecdotes about discrepancies: fish orders arriving short (Bistro), signing delivery notes blindly and losing £600/month on ghost items (Gastropub), spending days chasing credit notes (Hotel), and abandoning dispute processes because "I just pay it, I don’t have time" (Steakhouse). * Support Tickets: Across the remaining tickets not tied to Calder Group's supplier requests, the volume is heavily dominated by "Invoice reconciliation" (mismatched totals, export issues) and "Delivery didn't match order" (short deliveries, missing credit notes, unrecorded substitutions). * Evidence Strength: Strong. Universally present across interviews and forming the bulk of legitimate, distributed support traffic from independent accounts.

#### Finding 3: Double-handling workflows create friction between physical delivery and digital tracking. * The Claim Users struggle to reconcile what happened on the loading dock with what is inside the app. * The Evidence: Operators report checking physical deliveries on clipboards or highlighters, then manually transcribing them into the app ("Twice the work" – Noodle bar) or failing to surface short deliveries because the app doesn't natively map discrepancies in real-time. * Evidence Strength: Strong. Directly observed in multiple site interviews.

---

What This Means for Next Quarter (Product Recommendations)

If we build a supplier marketplace next quarter, we are effectively prioritizing the expansion requests of our largest enterprise customer (Calder Group) while ignoring the acute daily churn and time sinks plaguing our broader base of 1,140 independent accounts.

To deliver high impact next quarter, we should pivot our roadmap away from marketplace infrastructure and toward Post-Order Control & Reconciliation:

  1. Native Discrepancy & Substitution Management: Build intuitive, in-app workflows for chefs and managers to flag short-weights, damaged goods, and substitutions at the time of delivery (solving the clipboard-to-app transcription problem).
  2. Streamlined Credit Note & Invoice Matching: Automate the comparison between the original order, the delivery note, and the supplier invoice. Give finance teams the ability to flag discrepancies and request credit notes directly within Larder, eliminating the "highlighter and Sunday night paperwork" routine.
  3. Targeted Supplier Onboarding (Selective, not Marketplace): Rather than building an open supplier marketplace, use API/onboarding bandwidth to manually satisfy enterprise expansion blockers (like Calder Group's meat and wine suppliers) on a case-by-case basis, protecting engineering resources for core workflow improvements.

Claims the judge couldn’t find in the brief

  • A raw count of last quarter's tickets shows roughly 45 tickets categorized under 'Add more suppliers'
  • 22 of those tickets come from Calder Group
  • 7 out of 10 interviewed operators independently volunteered painful anecdotes about discrepancies
  • The app doesn't natively map discrepancies in real-time

Grades and run details

Decision model 75 · LLM judge 62
Decision model checks
  • failUses the supplied evidence correctly65%
  • passAddresses the actual decision32%
  • passRespects explicit constraints37%
  • failIdentifies material uncertainty60%
  • partialAvoids unsupported claims30%
  • passProduces the required deliverable75%
  • passSees one account behind the top ticket53%
  • passCorrects the draft's claim32%
  • passWeights behaviour over opinion98%
  • partialSays how many sources support each finding53%
  • passFrames findings for the decision92%
  • passReports what cuts against the plan100%
Run
Run
#1
Time to output
6 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 7

Uses the supplied evidence correctlyMixedMixedWrong
Sonnet 5.5 · API

The Ordering ticket count is stated as 31, but the CSV contains 30 Ordering tickets, so a factual statement about the current situation is incorrect.

Opus 5.5 · Claude

Two factual numbers are wrong: ordering tickets are 30 not 31, and credits export tickets are 10 not 8.

Gemini 3.5 Flash-Lite · Gemini

Output states incorrect ticket counts (45 total, 22 from Calder) and wrong interview count (7 of 10), which are not in the evidence and do not follow from arithmetic.

Addresses the actual decisionRightRightMixed
Sonnet 5.5 · API

Commits early to building delivery checking and reconciliation, and says what would change the view (finding a meaningful segment of multi-site groups like Calder).

Opus 5.5 · Claude

Commits to building delivery check-in, treating Calder as an account project, and not building a marketplace, with conditions to revisit.

Gemini 3.5 Flash-Lite · Gemini

Output commits to a recommendation but does not say what result or condition would change that decision.

Respects explicit constraintsRightMixedRight
Sonnet 5.5 · API

The output is a synthesis for the CPO, well under 800 words, and respects the requested form and reader.

Opus 5.5 · Claude

The output exceeds the 800-word limit (approximately 826 words).

Gemini 3.5 Flash-Lite · Gemini

Output is a synthesis for the CPO, within 800 words, and includes findings, evidence strength, and next-quarter implications.

Identifies material uncertaintyRightRightWrong
Sonnet 5.5 · API

Names unknowns (interview recruitment, ticket bias, inconsistent account_sites, anecdotal £600, need to size cost) and says how they would be resolved or what would change the call.

Opus 5.5 · Claude

Names specific unknowns (sample size, ticket bias, data quality, no revenue data) and how to resolve them.

Gemini 3.5 Flash-Lite · Gemini

Output does not name any unknowns that could change the decision or how they would be resolved.

Avoids unsupported claimsMixedRightWrong
Sonnet 5.5 · API

The claim 'Ordering (31 tickets)' is presented as an established fact, but the evidence shows 30 Ordering tickets, so it is an unsupported confident claim.

Opus 5.5 · Claude

Hypotheses and forecasts are labelled (e.g., 'untested', 'probably under-counted'), and confident claims are supported.

Gemini 3.5 Flash-Lite · Gemini

Presents '7 out of 10' as fact when it is 8, and claims the app doesn't natively map discrepancies without supporting evidence.

Produces the required deliverableRightMixedRight
Sonnet 5.5 · API

The synthesis is complete, in the right form for the CPO, within the word limit, and actionable with light edits.

Opus 5.5 · Claude

The synthesis is present but exceeds the 800-word limit, so it is not within the requested length.

Gemini 3.5 Flash-Lite · Gemini

Delivers a synthesis with main findings, evidence strength, and next-quarter recommendations, usable for the CPO.

Says how many sources support each findingMixedRightRight
Sonnet 5.5 · API

The finding on smaller issues sizes Ordering as 31 tickets, but the actual count is 30, so the finding is not accurately sized.

Opus 5.5 · Claude

Every finding is sized with numbers (8 of 10 interviews, 63 tickets, 58 from Calder, 17 from 17 accounts).

Gemini 3.5 Flash-Lite · Gemini

Each finding includes a count (e.g., '7 out of 10', '22 of those tickets'), even if some numbers are inaccurate.

All got right 5

Sees one account behind the top ticketRightRightRight
Sonnet 5.5 · API

Clearly shows that 58 of 75 'Add more suppliers' tickets come from Calder Group, and that delivery/invoice issues are spread across many accounts.

Opus 5.5 · Claude

Clearly shows Calder Group accounts for 58 of 75 supplier tickets, while delivery/invoice issues are spread across many accounts.

Gemini 3.5 Flash-Lite · Gemini

Identifies Calder Group as the source of most supplier tickets and contrasts with delivery/invoice issues spread across many accounts.

Corrects the draft's claimRightRightRight
Sonnet 5.5 · API

Counts the interviews (2 of 10 asked for more suppliers) and corrects the headline, showing that most interviews describe delivery and invoice pain.

Opus 5.5 · Claude

Corrects the draft by counting interviews: only 2 of 10 want a marketplace, and 8 describe delivery/invoice pain.

Gemini 3.5 Flash-Lite · Gemini

Corrects the draft's headline by showing marketplace demand is skewed and interviews mostly describe delivery/invoice pain.

Weights behaviour over opinionRightRightRight
Sonnet 5.5 · API

Distinguishes specific past instances of delivery/invoice pain from stated preferences and feature requests, and flags Calder's need as 'stated, conditional intent'.

Opus 5.5 · Claude

Weights concrete costs and observed behaviour (money lost, time spent) over stated wishes ('would love', 'I suppose').

Gemini 3.5 Flash-Lite · Gemini

Distinguishes stated marketplace wishes from observed operational pain and weights the latter more heavily.

Frames findings for the decisionRightRightRight
Sonnet 5.5 · API

Each main finding is framed as an opportunity and linked to the decision, with clear implications for what to build next quarter.

Opus 5.5 · Claude

Each finding is framed as an opportunity and linked to the decision (build check-in, don't build marketplace, treat Calder as account project).

Gemini 3.5 Flash-Lite · Gemini

Each finding is framed as an opportunity or pain point and linked to the decision about next quarter's build.

Reports what cuts against the planRightRightRight
Sonnet 5.5 · API

Reports evidence that cuts against the marketplace hypothesis, including the concentration of supplier requests in one account and the prevalence of delivery/invoice pain.

Opus 5.5 · Claude

Reports evidence against the marketplace plan: concentration in one account, interviews mostly about mismatch, café group's statement.

Gemini 3.5 Flash-Lite · Gemini

Reports evidence that challenges the marketplace plan and explains what it means for the roadmap.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 74% of 62 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT95.595.82None
2GPT-6.1 SolwithAPI97.792.02None
3GPT-6 LunawithAPI93.284.02None
4Sonnet 5.5withAPI95.567.62None
5Haiku 5.5withAPI86.471.22None
6Opus 5.5withClaude86.463.82None
7Gemini 3.8 FlashwithAPI71.063.12None
8Gemini 3.5 Flash-LitewithGemini69.347.42None

What you get for the cost

GPT-6.1 Sol · API scores within a point of GPT-6 Astra · ChatGPT at a ninth of the cost.

Task score against relative cost for each setup. The table below has the same figures.50607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 96, 194× the cheapest (1 of 2 outputs estimated)GPT-6.1 Sol · API: task score 95, 21× the cheapestGPT-6 Luna · API: task score 87, the cheapestSonnet 5.5 · API: task score 80, 62× the cheapestHaiku 5.5 · API: task score 77, 3.4× the cheapestOpus 5.5 · Claude: task score 72, 167× the cheapest (1 of 2 outputs estimated)Gemini 3.8 Flash · API: task score 66, 19× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 56, 6.5× the cheapest (1 of 2 outputs estimated)GPT-6 Astra · ChatGPTGPT-6.1 Sol · APIGPT-6 Luna · APISonnet 5.5 · APIHaiku 5.5 · APIOpus 5.5 · ClaudeGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API871.0×
Haiku 5.5 · API773.4×
Gemini 3.5 Flash-Lite · Gemini566.5× (partly estimated)
Gemini 3.8 Flash · API6619×
GPT-6.1 Sol · API9521×
Sonnet 5.5 · API8062×
Opus 5.5 · Claude72167× (partly estimated)
GPT-6 Astra · ChatGPT96194× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Turning a stack of call transcripts into what we actually learned.

Why it matters

Synthesis is where teams fool themselves. A model that smooths away dissent or turns one loud customer into a trend produces confident, wrong roadmaps.

What good looks like

  • Quotes evidence for each theme and counts sources honestly
  • Weights what customers did above what they say they'd do
  • Keeps important dissent visible
  • Labels hypotheses as hypotheses
  • Says what the research cannot tell us

Deliberately not measured

  • Transcript clean-up
  • Persona illustration
Capability tested

Faithful synthesis of qualitative research

The failure we’re looking for

Invents customer consensus or loses important dissent

Grading

Decision model and LLM judge, calibrated against a blind PM review