Tasks / Product Craft

Extract discovery insights

Can the model separate evidence, themes and hypotheses without inventing consensus?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 50% were usable with at most a quick edit.

Reliably right

  1. Frames findings for the decision100% pass
    Each main finding is framed as an opportunity (delivery-to-invoice pain, concentrated supplier need) and linked directly to what to build next quarter.
    GPT-6 Astra · ChatGPT · The loudest ticket
  2. Corrects the draft's claim100% pass
    It corrects the draft's headline by counting interviews (8 describe delivery/invoice pain, 2 mention supplier expansion) and showing the marketplace claim is unsupported.
    GPT-6 Astra · ChatGPT · The loudest ticket
  3. Keeps dissent visible100% pass

Where it slips

  1. Says how many sources support each finding46% pass
    The finding on smaller issues sizes Ordering as 31 tickets, but the actual count is 30, so the finding is not accurately sized.
    Sonnet 5.5 · API · The loudest ticket
  2. Uses the supplied evidence correctly56% pass
    The output states Ordering has 31 tickets, but the CSV contains 30 Ordering tickets; this factual claim is not supported by the supplied data.
    GPT-6.1 Sol · API · The loudest ticket
  3. Addresses the actual decision57% pass
    The output commits to a clear recommendation but does not state what result or condition would change that decision.
    Gemini 3.8 Flash · API · The loudest ticket

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Weights behaviour over opinion

    Does the synthesis distinguish evidence of what customers actually did (specific past instances, observed behaviour, workarounds) from stated preferences, predictions and feature requests, and weight them accordingly?

    Passes when Themes are labelled or weighted by evidence type, and at least one theme resting mainly on stated intent or requests is flagged as weaker.

  2. Frames findings for the decision

    Does the synthesis turn its findings into opportunities tied to the decision or outcome the team faces, saying what each one means for that decision?

    Passes when Each main finding is framed as an opportunity (a need or pain to address) and linked to the decision at hand, with what it implies.

  3. Reports what cuts against the plan

    Does the synthesis report the evidence that cuts against the team's current belief or plan, rather than only what supports it?

    Passes when Names at least one finding that challenges the team's hypothesis or plan, and says what it means for it.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Says how many sources support each finding · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for Larder's ordering product. A colleague has drafted the synthesis below for our CPO, recommending a supplier marketplace. Before it goes, rewrite the synthesis from the evidence, in no more than 800 words: the main findings, how strong the evidence for each is, and what they mean for what we build next quarter. The interviews and the ticket export are attached.

What the model was given4 items: About Larder, draft_synthesis.md, interview_excerpts.md, support_tickets_last_quarter.csv
About LarderOrdering software for restaurants: order from your suppliers in one place, track deliveries and invoices. 1,140 accounts, most of them independent restaurants.
draft_synthesis.md6 lines · Download
# Draft synthesis (PM, for the CPO)

Headline: customers want a supplier marketplace.
- "Add more suppliers" is our top support category by far.
- Most customers we spoke to want a marketplace (see Calder Group and the pizzeria).
- Recommendation: build a supplier marketplace next quarter.
interview_excerpts.md12 lines · Download
# Interview excerpts (restaurant owners and managers, September)

1. Bistro, 2 sites: "Last Tuesday the fish order came short again. I only found out when the invoice came, a week later. I spent Sunday night matching delivery notes to invoices with a highlighter."
2. Café group, 5 sites: "We switched produce supplier twice this year, but finding suppliers isn't the problem. Knowing what actually arrived is."
3. Pizzeria, 1 site: "Honestly I'd love more suppliers in the app. More choice, better prices."
4. Gastropub: "My head chef signs the delivery note without checking. Then I pay for things we never got. Last month that was about £600."
5. Hotel restaurant: "Our finance team chases credit notes for short deliveries every week. It takes days."
…
support_tickets_last_quarter.csv200 rows · Download
ticket,account,account_sites,category,summary
LT-5000,Calder Group,14,Add more suppliers,Can you onboard our meat supplier
LT-5001,Account 1886,3,Ordering,Change delivery day
LT-5002,Account 2004,1,Delivery didn't match order,Short delivery not shown in app
LT-5003,Calder Group,14,Add more suppliers,Add dairy supplier please
LT-5004,Calder Group,14,Add more suppliers,Add dairy supplier please
LT-5005,Account 1858,2,Add more suppliers,Any plans for more suppliers?
…
What a strong answer doesThe answer key the graders mark against

Shows that the marketplace headline doesn't hold. 'Add more suppliers' is the biggest ticket category (75 of 200), but 58 of those come from one account, Calder Group; across other accounts, delivery and invoice problems ('delivery didn't match order' and 'invoice reconciliation', 63 tickets) are far more common than supplier requests (17). In the interviews, 7 of 10 describe a specific recent instance of the same pain (short deliveries, invoices that don't match, credit notes, re-typing paper checks), while the marketplace mentions are wishes (the pizzeria 'would love' it, the tapas bar 'I suppose') or come from Calder. Weights what people did over what they say they'd want, counts how many sources support each finding, and recommends focusing next quarter on checking deliveries and reconciling invoices, with the marketplace treated as one large customer's request to handle separately.

Critical failures (cap the score at 40)
  • Recommends building the supplier marketplace next quarter as the main finding
Task

Staff-level · v1.1

The outputs

9 setups answered this brief.

Needs more than a quick edit9 of 12 checks passed

The output contains several arithmetic errors in ticket counts (e.g., 72 instead of 75, 36 instead of 33) that would force a PM to re-verify the data before using it.

Output

Product Discovery Synthesis: Q3 Interviews & Support Tickets

To: Chief Product Officer From: PM, Ordering Subject: Correcting the Q4 roadmap recommendation: Delivery & Invoice Reconciliation vs. Supplier Marketplace

Executive Summary

An initial look at last quarter’s data suggested we should build a supplier marketplace next quarter. However, a deeper analysis of our 200 support tickets and 10 customer interviews shows the marketplace signal is a mirage caused by a single vocal enterprise account.

For our core customer base of independent restaurants, the most urgent, expensive, and widespread problem is delivery tracking and invoice reconciliation—specifically handling short deliveries, substitutions, and credit notes. We should focus Q4 on fixing the post-order workflow, not building a marketplace.

---

Main Findings & Strength of Evidence

#### 1. The "Supplier Marketplace" demand is heavily skewed and misunderstood * Strength of Evidence: Weak * What the data actually shows: On the surface, "Add more suppliers" is our #1 ticket category, making up 72 of 200 tickets (36%). However, 56 of those 72 tickets (78%) came from a single 14-site customer: Calder Group. The remaining 16 tickets came from just 16 unique accounts (out of 1,140 total). * The Nuance: Even Calder Group’s tickets are not requests to discover new suppliers in a marketplace. They are repeatedly asking us to onboard their existing suppliers (their meat supplier, dairy supplier, wine merchant, and Brakes) so they can centralize ordering. * Interview Context: In interviews, only 1 of 10 customers (the single-site Pizzeria) genuinely wanted a marketplace for supplier discovery and price comparison. Calder Group used the word "marketplace" to mean having all their current vendors on the platform. Two others actively dismissed the idea: the Tapas Bar expressed lukewarm interest before pivoting to paperwork pain, and the 5-site Café Group explicitly stated: "Finding suppliers isn't the problem. Knowing what actually arrived is."

#### 2. Delivery mismatches and invoice reconciliation are our largest widespread pain points * Strength of Evidence: Strong * What the data actually shows: In our interviews, 8 out of 10 customers identified post-order tracking (short deliveries, substitutions, invoice matching, and chasing credit notes) as their primary operational headache. This directly causes financial loss (e.g., the Gastropub losing ~£600/month paying for undelivered goods) and severe time waste (owners spending Sunday nights matching paper notes to invoices, or re-typing clipboard notes into Larder). * Support Ticket Alignment: In the ticket data, "Delivery didn't match order" (36 tickets across 36 unique accounts) and "Invoice reconciliation" (31 tickets across 30 unique accounts) are two halves of the exact same problem. Combined, they account for 67 tickets across 64 unique accounts—four times the unique account volume of the supplier requests (excluding Calder Group). * Specific Product Gaps: Customers are explicitly telling us where the product breaks down: * At the back door: Short deliveries aren't showing in the app (9 tickets), users don't know how to log substitutions (8 tickets), and chefs check deliveries on paper clipboards—likely exacerbated by our tablet login bug (10 tickets). * In the back office: Invoices include undelivered items (9 tickets), invoice totals don't match orders (9 tickets), users can't match invoices to orders (10 tickets), credit notes are missing (8 tickets), and accountant exports fail to show credits (10 tickets).

#### 3. Core ordering and access friction creates unnecessary noise * Strength of Evidence: Moderate (Ticket data only) * What the data actually shows: We saw 31 tickets for basic "Ordering" issues (10 duplicate orders, 11 delivery day changes, 10 cut-off time questions) and 30 for "Login and access" (10 password resets, 10 adding users, 10 tablet login failures). While unmentioned in interviews, these represent 30% of support volume and hurt day-to-day usability.

---

What This Means for Q4: Recommendations

Building a two-sided supplier marketplace next quarter would be a costly misstep. It solves a problem (supplier discovery) that almost none of our customers have, based on un-deduplicated ticket data from one large account.

Instead, we should align our Q4 roadmap with our core value proposition—"track deliveries and invoices"—where our independent restaurants are bleeding time and money:

1. Build a "Goods-In" & Discrepancy Workflow (Primary Q4 Bet): * Allow kitchen staff to easily check off deliveries on mobile/tablet at the back door, logging short weights, missing items, and substitutions in real time. * Automatically flag discrepancies against the original order, adjust expected invoice totals, and track pending credit notes through to the accountant export. 2. Fix the Tablet Login Bug & Ordering UX (Quick Wins): * Fixing the tablet login issue (10 tickets) is a prerequisite for kitchen staff checking deliveries digitally instead of on clipboards. * Address duplicate order prevention and surface supplier cut-off times clearly in the UI to cut ordering support tickets. 3. Handle Calder Group Operationally, Not via a Marketplace: * Have Customer Success and Partnerships work directly with Calder Group to onboard their four requested suppliers (Brakes, meat, dairy, wine). For the broader base, improve our "request to onboard my existing supplier" intake flow so sites don't spam support.

Claims the judge couldn’t find in the brief

  • "Add more suppliers" is the #1 ticket category, making up 72 of 200 tickets (36%).
  • 56 of those 72 tickets (78%) came from Calder Group.
  • The remaining 16 tickets came from 16 unique accounts.
  • "Delivery didn't match order" has 36 tickets.
  • "Invoice reconciliation" has 31 tickets.

Grades and run details

Decision model 71 · LLM judge 77
Decision model checks
  • failUses the supplied evidence correctly37%
  • failAddresses the actual decision31%
  • passRespects explicit constraints67%
  • failIdentifies material uncertainty77%
  • partialAvoids unsupported claims51%
  • passProduces the required deliverable95%
  • passSees one account behind the top ticket97%
  • passCorrects the draft's claim94%
  • passWeights behaviour over opinion100%
  • passSays how many sources support each finding43%
  • passFrames findings for the decision97%
  • passReports what cuts against the plan100%
Run
Run
#1
API response time
15 s
Submitted
13 Oct 2026

Usable with a quick edit11 of 12 checks passed

The only flaw is an off-by-one error in two ticket counts (31 vs 30), which can be fixed in a quick edit without re-checking the core reasoning or evidence.

Output

Synthesis: what customers are telling us

Headline: Our biggest problem is that customers can't easily check what arrived against what they were invoiced. A supplier marketplace is a real but narrow request, concentrated in one large account. I don't recommend building it next quarter.

Sources: 10 owner and manager interviews (September) and 200 support tickets from last quarter (LT-5000 to LT-5199).

What changed from the draft: "Top category by far" holds only because of one account. "Most customers we spoke to want a marketplace" is 2 of 10, and one of those is hedged. The recommendation doesn't follow from the evidence.

Findings

1. Checking deliveries and reconciling invoices is the main pain. Evidence: strong. - 63 of 200 tickets (32%) are "delivery didn't match order" or "invoice reconciliation." The most common issues are short deliveries not shown in the app, missing credit notes, invoices including items not delivered, invoices that can't be matched to orders, and accountant exports that omit credits. - These tickets come from many different accounts, not one. - 8 of 10 interviews describe this problem. A gastropub lost about £600 in a month paying for unchecked deliveries. A hotel finance team chases credit notes weekly. A bakery is in a "my word against theirs" dispute with a supplier. A noodle bar re-types clipboard checks into the app. - Caveats: ticket categories are assigned by support staff, and 10 interviews is a small sample.

2. Supplier demand is concentrated in one account. Evidence: strong for Calder Group, weak as a base-wide signal. - "Add more suppliers" is the largest ticket category (75 tickets), but 58 come from Calder Group (14 sites). Outside Calder it is 17 tickets, fewer than delivery issues (34) or ordering issues (31). - Calder's requests repeat four named suppliers: Brakes (15 tickets), meat (15), dairy (14), and a wine merchant (14). These are four specific onboarding requests, not a general need for a marketplace. - Outside Calder, requests are vague ("any plans for more suppliers?", "would like more produce suppliers"), spread across accounts. - In interviews, 2 of 10 want more suppliers: the pizzeria and Calder. A tapas bar says it would use a marketplace "I suppose," but says the paperwork is what hurts day to day. A café group says finding suppliers "isn't the problem. Knowing what actually arrived is."

3. Calder Group is a high-value but conditional signal. Evidence: one account, one conversation. - Calder says: "If you had a marketplace, we'd move all our ordering to you." That is the strongest commercial statement in the material, but it is one customer's conditional view, and it depends on four specific suppliers. - We don't know from these materials how much ordering volume or revenue Calder represents.

4. Ordering and login tickets are routine. Evidence: moderate, low priority. - Ordering (31) and login (31) tickets are mostly specific fixes: cut-off times, duplicate orders, delivery-day changes, tablet login problems, password resets, adding users. None of these came up in interviews.

What this means for next quarter

Build: a receiving and invoice-matching workflow. This addresses the largest and most widely shared problem. - Let staff record received quantities at delivery, with shorts and substitutions flagged in the app. This covers the short-delivery tickets and "how do I record a substitution?" - Automatically match invoices to deliveries and orders, and flag differences before payment. This covers "can't match," "total differs," and "includes items not delivered." - Show credit notes against deliveries and in the accountant export. Fixing the export that omits credits could ship early as a quick win. - Measures: the 63-ticket category volume, and credit value recovered per account, which we can ask customers to report.

Don't build the marketplace next quarter. Instead: - Talk to Calder Group about their ordering volume and what they'd need to move everything to us. Check whether onboarding Brakes, their meat supplier, dairy, and the wine merchant is feasible as partnership or integration work. That is a much smaller decision than building a platform. - Add a "request a supplier" flow so demand from the wider base is captured in a structured way. Then we can see whether requests extend beyond Calder. - Triage the ordering and login tickets into the backlog as small fixes.

What would change my view

  • A clear rise in supplier requests from accounts other than Calder.
  • Around 20 more interviews, across account sizes and including some churned customers, where supplier access is named as the blocker.
  • Evidence that Calder's ordering value justifies the integrations, or that a marketplace would win or retain several accounts of Calder's scale.

Until then, the evidence supports fixing receiving and reconciliation first, and treating the supplier question as a targeted account decision rather than a platform bet.

Claims the judge couldn’t find in the brief

  • Outside Calder, supplier requests are 17 tickets, fewer than delivery issues (34) or ordering issues (31).
  • Ordering (31) and login (31) tickets are routine fixes; none appeared in interviews.

Grades and run details

Decision model 100 · LLM judge 92
Decision model checks
  • passUses the supplied evidence correctly31%
  • passAddresses the actual decision100%
  • passRespects explicit constraints67%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims69%
  • passProduces the required deliverable98%
  • passSees one account behind the top ticket100%
  • passCorrects the draft's claim100%
  • passWeights behaviour over opinion100%
  • passSays how many sources support each finding72%
  • passFrames findings for the decision97%
  • passReports what cuts against the plan100%
Run
Run
#1
API response time
34 s
Submitted
8 Oct 2026

Needs more than a quick edit8 of 12 checks passed

The output contains incorrect ticket and interview counts that would require re-checking the source data. A PM cannot use it without verifying and correcting those numbers.

Output

Product Synthesis: Customer Research & Support Ticket Analysis

To: Chief Product Officer From: Product Manager, Ordering Date: October 2023 Subject: Evidence-based review of next quarter’s roadmap (revising the supplier marketplace proposal)

---

Executive Summary

Our draft synthesis concluded that customers want a supplier marketplace and recommended building one next quarter. A rigorous re-examination of our customer interviews and last quarter's support tickets reveals this was a misreading of the data.

While "Add more suppliers" appears frequently in our ticket logs, over 45% of those tickets originate from a single multi-site account (Calder Group). Across our broader base of 1,140 mostly independent accounts, the core operational pain points are invoice reconciliation, short deliveries, and tracking actuals against orders.

Building a marketplace addresses a growth/acquisition lever heavily championed by enterprise accounts, but it does not solve the acute, daily operational friction draining time and money for our core user base. We should refocus next quarter’s build on delivery discrepancy tracking and invoice matching.

---

Main Findings & Evidence Strength

#### Finding 1: The "Supplier Marketplace" demand is heavily skewed by a single enterprise account. * The Claim: "Add more suppliers" is our top support category by far, proving a marketplace is our most urgent need. * The Evidence: A raw count of last quarter’s tickets (LT-5000 to LT-5199) shows roughly 45 tickets categorized under "Add more suppliers." However, a closer look at the account data reveals that 22 of those tickets come from Calder Group (a 14-site group). Furthermore, in customer interviews, while one pizzeria and Calder Group explicitly asked for a marketplace, other multi-site and independent operators stated otherwise: "Finding suppliers isn't the problem. Knowing what actually arrived is" (Café group, 5 sites); "What kills me is the paperwork" (Tapas bar). * Evidence Strength: Moderate to Weak. The support volume for suppliers is inflated by one outlier account. Independent operators view suppliers as readily available; their bottleneck is post-order administration.

#### Finding 2: Invoice reconciliation and short deliveries are massive, recurring operational drains. * The Claim: Customers spend significant uncompensated time matching invoices, handling missing credit notes, and absorbing losses on unfulfilled goods. * The Evidence: * Interviews: 7 out of 10 interviewed operators independently volunteered painful anecdotes about discrepancies: fish orders arriving short (Bistro), signing delivery notes blindly and losing £600/month on ghost items (Gastropub), spending days chasing credit notes (Hotel), and abandoning dispute processes because "I just pay it, I don’t have time" (Steakhouse). * Support Tickets: Across the remaining tickets not tied to Calder Group's supplier requests, the volume is heavily dominated by "Invoice reconciliation" (mismatched totals, export issues) and "Delivery didn't match order" (short deliveries, missing credit notes, unrecorded substitutions). * Evidence Strength: Strong. Universally present across interviews and forming the bulk of legitimate, distributed support traffic from independent accounts.

#### Finding 3: Double-handling workflows create friction between physical delivery and digital tracking. * The Claim Users struggle to reconcile what happened on the loading dock with what is inside the app. * The Evidence: Operators report checking physical deliveries on clipboards or highlighters, then manually transcribing them into the app ("Twice the work" – Noodle bar) or failing to surface short deliveries because the app doesn't natively map discrepancies in real-time. * Evidence Strength: Strong. Directly observed in multiple site interviews.

---

What This Means for Next Quarter (Product Recommendations)

If we build a supplier marketplace next quarter, we are effectively prioritizing the expansion requests of our largest enterprise customer (Calder Group) while ignoring the acute daily churn and time sinks plaguing our broader base of 1,140 independent accounts.

To deliver high impact next quarter, we should pivot our roadmap away from marketplace infrastructure and toward Post-Order Control & Reconciliation:

  1. Native Discrepancy & Substitution Management: Build intuitive, in-app workflows for chefs and managers to flag short-weights, damaged goods, and substitutions at the time of delivery (solving the clipboard-to-app transcription problem).
  2. Streamlined Credit Note & Invoice Matching: Automate the comparison between the original order, the delivery note, and the supplier invoice. Give finance teams the ability to flag discrepancies and request credit notes directly within Larder, eliminating the "highlighter and Sunday night paperwork" routine.
  3. Targeted Supplier Onboarding (Selective, not Marketplace): Rather than building an open supplier marketplace, use API/onboarding bandwidth to manually satisfy enterprise expansion blockers (like Calder Group's meat and wine suppliers) on a case-by-case basis, protecting engineering resources for core workflow improvements.

Claims the judge couldn’t find in the brief

  • A raw count of last quarter's tickets shows roughly 45 tickets categorized under 'Add more suppliers'
  • 22 of those tickets come from Calder Group
  • 7 out of 10 interviewed operators independently volunteered painful anecdotes about discrepancies
  • The app doesn't natively map discrepancies in real-time

Grades and run details

Decision model 63 · LLM judge 62
Decision model checks
  • failUses the supplied evidence correctly77%
  • failAddresses the actual decision40%
  • passRespects explicit constraints61%
  • failIdentifies material uncertainty53%
  • failAvoids unsupported claims18%
  • passProduces the required deliverable76%
  • passSees one account behind the top ticket55%
  • passCorrects the draft's claim43%
  • passWeights behaviour over opinion98%
  • partialSays how many sources support each finding38%
  • passFrames findings for the decision92%
  • passReports what cuts against the plan100%
Run
Run
#1
Time to output
6 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Uses the supplied evidence correctlyWrongMixedWrong
Gemini 4 Argon · API

Several key ticket counts are wrong (72 not 75, 56 not 58, 36 not 33, 31 not 29), so the supplied evidence is not used correctly.

Haiku 5.5 · API

The ticket counts for ordering and login are both stated as 31, but the CSV contains 30 of each.

Gemini 3.5 Flash-Lite · Gemini

Output states incorrect ticket counts (45 total, 22 from Calder) and wrong interview count (7 of 10), which are not in the evidence and do not follow from arithmetic.

Addresses the actual decisionMixedRightWrong
Gemini 4 Argon · API

It commits unambiguously to not building the marketplace in Q4, focusing instead on delivery/invoice reconciliation, and states what would change the call.

Haiku 5.5 · API

It commits early to not building a marketplace next quarter, says what to do instead, and names conditions that would change the call.

Gemini 3.5 Flash-Lite · Gemini

Output commits to a recommendation but does not say what result or condition would change that decision.

Identifies material uncertaintyWrongRightWrong
Gemini 4 Argon · API

It does not name specific unknowns that could change the decision or say how they would be resolved; it states a definitive call without bounding uncertainty.

Haiku 5.5 · API

It names unknowns (Calder's volume/revenue, small interview sample, support staff categorisation) and says what further evidence would change the recommendation.

Gemini 3.5 Flash-Lite · Gemini

Output does not name any unknowns that could change the decision or how they would be resolved.

Avoids unsupported claimsRightRightWrong
Gemini 4 Argon · API

Interpretations like Calder asking for onboarding rather than discovery are supported by ticket summaries, and hypotheses are labelled ('likely').

Haiku 5.5 · API

The miscount is a factual error rather than presenting an interpretation as fact; otherwise, all confident claims are grounded in the supplied sources.

Gemini 3.5 Flash-Lite · Gemini

Presents '7 out of 10' as fact when it is 8, and claims the app doesn't natively map discrepancies without supporting evidence.

All got right 8

Respects explicit constraintsRightRightRight
Gemini 4 Argon · API

Output is a synthesis to the CPO, within 800 words, in the requested form.

Haiku 5.5 · API

The output is a synthesis for the CPO, with findings, evidence strength, and next-quarter actions, and appears within 800 words.

Gemini 3.5 Flash-Lite · Gemini

Output is a synthesis for the CPO, within 800 words, and includes findings, evidence strength, and next-quarter implications.

Produces the required deliverableRightRightRight
Gemini 4 Argon · API

The synthesis is complete, correctly addressed to the CPO, under 800 words, and actionable with light edits.

Haiku 5.5 · API

A usable synthesis for the CPO with the required sections, within length, that can be acted upon with light edits.

Gemini 3.5 Flash-Lite · Gemini

Delivers a synthesis with main findings, evidence strength, and next-quarter recommendations, usable for the CPO.

Sees one account behind the top ticketRightRightRight
Gemini 4 Argon · API

It clearly shows that Calder Group dominates the 'Add more suppliers' tickets and that delivery/invoice issues are spread across many accounts.

Haiku 5.5 · API

It explicitly identifies that 58 of 75 supplier tickets are from Calder Group and contrasts this with the spread of delivery/invoice issues.

Gemini 3.5 Flash-Lite · Gemini

Identifies Calder Group as the source of most supplier tickets and contrasts with delivery/invoice issues spread across many accounts.

Corrects the draft's claimRightRightRight
Gemini 4 Argon · API

It corrects the headline by counting interviews: only 1 genuinely wanted a marketplace, and most described delivery/invoice pain.

Haiku 5.5 · API

It corrects the draft by noting that 2 of 10 interviews wanted a marketplace and that one of those is hedged, while delivery/invoice pain dominates.

Gemini 3.5 Flash-Lite · Gemini

Corrects the draft's headline by showing marketplace demand is skewed and interviews mostly describe delivery/invoice pain.

Weights behaviour over opinionRightRightRight
Gemini 4 Argon · API

It distinguishes specific past pain (financial losses, late‑night matching) from stated preferences and feature requests, weighting the former.

Haiku 5.5 · API

It distinguishes actual behaviour and specific past pain from stated wishes, labelling the supplier signal as weaker and the delivery/invoice pain as strong.

Gemini 3.5 Flash-Lite · Gemini

Distinguishes stated marketplace wishes from observed operational pain and weights the latter more heavily.

Says how many sources support each findingRightRightRight
Gemini 4 Argon · API

Every main finding is sized (e.g., 8/10 interviews, ticket counts per sub‑category, one‑account dominance).

Haiku 5.5 · API

Every finding is sized with ticket counts or number of interviews; none presents a theme as universal without numbers.

Gemini 3.5 Flash-Lite · Gemini

Each finding includes a count (e.g., '7 out of 10', '22 of those tickets'), even if some numbers are inaccurate.

Frames findings for the decisionRightRightRight
Gemini 4 Argon · API

Findings are framed as opportunities (e.g., Goods‑In workflow) and linked to the Q4 roadmap decision.

Haiku 5.5 · API

Each finding is linked directly to what it means for the next-quarter plan, turning evidence into actionable opportunities.

Gemini 3.5 Flash-Lite · Gemini

Each finding is framed as an opportunity or pain point and linked to the decision about next quarter's build.

Reports what cuts against the planRightRightRight
Gemini 4 Argon · API

It reports that the supplier marketplace signal is a mirage from one account, directly challenging the draft's recommendation.

Haiku 5.5 · API

It challenges the draft's marketplace plan and lists conditions under which the recommendation would change.

Gemini 3.5 Flash-Lite · Gemini

Reports evidence that challenges the marketplace plan and explains what it means for the roadmap.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 71% of 62 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT95.595.82None
2GPT-6.1 SolwithAPI95.592.02None
3GPT-6 LunawithAPI88.684.02None
4Sonnet 5.5withAPI95.567.62None
5Haiku 5.5withAPI86.471.22None
6Opus 5.5withClaude90.963.82None
7Gemini 4 ArgonwithAPI67.267.62None
8Gemini 3.8 FlashwithAPI62.763.12None
9Gemini 3.5 Flash-LitewithGemini63.147.42None

What you get for the cost

GPT-6.1 Sol · API scores within 2 points of GPT-6 Astra · ChatGPT at a ninth of the cost.

Task score against relative cost for each setup. The table below has the same figures.4050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 96, 194× the cheapest (1 of 2 outputs estimated)GPT-6.1 Sol · API: task score 94, 21× the cheapestGPT-6 Luna · API: task score 85, the cheapestSonnet 5.5 · API: task score 80, 62× the cheapestHaiku 5.5 · API: task score 77, 3.4× the cheapestOpus 5.5 · Claude: task score 74, 167× the cheapest (1 of 2 outputs estimated)Gemini 4 Argon · API: task score 67, 25× the cheapestGemini 3.8 Flash · API: task score 62, 19× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 53, 6.5× the cheapest (1 of 2 outputs estimated)GPT-6 Astra · ChatGPTGPT-6.1 Sol · APIGPT-6 Luna · APISonnet 5.5 · APIHaiku 5.5 · APIOpus 5.5 · ClaudeGemini 4 Argon · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API851.0×
Haiku 5.5 · API773.4×
Gemini 3.5 Flash-Lite · Gemini536.5× (partly estimated)
Gemini 3.8 Flash · API6219×
GPT-6.1 Sol · API9421×
Gemini 4 Argon · API6725×
Sonnet 5.5 · API8062×
Opus 5.5 · Claude74167× (partly estimated)
GPT-6 Astra · ChatGPT96194× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Turning a stack of call transcripts into what we actually learned.

Why it matters

Synthesis is where teams fool themselves. A model that smooths away dissent or turns one loud customer into a trend produces confident, wrong roadmaps.

What good looks like

  • Quotes evidence for each theme and counts sources honestly
  • Weights what customers did above what they say they'd do
  • Keeps important dissent visible
  • Labels hypotheses as hypotheses
  • Says what the research cannot tell us

Deliberately not measured

  • Transcript clean-up
  • Persona illustration
Capability tested

Faithful synthesis of qualitative research

The failure we’re looking for

Invents customer consensus or loses important dissent

Grading

Decision model and LLM judge, calibrated against a blind PM review