Tasks / Product Craft

Extract discovery insights

Can the model separate evidence, themes and hypotheses without inventing consensus?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 50% were usable with at most a quick edit.

Reliably right

  1. Frames findings for the decision100% pass
    Each main finding is framed as an opportunity (delivery-to-invoice pain, concentrated supplier need) and linked directly to what to build next quarter.
    GPT-6 Astra · ChatGPT · The loudest ticket
  2. Corrects the draft's claim100% pass
    It corrects the draft's headline by counting interviews (8 describe delivery/invoice pain, 2 mention supplier expansion) and showing the marketplace claim is unsupported.
    GPT-6 Astra · ChatGPT · The loudest ticket
  3. Keeps dissent visible100% pass

Where it slips

  1. Says how many sources support each finding52% pass
    The finding on smaller issues sizes Ordering as 31 tickets, but the actual count is 30, so the finding is not accurately sized.
    Sonnet 5.5 · API · The loudest ticket
  2. Uses the supplied evidence correctly59% pass
    The output states Ordering has 31 tickets, but the CSV contains 30 Ordering tickets; this factual claim is not supported by the supplied data.
    GPT-6.1 Sol · API · The loudest ticket
  3. Avoids unsupported claims63% pass
    The claim 'Ordering (31 tickets)' is presented as an established fact, but the evidence shows 30 Ordering tickets, so it is an unsupported confident claim.
    Sonnet 5.5 · API · The loudest ticket

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Weights behaviour over opinion

    Does the synthesis distinguish evidence of what customers actually did (specific past instances, observed behaviour, workarounds) from stated preferences, predictions and feature requests, and weight them accordingly?

    Passes when Themes are labelled or weighted by evidence type, and at least one theme resting mainly on stated intent or requests is flagged as weaker.

  2. Frames findings for the decision

    Does the synthesis turn its findings into opportunities tied to the decision or outcome the team faces, saying what each one means for that decision?

    Passes when Each main finding is framed as an opportunity (a need or pain to address) and linked to the decision at hand, with what it implies.

  3. Reports what cuts against the plan

    Does the synthesis report the evidence that cuts against the team's current belief or plan, rather than only what supports it?

    Passes when Names at least one finding that challenges the team's hypothesis or plan, and says what it means for it.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Says how many sources support each finding · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for Larder's ordering product. A colleague has drafted the synthesis below for our CPO, recommending a supplier marketplace. Before it goes, rewrite the synthesis from the evidence, in no more than 800 words: the main findings, how strong the evidence for each is, and what they mean for what we build next quarter. The interviews and the ticket export are attached.

What the model was given4 items: About Larder, draft_synthesis.md, interview_excerpts.md, support_tickets_last_quarter.csv
About LarderOrdering software for restaurants: order from your suppliers in one place, track deliveries and invoices. 1,140 accounts, most of them independent restaurants.
draft_synthesis.md6 lines · Download
# Draft synthesis (PM, for the CPO)

Headline: customers want a supplier marketplace.
- "Add more suppliers" is our top support category by far.
- Most customers we spoke to want a marketplace (see Calder Group and the pizzeria).
- Recommendation: build a supplier marketplace next quarter.
interview_excerpts.md12 lines · Download
# Interview excerpts (restaurant owners and managers, September)

1. Bistro, 2 sites: "Last Tuesday the fish order came short again. I only found out when the invoice came, a week later. I spent Sunday night matching delivery notes to invoices with a highlighter."
2. Café group, 5 sites: "We switched produce supplier twice this year, but finding suppliers isn't the problem. Knowing what actually arrived is."
3. Pizzeria, 1 site: "Honestly I'd love more suppliers in the app. More choice, better prices."
4. Gastropub: "My head chef signs the delivery note without checking. Then I pay for things we never got. Last month that was about £600."
5. Hotel restaurant: "Our finance team chases credit notes for short deliveries every week. It takes days."
…
support_tickets_last_quarter.csv200 rows · Download
ticket,account,account_sites,category,summary
LT-5000,Calder Group,14,Add more suppliers,Can you onboard our meat supplier
LT-5001,Account 1886,3,Ordering,Change delivery day
LT-5002,Account 2004,1,Delivery didn't match order,Short delivery not shown in app
LT-5003,Calder Group,14,Add more suppliers,Add dairy supplier please
LT-5004,Calder Group,14,Add more suppliers,Add dairy supplier please
LT-5005,Account 1858,2,Add more suppliers,Any plans for more suppliers?
…
What a strong answer doesThe answer key the graders mark against

Shows that the marketplace headline doesn't hold. 'Add more suppliers' is the biggest ticket category (75 of 200), but 58 of those come from one account, Calder Group; across other accounts, delivery and invoice problems ('delivery didn't match order' and 'invoice reconciliation', 63 tickets) are far more common than supplier requests (17). In the interviews, 7 of 10 describe a specific recent instance of the same pain (short deliveries, invoices that don't match, credit notes, re-typing paper checks), while the marketplace mentions are wishes (the pizzeria 'would love' it, the tapas bar 'I suppose') or come from Calder. Weights what people did over what they say they'd want, counts how many sources support each finding, and recommends focusing next quarter on checking deliveries and reconciling invoices, with the marketplace treated as one large customer's request to handle separately.

Critical failures (cap the score at 40)
  • Recommends building the supplier marketplace next quarter as the main finding
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Usable with a quick edit10 of 12 checks passed

The output is well-structured and evidence-based; a quick edit to add a condition for changing the decision would make it fully usable without re-checking evidence or redoing reasoning.

Output

To: Chief Product Officer From: Product Manager, Ordering Subject: Q4 Product Strategy Synthesis: Debunking the Marketplace Hypothesis

---

Executive Summary

The recommendation in the draft synthesis—to build a supplier marketplace—is based on a misinterpretation of support data and interview signals.

A closer analysis reveals that demand for a discovery marketplace is weak. Instead, the data reveals a critical, systemic gap in our core product: delivery verification and invoice reconciliation. Restaurants are losing hundreds of pounds monthly and spending hours manually reconciling short deliveries, substitutions, and incorrect invoices.

Next quarter, we should not build a marketplace. We should build an end-to-end delivery check-in and automated reconciliation workflow.

---

Key Findings & Evidence Strength

#### 1. Demand for a "Supplier Marketplace" is an artifact of a single customer * Finding: Customers are not asking for supplier discovery. What looks like our highest-volume ticket category is heavily skewed by one account, and the few requests for "more suppliers" reflect a desire to bring existing suppliers onto Larder, not discover new ones. * Evidence Strength: High (Quantitative & Qualitative) * Ticket Skew: "Add more suppliers" represents 75 of 200 tickets (37.5%). However, 58 of those 75 tickets (77%) were submitted by a single customer: Calder Group (14 sites), repeatedly requesting onboarding for four specific existing vendors (Brakes, meat, dairy, wine). * Only 17 other accounts (out of 1,140 total accounts, ~1.5%) submitted requests in this category. * Interviews: Only 1 of 10 respondents (Pizzeria) actively asked for a marketplace for choice/pricing. In contrast, multi-site operators explicitly stated: "Finding suppliers isn't the problem. Knowing what actually arrived is" (Café group) and "I'd use a marketplace if it existed, I suppose. But day to day, what kills me is the paperwork" (Tapas bar). Even Calder Group’s interview makes clear their goal is centralizing their existing supply base onto one system.

#### 2. The critical operational pain is post-order delivery check-in and invoice reconciliation * Finding: Restaurants face daily operational chaos and margin loss between order placement and invoice payment. Deliveries arrive short or substituted, chefs lack a frictionless way to record discrepancies, and finance teams waste days chasing credit notes. * Evidence Strength: Very High (Triangulated across support & interviews) * Support Tickets: Combined, Delivery didn't match order (34 tickets) and Invoice reconciliation (29 tickets) account for 63 tickets across ~60 distinct accounts—nearly 4x the unique account breadth of general supplier requests. Core sub-issues include: "Invoice includes items not delivered", "Short delivery not shown in app", "Credit note missing", and "Export for accountant doesn't show credits". * Interviews: 8 out of 10 interviewees cited this as their primary operational bottleneck. The impact is acute and financial: * Direct losses: Gastropub lost ~£600 last month paying for unchecked deliveries; Steakhouse pays unmatched invoices due to lack of time. * Wasted admin: Hotel finance team spends days chasing credit notes; Bistro owner spends Sundays matching notes with highlighters; Noodle bar re-types paper notes into the app. * Dispute friction: Bakery notes supplier disputes are "my word against theirs."

---

What This Means for Next Quarter

Building a two-sided marketplace would require massive operational lift (supplier acquisition, vetting, liquidity) while failing to solve the core retention and efficiency problems of our existing 1,140 accounts.

Instead, we must complete our core product loop: Order $\rightarrow$ Receive $\rightarrow$ Reconcile.

#### Next Quarter Roadmap Priorities

1. P0: Mobile/Tablet Delivery Check-In Flow * Problem solved: Staff currently check goods on paper clipboards or sign blindly. * Feature: Fast, touch-friendly check-in workflow at the delivery door. Staff can mark line items as accepted, short, rejected, or substituted, with photo capture for damaged goods.

2. P1: Discrepancy & Credit Note Tracking * Problem solved: Finance teams manually chase suppliers for credit notes; owners lack proof of short deliveries. * Feature: Automatically generate and email a discrepancy report to the supplier upon delivery check-in, creating an audit trail and tracking outstanding credit notes until resolved.

3. P2: 3-Way Invoice Reconciliation & Accounting Export * Problem solved: Invoices don't match orders, and exports fail to reflect credits. * Feature: Match PO $\rightarrow$ Delivery Note $\rightarrow$ Final Invoice. Highlight variances automatically and update accounting exports to include verified credits.

4. Fast-Follow / Targeted: Self-Serve Supplier Onboarding Portal * Problem solved: Satisfies Calder Group and similar multi-site accounts wanting their off-platform suppliers integrated. * Feature: A lightweight "Invite Supplier" flow where restaurants send an onboarding link directly to their existing vendors to set up a digital catalog.

Grades and run details

Decision model 88 · LLM judge 85
Decision model checks
  • passUses the supplied evidence correctly18%
  • passAddresses the actual decision44%
  • passRespects explicit constraints46%
  • failIdentifies material uncertainty93%
  • partialAvoids unsupported claims57%
  • passProduces the required deliverable94%
  • passSees one account behind the top ticket99%
  • passCorrects the draft's claim88%
  • passWeights behaviour over opinion100%
  • passSays how many sources support each finding54%
  • passFrames findings for the decision100%
  • passReports what cuts against the plan100%
Run
Run
#1
API response time
36 s
Submitted
6 Oct 2026

Usable with a quick edit11 of 12 checks passed

The output is usable with a quick edit to correct the Ordering ticket count from 31 to 30; the reasoning and recommendation remain sound and do not need redoing.

Output

Synthesis for the CPO: prioritise delivery-to-invoice accuracy, not a marketplace

Recommendation: Next quarter, prioritise a lightweight workflow for recording what arrived, resolving discrepancies and tracking credits. Continue targeted supplier onboarding, but do not commit to a supplier marketplace on this evidence.

Main findings and evidence strength

1. Delivery discrepancies and reconciliation are the clearest recurring customer problem

Eight of ten interviewees describe problems spanning delivery checking, missing items, substitutions, invoice matching, disputes or paperwork. These problems have concrete consequences: the gastropub reports paying roughly £600 for undelivered goods last month; the bistro spends Sunday evening reconciling documents; the hotel’s finance team chases credits weekly. These are self-reported examples, not verified savings estimates.

The ticket export independently supports this pattern:

Support categoryTicketsDistinct accounts within category
------:---:
Add more suppliers7518
Delivery didn’t match order3434
Invoice reconciliation2928
Login and access3131
Ordering3129

Delivery and reconciliation together account for 63 tickets from 62 distinct accounts. Treating them as one related workflow reveals a broader account-level signal than supplier requests.

Evidence strength: Strong evidence of a recurring problem among the customers represented, supported by both interviews and tickets. Moderate evidence for prioritising it across the customer base: ten interviews and support contacts are not representative measures of prevalence among all 1,140 accounts. We also cannot yet tell which part of the workflow offers the greatest benefit.

2. Supplier coverage matters, but ticket volume overstates the breadth of demand

“Add more suppliers” is the largest individual category, at 75 of 200 tickets. However, 58 of those 75 tickets—77%—come from Calder Group, one 14-site account, repeatedly requesting four named suppliers. The remaining 17 tickets come from 17 accounts.

That concentration matters. Repeated requests demonstrate an unresolved, potentially important need for Calder; they do not represent 58 independent customers. Calder is also larger than Larder’s typical independent restaurant account. Its commercial importance should be assessed separately rather than inferred from ticket volume.

Evidence strength: Strong evidence that Calder needs broader coverage, and credible evidence of interest from other accounts. Weak evidence that supplier expansion is the most widespread customer need.

3. Supplier coverage is not the same as a marketplace

The pizzeria explicitly wants more choice and better prices. Calder explicitly requests a marketplace, but its stated goal—ordering from every supplier in one place—could also be addressed by onboarding its existing suppliers. The tapas bar expresses only qualified interest and prioritises paperwork; the café group says finding suppliers is not the problem.

Thus, two interviewees express clear enthusiasm, not “most customers.” Most supplier tickets ask for named existing suppliers or “more suppliers”; they do not establish demand for discovery, comparison or transacting with new suppliers.

Evidence strength: Limited evidence for a marketplace specifically. The evidence does not establish likely adoption, willingness to pay or the supply-side requirements for making it useful.

What we should build next quarter

Make delivery-to-invoice accuracy the main product investment. Begin with a focused pilot covering:

  • Recording actual quantities received, shortages and substitutions at delivery, without retyping clipboard records.
  • Preserving a discrepancy record and supporting evidence that can be shared with suppliers.
  • Showing differences between ordered, received and invoiced items, with outstanding credits tracked through resolution.

Validate the workflow with independent and small multi-site restaurants before scaling. Prioritise the smallest useful end-to-end slice rather than assuming full reconciliation automation is necessary.

Measure delivery-recording adoption, time spent reconciling, unresolved discrepancy age and credit-resolution time against pilot baselines. Validate reported financial benefits rather than extrapolating the £600 anecdote.

Keep supplier coverage as a bounded parallel effort. Assess onboarding Calder’s four requested suppliers and the associated retention or expansion opportunity. Separately test whether other restaurants primarily need existing-supplier coverage or genuinely want to discover and switch suppliers.

Reconsider a marketplace only after broader, account-level research and behavioural tests demonstrate demand beyond Calder. Continue addressing ordering and access issues, each responsible for 31 tickets; this recommendation is a prioritisation decision, not evidence that those basics are solved.

Claims the judge couldn’t find in the brief

  • Ordering category has 31 tickets from 29 distinct accounts.

Grades and run details

Decision model 100 · LLM judge 92
Decision model checks
  • passUses the supplied evidence correctly48%
  • passAddresses the actual decision98%
  • passRespects explicit constraints67%
  • passIdentifies material uncertainty93%
  • passAvoids unsupported claims74%
  • passProduces the required deliverable92%
  • passSees one account behind the top ticket99%
  • passCorrects the draft's claim91%
  • passWeights behaviour over opinion100%
  • passSays how many sources support each finding68%
  • passFrames findings for the decision98%
  • passReports what cuts against the plan100%
Run
Run
#1
API response time
49 s
Submitted
6 Oct 2026

Usable with a quick edit9 of 12 checks passed

The output is clear, well-structured, and only needs a minor correction to the Ordering ticket count (30, not 31) to be fully accurate and usable.

Output

Synthesis for the CPO: what to build next quarter

Headline: The evidence doesn't support a marketplace. It points to the gap between what we order, what arrives and what we're invoiced. I recommend building delivery checking and reconciliation next quarter.

Corrections to the earlier draft

  • "Add more suppliers is our top category by far" is an artefact of one account. Of 200 tickets, 75 are in that category, and 58 of those come from Calder Group, which repeats four requests: Brakes, a meat supplier, a dairy supplier and a wine merchant. The other 17 come from 17 different accounts. Without Calder, it is our smallest category (17 of 142 tickets, 12%).
  • "Most customers want a marketplace" is wrong. Only 2 of 10 interviewees asked for more suppliers (Calder and the pizzeria). The tapas bar was lukewarm ("I suppose") and went straight to paperwork. The café group said finding suppliers "isn't the problem."

Findings and strength of evidence

1. Customers can't easily verify what was delivered against what they ordered and were invoiced for. Strength: strong. - Interviews: 7 of 10 describe this unprompted, and an eighth (tapas bar) says "paperwork" without detail. Examples are short fish orders found a week later, a chef signing notes unchecked (~£600 in a month), finance chasing credit notes, "my word against theirs," clipboard re-typing, and invoices that never match. - Tickets: "Delivery didn't match order" (34) and "Invoice reconciliation" (29) total 63 tickets, 44% of non-Calder volume, from about 62 distinct accounts. The ticket summaries include "short delivery not shown in app," "invoice includes items not delivered," "credit note missing" and "export doesn't show credits." - Interviews and tickets are independent sources and agree. The combined grouping is my judgement, but delivery tickets alone (34) are still twice the non-Calder supplier requests.

2. Generic demand for more suppliers is small and vague. Strength: moderate. - The 17 non-Calder tickets are "any plans for more suppliers?" (8) and "more produce suppliers" (9). They show mild interest, not a pain point. One interviewee (the pizzeria) wants more choice and better prices.

3. Calder Group has a real, specific need. Strength: strong for Calder, weak as a market signal. - Calder is one account with 14 sites, which says it would move all ordering to us with a marketplace. This is a stated, conditional intent. Its tickets name four suppliers, which is an onboarding request, not evidence for a marketplace. We don't yet know Calder's revenue or how many similar multi-site groups we have.

4. Smaller issues: ordering, logins and tablets. Strength: moderate, low priority. - Ordering (31 tickets) and login/access (31 tickets) are mostly routine. "Can't log in on tablet" matters because delivery checking will likely happen on a tablet at the back door.

What this means for next quarter

Build: a receiving and reconciliation flow. - Staff record what actually arrived at the door: shorts, substitutions and weights. - The app flags mismatches against the order and the invoice. - It produces a timestamped record the owner can send to the supplier as a credit-note claim. - Credits flow into the accountant export.

This addresses the main complaints: finding out a week late, the "my word against theirs" disputes, and double entry. It also helps the people who file tickets about short deliveries and missing credit notes.

Don't build: a marketplace. It's a large investment backed by one account and a few vague requests.

Handle Calder separately. Onboard their four named suppliers through our existing process if feasible. Then ask Calder whether that satisfies their "move everything to you" condition, and get their revenue.

Fix before building: tablet login reliability, since the new flow depends on it.

Caveats

  • The sample is 10 interviews from September, and I don't know how they were recruited. Eight of ten raising the same pain is encouraging but not a measured prevalence.
  • Tickets show what people contact support about, not everything that hurts. Customers with delivery problems may just pay the invoice, as the steakhouse does.
  • Ticket data is one quarter, and the `account_sites` values are inconsistent for a few accounts (for example 1418 and 1625), so I haven't segmented by size.
  • The £600 loss is one anecdote. Before committing scope, we should size the cost of short deliveries and unclaimed credits across a sample of accounts.

What would change my view

If we find a meaningful segment of multi-site groups like Calder, with material revenue that is blocked on supplier coverage, supplier breadth becomes a segment-specific bet. It would then be worth testing through targeted onboarding before any full marketplace.

Claims the judge couldn’t find in the brief

  • Ordering (31 tickets) and login/access (31 tickets) are mostly routine.

Grades and run details

Decision model 100 · LLM judge 77
Decision model checks
  • passUses the supplied evidence correctly36%
  • passAddresses the actual decision100%
  • passRespects explicit constraints53%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims56%
  • passProduces the required deliverable95%
  • passSees one account behind the top ticket100%
  • passCorrects the draft's claim99%
  • passWeights behaviour over opinion100%
  • passSays how many sources support each finding87%
  • passFrames findings for the decision99%
  • passReports what cuts against the plan100%
Run
Run
#1
API response time
47 s
Submitted
6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 5

Uses the supplied evidence correctlyRightMixedMixed
Gemini 3.8 Flash · API

All factual claims about the current situation are directly supported by the supplied context or arithmetic from it.

GPT-6.1 Sol · API

The output states Ordering has 31 tickets, but the CSV contains 30 Ordering tickets; this factual claim is not supported by the supplied data.

Sonnet 5.5 · API

The Ordering ticket count is stated as 31, but the CSV contains 30 Ordering tickets, so a factual statement about the current situation is incorrect.

Addresses the actual decisionMixedRightRight
Gemini 3.8 Flash · API

The output commits to a clear recommendation but does not state what result or condition would change that decision.

GPT-6.1 Sol · API

Commits early to prioritising delivery-to-invoice accuracy over a marketplace, and states what would change the call (broader research demonstrating demand beyond Calder).

Sonnet 5.5 · API

Commits early to building delivery checking and reconciliation, and says what would change the view (finding a meaningful segment of multi-site groups like Calder).

Identifies material uncertaintyWrongRightRight
Gemini 3.8 Flash · API

The output does not name specific unknowns that could change the decision or say how they would be resolved.

GPT-6.1 Sol · API

Names unknowns (which part of the workflow offers greatest benefit, adoption/willingness to pay for a marketplace) and says how to resolve them.

Sonnet 5.5 · API

Names unknowns (interview recruitment, ticket bias, inconsistent account_sites, anecdotal £600, need to size cost) and says how they would be resolved or what would change the call.

Avoids unsupported claimsRightRightMixed
Gemini 3.8 Flash · API

Interpretations are clearly tied to evidence, and no confident claims go beyond what the evidence establishes.

GPT-6.1 Sol · API

Interpretations are labelled as such (e.g., self-reported examples, evidence strength ratings), and no unsupported causes or forecasts are presented as fact.

Sonnet 5.5 · API

The claim 'Ordering (31 tickets)' is presented as an established fact, but the evidence shows 30 Ordering tickets, so it is an unsupported confident claim.

Says how many sources support each findingRightRightMixed
Gemini 3.8 Flash · API

Every finding is sized with numbers (8/10 interviewees, 58/75 tickets, 63 tickets across ~60 accounts).

GPT-6.1 Sol · API

Every finding is sized (e.g., 'eight of ten interviewees', '58 of 75 tickets', 'two interviewees').

Sonnet 5.5 · API

The finding on smaller issues sizes Ordering as 31 tickets, but the actual count is 30, so the finding is not accurately sized.

All got right 7

Respects explicit constraintsRightRightRight
Gemini 3.8 Flash · API

The output is a synthesis within 800 words, addressed to the CPO, covering main findings, evidence strength, and next-quarter implications.

GPT-6.1 Sol · API

Delivers a synthesis for the CPO in under 800 words, respecting the requested form and reader.

Sonnet 5.5 · API

The output is a synthesis for the CPO, well under 800 words, and respects the requested form and reader.

Produces the required deliverableRightRightRight
Gemini 3.8 Flash · API

The memo is complete, correctly formatted for the CPO, within the word limit, and actionable with light edits.

GPT-6.1 Sol · API

Provides a complete, actionable synthesis with findings, evidence strength, and a clear next-quarter recommendation.

Sonnet 5.5 · API

The synthesis is complete, in the right form for the CPO, within the word limit, and actionable with light edits.

Sees one account behind the top ticketRightRightRight
Gemini 3.8 Flash · API

The synthesis explicitly counts Calder Group's tickets (58 of 75) and contrasts the spread of delivery/invoice issues across many accounts.

GPT-6.1 Sol · API

Shows that 58 of 75 supplier-request tickets come from Calder Group, while delivery and invoice issues are spread across many accounts.

Sonnet 5.5 · API

Clearly shows that 58 of 75 'Add more suppliers' tickets come from Calder Group, and that delivery/invoice issues are spread across many accounts.

Corrects the draft's claimRightRightRight
Gemini 3.8 Flash · API

It corrects the draft by showing only two interviews mention a marketplace and that most interviews describe delivery/invoice pain.

GPT-6.1 Sol · API

Corrects the draft by counting interviews: only two express clear marketplace enthusiasm, while eight describe delivery/invoice pain.

Sonnet 5.5 · API

Counts the interviews (2 of 10 asked for more suppliers) and corrects the headline, showing that most interviews describe delivery and invoice pain.

Weights behaviour over opinionRightRightRight
Gemini 3.8 Flash · API

The output weights specific past instances (e.g., £600 loss, Sunday reconciliation) over stated wishes, and flags marketplace demand as weak.

GPT-6.1 Sol · API

Distinguishes self-reported specific instances (delivery/invoice problems) from stated preferences (marketplace wishes) and weights them accordingly.

Sonnet 5.5 · API

Distinguishes specific past instances of delivery/invoice pain from stated preferences and feature requests, and flags Calder's need as 'stated, conditional intent'.

Frames findings for the decisionRightRightRight
Gemini 3.8 Flash · API

Findings are framed as opportunities (delivery check-in, reconciliation) and directly linked to the decision not to build a marketplace.

GPT-6.1 Sol · API

Frames findings as opportunities (delivery-to-invoice accuracy, bounded supplier coverage) and links each to the next-quarter decision.

Sonnet 5.5 · API

Each main finding is framed as an opportunity and linked to the decision, with clear implications for what to build next quarter.

Reports what cuts against the planRightRightRight
Gemini 3.8 Flash · API

The output reports evidence that cuts against the marketplace hypothesis, showing it is driven by one account and not the broader user base.

GPT-6.1 Sol · API

Reports evidence that cuts against the marketplace plan (ticket concentration, interview counts, qualified interest) and says what it means.

Sonnet 5.5 · API

Reports evidence that cuts against the marketplace hypothesis, including the concentration of supplier requests in one account and the prevalence of delivery/invoice pain.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 74% of 62 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT95.595.82None
2GPT-6.1 SolwithAPI97.792.02None
3GPT-6 LunawithAPI93.284.02None
4Sonnet 5.5withAPI95.567.62None
5Haiku 5.5withAPI86.471.22None
6Opus 5.5withClaude86.463.82None
7Gemini 3.8 FlashwithAPI71.063.12None
8Gemini 3.5 Flash-LitewithGemini69.347.42None

What you get for the cost

GPT-6.1 Sol · API scores within a point of GPT-6 Astra · ChatGPT at a ninth of the cost.

Task score against relative cost for each setup. The table below has the same figures.50607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 96, 194× the cheapest (1 of 2 outputs estimated)GPT-6.1 Sol · API: task score 95, 21× the cheapestGPT-6 Luna · API: task score 87, the cheapestSonnet 5.5 · API: task score 80, 62× the cheapestHaiku 5.5 · API: task score 77, 3.4× the cheapestOpus 5.5 · Claude: task score 72, 167× the cheapest (1 of 2 outputs estimated)Gemini 3.8 Flash · API: task score 66, 19× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 56, 6.5× the cheapest (1 of 2 outputs estimated)GPT-6 Astra · ChatGPTGPT-6.1 Sol · APIGPT-6 Luna · APISonnet 5.5 · APIHaiku 5.5 · APIOpus 5.5 · ClaudeGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API871.0×
Haiku 5.5 · API773.4×
Gemini 3.5 Flash-Lite · Gemini566.5× (partly estimated)
Gemini 3.8 Flash · API6619×
GPT-6.1 Sol · API9521×
Sonnet 5.5 · API8062×
Opus 5.5 · Claude72167× (partly estimated)
GPT-6 Astra · ChatGPT96194× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Turning a stack of call transcripts into what we actually learned.

Why it matters

Synthesis is where teams fool themselves. A model that smooths away dissent or turns one loud customer into a trend produces confident, wrong roadmaps.

What good looks like

  • Quotes evidence for each theme and counts sources honestly
  • Weights what customers did above what they say they'd do
  • Keeps important dissent visible
  • Labels hypotheses as hypotheses
  • Says what the research cannot tell us

Deliberately not measured

  • Transcript clean-up
  • Persona illustration
Capability tested

Faithful synthesis of qualitative research

The failure we’re looking for

Invents customer consensus or loses important dissent

Grading

Decision model and LLM judge, calibrated against a blind PM review