Tasks / Product Craft

Extract discovery insights

Can the model separate evidence, themes and hypotheses without inventing consensus?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 50% were usable with at most a quick edit.

Reliably right

  1. Frames findings for the decision100% pass
    Each main finding is framed as an opportunity (delivery-to-invoice pain, concentrated supplier need) and linked directly to what to build next quarter.
    GPT-6 Astra · ChatGPT · The loudest ticket
  2. Corrects the draft's claim100% pass
    It corrects the draft's headline by counting interviews (8 describe delivery/invoice pain, 2 mention supplier expansion) and showing the marketplace claim is unsupported.
    GPT-6 Astra · ChatGPT · The loudest ticket
  3. Keeps dissent visible100% pass

Where it slips

  1. Says how many sources support each finding52% pass
    The finding on smaller issues sizes Ordering as 31 tickets, but the actual count is 30, so the finding is not accurately sized.
    Sonnet 5.5 · API · The loudest ticket
  2. Uses the supplied evidence correctly59% pass
    The output states Ordering has 31 tickets, but the CSV contains 30 Ordering tickets; this factual claim is not supported by the supplied data.
    GPT-6.1 Sol · API · The loudest ticket
  3. Avoids unsupported claims63% pass
    The claim 'Ordering (31 tickets)' is presented as an established fact, but the evidence shows 30 Ordering tickets, so it is an unsupported confident claim.
    Sonnet 5.5 · API · The loudest ticket

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Weights behaviour over opinion

    Does the synthesis distinguish evidence of what customers actually did (specific past instances, observed behaviour, workarounds) from stated preferences, predictions and feature requests, and weight them accordingly?

    Passes when Themes are labelled or weighted by evidence type, and at least one theme resting mainly on stated intent or requests is flagged as weaker.

  2. Frames findings for the decision

    Does the synthesis turn its findings into opportunities tied to the decision or outcome the team faces, saying what each one means for that decision?

    Passes when Each main finding is framed as an opportunity (a need or pain to address) and linked to the decision at hand, with what it implies.

  3. Reports what cuts against the plan

    Does the synthesis report the evidence that cuts against the team's current belief or plan, rather than only what supports it?

    Passes when Names at least one finding that challenges the team's hypothesis or plan, and says what it means for it.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Says how many sources support each finding · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for Larder's ordering product. A colleague has drafted the synthesis below for our CPO, recommending a supplier marketplace. Before it goes, rewrite the synthesis from the evidence, in no more than 800 words: the main findings, how strong the evidence for each is, and what they mean for what we build next quarter. The interviews and the ticket export are attached.

What the model was given4 items: About Larder, draft_synthesis.md, interview_excerpts.md, support_tickets_last_quarter.csv
About LarderOrdering software for restaurants: order from your suppliers in one place, track deliveries and invoices. 1,140 accounts, most of them independent restaurants.
draft_synthesis.md6 lines · Download
# Draft synthesis (PM, for the CPO)

Headline: customers want a supplier marketplace.
- "Add more suppliers" is our top support category by far.
- Most customers we spoke to want a marketplace (see Calder Group and the pizzeria).
- Recommendation: build a supplier marketplace next quarter.
interview_excerpts.md12 lines · Download
# Interview excerpts (restaurant owners and managers, September)

1. Bistro, 2 sites: "Last Tuesday the fish order came short again. I only found out when the invoice came, a week later. I spent Sunday night matching delivery notes to invoices with a highlighter."
2. Café group, 5 sites: "We switched produce supplier twice this year, but finding suppliers isn't the problem. Knowing what actually arrived is."
3. Pizzeria, 1 site: "Honestly I'd love more suppliers in the app. More choice, better prices."
4. Gastropub: "My head chef signs the delivery note without checking. Then I pay for things we never got. Last month that was about £600."
5. Hotel restaurant: "Our finance team chases credit notes for short deliveries every week. It takes days."
…
support_tickets_last_quarter.csv200 rows · Download
ticket,account,account_sites,category,summary
LT-5000,Calder Group,14,Add more suppliers,Can you onboard our meat supplier
LT-5001,Account 1886,3,Ordering,Change delivery day
LT-5002,Account 2004,1,Delivery didn't match order,Short delivery not shown in app
LT-5003,Calder Group,14,Add more suppliers,Add dairy supplier please
LT-5004,Calder Group,14,Add more suppliers,Add dairy supplier please
LT-5005,Account 1858,2,Add more suppliers,Any plans for more suppliers?
…
What a strong answer doesThe answer key the graders mark against

Shows that the marketplace headline doesn't hold. 'Add more suppliers' is the biggest ticket category (75 of 200), but 58 of those come from one account, Calder Group; across other accounts, delivery and invoice problems ('delivery didn't match order' and 'invoice reconciliation', 63 tickets) are far more common than supplier requests (17). In the interviews, 7 of 10 describe a specific recent instance of the same pain (short deliveries, invoices that don't match, credit notes, re-typing paper checks), while the marketplace mentions are wishes (the pizzeria 'would love' it, the tapas bar 'I suppose') or come from Calder. Weights what people did over what they say they'd want, counts how many sources support each finding, and recommends focusing next quarter on checking deliveries and reconciling invoices, with the marketplace treated as one large customer's request to handle separately.

Critical failures (cap the score at 40)
  • Recommends building the supplier marketplace next quarter as the main finding
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Needs more than a quick edit11 of 12 checks passed

The output fails to show that delivery and invoice issues are spread across many accounts, a key comparison a PM would need to add by re-checking the ticket data.

Output

Synthesis for the CPO

Headline

The evidence points more strongly to a delivery-reconciliation problem than to a general marketplace need. Supplier coverage matters—especially to Calder Group—but the interviews suggest the bigger day-to-day pain is knowing what arrived, resolving discrepancies, and matching invoices and credits. Build toward that workflow next quarter; treat a broad marketplace as an unproven solution.

Findings and evidence

1. Receiving and invoice reconciliation are recurring operational pains. — Moderate evidence Eight of the ten interview excerpts raise delivery checking, discrepancies, invoice matching, credit notes, or the paperwork around them. Examples include paying for undelivered goods, learning about short deliveries only when the invoice arrives, and re-entering clipboard checks into Larder. Several customers describe financial impact or repeated manual work. The support export also contains tickets in both “Delivery didn’t match order” and “Invoice reconciliation.”

This is a consistent signal across the interviews, but ten excerpts cannot establish how common the problem is across Larder’s 1,140 accounts or which part of the workflow is most valuable to solve first.

2. Supplier availability is a genuine need, but the ticket count overstates its breadth. — Moderate evidence for demand; weak evidence for a marketplace “Add more suppliers” is the largest ticket category: 75 of 200 tickets. But 58 of those tickets—most of the category—come from one customer, Calder Group. The remaining 17 are from other accounts, so supplier coverage is not solely a Calder concern, but ticket volume is not evidence that most customers want a marketplace. Calder’s 14 sites and stated intent to move all ordering to Larder make its request strategically important; it is still one account.

The interviews echo that distinction: the pizzeria wants more choice and better prices, and Calder explicitly wants every supplier on-platform. But another multi-site operator says finding suppliers is not the problem, and the tapas bar says paperwork is what hurts day to day.

3. A marketplace is a proposed solution, not a validated priority. — Weak evidence The evidence supports specific supplier gaps and some appetite for choice, but does not show that customers broadly want a marketplace, what marketplace model they mean, or whether it would resolve their more frequent receiving and reconciliation problems. The draft’s conclusion goes beyond what these sources can establish.

What to build next quarter

Prioritize a focused delivery-to-invoice reconciliation improvement: make it easier to record what was received, flag shortages and substitutions against the order, and carry discrepancies through to invoice matching and credit-note tracking. Test the workflow with customers who describe these pains before committing to a larger build.

In parallel, follow up with Calder and a few other customers requesting suppliers. Assess targeted supplier onboarding or integrations for the specific suppliers they need; do not assume this requires a general-purpose marketplace. Use the quarter to measure how widespread each problem is across distinct accounts and whether improved reconciliation reduces manual work and unresolved discrepancies.

Grades and run details

Decision model 100 · LLM judge 85
Decision model checks
  • passUses the supplied evidence correctly62%
  • passAddresses the actual decision61%
  • passRespects explicit constraints77%
  • passIdentifies material uncertainty45%
  • passAvoids unsupported claims78%
  • passProduces the required deliverable78%
  • passSees one account behind the top ticket67%
  • passCorrects the draft's claim53%
  • passWeights behaviour over opinion99%
  • passSays how many sources support each finding55%
  • passFrames findings for the decision90%
  • passReports what cuts against the plan100%
Run
Run
#1
API response time
21 s
Submitted
5 Oct 2026

Usable with a quick edit12 of 12 checks passed

The synthesis is clear, evidence-based, and directly usable by a product manager with at most light editing for emphasis or wording.

Output

Next-quarter synthesis: prioritise delivery-to-invoice accuracy, not a marketplace

Recommendation: Build and validate a focused workflow for recording what arrived, flagging discrepancies and resolving invoice/credit issues. Continue targeted supplier onboarding, but do not commit next quarter to a broad marketplace. The evidence supports operational accuracy more broadly than supplier discovery.

Main findings and evidence strength

1. Supplier requests lead ticket volume—but are heavily concentrated in one account

The export contains 200 tickets:

CategoryTicketsDistinct accounts
------:---:
Add more suppliers7518
Delivery didn’t match order3434
Invoice reconciliation2928
Ordering3129
Login and access3131

Account counts are deduplicated within categories; they should not be summed across categories.

Calder Group generated 58 of the 75 supplier tickets (77%), repeatedly requesting onboarding for four named suppliers or supplier types. The other 17 supplier tickets came from 17 accounts.

Evidence strength: Strong evidence of a concentrated supplier-coverage need; weak evidence that a marketplace is the right solution. Ticket volume measures support demand, not independent customer votes. Repeated requests may indicate an important unresolved problem, but not broad market demand.

Calder’s 14 sites and stated willingness to move all ordering to Larder warrant commercial follow-up. However, this is a conditional intention—not a commitment—and we have no revenue or order-volume evidence to size the opportunity. Requests to add existing suppliers must also be distinguished from discovering new ones.

2. Delivery and invoice problems affect a broader set of accounts

Together, delivery discrepancies and invoice reconciliation account for 63 tickets from 62 distinct accounts, compared with supplier requests from 18 accounts.

The interviews reinforce this pattern. Eight of ten excerpts describe delivery, reconciliation or paperwork pain, although the tapas bar’s comment is less specific. Examples include:

  • Discovering a short delivery only when the invoice arrives.
  • Paying for undelivered goods; one gastropub reported approximately £600 in one month.
  • Chasing credit notes weekly.
  • Having no shared evidence when suppliers dispute shortages.
  • Recording deliveries on paper, then re-entering them in Larder.

The café group explicitly says finding suppliers is not the problem; knowing what arrived is.

Evidence strength: Strong convergence across these two sources that this is a meaningful problem spanning multiple restaurant formats. Evidence for its prevalence across all 1,140 accounts, typical financial impact or the best solution remains limited. The £600 figure is one customer’s report, not an estimate we can extrapolate.

3. Marketplace interest exists, but “most customers want one” is unsupported

The pizzeria strongly wants more choice and better prices. Calder explicitly mentions a marketplace, but frames its need as getting every supplier onto the platform. The tapas bar expresses tentative interest, then prioritises paperwork.

Evidence strength: Directional evidence for supplier expansion, but weak evidence for broad marketplace demand. Only Calder explicitly advocates a marketplace; the pizzeria’s desired outcome could potentially be served in other ways. These excerpts do not establish expected adoption, switching behaviour or willingness to pay.

What we should build next quarter

Prioritise a narrow delivery-to-invoice pilot, rather than attempting full reconciliation automation immediately:

  1. Capture actual receipt against an order: quantities received, shortages and substitutions, without duplicate entry.
  2. Carry discrepancies forward: make them visible when checking invoices rather than leaving users to reconstruct events.
  3. Track resolution: preserve a discrepancy record and show outstanding supplier responses or credits.

Start with observing receiving and invoice-checking in a varied set of independent restaurants. In particular, test whether busy staff will actually record discrepancies: the unchecked delivery-note example suggests software alone may not solve the problem.

Measure receiving-workflow adoption, duplicate entry, reconciliation time, discrepancies identified before payment and time to resolve credits. Establish baselines before setting numerical success targets; use pilot results to decide how far to expand.

Keep supplier coverage as a separate, bounded workstream. Clarify Calder’s required suppliers, onboarding feasibility and commercial commitment. Test targeted onboarding before assuming we need marketplace discovery, comparison and transaction capabilities.

Confidence and limits: This is a prioritisation recommendation, not a proven business case. Ten interview excerpts and self-selected support tickets cannot establish population prevalence. Some accounts also have inconsistent site counts in the export, so site-weighted conclusions would be unreliable. Validate the proposed workflow and its value before a broad rollout.

Grades and run details

Decision model 100 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly47%
  • passAddresses the actual decision88%
  • passRespects explicit constraints56%
  • passIdentifies material uncertainty86%
  • passAvoids unsupported claims84%
  • passProduces the required deliverable88%
  • passSees one account behind the top ticket100%
  • passCorrects the draft's claim68%
  • passWeights behaviour over opinion100%
  • passSays how many sources support each finding75%
  • passFrames findings for the decision98%
  • passReports what cuts against the plan100%
Run
Run
#1
Time to output
48 s
Submitted
5 Oct 2026

Needs more than a quick edit9 of 12 checks passed

Factual errors in ticket counts (ordering 31 vs 30, credits export 8 vs 10) and word count exceedance mean a PM would need to re-check evidence and edit for length.

Output

Synthesis: what customers need from Larder next (for the CPO)

Headline: The broadest and best-evidenced problem is that customers can't easily tell what actually arrived, and it costs them money and hours every week. Demand for "more suppliers" is real, but nearly all of it comes from one account. That account wants its own four suppliers connected, which is not the same as a marketplace. Next quarter we should build delivery check-in and discrepancy tracking, and handle Calder Group's request as a bounded onboarding project.

Finding 1: Delivery and invoice mismatches are the most widespread pain

Evidence: strong. It is consistent across both sources, comes from many accounts, and customers describe concrete costs.

  • Tickets: "Delivery didn't match order" (34) and "Invoice reconciliation" (29) describe the same problem from two ends: ordered, received and invoiced don't line up. Together they account for 63 of 200 tickets (32%) from about 62 different accounts. By number of customers affected, this is our largest problem.
  • Interviews: 8 of 10 describe it. Examples:
  • About £600 lost in a month on undelivered items (gastropub).
  • A finance team spending days every week chasing credit notes (hotel).
  • Sunday nights reconciling invoices with a highlighter (bistro).
  • Disputes with suppliers that can't be won without proof (bakery).
  • Checking deliveries on a clipboard, then re-typing everything (noodle bar).
  • Paying mismatched invoices because there's no time to check (steakhouse).

Finding 2: "Add more suppliers" is the top ticket category, but it is essentially one customer

Evidence: strong that the demand is concentrated; weak that there is broad demand for a marketplace.

- Calder Group's tickets: 75 tickets (37.5%) fall in this category, but 58 come from Calder Group alone. They repeatedly ask for the same four suppliers: - Brakes (15 tickets) - Their meat supplier (15) - Their dairy supplier (14) - Their wine merchant (14) They want suppliers they already use connected to Larder. They are not asking to discover new suppliers. - Everyone else: The remaining 17 tickets come from 17 accounts and are mostly generic ("Any plans for more suppliers?", "Would like more produce suppliers"). - Interviews: Only 2 of 10 clearly want this (the pizzeria and Calder). The tapas bar is lukewarm ("I suppose") and says paperwork is the real problem. The café group says outright that "finding suppliers isn't the problem."

The earlier claim that "most customers we spoke to want a marketplace" is not supported.

Finding 3: Calder is a real commercial opportunity of a specific kind

Evidence: moderate. This is one account's clear statement, untested.

  • Calder has 14 sites and says it would move all its ordering to us if its suppliers were on the platform.
  • We don't yet know how much of its ordering runs through Larder today or what the account is worth. We need that before sizing the effort.
  • Their need is met by onboarding four named suppliers, not by building a marketplace.

Other signals

  • Login and access (31 tickets) and ordering questions (31) look routine.
  • One subset matters for the recommendation below: 10 tickets are "can't log in on tablet," and the tablet is where delivery check-in would happen.

Limits of this evidence

  • Small interview sample. There are 10 interviews and we don't know how participants were chosen.
  • One quarter of tickets. Ticket volume measures who contacts support, not who has the problem. Customers who silently pay wrong invoices, like the steakhouse, won't file tickets, so mismatch pain is probably under-counted.
  • Data quality. Support staff assign the categories, and a few accounts show inconsistent site counts across tickets.
  • No usage or revenue data yet. We can't weigh Calder against the long tail of smaller accounts.

What this means for next quarter

1. Build delivery check-in and discrepancy tracking. This is our core promise ("track deliveries and invoices") and is where the broad pain sits. Scope: - Check a delivery against the order at the door, on phone or tablet, replacing the clipboard and the re-typing. - Record shorts, substitutions and short weights, with a photo and timestamp to use as evidence in disputes. - Flag invoice lines that don't match what was received. - Raise credit notes and track them until they arrive. - Show credits in the accountant export (8 tickets ask for this). - Fix tablet login as part of the work.

Success measures: - Share of deliveries checked in. - £ of discrepancies recorded and credited. - Fewer tickets in the two mismatch categories. - Time spent on reconciliation, re-checked with these interviewees.

2. Treat Calder as an account project, not a platform bet. - Work with sales to confirm the account's value and how much of its ordering would move. - Then scope onboarding of the four requested suppliers. This is bounded work that doesn't require a marketplace.

3. Don't build a marketplace next quarter. - If we want to test demand, do it cheaply. Follow up with the 17 non-Calder accounts or run a fake-door test in the app. - Revisit with that data.

Claims the judge couldn’t find in the brief

  • Login and access tickets: 31; ordering questions: 31.
  • Show credits in the accountant export (8 tickets ask for this).

Grades and run details

Decision model 100 · LLM judge 69
Decision model checks
  • passUses the supplied evidence correctly39%
  • passAddresses the actual decision95%
  • passRespects explicit constraints52%
  • passIdentifies material uncertainty92%
  • passAvoids unsupported claims47%
  • passProduces the required deliverable95%
  • passSees one account behind the top ticket100%
  • passCorrects the draft's claim98%
  • passWeights behaviour over opinion100%
  • passSays how many sources support each finding72%
  • passFrames findings for the decision99%
  • passReports what cuts against the plan100%
Run
Run
#1
Time to output
72 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Uses the supplied evidence correctlyRightRightMixed
GPT-6 Luna · API

All factual claims about the current situation are directly supported by the supplied interview excerpts or ticket data.

GPT-6 Astra · ChatGPT

All claims about the current situation are directly supported by the supplied context or derived by correct arithmetic.

Opus 5.5 · Claude

Two factual numbers are wrong: ordering tickets are 30 not 31, and credits export tickets are 10 not 8.

Respects explicit constraintsRightRightMixed
GPT-6 Luna · API

The output is a synthesis under 800 words, includes main findings with evidence strength, and states what to build next quarter, all for the CPO.

GPT-6 Astra · ChatGPT

The output is a synthesis under 800 words, includes main findings with evidence strength, and says what to build next quarter, as requested.

Opus 5.5 · Claude

The output exceeds the 800-word limit (approximately 826 words).

Produces the required deliverableRightRightMixed
GPT-6 Luna · API

The synthesis is complete, within 800 words, addressed to the CPO, and actionable with findings, evidence strength, and a next-quarter plan.

GPT-6 Astra · ChatGPT

The synthesis is complete, in the right form for the CPO, within the word limit, and actionable without major gaps.

Opus 5.5 · Claude

The synthesis is present but exceeds the 800-word limit, so it is not within the requested length.

Sees one account behind the top ticketMixedRightRight
GPT-6 Luna · API

The output counts Calder's tickets and notes the remaining 17 are from others, but it does not show that delivery and invoice issues are spread across many accounts, only that those categories exist in the export.

GPT-6 Astra · ChatGPT

It counts tickets by account, shows Calder Group is 58 of 75 supplier tickets, and contrasts with delivery/invoice issues spread across many accounts.

Opus 5.5 · Claude

Clearly shows Calder Group accounts for 58 of 75 supplier tickets, while delivery/invoice issues are spread across many accounts.

All got right 8

Addresses the actual decisionRightRightRight
GPT-6 Luna · API

The output commits to a clear recommendation (prioritize delivery-to-invoice reconciliation, treat marketplace as unproven) and says to measure problem spread to inform future decisions.

GPT-6 Astra · ChatGPT

The output commits to a clear recommendation (prioritise delivery-to-invoice accuracy, not a marketplace) and states that validation results would change the call.

Opus 5.5 · Claude

Commits to building delivery check-in, treating Calder as an account project, and not building a marketplace, with conditions to revisit.

Identifies material uncertaintyRightRightRight
GPT-6 Luna · API

It names unknowns (how common the problem is across all accounts, which part of the workflow is most valuable, what marketplace model) and says to test and measure to resolve them.

GPT-6 Astra · ChatGPT

It names specific unknowns (revenue impact, adoption, prevalence across all accounts) and says how to resolve them through pilot and measurement.

Opus 5.5 · Claude

Names specific unknowns (sample size, ticket bias, data quality, no revenue data) and how to resolve them.

Avoids unsupported claimsRightRightRight
GPT-6 Luna · API

Interpretations are clearly tied to evidence, and no confident claim goes beyond what the supplied data establishes.

GPT-6 Astra · ChatGPT

Interpretations and forecasts are clearly labelled as such, and confident claims are limited to what the evidence establishes.

Opus 5.5 · Claude

Hypotheses and forecasts are labelled (e.g., 'untested', 'probably under-counted'), and confident claims are supported.

Corrects the draft's claimRightRightRight
GPT-6 Luna · API

It corrects the draft by counting eight interviews on delivery/invoice pain versus two on marketplace, and shows the draft's headline rests on a minority of interviews.

GPT-6 Astra · ChatGPT

It corrects the draft's headline by counting interviews (8 describe delivery/invoice pain, 2 mention supplier expansion) and showing the marketplace claim is unsupported.

Opus 5.5 · Claude

Corrects the draft by counting interviews: only 2 of 10 want a marketplace, and 8 describe delivery/invoice pain.

Weights behaviour over opinionRightRightRight
GPT-6 Luna · API

The output flags the marketplace theme as 'weak evidence' and a 'proposed solution, not a validated priority,' while the delivery/invoice pains are backed by specific instances and financial impact.

GPT-6 Astra · ChatGPT

It distinguishes specific past instances and workarounds (delivery/invoice pain) from stated preferences (marketplace wishes) and weights the former more heavily.

Opus 5.5 · Claude

Weights concrete costs and observed behaviour (money lost, time spent) over stated wishes ('would love', 'I suppose').

Says how many sources support each findingRightRightRight
GPT-6 Luna · API

Findings are sized: eight of ten interviews, 75 of 200 tickets, 58 from Calder, 17 from others; no finding is presented as universal without a count.

GPT-6 Astra · ChatGPT

Every finding is sized with ticket counts, account counts, or interview counts, and nothing is presented as universal without support.

Opus 5.5 · Claude

Every finding is sized with numbers (8 of 10 interviews, 63 tickets, 58 from Calder, 17 from 17 accounts).

Frames findings for the decisionRightRightRight
GPT-6 Luna · API

Each finding is linked to the decision: the reconciliation pain drives the build priority, the supplier concentration shapes the Calder follow-up, and the weak marketplace evidence justifies not building it.

GPT-6 Astra · ChatGPT

Each main finding is framed as an opportunity (delivery-to-invoice pain, concentrated supplier need) and linked directly to what to build next quarter.

Opus 5.5 · Claude

Each finding is framed as an opportunity and linked to the decision (build check-in, don't build marketplace, treat Calder as account project).

Reports what cuts against the planRightRightRight
GPT-6 Luna · API

It reports that the top ticket category is driven by one account and that the marketplace is an unvalidated solution, directly challenging the draft's plan.

GPT-6 Astra · ChatGPT

It reports that supplier requests are concentrated in one account and that the café group says finding suppliers isn't the problem, both cutting against the marketplace plan.

Opus 5.5 · Claude

Reports evidence against the marketplace plan: concentration in one account, interviews mostly about mismatch, café group's statement.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 74% of 62 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT95.595.82None
2GPT-6.1 SolwithAPI97.792.02None
3GPT-6 LunawithAPI93.284.02None
4Sonnet 5.5withAPI95.567.62None
5Haiku 5.5withAPI86.471.22None
6Opus 5.5withClaude86.463.82None
7Gemini 3.8 FlashwithAPI71.063.12None
8Gemini 3.5 Flash-LitewithGemini69.347.42None

What you get for the cost

GPT-6.1 Sol · API scores within a point of GPT-6 Astra · ChatGPT at a ninth of the cost.

Task score against relative cost for each setup. The table below has the same figures.50607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 96, 194× the cheapest (1 of 2 outputs estimated)GPT-6.1 Sol · API: task score 95, 21× the cheapestGPT-6 Luna · API: task score 87, the cheapestSonnet 5.5 · API: task score 80, 62× the cheapestHaiku 5.5 · API: task score 77, 3.4× the cheapestOpus 5.5 · Claude: task score 72, 167× the cheapest (1 of 2 outputs estimated)Gemini 3.8 Flash · API: task score 66, 19× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 56, 6.5× the cheapest (1 of 2 outputs estimated)GPT-6 Astra · ChatGPTGPT-6.1 Sol · APIGPT-6 Luna · APISonnet 5.5 · APIHaiku 5.5 · APIOpus 5.5 · ClaudeGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API871.0×
Haiku 5.5 · API773.4×
Gemini 3.5 Flash-Lite · Gemini566.5× (partly estimated)
Gemini 3.8 Flash · API6619×
GPT-6.1 Sol · API9521×
Sonnet 5.5 · API8062×
Opus 5.5 · Claude72167× (partly estimated)
GPT-6 Astra · ChatGPT96194× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Turning a stack of call transcripts into what we actually learned.

Why it matters

Synthesis is where teams fool themselves. A model that smooths away dissent or turns one loud customer into a trend produces confident, wrong roadmaps.

What good looks like

  • Quotes evidence for each theme and counts sources honestly
  • Weights what customers did above what they say they'd do
  • Keeps important dissent visible
  • Labels hypotheses as hypotheses
  • Says what the research cannot tell us

Deliberately not measured

  • Transcript clean-up
  • Persona illustration
Capability tested

Faithful synthesis of qualitative research

The failure we’re looking for

Invents customer consensus or loses important dissent

Grading

Decision model and LLM judge, calibrated against a blind PM review