Humanity’s Last PRD
Latest drop: Haiku 5.5 · 8 Oct
Tasks / Product Craft

Extract discovery insights

Can the model separate evidence, themes and hypotheses without inventing consensus?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 50% were usable with at most a quick edit.

Reliably right

  1. Frames findings for the decision100% pass
    Each main finding is framed as an opportunity (delivery-to-invoice pain, concentrated supplier need) and linked directly to what to build next quarter.
    GPT-6 Astra · ChatGPT · The loudest ticket
  2. Corrects the draft's claim100% pass
    It corrects the draft's headline by counting interviews (8 describe delivery/invoice pain, 2 mention supplier expansion) and showing the marketplace claim is unsupported.
    GPT-6 Astra · ChatGPT · The loudest ticket
  3. Keeps dissent visible100% pass

Where it slips

  1. Says how many sources support each finding52% pass
    The finding on smaller issues sizes Ordering as 31 tickets, but the actual count is 30, so the finding is not accurately sized.
    Sonnet 5.5 · API · The loudest ticket
  2. Uses the supplied evidence correctly59% pass
    The output states Ordering has 31 tickets, but the CSV contains 30 Ordering tickets; this factual claim is not supported by the supplied data.
    GPT-6.1 Sol · API · The loudest ticket
  3. Avoids unsupported claims63% pass
    The claim 'Ordering (31 tickets)' is presented as an established fact, but the evidence shows 30 Ordering tickets, so it is an unsupported confident claim.
    Sonnet 5.5 · API · The loudest ticket

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Weights behaviour over opinion

    Does the synthesis distinguish evidence of what customers actually did (specific past instances, observed behaviour, workarounds) from stated preferences, predictions and feature requests, and weight them accordingly?

    Passes when Themes are labelled or weighted by evidence type, and at least one theme resting mainly on stated intent or requests is flagged as weaker.

    Bob MoestaLenny’s Podcast

  2. Frames findings for the decision

    Does the synthesis turn its findings into opportunities tied to the decision or outcome the team faces, saying what each one means for that decision?

    Passes when Each main finding is framed as an opportunity (a need or pain to address) and linked to the decision at hand, with what it implies.

    Teresa TorresLenny’s Podcast

  3. Reports what cuts against the plan

    Does the synthesis report the evidence that cuts against the team's current belief or plan, rather than only what supports it?

    Passes when Names at least one finding that challenges the team's hypothesis or plan, and says what it means for it.

    Judd AntinLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Says how many sources support each finding · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for Larder's ordering product. A colleague has drafted the synthesis below for our CPO, recommending a supplier marketplace. Before it goes, rewrite the synthesis from the evidence, in no more than 800 words: the main findings, how strong the evidence for each is, and what they mean for what we build next quarter. The interviews and the ticket export are attached.

What the model was given4 items: About Larder, draft_synthesis.md, interview_excerpts.md, support_tickets_last_quarter.csv
About LarderOrdering software for restaurants: order from your suppliers in one place, track deliveries and invoices. 1,140 accounts, most of them independent restaurants.
draft_synthesis.md6 lines · Download
# Draft synthesis (PM, for the CPO)

Headline: customers want a supplier marketplace.
- "Add more suppliers" is our top support category by far.
- Most customers we spoke to want a marketplace (see Calder Group and the pizzeria).
- Recommendation: build a supplier marketplace next quarter.
interview_excerpts.md12 lines · Download
# Interview excerpts (restaurant owners and managers, September)

1. Bistro, 2 sites: "Last Tuesday the fish order came short again. I only found out when the invoice came, a week later. I spent Sunday night matching delivery notes to invoices with a highlighter."
2. Café group, 5 sites: "We switched produce supplier twice this year, but finding suppliers isn't the problem. Knowing what actually arrived is."
3. Pizzeria, 1 site: "Honestly I'd love more suppliers in the app. More choice, better prices."
4. Gastropub: "My head chef signs the delivery note without checking. Then I pay for things we never got. Last month that was about £600."
5. Hotel restaurant: "Our finance team chases credit notes for short deliveries every week. It takes days."
…
support_tickets_last_quarter.csv200 rows · Download
ticket,account,account_sites,category,summary
LT-5000,Calder Group,14,Add more suppliers,Can you onboard our meat supplier
LT-5001,Account 1886,3,Ordering,Change delivery day
LT-5002,Account 2004,1,Delivery didn't match order,Short delivery not shown in app
LT-5003,Calder Group,14,Add more suppliers,Add dairy supplier please
LT-5004,Calder Group,14,Add more suppliers,Add dairy supplier please
LT-5005,Account 1858,2,Add more suppliers,Any plans for more suppliers?
…
What a strong answer doesThe answer key the graders mark against

Shows that the marketplace headline doesn't hold. 'Add more suppliers' is the biggest ticket category (75 of 200), but 58 of those come from one account, Calder Group; across other accounts, delivery and invoice problems ('delivery didn't match order' and 'invoice reconciliation', 63 tickets) are far more common than supplier requests (17). In the interviews, 7 of 10 describe a specific recent instance of the same pain (short deliveries, invoices that don't match, credit notes, re-typing paper checks), while the marketplace mentions are wishes (the pizzeria 'would love' it, the tapas bar 'I suppose') or come from Calder. Weights what people did over what they say they'd want, counts how many sources support each finding, and recommends focusing next quarter on checking deliveries and reconciling invoices, with the marketplace treated as one large customer's request to handle separately.

Critical failures (cap the score at 40)
  • Recommends building the supplier marketplace next quarter as the main finding
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

GPT-6 AstraChatGPT100Usable with a quick editGPT-6.1 SolAPI96Usable with a quick editHaiku 5.5API96Usable with a quick editGPT-6 LunaAPI88Needs more workSonnet 5.5API88Usable with a quick editGemini 3.8 FlashAPI87Usable with a quick editOpus 5.5Claude81Needs more workGemini 3.5 Flash-LiteGemini65Needs more work
GPT-6 LunaAPI

Needs more than a quick edit11 of 12 checks passed

The output fails to show that delivery and invoice issues are spread across many accounts, a key comparison a PM would need to add by re-checking the ticket data.

Output

Synthesis for the CPO

Headline

The evidence points more strongly to a delivery-reconciliation problem than to a general marketplace need. Supplier coverage matters—especially to Calder Group—but the interviews suggest the bigger day-to-day pain is knowing what arrived, resolving discrepancies, and matching invoices and credits. Build toward that workflow next quarter; treat a broad marketplace as an unproven solution.

Findings and evidence

1. Receiving and invoice reconciliation are recurring operational pains. — Moderate evidence Eight of the ten interview excerpts raise delivery checking, discrepancies, invoice matching, credit notes, or the paperwork around them. Examples include paying for undelivered goods, learning about short deliveries only when the invoice arrives, and re-entering clipboard checks into Larder. Several customers describe financial impact or repeated manual work. The support export also contains tickets in both “Delivery didn’t match order” and “Invoice reconciliation.”

This is a consistent signal across the interviews, but ten excerpts cannot establish how common the problem is across Larder’s 1,140 accounts or which part of the workflow is most valuable to solve first.

2. Supplier availability is a genuine need, but the ticket count overstates its breadth. — Moderate evidence for demand; weak evidence for a marketplace “Add more suppliers” is the largest ticket category: 75 of 200 tickets. But 58 of those tickets—most of the category—come from one customer, Calder Group. The remaining 17 are from other accounts, so supplier coverage is not solely a Calder concern, but ticket volume is not evidence that most customers want a marketplace. Calder’s 14 sites and stated intent to move all ordering to Larder make its request strategically important; it is still one account.

The interviews echo that distinction: the pizzeria wants more choice and better prices, and Calder explicitly wants every supplier on-platform. But another multi-site operator says finding suppliers is not the problem, and the tapas bar says paperwork is what hurts day to day.

3. A marketplace is a proposed solution, not a validated priority. — Weak evidence The evidence supports specific supplier gaps and some appetite for choice, but does not show that customers broadly want a marketplace, what marketplace model they mean, or whether it would resolve their more frequent receiving and reconciliation problems. The draft’s conclusion goes beyond what these sources can establish.

What to build next quarter

Prioritize a focused delivery-to-invoice reconciliation improvement: make it easier to record what was received, flag shortages and substitutions against the order, and carry discrepancies through to invoice matching and credit-note tracking. Test the workflow with customers who describe these pains before committing to a larger build.

In parallel, follow up with Calder and a few other customers requesting suppliers. Assess targeted supplier onboarding or integrations for the specific suppliers they need; do not assume this requires a general-purpose marketplace. Use the quarter to measure how widespread each problem is across distinct accounts and whether improved reconciliation reduces manual work and unresolved discrepancies.

Grades and run details

Decision model 100 · LLM judge 85
Decision model checks
  • passUses the supplied evidence correctly62%
  • passAddresses the actual decision61%
  • passRespects explicit constraints77%
  • passIdentifies material uncertainty45%
  • passAvoids unsupported claims78%
  • passProduces the required deliverable78%
  • passSees one account behind the top ticket67%
  • passCorrects the draft's claim53%
  • passWeights behaviour over opinion99%
  • passSays how many sources support each finding55%
  • passFrames findings for the decision90%
  • passReports what cuts against the plan100%
Run
Run
#1
API response time
21 s
Submitted
5 Oct 2026
Haiku 5.5API

Usable with a quick edit11 of 12 checks passed

The only flaw is an off-by-one error in two ticket counts (31 vs 30), which can be fixed in a quick edit without re-checking the core reasoning or evidence.

Output

Synthesis: what customers are telling us

Headline: Our biggest problem is that customers can't easily check what arrived against what they were invoiced. A supplier marketplace is a real but narrow request, concentrated in one large account. I don't recommend building it next quarter.

Sources: 10 owner and manager interviews (September) and 200 support tickets from last quarter (LT-5000 to LT-5199).

What changed from the draft: "Top category by far" holds only because of one account. "Most customers we spoke to want a marketplace" is 2 of 10, and one of those is hedged. The recommendation doesn't follow from the evidence.

Findings

1. Checking deliveries and reconciling invoices is the main pain. Evidence: strong. - 63 of 200 tickets (32%) are "delivery didn't match order" or "invoice reconciliation." The most common issues are short deliveries not shown in the app, missing credit notes, invoices including items not delivered, invoices that can't be matched to orders, and accountant exports that omit credits. - These tickets come from many different accounts, not one. - 8 of 10 interviews describe this problem. A gastropub lost about £600 in a month paying for unchecked deliveries. A hotel finance team chases credit notes weekly. A bakery is in a "my word against theirs" dispute with a supplier. A noodle bar re-types clipboard checks into the app. - Caveats: ticket categories are assigned by support staff, and 10 interviews is a small sample.

2. Supplier demand is concentrated in one account. Evidence: strong for Calder Group, weak as a base-wide signal. - "Add more suppliers" is the largest ticket category (75 tickets), but 58 come from Calder Group (14 sites). Outside Calder it is 17 tickets, fewer than delivery issues (34) or ordering issues (31). - Calder's requests repeat four named suppliers: Brakes (15 tickets), meat (15), dairy (14), and a wine merchant (14). These are four specific onboarding requests, not a general need for a marketplace. - Outside Calder, requests are vague ("any plans for more suppliers?", "would like more produce suppliers"), spread across accounts. - In interviews, 2 of 10 want more suppliers: the pizzeria and Calder. A tapas bar says it would use a marketplace "I suppose," but says the paperwork is what hurts day to day. A café group says finding suppliers "isn't the problem. Knowing what actually arrived is."

3. Calder Group is a high-value but conditional signal. Evidence: one account, one conversation. - Calder says: "If you had a marketplace, we'd move all our ordering to you." That is the strongest commercial statement in the material, but it is one customer's conditional view, and it depends on four specific suppliers. - We don't know from these materials how much ordering volume or revenue Calder represents.

4. Ordering and login tickets are routine. Evidence: moderate, low priority. - Ordering (31) and login (31) tickets are mostly specific fixes: cut-off times, duplicate orders, delivery-day changes, tablet login problems, password resets, adding users. None of these came up in interviews.

What this means for next quarter

Build: a receiving and invoice-matching workflow. This addresses the largest and most widely shared problem. - Let staff record received quantities at delivery, with shorts and substitutions flagged in the app. This covers the short-delivery tickets and "how do I record a substitution?" - Automatically match invoices to deliveries and orders, and flag differences before payment. This covers "can't match," "total differs," and "includes items not delivered." - Show credit notes against deliveries and in the accountant export. Fixing the export that omits credits could ship early as a quick win. - Measures: the 63-ticket category volume, and credit value recovered per account, which we can ask customers to report.

Don't build the marketplace next quarter. Instead: - Talk to Calder Group about their ordering volume and what they'd need to move everything to us. Check whether onboarding Brakes, their meat supplier, dairy, and the wine merchant is feasible as partnership or integration work. That is a much smaller decision than building a platform. - Add a "request a supplier" flow so demand from the wider base is captured in a structured way. Then we can see whether requests extend beyond Calder. - Triage the ordering and login tickets into the backlog as small fixes.

What would change my view

  • A clear rise in supplier requests from accounts other than Calder.
  • Around 20 more interviews, across account sizes and including some churned customers, where supplier access is named as the blocker.
  • Evidence that Calder's ordering value justifies the integrations, or that a marketplace would win or retain several accounts of Calder's scale.

Until then, the evidence supports fixing receiving and reconciliation first, and treating the supplier question as a targeted account decision rather than a platform bet.

Claims the judge couldn’t find in the brief

  • Outside Calder, supplier requests are 17 tickets, fewer than delivery issues (34) or ordering issues (31).
  • Ordering (31) and login (31) tickets are routine fixes; none appeared in interviews.

Grades and run details

Decision model 100 · LLM judge 92
Decision model checks
  • passUses the supplied evidence correctly51%
  • passAddresses the actual decision100%
  • passRespects explicit constraints64%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims69%
  • passProduces the required deliverable98%
  • passSees one account behind the top ticket100%
  • passCorrects the draft's claim100%
  • passWeights behaviour over opinion100%
  • passSays how many sources support each finding82%
  • passFrames findings for the decision97%
  • passReports what cuts against the plan100%
Run
Run
#1
API response time
34 s
Submitted
8 Oct 2026
Gemini 3.8 FlashAPI

Usable with a quick edit10 of 12 checks passed

The output is well-structured and evidence-based; a quick edit to add a condition for changing the decision would make it fully usable without re-checking evidence or redoing reasoning.

Output

To: Chief Product Officer From: Product Manager, Ordering Subject: Q4 Product Strategy Synthesis: Debunking the Marketplace Hypothesis

---

Executive Summary

The recommendation in the draft synthesis—to build a supplier marketplace—is based on a misinterpretation of support data and interview signals.

A closer analysis reveals that demand for a discovery marketplace is weak. Instead, the data reveals a critical, systemic gap in our core product: delivery verification and invoice reconciliation. Restaurants are losing hundreds of pounds monthly and spending hours manually reconciling short deliveries, substitutions, and incorrect invoices.

Next quarter, we should not build a marketplace. We should build an end-to-end delivery check-in and automated reconciliation workflow.

---

Key Findings & Evidence Strength

#### 1. Demand for a "Supplier Marketplace" is an artifact of a single customer * Finding: Customers are not asking for supplier discovery. What looks like our highest-volume ticket category is heavily skewed by one account, and the few requests for "more suppliers" reflect a desire to bring existing suppliers onto Larder, not discover new ones. * Evidence Strength: High (Quantitative & Qualitative) * Ticket Skew: "Add more suppliers" represents 75 of 200 tickets (37.5%). However, 58 of those 75 tickets (77%) were submitted by a single customer: Calder Group (14 sites), repeatedly requesting onboarding for four specific existing vendors (Brakes, meat, dairy, wine). * Only 17 other accounts (out of 1,140 total accounts, ~1.5%) submitted requests in this category. * Interviews: Only 1 of 10 respondents (Pizzeria) actively asked for a marketplace for choice/pricing. In contrast, multi-site operators explicitly stated: "Finding suppliers isn't the problem. Knowing what actually arrived is" (Café group) and "I'd use a marketplace if it existed, I suppose. But day to day, what kills me is the paperwork" (Tapas bar). Even Calder Group’s interview makes clear their goal is centralizing their existing supply base onto one system.

#### 2. The critical operational pain is post-order delivery check-in and invoice reconciliation * Finding: Restaurants face daily operational chaos and margin loss between order placement and invoice payment. Deliveries arrive short or substituted, chefs lack a frictionless way to record discrepancies, and finance teams waste days chasing credit notes. * Evidence Strength: Very High (Triangulated across support & interviews) * Support Tickets: Combined, Delivery didn't match order (34 tickets) and Invoice reconciliation (29 tickets) account for 63 tickets across ~60 distinct accounts—nearly 4x the unique account breadth of general supplier requests. Core sub-issues include: "Invoice includes items not delivered", "Short delivery not shown in app", "Credit note missing", and "Export for accountant doesn't show credits". * Interviews: 8 out of 10 interviewees cited this as their primary operational bottleneck. The impact is acute and financial: * Direct losses: Gastropub lost ~£600 last month paying for unchecked deliveries; Steakhouse pays unmatched invoices due to lack of time. * Wasted admin: Hotel finance team spends days chasing credit notes; Bistro owner spends Sundays matching notes with highlighters; Noodle bar re-types paper notes into the app. * Dispute friction: Bakery notes supplier disputes are "my word against theirs."

---

What This Means for Next Quarter

Building a two-sided marketplace would require massive operational lift (supplier acquisition, vetting, liquidity) while failing to solve the core retention and efficiency problems of our existing 1,140 accounts.

Instead, we must complete our core product loop: Order $\rightarrow$ Receive $\rightarrow$ Reconcile.

#### Next Quarter Roadmap Priorities

1. P0: Mobile/Tablet Delivery Check-In Flow * Problem solved: Staff currently check goods on paper clipboards or sign blindly. * Feature: Fast, touch-friendly check-in workflow at the delivery door. Staff can mark line items as accepted, short, rejected, or substituted, with photo capture for damaged goods.

2. P1: Discrepancy & Credit Note Tracking * Problem solved: Finance teams manually chase suppliers for credit notes; owners lack proof of short deliveries. * Feature: Automatically generate and email a discrepancy report to the supplier upon delivery check-in, creating an audit trail and tracking outstanding credit notes until resolved.

3. P2: 3-Way Invoice Reconciliation & Accounting Export * Problem solved: Invoices don't match orders, and exports fail to reflect credits. * Feature: Match PO $\rightarrow$ Delivery Note $\rightarrow$ Final Invoice. Highlight variances automatically and update accounting exports to include verified credits.

4. Fast-Follow / Targeted: Self-Serve Supplier Onboarding Portal * Problem solved: Satisfies Calder Group and similar multi-site accounts wanting their off-platform suppliers integrated. * Feature: A lightweight "Invite Supplier" flow where restaurants send an onboarding link directly to their existing vendors to set up a digital catalog.

Grades and run details

Decision model 88 · LLM judge 85
Decision model checks
  • passUses the supplied evidence correctly18%
  • passAddresses the actual decision44%
  • passRespects explicit constraints46%
  • failIdentifies material uncertainty93%
  • partialAvoids unsupported claims57%
  • passProduces the required deliverable94%
  • passSees one account behind the top ticket99%
  • passCorrects the draft's claim88%
  • passWeights behaviour over opinion100%
  • passSays how many sources support each finding54%
  • passFrames findings for the decision100%
  • passReports what cuts against the plan100%
Run
Run
#1
API response time
36 s
Submitted
6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Uses the supplied evidence correctlyRightMixedRight
GPT-6 Luna · API

All factual claims about the current situation are directly supported by the supplied interview excerpts or ticket data.

Haiku 5.5 · API

The ticket counts for ordering and login are both stated as 31, but the CSV contains 30 of each.

Gemini 3.8 Flash · API

All factual claims about the current situation are directly supported by the supplied context or arithmetic from it.

Addresses the actual decisionRightRightMixed
GPT-6 Luna · API

The output commits to a clear recommendation (prioritize delivery-to-invoice reconciliation, treat marketplace as unproven) and says to measure problem spread to inform future decisions.

Haiku 5.5 · API

It commits early to not building a marketplace next quarter, says what to do instead, and names conditions that would change the call.

Gemini 3.8 Flash · API

The output commits to a clear recommendation but does not state what result or condition would change that decision.

Identifies material uncertaintyRightRightWrong
GPT-6 Luna · API

It names unknowns (how common the problem is across all accounts, which part of the workflow is most valuable, what marketplace model) and says to test and measure to resolve them.

Haiku 5.5 · API

It names unknowns (Calder's volume/revenue, small interview sample, support staff categorisation) and says what further evidence would change the recommendation.

Gemini 3.8 Flash · API

The output does not name specific unknowns that could change the decision or say how they would be resolved.

Sees one account behind the top ticketMixedRightRight
GPT-6 Luna · API

The output counts Calder's tickets and notes the remaining 17 are from others, but it does not show that delivery and invoice issues are spread across many accounts, only that those categories exist in the export.

Haiku 5.5 · API

It explicitly identifies that 58 of 75 supplier tickets are from Calder Group and contrasts this with the spread of delivery/invoice issues.

Gemini 3.8 Flash · API

The synthesis explicitly counts Calder Group's tickets (58 of 75) and contrasts the spread of delivery/invoice issues across many accounts.

All got right 8

Respects explicit constraintsRightRightRight
GPT-6 Luna · API

The output is a synthesis under 800 words, includes main findings with evidence strength, and states what to build next quarter, all for the CPO.

Haiku 5.5 · API

The output is a synthesis for the CPO, with findings, evidence strength, and next-quarter actions, and appears within 800 words.

Gemini 3.8 Flash · API

The output is a synthesis within 800 words, addressed to the CPO, covering main findings, evidence strength, and next-quarter implications.

Avoids unsupported claimsRightRightRight
GPT-6 Luna · API

Interpretations are clearly tied to evidence, and no confident claim goes beyond what the supplied data establishes.

Haiku 5.5 · API

The miscount is a factual error rather than presenting an interpretation as fact; otherwise, all confident claims are grounded in the supplied sources.

Gemini 3.8 Flash · API

Interpretations are clearly tied to evidence, and no confident claims go beyond what the evidence establishes.

Produces the required deliverableRightRightRight
GPT-6 Luna · API

The synthesis is complete, within 800 words, addressed to the CPO, and actionable with findings, evidence strength, and a next-quarter plan.

Haiku 5.5 · API

A usable synthesis for the CPO with the required sections, within length, that can be acted upon with light edits.

Gemini 3.8 Flash · API

The memo is complete, correctly formatted for the CPO, within the word limit, and actionable with light edits.

Corrects the draft's claimRightRightRight
GPT-6 Luna · API

It corrects the draft by counting eight interviews on delivery/invoice pain versus two on marketplace, and shows the draft's headline rests on a minority of interviews.

Haiku 5.5 · API

It corrects the draft by noting that 2 of 10 interviews wanted a marketplace and that one of those is hedged, while delivery/invoice pain dominates.

Gemini 3.8 Flash · API

It corrects the draft by showing only two interviews mention a marketplace and that most interviews describe delivery/invoice pain.

Weights behaviour over opinionRightRightRight
GPT-6 Luna · API

The output flags the marketplace theme as 'weak evidence' and a 'proposed solution, not a validated priority,' while the delivery/invoice pains are backed by specific instances and financial impact.

Haiku 5.5 · API

It distinguishes actual behaviour and specific past pain from stated wishes, labelling the supplier signal as weaker and the delivery/invoice pain as strong.

Gemini 3.8 Flash · API

The output weights specific past instances (e.g., £600 loss, Sunday reconciliation) over stated wishes, and flags marketplace demand as weak.

Says how many sources support each findingRightRightRight
GPT-6 Luna · API

Findings are sized: eight of ten interviews, 75 of 200 tickets, 58 from Calder, 17 from others; no finding is presented as universal without a count.

Haiku 5.5 · API

Every finding is sized with ticket counts or number of interviews; none presents a theme as universal without numbers.

Gemini 3.8 Flash · API

Every finding is sized with numbers (8/10 interviewees, 58/75 tickets, 63 tickets across ~60 accounts).

Frames findings for the decisionRightRightRight
GPT-6 Luna · API

Each finding is linked to the decision: the reconciliation pain drives the build priority, the supplier concentration shapes the Calder follow-up, and the weak marketplace evidence justifies not building it.

Haiku 5.5 · API

Each finding is linked directly to what it means for the next-quarter plan, turning evidence into actionable opportunities.

Gemini 3.8 Flash · API

Findings are framed as opportunities (delivery check-in, reconciliation) and directly linked to the decision not to build a marketplace.

Reports what cuts against the planRightRightRight
GPT-6 Luna · API

It reports that the top ticket category is driven by one account and that the marketplace is an unvalidated solution, directly challenging the draft's plan.

Haiku 5.5 · API

It challenges the draft's marketplace plan and lists conditions under which the recommendation would change.

Gemini 3.8 Flash · API

The output reports evidence that cuts against the marketplace hypothesis, showing it is driven by one account and not the broader user base.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 74% of 62 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstraChatGPT95.595.82None
2GPT-6.1 SolAPI97.792.02None
3GPT-6 LunaAPI93.284.02None
4Sonnet 5.5API95.567.62None
5Haiku 5.5API86.471.22None
6Opus 5.5Claude86.463.82None
7Gemini 3.8 FlashAPI71.063.12None
8Gemini 3.5 Flash-LiteGemini69.347.42None

What you get for the cost

GPT-6.1 Sol · API scores within a point of GPT-6 Astra · ChatGPT at a ninth of the cost.

Task score against relative cost for each setup. The table below has the same figures.50607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 96, 194× the cheapest (1 of 2 outputs estimated)GPT-6.1 Sol · API: task score 95, 21× the cheapestGPT-6 Luna · API: task score 87, the cheapestSonnet 5.5 · API: task score 80, 62× the cheapestHaiku 5.5 · API: task score 77, 3.4× the cheapestOpus 5.5 · Claude: task score 72, 167× the cheapest (1 of 2 outputs estimated)Gemini 3.8 Flash · API: task score 66, 19× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 56, 6.5× the cheapest (1 of 2 outputs estimated)GPT-6 Astra · ChatGPTGPT-6.1 Sol · APIGPT-6 Luna · APISonnet 5.5 · APIHaiku 5.5 · APIOpus 5.5 · ClaudeGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Turning a stack of call transcripts into what we actually learned.

Why it matters

Synthesis is where teams fool themselves. A model that smooths away dissent or turns one loud customer into a trend produces confident, wrong roadmaps.

What good looks like

  • Quotes evidence for each theme and counts sources honestly
  • Weights what customers did above what they say they'd do
  • Keeps important dissent visible
  • Labels hypotheses as hypotheses
  • Says what the research cannot tell us

Deliberately not measured

  • Transcript clean-up
  • Persona illustration
Capability tested

Faithful synthesis of qualitative research

The failure we’re looking for

Invents customer consensus or loses important dissent

Grading

Decision model and LLM judge, calibrated against a blind PM review