Humanity’s Last PRD
Latest drop: Gemini 4 Argon · 13 Oct
Tasks / Product Craft

Extract discovery insights

Can the model separate evidence, themes and hypotheses without inventing consensus?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 50% were usable with at most a quick edit.

Reliably right

  1. Frames findings for the decision100% pass
    Each main finding is framed as an opportunity (delivery-to-invoice pain, concentrated supplier need) and linked directly to what to build next quarter.
    GPT-6 Astra · ChatGPT · The loudest ticket
  2. Corrects the draft's claim100% pass
    It corrects the draft's headline by counting interviews (8 describe delivery/invoice pain, 2 mention supplier expansion) and showing the marketplace claim is unsupported.
    GPT-6 Astra · ChatGPT · The loudest ticket
  3. Keeps dissent visible100% pass

Where it slips

  1. Says how many sources support each finding46% pass
    The finding on smaller issues sizes Ordering as 31 tickets, but the actual count is 30, so the finding is not accurately sized.
    Sonnet 5.5 · API · The loudest ticket
  2. Uses the supplied evidence correctly56% pass
    The output states Ordering has 31 tickets, but the CSV contains 30 Ordering tickets; this factual claim is not supported by the supplied data.
    GPT-6.1 Sol · API · The loudest ticket
  3. Addresses the actual decision57% pass
    The output commits to a clear recommendation but does not state what result or condition would change that decision.
    Gemini 3.8 Flash · API · The loudest ticket

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Weights behaviour over opinion

    Does the synthesis distinguish evidence of what customers actually did (specific past instances, observed behaviour, workarounds) from stated preferences, predictions and feature requests, and weight them accordingly?

    Passes when Themes are labelled or weighted by evidence type, and at least one theme resting mainly on stated intent or requests is flagged as weaker.

    Bob MoestaLenny’s Podcast

  2. Frames findings for the decision

    Does the synthesis turn its findings into opportunities tied to the decision or outcome the team faces, saying what each one means for that decision?

    Passes when Each main finding is framed as an opportunity (a need or pain to address) and linked to the decision at hand, with what it implies.

    Teresa TorresLenny’s Podcast

  3. Reports what cuts against the plan

    Does the synthesis report the evidence that cuts against the team's current belief or plan, rather than only what supports it?

    Passes when Names at least one finding that challenges the team's hypothesis or plan, and says what it means for it.

    Judd AntinLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Says how many sources support each finding · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for Larder's ordering product. A colleague has drafted the synthesis below for our CPO, recommending a supplier marketplace. Before it goes, rewrite the synthesis from the evidence, in no more than 800 words: the main findings, how strong the evidence for each is, and what they mean for what we build next quarter. The interviews and the ticket export are attached.

What the model was given4 items: About Larder, draft_synthesis.md, interview_excerpts.md, support_tickets_last_quarter.csv
About LarderOrdering software for restaurants: order from your suppliers in one place, track deliveries and invoices. 1,140 accounts, most of them independent restaurants.
draft_synthesis.md6 lines · Download
# Draft synthesis (PM, for the CPO)

Headline: customers want a supplier marketplace.
- "Add more suppliers" is our top support category by far.
- Most customers we spoke to want a marketplace (see Calder Group and the pizzeria).
- Recommendation: build a supplier marketplace next quarter.
interview_excerpts.md12 lines · Download
# Interview excerpts (restaurant owners and managers, September)

1. Bistro, 2 sites: "Last Tuesday the fish order came short again. I only found out when the invoice came, a week later. I spent Sunday night matching delivery notes to invoices with a highlighter."
2. Café group, 5 sites: "We switched produce supplier twice this year, but finding suppliers isn't the problem. Knowing what actually arrived is."
3. Pizzeria, 1 site: "Honestly I'd love more suppliers in the app. More choice, better prices."
4. Gastropub: "My head chef signs the delivery note without checking. Then I pay for things we never got. Last month that was about £600."
5. Hotel restaurant: "Our finance team chases credit notes for short deliveries every week. It takes days."
…
support_tickets_last_quarter.csv200 rows · Download
ticket,account,account_sites,category,summary
LT-5000,Calder Group,14,Add more suppliers,Can you onboard our meat supplier
LT-5001,Account 1886,3,Ordering,Change delivery day
LT-5002,Account 2004,1,Delivery didn't match order,Short delivery not shown in app
LT-5003,Calder Group,14,Add more suppliers,Add dairy supplier please
LT-5004,Calder Group,14,Add more suppliers,Add dairy supplier please
LT-5005,Account 1858,2,Add more suppliers,Any plans for more suppliers?
…
What a strong answer doesThe answer key the graders mark against

Shows that the marketplace headline doesn't hold. 'Add more suppliers' is the biggest ticket category (75 of 200), but 58 of those come from one account, Calder Group; across other accounts, delivery and invoice problems ('delivery didn't match order' and 'invoice reconciliation', 63 tickets) are far more common than supplier requests (17). In the interviews, 7 of 10 describe a specific recent instance of the same pain (short deliveries, invoices that don't match, credit notes, re-typing paper checks), while the marketplace mentions are wishes (the pizzeria 'would love' it, the tapas bar 'I suppose') or come from Calder. Weights what people did over what they say they'd want, counts how many sources support each finding, and recommends focusing next quarter on checking deliveries and reconciling invoices, with the marketplace treated as one large customer's request to handle separately.

Critical failures (cap the score at 40)
  • Recommends building the supplier marketplace next quarter as the main finding
Task

Staff-level · v1.1

The outputs

9 setups answered this brief.

GPT-6 AstraChatGPT100Usable with a quick editGPT-6.1 SolAPI96Usable with a quick editHaiku 5.5API96Usable with a quick editGPT-6 LunaAPI88Needs more workSonnet 5.5API88Usable with a quick editOpus 5.5Claude81Needs more workGemini 3.8 FlashAPI79Usable with a quick editGemini 4 ArgonAPI71Needs more workGemini 3.5 Flash-LiteGemini60Needs more work
Gemini 4 ArgonAPI

Needs more than a quick edit9 of 12 checks passed

The output contains several arithmetic errors in ticket counts (e.g., 72 instead of 75, 36 instead of 33) that would force a PM to re-verify the data before using it.

Output

Product Discovery Synthesis: Q3 Interviews & Support Tickets

To: Chief Product Officer From: PM, Ordering Subject: Correcting the Q4 roadmap recommendation: Delivery & Invoice Reconciliation vs. Supplier Marketplace

Executive Summary

An initial look at last quarter’s data suggested we should build a supplier marketplace next quarter. However, a deeper analysis of our 200 support tickets and 10 customer interviews shows the marketplace signal is a mirage caused by a single vocal enterprise account.

For our core customer base of independent restaurants, the most urgent, expensive, and widespread problem is delivery tracking and invoice reconciliation—specifically handling short deliveries, substitutions, and credit notes. We should focus Q4 on fixing the post-order workflow, not building a marketplace.

---

Main Findings & Strength of Evidence

#### 1. The "Supplier Marketplace" demand is heavily skewed and misunderstood * Strength of Evidence: Weak * What the data actually shows: On the surface, "Add more suppliers" is our #1 ticket category, making up 72 of 200 tickets (36%). However, 56 of those 72 tickets (78%) came from a single 14-site customer: Calder Group. The remaining 16 tickets came from just 16 unique accounts (out of 1,140 total). * The Nuance: Even Calder Group’s tickets are not requests to discover new suppliers in a marketplace. They are repeatedly asking us to onboard their existing suppliers (their meat supplier, dairy supplier, wine merchant, and Brakes) so they can centralize ordering. * Interview Context: In interviews, only 1 of 10 customers (the single-site Pizzeria) genuinely wanted a marketplace for supplier discovery and price comparison. Calder Group used the word "marketplace" to mean having all their current vendors on the platform. Two others actively dismissed the idea: the Tapas Bar expressed lukewarm interest before pivoting to paperwork pain, and the 5-site Café Group explicitly stated: "Finding suppliers isn't the problem. Knowing what actually arrived is."

#### 2. Delivery mismatches and invoice reconciliation are our largest widespread pain points * Strength of Evidence: Strong * What the data actually shows: In our interviews, 8 out of 10 customers identified post-order tracking (short deliveries, substitutions, invoice matching, and chasing credit notes) as their primary operational headache. This directly causes financial loss (e.g., the Gastropub losing ~£600/month paying for undelivered goods) and severe time waste (owners spending Sunday nights matching paper notes to invoices, or re-typing clipboard notes into Larder). * Support Ticket Alignment: In the ticket data, "Delivery didn't match order" (36 tickets across 36 unique accounts) and "Invoice reconciliation" (31 tickets across 30 unique accounts) are two halves of the exact same problem. Combined, they account for 67 tickets across 64 unique accounts—four times the unique account volume of the supplier requests (excluding Calder Group). * Specific Product Gaps: Customers are explicitly telling us where the product breaks down: * At the back door: Short deliveries aren't showing in the app (9 tickets), users don't know how to log substitutions (8 tickets), and chefs check deliveries on paper clipboards—likely exacerbated by our tablet login bug (10 tickets). * In the back office: Invoices include undelivered items (9 tickets), invoice totals don't match orders (9 tickets), users can't match invoices to orders (10 tickets), credit notes are missing (8 tickets), and accountant exports fail to show credits (10 tickets).

#### 3. Core ordering and access friction creates unnecessary noise * Strength of Evidence: Moderate (Ticket data only) * What the data actually shows: We saw 31 tickets for basic "Ordering" issues (10 duplicate orders, 11 delivery day changes, 10 cut-off time questions) and 30 for "Login and access" (10 password resets, 10 adding users, 10 tablet login failures). While unmentioned in interviews, these represent 30% of support volume and hurt day-to-day usability.

---

What This Means for Q4: Recommendations

Building a two-sided supplier marketplace next quarter would be a costly misstep. It solves a problem (supplier discovery) that almost none of our customers have, based on un-deduplicated ticket data from one large account.

Instead, we should align our Q4 roadmap with our core value proposition—"track deliveries and invoices"—where our independent restaurants are bleeding time and money:

1. Build a "Goods-In" & Discrepancy Workflow (Primary Q4 Bet): * Allow kitchen staff to easily check off deliveries on mobile/tablet at the back door, logging short weights, missing items, and substitutions in real time. * Automatically flag discrepancies against the original order, adjust expected invoice totals, and track pending credit notes through to the accountant export. 2. Fix the Tablet Login Bug & Ordering UX (Quick Wins): * Fixing the tablet login issue (10 tickets) is a prerequisite for kitchen staff checking deliveries digitally instead of on clipboards. * Address duplicate order prevention and surface supplier cut-off times clearly in the UI to cut ordering support tickets. 3. Handle Calder Group Operationally, Not via a Marketplace: * Have Customer Success and Partnerships work directly with Calder Group to onboard their four requested suppliers (Brakes, meat, dairy, wine). For the broader base, improve our "request to onboard my existing supplier" intake flow so sites don't spam support.

Claims the judge couldn’t find in the brief

  • "Add more suppliers" is the #1 ticket category, making up 72 of 200 tickets (36%).
  • 56 of those 72 tickets (78%) came from Calder Group.
  • The remaining 16 tickets came from 16 unique accounts.
  • "Delivery didn't match order" has 36 tickets.
  • "Invoice reconciliation" has 31 tickets.

Grades and run details

Decision model 71 · LLM judge 77
Decision model checks
  • failUses the supplied evidence correctly37%
  • failAddresses the actual decision31%
  • passRespects explicit constraints67%
  • failIdentifies material uncertainty77%
  • partialAvoids unsupported claims51%
  • passProduces the required deliverable95%
  • passSees one account behind the top ticket97%
  • passCorrects the draft's claim94%
  • passWeights behaviour over opinion100%
  • passSays how many sources support each finding43%
  • passFrames findings for the decision97%
  • passReports what cuts against the plan100%
Run
Run
#1
API response time
15 s
Submitted
13 Oct 2026
GPT-6 AstraChatGPT

Usable with a quick edit12 of 12 checks passed

The synthesis is clear, evidence-based, and directly usable by a product manager with at most light editing for emphasis or wording.

Output

Next-quarter synthesis: prioritise delivery-to-invoice accuracy, not a marketplace

Recommendation: Build and validate a focused workflow for recording what arrived, flagging discrepancies and resolving invoice/credit issues. Continue targeted supplier onboarding, but do not commit next quarter to a broad marketplace. The evidence supports operational accuracy more broadly than supplier discovery.

Main findings and evidence strength

1. Supplier requests lead ticket volume—but are heavily concentrated in one account

The export contains 200 tickets:

CategoryTicketsDistinct accounts
------:---:
Add more suppliers7518
Delivery didn’t match order3434
Invoice reconciliation2928
Ordering3129
Login and access3131

Account counts are deduplicated within categories; they should not be summed across categories.

Calder Group generated 58 of the 75 supplier tickets (77%), repeatedly requesting onboarding for four named suppliers or supplier types. The other 17 supplier tickets came from 17 accounts.

Evidence strength: Strong evidence of a concentrated supplier-coverage need; weak evidence that a marketplace is the right solution. Ticket volume measures support demand, not independent customer votes. Repeated requests may indicate an important unresolved problem, but not broad market demand.

Calder’s 14 sites and stated willingness to move all ordering to Larder warrant commercial follow-up. However, this is a conditional intention—not a commitment—and we have no revenue or order-volume evidence to size the opportunity. Requests to add existing suppliers must also be distinguished from discovering new ones.

2. Delivery and invoice problems affect a broader set of accounts

Together, delivery discrepancies and invoice reconciliation account for 63 tickets from 62 distinct accounts, compared with supplier requests from 18 accounts.

The interviews reinforce this pattern. Eight of ten excerpts describe delivery, reconciliation or paperwork pain, although the tapas bar’s comment is less specific. Examples include:

  • Discovering a short delivery only when the invoice arrives.
  • Paying for undelivered goods; one gastropub reported approximately £600 in one month.
  • Chasing credit notes weekly.
  • Having no shared evidence when suppliers dispute shortages.
  • Recording deliveries on paper, then re-entering them in Larder.

The café group explicitly says finding suppliers is not the problem; knowing what arrived is.

Evidence strength: Strong convergence across these two sources that this is a meaningful problem spanning multiple restaurant formats. Evidence for its prevalence across all 1,140 accounts, typical financial impact or the best solution remains limited. The £600 figure is one customer’s report, not an estimate we can extrapolate.

3. Marketplace interest exists, but “most customers want one” is unsupported

The pizzeria strongly wants more choice and better prices. Calder explicitly mentions a marketplace, but frames its need as getting every supplier onto the platform. The tapas bar expresses tentative interest, then prioritises paperwork.

Evidence strength: Directional evidence for supplier expansion, but weak evidence for broad marketplace demand. Only Calder explicitly advocates a marketplace; the pizzeria’s desired outcome could potentially be served in other ways. These excerpts do not establish expected adoption, switching behaviour or willingness to pay.

What we should build next quarter

Prioritise a narrow delivery-to-invoice pilot, rather than attempting full reconciliation automation immediately:

  1. Capture actual receipt against an order: quantities received, shortages and substitutions, without duplicate entry.
  2. Carry discrepancies forward: make them visible when checking invoices rather than leaving users to reconstruct events.
  3. Track resolution: preserve a discrepancy record and show outstanding supplier responses or credits.

Start with observing receiving and invoice-checking in a varied set of independent restaurants. In particular, test whether busy staff will actually record discrepancies: the unchecked delivery-note example suggests software alone may not solve the problem.

Measure receiving-workflow adoption, duplicate entry, reconciliation time, discrepancies identified before payment and time to resolve credits. Establish baselines before setting numerical success targets; use pilot results to decide how far to expand.

Keep supplier coverage as a separate, bounded workstream. Clarify Calder’s required suppliers, onboarding feasibility and commercial commitment. Test targeted onboarding before assuming we need marketplace discovery, comparison and transaction capabilities.

Confidence and limits: This is a prioritisation recommendation, not a proven business case. Ten interview excerpts and self-selected support tickets cannot establish population prevalence. Some accounts also have inconsistent site counts in the export, so site-weighted conclusions would be unreliable. Validate the proposed workflow and its value before a broad rollout.

Grades and run details

Decision model 100 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly23%
  • passAddresses the actual decision40%
  • passRespects explicit constraints62%
  • passIdentifies material uncertainty86%
  • passAvoids unsupported claims70%
  • passProduces the required deliverable88%
  • passSees one account behind the top ticket100%
  • passCorrects the draft's claim73%
  • passWeights behaviour over opinion100%
  • passSays how many sources support each finding58%
  • passFrames findings for the decision97%
  • passReports what cuts against the plan100%
Run
Run
#1
Time to output
48 s
Submitted
5 Oct 2026
Sonnet 5.5API

Usable with a quick edit9 of 12 checks passed

The output is clear, well-structured, and only needs a minor correction to the Ordering ticket count (30, not 31) to be fully accurate and usable.

Output

Synthesis for the CPO: what to build next quarter

Headline: The evidence doesn't support a marketplace. It points to the gap between what we order, what arrives and what we're invoiced. I recommend building delivery checking and reconciliation next quarter.

Corrections to the earlier draft

  • "Add more suppliers is our top category by far" is an artefact of one account. Of 200 tickets, 75 are in that category, and 58 of those come from Calder Group, which repeats four requests: Brakes, a meat supplier, a dairy supplier and a wine merchant. The other 17 come from 17 different accounts. Without Calder, it is our smallest category (17 of 142 tickets, 12%).
  • "Most customers want a marketplace" is wrong. Only 2 of 10 interviewees asked for more suppliers (Calder and the pizzeria). The tapas bar was lukewarm ("I suppose") and went straight to paperwork. The café group said finding suppliers "isn't the problem."

Findings and strength of evidence

1. Customers can't easily verify what was delivered against what they ordered and were invoiced for. Strength: strong. - Interviews: 7 of 10 describe this unprompted, and an eighth (tapas bar) says "paperwork" without detail. Examples are short fish orders found a week later, a chef signing notes unchecked (~£600 in a month), finance chasing credit notes, "my word against theirs," clipboard re-typing, and invoices that never match. - Tickets: "Delivery didn't match order" (34) and "Invoice reconciliation" (29) total 63 tickets, 44% of non-Calder volume, from about 62 distinct accounts. The ticket summaries include "short delivery not shown in app," "invoice includes items not delivered," "credit note missing" and "export doesn't show credits." - Interviews and tickets are independent sources and agree. The combined grouping is my judgement, but delivery tickets alone (34) are still twice the non-Calder supplier requests.

2. Generic demand for more suppliers is small and vague. Strength: moderate. - The 17 non-Calder tickets are "any plans for more suppliers?" (8) and "more produce suppliers" (9). They show mild interest, not a pain point. One interviewee (the pizzeria) wants more choice and better prices.

3. Calder Group has a real, specific need. Strength: strong for Calder, weak as a market signal. - Calder is one account with 14 sites, which says it would move all ordering to us with a marketplace. This is a stated, conditional intent. Its tickets name four suppliers, which is an onboarding request, not evidence for a marketplace. We don't yet know Calder's revenue or how many similar multi-site groups we have.

4. Smaller issues: ordering, logins and tablets. Strength: moderate, low priority. - Ordering (31 tickets) and login/access (31 tickets) are mostly routine. "Can't log in on tablet" matters because delivery checking will likely happen on a tablet at the back door.

What this means for next quarter

Build: a receiving and reconciliation flow. - Staff record what actually arrived at the door: shorts, substitutions and weights. - The app flags mismatches against the order and the invoice. - It produces a timestamped record the owner can send to the supplier as a credit-note claim. - Credits flow into the accountant export.

This addresses the main complaints: finding out a week late, the "my word against theirs" disputes, and double entry. It also helps the people who file tickets about short deliveries and missing credit notes.

Don't build: a marketplace. It's a large investment backed by one account and a few vague requests.

Handle Calder separately. Onboard their four named suppliers through our existing process if feasible. Then ask Calder whether that satisfies their "move everything to you" condition, and get their revenue.

Fix before building: tablet login reliability, since the new flow depends on it.

Caveats

  • The sample is 10 interviews from September, and I don't know how they were recruited. Eight of ten raising the same pain is encouraging but not a measured prevalence.
  • Tickets show what people contact support about, not everything that hurts. Customers with delivery problems may just pay the invoice, as the steakhouse does.
  • Ticket data is one quarter, and the `account_sites` values are inconsistent for a few accounts (for example 1418 and 1625), so I haven't segmented by size.
  • The £600 loss is one anecdote. Before committing scope, we should size the cost of short deliveries and unclaimed credits across a sample of accounts.

What would change my view

If we find a meaningful segment of multi-site groups like Calder, with material revenue that is blocked on supplier coverage, supplier breadth becomes a segment-specific bet. It would then be worth testing through targeted onboarding before any full marketplace.

Claims the judge couldn’t find in the brief

  • Ordering (31 tickets) and login/access (31 tickets) are mostly routine.

Grades and run details

Decision model 100 · LLM judge 77
Decision model checks
  • passUses the supplied evidence correctly16%
  • passAddresses the actual decision100%
  • passRespects explicit constraints73%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims58%
  • passProduces the required deliverable95%
  • passSees one account behind the top ticket100%
  • passCorrects the draft's claim100%
  • passWeights behaviour over opinion100%
  • passSays how many sources support each finding77%
  • passFrames findings for the decision98%
  • passReports what cuts against the plan100%
Run
Run
#1
API response time
47 s
Submitted
6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 5

Uses the supplied evidence correctlyWrongRightMixed
Gemini 4 Argon · API

Several key ticket counts are wrong (72 not 75, 56 not 58, 36 not 33, 31 not 29), so the supplied evidence is not used correctly.

GPT-6 Astra · ChatGPT

All claims about the current situation are directly supported by the supplied context or derived by correct arithmetic.

Sonnet 5.5 · API

The Ordering ticket count is stated as 31, but the CSV contains 30 Ordering tickets, so a factual statement about the current situation is incorrect.

Addresses the actual decisionMixedRightRight
Gemini 4 Argon · API

It commits unambiguously to not building the marketplace in Q4, focusing instead on delivery/invoice reconciliation, and states what would change the call.

GPT-6 Astra · ChatGPT

The output commits to a clear recommendation (prioritise delivery-to-invoice accuracy, not a marketplace) and states that validation results would change the call.

Sonnet 5.5 · API

Commits early to building delivery checking and reconciliation, and says what would change the view (finding a meaningful segment of multi-site groups like Calder).

Identifies material uncertaintyWrongRightRight
Gemini 4 Argon · API

It does not name specific unknowns that could change the decision or say how they would be resolved; it states a definitive call without bounding uncertainty.

GPT-6 Astra · ChatGPT

It names specific unknowns (revenue impact, adoption, prevalence across all accounts) and says how to resolve them through pilot and measurement.

Sonnet 5.5 · API

Names unknowns (interview recruitment, ticket bias, inconsistent account_sites, anecdotal £600, need to size cost) and says how they would be resolved or what would change the call.

Avoids unsupported claimsRightRightMixed
Gemini 4 Argon · API

Interpretations like Calder asking for onboarding rather than discovery are supported by ticket summaries, and hypotheses are labelled ('likely').

GPT-6 Astra · ChatGPT

Interpretations and forecasts are clearly labelled as such, and confident claims are limited to what the evidence establishes.

Sonnet 5.5 · API

The claim 'Ordering (31 tickets)' is presented as an established fact, but the evidence shows 30 Ordering tickets, so it is an unsupported confident claim.

Says how many sources support each findingRightRightMixed
Gemini 4 Argon · API

Every main finding is sized (e.g., 8/10 interviews, ticket counts per sub‑category, one‑account dominance).

GPT-6 Astra · ChatGPT

Every finding is sized with ticket counts, account counts, or interview counts, and nothing is presented as universal without support.

Sonnet 5.5 · API

The finding on smaller issues sizes Ordering as 31 tickets, but the actual count is 30, so the finding is not accurately sized.

All got right 7

Respects explicit constraintsRightRightRight
Gemini 4 Argon · API

Output is a synthesis to the CPO, within 800 words, in the requested form.

GPT-6 Astra · ChatGPT

The output is a synthesis under 800 words, includes main findings with evidence strength, and says what to build next quarter, as requested.

Sonnet 5.5 · API

The output is a synthesis for the CPO, well under 800 words, and respects the requested form and reader.

Produces the required deliverableRightRightRight
Gemini 4 Argon · API

The synthesis is complete, correctly addressed to the CPO, under 800 words, and actionable with light edits.

GPT-6 Astra · ChatGPT

The synthesis is complete, in the right form for the CPO, within the word limit, and actionable without major gaps.

Sonnet 5.5 · API

The synthesis is complete, in the right form for the CPO, within the word limit, and actionable with light edits.

Sees one account behind the top ticketRightRightRight
Gemini 4 Argon · API

It clearly shows that Calder Group dominates the 'Add more suppliers' tickets and that delivery/invoice issues are spread across many accounts.

GPT-6 Astra · ChatGPT

It counts tickets by account, shows Calder Group is 58 of 75 supplier tickets, and contrasts with delivery/invoice issues spread across many accounts.

Sonnet 5.5 · API

Clearly shows that 58 of 75 'Add more suppliers' tickets come from Calder Group, and that delivery/invoice issues are spread across many accounts.

Corrects the draft's claimRightRightRight
Gemini 4 Argon · API

It corrects the headline by counting interviews: only 1 genuinely wanted a marketplace, and most described delivery/invoice pain.

GPT-6 Astra · ChatGPT

It corrects the draft's headline by counting interviews (8 describe delivery/invoice pain, 2 mention supplier expansion) and showing the marketplace claim is unsupported.

Sonnet 5.5 · API

Counts the interviews (2 of 10 asked for more suppliers) and corrects the headline, showing that most interviews describe delivery and invoice pain.

Weights behaviour over opinionRightRightRight
Gemini 4 Argon · API

It distinguishes specific past pain (financial losses, late‑night matching) from stated preferences and feature requests, weighting the former.

GPT-6 Astra · ChatGPT

It distinguishes specific past instances and workarounds (delivery/invoice pain) from stated preferences (marketplace wishes) and weights the former more heavily.

Sonnet 5.5 · API

Distinguishes specific past instances of delivery/invoice pain from stated preferences and feature requests, and flags Calder's need as 'stated, conditional intent'.

Frames findings for the decisionRightRightRight
Gemini 4 Argon · API

Findings are framed as opportunities (e.g., Goods‑In workflow) and linked to the Q4 roadmap decision.

GPT-6 Astra · ChatGPT

Each main finding is framed as an opportunity (delivery-to-invoice pain, concentrated supplier need) and linked directly to what to build next quarter.

Sonnet 5.5 · API

Each main finding is framed as an opportunity and linked to the decision, with clear implications for what to build next quarter.

Reports what cuts against the planRightRightRight
Gemini 4 Argon · API

It reports that the supplier marketplace signal is a mirage from one account, directly challenging the draft's recommendation.

GPT-6 Astra · ChatGPT

It reports that supplier requests are concentrated in one account and that the café group says finding suppliers isn't the problem, both cutting against the marketplace plan.

Sonnet 5.5 · API

Reports evidence that cuts against the marketplace hypothesis, including the concentration of supplier requests in one account and the prevalence of delivery/invoice pain.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 71% of 62 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstraChatGPT95.595.82None
2GPT-6.1 SolAPI95.592.02None
3GPT-6 LunaAPI88.684.02None
4Sonnet 5.5API95.567.62None
5Haiku 5.5API86.471.22None
6Opus 5.5Claude90.963.82None
7Gemini 4 ArgonAPI67.267.62None
8Gemini 3.8 FlashAPI62.763.12None
9Gemini 3.5 Flash-LiteGemini63.147.42None

What you get for the cost

GPT-6.1 Sol · API scores within 2 points of GPT-6 Astra · ChatGPT at a ninth of the cost.

Task score against relative cost for each setup. The table below has the same figures.4050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 96, 194× the cheapest (1 of 2 outputs estimated)GPT-6.1 Sol · API: task score 94, 21× the cheapestGPT-6 Luna · API: task score 85, the cheapestSonnet 5.5 · API: task score 80, 62× the cheapestHaiku 5.5 · API: task score 77, 3.4× the cheapestOpus 5.5 · Claude: task score 74, 167× the cheapest (1 of 2 outputs estimated)Gemini 4 Argon · API: task score 67, 25× the cheapestGemini 3.8 Flash · API: task score 62, 19× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 53, 6.5× the cheapest (1 of 2 outputs estimated)GPT-6 Astra · ChatGPTGPT-6.1 Sol · APIGPT-6 Luna · APISonnet 5.5 · APIHaiku 5.5 · APIOpus 5.5 · ClaudeGemini 4 Argon · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Turning a stack of call transcripts into what we actually learned.

Why it matters

Synthesis is where teams fool themselves. A model that smooths away dissent or turns one loud customer into a trend produces confident, wrong roadmaps.

What good looks like

  • Quotes evidence for each theme and counts sources honestly
  • Weights what customers did above what they say they'd do
  • Keeps important dissent visible
  • Labels hypotheses as hypotheses
  • Says what the research cannot tell us

Deliberately not measured

  • Transcript clean-up
  • Persona illustration
Capability tested

Faithful synthesis of qualitative research

The failure we’re looking for

Invents customer consensus or loses important dissent

Grading

Decision model and LLM judge, calibrated against a blind PM review