Tasks / Product Craft

Build a roadmap

Can the model sequence bets against capacity and dependencies, and explain the order?

Task type v1.0 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 69% were usable with at most a quick edit.

Reliably right

  1. Produces the required deliverable95% pass
    The roadmap is complete, in the right form for a Group PM, within the word limit, and actionable without major gaps.
    GPT-6.1 Sol · API · Fourteen requests from one customer
  2. Outcomes, with certainty that falls with distance94% pass
    Every item names its outcome (e.g., reduce paper transcription, reduce manual reconciliation), and near-term items are specific while later ones are deliberately looser (e.g., '7 weeks for rollout, workflow improvement and contingency').
    GPT-6.1 Sol · API · Fourteen requests from one customer
  3. Handles the big customer with evidence94% pass
    It uses Calder's rising usage and the second-hand 'might leave' to propose a conversation first, not committing the report builder, and offers a bounded export if needed.
    GPT-6.1 Sol · API · Fourteen requests from one customer

Where it slips

  1. Sees through the request count72% pass
    The output never counts requests by requester, does not mention that 14 requests come from Calder or that several are duplicates, and does not compare themes by customer count.
    GPT-6 Luna · API · Fourteen requests from one customer
  2. Uses the supplied evidence correctly72% pass
    States credit notes as 4 customers, but the request log contains 5 credit note tracking requests.
    Sonnet 5.5 · API · Fourteen requests from one customer
  3. Fits the stated capacity73% pass
    Q3 Expenses row lacks squad-week allocation, so the committed plan cannot be fully checked against the 11-week capacity.
    Opus 5.5 · Claude · A year of spend management, with a hard deadline

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Outcomes, with certainty that falls with distance

    Does each roadmap item say the problem or outcome it serves, and does precision (dates, scope) decrease for later items?

    Passes when Every item names its outcome or problem. Near-term items are specific and later ones are deliberately looser.

  2. Fits the stated capacity

    Does the committed plan fit the team capacity the brief gives, with sums a reader could check, and does it say what was cut or deferred to make it fit?

    Passes when The committed work adds up to no more than the stated capacity, with some slack, the sums can be checked, and it names what was cut or pushed later and why.

  3. Tells the story of the bets

    Does the roadmap group the work into a few themes and explain why those are the biggest things to invest in now, rather than presenting a scored or ranked list of projects?

    Passes when A handful of themes, each with the reason it matters most now, and the items sit under them.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Sequences around dependencies · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Group PM at Fernhill. Build the roadmap for the next half (Q1 and Q2) for our two squads, in no more than 900 words plus a table: what each squad builds, in what order, the outcome each item serves, what we're not doing and why, and what we tell Calder Group. The request log, usage, capacity and goal are attached.

What the model was given5 items: About Fernhill, company_goal_h1.md, feature_requests_last_6_months.csv, feature_usage.md, capacity_and_estimates.md
About FernhillPurchasing and inventory software for restaurants and hotels: orders, deliveries, invoices and stock.
company_goal_h1.md5 lines · Download
# Company goal for the half

Cut churn among independent restaurants (82% of accounts, 61% of revenue) from 2.4% to 1.6% a month. Top churn reason in exit surveys: "too much admin; didn't save us time on deliveries and invoices".

Calder Group renews in May. Their account manager says they "might" leave without the report builder; they haven't raised it with Customer Success, and their usage is up 20% this year.
feature_requests_last_6_months.csv53 rows · Download
request,requester,requester_arr_usd,source,title,theme
FR-300,Customer 553,4353,Support ticket,Credit note tracking,Invoicing
FR-301,Customer 391,9773,Sales call,Match invoices to deliveries automatically,Invoicing
FR-302,Customer 341,4856,Support ticket,Match invoices to deliveries automatically,Invoicing
FR-303,Calder Group,186000,Account manager,ERP export (CSV variant),Integrations
FR-304,Customer 895,9595,Sales call,Mobile receiving (check deliveries on a phone),Receiving
FR-305,Calder Group,186000,Account manager,Report builder with pivots,Reporting
…
feature_usage.md8 lines · Download
# Usage of related features today (last 30 days, 1,140 accounts)

| Feature | Accounts using it | Note |
|---|---|---|
| Standard reports | 212 (19%) | Calder Group uses them daily |
| Receiving checklist (desktop only) | 640 (56%) | 71% of uses start on a desktop, then get re-typed from paper |
| Invoice upload | 870 (76%) | Matching is manual |
…
capacity_and_estimates.md5 lines · Download
# Capacity, Q1 to Q2

Two squads, about 10 squad-weeks of roadmap work each per quarter (40 in total over the half).

Estimates (squad-weeks): Mobile receiving 8. Flag short deliveries 3 (needs mobile receiving). Invoice matching 9 (needs the new invoice parser, 5). Credit note tracking 3 (needs invoice matching). Accounting export with credits 2 (needs credit note tracking). Custom report builder 12. ERP export 4. Approval chains 6. Site-level permissions 4. SSO 3.
What a strong answer doesThe answer key the graders mark against

Builds the half around the churn goal (independent restaurants, whose top exit reason is admin around deliveries and invoices), not the request count. The log has 53 requests, but 14 come from one customer, Calder Group, and several of theirs are the same thing under different names (report builder three times, ERP export twice, bulk SKU twice, scheduled reports twice); across everyone else, receiving and invoicing requests dominate (mobile receiving 9, invoice matching 8, flag short deliveries 7, credit notes 5, accounting export 4). Sequences the dependency chain: mobile receiving (8), then flag short deliveries (3); the invoice parser (5), then invoice matching (9), then credit note tracking (3), then the accounting export (2): about 30 squad-weeks, inside the 40 available with slack. Handles Calder with evidence: their usage is up 20% and they haven't raised leaving with Customer Success, so talk to them before committing the 12-week report builder; offer a smaller item (SSO, 3, or the ERP export, 4) if it fits. Says what's not being done (report builder, approval chains, dark mode, multi-currency) and why, with near-term items specific and later ones looser.

Critical failures (cap the score at 40)
  • Schedules an item before the item it depends on (for example invoice matching before the parser)
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Usable with a quick edit11 of 12 checks passed

The output is a complete, evidence-based roadmap with a clear story, actionable table and Calder plan. A PM could use it with light edits and no need to re-verify facts or restructure the reasoning.

Output

Fernhill roadmap, Q1 and Q2

The call: Both squads go to the company goal: deliveries and invoices, the top reasons independents give for leaving. Calder gets a bounded share of capacity, not the roadmap. Calder's report builder can't ship before May, so we tell them that directly and commit to a scoped plan.

Why this split: - The receiving and invoice chains (30 of 40 squad-weeks) serve 32 different non-Calder accounts. Mobile receiving and flagging short deliveries touch 16 accounts. Invoice matching, credit notes and the accounting export touch 16. - Calder has 14 of the 53 requests, but they come from one account. Every Calder-only item totals 29 squad-weeks (report builder, ERP export, site permissions, SSO, approval chains). Taking all of it would consume 73% of the half. - Usage backs the sequence. 56% of accounts use the receiving checklist, but 71% of uses start on a desktop and get re-typed from paper, so mobile receiving removes that step. Invoice matching is manual for the 76% of accounts that upload invoices.

Capacity: 2 squads × 10 squad-weeks per quarter. The plan uses 40 of 40, with a 1-week spike per squad left at the end of Q2 for report builder scoping.

OrderSquadItemWeeksTimingOutcome it servesNotes
11Mobile receiving8Q1 wk 1–8Less time per delivery; ends paper re-typing9 requests, 9 accounts
22Invoice parser5Q1 wk 1–5Prerequisite for invoice matchingOnly item with no visible customer output; it gates the invoice chain
32Invoice matching9Q1 wk 6–10 to Q2 wk 4Less manual invoice admin8 accounts; biggest admin saving
41Flag short deliveries3Q1 wk 9–10 to Q2 wk 1Catch shortages at the door, not on the invoiceNeeds item 1; 7 accounts
51ERP export (CSV/ERP format)4Q2 wk 2–5Calder renewal; integrationCalder only; ships before May renewal
62Credit note tracking3Q2 wk 5–7Credits matched to invoicesNeeds item 3; 4 accounts
72Accounting export with credits2Q2 wk 8–9Invoices and credits reach accounting without re-keyingNeeds item 6; 4 accounts
81Site-level permissions4Q2 wk 6–9Calder multi-site accessCalder only; see below
91 + 2Report builder scoping spike1 eachQ2 wk 10Scoped plan and H2 date for CalderDesign only, no build

Dependencies are the risk. Item 2 gates the invoice chain. If the parser slips, items 3, 6 and 7 slip with it, and the invoice outcome misses the half. I'd rather flag that now than hide it inside the plan. If the parser runs over, the first move is to pull the Calder-only site permissions work (item 8) back into the buffer.

Measures: Time per delivery receipt and percentage of invoices auto-matched, tracked per cohort. Independent-restaurant churn (2.4% → 1.6% a month) is the outcome. We won't claim the roadmap alone moves it.

What we're not doing, and why

  • Full custom report builder (12 weeks). It's 60% of one squad's half, for one account, and the earliest it ships is around the May renewal with no slack. We do a scoping spike and commit to a decision date, not a build date.
  • SSO (3) and approval chains with 5 levels (6). Calder only, and the 5-level approvals request is a niche configuration. H2 candidates, to be revisited with Calder.
  • Custom PDF invoices, custom order fields, per-site dashboards, bulk SKU import v2 and bulk SKU edit. All Calder-only, none estimated. Not in H1.
  • Dark mode (2 accounts), multi-currency (1 account), more suppliers (3 accounts). Low demand, no estimates. Parked, not rejected.
  • Scheduled reports by email (Calder, 2 requests). Not estimated. It goes into the spike as a candidate for a smaller Calder deliverable.

What we tell Calder Group

  1. Customer Success calls them this week, not the account manager alone. They haven't raised the report builder with CS, and "might leave" is the kind of risk that surfaces too late.
  2. Be direct about the report builder: it's a substantial build, it won't land before May, and we won't promise a date we can't keep.
  3. What they get in H1: the ERP export in the format they asked for, shipping around early May, and site-level permissions by the end of June.
  4. The report builder: we scope it with them in June, and we commit to an H2 decision date by the end of June. We don't commit to a build date until the spike returns.
  5. Ask them two things: which reports they run daily (we know they use standard reports), and what minimum output would cover them until the full builder ships. Their answer decides whether scheduled email reports move up.

Asks for leadership: confirm Calder's renewal status and ARR before the CS call, and approve the spike's two squad-weeks. If Calder won't engage on the scoped plan, we should know before the roadmap is shared externally.

Grades and run details

Decision model 79 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly6%
  • passAddresses the actual decision95%
  • partialRespects explicit constraints23%
  • passIdentifies material uncertainty91%
  • partialAvoids unsupported claims56%
  • passProduces the required deliverable69%
  • passSees through the request count88%
  • passHandles the big customer with evidence95%
  • passOutcomes, with certainty that falls with distance80%
  • partialFits the stated capacity18%
  • passSequences around dependencies73%
  • passTells the story of the bets86%
Run
Run
#1
API response time
56 s
Submitted
8 Oct 2026

Usable with a quick edit11 of 12 checks passed

The roadmap is clear, actionable, and covers all required elements. The credit note count error is minor and can be corrected with a quick edit without redoing the reasoning.

Output

Fernhill H1 roadmap (Q1–Q2)

The plan: 30 of our 40 squad-weeks go to the delivery-to-invoice workflow that exit surveys blame for churn. 4 go to Calder's ERP export, and 6 are held as buffer. We are not building the report builder.

Assumption: I've treated the invoice parser (5) as additional to invoice matching (9), so matching is 14 weeks in total. If the 9 already includes the parser, we gain 5 weeks of buffer and nothing else changes.

Roadmap

OrderSquadItemWeeksTimingOutcome servedDepends on
1AMobile receiving8Q1 W1–8Less admin on deliveries: 71% of checklist uses start on desktop and get re-typed from paper. Measure: share of receiving done on mobile.None
1BInvoice parser5Q1 W1–5Foundation for matchingNone
2BInvoice matching9Q1 W6 – Q2 W4Less admin on invoices: 870 accounts upload invoices and match them by hand. Measure: % of invoices auto-matched.Parser
2AERP export (Calder's format)4Q1 W9 – Q2 W2Protects the $186k Calder renewal in May. Needs format confirmed in January.None
3AFlag short deliveries3Q2 W3–5Catches shortages at the door, saving time recovering money laterMobile receiving
3BCredit note tracking3Q2 W5–7Less admin on creditsMatching
4BAccounting export with credits2Q2 W8–9Less bookkeeping reworkCredit notes
–A (5 wks), B (1 wk)Buffer6Q2Covers slippage on the new parser and matching, then the Calder reporting slice below–

Why this order

  • Outcome. The goal is churn among independent restaurants, and the top exit reason is admin on deliveries and invoices. Receiving and invoicing are exactly that workflow.
  • Demand. The request log is consistent with the usage data, though it shows who asks rather than who churns, so the exit survey and usage numbers carry more weight.
  • Mobile receiving: 9 customers, about $60k ARR.
  • Matching: 8 customers, about $45k.
  • Short deliveries: 7 customers, about $48k.
  • Credit notes: 4 customers.
  • Export with credits: 4 customers, about $28k.
  • Sequencing. The invoicing chain is strictly sequential, so it takes one squad for 19 of its 20 weeks. The receiving chain runs in parallel on the other squad.
  • ERP export before flagging. Calder's renewal has a hard date, and flagging doesn't. The export is two requests from Calder and the cheapest of their asks. It lands mid-April.
  • Timing risk. Mobile receiving ships around the end of March and matching around the end of April. Churn lags adoption, so I expect only partial movement toward 1.6% by June. I'd report leading indicators monthly (mobile share of receiving, auto-match rate) alongside churn, and set that expectation with the exec team now.

What we're not doing, and why

  • Custom report builder (12 weeks), plus dashboards, scheduling and custom PDFs.
  • Calder is the only requester of anything in the Reporting theme. Only 19% of accounts use standard reports at all.
  • The cost is 30% of H1 capacity. One squad would need all of Q1 plus two weeks of Q2, which pushes about 10 weeks of churn work out of H1.
  • The churn threat behind it is unverified (see below).
  • Approval chains (6), site-level permissions (4), SSO (3), custom fields, bulk SKU import v2 and bulk edit. These are all Calder-only. The three that have estimates total 13 weeks, with no independent-restaurant demand and no link to the churn reason.
  • More suppliers (3 customers), dark mode (2), multi-currency (1). They're unestimated and unconnected to the admin-time problem. I'll revisit them in the H2 planning.

The Calder rows are 14 of the 53 log entries (26%), but they come from one account via its account manager. Counted by customer, they are one voice. The log also has small inconsistencies, such as Customer 553 appearing twice with different ARR, so treat its figures as approximate.

Calder Group

What we know. Their account manager says they "might" leave without the report builder. They haven't said this to Customer Success, and their usage is up 20%, which looks like engagement rather than exit. It's a risk to test, not a requirement.

What we do internally 1. In the first two weeks of January, Customer Success and I meet Calder directly. We ask which decisions they need reports for, what standard reports lack, and whether the ERP export plus raw data in their own tools meets the need. 2. Ask the account manager to give us one ranked list from Calder instead of 14 separate asks. 3. Have engineering size a thin reporting slice (for example, scheduled standard reports). There's no estimate today. It can only be funded from the 6-week buffer, and only if the January conversation shows reporting is a real blocker. 4. Checkpoint at the end of February. If Calder confirms in writing that reporting decides the renewal, I bring the exec team a trade-off. The full builder means dropping the credit-note tail and the ERP export and running with zero buffer, and it still wouldn't ship by May.

What we tell Calder - "We've heard your reporting needs and want to understand them properly. We'd like an hour with your team in January." - "Your ERP export is on our roadmap and is due mid-April, ahead of your May renewal. We'll confirm the format with you first." - "The full report builder isn't in our plan for the first half. We'll tell you by the end of February whether a smaller reporting improvement is feasible."

We make no date commitment for the builder and no promise on SSO, permissions or approval chains. If the ERP export slips, Calder hears it from us before they find out themselves.

Risks to watch

  • Parser or matching slips. The buffer absorbs up to six weeks. Beyond that, credit notes and the accounting export move to H2, not receiving.
  • Calder's real blocker turns out to be reporting. That is the trade-off decision at the end-of-February checkpoint, made with real information instead of a "might".

Claims the judge couldn’t find in the brief

  • Credit notes: 4 customers

Grades and run details

Decision model 96 · LLM judge 92
Decision model checks
  • passUses the supplied evidence correctly4%
  • passAddresses the actual decision98%
  • partialRespects explicit constraints20%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims37%
  • passProduces the required deliverable75%
  • passSees through the request count89%
  • passHandles the big customer with evidence100%
  • passOutcomes, with certainty that falls with distance76%
  • passFits the stated capacity60%
  • passSequences around dependencies79%
  • passTells the story of the bets95%
Run
Run
#1
API response time
81 s
Submitted
6 Oct 2026

Usable with a quick edit12 of 12 checks passed

The output is well-structured, evidence-based, and complete; a PM could use it with only minor wording edits, without re-checking facts or redoing reasoning.

Output

H1 roadmap: remove delivery and invoice admin

Prioritise the receiving-to-invoice workflow, not a general reporting platform. Our goal is to reduce independent-restaurant monthly churn from 2.4% to 1.6%. These customers represent 82% of accounts and 61% of revenue; their leading exit reason directly names delivery and invoice admin.

The evidence supports this choice: - Invoice upload reaches 76% of accounts, but matching remains manual. - Receiving reaches 56%, with substantial paper-to-desktop re-entry. - Mobile receiving, invoice matching and shortage flagging attracted requests from 9, 8 and 7 customers, respectively. - Calder’s multiple requests represent one $186,000 account, not repeated incremental ARR. Its renewal risk warrants investigation, but the report-builder threat is currently second-hand and usage is growing.

Sequenced delivery plan

Squad names below describe their H1 ownership. Estimates are squad-weeks; each squad has ten per quarter.

Squad / quarterBuild order and allocationOutcome served / release gate
Invoice Automation — Q11. New invoice parser: 5 weeks. 2. Begin invoice matching: 5 of 9 weeks.Establish reliable invoice data, then build matching against deliveries. Validate representative invoices before relying on parsed data. Matching carries into Q2; do not promise a production release in Q1.
Receiving — Q11. Mobile receiving: 8 weeks. 2. Pilot, hardening and adoption: 2 weeks.Let staff record deliveries where they happen, reducing paper transcription. Pilot on real delivery shifts before wider rollout.
Invoice Automation — Q23. Complete invoice matching: remaining 4 weeks. 4. Credit note tracking: 3 weeks. 5. Accounting export with credits: 2 weeks. 1 week reserved.Reduce manual reconciliation, make outstanding credits visible, and avoid re-entering credit adjustments in accounting. Release sequentially; ambiguous matches require review, not silent acceptance.
Receiving — Q23. Flag short deliveries: 3 weeks, after mobile receiving. 7 weeks for rollout, workflow improvement and contingency.Capture discrepancies at the door so invoice reconciliation starts with accurate received quantities. Use remaining capacity to improve adoption and fix measured workflow friction, not add speculative features.

Capacity: committed feature builds total 30 of 40 squad-weeks. The other ten are explicitly allocated to rollout, quality, adoption and contingency—not assumed available for an extra large feature. Each squad stays within its quarterly limit.

The two squads must agree early on delivery-line identifiers, received quantities and discrepancy states. Invoice matching must still handle ordinary deliveries without waiting for shortage flagging.

Success measures and checkpoints

The primary measure is independent-restaurant monthly churn, tracked monthly against the 1.6% goal. Report results by adoption cohort, while recognising that adoption alone does not prove causation.

In the first two weeks, baseline: - Admin minutes per delivery and invoice. - Paper/re-entry frequency. - Manual matching effort, match accuracy and exception rate. - Credit resolution time.

Set numerical improvement targets after that baseline, rather than inventing savings from request counts. Confirm pilot customers include independent restaurants; the request log does not establish every requester’s segment.

At the Q1 review, assess mobile adoption, time saved and parser readiness. During Q2, expand releases only when accuracy and support burden are acceptable. If adoption is weak, spend the reserved capacity removing friction before widening scope.

What we are not doing

  • Custom report builder this half: 12 weeks would take the selected feature programme to 42 weeks before rollout or contingency. Standard reporting has 19% penetration; Calder’s daily use makes its needs worth understanding, but does not establish broad demand for a builder.
  • ERP export: four weeks is feasible in isolation, but is not presently justified ahead of workflow adoption and reliability. Revisit only with verified renewal requirements and an explicit capacity trade-off.
  • Approval chains, site permissions and SSO: defer; they do not directly address the leading churn reason.
  • Custom dashboards, scheduled reports, custom PDFs/fields, SKU tooling, more suppliers, dark mode and multi-currency: no H1 commitment. Several lack estimates, and universal catalogue usage is not evidence that catalogue enhancements will reduce churn.

What we tell Calder Group

Hold a joint PM, CS and account-manager conversation early in Q1—not just before May. Validate who requires the builder, which decisions it supports, and whether renewal actually depends on it.

Tell them:

> “Our H1 investment is reducing delivery and invoice administration. A custom report builder is not committed for this half. We want to understand your specific reporting gaps and test whether existing reports or a bounded export can meet them before renewal.”

Agree a written reporting-gap assessment and a pre-renewal checkpoint. Any newly verified retention requirement triggers a leadership trade-off decision—not an uncosted promise or a quiet roadmap diversion.

Grades and run details

Decision model 100 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly56%
  • passAddresses the actual decision98%
  • passRespects explicit constraints42%
  • passIdentifies material uncertainty96%
  • passAvoids unsupported claims50%
  • passProduces the required deliverable82%
  • passSees through the request count68%
  • passHandles the big customer with evidence98%
  • passOutcomes, with certainty that falls with distance85%
  • passFits the stated capacity87%
  • passSequences around dependencies76%
  • passTells the story of the bets100%
Run
Run
#1
API response time
43 s
Submitted
6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 1

Uses the supplied evidence correctlyMixedMixedRight
Haiku 5.5 · API

Every statement about the current situation can be traced to the supplied context or is derived by clear arithmetic from it, with no invented facts.

Sonnet 5.5 · API

States credit notes as 4 customers, but the request log contains 5 credit note tracking requests.

GPT-6.1 Sol · API

All factual claims about the current situation are directly supported by the supplied context, with no invented numbers or facts.

All got right 11

Addresses the actual decisionRightRightRight
Haiku 5.5 · API

The roadmap commits unambiguously to investing in deliveries and invoices to address churn, with a clear call for Calder, and states what would change the plan (parser slippage).

Sonnet 5.5 · API

Commits to not building the report builder, offers a clear plan, and says what would change the call (Calder confirming reporting is a real blocker).

GPT-6.1 Sol · API

The output commits to a clear roadmap (no report builder, focus on receiving/invoice workflow) and states that a verified retention requirement would trigger a leadership trade-off.

Respects explicit constraintsRightRightRight
Haiku 5.5 · API

The output is well under 900 words plus a table, includes the required elements (ordered items, outcomes, not-do list, Calder plan), and the proposed plan respects the squad and capacity limits.

Sonnet 5.5 · API

Stays within 900 words plus table, includes all required sections (table, what each squad builds, order, outcomes, what's not done, what to tell Calder).

GPT-6.1 Sol · API

The output is under 900 words (536 words), includes the required table, and covers what each squad builds, order, outcomes, what's not done and why, and what to tell Calder.

Identifies material uncertaintyRightRightRight
Haiku 5.5 · API

It names the key dependency risk (invoice parser slip) and how it would be resolved (pull site permissions), and acknowledges the roadmap alone won't guarantee the churn outcome.

Sonnet 5.5 · API

Names the parser assumption, Calder's real blocker, and slippage risks, with resolution steps (January meeting, February checkpoint).

GPT-6.1 Sol · API

It names specific unknowns (Calder's actual renewal dependency, adoption rates, parser accuracy) and describes how they will be resolved (joint conversation, pilot baselines, Q1 review).

Avoids unsupported claimsRightRightRight
Haiku 5.5 · API

Interpretations and forecasts (e.g., why mobile receiving reduces admin) are clearly linked to the data, and no claim is presented as fact that isn't supported.

Sonnet 5.5 · API

Interpretations are labelled as such; no confident claims are presented as fact without evidence.

GPT-6.1 Sol · API

Interpretations like Calder's threat being second-hand are clearly presented as analysis, not established fact, and no unsupported causal claims are made.

Produces the required deliverableRightRightRight
Haiku 5.5 · API

The roadmap includes a detailed table, explicit not-doing list, and a specific Calder communication plan, all usable by a Group PM as-is.

Sonnet 5.5 · API

Provides a complete H1 roadmap with table, outcomes, cut items, and Calder messaging, usable by a Group PM.

GPT-6.1 Sol · API

The roadmap is complete, in the right form for a Group PM, within the word limit, and actionable without major gaps.

Sees through the request countRightRightRight
Haiku 5.5 · API

It explicitly notes Calder's 14 requests come from a single account, de-duplicates themes, and weights demand by distinct customer count rather than raw request volume.

Sonnet 5.5 · API

Explicitly notes 14 Calder requests are from one customer, de-duplicates themes, and weighs demand by number of customers.

GPT-6.1 Sol · API

The output explicitly notes Calder's 14 requests are from one account and counts demand by distinct customers (9, 8, 7) rather than raw request count.

Handles the big customer with evidenceRightRightRight
Haiku 5.5 · API

It uses Calder's rising usage and the second-hand 'might leave' to propose a direct CS conversation and a scoping spike, instead of committing the full report builder.

Sonnet 5.5 · API

Uses rising usage and second-hand 'might leave' to propose a conversation first, offers ERP export, and does not commit the report builder.

GPT-6.1 Sol · API

It uses Calder's rising usage and the second-hand 'might leave' to propose a conversation first, not committing the report builder, and offers a bounded export if needed.

Outcomes, with certainty that falls with distanceRightRightRight
Haiku 5.5 · API

Every roadmap item states the outcome or problem it serves; early items have precise timing, and the later report-builder spike is deliberately scoped rather than committed.

Sonnet 5.5 · API

Every item names its outcome; later items (credit notes, accounting export) are explicitly contingent on buffer and may move to H2, making them deliberately looser.

GPT-6.1 Sol · API

Every item names its outcome (e.g., reduce paper transcription, reduce manual reconciliation), and near-term items are specific while later ones are deliberately looser (e.g., '7 weeks for rollout, workflow improvement and contingency').

Fits the stated capacityRightRightRight
Haiku 5.5 · API

The committed work adds up to exactly 40 squad-weeks, the sums are verifiable, it names the items deferred (full report builder, SSO, approval chains, etc.), and includes a buffer/retreat plan.

Sonnet 5.5 · API

Committed work sums to 34 weeks plus 6 buffer = 40, matching capacity; cut items are named with reasons.

GPT-6.1 Sol · API

Committed feature builds sum to 30 of 40 squad-weeks, with 10 explicitly reserved for rollout and contingency, and it names what was cut (report builder, ERP export, etc.).

Sequences around dependenciesRightRightRight
Haiku 5.5 · API

All dependencies (flag short deliveries after mobile, invoice matching after parser, credit notes after matching, accounting export after credit notes) are respected and explicitly named.

Sonnet 5.5 · API

All dependencies respected: mobile receiving before flag short deliveries, parser before matching, matching before credit notes, credit notes before accounting export.

GPT-6.1 Sol · API

All dependencies are respected (parser before matching, mobile receiving before flagging, matching before credit notes, credit notes before accounting export) and named.

Tells the story of the betsRightRightRight
Haiku 5.5 · API

The roadmap groups work into the themes of deliveries, invoices, and Calder retention, and explains why each is the most important investment now to cut churn and manage the renewal.

Sonnet 5.5 · API

Groups work into the delivery-to-invoice workflow theme and Calder ERP export, explaining why these bets matter most for churn and renewal.

GPT-6.1 Sol · API

The work is grouped into two themes (Invoice Automation, Receiving) with a clear narrative about reducing delivery and invoice admin to cut churn, not a scored list.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 79% of 150 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6.1 SolwithAPI100.096.42None
2GPT-6 AstrawithChatGPT93.9100.02None
3Opus 5.5withClaude90.296.42None
4Sonnet 5.5withAPI94.192.62None
5GPT-6 LunawithAPI92.085.42None
6Haiku 5.5withAPI74.296.42None
7Gemini 3.8 FlashwithAPI74.266.82None
8Gemini 3.5 Flash-LitewithGemini56.936.521 capped

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6.1 Sol · API: task score 98, 6.9× the cheapestGPT-6 Astra · ChatGPT: task score 97, 39× the cheapestOpus 5.5 · Claude: task score 94, 37× the cheapestSonnet 5.5 · API: task score 94, 36× the cheapestGPT-6 Luna · API: task score 87, the cheapestHaiku 5.5 · API: task score 86, 2.1× the cheapestGemini 3.8 Flash · API: task score 68, 5.5× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 44, the cheapest, 1 critical failureGPT-6.1 Sol · APIGPT-6 Astra · ChatGPTOpus 5.5 · ClaudeSonnet 5.5 · APIGPT-6 Luna · APIHaiku 5.5 · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and costAt least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API871.0×
Gemini 3.5 Flash-Lite · Gemini441.1×
Haiku 5.5 · API862.1×
Gemini 3.8 Flash · API685.5×
GPT-6.1 Sol · API986.9×
Sonnet 5.5 · API9436×
Opus 5.5 · Claude9437×
GPT-6 Astra · ChatGPT9739×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Turning strategy into a sequenced plan.

Why it matters

A roadmap is where strategy meets capacity. Dated feature lists turn guesses into promises.

What good looks like

  • Items are problems or outcomes, not just features
  • Sequencing reflects dependencies
  • Explicit trade-offs
  • Commitment falls with distance

Deliberately not measured

  • Gantt formatting
Capability tested

Sequencing under constraints

The failure we’re looking for

A dated wishlist sorted by excitement

Grading

Decision model and LLM judge, calibrated against a blind PM review