Tasks / Product Craft

Build a roadmap

Can the model sequence bets against capacity and dependencies, and explain the order?

Task type v1.0 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 69% were usable with at most a quick edit.

Reliably right

  1. Produces the required deliverable95% pass
    The roadmap is complete, in the right form for a Group PM, within the word limit, and actionable without major gaps.
    GPT-6.1 Sol · API · Fourteen requests from one customer
  2. Outcomes, with certainty that falls with distance94% pass
    Every item names its outcome (e.g., reduce paper transcription, reduce manual reconciliation), and near-term items are specific while later ones are deliberately looser (e.g., '7 weeks for rollout, workflow improvement and contingency').
    GPT-6.1 Sol · API · Fourteen requests from one customer
  3. Handles the big customer with evidence94% pass
    It uses Calder's rising usage and the second-hand 'might leave' to propose a conversation first, not committing the report builder, and offers a bounded export if needed.
    GPT-6.1 Sol · API · Fourteen requests from one customer

Where it slips

  1. Sees through the request count72% pass
    The output never counts requests by requester, does not mention that 14 requests come from Calder or that several are duplicates, and does not compare themes by customer count.
    GPT-6 Luna · API · Fourteen requests from one customer
  2. Uses the supplied evidence correctly72% pass
    States credit notes as 4 customers, but the request log contains 5 credit note tracking requests.
    Sonnet 5.5 · API · Fourteen requests from one customer
  3. Fits the stated capacity73% pass
    Q3 Expenses row lacks squad-week allocation, so the committed plan cannot be fully checked against the 11-week capacity.
    Opus 5.5 · Claude · A year of spend management, with a hard deadline

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Outcomes, with certainty that falls with distance

    Does each roadmap item say the problem or outcome it serves, and does precision (dates, scope) decrease for later items?

    Passes when Every item names its outcome or problem. Near-term items are specific and later ones are deliberately looser.

  2. Fits the stated capacity

    Does the committed plan fit the team capacity the brief gives, with sums a reader could check, and does it say what was cut or deferred to make it fit?

    Passes when The committed work adds up to no more than the stated capacity, with some slack, the sums can be checked, and it names what was cut or pushed later and why.

  3. Tells the story of the bets

    Does the roadmap group the work into a few themes and explain why those are the biggest things to invest in now, rather than presenting a scored or ranked list of projects?

    Passes when A handful of themes, each with the reason it matters most now, and the items sit under them.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Sequences around dependencies · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Group PM at Fernhill. Build the roadmap for the next half (Q1 and Q2) for our two squads, in no more than 900 words plus a table: what each squad builds, in what order, the outcome each item serves, what we're not doing and why, and what we tell Calder Group. The request log, usage, capacity and goal are attached.

What the model was given5 items: About Fernhill, company_goal_h1.md, feature_requests_last_6_months.csv, feature_usage.md, capacity_and_estimates.md
About FernhillPurchasing and inventory software for restaurants and hotels: orders, deliveries, invoices and stock.
company_goal_h1.md5 lines · Download
# Company goal for the half

Cut churn among independent restaurants (82% of accounts, 61% of revenue) from 2.4% to 1.6% a month. Top churn reason in exit surveys: "too much admin; didn't save us time on deliveries and invoices".

Calder Group renews in May. Their account manager says they "might" leave without the report builder; they haven't raised it with Customer Success, and their usage is up 20% this year.
feature_requests_last_6_months.csv53 rows · Download
request,requester,requester_arr_usd,source,title,theme
FR-300,Customer 553,4353,Support ticket,Credit note tracking,Invoicing
FR-301,Customer 391,9773,Sales call,Match invoices to deliveries automatically,Invoicing
FR-302,Customer 341,4856,Support ticket,Match invoices to deliveries automatically,Invoicing
FR-303,Calder Group,186000,Account manager,ERP export (CSV variant),Integrations
FR-304,Customer 895,9595,Sales call,Mobile receiving (check deliveries on a phone),Receiving
FR-305,Calder Group,186000,Account manager,Report builder with pivots,Reporting
…
feature_usage.md8 lines · Download
# Usage of related features today (last 30 days, 1,140 accounts)

| Feature | Accounts using it | Note |
|---|---|---|
| Standard reports | 212 (19%) | Calder Group uses them daily |
| Receiving checklist (desktop only) | 640 (56%) | 71% of uses start on a desktop, then get re-typed from paper |
| Invoice upload | 870 (76%) | Matching is manual |
…
capacity_and_estimates.md5 lines · Download
# Capacity, Q1 to Q2

Two squads, about 10 squad-weeks of roadmap work each per quarter (40 in total over the half).

Estimates (squad-weeks): Mobile receiving 8. Flag short deliveries 3 (needs mobile receiving). Invoice matching 9 (needs the new invoice parser, 5). Credit note tracking 3 (needs invoice matching). Accounting export with credits 2 (needs credit note tracking). Custom report builder 12. ERP export 4. Approval chains 6. Site-level permissions 4. SSO 3.
What a strong answer doesThe answer key the graders mark against

Builds the half around the churn goal (independent restaurants, whose top exit reason is admin around deliveries and invoices), not the request count. The log has 53 requests, but 14 come from one customer, Calder Group, and several of theirs are the same thing under different names (report builder three times, ERP export twice, bulk SKU twice, scheduled reports twice); across everyone else, receiving and invoicing requests dominate (mobile receiving 9, invoice matching 8, flag short deliveries 7, credit notes 5, accounting export 4). Sequences the dependency chain: mobile receiving (8), then flag short deliveries (3); the invoice parser (5), then invoice matching (9), then credit note tracking (3), then the accounting export (2): about 30 squad-weeks, inside the 40 available with slack. Handles Calder with evidence: their usage is up 20% and they haven't raised leaving with Customer Success, so talk to them before committing the 12-week report builder; offer a smaller item (SSO, 3, or the ERP export, 4) if it fits. Says what's not being done (report builder, approval chains, dark mode, multi-currency) and why, with near-term items specific and later ones looser.

Critical failures (cap the score at 40)
  • Schedules an item before the item it depends on (for example invoice matching before the parser)
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Usable with a quick edit12 of 12 checks passed

The output is evidence-based, respects all constraints, and provides a complete, actionable plan that a PM could use with only minor wording adjustments.

Output

Fernhill H1 Roadmap: Receiving and Invoicing First

The call

Both squads spend most of the half on the two jobs independent restaurants say we don't save them time on: checking deliveries and reconciling invoices. That accounts for 30 of our 40 squad-weeks. Of the remaining 10, 8 go to Calder Group's two most concrete asks (ERP export and site-level permissions) and 2 stay in reserve. We are not building the custom report builder this half.

Why

  • The goal is independent-restaurant churn, and the exit reason is admin on deliveries and invoices. Those two themes make up 33 of the 53 requests in the log, from about 30 accounts across support, sales and in-app feedback:
  • Mobile receiving: 9 requests
  • Flag short deliveries: 7
  • Invoice matching: 8
  • Credit notes: 5
  • Accounting export with credits: 4
  • Usage shows where the admin sits.
  • 56% of accounts use the receiving checklist, but 71% of those checks are re-typed from paper.
  • 76% upload invoices, and every one is matched by hand.
  • Standard reports, the nearest thing to a report builder, are used by 19%.
  • Calder's 14 requests come from one person: their account manager. Calder hasn't raised the report builder with Customer Success. Their usage is up 20% and they use standard reports daily. "Might leave" is a signal worth acting on, but it doesn't justify 12 squad-weeks (30% of the half) and pushing back a churn lever. ERP export is the one Calder ask that appears twice in the log (FR-303, FR-339), and it costs 4 weeks.

Order of work

I'm assuming calendar quarters (Q1 Jan–Mar, Q2 Apr–Jun) and May as the renewal month.

Squad A (Receiving) 1. Mobile receiving (weeks 1–8). This is the biggest single fix for paper re-typing, and flag short deliveries depends on it. 2. ERP export for Calder (weeks 9–12, finishing about mid-April). It goes before short-delivery flags so it lands before Calder's May renewal rather than during it. 3. Flag short deliveries (weeks 13–15). 4. Site-level permissions (weeks 16–19). This is the default. If the Calder working session (below) surfaces something cheaper that matters more to them, such as scheduled report emails, we swap it in. Report scheduling was asked for twice but hasn't been sized yet. 5. 1 week buffer.

Squad B (Invoicing) 1. New invoice parser (weeks 1–5). 2. Invoice matching (weeks 6–14). 3. Credit note tracking (weeks 15–17). 4. Accounting export with credits (weeks 18–19). 5. 1 week buffer.

Squad B's chain is strictly sequential, so any slip pushes everything after it. The 1-week buffer is thin. If the parser runs long, we drop the accounting export to H2 rather than squeeze matching.

How we'll know it's working

Churn moves slowly, and the big items ship mid-half, so we won't get all the way from 2.4% to 1.6% from features alone by June. We'll track leading indicators monthly:

  • Receiving: share of checks started on a phone, and share re-typed from paper. Target: re-typing down from 71% to under 30% by June.
  • Invoicing: share of uploaded invoices auto-matched, and time from invoice upload to approval.
  • Churn: monthly churn among accounts using mobile receiving or matching, compared with those that don't.

What we're not doing, and why

  • Custom report builder, custom dashboards per site (Calder). 12+ weeks for one account. It would push invoice matching into H2. The low usage of standard reports doesn't suggest wider demand. We'll first find out what Calder actually needs.
  • Approval chains with 5 levels (6), SSO (3), custom fields on orders, custom PDF invoices (Calder). Single-account asks with no link to the churn goal. We'll revisit in H2 with Calder's input.
  • Bulk SKU import v2 and bulk edit for SKUs (Calder). Not sized and single-account. We'll size them in H2 if Calder confirms they matter.
  • More suppliers (3 requests). Not sized, and not tied to the churn reason.
  • Dark mode (2) and multi-currency (1). Low demand and off-goal.

A data note: Customers 553 and 294 each appear with two different ARR figures. It doesn't change any decision here, but RevOps should fix it.

What we tell Calder Group

The account manager and Customer Success should go together, in the first two weeks of Q1, well ahead of May. Customer Success needs to be in the room because Calder hasn't raised any of this with them. The message:

> "You've asked for a lot through your account manager, and we want to get the important things right before your renewal. Two commitments this half: ERP export in your format by mid-April, and site-level permissions by end of June. We're not building a full custom report builder in this period, and we'd rather tell you that now than slip it later. You use our standard reports every day, so we'd like a working session with your team to see the reports you run and the questions you can't answer yet. Scheduled report emails or saved views may get you most of the way sooner, and what we learn shapes our H2 reporting plans."

We don't promise a report builder date. If Calder makes it a hard condition of renewal after that session, it comes back to me and leadership as an explicit trade: the 12 weeks would come out of credit notes, the accounting export and short-delivery flags, which would weaken the churn goal.

Roadmap table

#ItemSquadQuarterWeeksDepends onOutcome served
1Mobile receivingAQ18noneChurn: cut paper re-typing (71% today)
2ERP exportAQ1→Q2 (done ~mid-Apr)4noneCalder May renewal
3Flag short deliveriesAQ23Mobile receivingChurn: delivery admin, credits owed
4Site-level permissions (swappable after Calder session)AQ24noneCalder retention
5Invoice parserBQ15noneEnables matching
6Invoice matchingBQ1→Q29ParserChurn: manual matching for 76% of accounts
7Credit note trackingBQ23MatchingChurn: invoice admin
8Accounting export with creditsBQ22Credit notesChurn: month-end admin (first to drop if B slips)
BufferA + BQ22Absorb slip on B's sequential chain
Total40

Grades and run details

Decision model 96 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly32%
  • passAddresses the actual decision99%
  • passRespects explicit constraints29%
  • passIdentifies material uncertainty95%
  • partialAvoids unsupported claims41%
  • passProduces the required deliverable82%
  • passSees through the request count68%
  • passHandles the big customer with evidence100%
  • passOutcomes, with certainty that falls with distance79%
  • passFits the stated capacity73%
  • passSequences around dependencies92%
  • passTells the story of the bets100%
Run
Run
#1
Time to output
53 s
Submitted
5 Oct 2026

Usable with a quick edit11 of 12 checks passed

The output is a complete, evidence-based roadmap with a clear story, actionable table and Calder plan. A PM could use it with light edits and no need to re-verify facts or restructure the reasoning.

Output

Fernhill roadmap, Q1 and Q2

The call: Both squads go to the company goal: deliveries and invoices, the top reasons independents give for leaving. Calder gets a bounded share of capacity, not the roadmap. Calder's report builder can't ship before May, so we tell them that directly and commit to a scoped plan.

Why this split: - The receiving and invoice chains (30 of 40 squad-weeks) serve 32 different non-Calder accounts. Mobile receiving and flagging short deliveries touch 16 accounts. Invoice matching, credit notes and the accounting export touch 16. - Calder has 14 of the 53 requests, but they come from one account. Every Calder-only item totals 29 squad-weeks (report builder, ERP export, site permissions, SSO, approval chains). Taking all of it would consume 73% of the half. - Usage backs the sequence. 56% of accounts use the receiving checklist, but 71% of uses start on a desktop and get re-typed from paper, so mobile receiving removes that step. Invoice matching is manual for the 76% of accounts that upload invoices.

Capacity: 2 squads × 10 squad-weeks per quarter. The plan uses 40 of 40, with a 1-week spike per squad left at the end of Q2 for report builder scoping.

OrderSquadItemWeeksTimingOutcome it servesNotes
11Mobile receiving8Q1 wk 1–8Less time per delivery; ends paper re-typing9 requests, 9 accounts
22Invoice parser5Q1 wk 1–5Prerequisite for invoice matchingOnly item with no visible customer output; it gates the invoice chain
32Invoice matching9Q1 wk 6–10 to Q2 wk 4Less manual invoice admin8 accounts; biggest admin saving
41Flag short deliveries3Q1 wk 9–10 to Q2 wk 1Catch shortages at the door, not on the invoiceNeeds item 1; 7 accounts
51ERP export (CSV/ERP format)4Q2 wk 2–5Calder renewal; integrationCalder only; ships before May renewal
62Credit note tracking3Q2 wk 5–7Credits matched to invoicesNeeds item 3; 4 accounts
72Accounting export with credits2Q2 wk 8–9Invoices and credits reach accounting without re-keyingNeeds item 6; 4 accounts
81Site-level permissions4Q2 wk 6–9Calder multi-site accessCalder only; see below
91 + 2Report builder scoping spike1 eachQ2 wk 10Scoped plan and H2 date for CalderDesign only, no build

Dependencies are the risk. Item 2 gates the invoice chain. If the parser slips, items 3, 6 and 7 slip with it, and the invoice outcome misses the half. I'd rather flag that now than hide it inside the plan. If the parser runs over, the first move is to pull the Calder-only site permissions work (item 8) back into the buffer.

Measures: Time per delivery receipt and percentage of invoices auto-matched, tracked per cohort. Independent-restaurant churn (2.4% → 1.6% a month) is the outcome. We won't claim the roadmap alone moves it.

What we're not doing, and why

  • Full custom report builder (12 weeks). It's 60% of one squad's half, for one account, and the earliest it ships is around the May renewal with no slack. We do a scoping spike and commit to a decision date, not a build date.
  • SSO (3) and approval chains with 5 levels (6). Calder only, and the 5-level approvals request is a niche configuration. H2 candidates, to be revisited with Calder.
  • Custom PDF invoices, custom order fields, per-site dashboards, bulk SKU import v2 and bulk SKU edit. All Calder-only, none estimated. Not in H1.
  • Dark mode (2 accounts), multi-currency (1 account), more suppliers (3 accounts). Low demand, no estimates. Parked, not rejected.
  • Scheduled reports by email (Calder, 2 requests). Not estimated. It goes into the spike as a candidate for a smaller Calder deliverable.

What we tell Calder Group

  1. Customer Success calls them this week, not the account manager alone. They haven't raised the report builder with CS, and "might leave" is the kind of risk that surfaces too late.
  2. Be direct about the report builder: it's a substantial build, it won't land before May, and we won't promise a date we can't keep.
  3. What they get in H1: the ERP export in the format they asked for, shipping around early May, and site-level permissions by the end of June.
  4. The report builder: we scope it with them in June, and we commit to an H2 decision date by the end of June. We don't commit to a build date until the spike returns.
  5. Ask them two things: which reports they run daily (we know they use standard reports), and what minimum output would cover them until the full builder ships. Their answer decides whether scheduled email reports move up.

Asks for leadership: confirm Calder's renewal status and ARR before the CS call, and approve the spike's two squad-weeks. If Calder won't engage on the scoped plan, we should know before the roadmap is shared externally.

Grades and run details

Decision model 79 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly6%
  • passAddresses the actual decision95%
  • partialRespects explicit constraints23%
  • passIdentifies material uncertainty91%
  • partialAvoids unsupported claims56%
  • passProduces the required deliverable69%
  • passSees through the request count88%
  • passHandles the big customer with evidence95%
  • passOutcomes, with certainty that falls with distance80%
  • partialFits the stated capacity18%
  • passSequences around dependencies73%
  • passTells the story of the bets86%
Run
Run
#1
API response time
56 s
Submitted
8 Oct 2026

Usable with a quick edit12 of 12 checks passed

The output is complete, evidence-based, and clearly structured. A PM could use it with minor wording tweaks, without re-checking facts or redoing the reasoning.

Output

H1 decision: remove delivery-to-invoice admin

Prioritise the independent-restaurant workflow: receive once, capture shortages, match invoices, track credits, export without rework. The company outcome is monthly churn falling from 2.4% to 1.6% in the segment representing 82% of accounts and 61% of revenue.

The evidence supports this over enterprise reporting: - Receiving has 640 active accounts; 71% of uses involve desktop entry followed by retyping from paper. - Invoice upload reaches 870 accounts, but matching remains manual. - These workflow requests account for 33 log entries from 32 distinct non-Calder customers. That shows breadth, although we must verify which are independent restaurants. - Calder’s 14 entries represent one $186k account, not 14 independent demand signals. Duplicate requests and inconsistent ARR entries must not inflate priority.

Roadmap and capacity

Assign stable ownership: Squad A—Receiving & Integrations; Squad B—Invoice Automation. Dates are planning targets, subject to initial technical validation.

Squad / quarterBuild order and scopeSquad-weeksOutcome / release target
---------:---
A — Q11. Mobile receiving: phone-based delivery checks and saved received quantities8Eliminate paper-to-desktop re-entry. Pilot, then release by March.
A — Q1Instrumentation, restaurant onboarding, pilot fixes2Establish baseline admin time and turn release into repeated use.
B — Q11. New invoice parser, then 2. invoice matching, first tranche5 + 5Establish dependable invoice data and build matching. No promise of completed matching in Q1.
A — Q22. Flag short deliveries, after mobile receiving3Capture discrepancies at the door rather than reconstructing them later. Target April.
A — Q23. ERP export, conditional on the Calder discovery gate below4 heldRemove a verified export workaround; potentially support renewal without building a reporting platform.
A — Q2Receiving rollout, reliability, adoption experiments3Reduce failed/incomplete receiving sessions and expand sustained restaurant adoption.
B — Q22. Finish invoice matching → 3. credit note tracking → 4. accounting export with credits4 → 3 → 2Target April matching, May credits, June export. Reduce reconciliation, credit chasing and duplicate accounting entry.
B — Q2Quality and rollout contingency1Resolve matching/export issues without cutting correctness safeguards.

Capacity: each squad uses 10 weeks per quarter; total 40. This comprises 30 weeks of core feature work, four conditionally held for ERP export, and six for instrumentation, adoption and contingency. No hidden third squad or assumed spare engineering capacity.

Matching should start with high-confidence suggestions and explicit review of exceptions—not unattended financial decisions. Credit tracking follows completed matching; accounting export follows credits. Mobile receiving precedes shortage capture. The squads should agree shared delivery/invoice identifiers early, without inventing an additional prerequisite for matching.

Outcomes and decision gates

In January, baseline independent-restaurant admin time, receiving completion, matching effort and cohort churn. Recruit restaurants doing significant delivery/invoice work, including recently disengaged accounts—not just enthusiastic requesters.

Proposed release-success thresholds, to validate against those baselines: - Receiving: at least 30% less median time per delivery, including subsequent re-entry. - Invoices: at least 30% less manual reconciliation time, with no increase in material errors. - Adoption: at least half of pilot accounts repeatedly use the relevant workflow within four weeks.

Use a staggered rollout with comparable restaurant cohorts to assess improvement, rather than attributing every churn change to launches. Review workflow outcomes monthly and segment churn throughout H1; June churn alone may not capture the full effect of later releases.

If matching or mobile receiving slips, protect the core sequence: release the conditional ERP allocation first, then move later scope rather than bypassing quality checks.

ERP gate: by end-January, PM, CS and the account manager meet Calder’s actual users and renewal decision-maker. Confirm the job, current workaround, required fields, renewal significance and whether the four-week estimate covers acceptance criteria. Commit only if this is a valuable, bounded export—not a disguised report builder. Otherwise redirect those four weeks to core-workflow quality and adoption.

What we are not doing

  • Custom report builder, pivots, site dashboards, scheduling and custom PDFs: insufficient breadth and weaker connection to the churn goal. The builder alone costs 12 weeks; it cannot fit alongside the core programme even if all ten non-core weeks were reclaimed.
  • Approval chains, permissions, SSO and custom order fields: defer pending demonstrated retention, security or contractual necessity; none is established here.
  • Catalogue expansion, bulk SKU tools, dark mode and multi-currency: defer. Catalogue’s universal usage does not establish that these additions reduce admin-related churn. Unestimated requests are not free extras.

What we tell Calder Group

“You’re important to Fernhill, and we want to understand what reporting prevents you from doing today. We are not committing to a custom report builder before your May renewal. Our H1 investment is delivery and invoice automation. We will assess whether a bounded ERP export or existing standard reports can meet your immediate need, and return with a written scope and credible timing.”

CS owns the renewal plan, with a January discovery meeting and an April executive checkpoint. Daily report usage and 20% overall usage growth do not prove safety—but an unverified “might leave” does not justify displacing the company’s retention strategy.

Grades and run details

Decision model 92 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly12%
  • passAddresses the actual decision98%
  • partialRespects explicit constraints27%
  • passIdentifies material uncertainty99%
  • partialAvoids unsupported claims38%
  • passProduces the required deliverable61%
  • passSees through the request count92%
  • passHandles the big customer with evidence100%
  • passOutcomes, with certainty that falls with distance66%
  • passFits the stated capacity56%
  • passSequences around dependencies76%
  • passTells the story of the bets99%
Run
Run
#1
Time to output
46 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 1

Uses the supplied evidence correctlyRightMixedRight
Opus 5.5 · Claude

All factual claims about the current situation are directly supported by the supplied context, with no invented numbers or events.

Haiku 5.5 · API

Every statement about the current situation can be traced to the supplied context or is derived by clear arithmetic from it, with no invented facts.

GPT-6 Astra · ChatGPT

All factual claims about the current situation are directly from the supplied context or derived by arithmetic from it, with no invented facts.

All got right 11

Addresses the actual decisionRightRightRight
Opus 5.5 · Claude

The output commits early to a clear plan (receiving and invoicing first, no report builder) and states the condition (Calder hard condition) that would change it.

Haiku 5.5 · API

The roadmap commits unambiguously to investing in deliveries and invoices to address churn, with a clear call for Calder, and states what would change the plan (parser slippage).

GPT-6 Astra · ChatGPT

The output commits to a clear roadmap decision (delivery-to-invoice admin focus) and includes a conditional gate for Calder that says what would change the call.

Respects explicit constraintsRightRightRight
Opus 5.5 · Claude

The output is under 900 words plus a table, includes the required sections (what each squad builds, order, outcomes, what's not done, what to tell Calder), and respects the reader and form.

Haiku 5.5 · API

The output is well under 900 words plus a table, includes the required elements (ordered items, outcomes, not-do list, Calder plan), and the proposed plan respects the squad and capacity limits.

GPT-6 Astra · ChatGPT

The output is well under 900 words plus a table, includes all requested elements (squad builds, order, outcomes, what's not done and why, what to tell Calder), and the plan would enforce the constraints.

Identifies material uncertaintyRightRightRight
Opus 5.5 · Claude

It names specific unknowns (Calder's real needs, report scheduling sizing, parser slip risk) and says how they'd be resolved or what would change the call.

Haiku 5.5 · API

It names the key dependency risk (invoice parser slip) and how it would be resolved (pull site permissions), and acknowledges the roadmap alone won't guarantee the churn outcome.

GPT-6 Astra · ChatGPT

It names specific unknowns (whether requesters are independent restaurants, whether Calder's ERP export is truly needed, risk of slips) and describes how they will be resolved or what would change the plan.

Avoids unsupported claimsRightRightRight
Opus 5.5 · Claude

Interpretations and forecasts are clearly presented as judgments or plans, not as established facts.

Haiku 5.5 · API

Interpretations and forecasts (e.g., why mobile receiving reduces admin) are clearly linked to the data, and no claim is presented as fact that isn't supported.

GPT-6 Astra · ChatGPT

Interpretations and forecasts are clearly labelled as such (e.g., 'we must verify', 'subject to initial technical validation'), and no unsupported causal claims are presented as fact.

Produces the required deliverableRightRightRight
Opus 5.5 · Claude

The roadmap is complete with a table, narrative, and Calder message, all within the word limit, and a PM could act on it with light edits.

Haiku 5.5 · API

The roadmap includes a detailed table, explicit not-doing list, and a specific Calder communication plan, all usable by a Group PM as-is.

GPT-6 Astra · ChatGPT

The roadmap table, not-doing list, and Calder message are all present, within the word limit, and usable by a product manager with light edits.

Sees through the request countRightRightRight
Opus 5.5 · Claude

It explicitly notes Calder's 14 requests come from one person, de-duplicates ERP export, and weighs demand by distinct customers rather than raw count.

Haiku 5.5 · API

It explicitly notes Calder's 14 requests come from a single account, de-duplicates themes, and weights demand by distinct customer count rather than raw request volume.

GPT-6 Astra · ChatGPT

The output explicitly notes Calder's 14 entries are one account, counts distinct non-Calder customers, and weighs demand by breadth rather than raw count.

Handles the big customer with evidenceRightRightRight
Opus 5.5 · Claude

It uses Calder's rising usage and second-hand 'might leave' to propose a conversation first, offering smaller items (ERP export, site-level permissions) instead of the report builder.

Haiku 5.5 · API

It uses Calder's rising usage and the second-hand 'might leave' to propose a direct CS conversation and a scoping spike, instead of committing the full report builder.

GPT-6 Astra · ChatGPT

It uses Calder's rising usage and the second-hand 'might leave' to propose a discovery meeting and a conditional ERP export, not the report builder, and explains the reasoning.

Outcomes, with certainty that falls with distanceRightRightRight
Opus 5.5 · Claude

Every item names its outcome, near-term items are specific, and later items (site-level permissions, accounting export) are explicitly swappable or droppable.

Haiku 5.5 · API

Every roadmap item states the outcome or problem it serves; early items have precise timing, and the later report-builder spike is deliberately scoped rather than committed.

GPT-6 Astra · ChatGPT

Every roadmap item names its outcome or problem, and later items have less precise dates and scope (e.g., 'Target April', 'conditional on the Calder discovery gate').

Fits the stated capacityRightRightRight
Opus 5.5 · Claude

The committed work sums to exactly 40 squad-weeks with 2 weeks of buffer, the arithmetic is checkable, and it names what was cut (report builder, approval chains, etc.) and why.

Haiku 5.5 · API

The committed work adds up to exactly 40 squad-weeks, the sums are verifiable, it names the items deferred (full report builder, SSO, approval chains, etc.), and includes a buffer/retreat plan.

GPT-6 Astra · ChatGPT

The committed work sums to 40 squad-weeks with slack (30 core, 4 conditional, 6 instrumentation/adoption/contingency), and the output names what was cut (report builder, approval chains, etc.).

Sequences around dependenciesRightRightRight
Opus 5.5 · Claude

All dependencies from the brief are respected (mobile receiving before flag short deliveries, parser before matching, matching before credit notes, credit notes before accounting export) and named.

Haiku 5.5 · API

All dependencies (flag short deliveries after mobile, invoice matching after parser, credit notes after matching, accounting export after credit notes) are respected and explicitly named.

GPT-6 Astra · ChatGPT

All dependencies from the brief are respected (parser before matching, mobile receiving before flag short, matching before credits, credits before accounting export) and the order is explained.

Tells the story of the betsRightRightRight
Opus 5.5 · Claude

The work is grouped into two clear themes (receiving and invoicing) tied directly to the churn goal, not presented as a scored list.

Haiku 5.5 · API

The roadmap groups work into the themes of deliveries, invoices, and Calder retention, and explains why each is the most important investment now to cut churn and manage the renewal.

GPT-6 Astra · ChatGPT

The roadmap groups work into two themes (Receiving & Integrations, Invoice Automation) under the overarching bet of removing delivery-to-invoice admin to reduce churn, with clear rationale.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 79% of 150 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6.1 SolwithAPI100.096.42None
2GPT-6 AstrawithChatGPT93.9100.02None
3Opus 5.5withClaude90.296.42None
4Sonnet 5.5withAPI94.192.62None
5GPT-6 LunawithAPI92.085.42None
6Haiku 5.5withAPI74.296.42None
7Gemini 3.8 FlashwithAPI74.266.82None
8Gemini 3.5 Flash-LitewithGemini56.936.521 capped

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6.1 Sol · API: task score 98, 6.9× the cheapestGPT-6 Astra · ChatGPT: task score 97, 39× the cheapestOpus 5.5 · Claude: task score 94, 37× the cheapestSonnet 5.5 · API: task score 94, 36× the cheapestGPT-6 Luna · API: task score 87, the cheapestHaiku 5.5 · API: task score 86, 2.1× the cheapestGemini 3.8 Flash · API: task score 68, 5.5× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 44, the cheapest, 1 critical failureGPT-6.1 Sol · APIGPT-6 Astra · ChatGPTOpus 5.5 · ClaudeSonnet 5.5 · APIGPT-6 Luna · APIHaiku 5.5 · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and costAt least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API871.0×
Gemini 3.5 Flash-Lite · Gemini441.1×
Haiku 5.5 · API862.1×
Gemini 3.8 Flash · API685.5×
GPT-6.1 Sol · API986.9×
Sonnet 5.5 · API9436×
Opus 5.5 · Claude9437×
GPT-6 Astra · ChatGPT9739×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Turning strategy into a sequenced plan.

Why it matters

A roadmap is where strategy meets capacity. Dated feature lists turn guesses into promises.

What good looks like

  • Items are problems or outcomes, not just features
  • Sequencing reflects dependencies
  • Explicit trade-offs
  • Commitment falls with distance

Deliberately not measured

  • Gantt formatting
Capability tested

Sequencing under constraints

The failure we’re looking for

A dated wishlist sorted by excitement

Grading

Decision model and LLM judge, calibrated against a blind PM review