Tasks / Product Craft

Build a roadmap

Can the model sequence bets against capacity and dependencies, and explain the order?

Task type v1.0 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 69% were usable with at most a quick edit.

Reliably right

  1. Produces the required deliverable95% pass
    The roadmap is complete, in the right form for a Group PM, within the word limit, and actionable without major gaps.
    GPT-6.1 Sol · API · Fourteen requests from one customer
  2. Outcomes, with certainty that falls with distance94% pass
    Every item names its outcome (e.g., reduce paper transcription, reduce manual reconciliation), and near-term items are specific while later ones are deliberately looser (e.g., '7 weeks for rollout, workflow improvement and contingency').
    GPT-6.1 Sol · API · Fourteen requests from one customer
  3. Handles the big customer with evidence94% pass
    It uses Calder's rising usage and the second-hand 'might leave' to propose a conversation first, not committing the report builder, and offers a bounded export if needed.
    GPT-6.1 Sol · API · Fourteen requests from one customer

Where it slips

  1. Sees through the request count72% pass
    The output never counts requests by requester, does not mention that 14 requests come from Calder or that several are duplicates, and does not compare themes by customer count.
    GPT-6 Luna · API · Fourteen requests from one customer
  2. Uses the supplied evidence correctly72% pass
    States credit notes as 4 customers, but the request log contains 5 credit note tracking requests.
    Sonnet 5.5 · API · Fourteen requests from one customer
  3. Fits the stated capacity73% pass
    Q3 Expenses row lacks squad-week allocation, so the committed plan cannot be fully checked against the 11-week capacity.
    Opus 5.5 · Claude · A year of spend management, with a hard deadline

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Outcomes, with certainty that falls with distance

    Does each roadmap item say the problem or outcome it serves, and does precision (dates, scope) decrease for later items?

    Passes when Every item names its outcome or problem. Near-term items are specific and later ones are deliberately looser.

  2. Fits the stated capacity

    Does the committed plan fit the team capacity the brief gives, with sums a reader could check, and does it say what was cut or deferred to make it fit?

    Passes when The committed work adds up to no more than the stated capacity, with some slack, the sums can be checked, and it names what was cut or pushed later and why.

  3. Tells the story of the bets

    Does the roadmap group the work into a few themes and explain why those are the biggest things to invest in now, rather than presenting a scored or ranked list of projects?

    Passes when A handful of themes, each with the reason it matters most now, and the items sit under them.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Sequences around dependencies · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Group PM at Fernhill. Build the roadmap for the next half (Q1 and Q2) for our two squads, in no more than 900 words plus a table: what each squad builds, in what order, the outcome each item serves, what we're not doing and why, and what we tell Calder Group. The request log, usage, capacity and goal are attached.

What the model was given5 items: About Fernhill, company_goal_h1.md, feature_requests_last_6_months.csv, feature_usage.md, capacity_and_estimates.md
About FernhillPurchasing and inventory software for restaurants and hotels: orders, deliveries, invoices and stock.
company_goal_h1.md5 lines · Download
# Company goal for the half

Cut churn among independent restaurants (82% of accounts, 61% of revenue) from 2.4% to 1.6% a month. Top churn reason in exit surveys: "too much admin; didn't save us time on deliveries and invoices".

Calder Group renews in May. Their account manager says they "might" leave without the report builder; they haven't raised it with Customer Success, and their usage is up 20% this year.
feature_requests_last_6_months.csv53 rows · Download
request,requester,requester_arr_usd,source,title,theme
FR-300,Customer 553,4353,Support ticket,Credit note tracking,Invoicing
FR-301,Customer 391,9773,Sales call,Match invoices to deliveries automatically,Invoicing
FR-302,Customer 341,4856,Support ticket,Match invoices to deliveries automatically,Invoicing
FR-303,Calder Group,186000,Account manager,ERP export (CSV variant),Integrations
FR-304,Customer 895,9595,Sales call,Mobile receiving (check deliveries on a phone),Receiving
FR-305,Calder Group,186000,Account manager,Report builder with pivots,Reporting
…
feature_usage.md8 lines · Download
# Usage of related features today (last 30 days, 1,140 accounts)

| Feature | Accounts using it | Note |
|---|---|---|
| Standard reports | 212 (19%) | Calder Group uses them daily |
| Receiving checklist (desktop only) | 640 (56%) | 71% of uses start on a desktop, then get re-typed from paper |
| Invoice upload | 870 (76%) | Matching is manual |
…
capacity_and_estimates.md5 lines · Download
# Capacity, Q1 to Q2

Two squads, about 10 squad-weeks of roadmap work each per quarter (40 in total over the half).

Estimates (squad-weeks): Mobile receiving 8. Flag short deliveries 3 (needs mobile receiving). Invoice matching 9 (needs the new invoice parser, 5). Credit note tracking 3 (needs invoice matching). Accounting export with credits 2 (needs credit note tracking). Custom report builder 12. ERP export 4. Approval chains 6. Site-level permissions 4. SSO 3.
What a strong answer doesThe answer key the graders mark against

Builds the half around the churn goal (independent restaurants, whose top exit reason is admin around deliveries and invoices), not the request count. The log has 53 requests, but 14 come from one customer, Calder Group, and several of theirs are the same thing under different names (report builder three times, ERP export twice, bulk SKU twice, scheduled reports twice); across everyone else, receiving and invoicing requests dominate (mobile receiving 9, invoice matching 8, flag short deliveries 7, credit notes 5, accounting export 4). Sequences the dependency chain: mobile receiving (8), then flag short deliveries (3); the invoice parser (5), then invoice matching (9), then credit note tracking (3), then the accounting export (2): about 30 squad-weeks, inside the 40 available with slack. Handles Calder with evidence: their usage is up 20% and they haven't raised leaving with Customer Success, so talk to them before committing the 12-week report builder; offer a smaller item (SSO, 3, or the ERP export, 4) if it fits. Says what's not being done (report builder, approval chains, dark mode, multi-currency) and why, with near-term items specific and later ones looser.

Critical failures (cap the score at 40)
  • Schedules an item before the item it depends on (for example invoice matching before the parser)
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Usable with a quick edit11 of 12 checks passed

The roadmap is clear, actionable, and covers all required elements. The credit note count error is minor and can be corrected with a quick edit without redoing the reasoning.

Output

Fernhill H1 roadmap (Q1–Q2)

The plan: 30 of our 40 squad-weeks go to the delivery-to-invoice workflow that exit surveys blame for churn. 4 go to Calder's ERP export, and 6 are held as buffer. We are not building the report builder.

Assumption: I've treated the invoice parser (5) as additional to invoice matching (9), so matching is 14 weeks in total. If the 9 already includes the parser, we gain 5 weeks of buffer and nothing else changes.

Roadmap

OrderSquadItemWeeksTimingOutcome servedDepends on
1AMobile receiving8Q1 W1–8Less admin on deliveries: 71% of checklist uses start on desktop and get re-typed from paper. Measure: share of receiving done on mobile.None
1BInvoice parser5Q1 W1–5Foundation for matchingNone
2BInvoice matching9Q1 W6 – Q2 W4Less admin on invoices: 870 accounts upload invoices and match them by hand. Measure: % of invoices auto-matched.Parser
2AERP export (Calder's format)4Q1 W9 – Q2 W2Protects the $186k Calder renewal in May. Needs format confirmed in January.None
3AFlag short deliveries3Q2 W3–5Catches shortages at the door, saving time recovering money laterMobile receiving
3BCredit note tracking3Q2 W5–7Less admin on creditsMatching
4BAccounting export with credits2Q2 W8–9Less bookkeeping reworkCredit notes
–A (5 wks), B (1 wk)Buffer6Q2Covers slippage on the new parser and matching, then the Calder reporting slice below–

Why this order

  • Outcome. The goal is churn among independent restaurants, and the top exit reason is admin on deliveries and invoices. Receiving and invoicing are exactly that workflow.
  • Demand. The request log is consistent with the usage data, though it shows who asks rather than who churns, so the exit survey and usage numbers carry more weight.
  • Mobile receiving: 9 customers, about $60k ARR.
  • Matching: 8 customers, about $45k.
  • Short deliveries: 7 customers, about $48k.
  • Credit notes: 4 customers.
  • Export with credits: 4 customers, about $28k.
  • Sequencing. The invoicing chain is strictly sequential, so it takes one squad for 19 of its 20 weeks. The receiving chain runs in parallel on the other squad.
  • ERP export before flagging. Calder's renewal has a hard date, and flagging doesn't. The export is two requests from Calder and the cheapest of their asks. It lands mid-April.
  • Timing risk. Mobile receiving ships around the end of March and matching around the end of April. Churn lags adoption, so I expect only partial movement toward 1.6% by June. I'd report leading indicators monthly (mobile share of receiving, auto-match rate) alongside churn, and set that expectation with the exec team now.

What we're not doing, and why

  • Custom report builder (12 weeks), plus dashboards, scheduling and custom PDFs.
  • Calder is the only requester of anything in the Reporting theme. Only 19% of accounts use standard reports at all.
  • The cost is 30% of H1 capacity. One squad would need all of Q1 plus two weeks of Q2, which pushes about 10 weeks of churn work out of H1.
  • The churn threat behind it is unverified (see below).
  • Approval chains (6), site-level permissions (4), SSO (3), custom fields, bulk SKU import v2 and bulk edit. These are all Calder-only. The three that have estimates total 13 weeks, with no independent-restaurant demand and no link to the churn reason.
  • More suppliers (3 customers), dark mode (2), multi-currency (1). They're unestimated and unconnected to the admin-time problem. I'll revisit them in the H2 planning.

The Calder rows are 14 of the 53 log entries (26%), but they come from one account via its account manager. Counted by customer, they are one voice. The log also has small inconsistencies, such as Customer 553 appearing twice with different ARR, so treat its figures as approximate.

Calder Group

What we know. Their account manager says they "might" leave without the report builder. They haven't said this to Customer Success, and their usage is up 20%, which looks like engagement rather than exit. It's a risk to test, not a requirement.

What we do internally 1. In the first two weeks of January, Customer Success and I meet Calder directly. We ask which decisions they need reports for, what standard reports lack, and whether the ERP export plus raw data in their own tools meets the need. 2. Ask the account manager to give us one ranked list from Calder instead of 14 separate asks. 3. Have engineering size a thin reporting slice (for example, scheduled standard reports). There's no estimate today. It can only be funded from the 6-week buffer, and only if the January conversation shows reporting is a real blocker. 4. Checkpoint at the end of February. If Calder confirms in writing that reporting decides the renewal, I bring the exec team a trade-off. The full builder means dropping the credit-note tail and the ERP export and running with zero buffer, and it still wouldn't ship by May.

What we tell Calder - "We've heard your reporting needs and want to understand them properly. We'd like an hour with your team in January." - "Your ERP export is on our roadmap and is due mid-April, ahead of your May renewal. We'll confirm the format with you first." - "The full report builder isn't in our plan for the first half. We'll tell you by the end of February whether a smaller reporting improvement is feasible."

We make no date commitment for the builder and no promise on SSO, permissions or approval chains. If the ERP export slips, Calder hears it from us before they find out themselves.

Risks to watch

  • Parser or matching slips. The buffer absorbs up to six weeks. Beyond that, credit notes and the accounting export move to H2, not receiving.
  • Calder's real blocker turns out to be reporting. That is the trade-off decision at the end-of-February checkpoint, made with real information instead of a "might".

Claims the judge couldn’t find in the brief

  • Credit notes: 4 customers

Grades and run details

Decision model 96 · LLM judge 92
Decision model checks
  • passUses the supplied evidence correctly4%
  • passAddresses the actual decision98%
  • partialRespects explicit constraints20%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims37%
  • passProduces the required deliverable75%
  • passSees through the request count89%
  • passHandles the big customer with evidence100%
  • passOutcomes, with certainty that falls with distance76%
  • passFits the stated capacity60%
  • passSequences around dependencies79%
  • passTells the story of the bets95%
Run
Run
#1
API response time
81 s
Submitted
6 Oct 2026

Usable with a quick edit12 of 12 checks passed

The output is complete, evidence-based, and clearly structured. A PM could use it with minor wording tweaks, without re-checking facts or redoing the reasoning.

Output

H1 decision: remove delivery-to-invoice admin

Prioritise the independent-restaurant workflow: receive once, capture shortages, match invoices, track credits, export without rework. The company outcome is monthly churn falling from 2.4% to 1.6% in the segment representing 82% of accounts and 61% of revenue.

The evidence supports this over enterprise reporting: - Receiving has 640 active accounts; 71% of uses involve desktop entry followed by retyping from paper. - Invoice upload reaches 870 accounts, but matching remains manual. - These workflow requests account for 33 log entries from 32 distinct non-Calder customers. That shows breadth, although we must verify which are independent restaurants. - Calder’s 14 entries represent one $186k account, not 14 independent demand signals. Duplicate requests and inconsistent ARR entries must not inflate priority.

Roadmap and capacity

Assign stable ownership: Squad A—Receiving & Integrations; Squad B—Invoice Automation. Dates are planning targets, subject to initial technical validation.

Squad / quarterBuild order and scopeSquad-weeksOutcome / release target
---------:---
A — Q11. Mobile receiving: phone-based delivery checks and saved received quantities8Eliminate paper-to-desktop re-entry. Pilot, then release by March.
A — Q1Instrumentation, restaurant onboarding, pilot fixes2Establish baseline admin time and turn release into repeated use.
B — Q11. New invoice parser, then 2. invoice matching, first tranche5 + 5Establish dependable invoice data and build matching. No promise of completed matching in Q1.
A — Q22. Flag short deliveries, after mobile receiving3Capture discrepancies at the door rather than reconstructing them later. Target April.
A — Q23. ERP export, conditional on the Calder discovery gate below4 heldRemove a verified export workaround; potentially support renewal without building a reporting platform.
A — Q2Receiving rollout, reliability, adoption experiments3Reduce failed/incomplete receiving sessions and expand sustained restaurant adoption.
B — Q22. Finish invoice matching → 3. credit note tracking → 4. accounting export with credits4 → 3 → 2Target April matching, May credits, June export. Reduce reconciliation, credit chasing and duplicate accounting entry.
B — Q2Quality and rollout contingency1Resolve matching/export issues without cutting correctness safeguards.

Capacity: each squad uses 10 weeks per quarter; total 40. This comprises 30 weeks of core feature work, four conditionally held for ERP export, and six for instrumentation, adoption and contingency. No hidden third squad or assumed spare engineering capacity.

Matching should start with high-confidence suggestions and explicit review of exceptions—not unattended financial decisions. Credit tracking follows completed matching; accounting export follows credits. Mobile receiving precedes shortage capture. The squads should agree shared delivery/invoice identifiers early, without inventing an additional prerequisite for matching.

Outcomes and decision gates

In January, baseline independent-restaurant admin time, receiving completion, matching effort and cohort churn. Recruit restaurants doing significant delivery/invoice work, including recently disengaged accounts—not just enthusiastic requesters.

Proposed release-success thresholds, to validate against those baselines: - Receiving: at least 30% less median time per delivery, including subsequent re-entry. - Invoices: at least 30% less manual reconciliation time, with no increase in material errors. - Adoption: at least half of pilot accounts repeatedly use the relevant workflow within four weeks.

Use a staggered rollout with comparable restaurant cohorts to assess improvement, rather than attributing every churn change to launches. Review workflow outcomes monthly and segment churn throughout H1; June churn alone may not capture the full effect of later releases.

If matching or mobile receiving slips, protect the core sequence: release the conditional ERP allocation first, then move later scope rather than bypassing quality checks.

ERP gate: by end-January, PM, CS and the account manager meet Calder’s actual users and renewal decision-maker. Confirm the job, current workaround, required fields, renewal significance and whether the four-week estimate covers acceptance criteria. Commit only if this is a valuable, bounded export—not a disguised report builder. Otherwise redirect those four weeks to core-workflow quality and adoption.

What we are not doing

  • Custom report builder, pivots, site dashboards, scheduling and custom PDFs: insufficient breadth and weaker connection to the churn goal. The builder alone costs 12 weeks; it cannot fit alongside the core programme even if all ten non-core weeks were reclaimed.
  • Approval chains, permissions, SSO and custom order fields: defer pending demonstrated retention, security or contractual necessity; none is established here.
  • Catalogue expansion, bulk SKU tools, dark mode and multi-currency: defer. Catalogue’s universal usage does not establish that these additions reduce admin-related churn. Unestimated requests are not free extras.

What we tell Calder Group

“You’re important to Fernhill, and we want to understand what reporting prevents you from doing today. We are not committing to a custom report builder before your May renewal. Our H1 investment is delivery and invoice automation. We will assess whether a bounded ERP export or existing standard reports can meet your immediate need, and return with a written scope and credible timing.”

CS owns the renewal plan, with a January discovery meeting and an April executive checkpoint. Daily report usage and 20% overall usage growth do not prove safety—but an unverified “might leave” does not justify displacing the company’s retention strategy.

Grades and run details

Decision model 92 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly12%
  • passAddresses the actual decision98%
  • partialRespects explicit constraints27%
  • passIdentifies material uncertainty99%
  • partialAvoids unsupported claims38%
  • passProduces the required deliverable61%
  • passSees through the request count92%
  • passHandles the big customer with evidence100%
  • passOutcomes, with certainty that falls with distance66%
  • passFits the stated capacity56%
  • passSequences around dependencies76%
  • passTells the story of the bets99%
Run
Run
#1
Time to output
46 s
Submitted
5 Oct 2026

Needs more than a quick edit8 of 12 checks passed

The output fails to de-duplicate Calder's requests and leaves no capacity slack, so a PM would need to re-check the request log and adjust the plan before using it.

Output

Strategic Context & H1 Focus

Fernhill’s H1 north star is cutting independent restaurant monthly churn from 2.4% to 1.6%. Independent restaurants represent 82% of our accounts and 61% of revenue. Exit interviews clearly highlight the churn driver: administrative burden on deliveries and invoices.

Currently: * 56% of accounts use desktop receiving, but 71% write deliveries on paper at the door and re-type them later. * 76% of accounts upload invoices, but matching is entirely manual.

To reach our churn target, H1 execution is dedicated to eliminating manual reconciliation at the back door and in the back office. Calder Group ($186k ARR) is critical, but dedicating 12 squad-weeks to a custom report builder would derail the core churn goal and still risk missing their May renewal date. We address Calder with a targeted integration release and proactive commercial management.

---

H1 Roadmap

SquadQuarterInitiativeEstimateDependencyPrimary Outcome Served
Squad 1: Receiving & DataQ1Mobile Receiving8 wksNoneEliminates paper receiving; cuts ~30–45 mins of daily double-entry per site.
Squad 1: Receiving & DataQ1–Q2Flag Short Deliveries at the Door3 wksMobile ReceivingReal-time shortage capture; generates dispute records at check-in.
Squad 1: Receiving & DataQ2Automated ERP Export4 wksNoneDe-risks Calder renewal; automates multi-site enterprise reporting pipelines.
Squad 1: Receiving & DataQ2Buffer, Rollout & Mobile QA5 wks—Hardware testing, onboarding stability, restaurant floor adoption.
Squad 2: Invoicing & FinanceQ1New Invoice Parser5 wksNoneFoundation for OCR extraction; pre-requisite for matching engine.
Squad 2: Invoicing & FinanceQ1–Q2Automated Invoice Matching9 wksNew Invoice Parser3-way match (PO to delivery to invoice); solves top independent request.
Squad 2: Invoicing & FinanceQ2Credit Note Tracking3 wksInvoice MatchingCloses loop on short deliveries, automating balance deductions.
Squad 2: Invoicing & FinanceQ2Accounting Export with Credits2 wksCredit Note TrackingOne-click sync to Xero/QuickBooks with verified vendor credits.
Squad 2: Invoicing & FinanceQ2Buffer & Accounting Integration QA1 wk—Edge-case tax/rounding rules and accounting schema validation.

---

Squad Breakdown & Sequencing

#### Squad 1 (Receiving & Data) — 20 Squad-Weeks 1. Mobile Receiving (8 wks, Q1): Build a mobile-optimized receiving flow allowing kitchen staff to check off line items directly on their phone at the loading dock. Eliminates paper double-entry (71% of receiving workflows today). 2. Flag Short Deliveries at the Door (3 wks, Q1–Q2): Extends mobile receiving to instantly flag missing quantities, take a photo, and generate a shortage log directly linked to the order. 3. Automated ERP Export (4 wks, Q2): Scheduled and manual CSV/ERP data pipeline (FR-303, FR-339). Delivers automated raw financial/operational extracts before Calder’s May renewal. 4. Buffer & Mobile Rollout (5 wks, Q2): Ensures smooth adoption in noisy kitchen environments with flaky Wi-Fi.

#### Squad 2 (Invoicing & Finance) — 20 Squad-Weeks 1. New Invoice Parser (5 wks, Q1): Modernizes parsing infrastructure to extract line items, taxes, unit costs, and invoice numbers reliably. 2. Automated Invoice Matching (9 wks, Q1–Q2): Automatically matches invoice line items against purchase orders and received delivery logs. Discrepancies are flagged automatically. 3. Credit Note Tracking (3 wks, Q2): Pairs short-delivery logs generated in Squad 1 with incoming vendor credit notes, stopping suppliers from overcharging for missing goods. 4. Accounting Export with Credits (2 wks, Q2): Exports matched invoices and net credit adjustments directly into standard general ledgers. 5. Buffer & Edge Cases (1 wk, Q2): Absorbs parser edge cases and vendor invoice layout variations.

---

What We Are NOT Doing (and Why)

1. Custom Report Builder with Pivots (12 weeks): * Why: At 12 weeks, this would consume 30% of total H1 capacity for a single account. It cannot ship and stabilize before Calder's May renewal, and standard report daily usage is already healthy (212 accounts, Calder included). 2. 5-Level Approval Chains & Site-Level Permissions (10 weeks combined): * Why: Pure enterprise overhead. Does not solve the operational time sink driving 82% of accounts to churn. 3. Dark Mode & Multi-Currency (Non-core): * Why: Zero correlation to churn reduction or operational savings. 4. Catalogue Bulk Tools & Expansion: * Why: Catalogue usage is 100% saturated. The friction point is not creating items; it is receiving and paying for them.

---

What We Tell Calder Group

#### 1. Commercial Reality Check (Internal) Calder’s usage is up 20% year-over-year, and they use Standard Reports daily. Their threat to leave originated from an Account Manager conversation, not Customer Success or executive escalation. We will not over-index engineering capacity on an AM negotiation bluff.

#### 2. External Messaging (Fernhill VP of Product & Head of CS to Calder Exec Sponsor in March) * Lead with High-Value Clean Data: “A reporting tool is only as good as the input data. By shipping Mobile Receiving and Automated Matching in Q1, Fernhill ensures every invoice, delivery quantity, and supplier credit across all Calder sites is 100% accurate and audited in real time.” * Deliver the ERP Export (Ships April, ahead of May renewal): “Rather than forcing your finance team into a proprietary, rigid reporting UI inside Fernhill, we are releasing the Automated ERP/CSV Data Export in April. This allows Calder to stream clean, site-level operational and invoice data directly into your central ERP/BI stack (PowerBI/Tableau/Excel) with your custom formulas and pivot models.” * White-Glove Support: CS will construct scheduled, bespoke standard reporting extracts tailored to their multi-site leadership team during renewal discussions.

Grades and run details

Decision model 79 · LLM judge 69
Decision model checks
  • failUses the supplied evidence correctly28%
  • passAddresses the actual decision53%
  • passRespects explicit constraints20%
  • partialIdentifies material uncertainty25%
  • partialAvoids unsupported claims32%
  • passProduces the required deliverable75%
  • partialSees through the request count80%
  • passHandles the big customer with evidence99%
  • passOutcomes, with certainty that falls with distance65%
  • passFits the stated capacity64%
  • passSequences around dependencies92%
  • passTells the story of the bets99%
Run
Run
#1
API response time
31 s
Submitted
6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Uses the supplied evidence correctlyMixedRightMixed
Sonnet 5.5 · API

States credit notes as 4 customers, but the request log contains 5 credit note tracking requests.

GPT-6 Astra · ChatGPT

All factual claims about the current situation are directly from the supplied context or derived by arithmetic from it, with no invented facts.

Gemini 3.8 Flash · API

All factual claims about the current situation are directly supported by the supplied context or simple arithmetic from it.

Identifies material uncertaintyRightRightWrong
Sonnet 5.5 · API

Names the parser assumption, Calder's real blocker, and slippage risks, with resolution steps (January meeting, February checkpoint).

GPT-6 Astra · ChatGPT

It names specific unknowns (whether requesters are independent restaurants, whether Calder's ERP export is truly needed, risk of slips) and describes how they will be resolved or what would change the plan.

Gemini 3.8 Flash · API

It does not name the specific unknowns that could change the Calder decision (e.g., executive escalation) or say how they would be resolved; it only states the decision not to over-index.

Sees through the request countRightRightWrong
Sonnet 5.5 · API

Explicitly notes 14 Calder requests are from one customer, de-duplicates themes, and weighs demand by number of customers.

GPT-6 Astra · ChatGPT

The output explicitly notes Calder's 14 entries are one account, counts distinct non-Calder customers, and weighs demand by breadth rather than raw count.

Gemini 3.8 Flash · API

The output never mentions that 14 requests come from Calder, several are duplicates, or weighs demand by unique customers; it relies on the churn goal instead of explicitly de-duplicating the request log.

Fits the stated capacityRightRightMixed
Sonnet 5.5 · API

Committed work sums to 34 weeks plus 6 buffer = 40, matching capacity; cut items are named with reasons.

GPT-6 Astra · ChatGPT

The committed work sums to 40 squad-weeks with slack (30 core, 4 conditional, 6 instrumentation/adoption/contingency), and the output names what was cut (report builder, approval chains, etc.).

Gemini 3.8 Flash · API

The committed work sums to exactly 40 squad-weeks with no slack; the buffers are allocated as work items, not as unallocated capacity for uncertainty.

All got right 8

Addresses the actual decisionRightRightRight
Sonnet 5.5 · API

Commits to not building the report builder, offers a clear plan, and says what would change the call (Calder confirming reporting is a real blocker).

GPT-6 Astra · ChatGPT

The output commits to a clear roadmap decision (delivery-to-invoice admin focus) and includes a conditional gate for Calder that says what would change the call.

Gemini 3.8 Flash · API

The output commits to a clear H1 roadmap with squad assignments, sequencing, and a specific plan for Calder, framed for the Group PM.

Respects explicit constraintsRightRightRight
Sonnet 5.5 · API

Stays within 900 words plus table, includes all required sections (table, what each squad builds, order, outcomes, what's not done, what to tell Calder).

GPT-6 Astra · ChatGPT

The output is well under 900 words plus a table, includes all requested elements (squad builds, order, outcomes, what's not done and why, what to tell Calder), and the plan would enforce the constraints.

Gemini 3.8 Flash · API

The output includes the required table, stays well under 900 words, and covers what each squad builds, order, outcomes, what's not done and why, and the Calder message.

Avoids unsupported claimsRightRightRight
Sonnet 5.5 · API

Interpretations are labelled as such; no confident claims are presented as fact without evidence.

GPT-6 Astra · ChatGPT

Interpretations and forecasts are clearly labelled as such (e.g., 'we must verify', 'subject to initial technical validation'), and no unsupported causal claims are presented as fact.

Gemini 3.8 Flash · API

Forecasts and interpretations are presented as roadmap outcomes or goals, not as established facts, and the few causal claims are clearly tied to the data.

Produces the required deliverableRightRightRight
Sonnet 5.5 · API

Provides a complete H1 roadmap with table, outcomes, cut items, and Calder messaging, usable by a Group PM.

GPT-6 Astra · ChatGPT

The roadmap table, not-doing list, and Calder message are all present, within the word limit, and usable by a product manager with light edits.

Gemini 3.8 Flash · API

The roadmap is complete, in the requested format, within the word limit, and a PM could act on it with light edits.

Handles the big customer with evidenceRightRightRight
Sonnet 5.5 · API

Uses rising usage and second-hand 'might leave' to propose a conversation first, offers ERP export, and does not commit the report builder.

GPT-6 Astra · ChatGPT

It uses Calder's rising usage and the second-hand 'might leave' to propose a discovery meeting and a conditional ERP export, not the report builder, and explains the reasoning.

Gemini 3.8 Flash · API

It uses Calder's rising usage and the second-hand 'might leave' to avoid committing the report builder, instead offering a smaller ERP export and proactive commercial conversation.

Outcomes, with certainty that falls with distanceRightRightRight
Sonnet 5.5 · API

Every item names its outcome; later items (credit notes, accounting export) are explicitly contingent on buffer and may move to H2, making them deliberately looser.

GPT-6 Astra · ChatGPT

Every roadmap item names its outcome or problem, and later items have less precise dates and scope (e.g., 'Target April', 'conditional on the Calder discovery gate').

Gemini 3.8 Flash · API

Every roadmap item names its outcome or problem, and near-term items are specific while later items (buffers, edge cases) are deliberately looser.

Sequences around dependenciesRightRightRight
Sonnet 5.5 · API

All dependencies respected: mobile receiving before flag short deliveries, parser before matching, matching before credit notes, credit notes before accounting export.

GPT-6 Astra · ChatGPT

All dependencies from the brief are respected (parser before matching, mobile receiving before flag short, matching before credits, credits before accounting export) and the order is explained.

Gemini 3.8 Flash · API

All dependencies from the brief are respected: mobile receiving before flag short deliveries, parser before matching, matching before credit notes, credit notes before accounting export.

Tells the story of the betsRightRightRight
Sonnet 5.5 · API

Groups work into the delivery-to-invoice workflow theme and Calder ERP export, explaining why these bets matter most for churn and renewal.

GPT-6 Astra · ChatGPT

The roadmap groups work into two themes (Receiving & Integrations, Invoice Automation) under the overarching bet of removing delivery-to-invoice admin to reduce churn, with clear rationale.

Gemini 3.8 Flash · API

The roadmap groups work into two themes (Receiving & Data, Invoicing & Finance) and explains why eliminating manual reconciliation is the biggest investment to reduce churn.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 79% of 150 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6.1 SolwithAPI100.096.42None
2GPT-6 AstrawithChatGPT93.9100.02None
3Opus 5.5withClaude90.296.42None
4Sonnet 5.5withAPI94.192.62None
5GPT-6 LunawithAPI92.085.42None
6Haiku 5.5withAPI74.296.42None
7Gemini 3.8 FlashwithAPI74.266.82None
8Gemini 3.5 Flash-LitewithGemini56.936.521 capped

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6.1 Sol · API: task score 98, 6.9× the cheapestGPT-6 Astra · ChatGPT: task score 97, 39× the cheapestOpus 5.5 · Claude: task score 94, 37× the cheapestSonnet 5.5 · API: task score 94, 36× the cheapestGPT-6 Luna · API: task score 87, the cheapestHaiku 5.5 · API: task score 86, 2.1× the cheapestGemini 3.8 Flash · API: task score 68, 5.5× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 44, the cheapest, 1 critical failureGPT-6.1 Sol · APIGPT-6 Astra · ChatGPTOpus 5.5 · ClaudeSonnet 5.5 · APIGPT-6 Luna · APIHaiku 5.5 · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and costAt least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API871.0×
Gemini 3.5 Flash-Lite · Gemini441.1×
Haiku 5.5 · API862.1×
Gemini 3.8 Flash · API685.5×
GPT-6.1 Sol · API986.9×
Sonnet 5.5 · API9436×
Opus 5.5 · Claude9437×
GPT-6 Astra · ChatGPT9739×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Turning strategy into a sequenced plan.

Why it matters

A roadmap is where strategy meets capacity. Dated feature lists turn guesses into promises.

What good looks like

  • Items are problems or outcomes, not just features
  • Sequencing reflects dependencies
  • Explicit trade-offs
  • Commitment falls with distance

Deliberately not measured

  • Gantt formatting
Capability tested

Sequencing under constraints

The failure we’re looking for

A dated wishlist sorted by excitement

Grading

Decision model and LLM judge, calibrated against a blind PM review