Tasks / Product Craft

Build a roadmap

Can the model sequence bets against capacity and dependencies, and explain the order?

Task type v1.0 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 69% were usable with at most a quick edit.

Reliably right

  1. Produces the required deliverable95% pass
    The roadmap is complete, in the right form for a Group PM, within the word limit, and actionable without major gaps.
    GPT-6.1 Sol · API · Fourteen requests from one customer
  2. Outcomes, with certainty that falls with distance94% pass
    Every item names its outcome (e.g., reduce paper transcription, reduce manual reconciliation), and near-term items are specific while later ones are deliberately looser (e.g., '7 weeks for rollout, workflow improvement and contingency').
    GPT-6.1 Sol · API · Fourteen requests from one customer
  3. Handles the big customer with evidence94% pass
    It uses Calder's rising usage and the second-hand 'might leave' to propose a conversation first, not committing the report builder, and offers a bounded export if needed.
    GPT-6.1 Sol · API · Fourteen requests from one customer

Where it slips

  1. Sees through the request count72% pass
    The output never counts requests by requester, does not mention that 14 requests come from Calder or that several are duplicates, and does not compare themes by customer count.
    GPT-6 Luna · API · Fourteen requests from one customer
  2. Uses the supplied evidence correctly72% pass
    States credit notes as 4 customers, but the request log contains 5 credit note tracking requests.
    Sonnet 5.5 · API · Fourteen requests from one customer
  3. Fits the stated capacity73% pass
    Q3 Expenses row lacks squad-week allocation, so the committed plan cannot be fully checked against the 11-week capacity.
    Opus 5.5 · Claude · A year of spend management, with a hard deadline

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Outcomes, with certainty that falls with distance

    Does each roadmap item say the problem or outcome it serves, and does precision (dates, scope) decrease for later items?

    Passes when Every item names its outcome or problem. Near-term items are specific and later ones are deliberately looser.

  2. Fits the stated capacity

    Does the committed plan fit the team capacity the brief gives, with sums a reader could check, and does it say what was cut or deferred to make it fit?

    Passes when The committed work adds up to no more than the stated capacity, with some slack, the sums can be checked, and it names what was cut or pushed later and why.

  3. Tells the story of the bets

    Does the roadmap group the work into a few themes and explain why those are the biggest things to invest in now, rather than presenting a scored or ranked list of projects?

    Passes when A handful of themes, each with the reason it matters most now, and the items sit under them.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Sequences around dependencies · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Group PM at Fernhill. Build the roadmap for the next half (Q1 and Q2) for our two squads, in no more than 900 words plus a table: what each squad builds, in what order, the outcome each item serves, what we're not doing and why, and what we tell Calder Group. The request log, usage, capacity and goal are attached.

What the model was given5 items: About Fernhill, company_goal_h1.md, feature_requests_last_6_months.csv, feature_usage.md, capacity_and_estimates.md
About FernhillPurchasing and inventory software for restaurants and hotels: orders, deliveries, invoices and stock.
company_goal_h1.md5 lines · Download
# Company goal for the half

Cut churn among independent restaurants (82% of accounts, 61% of revenue) from 2.4% to 1.6% a month. Top churn reason in exit surveys: "too much admin; didn't save us time on deliveries and invoices".

Calder Group renews in May. Their account manager says they "might" leave without the report builder; they haven't raised it with Customer Success, and their usage is up 20% this year.
feature_requests_last_6_months.csv53 rows · Download
request,requester,requester_arr_usd,source,title,theme
FR-300,Customer 553,4353,Support ticket,Credit note tracking,Invoicing
FR-301,Customer 391,9773,Sales call,Match invoices to deliveries automatically,Invoicing
FR-302,Customer 341,4856,Support ticket,Match invoices to deliveries automatically,Invoicing
FR-303,Calder Group,186000,Account manager,ERP export (CSV variant),Integrations
FR-304,Customer 895,9595,Sales call,Mobile receiving (check deliveries on a phone),Receiving
FR-305,Calder Group,186000,Account manager,Report builder with pivots,Reporting
…
feature_usage.md8 lines · Download
# Usage of related features today (last 30 days, 1,140 accounts)

| Feature | Accounts using it | Note |
|---|---|---|
| Standard reports | 212 (19%) | Calder Group uses them daily |
| Receiving checklist (desktop only) | 640 (56%) | 71% of uses start on a desktop, then get re-typed from paper |
| Invoice upload | 870 (76%) | Matching is manual |
…
capacity_and_estimates.md5 lines · Download
# Capacity, Q1 to Q2

Two squads, about 10 squad-weeks of roadmap work each per quarter (40 in total over the half).

Estimates (squad-weeks): Mobile receiving 8. Flag short deliveries 3 (needs mobile receiving). Invoice matching 9 (needs the new invoice parser, 5). Credit note tracking 3 (needs invoice matching). Accounting export with credits 2 (needs credit note tracking). Custom report builder 12. ERP export 4. Approval chains 6. Site-level permissions 4. SSO 3.
What a strong answer doesThe answer key the graders mark against

Builds the half around the churn goal (independent restaurants, whose top exit reason is admin around deliveries and invoices), not the request count. The log has 53 requests, but 14 come from one customer, Calder Group, and several of theirs are the same thing under different names (report builder three times, ERP export twice, bulk SKU twice, scheduled reports twice); across everyone else, receiving and invoicing requests dominate (mobile receiving 9, invoice matching 8, flag short deliveries 7, credit notes 5, accounting export 4). Sequences the dependency chain: mobile receiving (8), then flag short deliveries (3); the invoice parser (5), then invoice matching (9), then credit note tracking (3), then the accounting export (2): about 30 squad-weeks, inside the 40 available with slack. Handles Calder with evidence: their usage is up 20% and they haven't raised leaving with Customer Success, so talk to them before committing the 12-week report builder; offer a smaller item (SSO, 3, or the ERP export, 4) if it fits. Says what's not being done (report builder, approval chains, dark mode, multi-currency) and why, with near-term items specific and later ones looser.

Critical failures (cap the score at 40)
  • Schedules an item before the item it depends on (for example invoice matching before the parser)
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Usable with a quick edit12 of 12 checks passed

The output is well-structured, evidence-based, and complete; a PM could use it with only minor wording edits, without re-checking facts or redoing reasoning.

Output

H1 roadmap: remove delivery and invoice admin

Prioritise the receiving-to-invoice workflow, not a general reporting platform. Our goal is to reduce independent-restaurant monthly churn from 2.4% to 1.6%. These customers represent 82% of accounts and 61% of revenue; their leading exit reason directly names delivery and invoice admin.

The evidence supports this choice: - Invoice upload reaches 76% of accounts, but matching remains manual. - Receiving reaches 56%, with substantial paper-to-desktop re-entry. - Mobile receiving, invoice matching and shortage flagging attracted requests from 9, 8 and 7 customers, respectively. - Calder’s multiple requests represent one $186,000 account, not repeated incremental ARR. Its renewal risk warrants investigation, but the report-builder threat is currently second-hand and usage is growing.

Sequenced delivery plan

Squad names below describe their H1 ownership. Estimates are squad-weeks; each squad has ten per quarter.

Squad / quarterBuild order and allocationOutcome served / release gate
Invoice Automation — Q11. New invoice parser: 5 weeks. 2. Begin invoice matching: 5 of 9 weeks.Establish reliable invoice data, then build matching against deliveries. Validate representative invoices before relying on parsed data. Matching carries into Q2; do not promise a production release in Q1.
Receiving — Q11. Mobile receiving: 8 weeks. 2. Pilot, hardening and adoption: 2 weeks.Let staff record deliveries where they happen, reducing paper transcription. Pilot on real delivery shifts before wider rollout.
Invoice Automation — Q23. Complete invoice matching: remaining 4 weeks. 4. Credit note tracking: 3 weeks. 5. Accounting export with credits: 2 weeks. 1 week reserved.Reduce manual reconciliation, make outstanding credits visible, and avoid re-entering credit adjustments in accounting. Release sequentially; ambiguous matches require review, not silent acceptance.
Receiving — Q23. Flag short deliveries: 3 weeks, after mobile receiving. 7 weeks for rollout, workflow improvement and contingency.Capture discrepancies at the door so invoice reconciliation starts with accurate received quantities. Use remaining capacity to improve adoption and fix measured workflow friction, not add speculative features.

Capacity: committed feature builds total 30 of 40 squad-weeks. The other ten are explicitly allocated to rollout, quality, adoption and contingency—not assumed available for an extra large feature. Each squad stays within its quarterly limit.

The two squads must agree early on delivery-line identifiers, received quantities and discrepancy states. Invoice matching must still handle ordinary deliveries without waiting for shortage flagging.

Success measures and checkpoints

The primary measure is independent-restaurant monthly churn, tracked monthly against the 1.6% goal. Report results by adoption cohort, while recognising that adoption alone does not prove causation.

In the first two weeks, baseline: - Admin minutes per delivery and invoice. - Paper/re-entry frequency. - Manual matching effort, match accuracy and exception rate. - Credit resolution time.

Set numerical improvement targets after that baseline, rather than inventing savings from request counts. Confirm pilot customers include independent restaurants; the request log does not establish every requester’s segment.

At the Q1 review, assess mobile adoption, time saved and parser readiness. During Q2, expand releases only when accuracy and support burden are acceptable. If adoption is weak, spend the reserved capacity removing friction before widening scope.

What we are not doing

  • Custom report builder this half: 12 weeks would take the selected feature programme to 42 weeks before rollout or contingency. Standard reporting has 19% penetration; Calder’s daily use makes its needs worth understanding, but does not establish broad demand for a builder.
  • ERP export: four weeks is feasible in isolation, but is not presently justified ahead of workflow adoption and reliability. Revisit only with verified renewal requirements and an explicit capacity trade-off.
  • Approval chains, site permissions and SSO: defer; they do not directly address the leading churn reason.
  • Custom dashboards, scheduled reports, custom PDFs/fields, SKU tooling, more suppliers, dark mode and multi-currency: no H1 commitment. Several lack estimates, and universal catalogue usage is not evidence that catalogue enhancements will reduce churn.

What we tell Calder Group

Hold a joint PM, CS and account-manager conversation early in Q1—not just before May. Validate who requires the builder, which decisions it supports, and whether renewal actually depends on it.

Tell them:

> “Our H1 investment is reducing delivery and invoice administration. A custom report builder is not committed for this half. We want to understand your specific reporting gaps and test whether existing reports or a bounded export can meet them before renewal.”

Agree a written reporting-gap assessment and a pre-renewal checkpoint. Any newly verified retention requirement triggers a leadership trade-off decision—not an uncosted promise or a quiet roadmap diversion.

Grades and run details

Decision model 100 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly56%
  • passAddresses the actual decision98%
  • passRespects explicit constraints42%
  • passIdentifies material uncertainty96%
  • passAvoids unsupported claims50%
  • passProduces the required deliverable82%
  • passSees through the request count68%
  • passHandles the big customer with evidence98%
  • passOutcomes, with certainty that falls with distance85%
  • passFits the stated capacity87%
  • passSequences around dependencies76%
  • passTells the story of the bets100%
Run
Run
#1
API response time
43 s
Submitted
6 Oct 2026

Needs more than a quick edit8 of 12 checks passed

The output fails to de-duplicate Calder's requests and leaves no capacity slack, so a PM would need to re-check the request log and adjust the plan before using it.

Output

Strategic Context & H1 Focus

Fernhill’s H1 north star is cutting independent restaurant monthly churn from 2.4% to 1.6%. Independent restaurants represent 82% of our accounts and 61% of revenue. Exit interviews clearly highlight the churn driver: administrative burden on deliveries and invoices.

Currently: * 56% of accounts use desktop receiving, but 71% write deliveries on paper at the door and re-type them later. * 76% of accounts upload invoices, but matching is entirely manual.

To reach our churn target, H1 execution is dedicated to eliminating manual reconciliation at the back door and in the back office. Calder Group ($186k ARR) is critical, but dedicating 12 squad-weeks to a custom report builder would derail the core churn goal and still risk missing their May renewal date. We address Calder with a targeted integration release and proactive commercial management.

---

H1 Roadmap

SquadQuarterInitiativeEstimateDependencyPrimary Outcome Served
Squad 1: Receiving & DataQ1Mobile Receiving8 wksNoneEliminates paper receiving; cuts ~30–45 mins of daily double-entry per site.
Squad 1: Receiving & DataQ1–Q2Flag Short Deliveries at the Door3 wksMobile ReceivingReal-time shortage capture; generates dispute records at check-in.
Squad 1: Receiving & DataQ2Automated ERP Export4 wksNoneDe-risks Calder renewal; automates multi-site enterprise reporting pipelines.
Squad 1: Receiving & DataQ2Buffer, Rollout & Mobile QA5 wks—Hardware testing, onboarding stability, restaurant floor adoption.
Squad 2: Invoicing & FinanceQ1New Invoice Parser5 wksNoneFoundation for OCR extraction; pre-requisite for matching engine.
Squad 2: Invoicing & FinanceQ1–Q2Automated Invoice Matching9 wksNew Invoice Parser3-way match (PO to delivery to invoice); solves top independent request.
Squad 2: Invoicing & FinanceQ2Credit Note Tracking3 wksInvoice MatchingCloses loop on short deliveries, automating balance deductions.
Squad 2: Invoicing & FinanceQ2Accounting Export with Credits2 wksCredit Note TrackingOne-click sync to Xero/QuickBooks with verified vendor credits.
Squad 2: Invoicing & FinanceQ2Buffer & Accounting Integration QA1 wk—Edge-case tax/rounding rules and accounting schema validation.

---

Squad Breakdown & Sequencing

#### Squad 1 (Receiving & Data) — 20 Squad-Weeks 1. Mobile Receiving (8 wks, Q1): Build a mobile-optimized receiving flow allowing kitchen staff to check off line items directly on their phone at the loading dock. Eliminates paper double-entry (71% of receiving workflows today). 2. Flag Short Deliveries at the Door (3 wks, Q1–Q2): Extends mobile receiving to instantly flag missing quantities, take a photo, and generate a shortage log directly linked to the order. 3. Automated ERP Export (4 wks, Q2): Scheduled and manual CSV/ERP data pipeline (FR-303, FR-339). Delivers automated raw financial/operational extracts before Calder’s May renewal. 4. Buffer & Mobile Rollout (5 wks, Q2): Ensures smooth adoption in noisy kitchen environments with flaky Wi-Fi.

#### Squad 2 (Invoicing & Finance) — 20 Squad-Weeks 1. New Invoice Parser (5 wks, Q1): Modernizes parsing infrastructure to extract line items, taxes, unit costs, and invoice numbers reliably. 2. Automated Invoice Matching (9 wks, Q1–Q2): Automatically matches invoice line items against purchase orders and received delivery logs. Discrepancies are flagged automatically. 3. Credit Note Tracking (3 wks, Q2): Pairs short-delivery logs generated in Squad 1 with incoming vendor credit notes, stopping suppliers from overcharging for missing goods. 4. Accounting Export with Credits (2 wks, Q2): Exports matched invoices and net credit adjustments directly into standard general ledgers. 5. Buffer & Edge Cases (1 wk, Q2): Absorbs parser edge cases and vendor invoice layout variations.

---

What We Are NOT Doing (and Why)

1. Custom Report Builder with Pivots (12 weeks): * Why: At 12 weeks, this would consume 30% of total H1 capacity for a single account. It cannot ship and stabilize before Calder's May renewal, and standard report daily usage is already healthy (212 accounts, Calder included). 2. 5-Level Approval Chains & Site-Level Permissions (10 weeks combined): * Why: Pure enterprise overhead. Does not solve the operational time sink driving 82% of accounts to churn. 3. Dark Mode & Multi-Currency (Non-core): * Why: Zero correlation to churn reduction or operational savings. 4. Catalogue Bulk Tools & Expansion: * Why: Catalogue usage is 100% saturated. The friction point is not creating items; it is receiving and paying for them.

---

What We Tell Calder Group

#### 1. Commercial Reality Check (Internal) Calder’s usage is up 20% year-over-year, and they use Standard Reports daily. Their threat to leave originated from an Account Manager conversation, not Customer Success or executive escalation. We will not over-index engineering capacity on an AM negotiation bluff.

#### 2. External Messaging (Fernhill VP of Product & Head of CS to Calder Exec Sponsor in March) * Lead with High-Value Clean Data: “A reporting tool is only as good as the input data. By shipping Mobile Receiving and Automated Matching in Q1, Fernhill ensures every invoice, delivery quantity, and supplier credit across all Calder sites is 100% accurate and audited in real time.” * Deliver the ERP Export (Ships April, ahead of May renewal): “Rather than forcing your finance team into a proprietary, rigid reporting UI inside Fernhill, we are releasing the Automated ERP/CSV Data Export in April. This allows Calder to stream clean, site-level operational and invoice data directly into your central ERP/BI stack (PowerBI/Tableau/Excel) with your custom formulas and pivot models.” * White-Glove Support: CS will construct scheduled, bespoke standard reporting extracts tailored to their multi-site leadership team during renewal discussions.

Grades and run details

Decision model 79 · LLM judge 69
Decision model checks
  • failUses the supplied evidence correctly28%
  • passAddresses the actual decision53%
  • passRespects explicit constraints20%
  • partialIdentifies material uncertainty25%
  • partialAvoids unsupported claims32%
  • passProduces the required deliverable75%
  • partialSees through the request count80%
  • passHandles the big customer with evidence99%
  • passOutcomes, with certainty that falls with distance65%
  • passFits the stated capacity64%
  • passSequences around dependencies92%
  • passTells the story of the bets99%
Run
Run
#1
API response time
31 s
Submitted
6 Oct 2026

Needs more than a quick edit3 of 12 checks passed

The invoice matching estimate is incorrect, so the capacity plan is wrong and would need rework. A PM cannot use this without re-checking the evidence and recalculating.

Output

Fernhill H1 Product Roadmap (Q1–Q2)

Strategic Focus & Alignment

Our overriding company goal for H1 is to cut churn among independent restaurants (82% of accounts, 61% of revenue) from 2.4% to 1.6% a month. Exit surveys reveal the primary culprit: "too much admin; didn't save us time on deliveries and invoices."

Independent operators are drowning in manual paperwork at the back door and during invoice reconciliation. Therefore, both squads will be entirely laser-focused on automating receiving and invoice workflows in Q1 and Q2.

We have 40 total squad-weeks available (2 squads $\times$ 10 weeks per quarter). We are ruthlessly prioritizing features that directly target independent restaurant admin reduction, while deferring enterprise-heavy requests.

---

Squad 1: Receiving & Back-Door Ops

Squad 1 owns the physical receiving workflow, eliminating the friction of transcribing paper notes and catching vendor discrepancies at the door.

  • Q1 (Weeks 1–8): Mobile Receiving (8 squad-weeks)
  • Outcome: Enables staff to check deliveries directly on a phone or mobile device at the loading dock, eliminating the current clunky desktop-to-paper re-typing workflow used by 56% of accounts. Directly targets independent churn by saving hours of back-door admin.
  • Q1–Q2 (Overlap Weeks 9–10 & Q2 Weeks 1–3): Flag Short Deliveries at the Door (3 squad-weeks)
  • Outcome: Built immediately upon Mobile Receiving. Allows staff to instantly log missing or damaged items during the mobile check-in process, generating immediate vendor credit trails and saving reconciliation disputes.

---

Squad 2: Invoicing & Reconciliation

Squad 2 owns the financial workflow post-delivery, automating the tedious matching of paper/digital invoices against actual stock received.

  • Q1 (Weeks 1–9): Automated Invoice Matching (9 squad-weeks, including parser)
  • Outcome: Leverages the new invoice parser (5 weeks) alongside matching logic (4 weeks) to automatically reconcile uploaded invoices against verified deliveries. This attacks the second half of the independent churn bottleneck: manual invoice data entry.
  • Q2 (Weeks 1–4): Credit Note Tracking & Accounting Export (5 squad-weeks total)
  • Credit Note Tracking (3 wks): Tracks vendor credits resulting from short deliveries.
  • Accounting Export with Credits (2 wks): Seamlessly pushes reconciled invoices and credit notes to accounting software, removing end-of-month manual entry.

---

Roadmap Table

SquadQ1 InitiativeSquad-WeeksQ2 InitiativeSquad-WeeksPrimary Outcome
:---:---:---:---:---:---
Squad 1Mobile Receiving8Flag Short Deliveries3Eliminate paper re-typing at the back door; surface short deliveries instantly.
Squad 2Automated Invoice Matching9Credit Notes & Acct. Export5Automate invoice-to-delivery reconciliation and sync clean data to accounting.

---

What We Are Not Doing and Why

To hit our churn-reduction target, we are intentionally declining several high-visibility requests, notably from Calder Group (who accounts for $186k ARR, but whose account manager notes they "might" leave without the report builder).

1. Custom Report Builder (12 squad-weeks) & Calder’s Enterprise Requests (Approval chains, SSO, Site-level permissions, Custom PDF invoices): * Why: Building the Custom Report Builder would consume 30% of our entire company capacity for the half. While Calder Group represents significant ARR, their usage is actually up 20% this year, they have raised no complaints with Customer Success, and their renewal is in May. Diverting 12 weeks to appease an AM warning without CS validation while independent churn burns our revenue base is an unacceptable strategic risk. 2. More Suppliers / Catalogue Updates / Dark Mode / Multi-currency (FR-307, FR-315, FR-351, etc.): * Why: These are nice-to-haves that do not directly address the core operational admin burden cited in independent restaurant exit surveys.

---

What We Tell Calder Group

> "We value our partnership immensely and are thrilled to see your platform usage grow by 20% this year. For H1, our entire product and engineering organization is locked onto a company-wide mandate to eliminate back-office administrative overhead for operators—specifically delivering mobile receiving, short-delivery flagging, and automated invoice matching. > > Because these workflows directly accelerate daily operational efficiency, we aren't scheduling the Custom Report Builder or enterprise admin controls for Q1/Q2. However, your Account Manager and our Customer Success team are logging your specific reporting and export requirements so we can scope them for a future release cycle post-May renewal. We are fully committed to ensuring Fernhill continues to scale with Calder Group's operational needs."

Claims the judge couldn’t find in the brief

  • Automated Invoice Matching takes 9 squad-weeks including the parser (parser 5 weeks, matching logic 4 weeks).

Grades and run details

Decision model 79 · LLM judge 23
Decision model checks
  • passUses the supplied evidence correctly34%
  • passAddresses the actual decision32%
  • partialRespects explicit constraints9%
  • partialIdentifies material uncertainty28%
  • partialAvoids unsupported claims43%
  • passProduces the required deliverable26%
  • partialSees through the request count86%
  • passHandles the big customer with evidence37%
  • partialOutcomes, with certainty that falls with distance22%
  • passFits the stated capacity63%
  • passSequences around dependencies27%
  • passTells the story of the bets99%
Run
Run
#1
Time to output
6 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 9

Uses the supplied evidence correctlyRightMixedMixed
GPT-6.1 Sol · API

All factual claims about the current situation are directly supported by the supplied context, with no invented numbers or facts.

Gemini 3.8 Flash · API

All factual claims about the current situation are directly supported by the supplied context or simple arithmetic from it.

Gemini 3.5 Flash-Lite · Gemini

The output claims Automated Invoice Matching is 9 squad-weeks including the parser, but the supplied context indicates invoice matching is 9 weeks and the parser is a separate 5-week dependency, making the total 14.

Addresses the actual decisionRightRightMixed
GPT-6.1 Sol · API

The output commits to a clear roadmap (no report builder, focus on receiving/invoice workflow) and states that a verified retention requirement would trigger a leadership trade-off.

Gemini 3.8 Flash · API

The output commits to a clear H1 roadmap with squad assignments, sequencing, and a specific plan for Calder, framed for the Group PM.

Gemini 3.5 Flash-Lite · Gemini

The output does not state what result or condition would change the roadmap decision.

Identifies material uncertaintyRightWrongWrong
GPT-6.1 Sol · API

It names specific unknowns (Calder's actual renewal dependency, adoption rates, parser accuracy) and describes how they will be resolved (joint conversation, pilot baselines, Q1 review).

Gemini 3.8 Flash · API

It does not name the specific unknowns that could change the Calder decision (e.g., executive escalation) or say how they would be resolved; it only states the decision not to over-index.

Gemini 3.5 Flash-Lite · Gemini

The output does not name any unknowns that could change the decision or how they would be resolved.

Avoids unsupported claimsRightRightWrong
GPT-6.1 Sol · API

Interpretations like Calder's threat being second-hand are clearly presented as analysis, not established fact, and no unsupported causal claims are made.

Gemini 3.8 Flash · API

Forecasts and interpretations are presented as roadmap outcomes or goals, not as established facts, and the few causal claims are clearly tied to the data.

Gemini 3.5 Flash-Lite · Gemini

The output presents the invoice matching estimate (9 weeks including parser) as fact when the evidence does not support it.

Produces the required deliverableRightRightMixed
GPT-6.1 Sol · API

The roadmap is complete, in the right form for a Group PM, within the word limit, and actionable without major gaps.

Gemini 3.8 Flash · API

The roadmap is complete, in the requested format, within the word limit, and a PM could act on it with light edits.

Gemini 3.5 Flash-Lite · Gemini

The capacity error makes the roadmap unusable without re-checking the evidence and redoing the capacity math.

Sees through the request countRightWrongWrong
GPT-6.1 Sol · API

The output explicitly notes Calder's 14 requests are from one account and counts demand by distinct customers (9, 8, 7) rather than raw request count.

Gemini 3.8 Flash · API

The output never mentions that 14 requests come from Calder, several are duplicates, or weighs demand by unique customers; it relies on the churn goal instead of explicitly de-duplicating the request log.

Gemini 3.5 Flash-Lite · Gemini

The output does not count requests by requester or de-duplicate Calder Group's multiple requests; it only notes Calder's requests are enterprise-heavy.

Handles the big customer with evidenceRightRightMixed
GPT-6.1 Sol · API

It uses Calder's rising usage and the second-hand 'might leave' to propose a conversation first, not committing the report builder, and offers a bounded export if needed.

Gemini 3.8 Flash · API

It uses Calder's rising usage and the second-hand 'might leave' to avoid committing the report builder, instead offering a smaller ERP export and proactive commercial conversation.

Gemini 3.5 Flash-Lite · Gemini

The output does not offer Calder Group a smaller item (like SSO or ERP export) to address their concerns, only defers everything.

Outcomes, with certainty that falls with distanceRightRightWrong
GPT-6.1 Sol · API

Every item names its outcome (e.g., reduce paper transcription, reduce manual reconciliation), and near-term items are specific while later ones are deliberately looser (e.g., '7 weeks for rollout, workflow improvement and contingency').

Gemini 3.8 Flash · API

Every roadmap item names its outcome or problem, and near-term items are specific while later items (buffers, edge cases) are deliberately looser.

Gemini 3.5 Flash-Lite · Gemini

Q2 items are given specific week ranges (e.g., 'Q2 Weeks 1-4') rather than deliberately looser precision for later items.

Fits the stated capacityRightMixedMixed
GPT-6.1 Sol · API

Committed feature builds sum to 30 of 40 squad-weeks, with 10 explicitly reserved for rollout and contingency, and it names what was cut (report builder, ERP export, etc.).

Gemini 3.8 Flash · API

The committed work sums to exactly 40 squad-weeks with no slack; the buffers are allocated as work items, not as unallocated capacity for uncertainty.

Gemini 3.5 Flash-Lite · Gemini

The committed plan sums to 25 squad-weeks based on an incorrect estimate; the actual required work (parser 5 + matching 9 + credit notes 3 + export 2 + mobile 8 + flag 3) totals 30, and the output's capacity claim is therefore invalid.

All got right 3

Respects explicit constraintsRightRightRight
GPT-6.1 Sol · API

The output is under 900 words (536 words), includes the required table, and covers what each squad builds, order, outcomes, what's not done and why, and what to tell Calder.

Gemini 3.8 Flash · API

The output includes the required table, stays well under 900 words, and covers what each squad builds, order, outcomes, what's not done and why, and the Calder message.

Gemini 3.5 Flash-Lite · Gemini

The output is under 900 words, includes a table, and covers what each squad builds, order, outcomes, what's not being done, and what to tell Calder Group.

Sequences around dependenciesRightRightRight
GPT-6.1 Sol · API

All dependencies are respected (parser before matching, mobile receiving before flagging, matching before credit notes, credit notes before accounting export) and named.

Gemini 3.8 Flash · API

All dependencies from the brief are respected: mobile receiving before flag short deliveries, parser before matching, matching before credit notes, credit notes before accounting export.

Gemini 3.5 Flash-Lite · Gemini

The roadmap places flag short deliveries after mobile receiving, and the invoice matching item includes the parser, implying the parser is done first.

Tells the story of the betsRightRightRight
GPT-6.1 Sol · API

The work is grouped into two themes (Invoice Automation, Receiving) with a clear narrative about reducing delivery and invoice admin to cut churn, not a scored list.

Gemini 3.8 Flash · API

The roadmap groups work into two themes (Receiving & Data, Invoicing & Finance) and explains why eliminating manual reconciliation is the biggest investment to reduce churn.

Gemini 3.5 Flash-Lite · Gemini

The work is grouped into two themes (Receiving & Back-Door Ops, Invoicing & Reconciliation) with clear reasons tied to the churn goal.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 79% of 150 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6.1 SolwithAPI100.096.42None
2GPT-6 AstrawithChatGPT93.9100.02None
3Opus 5.5withClaude90.296.42None
4Sonnet 5.5withAPI94.192.62None
5GPT-6 LunawithAPI92.085.42None
6Haiku 5.5withAPI74.296.42None
7Gemini 3.8 FlashwithAPI74.266.82None
8Gemini 3.5 Flash-LitewithGemini56.936.521 capped

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6.1 Sol · API: task score 98, 6.9× the cheapestGPT-6 Astra · ChatGPT: task score 97, 39× the cheapestOpus 5.5 · Claude: task score 94, 37× the cheapestSonnet 5.5 · API: task score 94, 36× the cheapestGPT-6 Luna · API: task score 87, the cheapestHaiku 5.5 · API: task score 86, 2.1× the cheapestGemini 3.8 Flash · API: task score 68, 5.5× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 44, the cheapest, 1 critical failureGPT-6.1 Sol · APIGPT-6 Astra · ChatGPTOpus 5.5 · ClaudeSonnet 5.5 · APIGPT-6 Luna · APIHaiku 5.5 · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and costAt least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API871.0×
Gemini 3.5 Flash-Lite · Gemini441.1×
Haiku 5.5 · API862.1×
Gemini 3.8 Flash · API685.5×
GPT-6.1 Sol · API986.9×
Sonnet 5.5 · API9436×
Opus 5.5 · Claude9437×
GPT-6 Astra · ChatGPT9739×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Turning strategy into a sequenced plan.

Why it matters

A roadmap is where strategy meets capacity. Dated feature lists turn guesses into promises.

What good looks like

  • Items are problems or outcomes, not just features
  • Sequencing reflects dependencies
  • Explicit trade-offs
  • Commitment falls with distance

Deliberately not measured

  • Gantt formatting
Capability tested

Sequencing under constraints

The failure we’re looking for

A dated wishlist sorted by excitement

Grading

Decision model and LLM judge, calibrated against a blind PM review