Tasks / Product Craft

Build a roadmap

Can the model sequence bets against capacity and dependencies, and explain the order?

Task type v1.0 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 72% were usable with at most a quick edit.

Reliably right

  1. Produces the required deliverable96% pass
    The roadmap is complete, in the right form for a Group PM, within the word limit, and actionable without major gaps.
    GPT-6.1 Sol · API · Fourteen requests from one customer
  2. Outcomes, with certainty that falls with distance96% pass
    Every item names its outcome (e.g., reduce paper transcription, reduce manual reconciliation), and near-term items are specific while later ones are deliberately looser (e.g., '7 weeks for rollout, workflow improvement and contingency').
    GPT-6.1 Sol · API · Fourteen requests from one customer
  3. Handles the big customer with evidence94% pass
    It uses Calder's rising usage and the second-hand 'might leave' to propose a conversation first, not committing the report builder, and offers a bounded export if needed.
    GPT-6.1 Sol · API · Fourteen requests from one customer

Where it slips

  1. Fits the stated capacity53% pass
    Q3 Expenses row lacks squad-week allocation, so the committed plan cannot be fully checked against the 11-week capacity.
    Opus 5.5 · Claude · A year of spend management, with a hard deadline
  2. Tells the story of the bets54% pass
    The roadmap table is organized by squad, not grouped into a few themes, though the case provides thematic reasoning.
    Sonnet 5.5 · API · A year of spend management, with a hard deadline
  3. Sees through the request count72% pass
    The output never counts requests by requester, does not mention that 14 requests come from Calder or that several are duplicates, and does not compare themes by customer count.
    GPT-6 Luna · API · Fourteen requests from one customer

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Outcomes, with certainty that falls with distance

    Does each roadmap item say the problem or outcome it serves, and does precision (dates, scope) decrease for later items?

    Passes when Every item names its outcome or problem. Near-term items are specific and later ones are deliberately looser.

  2. Fits the stated capacity

    Does the committed plan fit the team capacity the brief gives, with sums a reader could check, and does it say what was cut or deferred to make it fit?

    Passes when The committed work adds up to no more than the stated capacity, with some slack, the sums can be checked, and it names what was cut or pushed later and why.

  3. Tells the story of the bets

    Does the roadmap group the work into a few themes and explain why those are the biggest things to invest in now, rather than presenting a scored or ranked list of projects?

    Passes when A handful of themes, each with the reason it matters most now, and the items sit under them.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Sequences around dependencies · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Group PM at Fernhill. Build the roadmap for the next half (Q1 and Q2) for our two squads, in no more than 900 words plus a table: what each squad builds, in what order, the outcome each item serves, what we're not doing and why, and what we tell Calder Group. The request log, usage, capacity and goal are attached.

What the model was given5 items: About Fernhill, company_goal_h1.md, feature_requests_last_6_months.csv, feature_usage.md, capacity_and_estimates.md
About FernhillPurchasing and inventory software for restaurants and hotels: orders, deliveries, invoices and stock.
company_goal_h1.md5 lines · Download
# Company goal for the half

Cut churn among independent restaurants (82% of accounts, 61% of revenue) from 2.4% to 1.6% a month. Top churn reason in exit surveys: "too much admin; didn't save us time on deliveries and invoices".

Calder Group renews in May. Their account manager says they "might" leave without the report builder; they haven't raised it with Customer Success, and their usage is up 20% this year.
feature_requests_last_6_months.csv53 rows · Download
request,requester,requester_arr_usd,source,title,theme
FR-300,Customer 553,4353,Support ticket,Credit note tracking,Invoicing
FR-301,Customer 391,9773,Sales call,Match invoices to deliveries automatically,Invoicing
FR-302,Customer 341,4856,Support ticket,Match invoices to deliveries automatically,Invoicing
FR-303,Calder Group,186000,Account manager,ERP export (CSV variant),Integrations
FR-304,Customer 895,9595,Sales call,Mobile receiving (check deliveries on a phone),Receiving
FR-305,Calder Group,186000,Account manager,Report builder with pivots,Reporting
…
feature_usage.md8 lines · Download
# Usage of related features today (last 30 days, 1,140 accounts)

| Feature | Accounts using it | Note |
|---|---|---|
| Standard reports | 212 (19%) | Calder Group uses them daily |
| Receiving checklist (desktop only) | 640 (56%) | 71% of uses start on a desktop, then get re-typed from paper |
| Invoice upload | 870 (76%) | Matching is manual |
…
capacity_and_estimates.md5 lines · Download
# Capacity, Q1 to Q2

Two squads, about 10 squad-weeks of roadmap work each per quarter (40 in total over the half).

Estimates (squad-weeks): Mobile receiving 8. Flag short deliveries 3 (needs mobile receiving). Invoice matching 9 (needs the new invoice parser, 5). Credit note tracking 3 (needs invoice matching). Accounting export with credits 2 (needs credit note tracking). Custom report builder 12. ERP export 4. Approval chains 6. Site-level permissions 4. SSO 3.
What a strong answer doesThe answer key the graders mark against

Builds the half around the churn goal (independent restaurants, whose top exit reason is admin around deliveries and invoices), not the request count. The log has 53 requests, but 14 come from one customer, Calder Group, and several of theirs are the same thing under different names (report builder three times, ERP export twice, bulk SKU twice, scheduled reports twice); across everyone else, receiving and invoicing requests dominate (mobile receiving 9, invoice matching 8, flag short deliveries 7, credit notes 5, accounting export 4). Sequences the dependency chain: mobile receiving (8), then flag short deliveries (3); the invoice parser (5), then invoice matching (9), then credit note tracking (3), then the accounting export (2): about 30 squad-weeks, inside the 40 available with slack. Handles Calder with evidence: their usage is up 20% and they haven't raised leaving with Customer Success, so talk to them before committing the 12-week report builder; offer a smaller item (SSO, 3, or the ERP export, 4) if it fits. Says what's not being done (report builder, approval chains, dark mode, multi-currency) and why, with near-term items specific and later ones looser.

Critical failures (cap the score at 40)
  • Schedules an item before the item it depends on (for example invoice matching before the parser)
Task

Staff-level · v1.1

The outputs

9 setups answered this brief.

Usable with a quick edit8 of 12 checks passed

The sole factual slip (40 vs 39 requests) and the overconfident interpretation about the account manager can be corrected with a quick wording edit; the core reasoning and roadmap remain intact and do not need re-checking.

Output

Strategic Overview

Our H1 goal is to reduce independent restaurant churn from 2.4% to 1.6% per month. Independents represent 82% of our accounts and 61% of our revenue. Exit surveys point to one glaring issue: "too much admin; didn't save us time on deliveries and invoices."

Usage and request data confirm this friction: * Deliveries: 56% of accounts use the desktop receiving checklist, and 71% of those print it on paper and manually re-type it later. * Invoices: 76% of accounts upload invoices, but 100% of matching to orders/deliveries is manual. * Demand: Out of 40 feature requests from independent restaurants in the last six months, 33 are for Receiving and Invoicing workflow improvements.

We have 40 squad-weeks across two squads in H1. We will dedicate one squad to Invoicing Automation and the other to Mobile Receiving, with a small allocation in Q2 to protect the Calder Group renewal without derailing our core strategy.

---

H1 Roadmap Table

SquadQuarterInitiativeEstimateDependenciesOutcome Served
:---:---:---:---:---:---
Squad A (Invoicing)Q1New Invoice Parser5 wksNoneUnlocks automated line-item extraction to eliminate manual data entry.
Squad A (Invoicing)Q1–Q2Automated Invoice-to-Delivery Matching9 wksInvoice ParserEliminates manual invoice reconciliation (76% of accounts); directly attacks #1 churn driver.
Squad A (Invoicing)Q2Credit Note Tracking3 wksInvoice MatchingAutomates discrepancy resolution when deliveries are short or overcharged.
Squad A (Invoicing)Q2Accounting Export with Credits2 wksCredit Note TrackingCloses the loop with accounting software, removing final manual bookkeeping steps.
Squad A TotalH119 wks(1 wk buffer)
Squad B (Receiving)Q1Mobile Receiving8 wksNoneEliminates paper-to-desktop double entry (affects 71% of receiving users).
Squad B (Receiving)Q1–Q2Flag Short Deliveries at the Door3 wksMobile ReceivingCaptures delivery errors at the loading dock, feeding directly into Credit Note Tracking.
Squad B (Receiving)Q2ERP Export (CSV Variant)4 wksNoneSecures Calder Group renewal ($186k ARR) by getting their raw data into their BI/ERP tools.
Squad B (Receiving)Q2Adoption & UX Polish (Receiving/Invoicing)5 wksMobile ReceivingDrives adoption of new H1 workflows among at-risk independent accounts to ensure churn reduction.
Squad B TotalH120 wks

---

Execution Plan & Sequencing

#### Squad A: End-to-End Invoicing (19 of 20 weeks) Squad A will spend the entire half building a seamless, automated invoicing pipeline. 1. New Invoice Parser (Q1, Weeks 1–5): Required foundation. 2. Automated Invoice Matching (Q1 Week 6 – Q2 Week 4): Our second most-requested feature by independents (8 requests). Automatically reconciles uploaded invoices against delivery records. 3. Credit Note Tracking (Q2, Weeks 5–7): Requested 5 times. Ensures restaurants actually recover money when suppliers shortchange them. 4. Accounting Export with Credits (Q2, Weeks 8–9): Requested 4 times. Pushes clean, reconciled data into their accounting tools. Week 10 is reserved as buffer for edge-case supplier invoice formats.

#### Squad B: Receiving & Strategic Retention (20 of 20 weeks) Squad B will solve physical delivery admin first, then pivot to a targeted retention play. 1. Mobile Receiving (Q1, Weeks 1–8): Our #1 most-requested feature (9 requests). Lets kitchen staff check off deliveries on their phones at the back door, killing the clipboards-and-retyping workflow. 2. Flag Short Deliveries at the Door (Q1 Week 9 – Q2 Week 1): Requested 7 times. Allows staff to log missing items on their phone during check-in. (Bonus: This creates the exact data needed for Squad A’s Automated Invoice Matching and Credit Note Tracking). 3. ERP Export CSV Variant (Q2, Weeks 2–5): Delivered by early May ahead of Calder Group’s renewal. 4. Workflow Polish & Onboarding (Q2, Weeks 6–10): Shipping features doesn't lower churn unless customers use them. Squad B will spend the final 5 weeks removing friction from mobile/invoicing onboarding to ensure we hit the 1.6% churn target.

---

What We Are Not Doing (and Why)

  • Custom Report Builder (12 wks) & Reporting Add-ons: Only 19% of accounts use standard reports today. Spending 30% of our H1 capacity here starves our core receiving/invoicing workflows and guarantees we miss our churn goal.
  • Enterprise Admin (Approval chains - 6 wks, Site permissions - 4 wks, SSO - 3 wks): Zero independent restaurants asked for these. Building them turns us into bespoke enterprise software at the expense of 82% of our user base.
  • Catalogue Additions / Bulk SKU Tools: Catalogue already has 100% usage and isn't driving churn. Our Ops team can handle urgent supplier catalogue additions manually.
  • Dark Mode & Multi-Currency: Low request volume (3 total) and zero impact on admin time savings.

---

What We Tell Calder Group ($186k ARR)

The Reality: Calder’s usage is up 20% YoY, they use standard reports daily, and they haven't complained to CS. The renewal risk is likely an Account Manager hedging their bets. However, $186k is significant, so we shouldn't ignore them entirely.

The Pitch (Delivered jointly by Product, CS, and the AM): 1. Lead with Operational Value: Show them Mobile Receiving and Automated Invoice Matching shipping in Q1/early Q2. Across their multiple sites, eliminating paper receiving and manual invoice reconciliation will save their site managers hundreds of hours a month. 2. Solve the Underlying Reporting Need: Explain that we are not building a custom pivot-table builder inside Fernhill this half because we are focused on core operational automation. Instead, we are building their requested Custom ERP CSV Export in April (before their May renewal). This allows them to pull granular, site-level data directly into their existing ERP/BI tools (or Excel) to build whatever custom pivots and dashboards they need. 3. Bridge the Gap with CS: Have Customer Success set up a weekly manual email of their daily standard reports so they don't have to pull them manually.

Claims the judge couldn’t find in the brief

  • Out of 40 feature requests from independent restaurants in the last six months, 33 are for Receiving and Invoicing workflow improvements

Grades and run details

Decision model 63 · LLM judge 77
Decision model checks
  • failUses the supplied evidence correctly10%
  • partialAddresses the actual decision61%
  • partialRespects explicit constraints18%
  • partialIdentifies material uncertainty69%
  • partialAvoids unsupported claims62%
  • passProduces the required deliverable64%
  • partialSees through the request count40%
  • passHandles the big customer with evidence94%
  • passOutcomes, with certainty that falls with distance64%
  • failFits the stated capacity16%
  • passSequences around dependencies76%
  • passTells the story of the bets38%
Run
Run
#1
API response time
17 s
Submitted
13 Oct 2026

Usable with a quick edit11 of 12 checks passed

The output is a complete, evidence-based roadmap with a clear story, actionable table and Calder plan. A PM could use it with light edits and no need to re-verify facts or restructure the reasoning.

Output

Fernhill roadmap, Q1 and Q2

The call: Both squads go to the company goal: deliveries and invoices, the top reasons independents give for leaving. Calder gets a bounded share of capacity, not the roadmap. Calder's report builder can't ship before May, so we tell them that directly and commit to a scoped plan.

Why this split: - The receiving and invoice chains (30 of 40 squad-weeks) serve 32 different non-Calder accounts. Mobile receiving and flagging short deliveries touch 16 accounts. Invoice matching, credit notes and the accounting export touch 16. - Calder has 14 of the 53 requests, but they come from one account. Every Calder-only item totals 29 squad-weeks (report builder, ERP export, site permissions, SSO, approval chains). Taking all of it would consume 73% of the half. - Usage backs the sequence. 56% of accounts use the receiving checklist, but 71% of uses start on a desktop and get re-typed from paper, so mobile receiving removes that step. Invoice matching is manual for the 76% of accounts that upload invoices.

Capacity: 2 squads × 10 squad-weeks per quarter. The plan uses 40 of 40, with a 1-week spike per squad left at the end of Q2 for report builder scoping.

OrderSquadItemWeeksTimingOutcome it servesNotes
11Mobile receiving8Q1 wk 1–8Less time per delivery; ends paper re-typing9 requests, 9 accounts
22Invoice parser5Q1 wk 1–5Prerequisite for invoice matchingOnly item with no visible customer output; it gates the invoice chain
32Invoice matching9Q1 wk 6–10 to Q2 wk 4Less manual invoice admin8 accounts; biggest admin saving
41Flag short deliveries3Q1 wk 9–10 to Q2 wk 1Catch shortages at the door, not on the invoiceNeeds item 1; 7 accounts
51ERP export (CSV/ERP format)4Q2 wk 2–5Calder renewal; integrationCalder only; ships before May renewal
62Credit note tracking3Q2 wk 5–7Credits matched to invoicesNeeds item 3; 4 accounts
72Accounting export with credits2Q2 wk 8–9Invoices and credits reach accounting without re-keyingNeeds item 6; 4 accounts
81Site-level permissions4Q2 wk 6–9Calder multi-site accessCalder only; see below
91 + 2Report builder scoping spike1 eachQ2 wk 10Scoped plan and H2 date for CalderDesign only, no build

Dependencies are the risk. Item 2 gates the invoice chain. If the parser slips, items 3, 6 and 7 slip with it, and the invoice outcome misses the half. I'd rather flag that now than hide it inside the plan. If the parser runs over, the first move is to pull the Calder-only site permissions work (item 8) back into the buffer.

Measures: Time per delivery receipt and percentage of invoices auto-matched, tracked per cohort. Independent-restaurant churn (2.4% → 1.6% a month) is the outcome. We won't claim the roadmap alone moves it.

What we're not doing, and why

  • Full custom report builder (12 weeks). It's 60% of one squad's half, for one account, and the earliest it ships is around the May renewal with no slack. We do a scoping spike and commit to a decision date, not a build date.
  • SSO (3) and approval chains with 5 levels (6). Calder only, and the 5-level approvals request is a niche configuration. H2 candidates, to be revisited with Calder.
  • Custom PDF invoices, custom order fields, per-site dashboards, bulk SKU import v2 and bulk SKU edit. All Calder-only, none estimated. Not in H1.
  • Dark mode (2 accounts), multi-currency (1 account), more suppliers (3 accounts). Low demand, no estimates. Parked, not rejected.
  • Scheduled reports by email (Calder, 2 requests). Not estimated. It goes into the spike as a candidate for a smaller Calder deliverable.

What we tell Calder Group

  1. Customer Success calls them this week, not the account manager alone. They haven't raised the report builder with CS, and "might leave" is the kind of risk that surfaces too late.
  2. Be direct about the report builder: it's a substantial build, it won't land before May, and we won't promise a date we can't keep.
  3. What they get in H1: the ERP export in the format they asked for, shipping around early May, and site-level permissions by the end of June.
  4. The report builder: we scope it with them in June, and we commit to an H2 decision date by the end of June. We don't commit to a build date until the spike returns.
  5. Ask them two things: which reports they run daily (we know they use standard reports), and what minimum output would cover them until the full builder ships. Their answer decides whether scheduled email reports move up.

Asks for leadership: confirm Calder's renewal status and ARR before the CS call, and approve the spike's two squad-weeks. If Calder won't engage on the scoped plan, we should know before the roadmap is shared externally.

Grades and run details

Decision model 83 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly7%
  • passAddresses the actual decision67%
  • passRespects explicit constraints25%
  • passIdentifies material uncertainty91%
  • partialAvoids unsupported claims51%
  • passProduces the required deliverable70%
  • passSees through the request count82%
  • passHandles the big customer with evidence95%
  • passOutcomes, with certainty that falls with distance82%
  • failFits the stated capacity28%
  • passSequences around dependencies74%
  • partialTells the story of the bets37%
Run
Run
#1
API response time
56 s
Submitted
8 Oct 2026

Needs more than a quick edit9 of 12 checks passed

The output fails to de-duplicate Calder's requests and leaves no capacity slack, so a PM would need to re-check the request log and adjust the plan before using it.

Output

Strategic Context & H1 Focus

Fernhill’s H1 north star is cutting independent restaurant monthly churn from 2.4% to 1.6%. Independent restaurants represent 82% of our accounts and 61% of revenue. Exit interviews clearly highlight the churn driver: administrative burden on deliveries and invoices.

Currently: * 56% of accounts use desktop receiving, but 71% write deliveries on paper at the door and re-type them later. * 76% of accounts upload invoices, but matching is entirely manual.

To reach our churn target, H1 execution is dedicated to eliminating manual reconciliation at the back door and in the back office. Calder Group ($186k ARR) is critical, but dedicating 12 squad-weeks to a custom report builder would derail the core churn goal and still risk missing their May renewal date. We address Calder with a targeted integration release and proactive commercial management.

---

H1 Roadmap

SquadQuarterInitiativeEstimateDependencyPrimary Outcome Served
Squad 1: Receiving & DataQ1Mobile Receiving8 wksNoneEliminates paper receiving; cuts ~30–45 mins of daily double-entry per site.
Squad 1: Receiving & DataQ1–Q2Flag Short Deliveries at the Door3 wksMobile ReceivingReal-time shortage capture; generates dispute records at check-in.
Squad 1: Receiving & DataQ2Automated ERP Export4 wksNoneDe-risks Calder renewal; automates multi-site enterprise reporting pipelines.
Squad 1: Receiving & DataQ2Buffer, Rollout & Mobile QA5 wks—Hardware testing, onboarding stability, restaurant floor adoption.
Squad 2: Invoicing & FinanceQ1New Invoice Parser5 wksNoneFoundation for OCR extraction; pre-requisite for matching engine.
Squad 2: Invoicing & FinanceQ1–Q2Automated Invoice Matching9 wksNew Invoice Parser3-way match (PO to delivery to invoice); solves top independent request.
Squad 2: Invoicing & FinanceQ2Credit Note Tracking3 wksInvoice MatchingCloses loop on short deliveries, automating balance deductions.
Squad 2: Invoicing & FinanceQ2Accounting Export with Credits2 wksCredit Note TrackingOne-click sync to Xero/QuickBooks with verified vendor credits.
Squad 2: Invoicing & FinanceQ2Buffer & Accounting Integration QA1 wk—Edge-case tax/rounding rules and accounting schema validation.

---

Squad Breakdown & Sequencing

#### Squad 1 (Receiving & Data) — 20 Squad-Weeks 1. Mobile Receiving (8 wks, Q1): Build a mobile-optimized receiving flow allowing kitchen staff to check off line items directly on their phone at the loading dock. Eliminates paper double-entry (71% of receiving workflows today). 2. Flag Short Deliveries at the Door (3 wks, Q1–Q2): Extends mobile receiving to instantly flag missing quantities, take a photo, and generate a shortage log directly linked to the order. 3. Automated ERP Export (4 wks, Q2): Scheduled and manual CSV/ERP data pipeline (FR-303, FR-339). Delivers automated raw financial/operational extracts before Calder’s May renewal. 4. Buffer & Mobile Rollout (5 wks, Q2): Ensures smooth adoption in noisy kitchen environments with flaky Wi-Fi.

#### Squad 2 (Invoicing & Finance) — 20 Squad-Weeks 1. New Invoice Parser (5 wks, Q1): Modernizes parsing infrastructure to extract line items, taxes, unit costs, and invoice numbers reliably. 2. Automated Invoice Matching (9 wks, Q1–Q2): Automatically matches invoice line items against purchase orders and received delivery logs. Discrepancies are flagged automatically. 3. Credit Note Tracking (3 wks, Q2): Pairs short-delivery logs generated in Squad 1 with incoming vendor credit notes, stopping suppliers from overcharging for missing goods. 4. Accounting Export with Credits (2 wks, Q2): Exports matched invoices and net credit adjustments directly into standard general ledgers. 5. Buffer & Edge Cases (1 wk, Q2): Absorbs parser edge cases and vendor invoice layout variations.

---

What We Are NOT Doing (and Why)

1. Custom Report Builder with Pivots (12 weeks): * Why: At 12 weeks, this would consume 30% of total H1 capacity for a single account. It cannot ship and stabilize before Calder's May renewal, and standard report daily usage is already healthy (212 accounts, Calder included). 2. 5-Level Approval Chains & Site-Level Permissions (10 weeks combined): * Why: Pure enterprise overhead. Does not solve the operational time sink driving 82% of accounts to churn. 3. Dark Mode & Multi-Currency (Non-core): * Why: Zero correlation to churn reduction or operational savings. 4. Catalogue Bulk Tools & Expansion: * Why: Catalogue usage is 100% saturated. The friction point is not creating items; it is receiving and paying for them.

---

What We Tell Calder Group

#### 1. Commercial Reality Check (Internal) Calder’s usage is up 20% year-over-year, and they use Standard Reports daily. Their threat to leave originated from an Account Manager conversation, not Customer Success or executive escalation. We will not over-index engineering capacity on an AM negotiation bluff.

#### 2. External Messaging (Fernhill VP of Product & Head of CS to Calder Exec Sponsor in March) * Lead with High-Value Clean Data: “A reporting tool is only as good as the input data. By shipping Mobile Receiving and Automated Matching in Q1, Fernhill ensures every invoice, delivery quantity, and supplier credit across all Calder sites is 100% accurate and audited in real time.” * Deliver the ERP Export (Ships April, ahead of May renewal): “Rather than forcing your finance team into a proprietary, rigid reporting UI inside Fernhill, we are releasing the Automated ERP/CSV Data Export in April. This allows Calder to stream clean, site-level operational and invoice data directly into your central ERP/BI stack (PowerBI/Tableau/Excel) with your custom formulas and pivot models.” * White-Glove Support: CS will construct scheduled, bespoke standard reporting extracts tailored to their multi-site leadership team during renewal discussions.

Grades and run details

Decision model 67 · LLM judge 69
Decision model checks
  • partialUses the supplied evidence correctly10%
  • partialAddresses the actual decision61%
  • passRespects explicit constraints34%
  • failIdentifies material uncertainty30%
  • partialAvoids unsupported claims58%
  • passProduces the required deliverable72%
  • partialSees through the request count80%
  • passHandles the big customer with evidence98%
  • passOutcomes, with certainty that falls with distance62%
  • partialFits the stated capacity14%
  • passSequences around dependencies91%
  • partialTells the story of the bets25%
Run
Run
#1
API response time
31 s
Submitted
6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 5

Uses the supplied evidence correctlyWrongRightRight
Gemini 4 Argon · API

The claim that there were '40 feature requests from independent restaurants' is false; 53 total minus 14 from Calder equals 39, so a statement about the current situation is not supported.

Haiku 5.5 · API

Every statement about the current situation can be traced to the supplied context or is derived by clear arithmetic from it, with no invented facts.

Gemini 3.8 Flash · API

All factual claims about the current situation are directly supported by the supplied context or simple arithmetic from it.

Identifies material uncertaintyWrongRightWrong
Gemini 4 Argon · API

The output does not name specific unknowns that could change the plan, nor does it say how they would be resolved or what result would change the call.

Haiku 5.5 · API

It names the key dependency risk (invoice parser slip) and how it would be resolved (pull site permissions), and acknowledges the roadmap alone won't guarantee the churn outcome.

Gemini 3.8 Flash · API

It does not name the specific unknowns that could change the Calder decision (e.g., executive escalation) or say how they would be resolved; it only states the decision not to over-index.

Avoids unsupported claimsWrongRightRight
Gemini 4 Argon · API

It presents 'The renewal risk is likely an Account Manager hedging their bets' as a confident reality ('The Reality'), but this is an interpretation not established by the supplied evidence.

Haiku 5.5 · API

Interpretations and forecasts (e.g., why mobile receiving reduces admin) are clearly linked to the data, and no claim is presented as fact that isn't supported.

Gemini 3.8 Flash · API

Forecasts and interpretations are presented as roadmap outcomes or goals, not as established facts, and the few causal claims are clearly tied to the data.

Sees through the request countRightRightWrong
Gemini 4 Argon · API

The output separates Calder's 14 requests, de-duplicates their repeated asks, and weighs demand by number of independent customers asking (33 receiving/invoicing requests).

Haiku 5.5 · API

It explicitly notes Calder's 14 requests come from a single account, de-duplicates themes, and weights demand by distinct customer count rather than raw request volume.

Gemini 3.8 Flash · API

The output never mentions that 14 requests come from Calder, several are duplicates, or weighs demand by unique customers; it relies on the churn goal instead of explicitly de-duplicating the request log.

Fits the stated capacityMixedMixedWrong
Gemini 4 Argon · API

Committed work adds up to 39 squad-weeks with 1 week slack, sums are checkable, and the output names what was cut (report builder, enterprise admin, etc.) and why.

Haiku 5.5 · API

The committed work adds up to exactly 40 squad-weeks, the sums are verifiable, it names the items deferred (full report builder, SSO, approval chains, etc.), and includes a buffer/retreat plan.

Gemini 3.8 Flash · API

The committed work sums to exactly 40 squad-weeks with no slack; the buffers are allocated as work items, not as unallocated capacity for uncertainty.

All got right 7

Addresses the actual decisionRightRightRight
Gemini 4 Argon · API

The output commits unambiguously to a roadmap split across two squads, addresses the churn goal, and specifies what to tell Calder Group.

Haiku 5.5 · API

The roadmap commits unambiguously to investing in deliveries and invoices to address churn, with a clear call for Calder, and states what would change the plan (parser slippage).

Gemini 3.8 Flash · API

The output commits to a clear H1 roadmap with squad assignments, sequencing, and a specific plan for Calder, framed for the Group PM.

Respects explicit constraintsRightRightRight
Gemini 4 Argon · API

The output stays within 900 words plus table, includes the required sections (what each squad builds, order, outcomes, what we're not doing, what we tell Calder), and frames the answer for the Group PM.

Haiku 5.5 · API

The output is well under 900 words plus a table, includes the required elements (ordered items, outcomes, not-do list, Calder plan), and the proposed plan respects the squad and capacity limits.

Gemini 3.8 Flash · API

The output includes the required table, stays well under 900 words, and covers what each squad builds, order, outcomes, what's not done and why, and the Calder message.

Produces the required deliverableRightRightRight
Gemini 4 Argon · API

The required roadmap table and narrative are present, within the length, and usable by the intended reader.

Haiku 5.5 · API

The roadmap includes a detailed table, explicit not-doing list, and a specific Calder communication plan, all usable by a Group PM as-is.

Gemini 3.8 Flash · API

The roadmap is complete, in the requested format, within the word limit, and a PM could act on it with light edits.

Handles the big customer with evidenceRightRightRight
Gemini 4 Argon · API

It uses Calder's rising usage and the second-hand 'might leave' to propose a smaller item (ERP CSV export) and a conversation, rather than committing the 12-week report builder by default.

Haiku 5.5 · API

It uses Calder's rising usage and the second-hand 'might leave' to propose a direct CS conversation and a scoping spike, instead of committing the full report builder.

Gemini 3.8 Flash · API

It uses Calder's rising usage and the second-hand 'might leave' to avoid committing the report builder, instead offering a smaller ERP export and proactive commercial conversation.

Outcomes, with certainty that falls with distanceRightRightRight
Gemini 4 Argon · API

Every item names an outcome or problem; near-term items are specific (e.g., mobile receiving outcome) and later items (adoption polish) are deliberately looser.

Haiku 5.5 · API

Every roadmap item states the outcome or problem it serves; early items have precise timing, and the later report-builder spike is deliberately scoped rather than committed.

Gemini 3.8 Flash · API

Every roadmap item names its outcome or problem, and near-term items are specific while later items (buffers, edge cases) are deliberately looser.

Sequences around dependenciesRightRightRight
Gemini 4 Argon · API

The roadmap sequences invoice parser before matching, matching before credit notes, credit notes before accounting export, and mobile receiving before flag short deliveries, respecting all stated dependencies.

Haiku 5.5 · API

All dependencies (flag short deliveries after mobile, invoice matching after parser, credit notes after matching, accounting export after credit notes) are respected and explicitly named.

Gemini 3.8 Flash · API

All dependencies from the brief are respected: mobile receiving before flag short deliveries, parser before matching, matching before credit notes, credit notes before accounting export.

Tells the story of the betsRightRightRight
Gemini 4 Argon · API

The work is grouped into two themes (Invoicing Automation and Mobile Receiving) tied directly to the churn reduction goal, and the narrative explains why those bets matter most now.

Haiku 5.5 · API

The roadmap groups work into the themes of deliveries, invoices, and Calder retention, and explains why each is the most important investment now to cut churn and manage the renewal.

Gemini 3.8 Flash · API

The roadmap groups work into two themes (Receiving & Data, Invoicing & Finance) and explains why eliminating manual reconciliation is the biggest investment to reduce churn.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 79% of 150 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT88.0100.02None
2Opus 5.5withClaude88.396.42None
3GPT-6.1 SolwithAPI88.396.42None
4Sonnet 5.5withAPI90.292.62None
5Haiku 5.5withAPI74.496.42None
6GPT-6 LunawithAPI84.085.42None
7Gemini 4 ArgonwithAPI62.088.52None
8Gemini 3.8 FlashwithAPI58.366.82None
9Gemini 3.5 Flash-LitewithGemini57.136.521 capped

What you get for the cost

GPT-6.1 Sol · API scores within a point of GPT-6 Astra · ChatGPT at a sixth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 94, 39× the cheapestOpus 5.5 · Claude: task score 93, 37× the cheapestGPT-6.1 Sol · API: task score 93, 6.9× the cheapestSonnet 5.5 · API: task score 92, 36× the cheapestHaiku 5.5 · API: task score 86, 2.1× the cheapestGPT-6 Luna · API: task score 84, the cheapestGemini 4 Argon · API: task score 77, 6.7× the cheapestGemini 3.8 Flash · API: task score 60, 5.5× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 45, the cheapest, 1 critical failureGPT-6 Astra · ChatGPTOpus 5.5 · ClaudeGPT-6.1 Sol · APISonnet 5.5 · APIHaiku 5.5 · APIGPT-6 Luna · APIGemini 4 Argon · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costAt least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API841.0×
Gemini 3.5 Flash-Lite · Gemini451.1×
Haiku 5.5 · API862.1×
Gemini 3.8 Flash · API605.5×
Gemini 4 Argon · API776.7×
GPT-6.1 Sol · API936.9×
Sonnet 5.5 · API9236×
Opus 5.5 · Claude9337×
GPT-6 Astra · ChatGPT9439×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Turning strategy into a sequenced plan.

Why it matters

A roadmap is where strategy meets capacity. Dated feature lists turn guesses into promises.

What good looks like

  • Items are problems or outcomes, not just features
  • Sequencing reflects dependencies
  • Explicit trade-offs
  • Commitment falls with distance

Deliberately not measured

  • Gantt formatting
Capability tested

Sequencing under constraints

The failure we’re looking for

A dated wishlist sorted by excitement

Grading

Decision model and LLM judge, calibrated against a blind PM review