Tasks / Product Craft

Build a roadmap

Can the model sequence bets against capacity and dependencies, and explain the order?

Task type v1.0 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 72% were usable with at most a quick edit.

Reliably right

  1. Produces the required deliverable96% pass
    The roadmap is complete, in the right form for a Group PM, within the word limit, and actionable without major gaps.
    GPT-6.1 Sol · API · Fourteen requests from one customer
  2. Outcomes, with certainty that falls with distance96% pass
    Every item names its outcome (e.g., reduce paper transcription, reduce manual reconciliation), and near-term items are specific while later ones are deliberately looser (e.g., '7 weeks for rollout, workflow improvement and contingency').
    GPT-6.1 Sol · API · Fourteen requests from one customer
  3. Handles the big customer with evidence94% pass
    It uses Calder's rising usage and the second-hand 'might leave' to propose a conversation first, not committing the report builder, and offers a bounded export if needed.
    GPT-6.1 Sol · API · Fourteen requests from one customer

Where it slips

  1. Fits the stated capacity53% pass
    Q3 Expenses row lacks squad-week allocation, so the committed plan cannot be fully checked against the 11-week capacity.
    Opus 5.5 · Claude · A year of spend management, with a hard deadline
  2. Tells the story of the bets54% pass
    The roadmap table is organized by squad, not grouped into a few themes, though the case provides thematic reasoning.
    Sonnet 5.5 · API · A year of spend management, with a hard deadline
  3. Sees through the request count72% pass
    The output never counts requests by requester, does not mention that 14 requests come from Calder or that several are duplicates, and does not compare themes by customer count.
    GPT-6 Luna · API · Fourteen requests from one customer

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Outcomes, with certainty that falls with distance

    Does each roadmap item say the problem or outcome it serves, and does precision (dates, scope) decrease for later items?

    Passes when Every item names its outcome or problem. Near-term items are specific and later ones are deliberately looser.

  2. Fits the stated capacity

    Does the committed plan fit the team capacity the brief gives, with sums a reader could check, and does it say what was cut or deferred to make it fit?

    Passes when The committed work adds up to no more than the stated capacity, with some slack, the sums can be checked, and it names what was cut or pushed later and why.

  3. Tells the story of the bets

    Does the roadmap group the work into a few themes and explain why those are the biggest things to invest in now, rather than presenting a scored or ranked list of projects?

    Passes when A handful of themes, each with the reason it matters most now, and the items sit under them.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Sequences around dependencies · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Group PM at Fernhill. Build the roadmap for the next half (Q1 and Q2) for our two squads, in no more than 900 words plus a table: what each squad builds, in what order, the outcome each item serves, what we're not doing and why, and what we tell Calder Group. The request log, usage, capacity and goal are attached.

What the model was given5 items: About Fernhill, company_goal_h1.md, feature_requests_last_6_months.csv, feature_usage.md, capacity_and_estimates.md
About FernhillPurchasing and inventory software for restaurants and hotels: orders, deliveries, invoices and stock.
company_goal_h1.md5 lines · Download
# Company goal for the half

Cut churn among independent restaurants (82% of accounts, 61% of revenue) from 2.4% to 1.6% a month. Top churn reason in exit surveys: "too much admin; didn't save us time on deliveries and invoices".

Calder Group renews in May. Their account manager says they "might" leave without the report builder; they haven't raised it with Customer Success, and their usage is up 20% this year.
feature_requests_last_6_months.csv53 rows · Download
request,requester,requester_arr_usd,source,title,theme
FR-300,Customer 553,4353,Support ticket,Credit note tracking,Invoicing
FR-301,Customer 391,9773,Sales call,Match invoices to deliveries automatically,Invoicing
FR-302,Customer 341,4856,Support ticket,Match invoices to deliveries automatically,Invoicing
FR-303,Calder Group,186000,Account manager,ERP export (CSV variant),Integrations
FR-304,Customer 895,9595,Sales call,Mobile receiving (check deliveries on a phone),Receiving
FR-305,Calder Group,186000,Account manager,Report builder with pivots,Reporting
…
feature_usage.md8 lines · Download
# Usage of related features today (last 30 days, 1,140 accounts)

| Feature | Accounts using it | Note |
|---|---|---|
| Standard reports | 212 (19%) | Calder Group uses them daily |
| Receiving checklist (desktop only) | 640 (56%) | 71% of uses start on a desktop, then get re-typed from paper |
| Invoice upload | 870 (76%) | Matching is manual |
…
capacity_and_estimates.md5 lines · Download
# Capacity, Q1 to Q2

Two squads, about 10 squad-weeks of roadmap work each per quarter (40 in total over the half).

Estimates (squad-weeks): Mobile receiving 8. Flag short deliveries 3 (needs mobile receiving). Invoice matching 9 (needs the new invoice parser, 5). Credit note tracking 3 (needs invoice matching). Accounting export with credits 2 (needs credit note tracking). Custom report builder 12. ERP export 4. Approval chains 6. Site-level permissions 4. SSO 3.
What a strong answer doesThe answer key the graders mark against

Builds the half around the churn goal (independent restaurants, whose top exit reason is admin around deliveries and invoices), not the request count. The log has 53 requests, but 14 come from one customer, Calder Group, and several of theirs are the same thing under different names (report builder three times, ERP export twice, bulk SKU twice, scheduled reports twice); across everyone else, receiving and invoicing requests dominate (mobile receiving 9, invoice matching 8, flag short deliveries 7, credit notes 5, accounting export 4). Sequences the dependency chain: mobile receiving (8), then flag short deliveries (3); the invoice parser (5), then invoice matching (9), then credit note tracking (3), then the accounting export (2): about 30 squad-weeks, inside the 40 available with slack. Handles Calder with evidence: their usage is up 20% and they haven't raised leaving with Customer Success, so talk to them before committing the 12-week report builder; offer a smaller item (SSO, 3, or the ERP export, 4) if it fits. Says what's not being done (report builder, approval chains, dark mode, multi-currency) and why, with near-term items specific and later ones looser.

Critical failures (cap the score at 40)
  • Schedules an item before the item it depends on (for example invoice matching before the parser)
Task

Staff-level · v1.1

The outputs

9 setups answered this brief.

Usable with a quick edit8 of 12 checks passed

The sole factual slip (40 vs 39 requests) and the overconfident interpretation about the account manager can be corrected with a quick wording edit; the core reasoning and roadmap remain intact and do not need re-checking.

Output

Strategic Overview

Our H1 goal is to reduce independent restaurant churn from 2.4% to 1.6% per month. Independents represent 82% of our accounts and 61% of our revenue. Exit surveys point to one glaring issue: "too much admin; didn't save us time on deliveries and invoices."

Usage and request data confirm this friction: * Deliveries: 56% of accounts use the desktop receiving checklist, and 71% of those print it on paper and manually re-type it later. * Invoices: 76% of accounts upload invoices, but 100% of matching to orders/deliveries is manual. * Demand: Out of 40 feature requests from independent restaurants in the last six months, 33 are for Receiving and Invoicing workflow improvements.

We have 40 squad-weeks across two squads in H1. We will dedicate one squad to Invoicing Automation and the other to Mobile Receiving, with a small allocation in Q2 to protect the Calder Group renewal without derailing our core strategy.

---

H1 Roadmap Table

SquadQuarterInitiativeEstimateDependenciesOutcome Served
:---:---:---:---:---:---
Squad A (Invoicing)Q1New Invoice Parser5 wksNoneUnlocks automated line-item extraction to eliminate manual data entry.
Squad A (Invoicing)Q1–Q2Automated Invoice-to-Delivery Matching9 wksInvoice ParserEliminates manual invoice reconciliation (76% of accounts); directly attacks #1 churn driver.
Squad A (Invoicing)Q2Credit Note Tracking3 wksInvoice MatchingAutomates discrepancy resolution when deliveries are short or overcharged.
Squad A (Invoicing)Q2Accounting Export with Credits2 wksCredit Note TrackingCloses the loop with accounting software, removing final manual bookkeeping steps.
Squad A TotalH119 wks(1 wk buffer)
Squad B (Receiving)Q1Mobile Receiving8 wksNoneEliminates paper-to-desktop double entry (affects 71% of receiving users).
Squad B (Receiving)Q1–Q2Flag Short Deliveries at the Door3 wksMobile ReceivingCaptures delivery errors at the loading dock, feeding directly into Credit Note Tracking.
Squad B (Receiving)Q2ERP Export (CSV Variant)4 wksNoneSecures Calder Group renewal ($186k ARR) by getting their raw data into their BI/ERP tools.
Squad B (Receiving)Q2Adoption & UX Polish (Receiving/Invoicing)5 wksMobile ReceivingDrives adoption of new H1 workflows among at-risk independent accounts to ensure churn reduction.
Squad B TotalH120 wks

---

Execution Plan & Sequencing

#### Squad A: End-to-End Invoicing (19 of 20 weeks) Squad A will spend the entire half building a seamless, automated invoicing pipeline. 1. New Invoice Parser (Q1, Weeks 1–5): Required foundation. 2. Automated Invoice Matching (Q1 Week 6 – Q2 Week 4): Our second most-requested feature by independents (8 requests). Automatically reconciles uploaded invoices against delivery records. 3. Credit Note Tracking (Q2, Weeks 5–7): Requested 5 times. Ensures restaurants actually recover money when suppliers shortchange them. 4. Accounting Export with Credits (Q2, Weeks 8–9): Requested 4 times. Pushes clean, reconciled data into their accounting tools. Week 10 is reserved as buffer for edge-case supplier invoice formats.

#### Squad B: Receiving & Strategic Retention (20 of 20 weeks) Squad B will solve physical delivery admin first, then pivot to a targeted retention play. 1. Mobile Receiving (Q1, Weeks 1–8): Our #1 most-requested feature (9 requests). Lets kitchen staff check off deliveries on their phones at the back door, killing the clipboards-and-retyping workflow. 2. Flag Short Deliveries at the Door (Q1 Week 9 – Q2 Week 1): Requested 7 times. Allows staff to log missing items on their phone during check-in. (Bonus: This creates the exact data needed for Squad A’s Automated Invoice Matching and Credit Note Tracking). 3. ERP Export CSV Variant (Q2, Weeks 2–5): Delivered by early May ahead of Calder Group’s renewal. 4. Workflow Polish & Onboarding (Q2, Weeks 6–10): Shipping features doesn't lower churn unless customers use them. Squad B will spend the final 5 weeks removing friction from mobile/invoicing onboarding to ensure we hit the 1.6% churn target.

---

What We Are Not Doing (and Why)

  • Custom Report Builder (12 wks) & Reporting Add-ons: Only 19% of accounts use standard reports today. Spending 30% of our H1 capacity here starves our core receiving/invoicing workflows and guarantees we miss our churn goal.
  • Enterprise Admin (Approval chains - 6 wks, Site permissions - 4 wks, SSO - 3 wks): Zero independent restaurants asked for these. Building them turns us into bespoke enterprise software at the expense of 82% of our user base.
  • Catalogue Additions / Bulk SKU Tools: Catalogue already has 100% usage and isn't driving churn. Our Ops team can handle urgent supplier catalogue additions manually.
  • Dark Mode & Multi-Currency: Low request volume (3 total) and zero impact on admin time savings.

---

What We Tell Calder Group ($186k ARR)

The Reality: Calder’s usage is up 20% YoY, they use standard reports daily, and they haven't complained to CS. The renewal risk is likely an Account Manager hedging their bets. However, $186k is significant, so we shouldn't ignore them entirely.

The Pitch (Delivered jointly by Product, CS, and the AM): 1. Lead with Operational Value: Show them Mobile Receiving and Automated Invoice Matching shipping in Q1/early Q2. Across their multiple sites, eliminating paper receiving and manual invoice reconciliation will save their site managers hundreds of hours a month. 2. Solve the Underlying Reporting Need: Explain that we are not building a custom pivot-table builder inside Fernhill this half because we are focused on core operational automation. Instead, we are building their requested Custom ERP CSV Export in April (before their May renewal). This allows them to pull granular, site-level data directly into their existing ERP/BI tools (or Excel) to build whatever custom pivots and dashboards they need. 3. Bridge the Gap with CS: Have Customer Success set up a weekly manual email of their daily standard reports so they don't have to pull them manually.

Claims the judge couldn’t find in the brief

  • Out of 40 feature requests from independent restaurants in the last six months, 33 are for Receiving and Invoicing workflow improvements

Grades and run details

Decision model 63 · LLM judge 77
Decision model checks
  • failUses the supplied evidence correctly10%
  • partialAddresses the actual decision61%
  • partialRespects explicit constraints18%
  • partialIdentifies material uncertainty69%
  • partialAvoids unsupported claims62%
  • passProduces the required deliverable64%
  • partialSees through the request count40%
  • passHandles the big customer with evidence94%
  • passOutcomes, with certainty that falls with distance64%
  • failFits the stated capacity16%
  • passSequences around dependencies76%
  • passTells the story of the bets38%
Run
Run
#1
API response time
17 s
Submitted
13 Oct 2026

Usable with a quick edit12 of 12 checks passed

The output is evidence-based, respects all constraints, and provides a complete, actionable plan that a PM could use with only minor wording adjustments.

Output

Fernhill H1 Roadmap: Receiving and Invoicing First

The call

Both squads spend most of the half on the two jobs independent restaurants say we don't save them time on: checking deliveries and reconciling invoices. That accounts for 30 of our 40 squad-weeks. Of the remaining 10, 8 go to Calder Group's two most concrete asks (ERP export and site-level permissions) and 2 stay in reserve. We are not building the custom report builder this half.

Why

  • The goal is independent-restaurant churn, and the exit reason is admin on deliveries and invoices. Those two themes make up 33 of the 53 requests in the log, from about 30 accounts across support, sales and in-app feedback:
  • Mobile receiving: 9 requests
  • Flag short deliveries: 7
  • Invoice matching: 8
  • Credit notes: 5
  • Accounting export with credits: 4
  • Usage shows where the admin sits.
  • 56% of accounts use the receiving checklist, but 71% of those checks are re-typed from paper.
  • 76% upload invoices, and every one is matched by hand.
  • Standard reports, the nearest thing to a report builder, are used by 19%.
  • Calder's 14 requests come from one person: their account manager. Calder hasn't raised the report builder with Customer Success. Their usage is up 20% and they use standard reports daily. "Might leave" is a signal worth acting on, but it doesn't justify 12 squad-weeks (30% of the half) and pushing back a churn lever. ERP export is the one Calder ask that appears twice in the log (FR-303, FR-339), and it costs 4 weeks.

Order of work

I'm assuming calendar quarters (Q1 Jan–Mar, Q2 Apr–Jun) and May as the renewal month.

Squad A (Receiving) 1. Mobile receiving (weeks 1–8). This is the biggest single fix for paper re-typing, and flag short deliveries depends on it. 2. ERP export for Calder (weeks 9–12, finishing about mid-April). It goes before short-delivery flags so it lands before Calder's May renewal rather than during it. 3. Flag short deliveries (weeks 13–15). 4. Site-level permissions (weeks 16–19). This is the default. If the Calder working session (below) surfaces something cheaper that matters more to them, such as scheduled report emails, we swap it in. Report scheduling was asked for twice but hasn't been sized yet. 5. 1 week buffer.

Squad B (Invoicing) 1. New invoice parser (weeks 1–5). 2. Invoice matching (weeks 6–14). 3. Credit note tracking (weeks 15–17). 4. Accounting export with credits (weeks 18–19). 5. 1 week buffer.

Squad B's chain is strictly sequential, so any slip pushes everything after it. The 1-week buffer is thin. If the parser runs long, we drop the accounting export to H2 rather than squeeze matching.

How we'll know it's working

Churn moves slowly, and the big items ship mid-half, so we won't get all the way from 2.4% to 1.6% from features alone by June. We'll track leading indicators monthly:

  • Receiving: share of checks started on a phone, and share re-typed from paper. Target: re-typing down from 71% to under 30% by June.
  • Invoicing: share of uploaded invoices auto-matched, and time from invoice upload to approval.
  • Churn: monthly churn among accounts using mobile receiving or matching, compared with those that don't.

What we're not doing, and why

  • Custom report builder, custom dashboards per site (Calder). 12+ weeks for one account. It would push invoice matching into H2. The low usage of standard reports doesn't suggest wider demand. We'll first find out what Calder actually needs.
  • Approval chains with 5 levels (6), SSO (3), custom fields on orders, custom PDF invoices (Calder). Single-account asks with no link to the churn goal. We'll revisit in H2 with Calder's input.
  • Bulk SKU import v2 and bulk edit for SKUs (Calder). Not sized and single-account. We'll size them in H2 if Calder confirms they matter.
  • More suppliers (3 requests). Not sized, and not tied to the churn reason.
  • Dark mode (2) and multi-currency (1). Low demand and off-goal.

A data note: Customers 553 and 294 each appear with two different ARR figures. It doesn't change any decision here, but RevOps should fix it.

What we tell Calder Group

The account manager and Customer Success should go together, in the first two weeks of Q1, well ahead of May. Customer Success needs to be in the room because Calder hasn't raised any of this with them. The message:

> "You've asked for a lot through your account manager, and we want to get the important things right before your renewal. Two commitments this half: ERP export in your format by mid-April, and site-level permissions by end of June. We're not building a full custom report builder in this period, and we'd rather tell you that now than slip it later. You use our standard reports every day, so we'd like a working session with your team to see the reports you run and the questions you can't answer yet. Scheduled report emails or saved views may get you most of the way sooner, and what we learn shapes our H2 reporting plans."

We don't promise a report builder date. If Calder makes it a hard condition of renewal after that session, it comes back to me and leadership as an explicit trade: the 12 weeks would come out of credit notes, the accounting export and short-delivery flags, which would weaken the churn goal.

Roadmap table

#ItemSquadQuarterWeeksDepends onOutcome served
1Mobile receivingAQ18noneChurn: cut paper re-typing (71% today)
2ERP exportAQ1→Q2 (done ~mid-Apr)4noneCalder May renewal
3Flag short deliveriesAQ23Mobile receivingChurn: delivery admin, credits owed
4Site-level permissions (swappable after Calder session)AQ24noneCalder retention
5Invoice parserBQ15noneEnables matching
6Invoice matchingBQ1→Q29ParserChurn: manual matching for 76% of accounts
7Credit note trackingBQ23MatchingChurn: invoice admin
8Accounting export with creditsBQ22Credit notesChurn: month-end admin (first to drop if B slips)
BufferA + BQ22Absorb slip on B's sequential chain
Total40

Grades and run details

Decision model 96 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly43%
  • passAddresses the actual decision95%
  • passRespects explicit constraints30%
  • passIdentifies material uncertainty95%
  • passAvoids unsupported claims31%
  • passProduces the required deliverable77%
  • passSees through the request count67%
  • passHandles the big customer with evidence100%
  • passOutcomes, with certainty that falls with distance79%
  • partialFits the stated capacity18%
  • passSequences around dependencies92%
  • passTells the story of the bets58%
Run
Run
#1
Time to output
53 s
Submitted
5 Oct 2026

Needs more than a quick edit3 of 12 checks passed

The invoice matching estimate is incorrect, so the capacity plan is wrong and would need rework. A PM cannot use this without re-checking the evidence and recalculating.

Output

Fernhill H1 Product Roadmap (Q1–Q2)

Strategic Focus & Alignment

Our overriding company goal for H1 is to cut churn among independent restaurants (82% of accounts, 61% of revenue) from 2.4% to 1.6% a month. Exit surveys reveal the primary culprit: "too much admin; didn't save us time on deliveries and invoices."

Independent operators are drowning in manual paperwork at the back door and during invoice reconciliation. Therefore, both squads will be entirely laser-focused on automating receiving and invoice workflows in Q1 and Q2.

We have 40 total squad-weeks available (2 squads $\times$ 10 weeks per quarter). We are ruthlessly prioritizing features that directly target independent restaurant admin reduction, while deferring enterprise-heavy requests.

---

Squad 1: Receiving & Back-Door Ops

Squad 1 owns the physical receiving workflow, eliminating the friction of transcribing paper notes and catching vendor discrepancies at the door.

  • Q1 (Weeks 1–8): Mobile Receiving (8 squad-weeks)
  • Outcome: Enables staff to check deliveries directly on a phone or mobile device at the loading dock, eliminating the current clunky desktop-to-paper re-typing workflow used by 56% of accounts. Directly targets independent churn by saving hours of back-door admin.
  • Q1–Q2 (Overlap Weeks 9–10 & Q2 Weeks 1–3): Flag Short Deliveries at the Door (3 squad-weeks)
  • Outcome: Built immediately upon Mobile Receiving. Allows staff to instantly log missing or damaged items during the mobile check-in process, generating immediate vendor credit trails and saving reconciliation disputes.

---

Squad 2: Invoicing & Reconciliation

Squad 2 owns the financial workflow post-delivery, automating the tedious matching of paper/digital invoices against actual stock received.

  • Q1 (Weeks 1–9): Automated Invoice Matching (9 squad-weeks, including parser)
  • Outcome: Leverages the new invoice parser (5 weeks) alongside matching logic (4 weeks) to automatically reconcile uploaded invoices against verified deliveries. This attacks the second half of the independent churn bottleneck: manual invoice data entry.
  • Q2 (Weeks 1–4): Credit Note Tracking & Accounting Export (5 squad-weeks total)
  • Credit Note Tracking (3 wks): Tracks vendor credits resulting from short deliveries.
  • Accounting Export with Credits (2 wks): Seamlessly pushes reconciled invoices and credit notes to accounting software, removing end-of-month manual entry.

---

Roadmap Table

SquadQ1 InitiativeSquad-WeeksQ2 InitiativeSquad-WeeksPrimary Outcome
:---:---:---:---:---:---
Squad 1Mobile Receiving8Flag Short Deliveries3Eliminate paper re-typing at the back door; surface short deliveries instantly.
Squad 2Automated Invoice Matching9Credit Notes & Acct. Export5Automate invoice-to-delivery reconciliation and sync clean data to accounting.

---

What We Are Not Doing and Why

To hit our churn-reduction target, we are intentionally declining several high-visibility requests, notably from Calder Group (who accounts for $186k ARR, but whose account manager notes they "might" leave without the report builder).

1. Custom Report Builder (12 squad-weeks) & Calder’s Enterprise Requests (Approval chains, SSO, Site-level permissions, Custom PDF invoices): * Why: Building the Custom Report Builder would consume 30% of our entire company capacity for the half. While Calder Group represents significant ARR, their usage is actually up 20% this year, they have raised no complaints with Customer Success, and their renewal is in May. Diverting 12 weeks to appease an AM warning without CS validation while independent churn burns our revenue base is an unacceptable strategic risk. 2. More Suppliers / Catalogue Updates / Dark Mode / Multi-currency (FR-307, FR-315, FR-351, etc.): * Why: These are nice-to-haves that do not directly address the core operational admin burden cited in independent restaurant exit surveys.

---

What We Tell Calder Group

> "We value our partnership immensely and are thrilled to see your platform usage grow by 20% this year. For H1, our entire product and engineering organization is locked onto a company-wide mandate to eliminate back-office administrative overhead for operators—specifically delivering mobile receiving, short-delivery flagging, and automated invoice matching. > > Because these workflows directly accelerate daily operational efficiency, we aren't scheduling the Custom Report Builder or enterprise admin controls for Q1/Q2. However, your Account Manager and our Customer Success team are logging your specific reporting and export requirements so we can scope them for a future release cycle post-May renewal. We are fully committed to ensuring Fernhill continues to scale with Calder Group's operational needs."

Claims the judge couldn’t find in the brief

  • Automated Invoice Matching takes 9 squad-weeks including the parser (parser 5 weeks, matching logic 4 weeks).

Grades and run details

Decision model 83 · LLM judge 23
Decision model checks
  • passUses the supplied evidence correctly48%
  • partialAddresses the actual decision61%
  • passRespects explicit constraints16%
  • partialIdentifies material uncertainty23%
  • partialAvoids unsupported claims51%
  • passProduces the required deliverable30%
  • partialSees through the request count85%
  • passHandles the big customer with evidence40%
  • passOutcomes, with certainty that falls with distance27%
  • passFits the stated capacity48%
  • passSequences around dependencies27%
  • passTells the story of the bets48%
Run
Run
#1
Time to output
6 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 9

Uses the supplied evidence correctlyWrongRightMixed
Gemini 4 Argon · API

The claim that there were '40 feature requests from independent restaurants' is false; 53 total minus 14 from Calder equals 39, so a statement about the current situation is not supported.

Opus 5.5 · Claude

All factual claims about the current situation are directly supported by the supplied context, with no invented numbers or events.

Gemini 3.5 Flash-Lite · Gemini

The output claims Automated Invoice Matching is 9 squad-weeks including the parser, but the supplied context indicates invoice matching is 9 weeks and the parser is a separate 5-week dependency, making the total 14.

Addresses the actual decisionRightRightWrong
Gemini 4 Argon · API

The output commits unambiguously to a roadmap split across two squads, addresses the churn goal, and specifies what to tell Calder Group.

Opus 5.5 · Claude

The output commits early to a clear plan (receiving and invoicing first, no report builder) and states the condition (Calder hard condition) that would change it.

Gemini 3.5 Flash-Lite · Gemini

The output does not state what result or condition would change the roadmap decision.

Identifies material uncertaintyWrongRightWrong
Gemini 4 Argon · API

The output does not name specific unknowns that could change the plan, nor does it say how they would be resolved or what result would change the call.

Opus 5.5 · Claude

It names specific unknowns (Calder's real needs, report scheduling sizing, parser slip risk) and says how they'd be resolved or what would change the call.

Gemini 3.5 Flash-Lite · Gemini

The output does not name any unknowns that could change the decision or how they would be resolved.

Avoids unsupported claimsWrongRightWrong
Gemini 4 Argon · API

It presents 'The renewal risk is likely an Account Manager hedging their bets' as a confident reality ('The Reality'), but this is an interpretation not established by the supplied evidence.

Opus 5.5 · Claude

Interpretations and forecasts are clearly presented as judgments or plans, not as established facts.

Gemini 3.5 Flash-Lite · Gemini

The output presents the invoice matching estimate (9 weeks including parser) as fact when the evidence does not support it.

Produces the required deliverableRightRightMixed
Gemini 4 Argon · API

The required roadmap table and narrative are present, within the length, and usable by the intended reader.

Opus 5.5 · Claude

The roadmap is complete with a table, narrative, and Calder message, all within the word limit, and a PM could act on it with light edits.

Gemini 3.5 Flash-Lite · Gemini

The capacity error makes the roadmap unusable without re-checking the evidence and redoing the capacity math.

Sees through the request countRightRightWrong
Gemini 4 Argon · API

The output separates Calder's 14 requests, de-duplicates their repeated asks, and weighs demand by number of independent customers asking (33 receiving/invoicing requests).

Opus 5.5 · Claude

It explicitly notes Calder's 14 requests come from one person, de-duplicates ERP export, and weighs demand by distinct customers rather than raw count.

Gemini 3.5 Flash-Lite · Gemini

The output does not count requests by requester or de-duplicate Calder Group's multiple requests; it only notes Calder's requests are enterprise-heavy.

Handles the big customer with evidenceRightRightMixed
Gemini 4 Argon · API

It uses Calder's rising usage and the second-hand 'might leave' to propose a smaller item (ERP CSV export) and a conversation, rather than committing the 12-week report builder by default.

Opus 5.5 · Claude

It uses Calder's rising usage and second-hand 'might leave' to propose a conversation first, offering smaller items (ERP export, site-level permissions) instead of the report builder.

Gemini 3.5 Flash-Lite · Gemini

The output does not offer Calder Group a smaller item (like SSO or ERP export) to address their concerns, only defers everything.

Outcomes, with certainty that falls with distanceRightRightMixed
Gemini 4 Argon · API

Every item names an outcome or problem; near-term items are specific (e.g., mobile receiving outcome) and later items (adoption polish) are deliberately looser.

Opus 5.5 · Claude

Every item names its outcome, near-term items are specific, and later items (site-level permissions, accounting export) are explicitly swappable or droppable.

Gemini 3.5 Flash-Lite · Gemini

Q2 items are given specific week ranges (e.g., 'Q2 Weeks 1-4') rather than deliberately looser precision for later items.

Fits the stated capacityMixedRightMixed
Gemini 4 Argon · API

Committed work adds up to 39 squad-weeks with 1 week slack, sums are checkable, and the output names what was cut (report builder, enterprise admin, etc.) and why.

Opus 5.5 · Claude

The committed work sums to exactly 40 squad-weeks with 2 weeks of buffer, the arithmetic is checkable, and it names what was cut (report builder, approval chains, etc.) and why.

Gemini 3.5 Flash-Lite · Gemini

The committed plan sums to 25 squad-weeks based on an incorrect estimate; the actual required work (parser 5 + matching 9 + credit notes 3 + export 2 + mobile 8 + flag 3) totals 30, and the output's capacity claim is therefore invalid.

All got right 3

Respects explicit constraintsRightRightRight
Gemini 4 Argon · API

The output stays within 900 words plus table, includes the required sections (what each squad builds, order, outcomes, what we're not doing, what we tell Calder), and frames the answer for the Group PM.

Opus 5.5 · Claude

The output is under 900 words plus a table, includes the required sections (what each squad builds, order, outcomes, what's not done, what to tell Calder), and respects the reader and form.

Gemini 3.5 Flash-Lite · Gemini

The output is under 900 words, includes a table, and covers what each squad builds, order, outcomes, what's not being done, and what to tell Calder Group.

Sequences around dependenciesRightRightRight
Gemini 4 Argon · API

The roadmap sequences invoice parser before matching, matching before credit notes, credit notes before accounting export, and mobile receiving before flag short deliveries, respecting all stated dependencies.

Opus 5.5 · Claude

All dependencies from the brief are respected (mobile receiving before flag short deliveries, parser before matching, matching before credit notes, credit notes before accounting export) and named.

Gemini 3.5 Flash-Lite · Gemini

The roadmap places flag short deliveries after mobile receiving, and the invoice matching item includes the parser, implying the parser is done first.

Tells the story of the betsRightRightRight
Gemini 4 Argon · API

The work is grouped into two themes (Invoicing Automation and Mobile Receiving) tied directly to the churn reduction goal, and the narrative explains why those bets matter most now.

Opus 5.5 · Claude

The work is grouped into two clear themes (receiving and invoicing) tied directly to the churn goal, not presented as a scored list.

Gemini 3.5 Flash-Lite · Gemini

The work is grouped into two themes (Receiving & Back-Door Ops, Invoicing & Reconciliation) with clear reasons tied to the churn goal.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 79% of 150 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT88.0100.02None
2Opus 5.5withClaude88.396.42None
3GPT-6.1 SolwithAPI88.396.42None
4Sonnet 5.5withAPI90.292.62None
5Haiku 5.5withAPI74.496.42None
6GPT-6 LunawithAPI84.085.42None
7Gemini 4 ArgonwithAPI62.088.52None
8Gemini 3.8 FlashwithAPI58.366.82None
9Gemini 3.5 Flash-LitewithGemini57.136.521 capped

What you get for the cost

GPT-6.1 Sol · API scores within a point of GPT-6 Astra · ChatGPT at a sixth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 94, 39× the cheapestOpus 5.5 · Claude: task score 93, 37× the cheapestGPT-6.1 Sol · API: task score 93, 6.9× the cheapestSonnet 5.5 · API: task score 92, 36× the cheapestHaiku 5.5 · API: task score 86, 2.1× the cheapestGPT-6 Luna · API: task score 84, the cheapestGemini 4 Argon · API: task score 77, 6.7× the cheapestGemini 3.8 Flash · API: task score 60, 5.5× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 45, the cheapest, 1 critical failureGPT-6 Astra · ChatGPTOpus 5.5 · ClaudeGPT-6.1 Sol · APISonnet 5.5 · APIHaiku 5.5 · APIGPT-6 Luna · APIGemini 4 Argon · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costAt least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API841.0×
Gemini 3.5 Flash-Lite · Gemini451.1×
Haiku 5.5 · API862.1×
Gemini 3.8 Flash · API605.5×
Gemini 4 Argon · API776.7×
GPT-6.1 Sol · API936.9×
Sonnet 5.5 · API9236×
Opus 5.5 · Claude9337×
GPT-6 Astra · ChatGPT9439×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Turning strategy into a sequenced plan.

Why it matters

A roadmap is where strategy meets capacity. Dated feature lists turn guesses into promises.

What good looks like

  • Items are problems or outcomes, not just features
  • Sequencing reflects dependencies
  • Explicit trade-offs
  • Commitment falls with distance

Deliberately not measured

  • Gantt formatting
Capability tested

Sequencing under constraints

The failure we’re looking for

A dated wishlist sorted by excitement

Grading

Decision model and LLM judge, calibrated against a blind PM review