Tasks / Product Craft

Build a roadmap

Can the model sequence bets against capacity and dependencies, and explain the order?

Task type v1.0 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 69% were usable with at most a quick edit.

Reliably right

Produces the required deliverable95% pass
The roadmap is complete, in the right form for a Group PM, within the word limit, and actionable without major gaps.
GPT-6.1 Sol · API · Fourteen requests from one customer
Outcomes, with certainty that falls with distance94% pass
Every item names its outcome (e.g., reduce paper transcription, reduce manual reconciliation), and near-term items are specific while later ones are deliberately looser (e.g., '7 weeks for rollout, workflow improvement and contingency').
GPT-6.1 Sol · API · Fourteen requests from one customer
Handles the big customer with evidence94% pass
It uses Calder's rising usage and the second-hand 'might leave' to propose a conversation first, not committing the report builder, and offers a bounded export if needed.
GPT-6.1 Sol · API · Fourteen requests from one customer

Where it slips

Sees through the request count72% pass
The output never counts requests by requester, does not mention that 14 requests come from Calder or that several are duplicates, and does not compare themes by customer count.
GPT-6 Luna · API · Fourteen requests from one customer
Uses the supplied evidence correctly72% pass
States credit notes as 4 customers, but the request log contains 5 credit note tracking requests.
Sonnet 5.5 · API · Fourteen requests from one customer
Fits the stated capacity73% pass
Q3 Expenses row lacks squad-week allocation, so the committed plan cannot be fully checked against the 11-week capacity.
Opus 5.5 · Claude · A year of spend management, with a hard deadline

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Outcomes, with certainty that falls with distance
Does each roadmap item say the problem or outcome it serves, and does precision (dates, scope) decrease for later items?
Passes when Every item names its outcome or problem. Near-term items are specific and later ones are deliberately looser.
Melissa PerriLenny’s Podcast Christian IdiodiLenny’s Podcast
Fits the stated capacity
Does the committed plan fit the team capacity the brief gives, with sums a reader could check, and does it say what was cut or deferred to make it fit?
Passes when The committed work adds up to no more than the stated capacity, with some slack, the sums can be checked, and it names what was cut or pushed later and why.
Gokul RajaramLenny’s Podcast
Tells the story of the bets
Does the roadmap group the work into a few themes and explain why those are the biggest things to invest in now, rather than presenting a scored or ranked list of projects?
Passes when A handful of themes, each with the reason it matters most now, and the items sit under them.
Jiaona ZhangLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Sequences around dependencies · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the Group PM at Fernhill. Build the roadmap for the next half (Q1 and Q2) for our two squads, in no more than 900 words plus a table: what each squad builds, in what order, the outcome each item serves, what we're not doing and why, and what we tell Calder Group. The request log, usage, capacity and goal are attached.

What the model was given5 items: About Fernhill, company_goal_h1.md, feature_requests_last_6_months.csv, feature_usage.md, capacity_and_estimates.md

About FernhillPurchasing and inventory software for restaurants and hotels: orders, deliveries, invoices and stock.

company_goal_h1.md5 lines · Download

# Company goal for the half

Cut churn among independent restaurants (82% of accounts, 61% of revenue) from 2.4% to 1.6% a month. Top churn reason in exit surveys: "too much admin; didn't save us time on deliveries and invoices".

Calder Group renews in May. Their account manager says they "might" leave without the report builder; they haven't raised it with Customer Success, and their usage is up 20% this year.

feature_requests_last_6_months.csv53 rows · Download

request,requester,requester_arr_usd,source,title,theme
FR-300,Customer 553,4353,Support ticket,Credit note tracking,Invoicing
FR-301,Customer 391,9773,Sales call,Match invoices to deliveries automatically,Invoicing
FR-302,Customer 341,4856,Support ticket,Match invoices to deliveries automatically,Invoicing
FR-303,Calder Group,186000,Account manager,ERP export (CSV variant),Integrations
FR-304,Customer 895,9595,Sales call,Mobile receiving (check deliveries on a phone),Receiving
FR-305,Calder Group,186000,Account manager,Report builder with pivots,Reporting
…

feature_usage.md8 lines · Download

# Usage of related features today (last 30 days, 1,140 accounts)

| Feature | Accounts using it | Note |
|---|---|---|
| Standard reports | 212 (19%) | Calder Group uses them daily |
| Receiving checklist (desktop only) | 640 (56%) | 71% of uses start on a desktop, then get re-typed from paper |
| Invoice upload | 870 (76%) | Matching is manual |
…

capacity_and_estimates.md5 lines · Download

# Capacity, Q1 to Q2

Two squads, about 10 squad-weeks of roadmap work each per quarter (40 in total over the half).

Estimates (squad-weeks): Mobile receiving 8. Flag short deliveries 3 (needs mobile receiving). Invoice matching 9 (needs the new invoice parser, 5). Credit note tracking 3 (needs invoice matching). Accounting export with credits 2 (needs credit note tracking). Custom report builder 12. ERP export 4. Approval chains 6. Site-level permissions 4. SSO 3.

What a strong answer doesThe answer key the graders mark against

Builds the half around the churn goal (independent restaurants, whose top exit reason is admin around deliveries and invoices), not the request count. The log has 53 requests, but 14 come from one customer, Calder Group, and several of theirs are the same thing under different names (report builder three times, ERP export twice, bulk SKU twice, scheduled reports twice); across everyone else, receiving and invoicing requests dominate (mobile receiving 9, invoice matching 8, flag short deliveries 7, credit notes 5, accounting export 4). Sequences the dependency chain: mobile receiving (8), then flag short deliveries (3); the invoice parser (5), then invoice matching (9), then credit note tracking (3), then the accounting export (2): about 30 squad-weeks, inside the 40 available with slack. Handles Calder with evidence: their usage is up 20% and they haven't raised leaving with Customer Success, so talk to them before committing the 12-week report builder; offer a smaller item (SSO, 3, or the ERP export, 4) if it fits. Says what's not being done (report builder, approval chains, dark mode, multi-currency) and why, with near-term items specific and later ones looser.

Critical failures (cap the score at 40)

Schedules an item before the item it depends on (for example invoice matching before the parser)

Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

GPT-6.1 SolAPI100Usable with a quick edit Opus 5.5Claude98Usable with a quick edit GPT-6 AstraChatGPT96Usable with a quick edit Sonnet 5.5API94Usable with a quick edit GPT-6 LunaAPI92Usable with a quick edit Haiku 5.5API90Usable with a quick edit Gemini 3.8 FlashAPI71Needs more work Gemini 3.5 Flash-LiteGemini48Needs more work

GPT-6 LunaAPI

Usable with a quick edit11 of 12 checks passed

The roadmap is complete, evidence-based, and directly usable; a PM could adopt it with minor wording tweaks and no re-checking of facts or logic.

Output

H1 roadmap

Priority: reduce admin that contributes to independent-restaurant churn. We’ll automate the invoice-to-credit workflow and make receiving work where deliveries happen. That addresses the stated churn reason and repeated requests across customers. Calder’s renewal matters, but the report-builder risk is unconfirmed, so we’ll validate it before committing 12 squad-weeks to a bespoke capability.

Order	Squad / timing	What we build	Outcome served
1	Squad A · Q1 (8 weeks)	Mobile receiving	Lets staff check deliveries on a phone instead of re-typing paper checklists. Desktop-to-paper re-entry is a clear source of admin.
2	Squad B · Q1 (5 weeks)	Invoice parser	Prerequisite for matching; prepares invoice data for automation.
3	Squad B · Q1–Q2 (9 weeks total: 5 in Q1, 4 in Q2)	Match invoices to deliveries	Reduces manual reconciliation, directly addressing “too much admin”; invoice upload is already used by 76% of accounts. Starts after the parser.
4	Squad A · Q2 (3 weeks)	Flag short deliveries	Surfaces discrepancies at receiving, before they become invoice errors. Starts after mobile receiving.
5	Squad B · Q2 (3 weeks)	Credit note tracking	Extends the automated invoice workflow to credits; starts after matching.
6	Squad B · Q2 (2 weeks)	Accounting export with credits	Makes the completed invoice-and-credit workflow easier to carry into accounting; starts after credit tracking.
—	Squad A · Q1 (2 weeks); Squad A · Q2 (7 weeks)	Field validation, rollout, adoption and measurement for the above	Validate the receiving workflow with restaurants, help teams adopt it, and measure whether it reduces re-entry and invoice admin. Include direct Calder discovery in Q1.
—	Squad B · Q2 (1 week)	Hardening and release buffer	Protects quality across the dependent invoice workflow before broad rollout.

This uses the 40 available squad-weeks: 20 per quarter. The invoice sequence is parser → matching → credit tracking → accounting export; receiving is mobile receiving → short-delivery flags.

Not doing this half

Custom report builder, dashboards and scheduled reports: 12 weeks, with the reported renewal risk coming from the account manager rather than Calder’s CS team; Calder’s usage is up 20% and it uses standard reports daily. We’ll validate the specific reporting job and whether it is truly a renewal blocker before reallocating capacity. A bespoke builder is not currently the best fit for the independent-restaurant churn goal.
ERP export, SSO, site permissions, approval chains, custom fields, catalogue work and other Calder requests: several are substantial or bespoke, and none has stronger evidence against the stated churn driver than the selected workflow improvements. We won’t imply these are committed.
More suppliers and dark mode: lower priority than reducing delivery and invoice admin.

What we tell Calder Group

“We’ve heard that reporting may affect your May renewal. We’re not committing to a custom report builder yet: your teams use standard reports daily, usage is growing, and we haven’t confirmed with you which reporting tasks are blocked. We’d like to meet with your reporting users and Customer Success early in Q1 to understand the must-have use cases and renewal criteria. We’ll come back with a clear proposal; meanwhile, we’ll be investing in reducing delivery and invoice admin across Fernhill.”

If that conversation confirms custom reporting is a hard renewal condition, we’ll bring the trade-off back for an explicit roadmap decision rather than promise a feature we haven’t scoped or prioritized.

Grades and run details

Decision model 92 · LLM judge 92

Decision model checks

passUses the supplied evidence correctly47%
passAddresses the actual decision95%
passRespects explicit constraints25%
passIdentifies material uncertainty98%
partialAvoids unsupported claims38%
passProduces the required deliverable58%
partialSees through the request count64%
passHandles the big customer with evidence97%
passOutcomes, with certainty that falls with distance67%
passFits the stated capacity15%
passSequences around dependencies88%
passTells the story of the bets87%

Run

Run: #1
API response time: 58 s
Submitted: 6 Oct 2026

Gemini 3.8 FlashAPI

Needs more than a quick edit8 of 12 checks passed

The output fails to de-duplicate Calder's requests and leaves no capacity slack, so a PM would need to re-check the request log and adjust the plan before using it.

Output

Strategic Context & H1 Focus

Fernhill’s H1 north star is cutting independent restaurant monthly churn from 2.4% to 1.6%. Independent restaurants represent 82% of our accounts and 61% of revenue. Exit interviews clearly highlight the churn driver: administrative burden on deliveries and invoices.

Currently: * 56% of accounts use desktop receiving, but 71% write deliveries on paper at the door and re-type them later. * 76% of accounts upload invoices, but matching is entirely manual.

To reach our churn target, H1 execution is dedicated to eliminating manual reconciliation at the back door and in the back office. Calder Group ($186k ARR) is critical, but dedicating 12 squad-weeks to a custom report builder would derail the core churn goal and still risk missing their May renewal date. We address Calder with a targeted integration release and proactive commercial management.

---

H1 Roadmap

Squad	Quarter	Initiative	Estimate	Dependency	Primary Outcome Served
Squad 1: Receiving & Data	Q1	Mobile Receiving	8 wks	None	Eliminates paper receiving; cuts ~30–45 mins of daily double-entry per site.
Squad 1: Receiving & Data	Q1–Q2	Flag Short Deliveries at the Door	3 wks	Mobile Receiving	Real-time shortage capture; generates dispute records at check-in.
Squad 1: Receiving & Data	Q2	Automated ERP Export	4 wks	None	De-risks Calder renewal; automates multi-site enterprise reporting pipelines.
Squad 1: Receiving & Data	Q2	Buffer, Rollout & Mobile QA	5 wks	—	Hardware testing, onboarding stability, restaurant floor adoption.
Squad 2: Invoicing & Finance	Q1	New Invoice Parser	5 wks	None	Foundation for OCR extraction; pre-requisite for matching engine.
Squad 2: Invoicing & Finance	Q1–Q2	Automated Invoice Matching	9 wks	New Invoice Parser	3-way match (PO to delivery to invoice); solves top independent request.
Squad 2: Invoicing & Finance	Q2	Credit Note Tracking	3 wks	Invoice Matching	Closes loop on short deliveries, automating balance deductions.
Squad 2: Invoicing & Finance	Q2	Accounting Export with Credits	2 wks	Credit Note Tracking	One-click sync to Xero/QuickBooks with verified vendor credits.
Squad 2: Invoicing & Finance	Q2	Buffer & Accounting Integration QA	1 wk	—	Edge-case tax/rounding rules and accounting schema validation.

---

Squad Breakdown & Sequencing

#### Squad 1 (Receiving & Data) — 20 Squad-Weeks 1. Mobile Receiving (8 wks, Q1): Build a mobile-optimized receiving flow allowing kitchen staff to check off line items directly on their phone at the loading dock. Eliminates paper double-entry (71% of receiving workflows today). 2. Flag Short Deliveries at the Door (3 wks, Q1–Q2): Extends mobile receiving to instantly flag missing quantities, take a photo, and generate a shortage log directly linked to the order. 3. Automated ERP Export (4 wks, Q2): Scheduled and manual CSV/ERP data pipeline (FR-303, FR-339). Delivers automated raw financial/operational extracts before Calder’s May renewal. 4. Buffer & Mobile Rollout (5 wks, Q2): Ensures smooth adoption in noisy kitchen environments with flaky Wi-Fi.

#### Squad 2 (Invoicing & Finance) — 20 Squad-Weeks 1. New Invoice Parser (5 wks, Q1): Modernizes parsing infrastructure to extract line items, taxes, unit costs, and invoice numbers reliably. 2. Automated Invoice Matching (9 wks, Q1–Q2): Automatically matches invoice line items against purchase orders and received delivery logs. Discrepancies are flagged automatically. 3. Credit Note Tracking (3 wks, Q2): Pairs short-delivery logs generated in Squad 1 with incoming vendor credit notes, stopping suppliers from overcharging for missing goods. 4. Accounting Export with Credits (2 wks, Q2): Exports matched invoices and net credit adjustments directly into standard general ledgers. 5. Buffer & Edge Cases (1 wk, Q2): Absorbs parser edge cases and vendor invoice layout variations.

---

What We Are NOT Doing (and Why)

1. Custom Report Builder with Pivots (12 weeks): * Why: At 12 weeks, this would consume 30% of total H1 capacity for a single account. It cannot ship and stabilize before Calder's May renewal, and standard report daily usage is already healthy (212 accounts, Calder included). 2. 5-Level Approval Chains & Site-Level Permissions (10 weeks combined): * Why: Pure enterprise overhead. Does not solve the operational time sink driving 82% of accounts to churn. 3. Dark Mode & Multi-Currency (Non-core): * Why: Zero correlation to churn reduction or operational savings. 4. Catalogue Bulk Tools & Expansion: * Why: Catalogue usage is 100% saturated. The friction point is not creating items; it is receiving and paying for them.

---

What We Tell Calder Group

#### 1. Commercial Reality Check (Internal) Calder’s usage is up 20% year-over-year, and they use Standard Reports daily. Their threat to leave originated from an Account Manager conversation, not Customer Success or executive escalation. We will not over-index engineering capacity on an AM negotiation bluff.

#### 2. External Messaging (Fernhill VP of Product & Head of CS to Calder Exec Sponsor in March) * Lead with High-Value Clean Data: “A reporting tool is only as good as the input data. By shipping Mobile Receiving and Automated Matching in Q1, Fernhill ensures every invoice, delivery quantity, and supplier credit across all Calder sites is 100% accurate and audited in real time.” * Deliver the ERP Export (Ships April, ahead of May renewal): “Rather than forcing your finance team into a proprietary, rigid reporting UI inside Fernhill, we are releasing the Automated ERP/CSV Data Export in April. This allows Calder to stream clean, site-level operational and invoice data directly into your central ERP/BI stack (PowerBI/Tableau/Excel) with your custom formulas and pivot models.” * White-Glove Support: CS will construct scheduled, bespoke standard reporting extracts tailored to their multi-site leadership team during renewal discussions.

Grades and run details

Decision model 79 · LLM judge 69

Decision model checks

failUses the supplied evidence correctly28%
passAddresses the actual decision53%
passRespects explicit constraints20%
partialIdentifies material uncertainty25%
partialAvoids unsupported claims32%
passProduces the required deliverable75%
partialSees through the request count80%
passHandles the big customer with evidence99%
passOutcomes, with certainty that falls with distance65%
passFits the stated capacity64%
passSequences around dependencies92%
passTells the story of the bets99%

Run

Run: #1
API response time: 31 s
Submitted: 6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 3

Uses the supplied evidence correctlyRightMixed

GPT-6 Luna · API

All factual claims about the current situation are directly supported by the supplied context.

Gemini 3.8 Flash · API

All factual claims about the current situation are directly supported by the supplied context or simple arithmetic from it.

Identifies material uncertaintyRightWrong

GPT-6 Luna · API

Identifies the unconfirmed report-builder risk and states that a conversation with Calder will resolve it and could change the plan.

Gemini 3.8 Flash · API

It does not name the specific unknowns that could change the Calder decision (e.g., executive escalation) or say how they would be resolved; it only states the decision not to over-index.

Fits the stated capacityRightMixed

GPT-6 Luna · API

The committed work sums to 40 squad-weeks (30 for features + 9 for validation/rollout + 1 buffer), and the output names what was cut and why.

Gemini 3.8 Flash · API

The committed work sums to exactly 40 squad-weeks with no slack; the buffers are allocated as work items, not as unallocated capacity for uncertainty.

All got wrong 1

Sees through the request countWrongWrong

GPT-6 Luna · API

The output never counts requests by requester, does not mention that 14 requests come from Calder or that several are duplicates, and does not compare themes by customer count.

Gemini 3.8 Flash · API

The output never mentions that 14 requests come from Calder, several are duplicates, or weighs demand by unique customers; it relies on the churn goal instead of explicitly de-duplicating the request log.

All got right 8

Addresses the actual decisionRightRight

GPT-6 Luna · API

Commits to a clear roadmap (receiving and invoicing automation) and says the Calder conversation would change the call if reporting is a hard renewal condition.

Gemini 3.8 Flash · API

The output commits to a clear H1 roadmap with squad assignments, sequencing, and a specific plan for Calder, framed for the Group PM.

Respects explicit constraintsRightRight

GPT-6 Luna · API

The output is under 900 words plus a table, includes the required sections, and addresses the Group PM reader.

Gemini 3.8 Flash · API

The output includes the required table, stays well under 900 words, and covers what each squad builds, order, outcomes, what's not done and why, and the Calder message.

Avoids unsupported claimsRightRight

GPT-6 Luna · API

Interpretations like 'desktop-to-paper re-entry is a clear source of admin' are grounded in the usage data; no unsupported confident claims are made.

Gemini 3.8 Flash · API

Forecasts and interpretations are presented as roadmap outcomes or goals, not as established facts, and the few causal claims are clearly tied to the data.

Produces the required deliverableRightRight

GPT-6 Luna · API

Provides a complete roadmap table with order, squad, timing, outcomes, a not-doing list with reasons, and a Calder message; a PM could act on it with light edits.

Gemini 3.8 Flash · API

The roadmap is complete, in the requested format, within the word limit, and a PM could act on it with light edits.

Handles the big customer with evidenceRightRight

GPT-6 Luna · API

Uses Calder's rising usage and the second-hand 'might leave' to propose a discovery conversation instead of committing the report builder; no small item fits the full capacity, so omission is acceptable.

Gemini 3.8 Flash · API

It uses Calder's rising usage and the second-hand 'might leave' to avoid committing the report builder, instead offering a smaller ERP export and proactive commercial conversation.

Outcomes, with certainty that falls with distanceRightRight

GPT-6 Luna · API

Every item names its outcome; near-term items are specific and later items (rollout, hardening) are deliberately looser.

Gemini 3.8 Flash · API

Every roadmap item names its outcome or problem, and near-term items are specific while later items (buffers, edge cases) are deliberately looser.

Sequences around dependenciesRightRight

GPT-6 Luna · API

All dependencies are respected (parser before matching, matching before credit notes, credit notes before accounting export, mobile receiving before flag short deliveries) and the chains are named.

Gemini 3.8 Flash · API

All dependencies from the brief are respected: mobile receiving before flag short deliveries, parser before matching, matching before credit notes, credit notes before accounting export.

Tells the story of the betsRightRight

GPT-6 Luna · API

Groups work into two themes (invoice-to-credit automation and mobile receiving) tied directly to the churn reason.

Gemini 3.8 Flash · API

The roadmap groups work into two themes (Receiving & Data, Invoicing & Finance) and explains why eliminating manual reconciliation is the biggest investment to reduce churn.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 79% of 150 checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	GPT-6.1 SolwithAPI	100.0	96.4	2	None
2	GPT-6 AstrawithChatGPT	93.9	100.0	2	None
3	Opus 5.5withClaude	90.2	96.4	2	None
4	Sonnet 5.5withAPI	94.1	92.6	2	None
5	GPT-6 LunawithAPI	92.0	85.4	2	None
6	Haiku 5.5withAPI	74.2	96.4	2	None
7	Gemini 3.8 FlashwithAPI	74.2	66.8	2	None
8	Gemini 3.5 Flash-LitewithGemini	56.9	36.5	2	1 capped

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Nobody beats these on both score and costAt least one critical failure

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
GPT-6 Luna · API	87	1.0×
Gemini 3.5 Flash-Lite · Gemini	44	1.1×
Haiku 5.5 · API	86	2.1×
Gemini 3.8 Flash · API	68	5.5×
GPT-6.1 Sol · API	98	6.9×
Sonnet 5.5 · API	94	36×
Opus 5.5 · Claude	94	37×
GPT-6 Astra · ChatGPT	97	39×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Turning strategy into a sequenced plan.

Why it matters

A roadmap is where strategy meets capacity. Dated feature lists turn guesses into promises.

What good looks like

Items are problems or outcomes, not just features
Sequencing reflects dependencies
Explicit trade-offs
Commitment falls with distance

Deliberately not measured

Gantt formatting

Capability tested

Sequencing under constraints

The failure we’re looking for

A dated wishlist sorted by excitement

Grading

Decision model and LLM judge, calibrated against a blind PM review

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

Outcomes, with certainty that falls with distance

Fits the stated capacity

Tells the story of the bets

Plus our standard checks

The tasks

The outputs

Output

H1 roadmap

Not doing this half

What we tell Calder Group

Grades and run details

Decision model checks

Run

Output

Strategic Context & H1 Focus

H1 Roadmap

Squad Breakdown & Sequencing

What We Are NOT Doing (and Why)

What We Tell Calder Group

Grades and run details

Decision model checks

Run

Check by check

Where they differ 3

All got wrong 1

All got right 8

Results

What you get for the cost

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured