Tasks / Product Craft

Build a roadmap

Can the model sequence bets against capacity and dependencies, and explain the order?

Task type v1.0 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 72% were usable with at most a quick edit.

Reliably right

  1. Produces the required deliverable96% pass
    The roadmap is complete, in the right form for a Group PM, within the word limit, and actionable without major gaps.
    GPT-6.1 Sol · API · Fourteen requests from one customer
  2. Outcomes, with certainty that falls with distance96% pass
    Every item names its outcome (e.g., reduce paper transcription, reduce manual reconciliation), and near-term items are specific while later ones are deliberately looser (e.g., '7 weeks for rollout, workflow improvement and contingency').
    GPT-6.1 Sol · API · Fourteen requests from one customer
  3. Handles the big customer with evidence94% pass
    It uses Calder's rising usage and the second-hand 'might leave' to propose a conversation first, not committing the report builder, and offers a bounded export if needed.
    GPT-6.1 Sol · API · Fourteen requests from one customer

Where it slips

  1. Fits the stated capacity53% pass
    Q3 Expenses row lacks squad-week allocation, so the committed plan cannot be fully checked against the 11-week capacity.
    Opus 5.5 · Claude · A year of spend management, with a hard deadline
  2. Tells the story of the bets54% pass
    The roadmap table is organized by squad, not grouped into a few themes, though the case provides thematic reasoning.
    Sonnet 5.5 · API · A year of spend management, with a hard deadline
  3. Sees through the request count72% pass
    The output never counts requests by requester, does not mention that 14 requests come from Calder or that several are duplicates, and does not compare themes by customer count.
    GPT-6 Luna · API · Fourteen requests from one customer

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Outcomes, with certainty that falls with distance

    Does each roadmap item say the problem or outcome it serves, and does precision (dates, scope) decrease for later items?

    Passes when Every item names its outcome or problem. Near-term items are specific and later ones are deliberately looser.

  2. Fits the stated capacity

    Does the committed plan fit the team capacity the brief gives, with sums a reader could check, and does it say what was cut or deferred to make it fit?

    Passes when The committed work adds up to no more than the stated capacity, with some slack, the sums can be checked, and it names what was cut or pushed later and why.

  3. Tells the story of the bets

    Does the roadmap group the work into a few themes and explain why those are the biggest things to invest in now, rather than presenting a scored or ranked list of projects?

    Passes when A handful of themes, each with the reason it matters most now, and the items sit under them.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Sequences around dependencies · and 3 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a Staff PM at Harbor. Tom Achebe, our CEO, has shared his draft roadmap for the next four quarters (Q4 2026 to Q3 2027), and asked you to turn it into the roadmap we take to next week's planning offsite. The exec team (CEO, CFO, CRO and CTO) will read it beforehand. Write: 1. The roadmap itself, by quarter or by now, next and later: what each squad works on and the outcome each item serves. 2. The case for it, in no more than 1,200 words: why this order, what you changed from Tom's draft and why, what we're not doing, and the risks that could change the plan. The pack below is everything we have. Not all of it matters equally.

What the model was given12 items: About Harbor, FY27 goal (approved by the board, September 2026), Squads and capacity, Hiring, Card processor deadline, Candidate work (estimates in squad-weeks, by owning squad), Tom's draft roadmap, Procurement evidence, Card spend evidence, Multi-entity evidence, Integrations evidence, Other asks
About HarborSpend management (corporate cards, expenses and approvals) for companies with 200 to 2,000 employees. 1,100 customers, $86M ARR, average 700 employees per customer. 58% of revenue is interchange on card spend; the rest is subscription. Net revenue retention (NRR) is 104%.
FY27 goal (approved by the board, September 2026)Raise NRR to 112% by the end of Q3 2027 by expanding inside existing customers: more spend on Harbor cards, more entities and more modules. New-logo growth matters, but it comes second this year.
Squads and capacityFive squads: Cards, Expenses, Approvals & Policy, Integrations and Platform. After support and on-call, each has about 11 squad-weeks of roadmap capacity a quarter. Squads can take work from another squad's area, but it takes them about 1.5× as long.
HiringTwo new squads were approved in September 2026. Tom's draft counts the first from Q1 2027 and the second from Q2 2027, both at full capacity. Last year, our three approved squads took 5, 6 and 8 months from approval to their first full sprint, and each delivered about half its capacity in its first quarter.
Card processor deadlineOur card processor retires its v1 API on 31 March 2027. Every card authorisation we process runs through v1 today. Migrating to v2 is about 18 squad-weeks of Cards work, and the processor must then run 4 weeks of certification testing before we can cut over. Certification is the processor's time, not ours, but nothing can change on our side while it runs.
Candidate work (estimates in squad-weeks, by owning squad)1. Processor v2 migration: Cards 18. Hard deadline above. 2. Virtual cards for software subscriptions: Cards 10. Needs v2. 3. Real-time spend controls (declines out-of-policy spend at the point of sale): Cards 8, Approvals & Policy 4. Needs v2. 4. Procurement module (purchase requests and approvals, a new paid add-on): Approvals & Policy 30, Integrations 6. 5. Multi-entity support (subsidiaries under one parent account): Platform 12, Approvals & Policy 10. 6. Sage Intacct integration: Integrations 9. 7. Microsoft Dynamics integration: Integrations 12. 8. SSO and SCIM provisioning: Platform 5. 9. Receipt-matching improvements: Expenses 6. 10. Mobile app rewrite: Expenses 22.
Tom's draft roadmapQ4 2026: Cards: v2 migration. Approvals & Policy: multi-entity. Integrations: Intacct. Platform: multi-entity. Expenses: receipt matching. Q1 2027: Cards: finish migration, start virtual cards. New squad A: procurement module. Integrations: Dynamics. Platform: SSO and SCIM. Q2 2027: Cards: virtual cards, spend controls. New squad A: procurement launch. New squad B: mobile rewrite. Q3 2027: Cards: spend controls. Everyone else: procurement adoption, mobile launch. Tom's note: “Procurement is our path to 112%. I've told the board it can add $6M of ARR in its first year.”
Procurement evidenceSix design partners have used a prototype since June. Two said they would pay for it; the other four already use a dedicated procurement tool and said they'd need a reason to switch. Proposed price: $8 per user per month, for users who raise purchase requests. At our customers, about 10% of employees raise purchase requests.
Card spend evidenceIn QBRs with our 60 largest customers, 26 asked for virtual cards for software subscriptions, and 31 CFOs asked for real-time spend controls. Finance estimates our customers pay about $410M a year of software subscriptions on other cards or by invoice (extrapolated from the 60 QBR accounts). Our net interchange is 1.1% of card spend.
Multi-entity evidence14 of our 60 largest customers have asked for it. Between them they have 52 subsidiaries not on Harbor, and 9 of the 14 have said in writing they would add their subsidiaries once it exists. Customer Success sized it at $2.9M ARR if all 52 joined at their parents' pricing.
Integrations evidence17% of customers use Sage Intacct through a CSV export. Their gross revenue churn is 13% a year, against 7% for customers on our NetSuite integration. Intacct was cited in 31% of lost new-logo deals last year, Dynamics in 8%. 3% of customers use Dynamics.
Other asksCRO: “Five enterprise deals worth $1.2M of pipeline need SSO and SCIM, and 11 existing accounts have it on their security review list.” CTO: “The mobile app is on a framework version that loses support in late 2027, and I want the rewrite done before then.” Support: receipt auto-match is at 71%, and unmatched receipts are the most common expense ticket.
What a strong answer doesThe answer key the graders mark against

A four-quarter roadmap that protects the 31 March deadline: the migration build has to finish by about early March to leave 4 weeks of certification, which is roughly all of Cards' capacity for Q4 and most of Q1 (18 of about 18 squad-weeks available before certification). So it adds help or buffer (Platform or another squad at 1.5×, or cutting everything else from Cards) and moves virtual cards out of Q1: both v2 features start only after cut-over (virtual cards Q2, spend controls Q2–Q3). It treats the new squads as upside: on past hiring they arrive around February to April 2027 and deliver half capacity in their first quarter, so nothing committed depends on them before Q2–Q3. It makes a clear call on procurement: the $6M claim needs about 890 accounts (81% of customers) paying $6,720 a year each (70 requesters × $8 × 12), and only 2 of 6 design partners would pay, so it replaces the 30-week build with a cheap paid test with a threshold, and keeps Approvals & Policy on multi-entity and spend controls, which have stronger expansion evidence ($2.9M sized, 9 of 14 committed in writing; about $1.5M a year of interchange from moving a third of the $410M of subscriptions onto virtual cards). Intacct comes early for its churn; Dynamics later or cut; SSO and SCIM are small and justified; the mobile rewrite waits for a new squad, before the framework's support ends. It lists what changed from Tom's draft, what's not being done, and the risks (certification slipping, hiring slipping), with commitment falling with distance.

Critical failures (cap the score at 40)
  • Leaves the processor migration finishing too late for 4 weeks of certification before 31 March 2027, or schedules virtual cards or real-time spend controls before it
Task

Staff-level · v1.7

The outputs

9 setups answered this brief.

Usable with a quick edit10 of 13 checks passed

The output is a well-argued, evidence-based roadmap and case that a PM could take to the offsite with only minor polish.

Output

FY27 Product Roadmap: Path to 112% NRR

To: Tom Achebe (CEO), CFO, CRO, CTO From: Staff PM Subject: Pre-read: Proposed Q4 2026 – Q3 2027 Product Roadmap

---

1. The FY27 Roadmap (Q4 2026 – Q3 2027)

To hit 112% NRR by the end of Q3 2027, we need $6.88M in net expansion ARR (8% of our $86M base). Because expansion revenue only counts toward Q3 NRR if it is live, rolled out, and generating spend before the end of Q3, this roadmap front-loads our highest-conviction NRR drivers and protects our core card infrastructure.

Note on Capacity: Each existing squad has 11 squad-weeks (SW) per quarter. Cross-squad work incurs a 1.5× penalty (e.g., 4 SW of native work takes 6 SW for a borrowing squad). New squads are modeled using our historical hiring baselines: 6 months to onboard and 50% capacity (5.5 SW) in their first active quarter.

Q4 2026: Clear the Decks & Lock In Quick Wins

  • Cards: Processor v2 Migration (11 of 18 SW). Outcome: Protect 58% of Harbor’s revenue ahead of the March 31 hard deadline.
  • Approvals & Policy (A&P): Multi-Entity Support (10 of 10 SW). Outcome: Unlock $2.9M ARR subsidiary expansion pipeline.
  • Platform: Multi-Entity Support (11 of 12 SW). Outcome: Same as above; ships in week 1 of Q1.
  • Integrations: Sage Intacct Integration (9 of 9 SW). Outcome: Slash gross churn from 13% to ~7% across 17% of our customer base (~$880k ARR saved) and remove the blocker in 31% of lost new-logo deals.
  • Expenses: Receipt-Matching Improvements (6 of 6 SW); begin Mobile App Rewrite (5 of 22 SW). Outcome: Cut our #1 support ticket driver to free up engineering on-call capacity; start retiring late-2027 mobile framework debt.

Q1 2027: Cutover v2 & Launch Multi-Entity

  • Cards: Finish Processor v2 Migration (3 SW in Weeks 1–3), Freeze for Certification (Weeks 4–7), Cutover (Week 8). Start Virtual Cards for SaaS (8 of 10 SW, built on v2 environment). Outcome: Zero card downtime on March 31; pull Virtual Cards launch into early Q2.
  • A&P: Assist Cards on Processor v2 Migration (6 SW = 4 native Cards SW in Weeks 1–6); Real-Time Spend Controls (4 of 4 native SW). Outcome: Crucial cross-squad assist ensures Cards hits the Feb 1 code freeze for v2 certification.
  • Platform: Finish Multi-Entity (1 of 12 SW); SSO & SCIM Provisioning (5 of 5 SW); Multi-Entity customer onboarding & scale hardening (5 SW). Outcome: Ship Multi-Entity in January for immediate CS expansion selling; unlock $1.2M new-logo pipeline and protect 11 enterprise renewals.
  • Integrations: Microsoft Dynamics Integration (11 of 12 SW). Outcome: Retain the 3% of base on Dynamics and remove the 8% new-logo blocker.
  • Expenses: Mobile App Rewrite (11 of 22 SW). Outcome: Hit 73% completion on mobile debt retirement without hiring dependencies.

Q2 2027: Capture the $410M Card Spend Opportunity

  • Cards: Finish Virtual Cards for SaaS (2 of 10 SW—Ships April); Real-Time Spend Controls (8 of 8 SW—Ships June). Outcome: Capture share of the $410M in off-Harbor software spend ($4.5M total interchange pool) requested by 43% of top customers; give 52% of top CFOs the point-of-sale controls needed to move remaining company spend to Harbor.
  • A&P: Spend Controls integration support & policy templates (4 SW); Discovery & v2 Architecture for Light Vendor/PO Workflows (7 SW). Outcome: Ensure rapid adoption of Spend Controls; re-architect our procurement approach (see The Case below).
  • Integrations: Finish Dynamics (1 of 12 SW); ERP syncing for Virtual Cards & Multi-Entity (10 SW). Outcome: Ensure seamless reconciliation for expanded multi-entity and SaaS card spend.
  • Platform: Card spend limit notifications, alerting & multi-entity RBAC (11 SW). Outcome: Drive self-serve adoption of Virtual Cards across existing accounts.
  • Expenses: Finish Mobile App Rewrite (6 of 22 SW—Ships May); Card spend receipt automation (5 SW). Outcome: Retire framework support risk 6 months ahead of CTO deadline.
  • New Squad A (Ramping: 5.5 SW): Build self-serve migration tools for recurring SaaS bills to Virtual Cards. Outcome: Accelerate time-to-interchange for Virtual Cards.

Q3 2027: Drive Adoption to Lock In 112% NRR

  • Cards & Platform: Automated SaaS vendor detection (flagging invoice spend in ERP to convert to Virtual Cards) & card limit auto-top-ups. Outcome: Maximize interchange realization before Q3 closes.
  • A&P, New Squad A (11 SW) & New Squad B (5.5 SW): Build redesigned intake-to-card workflows (issuing single-use PO virtual cards upon approval). Outcome: Monetize approval workflows via interchange and seat expansion rather than competing head-on with entrenched procurement suites.
  • Integrations & Expenses: In-app onboarding and bulk-provisioning tooling for Multi-Entity subsidiaries. Outcome: Help CS close the remaining 52 subsidiaries before EOY.

---

2. The Case for This Roadmap

Why This Order? (The Math to 112% NRR)

To add $6.88M in net ARR by the end of Q3 2027, we must prioritize high-margin revenue protection and high-velocity expansion that can land in H1 2027: 1. Stop the bleeding via Intacct & SSO (+$0.9M to $1.1M ARR retained): NRR is a net number; every dollar of churn avoided is a dollar of expansion we don't have to sell. 17% of our base (~$14.6M ARR) uses Intacct via CSV and churns at 13% vs. 7% for NetSuite. Fixing this in Q4 saves ~$880k/yr in gross churn and unlocks 31% of lost deals. Pairing this with SSO/SCIM in Q1 secures 11 enterprise accounts under security review. 2. Ship Multi-Entity in week 1 of Q1 (+$1.8M to $2.9M ARR): 9 top customers have committed in writing to add subsidiaries once live. Sizing all 52 subsidiaries in the top-60 cohort equals $2.9M ARR—and more exist across our other 1,040 customers. Shipping in January gives CS three full quarters to close and onboard these entities. 3. Unleash Card Spend in Q2 (+$2.5M to $3.5M ARR): Our customers spend $410M/yr on software off-Harbor. At 1.1% net interchange, that is a $4.51M high-margin revenue pool sitting inside our existing base. Nearly half (26 of 60) of our largest accounts are actively asking for Virtual Cards for subscriptions, and 31 CFOs want Real-Time Spend Controls. Unlike new modules that require procurement cycles and legal sign-off, turning on virtual cards for existing customers generates interchange revenue the day a card is swiped.

Combined with our baseline 104% NRR ($3.44M), capturing just half of our conservative targets across these three pillars puts us at 112.5% NRR ($96.8M ARR) by Q3.

What Changed from Tom’s Draft—and Why

1. Resolving the March 31 Card Processor Crisis Tom’s draft had the Cards squad doing the 18-SW v2 migration across Q4 and Q1. However, Cards only has 11 SW per quarter. Furthermore, the processor requires a 4-week certification freeze before the March 31 cutover. To cut over by March 31, code must be finished by February 26 (Week 8 of Q1). In 19 weeks of calendar time, the Cards squad only has ~17.5 SW of capacity—leaving zero margin for error and pushing Virtual Cards into late Q2/Q3. * The Fix: In Q1, Approvals & Policy gives 6 SW of capacity (yielding 4 native SW at the 1.5× cross-squad rate) to the Cards team. This finishes all 18 SW of v2 code by the end of January, allowing certification in February, safe cutover in early March, and pulling Virtual Cards forward by 6 weeks.

2. Replacing the $6M Procurement Bet with Card Expansion Tom’s draft relies on Procurement to add $6M ARR in year one. The customer and capacity math does not support this: * The Revenue Math: At $8/month ($96/yr) for the 10% of employees who raise purchase requests (70 users per average 700-person customer), 100% adoption across all 1,100 customers would only generate $7.39M ARR. To hit $6M in Year 1, we would have to sell an unbuilt add-on to 81% of our entire customer base in two quarters. Yet in our design partner trial, 4 of 6 customers already had a procurement tool and refused to switch, and only 2 (33%) were willing to pay. Realistic Year-1 ARR is ~$500k–$900k. * The Capacity Math: Procurement requires 30 SW from A&P and 6 SW from Integrations. Tom assigned this to "New Squad A" starting in Q1. Last year, our new squads took 5 to 8 months to reach their first sprint, and delivered 50% capacity in their first quarter. Approved in September, Squad A won't run a sprint until Q2 2027 (delivering ~5.5 SW), and as a brand-new squad working in A&P's domain, they face the 1.5× ramp penalty. Assigned to Squad A, a 36-SW Procurement module would not ship until Q1 2028—contributing $0 to FY27 NRR.

3. De-risking the Mobile App Rewrite Tom assigned the 22-SW Mobile Rewrite to "New Squad B" in Q2. Approved in September, Squad B likely won't start until Q3 2027 at half capacity (5.5 SW), missing the CTO’s late-2027 framework end-of-life deadline. By keeping our existing Expenses squad on Mobile starting in Q4 (after a 6-SW sprint on Receipt Matching), we complete the rewrite in Q2 2027 with zero hiring dependency.

What We Are Not Doing

  • We are not building a standalone, 36-SW Procurement Module in H1. We cannot justify starving Multi-Entity and Spend Controls to build a module that 67% of design partners rejected and that cannot ship in time to impact FY27 NRR. Instead, in Q2/Q3, we will explore lightweight PO-to-Virtual-Card workflows that drive card spend rather than competing with Coupa or Zip.
  • We are not counting on un-hired squads for critical-path H1 deliverables. New Squads A and B are scheduled for Q2 and Q3 respectively, focused on adoption tooling and H2 expansion where hiring delays won't threaten our board target or infrastructure deadlines.

Key Risks That Could Change This Plan

  1. Processor v2 Certification Rejections: If the processor finds bugs during the 4-week February certification window, our freeze extends and delays Virtual Cards. Mitigation: By using A&P to help finish the build in January, we buy a 4-week buffer before the March 31 cliff.
  2. Cross-Squad Ramp Friction: Having A&P take on 4 SW of Cards work (costing 6 SW) in Q1 requires clean work partitioning. Mitigation: Cards will own the core v2 authorization webhooks; A&P will own isolated migration scripts and fallback testing.
  3. Slower Multi-Entity Sales Cycles: While 9 accounts gave written intent, parent-subsidiary contract addendums can stall in legal. Mitigation: CS and Finance must pre-negotiate subsidiary pricing addendums with the 14 target accounts in Q4 so we can bill immediately upon the January release.

Grades and run details

Decision model 62 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly41%
  • passAddresses the actual decision70%
  • partialRespects explicit constraints11%
  • passIdentifies material uncertainty53%
  • partialAvoids unsupported claims46%
  • passProduces the required deliverable49%
  • partialProtects the processor deadline9%
  • passPlans on the squads we actually have89%
  • partialMakes the call on procurement73%
  • passOutcomes, with certainty that falls with distance67%
  • failFits the stated capacity64%
  • passSequences around dependencies11%
  • failTells the story of the bets74%
Run
Run
#1
API response time
32 s
Submitted
13 Oct 2026

Needs more than a quick edit7 of 13 checks passed

The Integrations capacity gap in Q1 and the missing procurement threshold would require rework before a PM could use it.

Output

Part 1: Product Roadmap (Q4 2026 – Q3 2027)

``` ==================================================================================================== SQUAD Q4 2026 Q1 2027 Q2 2027 Q3 2027 ==================================================================================================== Cards Processor v2 Migration Finish v2 (Wks 1-7) Virtual Cards for SaaS Real-Time Controls (11 wks) API Freeze/Cert (W8-11) (6 wks remaining) (Cards: 3 wks) Virtual Cards Prep Real-Time Controls Interchange Optimization (4 wks) (Cards: 5 wks) & Scaling (8 wks) ---------------------------------------------------------------------------------------------------- Platform Multi-Entity: Core Multi-Entity: Wrap-up Platform Scaling & Cross-Entity Reporting Data Architecture (1 wk) Org Hierarchies & Enterprise Admin (11 wks) SSO & SCIM (5 wks) (11 wks) (11 wks) Hardening (5 wks) ---------------------------------------------------------------------------------------------------- Approvals & Multi-Entity: Policy Multi-Entity: Release Real-Time Controls: Procurement Discovery Policy & Approval Routing (2 wks) Policy Engine (4 wks) & Scoping (MVP) (10 wks) A&P Rule Engine Complex Approval Rules (Approvals: 11 wks) Buffer (1 wk) Upgrades (9 wks) (7 wks) ---------------------------------------------------------------------------------------------------- Integrations Sage Intacct Native Sage Intacct GA (2 wks) MS Dynamics Integration MS Dynamics GA (3 wks) Integration MS Dynamics Integration (Integrations: 10 wks) ERP Webhooks & Custom (9 wks) + Buffer (2 wk) (Integrations: 2 wks) Sync (8 wks) ---------------------------------------------------------------------------------------------------- Expenses Receipt Auto-Match Mobile App Rewrite: Mobile App Rewrite: Mobile App GA & Optimization (6 wks) Phase 1 Core Workflows Phase 2 Parity/Release Expense Policy Polish Mobile Prep (5 wks) (11 wks) (11 wks) (11 wks) ---------------------------------------------------------------------------------------------------- New Squad A [Hiring / Onboarding] [Ramp / Training] Procurement Foundation Procurement Workflows (A&P Focus) (Ramped: ~5.5 wks) (Full capacity: 11 wks) ---------------------------------------------------------------------------------------------------- New Squad B [Hiring / Pipeline] [Hiring] [Onboarding / Ramp] Integrations / Core (Ecosystem Focus) (Ramped: ~5.5 wks) ==================================================================================================== ```

Squad-by-Squad Outcomes

  • Cards:
  • Processor v2 Migration: Protects 100% of Harbor’s card-authorisation volume ($49.9M existing ARR) against the hard 31 March 2027 processor shutoff; unlocks v2 capabilities required for virtual cards and real-time controls.
  • Virtual Cards for Software Subscriptions: Captures existing customer SaaS spend currently paid via invoice/other cards ($410M addressable pool), directly driving interchange revenue to expand NRR.
  • Real-Time Spend Controls: Enables 31 top-tier CFO accounts to enforce budgets at point-of-sale, unlocking enterprise card issuance and shifting unmanaged invoice spend to Harbor cards.
  • Platform:
  • Multi-Entity Architecture: Unlocks $2.9M contracted expansion ARR across 52 non-Harbor subsidiaries of top accounts.
  • SSO & SCIM Provisioning: Unblocks $1.2M in enterprise pipeline and closes compliance gaps for 11 critical renewal accounts.
  • Approvals & Policy:
  • Multi-Entity Approvals: Delivers cross-subsidiary approval chains required to close existing multi-entity commitments.
  • Real-Time Policy Engine: Co-delivers immediate POS budget validation with the Cards squad.
  • Procurement Discovery & MVP Architecture: Reframes procurement around verified buyer demand without starving committed delivery.
  • Integrations:
  • Sage Intacct Integration: Halves the 13% annual churn rate of the 17% of Harbor customers on Intacct (retaining ~$875k ARR annually) and addresses Harbor's #1 new-logo loss reason (31% of lost deals).
  • Microsoft Dynamics Integration: Expands ERP coverage for mid-market accounts, removing friction for 3% of existing customers and 8% of pipeline.
  • Expenses:
  • Receipt-Matching Improvements: Drives match rate above 71%, eliminating Harbor’s highest-volume support ticket category and improving user retention.
  • Mobile App Rewrite: Eliminates framework deprecation risk ahead of the late-2027 vendor cutoff without stealing capacity from revenue-generating squads.

---

Part 2: The Case for the Roadmap

Executive Summary

To achieve our board-approved target of 112% NRR by Q3 2027 ($86M ARR base expanding by ~$6.9M), Harbor must focus on the highest-probability, shortest-time-to-value expansion levers within our current 1,100 customers: multi-entity expansion ($2.9M ARR), capturing existing software spend via virtual cards ($1.3M–$2.2M ARR), and plugging Sage Intacct churn ($875k ARR retained).

Tom’s draft relied on an unbuilt Procurement module to generate $6M ARR in FY27. Ground-level capacity, vendor deadlines, and design partner data show that strategy carries severe revenue and operational risk. The proposed roadmap secures our existential card processor deadline, resolves major customer expansion requests, and establishes realistic hiring and product timelines.

---

What Changed from Tom’s Draft and Why

#### 1. Realignment on Procurement ($6M ARR Myth vs. Reality) * The Math: Harbor has 1,100 customers averaging 700 employees (770,000 total users). With 10% raising purchase requests (70 users/account) at $8/user/month ($96/year), an adopting customer generates $6,720 ARR. To generate $6M ARR, 893 out of 1,100 customers (81% of our entire base) would have to buy and deploy a v1 module in under 9 months. * The Customer Signal: 4 out of 6 design partners already use dedicated tools (e.g., Coupa, Zip) and explicitly stated they will not switch. Only 2 of 6 expressed willingness to pay. * The Decision: Procurement cannot carry our FY27 NRR goal. We have reallocated primary capacity to guaranteed expansion drivers (Multi-Entity and Virtual Cards), while initiating a structured discovery and phased MVP build with Squad A.

#### 2. Realistic Hiring Capacity * Tom’s Assumption: Squad A operates at 100% capacity in Q1 2027 (11 squad-weeks) and Squad B in Q2 2027 (11 squad-weeks). * Historical Reality: Approvals to first full sprint took 5–8 months historically, delivering ~50% capacity in quarter one. Squads approved in September 2026 will not reach operational productivity until late Q1 / early Q2 2027. * The Decision: No critical-path milestone or revenue target in Q4 or Q1 relies on unhired squads. Existing squads carry core commitments; Squad A ramps on Procurement foundations in Q2 (delivering ~5.5 squad-weeks) and reaches full delivery in Q3.

#### 3. Strict Compliance with the 31 March 2027 Processor Deadline * The Constraint: The v1 API deprecates on 31 March 2027. Migrating takes 18 squad-weeks plus a mandatory 4-week processor certification freeze during which our codebase cannot change. * Execution Window: The 4-week certification must run in March 2027. Cards code complete must occur by Week 7 of Q1 2027. * The Allocation: Cards dedicates all 11 squad-weeks in Q4 2026 and 7 squad-weeks in Q1 2027 (18 weeks total). During the 4-week March freeze, the Cards squad transitions to architectural prep for Virtual Cards, avoiding wasted idle time.

#### 4. Resequencing Multi-Entity, SSO/SCIM, and Sage Intacct * Multi-Entity (Platform + Approvals): Prioritised immediately in Q4. 14 of our 60 largest accounts requested this; 9 have signed written commitments to add 52 subsidiaries ($2.9M ARR). Shipping in early Q1 allows two full quarters of rollout and revenue recognition. * Sage Intacct (Integrations): 17% of our customers use Intacct via CSV, churning at 13% vs. NetSuite’s 7%. Cutting churn across these 187 accounts ($14.6M ARR base) saves ~$875k in ARR retention while eliminating our top sales loss reason (31% of lost deals). * SSO & SCIM (Platform): Sequenced for Q1 immediately following core Multi-Entity work. For a 5-week lift, this unlocks $1.2M in stalled enterprise pipeline and secures 11 major renewals.

---

How this Roadmap Hits 112% NRR

By the end of Q3 2027, this roadmap expands our $86M base by $7.0M+ in recurring expansion and retained ARR, exceeding the board's 112% target ($6.88M incremental):

InitiativeMechanismTarget ARR Impact
:---:---:---
Multi-Entity52 parent-account subsidiaries onboarding+$2.9M ARR
Virtual Cards (SaaS)Capturing 35% of $410M unmanaged SaaS spend @ 1.1% net interchange+$1.58M ARR
Real-Time Spend ControlsUnlocking enterprise card adoption across 31 QBR accounts+$0.80M ARR
Sage Intacct IntegrationEliminating 6% excess churn across 187 customer accounts+$0.88M ARR
SSO & SCIMEnterprise expansion & churn protection across 11 key accounts+$0.90M ARR
Total Net Expansion+$7.06M ARR (~112.2% NRR)

---

What We Are Not Doing (And Why)

  1. Not building Procurement as a hasty monolith in Q1: Building a broad module against negative design-partner feedback on an imaginary timeline would burn 36 squad-weeks with minimal revenue return. We are building the data model and workflows methodically using Squad A once ramped.
  2. Not assigning the Mobile Rewrite to an unhired squad: Tom assigned this 22-week project to Squad B in Q2. Squad B will not be staffed in time. Instead, the Expenses squad owns mobile: they finish Receipt Matching in Q4 (improving our #1 support issue) and execute the mobile rewrite across Q1 and Q2, completing it comfortably before late-2027 framework retirement.
  3. Not prioritising Microsoft Dynamics over Sage Intacct: Dynamics accounts for only 3% of our customer base and 8% of lost deals (vs. Intacct’s 17% base and 31% deal loss). Dynamics is scheduled for Q2 after Intacct is fully deployed.
  4. No cross-domain squad thrashing: Squads working out of domain suffer a 1.5× productivity penalty. We kept all squads focused within their domain expertise rather than forcing Expenses or Integrations to build Cards or Approvals features.

---

Risks and Mitigation Strategies

1. Card Processor Certification Delay: * Risk: Processor testing slips past 4 weeks, threatening the 31 March v1 shutoff. * Mitigation: We complete Harbor development by Week 7 of Q1 (mid-February), yielding a two-week safety buffer before the freeze. Cards works closely with processor engineers weekly starting in Q4. 2. Multi-Entity Delivery Bottlenecks: * Risk: Multi-Entity touches both Platform (12 wks) and Approvals & Policy (10 wks); cross-squad misalignment could delay Q1 rollout. * Mitigation: The technical spec and API contract between Platform and A&P will be finalized in Sprint 1 of Q4, enabling independent parallel development. 3. Hiring Lags for Squad A and B: * Risk: Talent acquisition takes longer than the historical 5–8 months. * Mitigation: The baseline revenue plan (Multi-Entity, Virtual Cards, Intacct, SSO) relies entirely on our five existing squads. Squad A’s onboarding pace dictates the launch timing of Procurement, not the attainment of our 112% NRR target.

Grades and run details

Decision model 50 · LLM judge 64
Decision model checks
  • failUses the supplied evidence correctly58%
  • partialAddresses the actual decision22%
  • partialRespects explicit constraints14%
  • partialIdentifies material uncertainty25%
  • partialAvoids unsupported claims33%
  • passProduces the required deliverable35%
  • passProtects the processor deadline34%
  • passPlans on the squads we actually have94%
  • partialMakes the call on procurement80%
  • passOutcomes, with certainty that falls with distance23%
  • failFits the stated capacity70%
  • failSequences around dependencies11%
  • failTells the story of the bets79%
Run
Run
#1
API response time
45 s
Submitted
1 Oct 2026

Needs more than a quick edit11 of 13 checks passed

The plan lacks any buffer for the processor migration, which is a critical risk, and the squad-based structure doesn't tell a thematic story for the exec offsite.

Output

Harbor FY27 roadmap — Q4 2026 to Q3 2027

Planning basis: Each existing squad has about 11 roadmap squad-weeks per quarter. Figures below are the candidate-work estimates; capacity not assigned to a feature is deliberately held for launch, customer activation, fixes and contingency. New squads are not counted at full capacity until their ramp is proven.

QuarterSquadWork and outcome
Q4 2026CardsProcessor v2 migration — 11 of 18 weeks. Start the mandatory migration and establish the path to retire v1 before the processor deadline.
Approvals & PolicyMulti-entity — 10 of 10 weeks. Build parent/subsidiary approval and policy support so existing customers can bring more entities onto Harbor.
PlatformMulti-entity — 11 of 12 weeks. Build the platform foundation for subsidiaries under a parent account.
IntegrationsSage Intacct — 9 weeks. Replace CSV workarounds to reduce a retention risk among existing customers.
ExpensesReceipt matching — 6 weeks. Improve the most common expense-ticket driver; use remaining capacity for fixes and customer feedback.
Q1 2027CardsFinish v2 — 7 weeks. Target implementation complete by the end of February, leaving March for the processor’s four-week certification. No card-side changes during certification.
Approvals & PolicyMulti-entity launch and customer activation. Support the first subsidiary rollouts. Use remaining capacity for a time-bounded procurement willingness-to-pay and switching validation; do not build the module.
PlatformFinish multi-entity — 1 week; SSO/SCIM — 5 weeks. Complete the entity foundation, then address security blockers affecting existing accounts and enterprise pipeline.
IntegrationsMicrosoft Dynamics — 11 of 12 weeks. Address a cited reason for lost new-logo deals; complete the final week in Q2.
ExpensesMobile rewrite — 11 of 22 weeks. Start early to reduce the risk of running out of time before the current framework loses support in late 2027.
Q2 2027CardsVirtual cards for software subscriptions — 10 weeks. After v2 certification, give customers a way to move subscription spend onto Harbor cards and capture more interchange.
Approvals & PolicyReal-time spend controls — 4 weeks. Build the policy logic for point-of-sale declines, in step with Cards’ Q3 work.
PlatformSSO/SCIM and multi-entity activation. Support adoption and resolve launch issues; no new major platform feature is precommitted.
IntegrationsFinish Dynamics — 1 week; customer rollout. Complete the integration and support implementation.
ExpensesFinish mobile rewrite — 11 weeks. Complete the rewrite by Q2, ahead of the framework support deadline.
Q3 2027CardsReal-time spend controls — 8 weeks. Complete and release with Approvals & Policy; help finance teams enforce policy at the point of sale.
Approvals & PolicyRelease and adoption for real-time controls; multi-entity activation. Focus on turning shipped capabilities into customer usage and expansion.
PlatformActivation and reliability for multi-entity and SSO/SCIM. Prioritize customer rollout and fixes over an unvalidated new feature.
IntegrationsIntacct and Dynamics adoption. Help customers get live and measure retention and deal impact before choosing further integration work.
ExpensesMobile rollout and stabilization; measure receipt-match performance. Address launch issues and confirm the support-ticket outcome.

New Squads A and B: Plan for onboarding and ramp, not full feature capacity. When productive, use them for bounded rollout, quality or activation work. No committed launch depends on their availability.

Case for the plan

Why this order

The board’s priority is to move NRR from 104% to 112% by the end of Q3, primarily through expansion in existing customers. The plan therefore favors credible opportunities to retain customers or deepen their use of Harbor, while keeping some room for new-logo needs.

First, v2 is a hard dependency and a hard deadline. All authorisations still run through v1, which retires on 31 March. The 18 Cards squad-weeks consume 11 weeks in Q4 and seven in Q1. That leaves time to finish implementation before the processor’s four-week certification and to keep the required freeze during testing. Virtual cards and real-time controls must wait until certification is complete.

Next, multi-entity has concrete expansion evidence: 14 of the 60 largest customers asked, and nine have said in writing that they would add subsidiaries. The $2.9M ARR estimate assumes all 52 subsidiaries join, so it is an upside case, not a commitment. Completing the Platform and Approvals & Policy work across Q4 and Q1 creates a path to test that demand with real rollouts.

Virtual cards are the first post-migration Cards feature because 26 large customers asked for them and Finance estimates $410M of annual subscription spend is currently elsewhere. At Harbor’s 1.1% net interchange, full capture would represent about $4.5M in annual interchange revenue, but that is a ceiling, not a forecast: the spend estimate is extrapolated, and adoption and capture are unknown. Real-time controls follow, with coordinated work across Cards and Approvals & Policy. The 31 CFO requests are strong demand evidence, but the plan avoids trying to fit both features into Cards’ 11-week quarterly capacity.

Intacct comes before Dynamics. Intacct is used by 17% of customers through CSV, and those customers have higher observed gross revenue churn than customers on NetSuite. That comparison is correlational, but it supports addressing a retention risk. Dynamics has lower existing use (3%) and a more limited new-logo case, but the 8% of lost deals citing it justifies completing the 12-week integration after Intacct.

SSO/SCIM is a relatively small, five-week Platform investment tied to five enterprise deals representing $1.2M of pipeline and 11 existing security reviews. It supports commercial outcomes without displacing the larger expansion bets.

What changed from Tom’s draft

  • Removed procurement from committed build and launch. Six design partners are not enough evidence for the board’s $6M first-year ARR claim: only two said they would pay, and four already use another tool. At the proposed price, an average 700-employee customer would generate about $6,720 ARR if 10% of employees were buyers. Reaching $6M would require roughly 893 such customer equivalents. We should validate willingness to switch and pricing before committing 30 Approvals & Policy and six Integrations squad-weeks.
  • Corrected the hiring assumption. Prior squads took five to eight months from approval to a first full sprint and delivered about half capacity in their first quarter. The plan does not count either new squad at full capacity in Q1 or Q2, or make a feature dependent on them.
  • Made the v2 critical path explicit. The migration spans Q4 and Q1, with implementation targeted ahead of March certification. Tom’s draft did not protect that freeze clearly enough.
  • Sequenced Cards work to fit capacity. Virtual cards and real-time controls require 18 Cards squad-weeks together, more than one quarter’s 11-week capacity. They are sequenced across Q2 and Q3.
  • Started the mobile rewrite with the existing Expenses squad. The work is 22 weeks, so starting in Q1 targets completion in Q2 without relying on an unproven new squad.

What we are not doing

We are not committing to build procurement, starting a mobile rewrite with a new squad at assumed full capacity, or taking on another major integration or platform feature in Q3 without evidence from the launches above. New-logo work remains in the plan, but expansion and retention come first.

Risks that could change the plan

  • Processor timing: Any migration slip puts card authorisations at risk. Track delivery and certification weekly; protect the Q1 card schedule and do not pull dependent features forward.
  • Expansion may not materialize: Multi-entity interest and card requests do not guarantee adoption or spend migration. Track subsidiary activation, card spend moved, and controls usage—not just releases.
  • Commercial estimates are uncertain: The $2.9M multi-entity estimate and $4.5M card-spend ceiling are not forecasts. Procurement’s $6M claim is especially unsupported by current evidence.
  • Mobile has little slack: The 22-week rewrite fills two quarters of Expenses capacity. Scope growth, delivery delays or an earlier-than-expected framework deadline would require replanning.
  • Hiring, integration and cross-squad delivery may slip: Ramp timing is uncertain; the Intacct churn difference may not be caused by the integration; and spend controls depend on coordinated Cards and Approvals & Policy work.

This roadmap gives Harbor credible bets aligned to the NRR goal, but it does not establish that the 112% target is guaranteed. We should review expansion and retention indicators at each quarterly planning checkpoint and change the plan if customer evidence or delivery risk materially shifts.

Grades and run details

Decision model 85 · LLM judge 79
Decision model checks
  • passUses the supplied evidence correctly64%
  • passAddresses the actual decision48%
  • passRespects explicit constraints14%
  • passIdentifies material uncertainty88%
  • passAvoids unsupported claims72%
  • passProduces the required deliverable77%
  • partialProtects the processor deadline38%
  • passPlans on the squads we actually have98%
  • partialMakes the call on procurement37%
  • passOutcomes, with certainty that falls with distance73%
  • passFits the stated capacity37%
  • passSequences around dependencies62%
  • failTells the story of the bets94%
Run
Run
#1
API response time
1.9 min
Submitted
1 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 7

Uses the supplied evidence correctlyMixedMixedRight
Gemini 4 Argon · API

All factual claims about current situation are drawn directly from the brief or context, or follow from arithmetic.

Gemini 3.8 Flash · API

All claims about the current situation are directly from the brief or supplied context.

GPT-6 Luna · API

All factual claims about the current situation are directly from the brief or derived by correct arithmetic.

Addresses the actual decisionRightWrongRight
Gemini 4 Argon · API

The output commits to a clear roadmap and a case that directly answers the CEO's request, with risks that could change the plan.

Gemini 3.8 Flash · API

The output commits to a roadmap but does not explicitly state what result or condition would change that decision (e.g., a threshold for procurement MVP to trigger full build).

GPT-6 Luna · API

Commits to a clear plan (no procurement build, validation instead) and states that quarterly review of expansion and retention indicators could change it.

Protects the processor deadlineRightRightWrong
Gemini 4 Argon · API

The migration builds finish by late January, leaving 4 weeks of certification plus buffer, and v2 features start only after cutover.

Gemini 3.8 Flash · API

Migration finishes by Week 7 of Q1, leaving 4 weeks of certification before 31 March, and virtual cards/spend controls start in Q2 after cut-over.

GPT-6 Luna · API

The plan uses all of Cards' capacity with no buffer or help from other squads, leaving no protection against slippage before the 31 March deadline.

Makes the call on procurementRightWrongRight
Gemini 4 Argon · API

It shows the $6M claim requires ~81% adoption, weighs the design partner rejection rate, assigns A&P to better-evidenced work, and proposes a lightweight card-driven test instead of the full module.

Gemini 3.8 Flash · API

It checks the $6M claim and commits to discovery/MVP, but does not propose a smaller test with a clear threshold that would justify the full build.

GPT-6 Luna · API

Checks the $6M claim against pricing and adoption evidence, shows it would need ~893 customers, and replaces the build with a paid validation test.

Fits the stated capacityMixedWrongRight
Gemini 4 Argon · API

Committed work fits within stated capacity with sums that can be checked, and it explains that the full procurement module and new-squad dependencies were cut.

Gemini 3.8 Flash · API

The Integrations squad in Q1 is allocated only 4 weeks (Sage Intacct GA 2 wks, MS Dynamics 2 wks) out of 11, leaving a capacity gap.

GPT-6 Luna · API

Committed work fits within 11 squad-weeks per quarter with slack held for launch and fixes, and it names what was cut (procurement build, additional features).

Sequences around dependenciesRightMixedRight
Gemini 4 Argon · API

v2 features start after migration and certification; multi-entity, Intacct and other items are sequenced around their dependencies.

Gemini 3.8 Flash · API

All dependencies (v2 before virtual cards/controls, multi-entity before cross-entity, Intacct before Dynamics) are respected and named.

GPT-6 Luna · API

Respects v2 dependency for virtual cards and spend controls, places them after certification, and names the dependencies that drive the order.

Tells the story of the betsMixedWrongWrong
Gemini 4 Argon · API

The roadmap is grouped into quarterly themes (Clear the Decks, Cutover v2, Capture Card Spend, Drive Adoption) that tell a coherent investment story.

Gemini 3.8 Flash · API

The roadmap is presented as a squad-by-quarter grid, not grouped into a few themes with the reason each matters most now.

GPT-6 Luna · API

The roadmap is organized by squad, not grouped into a few themes with a narrative about why those are the biggest bets now.

All got right 6

Respects explicit constraintsRightRightRight
Gemini 4 Argon · API

The roadmap is by quarter, the case is under 1,200 words, and the memo is addressed to the exec team as requested.

Gemini 3.8 Flash · API

The roadmap and case are present, within 1,200 words, and address the exec team as reader.

GPT-6 Luna · API

Delivers a quarterly roadmap with outcomes and a case under 1,200 words covering the required topics for the exec team.

Identifies material uncertaintyRightRightRight
Gemini 4 Argon · API

It names specific risks (certification rejections, cross-squad friction, multi-entity legal stalls) and describes how they could change the plan.

Gemini 3.8 Flash · API

It names specific risks (certification delay, multi-entity bottlenecks, hiring lags) and mitigation strategies.

GPT-6 Luna · API

Names specific risks (processor timing, adoption, commercial estimates, mobile slack, hiring) and how they would be tracked or resolved.

Avoids unsupported claimsRightRightRight
Gemini 4 Argon · API

Interpretations and forecasts are clearly labelled as estimates or conclusions, not as established facts.

Gemini 3.8 Flash · API

Forecasts are presented as targets, not established facts, and conclusions are based on supplied evidence.

GPT-6 Luna · API

Labels correlations and extrapolations clearly (e.g., 'correlational', 'ceiling, not a forecast') and avoids presenting them as established fact.

Produces the required deliverableRightRightRight
Gemini 4 Argon · API

The roadmap and case are complete, in the requested form, and could be used by the exec team with light edits.

Gemini 3.8 Flash · API

The roadmap table and case are complete and usable for the planning offsite.

GPT-6 Luna · API

The roadmap and case are complete, in the requested form, and usable by the exec team without major gaps.

Plans on the squads we actually haveRightRightRight
Gemini 4 Argon · API

New squads are modelled to arrive at half capacity in Q2/Q3, and no critical-path deliverables depend on them before then.

Gemini 3.8 Flash · API

Uses historical hiring data to place new squads at half capacity in Q2/Q3, and commits no critical-path work to them before then.

GPT-6 Luna · API

Does not count new squads at full capacity in Q1 or Q2 and commits no important work to them, consistent with past hiring delays and ramp.

Outcomes, with certainty that falls with distanceRightRightRight
Gemini 4 Argon · API

Every roadmap item states an outcome; near-term quarters are specific and later ones are deliberately looser.

Gemini 3.8 Flash · API

Every item names its outcome, and later items are deliberately looser (e.g., 'Cross-Entity Reporting & Enterprise Admin').

GPT-6 Luna · API

Every item names an outcome or problem, and later items are deliberately looser (e.g., 'focus on turning shipped capabilities into customer usage').

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 79% of 150 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT88.0100.02None
2Opus 5.5withClaude88.396.42None
3GPT-6.1 SolwithAPI88.396.42None
4Sonnet 5.5withAPI90.292.62None
5Haiku 5.5withAPI74.496.42None
6GPT-6 LunawithAPI84.085.42None
7Gemini 4 ArgonwithAPI62.088.52None
8Gemini 3.8 FlashwithAPI58.366.82None
9Gemini 3.5 Flash-LitewithGemini57.136.521 capped

What you get for the cost

GPT-6.1 Sol · API scores within a point of GPT-6 Astra · ChatGPT at a sixth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 94, 39× the cheapestOpus 5.5 · Claude: task score 93, 37× the cheapestGPT-6.1 Sol · API: task score 93, 6.9× the cheapestSonnet 5.5 · API: task score 92, 36× the cheapestHaiku 5.5 · API: task score 86, 2.1× the cheapestGPT-6 Luna · API: task score 84, the cheapestGemini 4 Argon · API: task score 77, 6.7× the cheapestGemini 3.8 Flash · API: task score 60, 5.5× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 45, the cheapest, 1 critical failureGPT-6 Astra · ChatGPTOpus 5.5 · ClaudeGPT-6.1 Sol · APISonnet 5.5 · APIHaiku 5.5 · APIGPT-6 Luna · APIGemini 4 Argon · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costAt least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API841.0×
Gemini 3.5 Flash-Lite · Gemini451.1×
Haiku 5.5 · API862.1×
Gemini 3.8 Flash · API605.5×
Gemini 4 Argon · API776.7×
GPT-6.1 Sol · API936.9×
Sonnet 5.5 · API9236×
Opus 5.5 · Claude9337×
GPT-6 Astra · ChatGPT9439×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Turning strategy into a sequenced plan.

Why it matters

A roadmap is where strategy meets capacity. Dated feature lists turn guesses into promises.

What good looks like

  • Items are problems or outcomes, not just features
  • Sequencing reflects dependencies
  • Explicit trade-offs
  • Commitment falls with distance

Deliberately not measured

  • Gantt formatting
Capability tested

Sequencing under constraints

The failure we’re looking for

A dated wishlist sorted by excitement

Grading

Decision model and LLM judge, calibrated against a blind PM review