Tasks / Product Craft

Build a roadmap

Can the model sequence bets against capacity and dependencies, and explain the order?

Task type v1.0 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 69% were usable with at most a quick edit.

Reliably right

  1. Produces the required deliverable95% pass
    The roadmap is complete, in the right form for a Group PM, within the word limit, and actionable without major gaps.
    GPT-6.1 Sol · API · Fourteen requests from one customer
  2. Outcomes, with certainty that falls with distance94% pass
    Every item names its outcome (e.g., reduce paper transcription, reduce manual reconciliation), and near-term items are specific while later ones are deliberately looser (e.g., '7 weeks for rollout, workflow improvement and contingency').
    GPT-6.1 Sol · API · Fourteen requests from one customer
  3. Handles the big customer with evidence94% pass
    It uses Calder's rising usage and the second-hand 'might leave' to propose a conversation first, not committing the report builder, and offers a bounded export if needed.
    GPT-6.1 Sol · API · Fourteen requests from one customer

Where it slips

  1. Sees through the request count72% pass
    The output never counts requests by requester, does not mention that 14 requests come from Calder or that several are duplicates, and does not compare themes by customer count.
    GPT-6 Luna · API · Fourteen requests from one customer
  2. Uses the supplied evidence correctly72% pass
    States credit notes as 4 customers, but the request log contains 5 credit note tracking requests.
    Sonnet 5.5 · API · Fourteen requests from one customer
  3. Fits the stated capacity73% pass
    Q3 Expenses row lacks squad-week allocation, so the committed plan cannot be fully checked against the 11-week capacity.
    Opus 5.5 · Claude · A year of spend management, with a hard deadline

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Outcomes, with certainty that falls with distance

    Does each roadmap item say the problem or outcome it serves, and does precision (dates, scope) decrease for later items?

    Passes when Every item names its outcome or problem. Near-term items are specific and later ones are deliberately looser.

  2. Fits the stated capacity

    Does the committed plan fit the team capacity the brief gives, with sums a reader could check, and does it say what was cut or deferred to make it fit?

    Passes when The committed work adds up to no more than the stated capacity, with some slack, the sums can be checked, and it names what was cut or pushed later and why.

  3. Tells the story of the bets

    Does the roadmap group the work into a few themes and explain why those are the biggest things to invest in now, rather than presenting a scored or ranked list of projects?

    Passes when A handful of themes, each with the reason it matters most now, and the items sit under them.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Sequences around dependencies · and 3 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a Staff PM at Harbor. Tom Achebe, our CEO, has shared his draft roadmap for the next four quarters (Q4 2026 to Q3 2027), and asked you to turn it into the roadmap we take to next week's planning offsite. The exec team (CEO, CFO, CRO and CTO) will read it beforehand. Write: 1. The roadmap itself, by quarter or by now, next and later: what each squad works on and the outcome each item serves. 2. The case for it, in no more than 1,200 words: why this order, what you changed from Tom's draft and why, what we're not doing, and the risks that could change the plan. The pack below is everything we have. Not all of it matters equally.

What the model was given12 items: About Harbor, FY27 goal (approved by the board, September 2026), Squads and capacity, Hiring, Card processor deadline, Candidate work (estimates in squad-weeks, by owning squad), Tom's draft roadmap, Procurement evidence, Card spend evidence, Multi-entity evidence, Integrations evidence, Other asks
About HarborSpend management (corporate cards, expenses and approvals) for companies with 200 to 2,000 employees. 1,100 customers, $86M ARR, average 700 employees per customer. 58% of revenue is interchange on card spend; the rest is subscription. Net revenue retention (NRR) is 104%.
FY27 goal (approved by the board, September 2026)Raise NRR to 112% by the end of Q3 2027 by expanding inside existing customers: more spend on Harbor cards, more entities and more modules. New-logo growth matters, but it comes second this year.
Squads and capacityFive squads: Cards, Expenses, Approvals & Policy, Integrations and Platform. After support and on-call, each has about 11 squad-weeks of roadmap capacity a quarter. Squads can take work from another squad's area, but it takes them about 1.5× as long.
HiringTwo new squads were approved in September 2026. Tom's draft counts the first from Q1 2027 and the second from Q2 2027, both at full capacity. Last year, our three approved squads took 5, 6 and 8 months from approval to their first full sprint, and each delivered about half its capacity in its first quarter.
Card processor deadlineOur card processor retires its v1 API on 31 March 2027. Every card authorisation we process runs through v1 today. Migrating to v2 is about 18 squad-weeks of Cards work, and the processor must then run 4 weeks of certification testing before we can cut over. Certification is the processor's time, not ours, but nothing can change on our side while it runs.
Candidate work (estimates in squad-weeks, by owning squad)1. Processor v2 migration: Cards 18. Hard deadline above. 2. Virtual cards for software subscriptions: Cards 10. Needs v2. 3. Real-time spend controls (declines out-of-policy spend at the point of sale): Cards 8, Approvals & Policy 4. Needs v2. 4. Procurement module (purchase requests and approvals, a new paid add-on): Approvals & Policy 30, Integrations 6. 5. Multi-entity support (subsidiaries under one parent account): Platform 12, Approvals & Policy 10. 6. Sage Intacct integration: Integrations 9. 7. Microsoft Dynamics integration: Integrations 12. 8. SSO and SCIM provisioning: Platform 5. 9. Receipt-matching improvements: Expenses 6. 10. Mobile app rewrite: Expenses 22.
Tom's draft roadmapQ4 2026: Cards: v2 migration. Approvals & Policy: multi-entity. Integrations: Intacct. Platform: multi-entity. Expenses: receipt matching. Q1 2027: Cards: finish migration, start virtual cards. New squad A: procurement module. Integrations: Dynamics. Platform: SSO and SCIM. Q2 2027: Cards: virtual cards, spend controls. New squad A: procurement launch. New squad B: mobile rewrite. Q3 2027: Cards: spend controls. Everyone else: procurement adoption, mobile launch. Tom's note: “Procurement is our path to 112%. I've told the board it can add $6M of ARR in its first year.”
Procurement evidenceSix design partners have used a prototype since June. Two said they would pay for it; the other four already use a dedicated procurement tool and said they'd need a reason to switch. Proposed price: $8 per user per month, for users who raise purchase requests. At our customers, about 10% of employees raise purchase requests.
Card spend evidenceIn QBRs with our 60 largest customers, 26 asked for virtual cards for software subscriptions, and 31 CFOs asked for real-time spend controls. Finance estimates our customers pay about $410M a year of software subscriptions on other cards or by invoice (extrapolated from the 60 QBR accounts). Our net interchange is 1.1% of card spend.
Multi-entity evidence14 of our 60 largest customers have asked for it. Between them they have 52 subsidiaries not on Harbor, and 9 of the 14 have said in writing they would add their subsidiaries once it exists. Customer Success sized it at $2.9M ARR if all 52 joined at their parents' pricing.
Integrations evidence17% of customers use Sage Intacct through a CSV export. Their gross revenue churn is 13% a year, against 7% for customers on our NetSuite integration. Intacct was cited in 31% of lost new-logo deals last year, Dynamics in 8%. 3% of customers use Dynamics.
Other asksCRO: “Five enterprise deals worth $1.2M of pipeline need SSO and SCIM, and 11 existing accounts have it on their security review list.” CTO: “The mobile app is on a framework version that loses support in late 2027, and I want the rewrite done before then.” Support: receipt auto-match is at 71%, and unmatched receipts are the most common expense ticket.
What a strong answer doesThe answer key the graders mark against

A four-quarter roadmap that protects the 31 March deadline: the migration build has to finish by about early March to leave 4 weeks of certification, which is roughly all of Cards' capacity for Q4 and most of Q1 (18 of about 18 squad-weeks available before certification). So it adds help or buffer (Platform or another squad at 1.5×, or cutting everything else from Cards) and moves virtual cards out of Q1: both v2 features start only after cut-over (virtual cards Q2, spend controls Q2–Q3). It treats the new squads as upside: on past hiring they arrive around February to April 2027 and deliver half capacity in their first quarter, so nothing committed depends on them before Q2–Q3. It makes a clear call on procurement: the $6M claim needs about 890 accounts (81% of customers) paying $6,720 a year each (70 requesters × $8 × 12), and only 2 of 6 design partners would pay, so it replaces the 30-week build with a cheap paid test with a threshold, and keeps Approvals & Policy on multi-entity and spend controls, which have stronger expansion evidence ($2.9M sized, 9 of 14 committed in writing; about $1.5M a year of interchange from moving a third of the $410M of subscriptions onto virtual cards). Intacct comes early for its churn; Dynamics later or cut; SSO and SCIM are small and justified; the mobile rewrite waits for a new squad, before the framework's support ends. It lists what changed from Tom's draft, what's not being done, and the risks (certification slipping, hiring slipping), with commitment falling with distance.

Critical failures (cap the score at 40)
  • Leaves the processor migration finishing too late for 4 weeks of certification before 31 March 2027, or schedules virtual cards or real-time spend controls before it
Task

Staff-level · v1.6

The outputs

8 setups answered this brief.

Usable with a quick edit9 of 13 checks passed

The output is complete, evidence-based, and addresses the CEO's request; a product manager could use it with light wording adjustments, without re-checking facts or redoing the logic.

Output

Harbor Roadmap, Q4 2026 to Q3 2027

Q4 2026 (now)

SquadWork (squad-weeks)Outcome it serves
CardsProcessor v2 migration (11 of 18)Keeps every card authorisation running. Hard deadline: cut over before 31 Mar 2027.
ExpensesReceipt-matching (6); mobile rewrite starts (5)Auto-match from 71% up; fewer expense tickets. Mobile rewrite ships before framework support ends.
Approvals & PolicyMulti-entity (10)NRR: subsidiaries join existing accounts.
PlatformSSO/SCIM (5); multi-entity (6)$1.2M enterprise pipeline; 11 accounts in security review. NRR via multi-entity.
IntegrationsSage Intacct (9)Retention: Intacct customers churn at 13% vs 7% on NetSuite. Fewer lost new-logo deals.

Q1 2027

SquadWorkOutcome it serves
CardsFinish v2 (7; code-complete by 15 Feb). Processor certification (4 weeks, change freeze). Cut over before 31 Mar. Virtual cards start (4).Authorisations protected. NRR: interchange on subscription spend.
ExpensesMobile rewrite (11)Mobile ships in Q3, ahead of late-2027 EOL.
Approvals & PolicyReal-time spend controls (4); procurement build (7)Spend controls: 31 CFOs asked for them. Procurement: pricing gate at end of Q1.
PlatformMulti-entity finish (6); 5 sw held for onboarding and reliabilityNRR: multi-entity GA.
IntegrationsProcurement integration (6); Dynamics starts (5)Procurement build. Dynamics is last in queue.
New squad A (Approvals & Policy home)Onboarding; procurement build from ~half capacityProcurement build.

Q2 2027

SquadWorkOutcome it serves
CardsVirtual cards (6); spend controls (5)NRR: software subscription spend moves onto Harbor cards.
ExpensesMobile rewrite (6 to finish); betaMobile feature-complete.
Approvals & Policy + new squad AProcurement build (11 plus squad A at ~half)Procurement build; design-partner beta prep.
PlatformCapacity held for multi-entity subsidiary onboardingNRR: subsidiaries of the 14 requesting customers.
IntegrationsDynamics (7 to finish). Squad B onboarding.Low-priority retention and new-logo coverage.

Q3 2027

SquadWorkOutcome it serves
CardsSpend controls (3); virtual cards hardening; 8 sw spareNRR: spend controls and virtual cards at scale.
ExpensesMobile launchMobile ships before EOL.
Approvals & Policy + new squad AProcurement build complete (base case ~August); design-partner paid pilots (September)Procurement: first paid pilots. Little or no FY27 ARR.
PlatformMulti-entity subsidiaries joinNRR.
IntegrationsDynamics complete; squad B at full capacity; next item chosen at Q3 reviewCoverage.

Later (Q4 2027 onward, gated): Procurement paid launch, only if the Q1 pricing gate is passed. Most procurement ARR lands in FY28.

---

The Case

The headline: 112% is not reachable in FY27

NRR is 104%. Reaching 112% by Q3 2027 means about $6.9M of additional net expansion ARR on our $86M base. My estimates for what this plan delivers inside the window:

  • Multi-entity: $1.2–1.9M. The ceiling is $2.9M if all 52 subsidiaries join. Nine of 14 customers have committed in writing, which is not the same as signed.
  • Virtual cards and spend controls: $0.4–0.7M. Interchange is 1.1%, so $410M of subscription spend is worth $4.5M at full capture. I assume 10–15% capture in-year. The $410M is extrapolated from 60 accounts.
  • Intacct churn reduction: $0.4–0.9M. This assumes Intacct-using ARR is roughly 17% of the base (~$15M) and that the 6-point churn gap is caused by the CSV workflow, not by the customers themselves.
  • Procurement: near zero.

That totals $2.0–3.5M, which takes NRR to roughly 106–108%. Procurement is the only item big enough to close the gap, and it cannot deliver inside FY27. Tom's note says procurement can add $6M in its first year. $6M is about 80% of the theoretical ceiling (1,100 customers × 70 requesters × $96 a year ≈ $7.4M), and only 2 of 6 design partners said they would pay. Even if the first-year number holds, the timing means it lands in FY28.

The board needs this correction before the offsite ends. Tom should brief the chair before the board sees a revised NRR path. I suggest we present 106–108% for FY27 with a path to 112% in FY28 and set out what would have to be true to close the gap.

Why this order

  1. Protect the base first. The v2 migration is the only hard deadline. Every card authorisation depends on it. Cards is planned with about two weeks of slack: migration complete by 15 Feb, 4 weeks of certification, cut-over by mid-March. The Cards squad is ring-fenced until cut-over, and I want a go/no-go on 15 Feb.
  2. Put the cheap, high-demand items first. SSO/SCIM is 5 squad-weeks and unlocks $1.2M of pipeline and 11 security reviews. Multi-entity is the largest in-year expansion lever and has written demand. Intacct is 9 squad-weeks against a 13% churn rate on 17% of customers.
  3. Use squads that exist. Expenses can build the mobile rewrite with its own capacity by Q2. That ships before the framework's late-2027 end of support without waiting for a new squad.
  4. Gate procurement. The build is 30 squad-weeks of Approvals & Policy work on a product where only a third of design partners have said they would pay. We build it, but the pricing decision comes at the end of Q1.

What I changed from Tom's draft, and why

Tom's draftOur planWhy
New squad A: procurement from Q1; launch in Q2.Procurement built by existing Approvals & Policy plus squad A; base-case build complete ~August; paid pilots in September.30 squad-weeks cannot be built and launched in Q2 by a squad that is still ramping.
New squads at full capacity from Q1 and Q2.Modelled at half in their first quarter.Last year's squads took 5–8 months to first full sprint and delivered half capacity in their first quarter. Tom's draft assumes about 55 squad-weeks from new squads in Q1–Q3. The realistic best case is about 44.
Mobile rewrite assigned to new squad B in Q2.Mobile rewrite in Expenses from Q4; launch in Q3.Expenses has the capacity now. Squad B is freed for Integrations.
Platform carries 12 squad-weeks of multi-entity in Q4 (above its 11 capacity); SSO in Q1.SSO and 6 weeks of multi-entity in Q4; multi-entity finishes in Q1.SSO moves forward to unlock $1.2M of pipeline. Multi-entity still reaches GA by end of Q1.
Dynamics in Q1.Dynamics runs in spare Integrations capacity from Q1 to Q2, and is the first item to cut.Dynamics is used by 3% of customers and was cited in 8% of lost deals. Intacct, at 17% of customers and 31% of lost deals, comes first.
Cards: finish migration in Q1 and start virtual cards.Same sequence, with a 15 February checkpoint.Tom's sequence was right. The checkpoint protects the certification window.

What we're not doing

  • Procurement at full scope before the Q1 pricing gate. If fewer than three of six design partners commit to a paid pilot at $8 per user, we cut scope and move capacity to other work.
  • Dynamics as a committed FY27 item. It fits in spare capacity. It does not get priority over anything else.
  • Mobile in a new squad. Expenses owns it.
  • Platform-led procurement work. Platform has about 16 spare squad-weeks in Q1–Q2 and could absorb procurement sub-components at 1.5× cost if the gate passes. We don't plan that yet.
  • Any NRR commitment above 108% for FY27 in board materials.

Risks that could change the plan

  1. The v2 migration slips past the certification window. This is the existential risk. If migration is not code-complete by about 22 February, we miss 31 March. Mitigations: ring-fence Cards from January, book a certification slot with the processor now, and agree a contingency with the processor in writing.
  2. New squads ramp slower than planned. Each month of delay pushes multi-entity onboarding, procurement and Dynamics later. Procurement is most exposed because Approvals & Policy is already at zero slack.
  3. Procurement demand is weaker than the evidence suggests. Two of six design partners would pay, and four already use a dedicated tool. The Q1 gate exists for this reason.
  4. Virtual card and interchange sizing is soft. The $410M is extrapolated from 60 accounts. Capture is an assumption.
  5. Intacct churn may not be caused by the CSV workflow. If the churn gap is about customer type rather than integration, the retention benefit is smaller.
  6. Multi-entity commitments are not contracts. Nine written commitments are real signals, but pricing for subsidiaries still needs to be set and sold.
  7. Mobile slips past Q3. Expenses has about 5 spare squad-weeks in Q2 and Q3 for launch fixes. A slip beyond Q3 puts us close to the framework's end of support.

Decisions needed at the offsite

  1. Approve the revised FY27 NRR range (106–108%) and agree how Tom briefs the board.
  2. Confirm the 15 February v2 checkpoint and the Cards ring-fence.
  3. Approve the procurement build with the Q1 pricing gate and a pre-agreed cut scope.
  4. Move mobile into Expenses and release squad B to Integrations.

Grades and run details

Decision model 69 · LLM judge 93
Decision model checks
  • failUses the supplied evidence correctly9%
  • passAddresses the actual decision92%
  • partialRespects explicit constraints18%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims19%
  • passProduces the required deliverable53%
  • partialProtects the processor deadline37%
  • passPlans on the squads we actually have49%
  • passMakes the call on procurement88%
  • passOutcomes, with certainty that falls with distance86%
  • failFits the stated capacity22%
  • failSequences around dependencies33%
  • passTells the story of the bets46%
Run
Run
#1
API response time
1.8 min
Submitted
8 Oct 2026
Critical failure: Leaves the processor migration finishing too late for 4 weeks of certification before 31 March 2027, or schedules virtual cards or real-time spend controls before it. Score capped at 40.

Needs more than a quick edit5 of 13 checks passed

The roadmap has capacity overruns, a critical dependency violation, and a miscalculated procurement case. A PM would need to redo the capacity planning and procurement analysis.

Output

Harbor FY27 Product Roadmap

Prepared by: Staff PM, Harbor For: Executive Team Review (CEO, CFO, CRO, CTO)

---

Part 1: The FY27 Roadmap (Q4 2026 – Q3 2027)

Our roadmap is structured into horizons (Now, Next, Later) to account for squad capacity constraints, hiring lead times, and critical infrastructure dependencies. Every initiative directly maps to our core FY27 mandate: raising NRR from 104% to 112%.

NOW (Q4 2026)

Focus: Mandatory infrastructure, unblocking enterprise sales security requirements, and protecting existing revenue.

  • Cards Squad (11 squad-weeks):
  • Processor v2 API Migration (11/18 sw used): Commencing the mandatory migration off the retiring v1 API.
  • Platform Squad (11 squad-weeks):
  • SSO and SCIM Provisioning (5 sw): Directly unblocks $1.2M in enterprise pipeline and clears 11 existing account security reviews (CRO request).
  • Multi-Entity Foundation (6/12 sw used): Architectural groundwork for parent-subsidiary accounts.
  • Integrations Squad (11 squad-weeks):
  • Sage Intacct Integration (9 sw): Addresses our highest-churn segment (13% churn vs. 7% on NetSuite) and eliminates a major barrier in lost new-logo deals (31% of losses).
  • Spillover buffer (2 sw).
  • Expenses Squad (11 squad-weeks):
  • Receipt-Matching Improvements (6 sw): Bumps auto-match above 71% to reduce the most common support ticket category.
  • Mobile App Rewrite - Phase 1 (5/22 sw used): Early scoping and architecture on the CTO's supported framework upgrade.
  • Approvals & Policy Squad (11 squad-weeks):
  • Multi-Entity Approval Workflows (10 sw): Pairing with Platform to enable cross-subsidiary governance.
  • Spillover buffer (1 sw).

---

NEXT (Q1 2027)

Focus: Completing technical debt, launching high-value expansion modules, and integrating critical ERPs.

  • Cards Squad (11 squad-weeks):
  • Processor v2 API Migration - Completion (7/18 sw used): Finalising code before the March 31 deadline, entering the 4-week certification freeze.
  • Virtual Cards for Software Subscriptions - Initiation (4/10 sw used).
  • Integrations Squad (11 squad-weeks):
  • Microsoft Dynamics Integration (12 sw adjusted to 11 sw capacity via minor scoping): Expanding our enterprise ERP footprint.
  • Platform Squad (11 squad-weeks):
  • Multi-Entity Support - Completion (6/12 sw used): Finalising multi-subsidiary entity structures.
  • Expenses Squad (11 squad-weeks):
  • Mobile App Rewrite - Phase 2 (11/22 sw used): Core feature porting and UI overhaul.
  • New Squad A (Hired Dec 2026, delivering at ~50% capacity = ~5.5 sw):
  • Procurement Module - Phase 1 (5.5/30 sw used): Setting up core purchase requests and initial approval logic.

(Note: During Q1, the card processor runs 4 weeks of certification testing. No changes can be deployed to card flows during this time).

---

LATER (Q2 – Q3 2027)

Focus: Unlocking high-margin card spend, launching the flagship procurement add-on, and modernizing mobile infrastructure.

  • Q2 2027 (Squads operating at full scale, plus New Squad B onboarding):
  • Cards Squad: Virtual Cards for Software Subscriptions (6 sw remaining) & Real-time Spend Controls (8 sw).
  • New Squad A (Procurement): Procurement Module - Phase 2 (18 sw total across Q2).
  • Expenses Squad: Mobile App Rewrite - Phase 3 / Launch Preparation (11 sw remaining of 22).
  • New Squad B (Hiring initiated Q1, arriving Q2, delivering ~5.5 sw): Deployed to accelerate Procurement integrations (Integrations/A&P support).
  • Q3 2027 (Fully ramped capacity across 7 squads):
  • Cards Squad: Real-time Spend Controls rollout and optimization.
  • New Squad A & B: Procurement Module Launch, GA, and initial adoption drives.
  • Expenses Squad: Mobile App GA (satisfying CTO's late-2027 framework deprecation timeline).
  • Platform / A&P / Integrations: Cross-functional adoption loops for Procurement, Multi-Entity, and Dynamics/Intacct.

---

Part 2: The Case for the Roadmap

1. Why This Order? (Balancing Reality and Ambition)

Our mandate is clear: raise NRR to 112% by the end of Q3 2027. To achieve this, we must unlock high-value expansion vectors (Procurement, Virtual Cards, Multi-Entity) while shoring up retention leaks (Sage Intacct churn, mobile app obsolescence, enterprise security blocks).

However, ambition must bow to physics. Tom’s draft roadmap contained a fatal flaw: it assumed new squads would ship code on day one and deliver 100% capacity immediately. Historical data at Harbor proves that newly approved squads take 5 to 8 months to reach their first full sprint and deliver roughly 50% capacity in their first quarter.

Furthermore, Tom’s draft scheduled the Procurement module to launch in Q2 2027 using a brand-new squad. Under realistic hiring timelines (approving the first squad in September means they only start shipping in earnest in Q1 at half-capacity), that timeline was impossible.

We have ordered this roadmap to respect three non-negotiable constraints: 1. The Card Processor Hard Deadline: On March 31, 2027, our processor kills the v1 API. If we do not finish the 18 squad-weeks of migration and the 4-week certification freeze by then, our card processing halts entirely—crushing 58% of our revenue. 2. Realistic Squad Ramp-Up: New Squad A (approved September) is phased in realistically for Q1 (delivering ~5.5 squad-weeks) and hits full stride in Q2. New Squad B is phased in for Q2. 3. Sequence of Dependencies: Virtual Cards and Real-time Spend Controls technically require the v2 API. Therefore, they cannot launch until Q2, immediately following the Q1 v2 migration and certification window.

2. What We Changed from Tom’s Draft and Why

  • Shifted Procurement Timelines and Squad Allocation: Tom’s draft had New Squad A building and launching Procurement in Q1/Q2. Given historical hiring lags and the massive scope of Procurement (30 sw for Approvals & Policy + 6 sw for Integrations), we have stretched Procurement development across Q1 (New Squad A partial capacity) and Q2 (full capacity + New Squad B support), targeting a GA in Q3. Trying to rush it in Q2 would result in a broken launch or squad burn-out.
  • Shifted Real-Time Spend Controls and Virtual Cards to Q2: Tom scheduled virtual cards in Q1. Because virtual cards and spend controls depend on the v2 API, and the processor locks us out for 4 weeks of certification testing during Q1, these items are now correctly sequenced for Q2 deployment, ensuring zero downtime risk.
  • Brought Forward Enterprise Security & Retention Fixes (Q4): We moved SSO/SCIM, Sage Intacct, and receipt-matching improvements into Q4. Why? Because NRR isn't just about selling new modules; it's about stopping the bleeding. Sage Intacct users churn at nearly double the rate of NetSuite users (13% vs 7%), and Intacct blocks 31% of new-logo deals. Fixing this now preserves our base. Similarly, securing $1.2M in enterprise pipeline via SSO/SCIM in Q4 directly supports our growth goals without waiting for unproven modules.

3. What We Are Not Doing (And Why)

To protect focus and deliver on our NRR targets, we are explicitly declining or deferring several candidate items: * Alternative ERP Integrations (Beyond Intacct and Dynamics): While long-tail ERP requests exist, our data shows Intacct (17% of customers, high churn) and Dynamics (enterprise blocker) are the revenue levers that matter. Other ERPs are deferred to FY28. * Ad-Hoc Feature Requests from QBRs: While 26 out of 60 top customers asked for software virtual cards, we are delivering that. However, we are saying no to custom reporting suites and bespoke workflow builders requested by individual accounts until multi-entity and procurement are fully bedded down.

4. Risks That Could Change the Plan

1. Card Processor Certification Delays: * Risk: The processor’s 4-week certification window uncovers critical bugs on our side, forcing us to rewrite v2 migration code during the freeze. * Mitigation: We have front-loaded the 18 squad-weeks of migration work into Q4 and early Q1 to ensure we submit for certification well ahead of the March 31 hard stop. 2. Hiring Velocity for New Squads: * Risk: Tech talent market tightening could delay the onboarding of New Squad A and B engineers. * Mitigation: If hiring slips by a quarter, New Squad B’s intake is dropped, and the Procurement launch slips from late Q2 to mid-Q3. We would backfill critical path items by temporarily reallocating the Platform squad. 3. Procurement Adoption Skepticism: * Risk: Design partner feedback indicates that 4 out of 6 design partners already use dedicated procurement tools and need strong convincing to switch. Tom’s assumption of $6M ARR in Year 1 may be aggressive if attach rates lag. * Mitigation: At $8/user/month (impacting ~10% of employees at a 700-employee average customer = ~$6,700 ARR per customer), we need roughly 90 enterprise customers adopting it in Year 1 to hit the $6M target. Success hinges on Customer Success and Sales enablement, which begins in Q2 alongside build completion.

Grades and run details

Decision model 35 · LLM judge 50
Decision model checks
  • failUses the supplied evidence correctly9%
  • partialAddresses the actual decision31%
  • failRespects explicit constraints83%
  • passIdentifies material uncertainty29%
  • partialAvoids unsupported claims24%
  • partialProduces the required deliverable53%
  • failProtects the processor deadline49%
  • partialPlans on the squads we actually have61%
  • failMakes the call on procurement71%
  • partialOutcomes, with certainty that falls with distance48%
  • failFits the stated capacity49%
  • failSequences around dependencies80%
  • passTells the story of the bets72%
Run
Run
#1
Time to output
10 s
Submitted
1 Oct 2026

Needs more than a quick edit7 of 13 checks passed

The Integrations capacity gap in Q1 and the missing procurement threshold would require rework before a PM could use it.

Output

Part 1: Product Roadmap (Q4 2026 – Q3 2027)

``` ==================================================================================================== SQUAD Q4 2026 Q1 2027 Q2 2027 Q3 2027 ==================================================================================================== Cards Processor v2 Migration Finish v2 (Wks 1-7) Virtual Cards for SaaS Real-Time Controls (11 wks) API Freeze/Cert (W8-11) (6 wks remaining) (Cards: 3 wks) Virtual Cards Prep Real-Time Controls Interchange Optimization (4 wks) (Cards: 5 wks) & Scaling (8 wks) ---------------------------------------------------------------------------------------------------- Platform Multi-Entity: Core Multi-Entity: Wrap-up Platform Scaling & Cross-Entity Reporting Data Architecture (1 wk) Org Hierarchies & Enterprise Admin (11 wks) SSO & SCIM (5 wks) (11 wks) (11 wks) Hardening (5 wks) ---------------------------------------------------------------------------------------------------- Approvals & Multi-Entity: Policy Multi-Entity: Release Real-Time Controls: Procurement Discovery Policy & Approval Routing (2 wks) Policy Engine (4 wks) & Scoping (MVP) (10 wks) A&P Rule Engine Complex Approval Rules (Approvals: 11 wks) Buffer (1 wk) Upgrades (9 wks) (7 wks) ---------------------------------------------------------------------------------------------------- Integrations Sage Intacct Native Sage Intacct GA (2 wks) MS Dynamics Integration MS Dynamics GA (3 wks) Integration MS Dynamics Integration (Integrations: 10 wks) ERP Webhooks & Custom (9 wks) + Buffer (2 wk) (Integrations: 2 wks) Sync (8 wks) ---------------------------------------------------------------------------------------------------- Expenses Receipt Auto-Match Mobile App Rewrite: Mobile App Rewrite: Mobile App GA & Optimization (6 wks) Phase 1 Core Workflows Phase 2 Parity/Release Expense Policy Polish Mobile Prep (5 wks) (11 wks) (11 wks) (11 wks) ---------------------------------------------------------------------------------------------------- New Squad A [Hiring / Onboarding] [Ramp / Training] Procurement Foundation Procurement Workflows (A&P Focus) (Ramped: ~5.5 wks) (Full capacity: 11 wks) ---------------------------------------------------------------------------------------------------- New Squad B [Hiring / Pipeline] [Hiring] [Onboarding / Ramp] Integrations / Core (Ecosystem Focus) (Ramped: ~5.5 wks) ==================================================================================================== ```

Squad-by-Squad Outcomes

  • Cards:
  • Processor v2 Migration: Protects 100% of Harbor’s card-authorisation volume ($49.9M existing ARR) against the hard 31 March 2027 processor shutoff; unlocks v2 capabilities required for virtual cards and real-time controls.
  • Virtual Cards for Software Subscriptions: Captures existing customer SaaS spend currently paid via invoice/other cards ($410M addressable pool), directly driving interchange revenue to expand NRR.
  • Real-Time Spend Controls: Enables 31 top-tier CFO accounts to enforce budgets at point-of-sale, unlocking enterprise card issuance and shifting unmanaged invoice spend to Harbor cards.
  • Platform:
  • Multi-Entity Architecture: Unlocks $2.9M contracted expansion ARR across 52 non-Harbor subsidiaries of top accounts.
  • SSO & SCIM Provisioning: Unblocks $1.2M in enterprise pipeline and closes compliance gaps for 11 critical renewal accounts.
  • Approvals & Policy:
  • Multi-Entity Approvals: Delivers cross-subsidiary approval chains required to close existing multi-entity commitments.
  • Real-Time Policy Engine: Co-delivers immediate POS budget validation with the Cards squad.
  • Procurement Discovery & MVP Architecture: Reframes procurement around verified buyer demand without starving committed delivery.
  • Integrations:
  • Sage Intacct Integration: Halves the 13% annual churn rate of the 17% of Harbor customers on Intacct (retaining ~$875k ARR annually) and addresses Harbor's #1 new-logo loss reason (31% of lost deals).
  • Microsoft Dynamics Integration: Expands ERP coverage for mid-market accounts, removing friction for 3% of existing customers and 8% of pipeline.
  • Expenses:
  • Receipt-Matching Improvements: Drives match rate above 71%, eliminating Harbor’s highest-volume support ticket category and improving user retention.
  • Mobile App Rewrite: Eliminates framework deprecation risk ahead of the late-2027 vendor cutoff without stealing capacity from revenue-generating squads.

---

Part 2: The Case for the Roadmap

Executive Summary

To achieve our board-approved target of 112% NRR by Q3 2027 ($86M ARR base expanding by ~$6.9M), Harbor must focus on the highest-probability, shortest-time-to-value expansion levers within our current 1,100 customers: multi-entity expansion ($2.9M ARR), capturing existing software spend via virtual cards ($1.3M–$2.2M ARR), and plugging Sage Intacct churn ($875k ARR retained).

Tom’s draft relied on an unbuilt Procurement module to generate $6M ARR in FY27. Ground-level capacity, vendor deadlines, and design partner data show that strategy carries severe revenue and operational risk. The proposed roadmap secures our existential card processor deadline, resolves major customer expansion requests, and establishes realistic hiring and product timelines.

---

What Changed from Tom’s Draft and Why

#### 1. Realignment on Procurement ($6M ARR Myth vs. Reality) * The Math: Harbor has 1,100 customers averaging 700 employees (770,000 total users). With 10% raising purchase requests (70 users/account) at $8/user/month ($96/year), an adopting customer generates $6,720 ARR. To generate $6M ARR, 893 out of 1,100 customers (81% of our entire base) would have to buy and deploy a v1 module in under 9 months. * The Customer Signal: 4 out of 6 design partners already use dedicated tools (e.g., Coupa, Zip) and explicitly stated they will not switch. Only 2 of 6 expressed willingness to pay. * The Decision: Procurement cannot carry our FY27 NRR goal. We have reallocated primary capacity to guaranteed expansion drivers (Multi-Entity and Virtual Cards), while initiating a structured discovery and phased MVP build with Squad A.

#### 2. Realistic Hiring Capacity * Tom’s Assumption: Squad A operates at 100% capacity in Q1 2027 (11 squad-weeks) and Squad B in Q2 2027 (11 squad-weeks). * Historical Reality: Approvals to first full sprint took 5–8 months historically, delivering ~50% capacity in quarter one. Squads approved in September 2026 will not reach operational productivity until late Q1 / early Q2 2027. * The Decision: No critical-path milestone or revenue target in Q4 or Q1 relies on unhired squads. Existing squads carry core commitments; Squad A ramps on Procurement foundations in Q2 (delivering ~5.5 squad-weeks) and reaches full delivery in Q3.

#### 3. Strict Compliance with the 31 March 2027 Processor Deadline * The Constraint: The v1 API deprecates on 31 March 2027. Migrating takes 18 squad-weeks plus a mandatory 4-week processor certification freeze during which our codebase cannot change. * Execution Window: The 4-week certification must run in March 2027. Cards code complete must occur by Week 7 of Q1 2027. * The Allocation: Cards dedicates all 11 squad-weeks in Q4 2026 and 7 squad-weeks in Q1 2027 (18 weeks total). During the 4-week March freeze, the Cards squad transitions to architectural prep for Virtual Cards, avoiding wasted idle time.

#### 4. Resequencing Multi-Entity, SSO/SCIM, and Sage Intacct * Multi-Entity (Platform + Approvals): Prioritised immediately in Q4. 14 of our 60 largest accounts requested this; 9 have signed written commitments to add 52 subsidiaries ($2.9M ARR). Shipping in early Q1 allows two full quarters of rollout and revenue recognition. * Sage Intacct (Integrations): 17% of our customers use Intacct via CSV, churning at 13% vs. NetSuite’s 7%. Cutting churn across these 187 accounts ($14.6M ARR base) saves ~$875k in ARR retention while eliminating our top sales loss reason (31% of lost deals). * SSO & SCIM (Platform): Sequenced for Q1 immediately following core Multi-Entity work. For a 5-week lift, this unlocks $1.2M in stalled enterprise pipeline and secures 11 major renewals.

---

How this Roadmap Hits 112% NRR

By the end of Q3 2027, this roadmap expands our $86M base by $7.0M+ in recurring expansion and retained ARR, exceeding the board's 112% target ($6.88M incremental):

InitiativeMechanismTarget ARR Impact
:---:---:---
Multi-Entity52 parent-account subsidiaries onboarding+$2.9M ARR
Virtual Cards (SaaS)Capturing 35% of $410M unmanaged SaaS spend @ 1.1% net interchange+$1.58M ARR
Real-Time Spend ControlsUnlocking enterprise card adoption across 31 QBR accounts+$0.80M ARR
Sage Intacct IntegrationEliminating 6% excess churn across 187 customer accounts+$0.88M ARR
SSO & SCIMEnterprise expansion & churn protection across 11 key accounts+$0.90M ARR
Total Net Expansion+$7.06M ARR (~112.2% NRR)

---

What We Are Not Doing (And Why)

  1. Not building Procurement as a hasty monolith in Q1: Building a broad module against negative design-partner feedback on an imaginary timeline would burn 36 squad-weeks with minimal revenue return. We are building the data model and workflows methodically using Squad A once ramped.
  2. Not assigning the Mobile Rewrite to an unhired squad: Tom assigned this 22-week project to Squad B in Q2. Squad B will not be staffed in time. Instead, the Expenses squad owns mobile: they finish Receipt Matching in Q4 (improving our #1 support issue) and execute the mobile rewrite across Q1 and Q2, completing it comfortably before late-2027 framework retirement.
  3. Not prioritising Microsoft Dynamics over Sage Intacct: Dynamics accounts for only 3% of our customer base and 8% of lost deals (vs. Intacct’s 17% base and 31% deal loss). Dynamics is scheduled for Q2 after Intacct is fully deployed.
  4. No cross-domain squad thrashing: Squads working out of domain suffer a 1.5× productivity penalty. We kept all squads focused within their domain expertise rather than forcing Expenses or Integrations to build Cards or Approvals features.

---

Risks and Mitigation Strategies

1. Card Processor Certification Delay: * Risk: Processor testing slips past 4 weeks, threatening the 31 March v1 shutoff. * Mitigation: We complete Harbor development by Week 7 of Q1 (mid-February), yielding a two-week safety buffer before the freeze. Cards works closely with processor engineers weekly starting in Q4. 2. Multi-Entity Delivery Bottlenecks: * Risk: Multi-Entity touches both Platform (12 wks) and Approvals & Policy (10 wks); cross-squad misalignment could delay Q1 rollout. * Mitigation: The technical spec and API contract between Platform and A&P will be finalized in Sprint 1 of Q4, enabling independent parallel development. 3. Hiring Lags for Squad A and B: * Risk: Talent acquisition takes longer than the historical 5–8 months. * Mitigation: The baseline revenue plan (Multi-Entity, Virtual Cards, Intacct, SSO) relies entirely on our five existing squads. Squad A’s onboarding pace dictates the launch timing of Procurement, not the attainment of our 112% NRR target.

Grades and run details

Decision model 69 · LLM judge 64
Decision model checks
  • failUses the supplied evidence correctly58%
  • passAddresses the actual decision79%
  • partialRespects explicit constraints19%
  • passIdentifies material uncertainty28%
  • failAvoids unsupported claims34%
  • passProduces the required deliverable45%
  • passProtects the processor deadline42%
  • passPlans on the squads we actually have95%
  • partialMakes the call on procurement77%
  • passOutcomes, with certainty that falls with distance26%
  • failFits the stated capacity49%
  • passSequences around dependencies16%
  • passTells the story of the bets75%
Run
Run
#1
API response time
45 s
Submitted
1 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 9

Addresses the actual decisionRightRightMixed
Haiku 5.5 · API

The output commits to a clear roadmap and a specific NRR range (106–108%), frames it for the exec team, and states what would change the call (pricing gate, migration checkpoint).

Gemini 3.5 Flash-Lite · Gemini

Commits to a clear roadmap and case, and states what would change the plan (risks).

Gemini 3.8 Flash · API

The output commits to a roadmap but does not explicitly state what result or condition would change that decision (e.g., a threshold for procurement MVP to trigger full build).

Respects explicit constraintsRightMixedRight
Haiku 5.5 · API

The requested deliverable (roadmap by quarter plus case ≤1,200 words) is present, and the roadmap enforces the processor deadline, capacity limits, and hiring histories.

Gemini 3.5 Flash-Lite · Gemini

Delivers a roadmap by now/next/later and a case under 1,200 words, as requested.

Gemini 3.8 Flash · API

The roadmap and case are present, within 1,200 words, and address the exec team as reader.

Avoids unsupported claimsRightWrongMixed
Haiku 5.5 · API

Causes and forecasts are explicitly labelled as assumptions (e.g., 'I assume 10–15% capture', 'This assumes the churn gap is caused by CSV'), and confident claims are based on supplied evidence.

Gemini 3.5 Flash-Lite · Gemini

Presents the forecast 'we need roughly 90 enterprise customers' as fact, but the correct arithmetic is ~893, making the claim unsupported.

Gemini 3.8 Flash · API

Forecasts are presented as targets, not established facts, and conclusions are based on supplied evidence.

Protects the processor deadlineRightWrongRight
Haiku 5.5 · API

The migration is code-complete by 15 Feb, leaves 4 weeks for certification, cut-over before 31 Mar, and virtual cards/spend controls start only after cut-over.

Gemini 3.5 Flash-Lite · Gemini

Starts virtual cards in Q1 before the v2 cut-over, and provides no buffer or help for the migration, risking the deadline.

Gemini 3.8 Flash · API

Migration finishes by Week 7 of Q1, leaving 4 weeks of certification before 31 March, and virtual cards/spend controls start in Q2 after cut-over.

Plans on the squads we actually haveRightWrongRight
Haiku 5.5 · API

New squads are modelled at half capacity in their first quarter, consistent with the hiring history; no critical commitments depend on them at full capacity until Q2–Q3.

Gemini 3.5 Flash-Lite · Gemini

Commits the procurement module to a new squad in Q1, relying on an unhired squad for important work before they are proven.

Gemini 3.8 Flash · API

Uses historical hiring data to place new squads at half capacity in Q2/Q3, and commits no critical-path work to them before then.

Makes the call on procurementRightWrongWrong
Haiku 5.5 · API

The output checks the $6M claim against pricing (implies ~80% adoption) and design partner evidence (2 of 6 would pay), commits Approvals & Policy to higher-evidence work, and proposes a gated build with a paid-pilot threshold before full launch.

Gemini 3.5 Flash-Lite · Gemini

Does not correctly check the $6M claim (miscalculates 90 customers instead of ~893), does not propose a smaller paid test, and does not commit Approvals & Policy to better-evidenced work.

Gemini 3.8 Flash · API

It checks the $6M claim and commits to discovery/MVP, but does not propose a smaller test with a clear threshold that would justify the full build.

Fits the stated capacityMixedWrongWrong
Haiku 5.5 · API

The committed work sums to ≤11 squad-weeks per squad per quarter, with visible slack in some quarters; the case explicitly lists what was cut or deferred (Dynamics, full-scope procurement, mobile in new squad).

Gemini 3.5 Flash-Lite · Gemini

Q2 Cards squad is over capacity (14 sw vs 11), New Squad A is over capacity (18 sw vs 11), and the required Approvals & Policy work for spend controls is missing.

Gemini 3.8 Flash · API

The Integrations squad in Q1 is allocated only 4 weeks (Sage Intacct GA 2 wks, MS Dynamics 2 wks) out of 11, leaving a capacity gap.

Sequences around dependenciesMixedWrongRight
Haiku 5.5 · API

v2 features only appear after migration and certification; mobile rewrite finishes before framework EOL; new squads are used only after ramping; the roadmap names the v2 dependency and hiring constraint.

Gemini 3.5 Flash-Lite · Gemini

Virtual cards start before the v2 cut-over, and the spend-controls dependency on Approvals & Policy is not respected.

Gemini 3.8 Flash · API

All dependencies (v2 before virtual cards/controls, multi-entity before cross-entity, Intacct before Dynamics) are respected and named.

Tells the story of the betsMixedRightMixed
Haiku 5.5 · API

The roadmap is presented as a squad-by-quarter table, not grouped into a few overarching themes; the case explains the bets but the roadmap itself is a list of projects.

Gemini 3.5 Flash-Lite · Gemini

Groups work into Now/Next/Later with clear themes and rationale.

Gemini 3.8 Flash · API

The roadmap is presented as a squad-by-quarter grid, not grouped into a few themes with the reason each matters most now.

All mixed 1

Uses the supplied evidence correctlyMixedMixedMixed
Haiku 5.5 · API

All statements about the current situation (numbers, dates, customer feedback) are drawn directly from the supplied context or derived from it by arithmetic.

Gemini 3.5 Flash-Lite · Gemini

All factual statements about the current situation are taken directly from the supplied context.

Gemini 3.8 Flash · API

All claims about the current situation are directly from the brief or supplied context.

All got right 3

Identifies material uncertaintyRightRightRight
Haiku 5.5 · API

The case identifies specific unknowns (migration slip, squad ramp speed, procurement demand, interchange capture) and how they would be resolved or would change the plan.

Gemini 3.5 Flash-Lite · Gemini

Names specific risks (certification delays, hiring velocity, procurement adoption) and how they would be resolved.

Gemini 3.8 Flash · API

It names specific risks (certification delay, multi-entity bottlenecks, hiring lags) and mitigation strategies.

Produces the required deliverableRightRightRight
Haiku 5.5 · API

The roadmap table and case together form a complete, actionable packet within the word limit that the exec team could read and act on.

Gemini 3.5 Flash-Lite · Gemini

The roadmap and case are present, complete, and usable for the exec team.

Gemini 3.8 Flash · API

The roadmap table and case are complete and usable for the planning offsite.

Outcomes, with certainty that falls with distanceRightRightRight
Haiku 5.5 · API

Every roadmap item names the problem or outcome it serves, and later items (e.g., 'virtual cards at scale', 'first paid pilots') are deliberately looser than near-term ones.

Gemini 3.5 Flash-Lite · Gemini

Every item names an outcome, and later items are deliberately looser.

Gemini 3.8 Flash · API

Every item names its outcome, and later items are deliberately looser (e.g., 'Cross-Entity Reporting & Enterprise Admin').

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 79% of 150 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6.1 SolwithAPI100.096.42None
2GPT-6 AstrawithChatGPT93.9100.02None
3Opus 5.5withClaude90.296.42None
4Sonnet 5.5withAPI94.192.62None
5GPT-6 LunawithAPI92.085.42None
6Haiku 5.5withAPI74.296.42None
7Gemini 3.8 FlashwithAPI74.266.82None
8Gemini 3.5 Flash-LitewithGemini56.936.521 capped

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6.1 Sol · API: task score 98, 6.9× the cheapestGPT-6 Astra · ChatGPT: task score 97, 39× the cheapestOpus 5.5 · Claude: task score 94, 37× the cheapestSonnet 5.5 · API: task score 94, 36× the cheapestGPT-6 Luna · API: task score 87, the cheapestHaiku 5.5 · API: task score 86, 2.1× the cheapestGemini 3.8 Flash · API: task score 68, 5.5× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 44, the cheapest, 1 critical failureGPT-6.1 Sol · APIGPT-6 Astra · ChatGPTOpus 5.5 · ClaudeSonnet 5.5 · APIGPT-6 Luna · APIHaiku 5.5 · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and costAt least one critical failure
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API871.0×
Gemini 3.5 Flash-Lite · Gemini441.1×
Haiku 5.5 · API862.1×
Gemini 3.8 Flash · API685.5×
GPT-6.1 Sol · API986.9×
Sonnet 5.5 · API9436×
Opus 5.5 · Claude9437×
GPT-6 Astra · ChatGPT9739×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Turning strategy into a sequenced plan.

Why it matters

A roadmap is where strategy meets capacity. Dated feature lists turn guesses into promises.

What good looks like

  • Items are problems or outcomes, not just features
  • Sequencing reflects dependencies
  • Explicit trade-offs
  • Commitment falls with distance

Deliberately not measured

  • Gantt formatting
Capability tested

Sequencing under constraints

The failure we’re looking for

A dated wishlist sorted by excitement

Grading

Decision model and LLM judge, calibrated against a blind PM review