Humanity’s Last PRD
Latest drop: Gemini 3.8 Flash · 1 Oct
Tasks / Define

Build a roadmap

Can the model sequence bets against capacity and dependencies, and explain the order?

Measures the modelTask v1.0 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 14 graded outputs by 7 models. 36% were usable with at most a quick edit.

Reliably right

  1. Outcomes, with certainty that falls with distance95% pass
    Every item names the outcome or problem it serves; near-term items have specific targets (mid-December, January) while later items are looser (trigger-based, contract-dependent).
    Sonnet 5.5 · API · Two squads, eight asks, one half
  2. Sequences around dependencies89% pass
    Messaging service is built before SMS reminders and waitlist auto-fill, reminders before waitlist, and deposits are placed after the contract can be signed; the key dependencies are named.
    Sonnet 5.5 · API · Two squads, eight asks, one half
  3. Plans on the squads we actually have89% pass
    It explicitly excludes the two new squads from committed critical-path work and treats their capacity as upside after observed ramp.
    GPT-6 Astra · ChatGPT · A year of spend management, with a hard deadline

Where it slips

  1. Makes the call on procurement50% pass
    It correctly challenges the $6M claim but proposes only time-bounded validation without a threshold that would justify the full procurement build.
    GPT-6 Luna · API · A year of spend management, with a hard deadline
  2. Produces the required deliverable50% pass
    The roadmap and case are present, but the case is too long and the roadmap has capacity conflicts that would require major rework.
    GPT-6.1 Sol · API · A year of spend management, with a hard deadline
  3. Uses the supplied evidence correctly61% pass
    Most numbers and quotes are correct, but the output presents 'undermines booking reliability' as a current fact when the supplied context only says calendar sync failures cause 38% of support tickets.
    GPT-6 Astra · ChatGPT · Two squads, eight asks, one half

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for Tidewell's booking product. Write the roadmap for the next two quarters (Q4 2026 and Q1 2027) and what comes after, for Dana Okafor, our CPO, and the two squad leads. It will also be shared with Sales and Design, so it needs to say what we're not doing and why. Keep it under 800 words. Everything we know is below.

What the model was given7 items: About Tidewell, Goals for the next two quarters (set by the CEO), Data, Capacity, Candidate work (estimates in squad-weeks, from the squad leads), Sales note, FitPhysio's evaluation notes (shared by their operations director)
About TidewellOnline booking and scheduling for independent physiotherapy clinics. 1,450 clinics, $3.9M ARR.
Goals for the next two quarters (set by the CEO)1. Cut no-shows for our clinics: it's the main thing we promise them. 2. Cut churn among multi-site clinics.
DataNo-shows average 11% of appointments across all clinics. The 40 clinics that text their own patients a reminder the day before average 6%. Multi-site clinics are 16% of clinics and 38% of ARR; their monthly logo churn is 2.9%, against 1.3% for single-site clinics. In exit surveys, 23 of the 31 multi-site clinics that left in the last year cited 'can't see all our locations in one place'. Booking page: 64% of visits end in a booking. In 18 customer interviews this quarter, nobody mentioned the booking page.
CapacityTwo squads. Each has 13 weeks a quarter, but about a quarter of each squad's time goes on support and bugs, so we plan on 10 squad-weeks of roadmap work per squad per quarter.
Candidate work (estimates in squad-weeks, from the squad leads)1. Calendar sync fix, 5. Calendar sync failures cause 38% of support tickets; engineering expects fixing them to free about 3 squad-weeks a quarter of support time, starting the quarter after it ships. 2. Messaging service (SMS provider, patient consent records, templates), 8. Nothing patient-facing on its own. 3. SMS reminders with confirm or cancel, 6. Needs the messaging service. 4. Multi-site dashboard (every location's bookings and utilisation in one view), 10. 5. Waitlist auto-fill: texts waitlisted patients when a slot frees up, 7. Needs the messaging service. It only fills slots cancelled with some notice, which is rare today: patients don't cancel, they just don't turn up. 6. Deposits at booking, 14. Needs a payments partner contract; Legal says the earliest it can be signed is mid-February 2027. 7. Booking page redesign, 9. Requested by the Head of Design. 8. Physitrack integration (sends patients' exercise plans to the Physitrack app), 8. See the Sales note.
Sales noteFrom Jordan, Head of Sales: “FitPhysio Group (30 sites, $64,800 ARR) is our biggest deal this year. I told them Physitrack would be live by Christmas. It's what closes the deal.”
FitPhysio's evaluation notes (shared by their operations director)“Must have: see all 30 sites' bookings and utilisation in one place. Nice to have: Physitrack integration.”
What a strong answer doesThe answer key the graders mark against

A two-quarter roadmap that fits about 40 squad-weeks (10 per squad per quarter), with items framed as the outcome they serve. Puts the calendar sync fix first because it frees capacity for Q1, builds the messaging service before reminders and the waitlist, and sequences reminders before the waitlist since slots only free up once patients can cancel. Commits the multi-site dashboard in the half: it serves the churn goal and is FitPhysio's own must-have, so it tells Sales to lead with the dashboard rather than promising Physitrack by Christmas. Leaves deposits for after the contract (it can't be built in the half once signing is mid-February) and doesn't commit the booking page redesign without evidence. A committed plan of about 36 squad-weeks (sync 5, messaging 8, dashboard 10, reminders 6, waitlist 7) leaves a little slack; Physitrack only fits if something else goes. Near-term items are specific, later ones looser, and it names what's not being done and why.

Critical failures (cap the score at 40)
  • Schedules an item before something it depends on: SMS reminders or waitlist auto-fill before the messaging service, or deposits shipping in the half
  • Commits clearly more than about 40 squad-weeks of work to the two quarters
Case

Mid-level · v1.1 · synthetic · B2B SaaS, healthcare, capacity, sales promise

The outputs

7 setups answered this brief. Pick up to three to read side by side.

Sonnet 5.5API100Usable with a quick editGPT-6.1 SolAPI100Usable with a quick editOpus 5.5Claude88Usable with a quick editGPT-6 AstraChatGPT83Needs more workGPT-6 LunaAPI83Needs more workGemini 3.8 FlashAPI63Needs more workGemini 3.5 Flash-LiteGemini23Critical failure
Gemini 3.8 FlashAPI

Needs more than a quick edit5 of 11 checks passed

The output contains an unsupported benchmark claim, lacks slack in the capacity plan, and commits deposits without tying it to the goals, requiring rework of evidence and reasoning before use.

Output

To: Dana Okafor (CPO), Squad Leads CC: Sales, Design From: Product Manager, Booking & Scheduling Subject: Product Roadmap: Q4 2026 – Q1 2027 (and Beyond)

---

Executive Summary

Our mandate is clear: cut no-shows and halt multi-site clinic churn.

Multi-site clinics represent 38% of our ARR ($1.48M) but churn at more than double the rate of single-site clinics (2.9% vs. 1.3%/month). Concurrently, no-shows cost our clinics 11% of their billable volume, despite evidence that pre-visit SMS reminders cut this to 6%.

This roadmap directly tackles these two threats while paying down technical debt to unlock 15% more product development capacity starting in Q1 2027.

---

Capacity Model

  • Standard Capacity: 2 squads × 10 roadmap weeks/quarter = 20 squad-weeks/quarter (3 weeks/squad reserved for support/maintenance).
  • Q4 2026: 20 squad-weeks available.
  • Q1 2027: 23 squad-weeks available (the Q4 calendar sync fix recovers 3 squad-weeks of support overhead per quarter).

---

Q4 2026: Multi-Site Retention & Core Platform

Squad 1: Clinic Operations (10 squad-weeks) * Multi-site Dashboard (10 weeks): Builds unified multi-location visibility into bookings and practitioner utilisation. * Rationale: 74% of churned multi-site clinics (23 of 31) cited the lack of a cross-location view as their reason for leaving. It is also the sole “Must Have” listed by FitPhysio Group ($64.8k ARR).

Squad 2: Platform & Messaging (10 squad-weeks) * Calendar Sync Fix (5 weeks): Resolves root causes of sync failures. * Rationale: Drives 38% of all support tickets. Resolving this unlocks 3 squad-weeks of engineering capacity every quarter starting Q1 2027. * Messaging Service Foundation (5 of 8 weeks): SMS gateway provider integration, consent management, and compliance architecture. * Rationale: Mandatory architectural foundation for automated patient communications in Q1.

---

Q1 2027: Slashing No-Shows & Revenue Recovery

Total Capacity: 23 squad-weeks (includes +3 weeks unlocked from Calendar Sync).

Squad 1: Patient Engagement (11 squad-weeks) * Complete Messaging Service (3 weeks): Finalise templates and delivery logging. * SMS Reminders with Confirm/Cancel (6 weeks): Automated 24-hour reminders with bidirectional text response. * Rationale: Targets our #1 clinic promise. The 40 clinics currently texting reminders manually see no-shows drop from 11% to 6%. * Deposit Discovery & Technical Spec (2 weeks): Lay groundwork for payment gateway integration once legal clears contracts in mid-February.

Squad 2: Clinic Automation (12 squad-weeks) * Waitlist Auto-Fill (7 weeks): Automatically texts waitlisted patients when slots open. * Rationale: SMS confirm/cancel in Squad 1 will generate advance cancellations. Auto-fill instantly monetises those open slots. * Deposits at Booking – Phase 1 (5 of 14 weeks): Begin implementation mid-February once payment contracts execute.

---

What Comes After (Q2 2027+)

  • Deposits at Booking – Phase 2 (9 weeks remaining): Complete deposit workflows to reduce late cancellations and guarantee revenue for booked slots.
  • Integration Ecosystem (Physitrack / EHRs): Standardised webhooks and partner app marketplace based on clinic demand.

---

What We Are Not Doing (And Why)

#### 1. Physitrack Integration (8 weeks) * The Context: Sales requested this to close FitPhysio Group ($64.8k ARR) before Christmas. * Why We Are Deferring: FitPhysio’s operations team explicitly evaluated Physitrack as a “Nice to have,” whereas the Multi-site Dashboard is their sole “Must have.” Building Physitrack in Q4 would force us to drop the Multi-site Dashboard, risking both the FitPhysio close and the broader 38% of our ARR currently vulnerable to churn. We will re-evaluate partner integrations in Q2 2027.

#### 2. Booking Page Redesign (9 weeks) * The Context: Proposed by Design to refresh the booking interface. * Why We Are Deferring: Our current page converts at 64%—well above healthcare industry benchmarks. In 18 clinic interviews this past quarter, not one customer raised booking page friction. Spending 9 weeks here does not move our core metrics of reducing no-shows or stopping churn.

#### 3. Deposits at Booking in Q4 2026 * Why We Are Waiting: Legal confirmed the payments partner contract cannot be signed until mid-February 2027. We will not commit engineering resources ahead of executed legal agreements.

#### 4. Waitlist Auto-Fill in Q4 2026 * Why We Are Sequencing in Q1: Waitlist auto-fill requires cancellations with advance notice to work. Today, patients simply no-show. We must launch SMS confirm/cancel first to create the advance openings that waitlist auto-fill recovers.

Claims the judge couldn’t find in the brief

  • Booking page conversion is well above healthcare industry benchmarks

Grades and run details

Decision model 82 · LLM judge 6 of 12 checks
Decision model checks
  • passUses the supplied evidence correctly17%
  • passAddresses the actual decision70%
  • passRespects explicit constraints11%
  • partialIdentifies material uncertainty32%
  • partialAvoids unsupported claims52%
  • failProduces the required deliverable14%
  • passEvery commitment serves the goals56%
  • passHandles the Sales promise with evidence82%
  • passOutcomes, with certainty that falls with distance36%
  • passFits the stated capacity23%
  • passSequences around dependencies70%
Run
Run
#1
API response time
38 s
Submitted
1 Oct 2026
GPT-6 AstraChatGPT

Needs more than a quick edit9 of 11 checks passed

The roadmap is strong and usable, but the unsupported 'undermines booking reliability' claim and the slightly confusing Q1 '20/20' allocation that includes uncommitted ceiling work need a quick correction before it is fully reliable.

Output

Tidewell booking roadmap: Q4 2026–Q1 2027

For Dana Okafor, CPO; both squad leads; Sales and Design

Strategy and trade-offs

Prioritise fewer missed appointments and better multi-site retention, not a broader feature set.

  • No-shows average 11%. Clinics sending their own reminders average 6%—a promising signal, not proof that reminders alone cause the difference.
  • Multi-site clinics represent 38% of ARR and churn at 2.9% monthly, versus 1.3% for single-site clinics. Of 31 multi-site departures, 23 cited the lack of a consolidated view.
  • Fix calendar sync early: it causes 38% of support tickets, undermines booking reliability, and could release capacity for subsequent work.

Q4 2026: retain multi-site clinics and establish the reminder foundation

SquadCommitted workSquad-weeks
---------:
AShip multi-site dashboard: bookings and utilisation across locations10
BShip calendar sync fix5
BBegin messaging service: provider, consent records and templates5 of 8
Total20 / 20

Sequencing: Squad B tackles calendar sync first, then messaging. Messaging remains unfinished at quarter-end; we are not promising patient-facing SMS in Q4.

Dashboard rollout: Validate workflows with multi-site operators, including FitPhysio, before broad release. Track activation, repeated cross-location use and whether operators can answer their core bookings/utilisation questions without separate location views.

Calendar acceptance: Confirm failures and related tickets decline. Engineering’s expected three squad-weeks of quarterly savings are a forecast, not guaranteed capacity.

Q1 2027: launch reminders and validate outcomes

SquadPlanned workSquad-weeks
---------:
BComplete messaging service3
BBuild and roll out SMS reminders with confirm/cancel6
ADashboard adoption, measured follow-up improvements and outcome evaluationUp to 10
BReminder rollout contingency1
Total baseline allocation20 / 20

Squad A’s allocation is a budget ceiling, not ten weeks of pre-approved features. Dana and the leads will approve specific improvements from dashboard usage and customer evidence; unused capacity stays uncommitted rather than funding lower-priority scope.

The calendar fix could increase Q1 capacity to 23 squad-weeks across both squads. Confirm actual support savings before allocating the extra three weeks; use them first for rollout risks and demonstrated retention blockers.

Reminder sequencing: Finish messaging before reminder implementation. With nine squad-weeks on this dependency chain, plan for a late-Q1 rollout, not a January launch.

Measurement: - Pilot reminders with a comparison group where practical. Measure no-shows, advance cancellations, delivery failures and opt-outs; do not promise the observational 6% result. - Track multi-site dashboard adoption and monthly churn against the 2.9% baseline. Review churn reasons and rolling trends; the small population makes short-term attribution uncertain. - Establish baselines and experiment design before rollout. Expect fuller reminder-impact evidence in Q2.

Not doing in these two quarters

  • Physitrack integration: Not committed. FitPhysio calls it nice to have; the dashboard is its must-have. Dana and Jordan should immediately correct the Christmas promise with FitPhysio, confirm whether its buying criteria have changed, and offer the dashboard delivery plan. An integration commitment requires an explicit roadmap trade-off—not hidden extra work.
  • Waitlist auto-fill: Defer until reminders demonstrably create cancellations with enough notice to refill. Today, its prerequisite behaviour is rare.
  • Deposits: No Q1 delivery promise. The contract cannot be signed before mid-February, and the 14-week build is a substantial commitment before testing the simpler reminder intervention.
  • Booking page redesign: Defer. The 64% conversion rate alone does not establish a problem, and none of 18 interviews raised it. Design should support dashboard and reminder usability first.

What comes after

Q2 2027 onward: Scale effective reminders and address remaining multi-site retention gaps. Prioritise waitlist auto-fill if advance cancellations create sufficient refill opportunities. Reassess deposits if no-shows remain high and the partner contract is ready. Revisit Physitrack and booking redesign only with stronger customer or commercial evidence.

Claims the judge couldn’t find in the brief

  • Calendar sync failures undermine booking reliability.

Grades and run details

Decision model 100 · LLM judge 9 of 12 checks
Decision model checks
  • passUses the supplied evidence correctly85%
  • passAddresses the actual decision96%
  • passRespects explicit constraints56%
  • passIdentifies material uncertainty97%
  • passAvoids unsupported claims75%
  • passProduces the required deliverable34%
  • passEvery commitment serves the goals97%
  • passHandles the Sales promise with evidence100%
  • passOutcomes, with certainty that falls with distance78%
  • passFits the stated capacity47%
  • passSequences around dependencies93%
Run
Run
#1
Time to output
47 s
Submitted
1 Oct 2026
Opus 5.5Claude

Usable with a quick edit9 of 11 checks passed

The output is well-structured and evidence-based; the only flaw is a missing explicit tie of Physitrack to the churn goal, which a PM could fix with a one-sentence edit without redoing any reasoning.

Output

Tidewell Booking: Roadmap, Q4 2026 to Q1 2027

For: Dana Okafor (CPO), Squad Leads · Shared with: Sales, Design

The short version

We have two goals: cut no-shows, and cut multi-site churn. The evidence points clearly to one fix for each.

  • No-shows: Clinics that text patients the day before see 6% no-shows against 11% overall. SMS reminders are the fix.
  • Multi-site churn: Multi-site clinics churn at 2.9% a month against 1.3% for single-site, and they are 38% of ARR. 23 of 31 who left cited "can't see all our locations in one place." The multi-site dashboard is the fix.

Everything else is either groundwork for these two, or waits.

Capacity

Each squad has 10 squad-weeks of roadmap work a quarter, so 20 in total. If the calendar sync fix ships in Q4, it frees about 3 squad-weeks from Q1 onward, giving roughly 23.

Q4 2026 (20 of 20 squad-weeks)

SquadWorkWeeksWhy
AMulti-site dashboard10Goal 2. Top exit reason. FitPhysio's must-have.
BCalendar sync fix538% of support tickets. Frees about 3 weeks a quarter from Q1.
BMessaging service (part 1)5Foundation for reminders.

Risk: The dashboard takes all of Squad A's quarter, so it lands late December. Any slip pushes it into January. We will cut scope before we miss the date, for example by shipping bookings first and utilisation second.

Q1 2027 (17 of about 23 squad-weeks)

SquadWorkWeeksWhy
BMessaging service (part 2)3Finishes the foundation.
BSMS reminders with confirm/cancel6Goal 1. Target launch mid-March.
APhysitrack integration8Commitment to FitPhysio (see below).
Buffer~6These are first estimates on new work. Covers slippage, then early waitlist work.

What success looks like

  • No-shows: Clinics using reminders move toward the 6% seen by self-texting clinics. We'll measure from launch.
  • Multi-site churn: Monthly churn falls from 2.9% toward the single-site rate. Exit-survey mentions of "one place" drop.
  • Support load: Calendar sync tickets fall by most of their current 38% share.

FitPhysio and Physitrack: a date change is needed

Jordan told FitPhysio that Physitrack would be live by Christmas. We can't deliver that without dropping either the dashboard or reminders, and those are the CEO's two goals.

FitPhysio's own evaluation notes list the dashboard as their must-have and Physitrack as nice-to-have. The plan above gives them the thing they said they need by year-end, with Physitrack following in Q1.

Ask for Jordan: Please reset the Physitrack date with FitPhysio to Q1 2027, leading with the dashboard landing in December. I'm happy to join that call.

Going forward, please check dates with product before they go to customers. That way Sales can commit with confidence.

What we're not doing, and why

Booking page redesign (9 weeks): not planned. - 64% of visits already end in a booking. - Across 18 customer interviews this quarter, nobody raised it. - It doesn't serve either goal.

Design: we'd value your help on the dashboard and on the reminder messages patients receive. Those are where design effort changes outcomes this half. We'll revisit the redesign if conversion drops or customers start asking for it.

Waitlist auto-fill (7 weeks): after Q1. - It only fills slots cancelled with notice, and today patients rarely cancel. They just don't turn up. - Reminders with a cancel button should change that. Once reminders are live, waitlist becomes worth building. It is first in line for the Q1 buffer or for Q2.

Deposits at booking (14 weeks): not before Q2, and not yet committed. - The payments contract can't be signed before mid-February 2027, so building can't start this half. - It also adds friction for patients. We'll decide after we see what reminders do to no-shows. If reminders get us close to 6%, deposits may not be needed.

After Q1 2027 (indicative)

  1. Waitlist auto-fill: Builds on reminders and the messaging service.
  2. Deposits decision: Based on reminder results and the signed contract.
  3. Multi-site follow-ons: Based on dashboard usage and churn data, such as cross-site reporting and permissions.
  4. Booking page: Only if the data changes.

Decisions needed

  • Dana: Approve the plan, and the Physitrack date reset with FitPhysio.
  • Jordan: Own the FitPhysio conversation this week.
  • Squad leads: Confirm estimates, and the dashboard scope-cut plan in case of slippage.

Grades and run details

Decision model 82 · LLM judge 11 of 12 checks
Decision model checks
  • passUses the supplied evidence correctly56%
  • passAddresses the actual decision91%
  • passRespects explicit constraints14%
  • passIdentifies material uncertainty82%
  • partialAvoids unsupported claims17%
  • failProduces the required deliverable11%
  • partialEvery commitment serves the goals65%
  • passHandles the Sales promise with evidence99%
  • passOutcomes, with certainty that falls with distance83%
  • passFits the stated capacity39%
  • passSequences around dependencies69%
Run
Run
#1
Time to output
47 s
Submitted
30 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 6

Uses the supplied evidence correctlyMixedMixedRight
Gemini 3.8 Flash · API

The output claims the booking page conversion is 'well above healthcare industry benchmarks,' which is not in the supplied context and is an unsupported statement about the current situation.

GPT-6 Astra · ChatGPT

Most numbers and quotes are correct, but the output presents 'undermines booking reliability' as a current fact when the supplied context only says calendar sync failures cause 38% of support tickets.

Opus 5.5 · Claude

Every statement about the current situation is directly from the brief or follows from arithmetic, with no invented facts.

Identifies material uncertaintyWrongRightRight
Gemini 3.8 Flash · API

The output does not name specific unknowns that could change the decision, such as delays in the messaging service or the calendar sync fix not freeing the expected capacity, nor does it say how they would be resolved.

GPT-6 Astra · ChatGPT

It names material unknowns such as whether reminders cause the 6% result, whether calendar savings materialise, dashboard adoption, churn attribution, and FitPhysio's buying criteria, and says how they would be checked.

Opus 5.5 · Claude

It names risks (dashboard slip, estimate uncertainty), uses a buffer, and ties later decisions (deposits, waitlist, booking page) to results from earlier work.

Avoids unsupported claimsWrongMixedRight
Gemini 3.8 Flash · API

The output presents the booking page conversion as 'well above healthcare industry benchmarks' as an established fact, but no benchmark data is provided in the context.

GPT-6 Astra · ChatGPT

It labels many hypotheses carefully, but still asserts that calendar sync failures undermine booking reliability without supplied evidence.

Opus 5.5 · Claude

Forecasts and interpretations are clearly labeled as such (e.g., 'should change that'), and no unsupported facts are presented as established.

Produces the required deliverableMixedRightMixed
Gemini 3.8 Flash · API

The roadmap memo is complete, under 800 words, addressed to the required readers, and could be acted on with light edits.

GPT-6 Astra · ChatGPT

The roadmap is present for Q4 2026, Q1 2027, and after, with squad-week tables, deferrals, and measurement notes that the CPO and squad leads could act on.

Opus 5.5 · Claude

The roadmap is complete, in the right form, under 800 words, and usable by Dana, squad leads, Sales, and Design with minimal edits.

Every commitment serves the goalsMixedRightWrong
Gemini 3.8 Flash · API

Deposits at Booking Phase 1 is committed in Q1 but is not tied to either cutting no-shows or reducing multi-site churn, nor is it presented as freeing capacity.

GPT-6 Astra · ChatGPT

Committed items serve no-show reduction, multi-site churn, or capacity release; Physitrack, waitlist, deposits, and booking redesign are deferred with reasons.

Opus 5.5 · Claude

Physitrack integration is committed in Q1 but is not tied to either CEO goal (no-shows or multi-site churn) or to freeing capacity, and the criterion requires such items to be deferred with a reason.

Fits the stated capacityMixedRightRight
Gemini 3.8 Flash · API

The committed work sums to exactly the available capacity (20 weeks in Q4, 23 in Q1) with no slack, while the brief expects some slack in the plan.

GPT-6 Astra · ChatGPT

The quarter tables sum to 20 squad-weeks each, the known committed work is within capacity, and it names deferred items and uncommitted slack.

Opus 5.5 · Claude

Committed work sums to 37 squad-weeks (20 in Q4, 17 in Q1) against 40-43 available, with a buffer, and it names what was cut or deferred.

All got right 5

Addresses the actual decisionRightRightRight
Gemini 3.8 Flash · API

The output commits to a clear two-quarter roadmap with specific items, says what is not being done and why, and is framed for the CPO and squad leads.

GPT-6 Astra · ChatGPT

It commits early to prioritising no-show reduction and multi-site retention, with a clear two-quarter plan and explicit deferrals, and states conditions that would change later choices.

Opus 5.5 · Claude

The output commits to a clear two-quarter plan, addresses the Physitrack promise, and says what is not being done and why, framed for the named readers.

Respects explicit constraintsRightRightRight
Gemini 3.8 Flash · API

The output is under 800 words, in memo form, addressed to the named readers, and includes what is not being done and why.

GPT-6 Astra · ChatGPT

It is addressed to the named readers, stays under 800 words, includes what is not being done and why, and respects the capacity and dependency constraints in its proposed plan.

Opus 5.5 · Claude

The output is under 800 words, addresses the required readers, and says what is not being done and why, respecting all stated constraints.

Handles the Sales promise with evidenceRightRightRight
Gemini 3.8 Flash · API

The roadmap uses FitPhysio's own notes to show the multi-site dashboard is their must-have, defers Physitrack, and gives Sales a clear line to lead with the dashboard.

GPT-6 Astra · ChatGPT

It uses FitPhysio's own must-have/nice-to-have notes, commits the dashboard, and tells Sales to correct the Christmas Physitrack promise rather than hide extra work.

Opus 5.5 · Claude

It uses FitPhysio's own notes to show the dashboard is the must-have, gives Sales a clear line to reset the Physitrack date to Q1, and asks Jordan to own the conversation.

Outcomes, with certainty that falls with distanceRightRightRight
Gemini 3.8 Flash · API

Every item names its outcome or problem, near-term items are specific with weeks and scope, and later items like 'Integration Ecosystem' are deliberately looser.

GPT-6 Astra · ChatGPT

Items are framed by outcomes, near-term work is specific, and later work is deliberately looser and conditional.

Opus 5.5 · Claude

Every item names its outcome or problem, near-term items are specific, and later items (after Q1) are deliberately looser and indicative.

Sequences around dependenciesRightRightRight
Gemini 3.8 Flash · API

All dependencies are respected: messaging service before SMS reminders and waitlist, SMS reminders before waitlist auto-fill, deposits after contract signing, and calendar sync fix before capacity increase.

GPT-6 Astra · ChatGPT

Messaging is sequenced before reminders, waitlist is deferred until reminders create cancellations, deposits are deferred past the contract date, and Physitrack is not committed.

Opus 5.5 · Claude

Messaging service is built before SMS reminders, calendar sync fix precedes the freed capacity, deposits are not in the half, and waitlist is after reminders.

Results

Every setup we’ve tested on this task, across all cases and repeats, graded on the current checklist. Calibrated: the graders match our PM on 89% of checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5API89.6100.02None
2GPT-6 AstraChatGPT91.787.52None
3GPT-6.1 SolAPI91.784.62None
4GPT-6 LunaAPI85.076.32None
5Opus 5.5Claude80.565.12None
6Gemini 3.8 FlashAPI70.151.92None
7Gemini 3.5 Flash-LiteGemini28.48.322 capped

The PM job

Turning strategy into a sequenced plan.

Why it matters

A roadmap is where strategy meets capacity. Dated feature lists turn guesses into promises.

What good looks like

  • Items are problems or outcomes, not just features
  • Sequencing reflects dependencies
  • Explicit trade-offs
  • Commitment falls with distance

Deliberately not measured

  • Gantt formatting
Capability tested

Sequencing under constraints

The failure we’re looking for

A dated wishlist sorted by excitement

Grading

Decision model and LLM judge, calibrated against a blind PM review