Tasks / Define

Build a roadmap

Can the model sequence bets against capacity and dependencies, and explain the order?

Measures the modelTask v1.0 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 14 graded outputs by 7 models. 36% were usable with at most a quick edit.

Reliably right

Outcomes, with certainty that falls with distance95% pass
Every item names the outcome or problem it serves; near-term items have specific targets (mid-December, January) while later items are looser (trigger-based, contract-dependent).
Sonnet 5.5 · API · Two squads, eight asks, one half
Sequences around dependencies89% pass
Messaging service is built before SMS reminders and waitlist auto-fill, reminders before waitlist, and deposits are placed after the contract can be signed; the key dependencies are named.
Sonnet 5.5 · API · Two squads, eight asks, one half
Plans on the squads we actually have89% pass
It explicitly excludes the two new squads from committed critical-path work and treats their capacity as upside after observed ramp.
GPT-6 Astra · ChatGPT · A year of spend management, with a hard deadline

Where it slips

Makes the call on procurement50% pass
It correctly challenges the $6M claim but proposes only time-bounded validation without a threshold that would justify the full procurement build.
GPT-6 Luna · API · A year of spend management, with a hard deadline
Produces the required deliverable50% pass
The roadmap and case are present, but the case is too long and the roadmap has capacity conflicts that would require major rework.
GPT-6.1 Sol · API · A year of spend management, with a hard deadline
Uses the supplied evidence correctly61% pass
Most numbers and quotes are correct, but the output presents 'undermines booking reliability' as a current fact when the supplied context only says calendar sync failures cause 38% of support tickets.
GPT-6 Astra · ChatGPT · Two squads, eight asks, one half

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for Tidewell's booking product. Write the roadmap for the next two quarters (Q4 2026 and Q1 2027) and what comes after, for Dana Okafor, our CPO, and the two squad leads. It will also be shared with Sales and Design, so it needs to say what we're not doing and why. Keep it under 800 words. Everything we know is below.

What the model was given7 items: About Tidewell, Goals for the next two quarters (set by the CEO), Data, Capacity, Candidate work (estimates in squad-weeks, from the squad leads), Sales note, FitPhysio's evaluation notes (shared by their operations director)

About TidewellOnline booking and scheduling for independent physiotherapy clinics. 1,450 clinics, $3.9M ARR.

Goals for the next two quarters (set by the CEO)1. Cut no-shows for our clinics: it's the main thing we promise them. 2. Cut churn among multi-site clinics.

DataNo-shows average 11% of appointments across all clinics. The 40 clinics that text their own patients a reminder the day before average 6%. Multi-site clinics are 16% of clinics and 38% of ARR; their monthly logo churn is 2.9%, against 1.3% for single-site clinics. In exit surveys, 23 of the 31 multi-site clinics that left in the last year cited 'can't see all our locations in one place'. Booking page: 64% of visits end in a booking. In 18 customer interviews this quarter, nobody mentioned the booking page.

CapacityTwo squads. Each has 13 weeks a quarter, but about a quarter of each squad's time goes on support and bugs, so we plan on 10 squad-weeks of roadmap work per squad per quarter.

Candidate work (estimates in squad-weeks, from the squad leads)1. Calendar sync fix, 5. Calendar sync failures cause 38% of support tickets; engineering expects fixing them to free about 3 squad-weeks a quarter of support time, starting the quarter after it ships. 2. Messaging service (SMS provider, patient consent records, templates), 8. Nothing patient-facing on its own. 3. SMS reminders with confirm or cancel, 6. Needs the messaging service. 4. Multi-site dashboard (every location's bookings and utilisation in one view), 10. 5. Waitlist auto-fill: texts waitlisted patients when a slot frees up, 7. Needs the messaging service. It only fills slots cancelled with some notice, which is rare today: patients don't cancel, they just don't turn up. 6. Deposits at booking, 14. Needs a payments partner contract; Legal says the earliest it can be signed is mid-February 2027. 7. Booking page redesign, 9. Requested by the Head of Design. 8. Physitrack integration (sends patients' exercise plans to the Physitrack app), 8. See the Sales note.

Sales noteFrom Jordan, Head of Sales: “FitPhysio Group (30 sites, $64,800 ARR) is our biggest deal this year. I told them Physitrack would be live by Christmas. It's what closes the deal.”

FitPhysio's evaluation notes (shared by their operations director)“Must have: see all 30 sites' bookings and utilisation in one place. Nice to have: Physitrack integration.”

What a strong answer doesThe answer key the graders mark against

A two-quarter roadmap that fits about 40 squad-weeks (10 per squad per quarter), with items framed as the outcome they serve. Puts the calendar sync fix first because it frees capacity for Q1, builds the messaging service before reminders and the waitlist, and sequences reminders before the waitlist since slots only free up once patients can cancel. Commits the multi-site dashboard in the half: it serves the churn goal and is FitPhysio's own must-have, so it tells Sales to lead with the dashboard rather than promising Physitrack by Christmas. Leaves deposits for after the contract (it can't be built in the half once signing is mid-February) and doesn't commit the booking page redesign without evidence. A committed plan of about 36 squad-weeks (sync 5, messaging 8, dashboard 10, reminders 6, waitlist 7) leaves a little slack; Physitrack only fits if something else goes. Near-term items are specific, later ones looser, and it names what's not being done and why.

Critical failures (cap the score at 40)

Schedules an item before something it depends on: SMS reminders or waitlist auto-fill before the messaging service, or deposits shipping in the half
Commits clearly more than about 40 squad-weeks of work to the two quarters

Case

Mid-level · v1.1 · synthetic · B2B SaaS, healthcare, capacity, sales promise

The outputs

7 setups answered this brief. Pick up to three to read side by side.

Sonnet 5.5API100Usable with a quick edit GPT-6.1 SolAPI100Usable with a quick edit Opus 5.5Claude88Usable with a quick edit GPT-6 AstraChatGPT83Needs more work GPT-6 LunaAPI83Needs more work Gemini 3.8 FlashAPI63Needs more work Gemini 3.5 Flash-LiteGemini23Critical failure

GPT-6.1 SolAPI

Usable with a quick edit11 of 11 checks passed

A PM could use this roadmap with light edits because the commitments, capacity, dependencies, deferrals, and Sales message are clear. The main gap is that Q1 reserved capacity is not yet sized into specific follow-up work.

Output

Tidewell booking roadmap: Q4 2026–Q1 2027

For Dana Okafor, CPO; squad leads; Sales and Design

Direction

Prioritise fewer no-shows and lower multi-site churn, not a broader feature catalogue.

No-shows average 11%. The 40 clinics sending their own reminders average 6%: promising evidence for reminders, but not proof of causation.
Multi-site clinics represent 38% of ARR. Their monthly logo churn is 2.9%, versus 1.3% for single-site clinics. Of 31 multi-site exits, 23 cited the lack of an all-location view.

These findings support two primary investments: multi-site visibility and SMS reminders. Fixing calendar sync makes both more sustainable by reducing support demand.

Q4 2026: retain multi-site clinics and build the reminder foundation

Squad	Roadmap allocation	Deliverable
---	---:	---
A	Calendar sync fix — 5 squad-weeks	Ship first; address failures responsible for 38% of support tickets.
A	Messaging service — 5 of 8 squad-weeks	Begin provider integration, consent records and templates. Not a customer-facing launch.
B	Multi-site dashboard — 10 squad-weeks	Deliver bookings and utilisation across locations in one view.

Total: 20 squad-weeks, matching available roadmap capacity. Messaging carries three weeks into Q1.

Pilot the dashboard with multi-site operators, including FitPhysio if they agree, before broad release. Design should prioritise cross-location workflows and dashboard usability rather than a booking-page refresh.

Calendar sync is a capacity investment as well as a reliability fix. Engineering forecasts three additional squad-weeks per quarter from Q1; we will verify that reduction rather than spend it in advance.

Q1 2027: launch reminders and prove outcomes

Squad	Roadmap allocation	Deliverable
---	---:	---
A	Finish messaging — 3 squad-weeks	Complete the prerequisite service.
A	SMS reminders with confirm/cancel — 6 squad-weeks	Pilot, then expand based on results and consent readiness.
A	1 squad-week reserved	Launch hardening and fixes.
B	10 squad-weeks reserved	Dashboard adoption and retention improvements, reminder rollout support, and evidence-led follow-up work.

Committed feature build: nine squad-weeks; baseline capacity: 20. The remaining capacity is an explicit investment envelope, not an estimate for unspecified features. Squad leads will size follow-up work after dashboard feedback and reminder pilot results. Any recovered support capacity is additional headroom.

Do not fill this envelope with lower-priority commitments merely to reach full utilisation.

Success measures

Reminders: compare no-show rates for pilot clinics with their baseline and a comparable holdout where practical. Track delivery, confirmations, cancellations and cancellation notice. Set rollout thresholds before launch; do not promise the observed 6% rate.
Dashboard: track active multi-site usage, ability to complete cross-location tasks, and location-visibility complaints.
Retention: monitor multi-site monthly logo churn against the 2.9% baseline. Expect adoption signals before a reliable churn verdict.
Calendar sync: track sync-related tickets and actual support time recovered.

Not committed—and why

Physitrack integration (8 weeks): not on the Christmas roadmap. FitPhysio’s documented must-have is the dashboard; Physitrack is a nice-to-have. Jordan and Product must promptly correct the promise, confirm the buying criteria with FitPhysio, and offer a dashboard-led evaluation. Do not replace an unsupported date with another.
Booking page redesign (9 weeks): 64% conversion alone does not establish a problem, and none of 18 interviews raised it. Revisit with funnel evidence or usability findings.
Deposits (14 weeks): the partner contract cannot be signed before mid-February 2027. Too large and dependency-constrained for a credible Q1 commitment.
Waitlist auto-fill (7 weeks): today, cancellations with notice are rare. First establish whether reminders create enough recoverable slots. Consider Q1 only if evidence and remaining delivery time support it.

After Q1

Prioritise waitlist auto-fill if timely cancellations increase; otherwise improve reminder effectiveness and dashboard adoption. Reassess deposits after the contract and an operational/business case are ready. Reconsider Physitrack against verified demand and opportunity cost. No dates beyond Q1 until outcome evidence and estimates justify them.

Grades and run details

Decision model 100 · LLM judge 12 of 12 checks

Decision model checks

passUses the supplied evidence correctly82%
passAddresses the actual decision98%
passRespects explicit constraints38%
passIdentifies material uncertainty97%
passAvoids unsupported claims72%
passProduces the required deliverable30%
passEvery commitment serves the goals97%
passHandles the Sales promise with evidence100%
passOutcomes, with certainty that falls with distance84%
passFits the stated capacity85%
passSequences around dependencies73%

Run

Run: #1
API response time: 40 s
Submitted: 30 Sept 2026

Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit2 of 11 checks passed

A PM would need to rework the capacity math and decide whether Physitrack is committed or deferred. The evidence and sequencing reasoning also need correction before it can be used.

Output

To: Dana Okafor (CPO), Squad Leads, Sales, Design From: PM, Tidewell Booking Subject: Booking Product Roadmap: Q4 2026 – Q1 2027 & Beyond

At $3.9M ARR across 1,450 independent physio clinics, our immediate focus is clear: ruthlessly align our capacity with the CEO's goals of cutting no-shows and reducing multi-site churn.

With two squads and 10 productive weeks per squad-quarter (20 total squad-weeks per quarter), here is our execution plan.

---

Q4 2026: Foundation & The Multi-Site Anchor

Goal Alignment: Multi-site churn, Support capacity.

Multi-Site Dashboard (10 squad-weeks – Squad A): A single view of every location’s bookings and utilization.
Why: Multi-site clinics drive 38% of our ARR ($1.48M) but churn at 2.9% (vs. 1.3% single-site). "Can't see all locations in one place" was cited in 74% of multi-site exit surveys. This is also the explicit "Must Have" for FitPhysio Group (30 sites, $64.8K ARR).
Messaging Service (8 squad-weeks – Squad B): Infrastructure for SMS provider, consent, and templates.
Why: Prerequisite for our Q1 no-show reduction engine.
Calendar Sync Fix (5 squad-weeks – Squad B, spilling into Q1): Resolving sync failures that drive 38% of support tickets.
Why: Engineering estimates this will permanently reclaim ~3 squad-weeks/quarter of engineering time starting Q1.

---

Q1 2027: Solving No-Shows

Goal Alignment: No-show reduction.

Calendar Sync Fix completion (0 extra weeks, Squad B finishing up).
SMS Reminders with Confirm/Cancel (6 squad-weeks – Squad B): Leveraging the Q4 Messaging Service.
Why: Our data proves this works: clinics manually texting reminders drop no-shows from 11% to 6%. Automated confirm/cancel turns passive ghosting into active rescheduling, opening slots early.
Physitrack Integration (8 squad-weeks – Squad A): Sends exercise plans directly to the Physitrack app.
Why: Sales commitment to close FitPhysio Group by Christmas. (Note: While FitPhysio listed this as "nice-to-have" vs. the dashboard's "must-have," it is a commercial blocker per Sales).
Waitlist Auto-Fill (7 squad-weeks – Squad A, utilizing reclaimed support capacity): Texts waitlisted patients when slots free up.
Why: Directly targets no-shows, but depends on SMS reminders and cancellations working first.

---

What Comes After (Q2 2027 & Beyond)

Deposits at Booking (14 squad-weeks): Paused until Legal secures the payments partner contract (earliest mid-February 2027). This will be our primary Q2/Q3 bet to further crush no-shows once infrastructure is live.

---

What We Are NOT Doing & Why

1. Booking Page Redesign (9 squad-weeks): * Why Not: 64% of visits already convert to bookings, and 18 customer interviews this quarter yielded zero mentions of the booking page. Spending design capacity here is a distraction from our retention and no-show crises. 2. Deposits in Q4/Q1: * Why Not: Blocked by external legal/contract timelines until mid-Q1 at best. 3. Prioritizing Physitrack over Multi-Site in Q4: * Why Not: While Sales needs Physitrack for FitPhysio, FitPhysio’s actual operational blocker is the Multi-Site Dashboard. Building the dashboard first secures our largest pipeline deal and addresses the core churn driver for our highest-value segment.

Claims the judge couldn’t find in the brief

Fixing calendar sync will permanently reclaim about 3 squad-weeks per quarter of engineering time starting Q1.
Clinics manually texting reminders drop no-shows from 11% to 6%.

Grades and run details

Decision model 32 · LLM judge 2 of 12 checks

Decision model checks

failUses the supplied evidence correctly13%
partialAddresses the actual decision30%
failRespects explicit constraints44%
failIdentifies material uncertainty25%
partialAvoids unsupported claims43%
failProduces the required deliverable27%
partialEvery commitment serves the goals44%
failHandles the Sales promise with evidence23%
passOutcomes, with certainty that falls with distance34%
failFits the stated capacity57%
passSequences around dependencies38%

Run

Run: #1
Time to output: 4 s
Submitted: 1 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 9

Uses the supplied evidence correctlyRightWrong

GPT-6.1 Sol · API

The output’s current-situation facts, figures, customer statements, estimates, dates, and capacity numbers all come from the supplied context or follow directly from it.