Tasks / Define

Develop product strategy

Can the model make a coherent choice grounded in the evidence, rather than list aspirations?

Measures the modelTask v1.1 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 14 graded outputs by 7 models. 64% were usable with at most a quick edit.

Reliably right

  1. Produces the required deliverable100% pass
    It is a complete one-page strategy for the CEO/board, with staged funding, metrics, and risk controls, usable as-is with light edits.
    GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
  2. Diagnosis before prescription100% pass
    It diagnoses the mismatch between abundant supply and failed matching, and the proposed actions directly target tutor choice and repeat bookings.
    GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
  3. Surfaces the central contradiction100% pass
    It explicitly states supply grew 60% while bookings grew only 8%, and that 70% of tutors get no bookings.
    GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace

Where it slips

  1. Proposes tests that could fail32% pass
    The months 7–12 gate says 'gains persist and contribution economics support expansion' without a numeric threshold or defined contribution target, so not every gate names a threshold.
    GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
  2. Uses the supplied evidence correctly39% pass
    The output makes unsupported claims about the current system (open-ended tutor browsing as default journey, existence of data on subject/exam-board segments) that are not in the supplied context.
    GPT-6.1 Sol · API · Supply or demand for a stalled marketplace
  3. Avoids unsupported claims41% pass
    Presents as fact that the current default journey is open-ended tutor browsing and that data on highest-volume subject/exam-board segments exists, neither of which is in the supplied evidence.
    GPT-6.1 Sol · API · Supply or demand for a stalled marketplace

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

Our tutoring marketplace has stalled at about 4,000 bookings a month. Write a one-page strategy for the next 12 months (about 500 words) for our CEO to take to the board. We can fund one major bet.

What the model was given4 items: Scenario, Metrics, Research, Board view
ScenarioTutorly connects secondary-school students with independent tutors for online lessons at £30–45 an hour; we take a 20% commission. There are 6,500 active tutors and 11,000 active students. We have 18 months of runway and about £1.2m to put behind the one bet.
MetricsTutor supply up 60% year on year; bookings up 8%. 70% of tutors get zero bookings in a typical month. Repeat booking rate is 64% for students who book twice.
ResearchStudents cite 'finding the right tutor' as their top problem. Tutors cite 'not enough students'.
Board viewThe board believes paid acquisition of students is the answer.
What a strong answer doesThe answer key the graders mark against

Recognise the contradiction: supply is abundant but matching fails, so buying demand into a broken match pours water into a leaky bucket. Choose matching quality and say what is given up.

Critical failures (cap the score at 40)
  • Recommends paid demand acquisition without addressing matching
Case

Staff-level · v1.9 · synthetic · marketplace, board pressure

The outputs

7 setups answered this brief. Pick up to three to read side by side.

Needs more than a quick edit3 of 10 checks passed

A PM would need to remove invented current-state claims and add a hard decision gate and explicit sacrifice before using it. The reasoning is directionally strong but not evidence-safe as written.

Output

MEMORANDUM

TO: Board of Directors, Tutorly FROM: Chief Executive Officer DATE: October 2023 SUBJECT: 12-Month Strategic Plan: Unlocking Growth via Managed Matching

---

Executive Summary: Diagnosing the Growth Plateau

Our marketplace has stalled at 4,000 bookings/month (~£30k monthly net revenue) despite healthy user numbers (11k students, 6.5k tutors).

The Board’s instinct to deploy our £1.2m reserve into paid student acquisition is understandable, but premature. The data reveals a liquidity and conversion bottleneck, not an awareness problem: * Supply is bloated and disengaged: Tutor supply grew 60% YoY, yet 70% receive zero bookings monthly. * Friction kills discovery: Students cite “finding the right tutor” as their primary obstacle. Our open-directory model forces parents to sift through 6,500 profiles, causing decision paralysis and drop-off. * Underlying product-market fit is proven: When students reach a second booking, 64% retain.

Pouring paid traffic into an unguided directory will burn our £1.2m on high CAC and low conversion. Our single bet for the next 12 months is transitioning from an open directory to an intelligent, Managed Matching Engine.

---

The Single Bet: "The Perfect Match" Engine

Instead of browsing endless profiles, parents will complete a 60-second diagnostic intake (curriculum tier, personality, learning barriers, availability). Our matching engine pairs them with the single best tutor—backed by a 100% “First Lesson Satisfaction Guarantee” (rebooked free if not a fit).

By actively directing demand to a curated, high-performing top quartile of our supply (~1,500 tutors), we solve both sides of the marketplace: students find immediate trust, and active tutors earn dependable income.

#### Allocation of the £1.2m Capital: 1. Product & Matching Tech (£500k): Algorithmic intake flow, automated calendar sync, and in-app instant booking to eliminate messaging back-and-forth. 2. Onboarding & Operations (£300k): Quality vetting, tiering supply, and a proactive concierge team to manually rescue failed first sessions. 3. Targeted Performance Marketing (£400k): Phased acquisition deployed only once the match-to-book conversion rate improves.

---

12-Month Execution Roadmap

  • Q1–Q2 (Fix the Engine):
  • Replace public browse with the guided diagnostic matching flow.
  • Restrict active routing to the top 25% highest-rated tutors to concentrate liquidity.
  • Launch the "Fit Guarantee" to de-risk the first booking.
  • Q3 (Solve the 2nd Booking):
  • Automate recurring weekly reservation slots right after lesson one (optimizing for the 64% retention threshold).
  • Introduce 4-lesson starter packs to mathematically guarantee lesson two.
  • Q4 (Scale What Works):
  • Deploy the remaining £400k into intent-driven paid channels (Google Search for GCSE/A-Level revision), scaling into a proven, high-converting funnel.

---

Targets & Milestones

MetricCurrent12-Month Target
:---:---:---
Monthly Bookings4,00010,000+
Search-to-First-Booking CVRBaseline+40%
% Tutors with ≥1 Booking/mo30%65% (of active tier)
Annualized Net Revenue (20%)£360k£900k+

Conclusion: Fixing search friction activates the high-retention engine we already possess. By funding matching instead of brute-force acquisition, we reach sustainable unit economics and double our run-rate within our existing 18-month runway.

Claims the judge couldn’t find in the brief

  • Tutorly currently operates an open-directory model in which parents sift through 6,500 profiles.
  • The current model causes decision paralysis and drop-off.
  • The problem is a liquidity and conversion bottleneck, not an awareness problem.
  • Paid traffic into an unguided directory will burn £1.2m on high CAC and low conversion.
  • There is a high-performing top quartile of supply of about 1,500 tutors.

Grades and run details

Decision model 65 · LLM judge 3 of 11 checks
Decision model checks
  • failUses the supplied evidence correctly76%
  • passAddresses the actual decision69%
  • passRespects explicit constraints40%
  • partialIdentifies material uncertainty42%
  • failAvoids unsupported claims85%
  • passProduces the required deliverable67%
  • passSurfaces the central contradiction100%
  • passMakes a clear choice95%
  • passDiagnosis before prescription100%
  • failProposes tests that could fail28%
Run
Run
#1
API response time
16 s
Submitted
1 Oct 2026

Usable with a quick edit8 of 10 checks passed

The strategy is complete, evidence-based, and actionable; the only small gap is a numeric threshold for the final expansion gate, which is a quick edit.

Output

Tutorly: turn tutor choice into sustained bookings

12-month strategy | Board decision requested

Recommendation

Commit our one major bet to helping students find a suitable tutor and establish a recurring lesson habit. Authorise up to £1.2m in staged funding. Defer broad paid student acquisition until we demonstrate that we can reliably convert demand into repeat bookings at sustainable economics.

Why this bet

Tutor supply has grown 60%, while bookings have grown just 8%. With 70% of tutors receiving no bookings in a typical month, recruiting more supply is unlikely to unlock growth. Students identify finding the right tutor as their biggest problem. Our working hypothesis is that Tutorly offers access without enough confidence to choose.

The board’s acquisition thesis remains plausible: tutors report insufficient students. But that describes their experience, not necessarily the underlying constraint. More traffic could help, or could send more students into an ineffective matching journey.

The reported 64% repeat rate among students who book twice suggests potential value after an initial relationship forms. It does not establish retention across all students or explain losses before the second booking. We must validate its definition and measurement window.

What we will build

Start with the subject and year-group segment showing the strongest combination of existing demand, available tutors and repeat behaviour. Offer students a short intake covering learning goals, syllabus, budget and availability, followed by three suitable, available tutors with clear reasons for each recommendation.

After the first lesson, check fit, offer a straightforward rematch where needed and make recurring booking easy. Begin with manually assisted matching to learn what predicts successful relationships before automating it. Broad tutor recruitment and unrelated marketplace expansion will be deferred.

Execution and investment

In months 1–2, release £150,000 to establish the funnel from qualified enquiry through first, second and subsequent lessons, interview students who abandoned booking and run a controlled matching pilot. Audit “active” user definitions: the headline counts do not establish usable demand or supply.

In months 3–6, release a further £350,000 only if the pilot improves second completed bookings within 30 days by at least 20% relative to the existing journey, without materially increasing refunds or complaints. This is a proposed investment threshold, not a forecast; sample size and measurement rules will be agreed before testing.

In months 7–12, release the remaining £700,000 only if gains persist and contribution economics support expansion. Automate proven matching steps and extend into adjacent segments. Small paid-acquisition tests can then establish whether additional demand scales profitably.

Success and downside protection

Target 8,000 completed monthly bookings by month 121, with term-time and holiday comparisons reported separately. Track contribution after payment costs, support, incentives and matching operations alongside booking growth.

Assuming one-hour bookings, current commission revenue is only £24,000–£36,000 monthly. Doubling bookings would not itself establish sustainability. Acquisition spending must therefore recover its cost from observed contribution within six months, rather than speculative lifetime value.

If matching fails the month-six gate, stop further funding and reassess the acquisition thesis. With 18 months of runway, preserving capital when evidence disappoints is part of the strategy.

What a PM had to fix

  1. 1Test or gate too weakTighten the testQuick edit

    What we’d changeGive the target a basis, a short bridge showing the conversion, booking-frequency or demand change it needs, and say exactly when the month-6 gate is judged.

Grades and run details

Decision model 85 · LLM judge 10 of 11 checks
Decision model checks
  • failUses the supplied evidence correctly11%
  • passAddresses the actual decision100%
  • passRespects explicit constraints58%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims76%
  • passProduces the required deliverable71%
  • passSurfaces the central contradiction100%
  • passMakes a clear choice100%
  • passDiagnosis before prescription100%
  • partialProposes tests that could fail50%
Run
Run
#1
Time to output
39 s
Submitted
24 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 6

Uses the supplied evidence correctlyWrongMixed
Gemini 3.8 Flash · API

It invents current-state facts such as an open-directory model, parents sifting through profiles, decision paralysis, and a high-performing top quartile of about 1,500 tutors.

GPT-6 Astra · ChatGPT

All current-situation facts used are from the metrics, research, scenario, or arithmetic from them, with assumptions clearly labelled.

Addresses the actual decisionMixedRight
Gemini 3.8 Flash · API

It commits to managed matching, but does not clearly state what result or condition would change that decision.

GPT-6 Astra · ChatGPT

It commits early to matching quality as the one bet and says the month-six failure would stop funding and reassess acquisition.

Respects explicit constraintsMixedRight
Gemini 3.8 Flash · API

It allocates £400k to paid acquisition as part of the plan without a concrete gate, so it does not fully enforce the one-major-bet constraint.

GPT-6 Astra · ChatGPT

It respects the one-bet funding constraint, 12-month horizon, CEO/board audience, and approximate 500-word length.

Identifies material uncertaintyWrongRight
Gemini 3.8 Flash · API

It does not name material unknowns such as whether matching improves conversion, whether top tutors have capacity, or whether the guarantee is affordable.

GPT-6 Astra · ChatGPT

It names the unproven matching hypothesis and repeat-rate measurement as uncertainties, with a numeric month-six gate to resolve them.

Avoids unsupported claimsWrongRight
Gemini 3.8 Flash · API

It presents several interpretations and forecasts as established fact, including high CAC, low conversion, decision paralysis, and guaranteed lesson two.

GPT-6 Astra · ChatGPT

Causal and forward-looking statements are framed as hypotheses, possibilities, or assumptions rather than established fact.

Makes a clear choiceMixedRight
Gemini 3.8 Flash · API

It chooses matching, but does not clearly name what is sacrificed, and it still funds paid acquisition without a hard threshold.

GPT-6 Astra · ChatGPT

It chooses matching quality as the one bet and names what it forgoes: broad paid acquisition, broad tutor recruitment, and unrelated marketplace expansion.

All got wrong 1

Proposes tests that could failWrongWrong
Gemini 3.8 Flash · API

The only gate, deploying paid marketing once match-to-book conversion improves, lacks a numeric threshold, measurement window, and explicit action for failure.

GPT-6 Astra · ChatGPT

The months 7–12 gate says 'gains persist and contribution economics support expansion' without a numeric threshold or defined contribution target, so not every gate names a threshold.

All got right 3

Produces the required deliverableRightRight
Gemini 3.8 Flash · API

It is a board-ready one-page strategy memo for the CEO, roughly within the requested length and usable with light edits.

GPT-6 Astra · ChatGPT

It is a complete one-page strategy for the CEO/board, with staged funding, metrics, and risk controls, usable as-is with light edits.

Surfaces the central contradictionRightRight
Gemini 3.8 Flash · API

It explicitly identifies that tutor supply grew 60% while 70% of tutors receive zero bookings and bookings grew only 8%.

GPT-6 Astra · ChatGPT

It explicitly states supply grew 60% while bookings grew only 8%, and that 70% of tutors get no bookings.

Diagnosis before prescriptionRightRight
Gemini 3.8 Flash · API

It diagnoses matching/search friction as the crux and proposes intake, routing, guarantee, and recurring booking actions aimed at that crux.

GPT-6 Astra · ChatGPT

It diagnoses the mismatch between abundant supply and failed matching, and the proposed actions directly target tutor choice and repeat bookings.

Results

Every setup we’ve tested on this task, across all cases and repeats, graded on the current checklist. Calibrated: the graders match our PM on 84% of checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 LunawithAPI87.290.52None
2GPT-6.1 SolwithAPI87.285.92None
3Sonnet 5.5withAPI79.285.92None
4GPT-6 AstrawithChatGPT89.775.52None
5Opus 5.5withClaude78.971.42None
6Gemini 3.5 Flash-LitewithGemini58.357.72None
7Gemini 3.8 FlashwithAPI71.438.62None

About the task

The PM job

Writing a strategy memo that leadership can act on.

Why it matters

Strategy that doesn't choose isn't strategy. Models write fluent strategic prose easily; making a choice the evidence supports and naming what it gives up is harder.

What good looks like

  • Diagnoses the one obstacle that matters before prescribing
  • Makes one clear choice and names what is sacrificed
  • Grounds the choice in the supplied evidence
  • Surfaces the tension or contradiction in the data
  • Defines how we would know it is working

Deliberately not measured

  • Market sizing accuracy beyond the supplied data
  • Financial modelling
Capability tested

Choosing where to play and what not to do, from supplied evidence

The failure we’re looking for

Strategic-sounding aspirations without a diagnosis or a choice

Grading

Decision model and LLM judge, calibrated against a blind PM review