Tasks / Strategy

Develop product strategy

Can the model make a coherent choice grounded in the evidence, rather than list aspirations?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 38% were usable with at most a quick edit.

Reliably right

  1. Produces the required deliverable100% pass
    Complete strategy document in the requested form, length, and for the named reader; the board could act on it.
    GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
  2. Diagnosis before prescription100% pass
    Diagnosis: students can't find the right tutor; actions (intake, recommendations, rematch, manual matching) directly address it.
    GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
  3. Tells the CEO plainly100% pass
    It clearly states the enterprise-first plan is wrong for next year, backs it with numbers, and presents the alternative.
    GPT-6 Astra · ChatGPT · Enterprise, says the CEO

Where it slips

  1. Names an advantage that's hard to copy11% pass
    Does not name a benefit and a barrier that a well-funded competitor couldn't copy; no moat is identified.
    GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
  2. Proposes tests that could fail42% pass
    The SMB pilot lacks a numeric threshold for success; while the payback guardrail has a threshold and action, not every proposed test/gate meets the full requirement.
    GPT-6 Astra · ChatGPT · Enterprise, says the CEO
  3. Avoids unsupported claims44% pass
    Presents 'root cause: we've let the loudest requests set the roadmap' and 'SMB remains our top-of-funnel' as facts without labelling them as hypotheses or citing evidence.
    Haiku 5.5 · API · Enterprise, says the CEO

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Diagnosis before prescription

    Does the strategy name the crux, the specific obstacle or opportunity the evidence points to, and do the proposed actions follow from that diagnosis?

    Passes when States one crux grounded in the supplied evidence, and each major action visibly addresses it.

  2. Names an advantage that's hard to copy

    Does the strategy say what will make its position hard for an informed, well-funded competitor to copy (a benefit customers value plus a barrier to copying it), rather than relying on being first, on features, or on funding?

    Passes when Names the benefit and the barrier, and why a capable competitor couldn't simply replicate it.

  3. Says what it gives up

    Does the strategy name what the company will stop doing, or deliberately not do, in order to pursue it?

    Passes when Names at least one real thing being dropped or declined, with the reason it loses out.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Proposes tests that could fail · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

Our tutoring marketplace has stalled at about 4,000 bookings a month. Write a one-page strategy for the next 12 months (about 500 words) for our CEO to take to the board. We can fund one major bet.

What the model was given4 items: Scenario, Metrics, Research, Board view
ScenarioTutorly connects secondary-school students with independent tutors for online lessons at £30–45 an hour; we take a 20% commission. There are 6,500 active tutors and 11,000 active students. We have 18 months of runway and about £1.2m to put behind the one bet.
MetricsTutor supply up 60% year on year; bookings up 8%. 70% of tutors get zero bookings in a typical month. Repeat booking rate is 64% for students who book twice.
ResearchStudents cite 'finding the right tutor' as their top problem. Tutors cite 'not enough students'.
Board viewThe board believes paid acquisition of students is the answer.
What a strong answer doesThe answer key the graders mark against

Recognise the contradiction: supply is abundant but matching fails, so buying demand into a broken match pours water into a leaky bucket. Choose matching quality and say what is given up.

Critical failures (cap the score at 40)
  • Recommends paid demand acquisition without addressing matching
Task

Staff-level · v1.12

The outputs

8 setups answered this brief.

Usable with a quick edit8 of 12 checks passed

The strategy is actionable, evidence-based, and framed for the board; a PM could add a numeric threshold to the Month 3 gate and a brief moat argument with a quick edit without redoing any reasoning.

Output

Tutorly: 12-Month Strategy Board paper, for decision

The problem Bookings have grown 8% while tutor supply grew 60%. We are not short of tutors or students. We are failing to connect them. Students say their top problem is finding the right tutor. Tutors say they lack students, and 70% get no bookings in a typical month. Once a student books twice, 64% keep booking, so the product works when a match happens. The leak is in the first match.

Why not paid student acquisition The board's instinct is to buy more students. I recommend against it for now: - Paid students would arrive at the same broken search and matching flow that already frustrates them. We would be paying to acquire customers we are likely to lose before their second booking. - Our tutor base is already larger than our demand. More students would mainly bring more tutors into the same competition for the few bookings available. - Acquisition spend scales with cost. Matching improvements scale with use.

The bet: fix first-match conversion (£1.2m) Over 12 months we will build a guided matching system and test it against the current search:

  1. Structured intake (Months 1–4). Students give subject, exam board, year group, goals, budget, and availability. Required before search.
  2. Curated shortlists (Months 2–6). Within one hour, students receive three ranked tutors with a free 15-minute intro call. A small concierge team handles hard cases such as niche subjects and last-minute exam prep.
  3. Ranking that rewards fit and responsiveness (Months 3–8). Ranking weighs subject expertise, response time, and student outcomes. New tutors get guaranteed early exposure so that supply does not stall at the bottom.
  4. Tutor activation (Months 4–12). Tutors with zero bookings in 30 days receive coaching on profile quality and availability, plus targeted introductions.

Budget (£1.2m) - Product and engineering: £600k - Concierge matching team: £300k - Tutor activation and targeted student tests: £200k - Measurement and contingency: £100k

Targets (12 months) - Bookings: 4,000 → 5,500 per month - Students with a first booking within 7 days of intake: baseline in Month 1, target +50% - Share of tutors with at least one booking per month: ~30% → 45% - Repeat rate (second booking): hold at or above 64%

Decision gates - Month 3: Intake and shortlist live for a test cohort. Confirm the conversion baseline and the uplift. - Month 6: If first-match conversion has not improved by at least 20% against control, stop further build and move remaining funds to a small, controlled paid-acquisition test. If it has, scale the concierge model and begin targeted acquisition in segments where our supply is deepest. - Month 9: Review unit economics. Each booking should contribute enough commission to cover concierge cost.

Runway This plan spends £1.2m over 12 months and leaves about six months of runway for the gates above. If the Month 6 gate fails, we retain the option to pivot without running out of cash.

What we are asking the board to approve 1. Fund the £1.2m matching-led plan. 2. Approve the Month 6 gate and the reallocation rule if it fails. 3. Defer paid student acquisition until conversion data supports it.

The board's instinct is right that growth needs more students. But we already have more tutors than we can match, and the evidence points to the match as the bottleneck. Fixing that first makes every future pound of acquisition work harder.

Grades and run details

Decision model 71 · LLM judge 85
Decision model checks
  • failUses the supplied evidence correctly18%
  • passAddresses the actual decision100%
  • passRespects explicit constraints35%
  • passIdentifies material uncertainty100%
  • failAvoids unsupported claims34%
  • passProduces the required deliverable89%
  • passSurfaces the central contradiction100%
  • passMakes a clear choice100%
  • passDiagnosis before prescription100%
  • partialProposes tests that could fail48%
  • failNames an advantage that's hard to copy46%
  • passSays what it gives up100%
Run
Run
#1
API response time
16 s
Submitted
8 Oct 2026

Needs more than a quick edit5 of 12 checks passed

The output contains unsupported claims and lacks uncertainty, tests, moat, and tradeoff discussion, requiring rework before use.

Output

Tutorly: 12-Month Growth Strategy (Board Briefing)

Executive Summary

Tutorly has reached a critical inflection point, plateauing at roughly 4,000 bookings per month. While tutor supply has surged by 60% year-on-year, demand has lagged behind, with bookings up only 8%. Consequently, 70% of our active tutors receive zero bookings in a typical month, threatening supply quality and platform retention.

With 18 months of runway and £1.2m in capital, we face a strategic choice. While the board traditionally advocates for paid student acquisition, qualitative research reveals that our primary bottleneck is not awareness, but friction in matching. Students cite “finding the right tutor” as their top hurdle, while tutors report “not enough students.” Throwing capital at top-of-funnel paid acquisition will merely leak budget into an inefficient discovery funnel.

Instead, our single major bet for the next 12 months is to re-engineer the marketplace discovery and matching engine, transforming Tutorly from a static directory into an intelligent, high-conversion matching platform.

---

The Core Problem & Diagnosis

Our core metrics tell a clear story of imbalance: * The Supply Glut: 6,500 active tutors vs. 11,000 active students, resulting in 70% of tutors experiencing zero-booking months. * The Demand Efficiency: Repeat booking is healthy at 64% for students who book twice, proving that product-market fit exists once a match is made4. The failure is entirely in initial activation and discovery.1

---

The 12-Month Strategic Bet: Intelligent Matchmaking & Activation

We will deploy our £1.2m budget across three synchronized pillars over the next 12 months to drive bookings from 4,000 to 10,000+ monthly transactions.

1. Algorithmic Matching & Guided Discovery (Months 1–4)

  • Action: Shift from manual search filters to an AI-driven, intent-based matching questionnaire for incoming students. Students specify learning style, goals, budget (£30–45/hr), and schedule, and receive top-3 instant tutor recommendations.
  • Investment: £400k in product engineering, UX redesign, and recommendation algorithms.

2. Supply-Side Activation & Tiered Visibility (Months 5–8)

  • Action: Address the 70% zero-booking tutor rate by introducing performance-backed visibility tiers, response-time incentives, and automated profile optimization tools. Tutors who maintain high responsiveness and trial conversion rates are prioritized in the new matching engine.
  • Investment: £300k in tutor success tooling, automated coaching, and CRM workflows.2

3. Targeted Demand Amplification (Months 9–12)

  • Action: With conversion rates optimized, deploy the remaining £500k into targeted performance marketing and SEO. Rather than broad, leaky acquisition campaigns, we will target high-intent secondary-school parents with hyper-specific landing pages tied directly to instant matching funnels.
  • Investment: £500k in performance channels, referral loops, and parent trust initiatives.

---

Financial Impact & Board Recommendation

By fixing the matching bottleneck first, we project a rise in trial-to-booking conversion rates from current lows to over 25%, lifting monthly bookings past 10,0003 and securing a sustainable, high-margin marketplace model well within our 18-month runway.

We request board approval to allocate the £1.2m capital to this intelligent matching strategy, prioritizing structural efficiency over superficial top-of-funnel spend.

What a PM had to fix

  1. 1Hypothesis stated as factReframe it as a hypothesisStart again

    Source: ResearchStudents cite 'finding the right tutor' as their top problem. Tutors cite 'not enough students'.

    What we’d changePresent matching friction as the hypothesis the research suggests, not a proven diagnosis, and stage the spending so the board learns before it commits the full £1.2m.

  2. 2Constraint missedRestore the constraintSubstantial rework

    Source: BriefWe can fund one major bet.

    What we’d changeCut back to one bet. Tutor tooling needs its own case when supply already far outstrips bookings.

  3. 3Invented evidenceVerify or remove the claimTargeted repair

    What we’d changeRemove the forecast or show the model: no conversion baseline is given, and nothing connects 25% to 10,000 bookings.

  4. 4Hypothesis stated as factReframe it as a hypothesisQuick edit

    Source: MetricsRepeat booking rate is 64% for students who book twice.

    What we’d changeA 64% repeat rate among two-time bookers suggests value after a match. It doesn't prove product-market fit.

Claims the judge couldn’t find in the brief

  • trial-to-booking conversion rates from current lows
  • The failure is entirely in initial activation and discovery.

Grades and run details

Decision model 50 · LLM judge 38
Decision model checks
  • failUses the supplied evidence correctly32%
  • partialAddresses the actual decision36%
  • partialRespects explicit constraints30%
  • failIdentifies material uncertainty83%
  • failAvoids unsupported claims87%
  • passProduces the required deliverable43%
  • passSurfaces the central contradiction99%
  • passMakes a clear choice67%
  • passDiagnosis before prescription100%
  • failProposes tests that could fail89%
  • failNames an advantage that's hard to copy34%
  • passSays what it gives up87%
Artefacts
Run
Run
#1
Time to output
48 s
Submitted
24 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 5

Uses the supplied evidence correctlyMixedWrong
Haiku 5.5 · API

All factual statements about the current situation appear in the brief or context, or follow from the supplied numbers.

Gemini 3.5 Flash-Lite · Gemini

Claims about 'trial-to-booking conversion rates from current lows' and that 'the failure is entirely in initial activation and discovery' are not supported by the supplied evidence.

Addresses the actual decisionRightWrong
Haiku 5.5 · API

Commits early to matching as the one bet, defers paid acquisition, and names the Month 6 conversion improvement condition that would change the answer.

Gemini 3.5 Flash-Lite · Gemini

Does not state what result or condition would change the decision.

Identifies material uncertaintyRightWrong
Haiku 5.5 · API

Month 6 gate sets a clear numeric threshold and action; the plan acknowledges the risk and a pivot option if the bet fails.

Gemini 3.5 Flash-Lite · Gemini

No unknowns, risks, or conditions that could change the decision are named.

Avoids unsupported claimsMixedWrong
Haiku 5.5 · API

Interpretations like 'not short of students' and 'leak is in the first match' are grounded in the supplied evidence and not presented as unsupported assertions.

Gemini 3.5 Flash-Lite · Gemini

Presents 'trial-to-booking conversion rates from current lows' as fact and 'the failure is entirely in initial activation and discovery' as established, without labelling as hypothesis.

Says what it gives upRightMixed
Haiku 5.5 · API

Explicitly defers paid student acquisition, naming the sacrificed growth lever and the reason (conversion data needed first).

Gemini 3.5 Flash-Lite · Gemini

Does not name any existing activity or investment that will be stopped or deliberately not done.

All got wrong 2

Proposes tests that could failWrongWrong
Haiku 5.5 · API

The Month 3 gate lacks a numeric threshold, does not specify what result would stop or continue, and thus doesn't fully meet the 'test that could fail' requirement.

Gemini 3.5 Flash-Lite · Gemini

No tests, gates, or kill criteria with numeric thresholds and measurement windows are proposed.

Names an advantage that's hard to copyWrongWrong
Haiku 5.5 · API

Does not name a benefit customers value plus a barrier that would make the position hard for a well-funded competitor to copy.

Gemini 3.5 Flash-Lite · Gemini

Does not name a benefit and barrier that would make the position hard for a well-funded competitor to copy.

All got right 5

Respects explicit constraintsRightRight
Haiku 5.5 · API

Delivers a ~500-word one-page board paper that respects the single-bet budget and runway.

Gemini 3.5 Flash-Lite · Gemini

Output is a one-page strategy within ~500 words, commits to one major bet, and is addressed to the CEO for the board.

Produces the required deliverableRightRight
Haiku 5.5 · API

Output is a complete board-ready paper within the length, formatted for the CEO to take to the board and act upon.

Gemini 3.5 Flash-Lite · Gemini

The output is a complete one-page strategy document that the CEO could take to the board with light edits.

Surfaces the central contradictionRightRight
Haiku 5.5 · API

Explicitly states that supply grew 60% while bookings grew only 8%, highlighting the disconnect that the matching bet addresses.

Gemini 3.5 Flash-Lite · Gemini

Identifies that tutor supply grew 60% but bookings only 8%, and 70% of tutors get zero bookings.

Makes a clear choiceRightRight
Haiku 5.5 · API

Chooses the matching bet and clearly defers paid student acquisition, naming what is given up.

Gemini 3.5 Flash-Lite · Gemini

Chooses re-engineering matching as the single major bet and implicitly forgoes paid acquisition as the primary bet.

Diagnosis before prescriptionRightRight
Haiku 5.5 · API

Diagnoses the first-match conversion as the crux and then proposes structured intake, curated shortlists, ranking, and tutor activation that all address it.

Gemini 3.5 Flash-Lite · Gemini

Diagnoses matching friction as the crux and proposes actions that directly address it.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 84% of 76 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT83.388.52None
2Haiku 5.5withAPI77.180.82None
3Sonnet 5.5withAPI77.180.82None
4GPT-6.1 SolwithAPI79.276.92None
5GPT-6 LunawithAPI77.176.92None
6Opus 5.5withClaude83.365.42None
7Gemini 3.8 FlashwithAPI68.857.72None
8Gemini 3.5 Flash-LitewithGemini60.442.32None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.4050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 87, 84× the cheapest (1 of 2 outputs estimated)Haiku 5.5 · API: task score 80, 1.7× the cheapestSonnet 5.5 · API: task score 78, 27× the cheapestGPT-6.1 Sol · API: task score 77, 12× the cheapestGPT-6 Luna · API: task score 74, the cheapestOpus 5.5 · Claude: task score 71, 68× the cheapest (1 of 2 outputs estimated)Gemini 3.8 Flash · API: task score 61, 8.1× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 49, 3.1× the cheapest (1 of 2 outputs estimated)GPT-6 Astra · ChatGPTHaiku 5.5 · APISonnet 5.5 · APIGPT-6.1 Sol · APIGPT-6 Luna · APIOpus 5.5 · ClaudeGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and costCost partly estimated (app runs with no usage meter)
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API741.0×
Haiku 5.5 · API801.7×
Gemini 3.5 Flash-Lite · Gemini493.1× (partly estimated)
Gemini 3.8 Flash · API618.1×
GPT-6.1 Sol · API7712×
Sonnet 5.5 · API7827×
Opus 5.5 · Claude7168× (partly estimated)
GPT-6 Astra · ChatGPT8784× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Writing a strategy memo that leadership can act on.

Why it matters

Strategy that doesn't choose isn't strategy. Models write fluent strategic prose easily; making a choice the evidence supports and naming what it gives up is harder.

What good looks like

  • Diagnoses the one obstacle that matters before prescribing
  • Makes one clear choice and names what is sacrificed
  • Grounds the choice in the supplied evidence
  • Surfaces the tension or contradiction in the data
  • Defines how we would know it is working

Deliberately not measured

  • Market sizing accuracy beyond the supplied data
  • Financial modelling
Capability tested

Choosing where to play and what not to do, from supplied evidence

The failure we’re looking for

Strategic-sounding aspirations without a diagnosis or a choice

Grading

Decision model and LLM judge, calibrated against a blind PM review