Tasks / Strategy

Develop product strategy

Can the model make a coherent choice grounded in the evidence, rather than list aspirations?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 33% were usable with at most a quick edit.

Reliably right

  1. Respects explicit constraints100% pass
    One-page, ~500 words, one major bet, addressed to the CEO for the board.
    GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
  2. Produces the required deliverable100% pass
    Complete strategy document in the requested form, length, and for the named reader; the board could act on it.
    GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
  3. Diagnosis before prescription100% pass
    Diagnosis: students can't find the right tutor; actions (intake, recommendations, rematch, manual matching) directly address it.
    GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace

Where it slips

  1. Names an advantage that's hard to copy10% pass
    Does not name a benefit and a barrier that a well-funded competitor couldn't copy; no moat is identified.
    GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
  2. Proposes tests that could fail46% pass
    The operating targets for mid-market and SMB lack explicit actions if missed; only the enterprise gate specifies what each result triggers.
    GPT-6.1 Sol · API · Enterprise, says the CEO
  3. Avoids unsupported claims54% pass
    Presents 'root cause: we've let the loudest requests set the roadmap' and 'SMB remains our top-of-funnel' as facts without labelling them as hypotheses or citing evidence.
    Haiku 5.5 · API · Enterprise, says the CEO

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Diagnosis before prescription

    Does the strategy name the crux, the specific obstacle or opportunity the evidence points to, and do the proposed actions follow from that diagnosis?

    Passes when States one crux grounded in the supplied evidence, and each major action visibly addresses it.

  2. Names an advantage that's hard to copy

    Does the strategy say what will make its position hard for an informed, well-funded competitor to copy (a benefit customers value plus a barrier to copying it), rather than relying on being first, on features, or on funding?

    Passes when Names the benefit and the barrier, and why a capable competitor couldn't simply replicate it.

  3. Says what it gives up

    Does the strategy name what the company will stop doing, or deliberately not do, in order to pursue it?

    Passes when Names at least one real thing being dropped or declined, with the reason it loses out.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Proposes tests that could fail · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

Our tutoring marketplace has stalled at about 4,000 bookings a month. Write a one-page strategy for the next 12 months (about 500 words) for our CEO to take to the board. We can fund one major bet.

What the model was given4 items: Scenario, Metrics, Research, Board view
ScenarioTutorly connects secondary-school students with independent tutors for online lessons at £30–45 an hour; we take a 20% commission. There are 6,500 active tutors and 11,000 active students. We have 18 months of runway and about £1.2m to put behind the one bet.
MetricsTutor supply up 60% year on year; bookings up 8%. 70% of tutors get zero bookings in a typical month. Repeat booking rate is 64% for students who book twice.
ResearchStudents cite 'finding the right tutor' as their top problem. Tutors cite 'not enough students'.
Board viewThe board believes paid acquisition of students is the answer.
What a strong answer doesThe answer key the graders mark against

Recognise the contradiction: supply is abundant but matching fails, so buying demand into a broken match pours water into a leaky bucket. Choose matching quality and say what is given up.

Critical failures (cap the score at 40)
  • Recommends paid demand acquisition without addressing matching
Task

Staff-level · v1.13

The outputs

9 setups answered this brief.

Needs more than a quick edit6 of 12 checks passed

Contains unsupported claims about dormant students and second-booking rate that would require fact-checking, making it unusable with just a quick edit.

Output

MEMORANDUM TO: The Board of Directors FROM: CEO, Tutorly DATE: October 24, 2023 SUBJECT: 12-Month Strategy: Unlocking Liquidity via Guided Matching

1. The Diagnosis: A Conversion Crisis, Not an Acquisition Crisis Tutorly has stalled at 4,000 monthly bookings, generating roughly £30,000 in monthly net revenue. While the Board has suggested deploying our £1.2m capital into paid student acquisition, the data indicates this would be a fatal mistake.

We do not have a top-of-funnel problem; we have a liquidity and matching problem. We currently have 11,000 active students on the platform, yet they generate only 4,000 bookings. Meanwhile, our tutor supply has grown 60% YoY to 6,500, leaving 70% of tutors with zero bookings a month.

Why are 11,000 active students not booking 6,500 available tutors? User research gives us the answer: students are paralyzed by the paradox of choice, citing "finding the right tutor" as their top barrier. Tutors think there aren't enough students, when in reality, bloated supply is actively hurting student conversion. Pouring £1.2m of paid marketing into a leaky bucket will only burn our 18-month runway faster.

However, our bright spot is retention: when a student books twice, their repeat rate is 64%. Our core strategic challenge is getting existing students from browsing to their second booking.

2. Our One Major Bet: "Tutorly Match" & Supply Curation Instead of buying traffic, we will invest our £1.2m in product-led conversion. We will pivot Tutorly from an open directory (like Craigslist) to a curated, algorithmically matched marketplace (like Uber).

Over the next 12 months, we will execute a three-part strategy:

  • Aggressive Supply Curation: We will freeze new tutor onboarding and delist the bottom 50% of inactive/low-converting tutors. By concentrating demand among the top 2,000 tutors, we instantly improve the average quality of a student’s first lesson and keep our best tutors earning and engaged.
  • The "Concierge" Matching Engine: We will replace our search bar with a diagnostic intake flow. Students will input their exam board, target grade, and learning style, and our algorithm will present only three ideal tutors.
  • The "Two-Lesson" Guarantee: Because a second booking triggers a 64% repeat rate, we will subsidize the friction of trial-and-error. If a student doesn’t click with their first matched tutor, Tutorly will fund the first lesson with a replacement tutor.

3. Capital Allocation (£1.2m) * £650k (Product & Data): Hiring a dedicated squad (PM, Data Scientist, 3 Engineers) to build the matching engine, diagnostic onboarding, and a revamped tutor rating system based on student retention. * £350k (The "Second Booking" Subsidy): Financing the "First-Match Guarantee" and discounting a bundled "First Two Lessons" package to force the habit loop. * £200k (Lifecycle Marketing): Automated CRM flows (SMS/Email) to re-engage the ~7,000 dormant students currently on our platform.

4. 12-Month Success Metrics By fixing the conversion bottleneck first, we build a machine that can profitably scale paid acquisition in Year 2. By Q4 next year, this bet will deliver: 1. Bookings: Increase from 4,000 to 12,000/month (tripling net revenue to ~£90k/month). 2. Student Activation: Increase the percentage of active students reaching their 2nd booking from ~15% to 40%. 3. Marketplace Health: Reduce the "zero-booking" tutor rate from 70% to under 20% (on a curated base of 2,500 tutors).

Claims the judge couldn’t find in the brief

  • ~7,000 dormant students currently on our platform
  • current percentage of active students reaching their 2nd booking is ~15%
  • students are paralyzed by the paradox of choice
  • bloated supply is actively hurting student conversion

Grades and run details

Decision model 58 · LLM judge 46
Decision model checks
  • failUses the supplied evidence correctly55%
  • failAddresses the actual decision45%
  • passRespects explicit constraints56%
  • failIdentifies material uncertainty66%
  • failAvoids unsupported claims69%
  • passProduces the required deliverable77%
  • passSurfaces the central contradiction100%
  • passMakes a clear choice100%
  • passDiagnosis before prescription100%
  • passProposes tests that could fail7%
  • failNames an advantage that's hard to copy27%
  • passSays what it gives up100%
Run
Run
#1
API response time
12 s
Submitted
13 Oct 2026

Needs more than a quick edit9 of 12 checks passed

The output invents a fact about not tracking conversion cleanly and presents several interpretations as established facts, so a PM would need to re-check evidence and correct unsupported claims before using it.

Output

Tutorly: 12-Month Strategy

Sep 24, 2026 · @Martin Slaney

The problem is matching, not demand1

Tutor supply grew 60% last year; bookings grew 8%. 70% of our 6,500 tutors get no bookings in a typical month, and our 11,000 active students generate only \~4,000 bookings a month. Both sides report the same failure from opposite ends: students can't find the right tutor; tutors can't find students. Adding supply has made the catalogue bigger and choosing harder.

The bright spot: students who book twice repeat at 64%. The relationship works once it starts.3 The leak is between arriving and the second lesson.

Why paid acquisition first is the wrong bet

Paid acquisition sends more students into the same funnel that already fails to convert most of the ones we have. We'd be buying students at full cost and losing them at the step before the relationship forms. At a 20% take on £30–45 lessons (\~£7.50 per booking), we only recover acquisition cost from students who stick; today too few get there. Fix conversion first, and every pound of acquisition afterwards is worth more.

The bet: managed matching

Replace browse-and-hope with matching we take responsibility for.

  • Guided intake. Subject, level, exam board, goals, learning style, budget, availability. Two minutes, not twenty profiles.
  • We recommend three tutors, ranked on fit and track record, not on who signed up most recently.
  • Low-risk first lesson. A discounted trial plus a "right tutor or your next lesson free" guarantee.
  • Curate supply. Stop broad tutor recruitment. Recruit only where demand outstrips supply (by subject and exam level); rank and surface tutors on conversion and repeat rate; move inactive tutors out of results.
  • Start with human matchers, then automate what works. The concierge data trains the algorithm.

Budget and milestones

Allocation£
Matching product, data and ranking (team of \~4)500k
Human matching concierge (pilot, then scale down)200k
Trial-lesson subsidy and guarantee150k
Gated acquisition tranche (released at month 6 only if targets hit)350k
Total1.2m

Month 3: matching live for new students in top 3 subjects. Baseline set for first-to-second-lesson conversion (we don't track this cleanly today2; it's the number that matters most).

Month 6 gate: first-to-second conversion up 50% on baseline; share of tutors with zero bookings below 55%. Hit it, and we release £350k into paid acquisition, now into a funnel that converts. Miss it, and we hold the cash as runway.

Month 12: 6,000 bookings a month (+50%), with growth coming from bookings per student, not student count alone.

Ask of the board

Back matching as the one bet, with acquisition as its reward, not its substitute. We're not against growth spend; we're against spending it before the product can hold the students it buys. With 18 months of runway, the month-6 gate gives us a clear read while two-thirds of the budget and a year of runway remain.

What a PM had to fix

  1. 1Hypothesis stated as factReframe it as a hypothesisTargeted repair

    Source: ResearchStudents cite 'finding the right tutor' as their top problem. Tutors cite 'not enough students'.

    What we’d changeFrame matching as the working hypothesis the research points to, and say what the first months will test, rather than stating it as the diagnosis.

  2. 2Invented evidenceVerify or remove the claimQuick edit

    What we’d changeRemove it: the brief doesn't say what's tracked. Say the baseline will be set in month 3.

  3. 3Hypothesis stated as factReframe it as a hypothesisQuick edit

    Source: MetricsRepeat booking rate is 64% for students who book twice.

    What we’d changeThe 64% applies to students who book twice. It doesn't show the relationship works for everyone who starts.

Claims the judge couldn’t find in the brief

  • today too few get there [students who stick]
  • we don't track this cleanly today [first-to-second-lesson conversion]
  • The relationship works once it starts.
  • Adding supply has made the catalogue bigger and choosing harder.
  • The leak is between arriving and the second lesson.

Grades and run details

Decision model 88 · LLM judge 69
Decision model checks
  • passUses the supplied evidence correctly59%
  • passAddresses the actual decision98%
  • passRespects explicit constraints61%
  • passIdentifies material uncertainty100%
  • failAvoids unsupported claims7%
  • passProduces the required deliverable89%
  • passSurfaces the central contradiction100%
  • passMakes a clear choice100%
  • passDiagnosis before prescription100%
  • passProposes tests that could fail49%
  • partialNames an advantage that's hard to copy38%
  • passSays what it gives up100%
Artefacts
Run
Run
#1
Time to output
62 s
Submitted
24 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Uses the supplied evidence correctlyWrongMixed
Gemini 4 Argon · API

Contains invented current data (7,000 dormant students, ~15% student second-booking rate) not supported by the brief or context.

Opus 5.5 · Claude

The output states as fact that 'we don't track this cleanly today' and presents interpretations like 'The relationship works once it starts' and 'today too few get there' as established facts, none of which appear in the supplied context.

Addresses the actual decisionWrongRight
Gemini 4 Argon · API

Does not state what result or condition would change the chosen bet.

Opus 5.5 · Claude

The output commits early to 'managed matching' as the one bet, frames it for the CEO/board, and specifies that missing the month-6 gate would change the call (hold cash as runway).

Identifies material uncertaintyWrongRight
Gemini 4 Argon · API

Does not name specific unknowns that could change the decision or how they would be resolved.

Opus 5.5 · Claude

It names the unknown first-to-second-lesson conversion rate, says it will set a baseline, and uses a month-6 gate with numeric thresholds to resolve whether to release acquisition spend.

Proposes tests that could failMixedRight
Gemini 4 Argon · API

No kill criteria, numeric thresholds, or measurement windows are specified for any test or pilot.

Opus 5.5 · Claude

The month-6 gate has numeric thresholds (conversion up 50%, zero-booking tutors below 55%), a 3-month measurement window, and clear actions for pass (release £350k) and fail (hold cash).

All got wrong 2

Avoids unsupported claimsWrongWrong
Gemini 4 Argon · API

Presents causal interpretations ('paralyzed by paradox of choice', 'bloated supply is actively hurting student conversion') as established fact without labelling them as hypotheses.

Opus 5.5 · Claude

Interpretations like 'The relationship works once it starts', 'The leak is between arriving and the second lesson', and 'today too few get there' are presented as confident facts without being labelled as hypotheses.

Names an advantage that's hard to copyWrongWrong
Gemini 4 Argon · API

Does not name a specific customer benefit and a barrier to copying; pivot to curated matching is described without explaining why it's defensible.

Opus 5.5 · Claude

The strategy does not name a benefit and a barrier that would make the position hard for a well-funded competitor to copy; it mentions human matchers and data training but never explains why a competitor couldn't replicate it.

All got right 6

Respects explicit constraintsRightRight
Gemini 4 Argon · API

Within ~500 words, presents one major bet, and is addressed to CEO for the board.

Opus 5.5 · Claude

The output is a one-page strategy of about 400 words, addresses the CEO for the board, and funds one major bet (matching) with a conditional acquisition tranche, respecting the brief's constraints.

Produces the required deliverableRightRight
Gemini 4 Argon · API

Delivers a complete one-page strategy memo usable by the CEO for the board.

Opus 5.5 · Claude

The output is a complete one-page strategy memo for the CEO to take to the board, within the requested length, and is usable with light edits.

Surfaces the central contradictionRightRight
Gemini 4 Argon · API

Clearly identifies that 60% supply growth only produced 8% bookings growth, showing mismatch.

Opus 5.5 · Claude

It explicitly notes that tutor supply grew 60% while bookings grew only 8%, surfacing the contradiction that supply growth hasn't produced bookings.

Makes a clear choiceRightRight
Gemini 4 Argon · API

Commits to matching quality bet and names what is sacrificed (paid acquisition, new tutor onboarding).

Opus 5.5 · Claude

The strategy chooses managed matching as the one bet and names what it forgoes: immediate paid acquisition and broad tutor recruitment.

Diagnosis before prescriptionRightRight
Gemini 4 Argon · API

States crux as matching/conversion problem, and all major actions (curation, matching engine, two‑lesson guarantee) address it.

Opus 5.5 · Claude

It diagnoses the crux as a matching failure between arrival and the second lesson, and all major actions (guided intake, recommendations, trial guarantee, supply curation) directly address that crux.

Says what it gives upRightRight
Gemini 4 Argon · API

States what is given up: paid student acquisition and new tutor onboarding/delisting inactive tutors.

Opus 5.5 · Claude

It explicitly says to stop broad tutor recruitment and defers paid acquisition, naming real things being given up with the reason they lose out.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 82% of 76 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT87.588.52None
2Haiku 5.5withAPI85.480.82None
3Sonnet 5.5withAPI87.580.82None
4GPT-6.1 SolwithAPI89.676.92None
5GPT-6 LunawithAPI83.376.92None
6Opus 5.5withClaude89.665.42None
7Gemini 3.8 FlashwithAPI62.557.72None
8Gemini 4 ArgonwithAPI66.753.82None
9Gemini 3.5 Flash-LitewithGemini66.742.32None

What you get for the cost

Haiku 5.5 · API scores within 4 points of GPT-6 Astra · ChatGPT at a fiftieth of the cost.

Task score against relative cost for each setup. The table below has the same figures.4050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 88, 84× the cheapest (1 of 2 outputs estimated)Haiku 5.5 · API: task score 84, 1.7× the cheapestSonnet 5.5 · API: task score 83, 27× the cheapestGPT-6.1 Sol · API: task score 82, 12× the cheapestGPT-6 Luna · API: task score 77, the cheapestOpus 5.5 · Claude: task score 74, 68× the cheapest (1 of 2 outputs estimated)Gemini 3.8 Flash · API: task score 58, 8.1× the cheapestGemini 4 Argon · API: task score 58, 12× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 52, 3.1× the cheapest (1 of 2 outputs estimated)GPT-6 Astra · ChatGPTHaiku 5.5 · APISonnet 5.5 · APIGPT-6.1 Sol · APIGPT-6 Luna · APIOpus 5.5 · ClaudeGemini 3.8 Flash · APIGemini 4 Argon · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API771.0×
Haiku 5.5 · API841.7×
Gemini 3.5 Flash-Lite · Gemini523.1× (partly estimated)
Gemini 3.8 Flash · API588.1×
GPT-6.1 Sol · API8212×
Gemini 4 Argon · API5812×
Sonnet 5.5 · API8327×
Opus 5.5 · Claude7468× (partly estimated)
GPT-6 Astra · ChatGPT8884× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Writing a strategy memo that leadership can act on.

Why it matters

Strategy that doesn't choose isn't strategy. Models write fluent strategic prose easily; making a choice the evidence supports and naming what it gives up is harder.

What good looks like

  • Diagnoses the one obstacle that matters before prescribing
  • Makes one clear choice and names what is sacrificed
  • Grounds the choice in the supplied evidence
  • Surfaces the tension or contradiction in the data
  • Defines how we would know it is working

Deliberately not measured

  • Market sizing accuracy beyond the supplied data
  • Financial modelling
Capability tested

Choosing where to play and what not to do, from supplied evidence

The failure we’re looking for

Strategic-sounding aspirations without a diagnosis or a choice

Grading

Decision model and LLM judge, calibrated against a blind PM review