Tasks / Strategy

Develop product strategy

Can the model make a coherent choice grounded in the evidence, rather than list aspirations?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 38% were usable with at most a quick edit.

Reliably right

  1. Produces the required deliverable100% pass
    Complete strategy document in the requested form, length, and for the named reader; the board could act on it.
    GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
  2. Diagnosis before prescription100% pass
    Diagnosis: students can't find the right tutor; actions (intake, recommendations, rematch, manual matching) directly address it.
    GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
  3. Tells the CEO plainly100% pass
    It clearly states the enterprise-first plan is wrong for next year, backs it with numbers, and presents the alternative.
    GPT-6 Astra · ChatGPT · Enterprise, says the CEO

Where it slips

  1. Names an advantage that's hard to copy11% pass
    Does not name a benefit and a barrier that a well-funded competitor couldn't copy; no moat is identified.
    GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
  2. Proposes tests that could fail42% pass
    The SMB pilot lacks a numeric threshold for success; while the payback guardrail has a threshold and action, not every proposed test/gate meets the full requirement.
    GPT-6 Astra · ChatGPT · Enterprise, says the CEO
  3. Avoids unsupported claims44% pass
    Presents 'root cause: we've let the loudest requests set the roadmap' and 'SMB remains our top-of-funnel' as facts without labelling them as hypotheses or citing evidence.
    Haiku 5.5 · API · Enterprise, says the CEO

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Diagnosis before prescription

    Does the strategy name the crux, the specific obstacle or opportunity the evidence points to, and do the proposed actions follow from that diagnosis?

    Passes when States one crux grounded in the supplied evidence, and each major action visibly addresses it.

  2. Names an advantage that's hard to copy

    Does the strategy say what will make its position hard for an informed, well-funded competitor to copy (a benefit customers value plus a barrier to copying it), rather than relying on being first, on features, or on funding?

    Passes when Names the benefit and the barrier, and why a capable competitor couldn't simply replicate it.

  3. Says what it gives up

    Does the strategy name what the company will stop doing, or deliberately not do, in order to pursue it?

    Passes when Names at least one real thing being dropped or declined, with the reason it loses out.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Proposes tests that could fail · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the VP Product at Cadence. Our CEO wants a strategy for next year the exec team can rally behind, and believes it's enterprise (the note is below). Write the strategy memo, in no more than 1,200 words: the diagnosis, the guiding policy (where we play and what we stop), the coherent actions for next year, and how we'll know it's working. Tell the CEO if you think they're wrong. The files are attached.

What the model was given6 items: About Cadence, ceo_note.md, unit_economics_by_segment.csv, enterprise_detail.md, mid_market_detail.md, smb_detail.md
About CadenceStaff scheduling and time tracking, sold to businesses with hourly workers.
ceo_note.md1 lines · Download
From the CEO: "I want a strategy for next year that the exec team can rally behind. I believe it's enterprise. Tell me if I'm wrong."
unit_economics_by_segment.csv3 rows · Download
segment,customers,avg_arr_usd,gross_margin_pct,support_cost_per_customer_year_usd,cac_usd,payback_months,net_revenue_retention_pct,logo_churn_year_pct
SMB self-serve,6200,1100,78,610,740,31,88,34
Mid-market,890,14500,81,1900,9800,9,118,7
Enterprise,34,96000,64,21000,118000,26,104,9
enterprise_detail.md4 lines · Download
# Enterprise detail

- Average sales cycle: 9 months. 5 of the last 8 deals needed custom features (built by the core team, about 14 engineer-weeks each).
- The CEO's plan: "Win enterprise. Hire 10 enterprise AEs and a solutions team, and build SSO, audit logs and custom workflows this year."
mid_market_detail.md5 lines · Download
# Mid-market detail

- Best-retaining segment. Expansion comes from adding locations and seats.
- Top requests: multi-location reporting, payroll integrations, shift templates.
- Win rate against our main competitor: 61%.
smb_detail.md4 lines · Download
# SMB self-serve detail

- 62% of support tickets come from SMB accounts. Free trial converts at 4%.
- Most SMB customers are on the $79/month plan with unlimited phone support.
What a strong answer doesThe answer key the graders mark against

Diagnoses from the unit economics: mid-market is the engine (payback 9 months, NRR 118%, churn 7%, wins 61% against the main competitor); SMB self-serve loses money once support and churn are counted (about $1,100 a year at 78% gross margin is about $860, minus $610 of support, leaves about $250 a year against $740 to acquire, with 34% leaving each year; payback 31 months, NRR 88%) and generates 62% of tickets; enterprise is slow and expensive (payback 26 months, 64% gross margin, 9-month cycles, 5 of 8 deals needing about 14 engineer-weeks of custom work). Tells the CEO the enterprise bet is wrong for next year: it would pull the core team into custom work and stretch payback. Policy: win mid-market, expanding on locations and seats; stop subsidising SMB support (change the plan or pricing, self-serve support, raise the entry price) rather than abandon it; serve enterprise only where it buys the product as is. Coherent actions: mid-market roadmap (multi-location reporting, payroll integrations, shift templates), SMB support and pricing changes, no custom enterprise work. Names what it gives up and how it'll know (mid-market NRR and payback, SMB support cost per customer).

Critical failures (cap the score at 40)
  • Recommends hiring the enterprise team and building custom workflows as next year's main strategy
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Usable with a quick edit11 of 12 checks passed

The memo is clear, evidence-based, and actionable; a product manager could use it with minor wording edits and no re-checking of evidence or reasoning.

Output

Cadence strategy memo: next year

To: CEO and exec team From: VP Product

Short answer

I don't think enterprise should be the strategy. I think mid-market should. Enterprise should stay an option we keep cheap and revisit with evidence. You're right that we need a higher ceiling than SMB gives us, but the data says mid-market is the route to it.

Diagnosis

Approximate ARR is customers × average ARR. Contribution is ARR × gross margin, minus support cost.

ARRContributionPaybackNRRLogo churn
SMB~$6.8M~$1.5M31 mo88%34%
Mid-market~$12.9M~$8.8M9 mo118%7%
Enterprise~$3.3M~$1.4M26 mo104%9%
  1. Mid-market is the business. It is about 56% of ARR and roughly 75% of contribution. It pays back in 9 months, expands on its own as customers add locations and seats, and we win 61% against our main competitor. Customers are telling us what they want: multi-location reporting, payroll integrations, shift templates. We are under-serving our best segment.
  1. SMB is a leaky bucket that we subsidize. It is 62% of tickets, which matches its ~61% share of support spend. The $79 plan includes unlimited phone support, so a $1,100 account costs $610 a year to serve. Trial converts at 4%, churn is 34%, and payback is 31 months. That is roughly the average customer lifetime, so we barely break even on the typical account. But SMB is still 30% of revenue and a cheap feeder. The fix is to repair it, not to exit.
  1. Enterprise, as currently run, has the worst unit economics we have. Gross margin is 64%, support costs $21k per customer, the sales cycle is 9 months, and payback is 26 months. Five of the last eight deals needed custom features at ~14 engineer-weeks each, which means we are paying for product work out of the core team's capacity. NRR of 104% and churn of 9% are fine but not special.
  1. The proposed plan scales the weakest model. Ten AEs plus a solutions team plausibly costs on the order of $3M+ a year fully loaded (my estimate, so finance should check it). That is about the size of today's entire enterprise ARR, with 9-month cycles and ramp time, so most revenue lands after next year. Custom workflows would deepen the bespoke-work problem. And 34 customers is a thin sample to bet the company on.

What I don't know. I have no enterprise pipeline data, no enterprise win rate, and no view of mid-market market size or channel saturation. I also don't know whether SSO and audit logs are blocking mid-market deals. Those gaps shape the tripwires below.

Guiding policy

Win multi-location mid-market. Make SMB profitable. Keep enterprise as a gated option.

Where we play - Core: hourly-workforce businesses with multiple locations, in the mid-market range. Growth comes from landing and expanding across locations. - Feeder: SMB self-serve, run for margin and for conversion into mid-market, not for volume. - Option: enterprise deals we can close with the standard product and little custom work.

What we stop - Hiring the 10-AE enterprise team and the dedicated solutions team. - Bespoke feature builds for individual deals. Any enterprise requirement must be built as a standard product capability or declined. - Unlimited phone support bundled into the $79 plan. - Spending acquisition dollars on SMB channels that can't pay back within about 18 months.

Coherent actions

1. Point the roadmap at mid-market (about 60% of product and engineering capacity). Ship multi-location reporting first, then payroll integrations (starting with the two or three most requested), then shift templates. Sequence by expansion impact: which features most increase locations and seats per account.

2. Put the sales investment where payback is 9 months. Redirect the headcount budget to mid-market: a handful of AEs plus customer success focused on expansion. Add a multi-location expansion motion, such as account reviews triggered when a customer opens a new location. Size the hiring to what mid-market pipeline can absorb. Prove that first with a small cohort before scaling.

3. Fix SMB economics (about 20% of capacity). - Move phone support off the $79 plan. Offer it as a paid add-on or a higher tier, with chat, email, and in-product help as the default. Grandfather existing customers for a period and test the change on a cohort before a full rollout, because churn is already 34%. - Work the trial: onboarding, templates, and time to first published schedule. Moving conversion from 4% to 6% beats buying more traffic. - Build a clear upgrade path for SMB customers who add locations.

4. Run enterprise as a disciplined option (about 10–15% of capacity). - Serve and renew the existing 34. Set a target for support cost per account and margin, and review the biggest cost drivers. - Pursue inbound and referral enterprise deals with a small team: one senior AE plus existing sales leadership, no new organization. - Build SSO and audit logs as standard platform features, but only after win/loss analysis confirms they block mid-market and upper-mid-market deals. If they do, they pay off in both segments. If not, they wait. - Stop custom workflows. Where customers need them, define a configurable, productized version or turn the deal down.

5. Instrument everything first (first 60 days). Get win/loss reasons by segment, enterprise pipeline and win rate, and fully loaded cost per enterprise deal (including those custom builds). This turns the open questions above into data.

How we'll know it's working

Proposed targets, to be pressure-tested with finance:

AreaMetricTarget
Mid-marketARR+35–45%
NRR≥115%
Win rate vs. main competitor≥60% as volume grows
CAC payback≤12 months
Adoption of multi-location reporting and payroll integrationsMajority of accounts with 2+ locations
SMBSupport cost per customer$610 → ≤$400
Share of support tickets62% → under 45%
Trial conversion4% → 6%
Logo churn34% → ≤30%, after any support change
EnterpriseCustom engineer-weeksZero new bespoke builds
NRR≥100%
MarginMoving toward SMB/mid-market levels

Tripwires. I'd revisit the enterprise question at mid-year, and invest more if all of these hold: - At least 3 enterprise-sized deals closed with minimal custom work (about 2 engineer-weeks or less each). - Enterprise support cost per account falling. - Pipeline data showing a repeatable motion, not one-off wins.

I'd pull back on the SMB support change if churn among affected accounts rises beyond the savings. I'd slow mid-market hiring if win rate or payback degrades as we scale.

The ask

Align on mid-market as the center of next year's plan, with SMB repaired and enterprise as an option. If you still believe enterprise is the bigger long-term prize, the way to get there is to make mid-market large and stable first. That funds the bet and gives enterprise buyers a product with SSO, audit logs, and multi-location reporting built for real customers. I'd rather be wrong about enterprise on a small budget than be right with the team stretched across three segments.

Grades and run details

Decision model 79 · LLM judge 92
Decision model checks
  • partialUses the supplied evidence correctly2%
  • passAddresses the actual decision100%
  • passRespects explicit constraints45%
  • passIdentifies material uncertainty100%
  • partialAvoids unsupported claims38%
  • passProduces the required deliverable72%
  • passReads the segment economics92%
  • passTells the CEO plainly100%
  • passDiagnosis before prescription100%
  • partialProposes tests that could fail81%
  • failNames an advantage that's hard to copy67%
  • passSays what it gives up100%
Run
Run
#1
API response time
46 s
Submitted
6 Oct 2026

Check by check

Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Got wrong 1

Names an advantage that's hard to copyWrong

Does not name a hard-to-copy advantage; relies on features and expansion without a barrier a capable competitor couldn't replicate.

Got right 11

Uses the supplied evidence correctlyRight

All factual claims about the current situation are taken directly from the supplied context or derived correctly by arithmetic.

Addresses the actual decisionRight

Commits early to mid-market over enterprise, and specifies tripwires that would change the answer.

Respects explicit constraintsRight

Delivers a strategy memo under 1,200 words, addressed to the CEO and exec team, with all requested sections.

Identifies material uncertaintyRight

Names missing data (pipeline, win rate, market size, SSO impact) and sets tripwires to resolve them.

Avoids unsupported claimsRight

Interpretations and estimates are clearly labelled as such, and confident claims are supported by the evidence.

Produces the required deliverableRight

Complete, usable strategy memo with diagnosis, policy, actions, and metrics; a reader could act on it with light edits.

Reads the segment economicsRight

Works through payback, retention, and support cost for all three segments, correctly identifying mid-market as strongest, SMB as unprofitable after support, and enterprise as slow and costly.

Tells the CEO plainlyRight

Directly states the enterprise-first plan is wrong, supports it with numbers, and offers a clear alternative.

Diagnosis before prescriptionRight

Diagnosis identifies mid-market as the under-served engine, and every major action addresses that crux.

Proposes tests that could failRight

Tripwires include numeric thresholds (3 deals, ≤2 engineer-weeks), a mid-year window, and actions for each outcome.

Says what it gives upRight

Explicitly stops enterprise AE hiring, custom builds, unlimited SMB phone support, and unprofitable SMB acquisition spend.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 84% of 76 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT83.388.52None
2Haiku 5.5withAPI77.180.82None
3Sonnet 5.5withAPI77.180.82None
4GPT-6.1 SolwithAPI79.276.92None
5GPT-6 LunawithAPI77.176.92None
6Opus 5.5withClaude83.365.42None
7Gemini 3.8 FlashwithAPI68.857.72None
8Gemini 3.5 Flash-LitewithGemini60.442.32None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.4050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 87, 84× the cheapest (1 of 2 outputs estimated)Haiku 5.5 · API: task score 80, 1.7× the cheapestSonnet 5.5 · API: task score 78, 27× the cheapestGPT-6.1 Sol · API: task score 77, 12× the cheapestGPT-6 Luna · API: task score 74, the cheapestOpus 5.5 · Claude: task score 71, 68× the cheapest (1 of 2 outputs estimated)Gemini 3.8 Flash · API: task score 61, 8.1× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 49, 3.1× the cheapest (1 of 2 outputs estimated)GPT-6 Astra · ChatGPTHaiku 5.5 · APISonnet 5.5 · APIGPT-6.1 Sol · APIGPT-6 Luna · APIOpus 5.5 · ClaudeGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and costCost partly estimated (app runs with no usage meter)
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API741.0×
Haiku 5.5 · API801.7×
Gemini 3.5 Flash-Lite · Gemini493.1× (partly estimated)
Gemini 3.8 Flash · API618.1×
GPT-6.1 Sol · API7712×
Sonnet 5.5 · API7827×
Opus 5.5 · Claude7168× (partly estimated)
GPT-6 Astra · ChatGPT8784× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Writing a strategy memo that leadership can act on.

Why it matters

Strategy that doesn't choose isn't strategy. Models write fluent strategic prose easily; making a choice the evidence supports and naming what it gives up is harder.

What good looks like

  • Diagnoses the one obstacle that matters before prescribing
  • Makes one clear choice and names what is sacrificed
  • Grounds the choice in the supplied evidence
  • Surfaces the tension or contradiction in the data
  • Defines how we would know it is working

Deliberately not measured

  • Market sizing accuracy beyond the supplied data
  • Financial modelling
Capability tested

Choosing where to play and what not to do, from supplied evidence

The failure we’re looking for

Strategic-sounding aspirations without a diagnosis or a choice

Grading

Decision model and LLM judge, calibrated against a blind PM review