Tasks / Strategy

Develop product strategy

Can the model make a coherent choice grounded in the evidence, rather than list aspirations?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 38% were usable with at most a quick edit.

Reliably right

  1. Produces the required deliverable100% pass
    Complete strategy document in the requested form, length, and for the named reader; the board could act on it.
    GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
  2. Diagnosis before prescription100% pass
    Diagnosis: students can't find the right tutor; actions (intake, recommendations, rematch, manual matching) directly address it.
    GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
  3. Tells the CEO plainly100% pass
    It clearly states the enterprise-first plan is wrong for next year, backs it with numbers, and presents the alternative.
    GPT-6 Astra · ChatGPT · Enterprise, says the CEO

Where it slips

  1. Names an advantage that's hard to copy11% pass
    Does not name a benefit and a barrier that a well-funded competitor couldn't copy; no moat is identified.
    GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
  2. Proposes tests that could fail42% pass
    The SMB pilot lacks a numeric threshold for success; while the payback guardrail has a threshold and action, not every proposed test/gate meets the full requirement.
    GPT-6 Astra · ChatGPT · Enterprise, says the CEO
  3. Avoids unsupported claims44% pass
    Presents 'root cause: we've let the loudest requests set the roadmap' and 'SMB remains our top-of-funnel' as facts without labelling them as hypotheses or citing evidence.
    Haiku 5.5 · API · Enterprise, says the CEO

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Diagnosis before prescription

    Does the strategy name the crux, the specific obstacle or opportunity the evidence points to, and do the proposed actions follow from that diagnosis?

    Passes when States one crux grounded in the supplied evidence, and each major action visibly addresses it.

  2. Names an advantage that's hard to copy

    Does the strategy say what will make its position hard for an informed, well-funded competitor to copy (a benefit customers value plus a barrier to copying it), rather than relying on being first, on features, or on funding?

    Passes when Names the benefit and the barrier, and why a capable competitor couldn't simply replicate it.

  3. Says what it gives up

    Does the strategy name what the company will stop doing, or deliberately not do, in order to pursue it?

    Passes when Names at least one real thing being dropped or declined, with the reason it loses out.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Proposes tests that could fail · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the VP Product at Cadence. Our CEO wants a strategy for next year the exec team can rally behind, and believes it's enterprise (the note is below). Write the strategy memo, in no more than 1,200 words: the diagnosis, the guiding policy (where we play and what we stop), the coherent actions for next year, and how we'll know it's working. Tell the CEO if you think they're wrong. The files are attached.

What the model was given6 items: About Cadence, ceo_note.md, unit_economics_by_segment.csv, enterprise_detail.md, mid_market_detail.md, smb_detail.md
About CadenceStaff scheduling and time tracking, sold to businesses with hourly workers.
ceo_note.md1 lines · Download
From the CEO: "I want a strategy for next year that the exec team can rally behind. I believe it's enterprise. Tell me if I'm wrong."
unit_economics_by_segment.csv3 rows · Download
segment,customers,avg_arr_usd,gross_margin_pct,support_cost_per_customer_year_usd,cac_usd,payback_months,net_revenue_retention_pct,logo_churn_year_pct
SMB self-serve,6200,1100,78,610,740,31,88,34
Mid-market,890,14500,81,1900,9800,9,118,7
Enterprise,34,96000,64,21000,118000,26,104,9
enterprise_detail.md4 lines · Download
# Enterprise detail

- Average sales cycle: 9 months. 5 of the last 8 deals needed custom features (built by the core team, about 14 engineer-weeks each).
- The CEO's plan: "Win enterprise. Hire 10 enterprise AEs and a solutions team, and build SSO, audit logs and custom workflows this year."
mid_market_detail.md5 lines · Download
# Mid-market detail

- Best-retaining segment. Expansion comes from adding locations and seats.
- Top requests: multi-location reporting, payroll integrations, shift templates.
- Win rate against our main competitor: 61%.
smb_detail.md4 lines · Download
# SMB self-serve detail

- 62% of support tickets come from SMB accounts. Free trial converts at 4%.
- Most SMB customers are on the $79/month plan with unlimited phone support.
What a strong answer doesThe answer key the graders mark against

Diagnoses from the unit economics: mid-market is the engine (payback 9 months, NRR 118%, churn 7%, wins 61% against the main competitor); SMB self-serve loses money once support and churn are counted (about $1,100 a year at 78% gross margin is about $860, minus $610 of support, leaves about $250 a year against $740 to acquire, with 34% leaving each year; payback 31 months, NRR 88%) and generates 62% of tickets; enterprise is slow and expensive (payback 26 months, 64% gross margin, 9-month cycles, 5 of 8 deals needing about 14 engineer-weeks of custom work). Tells the CEO the enterprise bet is wrong for next year: it would pull the core team into custom work and stretch payback. Policy: win mid-market, expanding on locations and seats; stop subsidising SMB support (change the plan or pricing, self-serve support, raise the entry price) rather than abandon it; serve enterprise only where it buys the product as is. Coherent actions: mid-market roadmap (multi-location reporting, payroll integrations, shift templates), SMB support and pricing changes, no custom enterprise work. Names what it gives up and how it'll know (mid-market NRR and payback, SMB support cost per customer).

Critical failures (cap the score at 40)
  • Recommends hiring the enterprise team and building custom workflows as next year's main strategy
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Needs more than a quick edit6 of 12 checks passed

The memo contains factual inaccuracies (incorrect SMB average price and net-negative margin claim) that a PM would need to correct before use; otherwise the reasoning is sound but the errors undermine trust in the evidence.

Output

MEMORANDUM

TO: CEO FROM: VP of Product DATE: October 24, 2023 SUBJECT: Product & Go-To-Market Strategy for Next Year

*

You asked for a strategy the executive team can rally behind, and stated your belief that our future lies in enterprise.

You are wrong. If we pivot our resources toward enterprise next year, we will stall our fastest-growing segment, burn cash on an unscalable product architecture, and distract the company from our true economic engine.

Here is our reality, our diagnosis, and the strategy we should actually pursue.

---

1. The Diagnosis: Where the Numbers Really Point

To evaluate your enterprise hypothesis, we have to look at the unit economics across our three segments (SMB, Mid-Market, Enterprise):

  • SMB Self-Serve (6,200 customers, $1,100 ARR): This segment is an anchor dragging us down. A 34% logo churn and 88% NRR means we are running to stay in place. Worse, 62% of our support tickets come from SMB accounts who pay an average of $79/month while demanding unlimited phone support. We are subsidizing their operations at a severe net-negative margin once support costs are factored in.
  • Enterprise (34 customers, $96,000 ARR): Enterprise looks glossy on the top line, but the mechanics are broken. Our payback period is 26 months, our gross margin has cratered to 64% (due to heavy support and custom engineering), and NRR is a mediocre 104%. As noted in the enterprise detail, 5 of our last 8 deals required custom features built by our core engineering team—averaging 14 engineer-weeks each. We are currently operating as a custom software consultancy for large accounts, not a scalable SaaS business.
  • Mid-Market (890 customers, $14,500 ARR): This is our crown jewel. Mid-market boasts an 81% gross margin, a lightning-fast 9-month payback period, a stellar 118% NRR, and a low 7% logo churn. We win 61% of deals against our main competitor here. Expansion happens organically as clients add locations and seats.

The Diagnosis: We are suffering from strategic schizophrenia. We are spending engineering cycles building bespoke features for 34 enterprise logos while burning support bandwidth on 6,200 low-end SMBs. Meanwhile, we are under-investing in the Mid-Market—the only segment where we have product-market fit, high retention, and clear competitive dominance.

---

2. The Guiding Policy

Where We Play: We will focus our go-to-market, product development, and company roadmap entirely on the Mid-Market (companies with roughly 100 to 1,000 hourly workers).

What We Stop: 1. We stop custom enterprise builds. We will not pull core engineers off the roadmap to build one-off workflows for enterprise deals. If a whale customer requires custom engineering, we walk away. 2. We stop optimizing for SMB acquisition. We will sunset active acquisition efforts for self-serve SMBs and strip away unscalable touchpoints (like unlimited phone support for $79/mo accounts) to stop the margin bleed. 3. We stop treating Enterprise as our primary growth vector. We will not hire 10 enterprise AEs or build a dedicated solutions team.

By pulling back from Enterprise and SMB, we concentrate all our capital on scaling what is already working brilliantly in the Mid-Market.

---

3. Coherent Actions for Next Year

To capture the mid-market opportunity, product and go-to-market will execute three synchronized moves:

#### Action 1: Productize for Mid-Market Expansion (Product & Engineering) Instead of building custom enterprise workflows (SSO, audit logs, and bespoke routing), our engineering team will build the top three requested features that drive multi-location expansion and stickiness in the Mid-Market: * Multi-location reporting: Giving regional managers roll-up visibility across disparate sites. * Robust payroll integrations: Deepening native syncs with major mid-market payroll providers to reduce administrative friction. * Advanced shift templates: Allowing complex scheduling rules to be deployed instantly across new locations.

#### Action 2: Pivot Go-To-Market to Land-and-Expand (Sales & Marketing) We will reallocate budget from SMB acquisition and enterprise pipe-dreams directly into Mid-Market account executive and customer success headcount. * Our motion will focus on landing a mid-market business at a single location or division, then leveraging our stellar NRR (118%) to expand horizontally into new locations. * We will introduce tiered packaging that naturally captures revenue as mid-market clients add locations and seats.

#### Action 3: Streamline or Harvest the SMB Tier (Operations & Support) We cannot drop our existing 6,200 SMB customers overnight, but we must stop bleeding margin on them. * Transition self-serve SMB support away from expensive phone queues to a self-service knowledge base and AI-driven chat. * Raise prices on legacy low-tier plans to match the true cost-to-serve, accepting natural churn as a feature, not a bug, which frees up support capacity for our mid-market clients.

---

4. How We'll Know It's Working

If we execute this focus on the Mid-Market, we will see specific operational shifts within the next 12 months:

  1. Mid-Market ARR Growth: Mid-market ARR growth rate accelerates by 35% year-over-year as we direct 80% of our sales and marketing capacity here.
  2. Payback and Margins: Our blended gross margin climbs back toward 80%+ as high-margin mid-market revenue outpaces low-margin enterprise custom work and unprofitable SMB volume. Payback periods across the active customer base remain under 10 months.
  3. NRR Expansion: Mid-market NRR increases from 118% to 122%+ driven by faster multi-location adoption of our new reporting and payroll features.
  4. Engineering Velocity: Feature delivery velocity increases by 40% because engineering is no longer interrupted by 14-week custom enterprise implementations.

Summary

Pivoting to enterprise feels ambitious, but the data tells us it is a trap. We would be trading a high-margin, highly defensible leadership position in the Mid-Market to become a low-margin, high-friction consultancy for a handful of giants.

Let’s double down on where we win, where retention is highest, and where our unit economics sing. Let's own the Mid-Market next year.

Grades and run details

Decision model 71 · LLM judge 46
Decision model checks
  • partialUses the supplied evidence correctly3%
  • passAddresses the actual decision45%
  • passRespects explicit constraints54%
  • failIdentifies material uncertainty99%
  • failAvoids unsupported claims42%
  • passProduces the required deliverable76%
  • passReads the segment economics71%
  • passTells the CEO plainly100%
  • passDiagnosis before prescription100%
  • partialProposes tests that could fail49%
  • partialNames an advantage that's hard to copy59%
  • passSays what it gives up100%
Run
Run
#1
Time to output
9 s
Submitted
5 Oct 2026

Usable with a quick edit9 of 12 checks passed

Minor unsupported claims can be removed or rephrased in an edit; the core evidence and logic are sound and the memo is actionable.

Output

To: CEO From: VP Product Re: Next-year strategy

Bottom line: I think you're wrong. We are a mid-market company, and the enterprise plan would spend our scarcest resources, engineering time and hiring budget, on the segment with the weakest economics. Enterprise can be a real option later. It shouldn't be the strategy for next year.

Diagnosis

Annualized from the unit economics file (my calculations, not reported figures):

SMB self-serveMid-marketEnterprise
Customers6,20089034
ARR (share)~$6.8M (30%)~$12.9M (56%)~$3.3M (14%)
Gross margin78%81%64%
Support cost / customer / yr$610$1,900$21,000
Gross profit less support / customer~$250~$9,850~$40,400
CAC$740$9,800$118,000
Payback (months)31926
NRR88%118%104%
Logo churn34%7%9%

Mid-market is our engine. It produces most of our ARR, expands at 118% NRR, pays back in 9 months, and churns at 7%. We win 61% of head-to-head deals against our main competitor. Customers are asking for multi-location reporting, payroll integrations, and shift templates, which are the features that drive expansion.

SMB is a support sink. SMB accounts generate 62% of support tickets. Support costs about $3.8M a year against roughly $6.8M of SMB ARR, so over half of SMB revenue goes to support. Most SMB customers are on a $79/month plan with unlimited phone support. Churn is 34%, trial conversion is 4%, and lifetime contribution barely covers CAC.

Enterprise is a real market but a poor fit for us today. We have 34 logos. Sales cycles run nine months. Gross margin is 64%, and payback is 26 months before custom work is counted. Five of our last eight enterprise deals needed custom features built by the core team at about 14 engineer-weeks each. That is roughly 70 engineer-weeks, or about 1.5 engineer-years, in eight deals. The plan doesn't account for that cost.

Root cause: We've let the loudest requests set the roadmap and support model. Enterprise asks for custom builds, SMB asks for phone support, and neither request is what drives our best economics.

Guiding policy

Where we play: Mid-market multi-location employers with hourly workforces. This is the customer we win, keep, and expand, and where our product is strongest.

What we stop: 1. Custom builds for individual deals. No bespoke engineering without a paid statement of work that covers full cost and passes a "does this generalize?" test. 2. Unlimited phone support for SMB. SMB moves to self-serve, chat, and knowledge-base support, with paid priority support as an option. 3. Dedicated enterprise hiring and outbound. Enterprise becomes inbound-only, served by the mid-market product. We don't build a headcount plan around it. 4. Single-segment edge-case features that don't help mid-market or the platform.

What we don't stop: The 34 enterprise customers keep full service. We don't reprice or abandon them. SMB remains our top-of-funnel, but it stops being a support-heavy product.

Coherent actions for next year

  1. Build the mid-market expansion product. Ship multi-location reporting first, then shift templates, then the two payroll integrations that come up most in mid-market accounts. This gets roughly 70% of core product capacity.
  1. Move the enterprise hiring budget to mid-market. Use it for mid-market AEs (where a 61% win rate and 9-month payback make each dollar work faster) and for expansion coverage on multi-location accounts.
  1. Restructure SMB support. Remove unlimited phone support. Create a clear upgrade path to mid-market plans when an SMB account adds a second location. Use product changes to deflect the most common tickets. The goal is to halve support cost per SMB customer.
  1. Productize enterprise table stakes, nothing more. Build SSO and audit logs once as standard features. Some mid-market buyers need them too, and building them once is cheaper than building them repeatedly for deals. Don't build custom workflows. Keep the 34 enterprise accounts well served through CS.
  1. Govern the roadmap. A product council with exec sign-off reviews any request that isn't on the mid-market or platform roadmap. Every custom request must pass the segment-fit test.

Capacity split (proposed): ~70% mid-market roadmap, ~15% platform (SSO, audit logs, reliability), ~15% SMB self-serve and ticket deflection.

How we'll know it's working

Mid-market (primary): - NRR ≥ 120% (from 118%) - Logo churn ≤ 7% - Win rate vs. main competitor ≥ 61% - Payback ≤ 9 months on new mid-market CAC - Share of expansion ARR from added locations and seats, tracked quarterly

SMB: - Support cost per SMB customer down from $610 to ≤ $300 - SMB tickets per customer down year over year - Gross profit after support positive per customer on the new support model - SMB churn trending down; we accept some SMB logo loss as the price of the change

Enterprise: - Zero unplanned custom engineering weeks outside approved SOWs - Enterprise gross margin trending up toward 70%+ - No loss in renewals among the 34 existing accounts

Company: - Blended gross margin up, driven by the mix shift - Total ARR growth on the mid-market base

Revisit enterprise in 12 months if mid-market hits its targets and we see a steady flow of inbound enterprise deals (for example, three per quarter) that fit the standard product without custom work.

Risks

  • We'll lose some SMB and enterprise revenue. The bet is that mid-market growth and margin improvement outweigh it. We should show the exec team the expected net effect before we commit.
  • Mid-market buyers may demand SSO and audit logs sooner. Building them as standard features in the platform bucket covers this.
  • The enterprise story matters to investors and some of the team. We should explain it as focus with a defined path back, not as retreat.

I'm glad to walk through the model and the capacity assumptions with you before the exec session. My recommendation is to take this to the team as the choice between being a strong mid-market company now and a weak enterprise company later.

Claims the judge couldn’t find in the brief

  • Root cause: 'We've let the loudest requests set the roadmap and support model. Enterprise asks for custom builds, SMB asks for phone support, and neither request is what drives our best economics.'
  • SMB remains our top-of-funnel.

Grades and run details

Decision model 83 · LLM judge 77
Decision model checks
  • passUses the supplied evidence correctly35%
  • passAddresses the actual decision100%
  • passRespects explicit constraints48%
  • passIdentifies material uncertainty51%
  • partialAvoids unsupported claims52%
  • passProduces the required deliverable89%
  • passReads the segment economics98%
  • passTells the CEO plainly100%
  • passDiagnosis before prescription100%
  • partialProposes tests that could fail79%
  • failNames an advantage that's hard to copy46%
  • passSays what it gives up100%
Run
Run
#1
API response time
36 s
Submitted
8 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Uses the supplied evidence correctlyWrongMixed
Gemini 3.5 Flash-Lite · Gemini

Claims about SMB average price ($79/mo) and net-negative margin are not supported by the supplied evidence; the average ARR is $91.67/mo and gross profit after support is positive.

Haiku 5.5 · API

Two statements about current situation (root cause and SMB as top-of-funnel) are not from the brief nor arithmetic.

Addresses the actual decisionMixedRight
Gemini 3.5 Flash-Lite · Gemini

The output does not state what result or condition would change the recommended decision.

Haiku 5.5 · API

Commits early to disagreeing with the CEO, says enterprise is the wrong bet for next year, and specifies conditions to revisit.

Identifies material uncertaintyWrongRight
Gemini 3.5 Flash-Lite · Gemini

No material unknowns that could change the decision are identified or discussed.

Haiku 5.5 · API

Names the risk of SMB/enterprise revenue loss and the condition (mid-market targets + inbound enterprise flow) to revisit the enterprise decision in 12 months.

Proposes tests that could failWrongRight
Gemini 3.5 Flash-Lite · Gemini

Success metrics are given with thresholds but no triggers or specified actions if they are not met.

Haiku 5.5 · API

Proposes numeric thresholds (NRR ≥ 120, churn ≤ 7%, support cost ≤ $300, zero custom weeks) and a 12-month revisit gate with specific conditions.

All got wrong 2

Avoids unsupported claimsWrongWrong
Gemini 3.5 Flash-Lite · Gemini

Interpretations such as 'severe net-negative margin' and 'we are under-investing' are presented as fact without hedging, and the net-negative claim is factually incorrect.

Haiku 5.5 · API

Presents 'root cause: we've let the loudest requests set the roadmap' and 'SMB remains our top-of-funnel' as facts without labelling them as hypotheses or citing evidence.

Names an advantage that's hard to copyWrongWrong
Gemini 3.5 Flash-Lite · Gemini

No distinct barrier to competition or moat is described; the strategy relies on features and market focus.

Haiku 5.5 · API

Does not name a benefit plus a barrier that would make the position hard for a well-funded competitor to copy; relies on features and current win rate.

All got right 6

Respects explicit constraintsRightRight
Gemini 3.5 Flash-Lite · Gemini

The memo respects the requested form (strategy memo with sections), addresses the CEO, and is within 1,200 words.

Haiku 5.5 · API

The memo contains diagnosis, guiding policy, actions, and measures, is under 1,200 words, and addresses the CEO.

Produces the required deliverableRightRight
Gemini 3.5 Flash-Lite · Gemini

The memo is complete, structured, and actionable for the CEO with at most light edits.

Haiku 5.5 · API

Complete strategy memo in the requested form, addressed to the CEO, within word limit, and directly usable by the exec team.

Reads the segment economicsRightRight
Gemini 3.5 Flash-Lite · Gemini

The memo uses numbers from all three segments, including SMB support burden, to conclude mid-market is strongest and enterprise is costly.

Haiku 5.5 · API

Works through payback, retention, and support cost for all three segments and concludes mid-market is the strongest, SMB loses after support, enterprise is slow and costly.

Tells the CEO plainlyRightRight
Gemini 3.5 Flash-Lite · Gemini

Directly tells the CEO the enterprise plan is wrong, supported by numbers, and offers an alternative.

Haiku 5.5 · API

States plainly and respectfully that the enterprise-first plan is wrong, backed by the segment numbers.

Diagnosis before prescriptionRightRight
Gemini 3.5 Flash-Lite · Gemini

Diagnosis of strategic schizophrenia is stated, and actions (productize for mid-market, pivot GTM, streamline SMB) address it.

Haiku 5.5 · API

Names a crux (resource allocation driven by loudest requests) and all major actions address it.

Says what it gives upRightRight
Gemini 3.5 Flash-Lite · Gemini

Explicitly stops custom enterprise builds, SMB acquisition optimization, and enterprise as primary growth vector.

Haiku 5.5 · API

Names clear stops: custom builds for deals, unlimited SMB phone support, dedicated enterprise hiring and outbound, single-segment edge-case features.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 84% of 76 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT83.388.52None
2Haiku 5.5withAPI77.180.82None
3Sonnet 5.5withAPI77.180.82None
4GPT-6.1 SolwithAPI79.276.92None
5GPT-6 LunawithAPI77.176.92None
6Opus 5.5withClaude83.365.42None
7Gemini 3.8 FlashwithAPI68.857.72None
8Gemini 3.5 Flash-LitewithGemini60.442.32None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.4050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 87, 84× the cheapest (1 of 2 outputs estimated)Haiku 5.5 · API: task score 80, 1.7× the cheapestSonnet 5.5 · API: task score 78, 27× the cheapestGPT-6.1 Sol · API: task score 77, 12× the cheapestGPT-6 Luna · API: task score 74, the cheapestOpus 5.5 · Claude: task score 71, 68× the cheapest (1 of 2 outputs estimated)Gemini 3.8 Flash · API: task score 61, 8.1× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 49, 3.1× the cheapest (1 of 2 outputs estimated)GPT-6 Astra · ChatGPTHaiku 5.5 · APISonnet 5.5 · APIGPT-6.1 Sol · APIGPT-6 Luna · APIOpus 5.5 · ClaudeGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and costCost partly estimated (app runs with no usage meter)
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API741.0×
Haiku 5.5 · API801.7×
Gemini 3.5 Flash-Lite · Gemini493.1× (partly estimated)
Gemini 3.8 Flash · API618.1×
GPT-6.1 Sol · API7712×
Sonnet 5.5 · API7827×
Opus 5.5 · Claude7168× (partly estimated)
GPT-6 Astra · ChatGPT8784× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Writing a strategy memo that leadership can act on.

Why it matters

Strategy that doesn't choose isn't strategy. Models write fluent strategic prose easily; making a choice the evidence supports and naming what it gives up is harder.

What good looks like

  • Diagnoses the one obstacle that matters before prescribing
  • Makes one clear choice and names what is sacrificed
  • Grounds the choice in the supplied evidence
  • Surfaces the tension or contradiction in the data
  • Defines how we would know it is working

Deliberately not measured

  • Market sizing accuracy beyond the supplied data
  • Financial modelling
Capability tested

Choosing where to play and what not to do, from supplied evidence

The failure we’re looking for

Strategic-sounding aspirations without a diagnosis or a choice

Grading

Decision model and LLM judge, calibrated against a blind PM review