Tasks / Strategy

Develop product strategy

Can the model make a coherent choice grounded in the evidence, rather than list aspirations?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 38% were usable with at most a quick edit.

Reliably right

  1. Produces the required deliverable100% pass
    Complete strategy document in the requested form, length, and for the named reader; the board could act on it.
    GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
  2. Diagnosis before prescription100% pass
    Diagnosis: students can't find the right tutor; actions (intake, recommendations, rematch, manual matching) directly address it.
    GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
  3. Tells the CEO plainly100% pass
    It clearly states the enterprise-first plan is wrong for next year, backs it with numbers, and presents the alternative.
    GPT-6 Astra · ChatGPT · Enterprise, says the CEO

Where it slips

  1. Names an advantage that's hard to copy11% pass
    Does not name a benefit and a barrier that a well-funded competitor couldn't copy; no moat is identified.
    GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
  2. Proposes tests that could fail42% pass
    The SMB pilot lacks a numeric threshold for success; while the payback guardrail has a threshold and action, not every proposed test/gate meets the full requirement.
    GPT-6 Astra · ChatGPT · Enterprise, says the CEO
  3. Avoids unsupported claims44% pass
    Presents 'root cause: we've let the loudest requests set the roadmap' and 'SMB remains our top-of-funnel' as facts without labelling them as hypotheses or citing evidence.
    Haiku 5.5 · API · Enterprise, says the CEO

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Diagnosis before prescription

    Does the strategy name the crux, the specific obstacle or opportunity the evidence points to, and do the proposed actions follow from that diagnosis?

    Passes when States one crux grounded in the supplied evidence, and each major action visibly addresses it.

  2. Names an advantage that's hard to copy

    Does the strategy say what will make its position hard for an informed, well-funded competitor to copy (a benefit customers value plus a barrier to copying it), rather than relying on being first, on features, or on funding?

    Passes when Names the benefit and the barrier, and why a capable competitor couldn't simply replicate it.

  3. Says what it gives up

    Does the strategy name what the company will stop doing, or deliberately not do, in order to pursue it?

    Passes when Names at least one real thing being dropped or declined, with the reason it loses out.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Proposes tests that could fail · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the VP Product at Cadence. Our CEO wants a strategy for next year the exec team can rally behind, and believes it's enterprise (the note is below). Write the strategy memo, in no more than 1,200 words: the diagnosis, the guiding policy (where we play and what we stop), the coherent actions for next year, and how we'll know it's working. Tell the CEO if you think they're wrong. The files are attached.

What the model was given6 items: About Cadence, ceo_note.md, unit_economics_by_segment.csv, enterprise_detail.md, mid_market_detail.md, smb_detail.md
About CadenceStaff scheduling and time tracking, sold to businesses with hourly workers.
ceo_note.md1 lines · Download
From the CEO: "I want a strategy for next year that the exec team can rally behind. I believe it's enterprise. Tell me if I'm wrong."
unit_economics_by_segment.csv3 rows · Download
segment,customers,avg_arr_usd,gross_margin_pct,support_cost_per_customer_year_usd,cac_usd,payback_months,net_revenue_retention_pct,logo_churn_year_pct
SMB self-serve,6200,1100,78,610,740,31,88,34
Mid-market,890,14500,81,1900,9800,9,118,7
Enterprise,34,96000,64,21000,118000,26,104,9
enterprise_detail.md4 lines · Download
# Enterprise detail

- Average sales cycle: 9 months. 5 of the last 8 deals needed custom features (built by the core team, about 14 engineer-weeks each).
- The CEO's plan: "Win enterprise. Hire 10 enterprise AEs and a solutions team, and build SSO, audit logs and custom workflows this year."
mid_market_detail.md5 lines · Download
# Mid-market detail

- Best-retaining segment. Expansion comes from adding locations and seats.
- Top requests: multi-location reporting, payroll integrations, shift templates.
- Win rate against our main competitor: 61%.
smb_detail.md4 lines · Download
# SMB self-serve detail

- 62% of support tickets come from SMB accounts. Free trial converts at 4%.
- Most SMB customers are on the $79/month plan with unlimited phone support.
What a strong answer doesThe answer key the graders mark against

Diagnoses from the unit economics: mid-market is the engine (payback 9 months, NRR 118%, churn 7%, wins 61% against the main competitor); SMB self-serve loses money once support and churn are counted (about $1,100 a year at 78% gross margin is about $860, minus $610 of support, leaves about $250 a year against $740 to acquire, with 34% leaving each year; payback 31 months, NRR 88%) and generates 62% of tickets; enterprise is slow and expensive (payback 26 months, 64% gross margin, 9-month cycles, 5 of 8 deals needing about 14 engineer-weeks of custom work). Tells the CEO the enterprise bet is wrong for next year: it would pull the core team into custom work and stretch payback. Policy: win mid-market, expanding on locations and seats; stop subsidising SMB support (change the plan or pricing, self-serve support, raise the entry price) rather than abandon it; serve enterprise only where it buys the product as is. Coherent actions: mid-market roadmap (multi-location reporting, payroll integrations, shift templates), SMB support and pricing changes, no custom enterprise work. Names what it gives up and how it'll know (mid-market NRR and payback, SMB support cost per customer).

Critical failures (cap the score at 40)
  • Recommends hiring the enterprise team and building custom workflows as next year's main strategy
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Usable with a quick edit10 of 12 checks passed

The memo is evidence-based, clear, and actionable; a PM could adopt it with minor wording tweaks without re-verifying facts or redoing the analysis.

Output

Cadence: Next Year’s Strategy

To: CEO and Executive Team From: VP Product Decision: Make multi-location mid-market businesses our growth engine—not enterprise.

1. Diagnosis: Our constraint is focus, not deal size

I believe the enterprise-first plan is wrong for next year. Enterprise may become an attractive market, but our current evidence does not justify hiring ten enterprise AEs or orienting the roadmap around their deals.

Cadence has three materially different businesses:

SegmentCurrent ARR*Reported CAC paybackNRRAnnual logo churn
------:---:---:---:
SMB self-serve$6.82M31 months88%34%
Mid-market$12.91M9 months118%7%
Enterprise$3.26M26 months104%9%

Customers × average ARR.

Mid-market already generates 56% of ARR. It combines our best retention, strongest expansion, fastest acquisition payback, and highest gross margin, at 81%. A 61% win rate against our main competitor suggests a defensible product advantage. Expansion through locations and seats gives us a repeatable growth mechanism.

Enterprise offers larger contracts, but not yet a better business. Gross margin is 64%, the sales cycle averages nine months, and five of the last eight deals required custom development. Those five deals consumed approximately 70 engineer-weeks. Hiring sellers before fixing this delivery model risks scaling commitments faster than reusable product capability.

SMB is a different problem: substantial revenue with weak retention and disproportionate service demand. It generates 62% of support tickets; annual support cost averages $610 against $1,100 in ARR. Unlimited phone support on a $79 monthly plan is misaligned with self-service economics. A 4% trial conversion rate compounds the acquisition problem.

The central challenge is therefore allocating scarce product, engineering, and commercial capacity toward repeatable, retained revenue. Enterprise customization and SMB service demands are drawing capacity away from the segment where our model works best.

2. Guiding policy: Win multi-location mid-market

Where we play

Our primary customer is a mid-market business with hourly workers across multiple locations, where scheduling consistency, consolidated visibility, and payroll handoffs create recurring operational pain.

Our promise: make running the next location easier than running the first.

We will deepen that advantage and turn successful initial deployments into additional locations and seats. We will first target industries and acquisition channels where our own mid-market cohorts show the strongest retention and payback; the supplied evidence does not yet support choosing a specific vertical.

Enterprise remains a selective adjacency. SMB remains a supported, increasingly self-service business—not our primary acquisition investment.

What we stop

  • Do not approve the ten-enterprise-AE hiring plan or a dedicated solutions-team buildout.
  • Stop letting individual enterprise deals dictate core-team custom workflows.
  • Do not commit to the entire enterprise feature bundle. SSO and audit logs must earn priority through demonstrated, reusable demand—not their “enterprise” label.
  • Stop scaling SMB paid acquisition under current economics.
  • End unlimited phone support as the default for new low-priced plans, while honoring existing commitments and managing any migration carefully.
  • Stop broad roadmap work that cannot demonstrate a connection to mid-market acquisition, retention, expansion, or essential platform health.

We will continue supporting existing enterprise and SMB customers. Focus is not abandonment.

3. Coherent actions for next year

A. Build the multi-location growth loop

Product and Engineering: Give mid-market the clear majority of discretionary roadmap capacity, after reliability, security, and contractual obligations.

Sequence three connected capabilities:

  1. Multi-location reporting: Give central operators visibility into staffing and hours across their estate.
  2. Payroll integrations: Prioritize providers used by the largest share of our target accounts; reduce recurring payroll friction.
  3. Shift templates: Make successful scheduling practices reusable across locations.

In Q1, validate scope with retained, expanding customers and establish adoption and workflow baselines. Release reporting and the first prioritized integration in the first half; expand integrations and template-based rollout in the second half. Measure customer outcomes and subsequent location adoption, not merely shipment.

B. Sell and expand the same repeatable product

Sales and Customer Success: Redirect incremental commercial investment toward mid-market rather than enterprise hiring.

Package a standard multi-location deployment, define onboarding milestones, and build expansion reviews around customers’ remaining locations. Align incentives with retained recurring revenue and successful activation, not bookings alone.

Increase acquisition spending in stages only while segment payback remains within our guardrail. Do not assume today’s economics survive unlimited scaling.

C. Repair SMB service economics

Support, Product, and Finance: Identify the largest ticket drivers in Q1. Improve onboarding and in-product guidance around those problems; test appropriately priced phone support and service tiers for new customers.

Pilot changes before broad rollout. Evaluate support savings alongside conversion, retention, and revenue effects. A lower ticket count caused by driving customers away is not success.

D. Keep enterprise as a bounded learning option

Sales and Product: Accept new enterprise opportunities only when they fit the standard product, require no bespoke core-team development, and meet Finance-approved deal economics.

Review enterprise quarterly. Reconsider a larger investment only after multiple customers buy and deploy substantially the same product, with improving margin and acquisition payback. One marquee logo is not proof.

E. Establish a shared economic scorecard

Finance: In Q1, reconcile acquisition costs, implementation costs, support allocation, and payback definitions across segments. Confirm whether support is already included in gross margin before subtracting it again. Track custom engineering separately so its opportunity cost remains visible.

4. How we will know it is working

These are proposed operating targets, not forecasts:

MeasureBaselineYear-end target / guardrail
------:---:
Mid-market ARR$12.91MAt least $16.1M, approximately 25% growth
Mid-market NRR118%At least 120%
Mid-market annual logo churn7%No worse than 7%
Mid-market CAC payback9 monthsAt most 12 months while scaling
Mid-market gross margin81%At least 80%
SMB tickets per active accountEstablish in Q1Reduce 25%, without worsening cohort revenue retention
New bespoke enterprise core-team workFive of last eight dealsZero

The executive team will review results monthly and resource allocation quarterly. Leading indicators are faster location activation, adoption of the new workflows, and expansion pipeline; financial outcomes remain the test.

The strategy is not “avoid enterprise.” It is “scale the repeatable business we have earned.” Mid-market gives us the strongest foundation for growth next year—and a healthier route into enterprise later.

Grades and run details

Decision model 88 · LLM judge 85
Decision model checks
  • passUses the supplied evidence correctly19%
  • passAddresses the actual decision99%
  • passRespects explicit constraints47%
  • passIdentifies material uncertainty45%
  • partialAvoids unsupported claims43%
  • passProduces the required deliverable69%
  • passReads the segment economics66%
  • passTells the CEO plainly100%
  • passDiagnosis before prescription100%
  • partialProposes tests that could fail74%
  • partialNames an advantage that's hard to copy76%
  • passSays what it gives up100%
Run
Run
#1
Time to output
39 s
Submitted
5 Oct 2026

Needs more than a quick edit8 of 12 checks passed

The memo contains an unsupported factual claim about market expectations, requiring evidence re-check, and lacks a clear competitive moat, so a PM would need to redo some reasoning.

Output

Memo: Next Year's Strategy

To: CEO From: VP Product Re: Where we play next year, and why I don't think it's enterprise

You asked me to tell you if you're wrong. I think you are, at least for next year. Enterprise isn't a bad market. But our numbers say our winning position is somewhere else, and an enterprise push would starve that position of the engineering time and sales capacity it needs. Below is the case and what I propose instead.

---

1. Diagnosis

Cadence runs three different businesses, and only one of them is working well.

SegmentARRAnnual gross profit after support, per customerCACPaybackNRRLogo churn
SMB self-serve$6.8M$248$74031 mo88%34%
Mid-market$12.9M$9,845$9,8009 mo118%7%
Enterprise$3.3M$40,440$118,00026 mo104%9%

(Gross profit after support = ARR × gross margin − support cost. Total ARR is about $23M.)

Mid-market is the business. - It is 56% of ARR. - It pays back CAC in 9 months. - It grows on its own: 118% NRR on $12.9M means about $2.3M of net new ARR next year before we sign a single new logo. - We win 61% of the time against our main competitor. - Expansion comes naturally as customers add locations and seats.

Enterprise looks big per logo but is weak as a business today. - Gross margin is 64%, versus 81% in mid-market. - Each customer costs $21K a year to support. - CAC is $118K, payback takes 26 months, and the sales cycle is 9 months. - The hidden cost is engineering. 5 of the last 8 deals needed custom features, at about 14 engineer-weeks each, built by the core team. That is roughly 70 engineer-weeks diverted from the product that 890 mid-market customers pay for.

SMB is roughly break-even and pulling capacity away from everything else. - Each customer generates $248 a year after support. - With 34% annual churn, the average customer lives about 3 years. Lifetime contribution is roughly $730, against $740 of CAC. - SMB produces 62% of our support tickets. The cause is structural: we offer unlimited phone support on a $79/month plan, so support costs ($610) eat more than half of revenue ($1,100). - Trial conversion is 4%.

The core problem: we are trying to be everything to everyone. The cost shows up in two places. The roadmap gets hijacked by single enterprise deals, and the support budget gets consumed by SMB.

Why the enterprise plan makes this worse. Assume 10 new AEs close 20 enterprise logos next year, which is optimistic given a 9-month cycle and ramp time. - That adds about $1.9M of ARR. - It costs about $2.4M in CAC. - If 5 of every 8 deals still need custom work, it consumes about 175 engineer-weeks, or 3–4 engineer-years, on custom features alone. - That is before the work on SSO, audit logs, and custom workflows. - Most of that ARR would land late in the year and pay back in year three.

Meanwhile, the mid-market roadmap would stall in the segment where we already win.

---

2. Guiding Policy

Where we play: We will be the best scheduling and time-tracking platform for multi-location operators with hourly workforces, from roughly 10 to a few hundred locations. Mid-market is the core. Every product, sales, and support decision should first answer one question: does this help a mid-market customer win, stay, or add locations?

What we stop: 1. Custom feature builds for single deals. If a feature isn't on the roadmap for a segment, we don't build it to close one logo. 2. The 10-AE enterprise hiring plan. We redeploy that budget (see below). 3. Unlimited phone support on the $79 plan. That plan moves to in-app help, chat, and a knowledge base. Phone support becomes a paid tier. 4. Custom workflows as a product initiative. This is a services business disguised as a feature.

What we deliberately keep: - Our 34 enterprise customers, served well on the standard product. - Inbound enterprise deals that can close without custom work. - SSO and audit logs. Upper mid-market buyers increasingly expect these, so they serve the core rather than a new segment.

---

3. Coherent Actions for Next Year

These actions reinforce each other. Product makes mid-market expansion easier. Go-to-market (GTM) captures that expansion. Fixing SMB frees money and support capacity to fund both.

Product (mid-market roadmap, Q1–Q3) - Multi-location reporting. This is the top request, and it directly drives location expansion, our main source of NRR. - Payroll integrations. We will prioritize the top 3–4 providers by customer count. Integrations increase switching costs and reduce churn. - Shift templates. This reduces setup time for each new location, so expansion becomes cheap for the customer and for us. - SSO and audit logs. These are table stakes for larger mid-market accounts. We will scope them tightly and avoid gold-plating. - Engineering capacity rule: zero engineer-weeks go to one-off custom builds.

Go-to-market - Redirect the enterprise AE budget. Fund roughly 4–5 mid-market AEs and 3–4 account managers whose job is location and seat expansion inside existing accounts. A 9-month payback and 118% NRR make this the most efficient growth dollar we have. - Build a competitive playbook. Codify why we win 61% against our main competitor, and target that competitor's mid-market base. - Handle enterprise inbound on the standard product. One senior AE and the existing team qualify enterprise leads. A deal requiring custom work is a no unless the feature is already on the roadmap.

SMB economics - Restructure support. Move the $79 plan to digital-first support and introduce a higher tier that includes phone support. The target is cutting SMB support cost per customer from $610 to under $350. - Overhaul onboarding and trials. A 4% conversion rate signals weak activation. Fixing onboarding should also reduce tickets. - Find out whether SMB feeds mid-market. We need to learn how many mid-market customers started on SMB. If SMB is a real feeder, we keep it as a low-cost funnel. If not, we run it purely for margin.

Enterprise - Improve margins on the existing base. Standardize their configurations and reduce bespoke support. - Track demand without chasing it. We log every enterprise request, so that if we return to enterprise later, we can build a standard product rather than custom one-offs.

---

4. How We'll Know It's Working

Leading indicators (quarterly) - Multi-location reporting and payroll integrations shipped by end of Q2. - Engineer-weeks spent on custom deal work: 0. - SMB support tickets per customer down 30% by Q3. - Mid-market win rate against our main competitor: 61% → 65%.

Outcomes (year-end) - Mid-market NRR ≥ 120%, up from 118%. - Mid-market new-logo ARR grows at least 25%, with payback held at ≤ 12 months. - SMB support cost per customer under $350, and SMB contribution per customer roughly doubled. - Enterprise gross margin above 70%, up from 64%, with no logo churn among the top 10.

When we would revisit enterprise: I'm not saying never. I'd reopen the question in Q3 if either of these happens: - Mid-market customers are outgrowing us into enterprise-scale deployments, meaning enterprise becomes the natural next step for our core. - We close at least 3 enterprise deals on the standard product.

Either signal would mean enterprise has become an extension of our strength rather than a separate bet. At that point, a focused enterprise motion built on a product we actually have would make sense.

---

Bottom line: The exec team can rally behind this: win multi-location mid-market, stop subsidizing segments that don't pay. It builds on where we already win, funds itself by fixing SMB economics, and keeps enterprise open as a future option rather than a costly bet. I'd welcome the chance to pressure-test this with you before the offsite.

Claims the judge couldn’t find in the brief

  • Upper mid-market buyers increasingly expect SSO and audit logs.
  • Integrations increase switching costs and reduce churn.

Grades and run details

Decision model 88 · LLM judge 62
Decision model checks
  • passUses the supplied evidence correctly25%
  • passAddresses the actual decision100%
  • passRespects explicit constraints49%
  • passIdentifies material uncertainty98%
  • partialAvoids unsupported claims51%
  • passProduces the required deliverable83%
  • passReads the segment economics90%
  • passTells the CEO plainly100%
  • passDiagnosis before prescription100%
  • partialProposes tests that could fail72%
  • partialNames an advantage that's hard to copy78%
  • passSays what it gives up100%
Run
Run
#1
Time to output
44 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 2

Uses the supplied evidence correctlyRightMixed
GPT-6 Astra · ChatGPT

All factual claims about the current situation are drawn directly from the supplied context or simple arithmetic.

Opus 5.5 · Claude

The memo claims that 'Upper mid-market buyers increasingly expect these' (SSO and audit logs) without support from the supplied context, inventing a fact about the current market.

Avoids unsupported claimsRightWrong
GPT-6 Astra · ChatGPT

Interpretations are presented as such (e.g., 'suggests a defensible product advantage'), and forecasts are clearly not stated as fact.

Opus 5.5 · Claude

Presents 'Upper mid-market buyers increasingly expect these' as fact without evidence, and 'Integrations increase switching costs' as an established cause, neither supported by the supplied context.

All got wrong 2

Proposes tests that could failWrongWrong
GPT-6 Astra · ChatGPT

The SMB pilot lacks a numeric threshold for success; while the payback guardrail has a threshold and action, not every proposed test/gate meets the full requirement.

Opus 5.5 · Claude

The leading indicators (e.g., SMB tickets down 30%, mid-market NRR ≥120%) lack stated actions for when they are not met; only the enterprise revisit gates have triggers and actions.

Names an advantage that's hard to copyWrongWrong
GPT-6 Astra · ChatGPT

The strategy relies on features and a high win rate but does not name a specific barrier that would prevent a well-funded competitor from copying the multi-location advantage.

Opus 5.5 · Claude

Does not name a specific advantage that is hard for a well-funded competitor to copy, nor explain why they couldn't simply replicate the mid-market focus.

All got right 8

Addresses the actual decisionRightRight
GPT-6 Astra · ChatGPT

The memo commits early to mid-market over enterprise, says the enterprise-first plan is wrong, and states conditions to reconsider enterprise investment.

Opus 5.5 · Claude

Commits early to disagreeing with the CEO's enterprise bet, proposes an alternative, and specifies conditions (mid-market outgrowing, 3 standard enterprise deals) that would reopen the question.

Respects explicit constraintsRightRight
GPT-6 Astra · ChatGPT

The output is a strategy memo under 1,200 words, includes diagnosis, guiding policy, actions, and metrics, and directly addresses the CEO.

Opus 5.5 · Claude

The memo is under 1200 words, addresses the CEO, includes diagnosis, guiding policy, actions, and how we'll know, and tells the CEO they are wrong.

Identifies material uncertaintyRightRight
GPT-6 Astra · ChatGPT

It notes the need to confirm support cost accounting, that vertical selection is not yet supported, and sets conditions for scaling acquisition and enterprise investment.

Opus 5.5 · Claude

Identifies the unknown of whether SMB feeds mid-market and proposes to learn it; bounds enterprise uncertainty with specific revisit triggers.

Produces the required deliverableRightRight
GPT-6 Astra · ChatGPT

The memo is complete, well-structured for the executive team, within length, and actionable without major gaps.

Opus 5.5 · Claude

Delivers a complete strategy memo in the requested form, with all required sections, and is actionable for the CEO.

Reads the segment economicsRightRight
GPT-6 Astra · ChatGPT

It explicitly uses payback, retention, and support cost for all three segments and concludes mid-market is strongest, SMB economics are poor, and enterprise is slow/costly.

Opus 5.5 · Claude

Works through payback, retention, and support cost for all three segments, correctly concluding mid-market is strongest, SMB loses money, and enterprise is slow and costly.

Tells the CEO plainlyRightRight
GPT-6 Astra · ChatGPT

It clearly states the enterprise-first plan is wrong for next year, backs it with numbers, and presents the alternative.

Opus 5.5 · Claude

Clearly and respectfully tells the CEO they are wrong for next year, with evidence from the numbers.

Diagnosis before prescriptionRightRight
GPT-6 Astra · ChatGPT

The crux (allocating scarce capacity toward repeatable retained revenue) is grounded in the evidence, and actions directly address enterprise custom work and SMB support drain.

Opus 5.5 · Claude

States the core problem (trying to be everything to everyone) grounded in the data, and each major action addresses that problem.

Says what it gives upRightRight
GPT-6 Astra · ChatGPT

It explicitly lists what it stops: enterprise AE hiring, custom enterprise work, SMB paid acquisition scaling, unlimited phone support, and broad roadmap work, with reasons.

Opus 5.5 · Claude

Explicitly lists what the company will stop: custom feature builds, enterprise AE hiring, unlimited phone support, custom workflows, with reasons.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 84% of 76 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT83.388.52None
2Haiku 5.5withAPI77.180.82None
3Sonnet 5.5withAPI77.180.82None
4GPT-6.1 SolwithAPI79.276.92None
5GPT-6 LunawithAPI77.176.92None
6Opus 5.5withClaude83.365.42None
7Gemini 3.8 FlashwithAPI68.857.72None
8Gemini 3.5 Flash-LitewithGemini60.442.32None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.4050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 87, 84× the cheapest (1 of 2 outputs estimated)Haiku 5.5 · API: task score 80, 1.7× the cheapestSonnet 5.5 · API: task score 78, 27× the cheapestGPT-6.1 Sol · API: task score 77, 12× the cheapestGPT-6 Luna · API: task score 74, the cheapestOpus 5.5 · Claude: task score 71, 68× the cheapest (1 of 2 outputs estimated)Gemini 3.8 Flash · API: task score 61, 8.1× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 49, 3.1× the cheapest (1 of 2 outputs estimated)GPT-6 Astra · ChatGPTHaiku 5.5 · APISonnet 5.5 · APIGPT-6.1 Sol · APIGPT-6 Luna · APIOpus 5.5 · ClaudeGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and costCost partly estimated (app runs with no usage meter)
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API741.0×
Haiku 5.5 · API801.7×
Gemini 3.5 Flash-Lite · Gemini493.1× (partly estimated)
Gemini 3.8 Flash · API618.1×
GPT-6.1 Sol · API7712×
Sonnet 5.5 · API7827×
Opus 5.5 · Claude7168× (partly estimated)
GPT-6 Astra · ChatGPT8784× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Writing a strategy memo that leadership can act on.

Why it matters

Strategy that doesn't choose isn't strategy. Models write fluent strategic prose easily; making a choice the evidence supports and naming what it gives up is harder.

What good looks like

  • Diagnoses the one obstacle that matters before prescribing
  • Makes one clear choice and names what is sacrificed
  • Grounds the choice in the supplied evidence
  • Surfaces the tension or contradiction in the data
  • Defines how we would know it is working

Deliberately not measured

  • Market sizing accuracy beyond the supplied data
  • Financial modelling
Capability tested

Choosing where to play and what not to do, from supplied evidence

The failure we’re looking for

Strategic-sounding aspirations without a diagnosis or a choice

Grading

Decision model and LLM judge, calibrated against a blind PM review