Tasks / Strategy

Develop product strategy

Can the model make a coherent choice grounded in the evidence, rather than list aspirations?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 38% were usable with at most a quick edit.

Reliably right

Produces the required deliverable100% pass
Complete strategy document in the requested form, length, and for the named reader; the board could act on it.
GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
Diagnosis before prescription100% pass
Diagnosis: students can't find the right tutor; actions (intake, recommendations, rematch, manual matching) directly address it.
GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
Tells the CEO plainly100% pass
It clearly states the enterprise-first plan is wrong for next year, backs it with numbers, and presents the alternative.
GPT-6 Astra · ChatGPT · Enterprise, says the CEO

Where it slips

Names an advantage that's hard to copy11% pass
Does not name a benefit and a barrier that a well-funded competitor couldn't copy; no moat is identified.
GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
Proposes tests that could fail42% pass
The SMB pilot lacks a numeric threshold for success; while the payback guardrail has a threshold and action, not every proposed test/gate meets the full requirement.
GPT-6 Astra · ChatGPT · Enterprise, says the CEO
Avoids unsupported claims44% pass
Presents 'root cause: we've let the loudest requests set the roadmap' and 'SMB remains our top-of-funnel' as facts without labelling them as hypotheses or citing evidence.
Haiku 5.5 · API · Enterprise, says the CEO

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Diagnosis before prescription
Does the strategy name the crux, the specific obstacle or opportunity the evidence points to, and do the proposed actions follow from that diagnosis?
Passes when States one crux grounded in the supplied evidence, and each major action visibly addresses it.
Bangaly KabaLenny’s Podcast
Names an advantage that's hard to copy
Does the strategy say what will make its position hard for an informed, well-funded competitor to copy (a benefit customers value plus a barrier to copying it), rather than relying on being first, on features, or on funding?
Passes when Names the benefit and the barrier, and why a capable competitor couldn't simply replicate it.
Hamilton HelmerLenny’s Podcast
Says what it gives up
Does the strategy name what the company will stop doing, or deliberately not do, in order to pursue it?
Passes when Names at least one real thing being dropped or declined, with the reason it loses out.
Gokul RajaramLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Proposes tests that could fail · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the VP Product at Cadence. Our CEO wants a strategy for next year the exec team can rally behind, and believes it's enterprise (the note is below). Write the strategy memo, in no more than 1,200 words: the diagnosis, the guiding policy (where we play and what we stop), the coherent actions for next year, and how we'll know it's working. Tell the CEO if you think they're wrong. The files are attached.

What the model was given6 items: About Cadence, ceo_note.md, unit_economics_by_segment.csv, enterprise_detail.md, mid_market_detail.md, smb_detail.md

About CadenceStaff scheduling and time tracking, sold to businesses with hourly workers.

ceo_note.md1 lines · Download

From the CEO: "I want a strategy for next year that the exec team can rally behind. I believe it's enterprise. Tell me if I'm wrong."

unit_economics_by_segment.csv3 rows · Download

segment,customers,avg_arr_usd,gross_margin_pct,support_cost_per_customer_year_usd,cac_usd,payback_months,net_revenue_retention_pct,logo_churn_year_pct
SMB self-serve,6200,1100,78,610,740,31,88,34
Mid-market,890,14500,81,1900,9800,9,118,7
Enterprise,34,96000,64,21000,118000,26,104,9

enterprise_detail.md4 lines · Download

# Enterprise detail

- Average sales cycle: 9 months. 5 of the last 8 deals needed custom features (built by the core team, about 14 engineer-weeks each).
- The CEO's plan: "Win enterprise. Hire 10 enterprise AEs and a solutions team, and build SSO, audit logs and custom workflows this year."

mid_market_detail.md5 lines · Download

# Mid-market detail

- Best-retaining segment. Expansion comes from adding locations and seats.
- Top requests: multi-location reporting, payroll integrations, shift templates.
- Win rate against our main competitor: 61%.

smb_detail.md4 lines · Download

# SMB self-serve detail

- 62% of support tickets come from SMB accounts. Free trial converts at 4%.
- Most SMB customers are on the $79/month plan with unlimited phone support.

What a strong answer doesThe answer key the graders mark against

Diagnoses from the unit economics: mid-market is the engine (payback 9 months, NRR 118%, churn 7%, wins 61% against the main competitor); SMB self-serve loses money once support and churn are counted (about $1,100 a year at 78% gross margin is about $860, minus $610 of support, leaves about $250 a year against $740 to acquire, with 34% leaving each year; payback 31 months, NRR 88%) and generates 62% of tickets; enterprise is slow and expensive (payback 26 months, 64% gross margin, 9-month cycles, 5 of 8 deals needing about 14 engineer-weeks of custom work). Tells the CEO the enterprise bet is wrong for next year: it would pull the core team into custom work and stretch payback. Policy: win mid-market, expanding on locations and seats; stop subsidising SMB support (change the plan or pricing, self-serve support, raise the entry price) rather than abandon it; serve enterprise only where it buys the product as is. Coherent actions: mid-market roadmap (multi-location reporting, payroll integrations, shift templates), SMB support and pricing changes, no custom enterprise work. Names what it gives up and how it'll know (mid-market NRR and payback, SMB support cost per customer).

Critical failures (cap the score at 40)

Recommends hiring the enterprise team and building custom workflows as next year's main strategy

Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

GPT-6 AstraChatGPT87Usable with a quick edit Sonnet 5.5API87Usable with a quick edit GPT-6.1 SolAPI85Usable with a quick edit Haiku 5.5API81Usable with a quick edit GPT-6 LunaAPI75Needs more work Opus 5.5Claude71Needs more work Gemini 3.8 FlashAPI67Needs more work Gemini 3.5 Flash-LiteGemini56Needs more work

Haiku 5.5API

Usable with a quick edit9 of 12 checks passed

Minor unsupported claims can be removed or rephrased in an edit; the core evidence and logic are sound and the memo is actionable.

Output

To: CEO From: VP Product Re: Next-year strategy

Bottom line: I think you're wrong. We are a mid-market company, and the enterprise plan would spend our scarcest resources, engineering time and hiring budget, on the segment with the weakest economics. Enterprise can be a real option later. It shouldn't be the strategy for next year.

Diagnosis

Annualized from the unit economics file (my calculations, not reported figures):

	SMB self-serve	Mid-market	Enterprise
Customers	6,200	890	34
ARR (share)	~$6.8M (30%)	~$12.9M (56%)	~$3.3M (14%)
Gross margin	78%	81%	64%
Support cost / customer / yr	$610	$1,900	$21,000
Gross profit less support / customer	~$250	~$9,850	~$40,400
CAC	$740	$9,800	$118,000
Payback (months)	31	9	26
NRR	88%	118%	104%
Logo churn	34%	7%	9%

Mid-market is our engine. It produces most of our ARR, expands at 118% NRR, pays back in 9 months, and churns at 7%. We win 61% of head-to-head deals against our main competitor. Customers are asking for multi-location reporting, payroll integrations, and shift templates, which are the features that drive expansion.

SMB is a support sink. SMB accounts generate 62% of support tickets. Support costs about $3.8M a year against roughly $6.8M of SMB ARR, so over half of SMB revenue goes to support. Most SMB customers are on a $79/month plan with unlimited phone support. Churn is 34%, trial conversion is 4%, and lifetime contribution barely covers CAC.

Enterprise is a real market but a poor fit for us today. We have 34 logos. Sales cycles run nine months. Gross margin is 64%, and payback is 26 months before custom work is counted. Five of our last eight enterprise deals needed custom features built by the core team at about 14 engineer-weeks each. That is roughly 70 engineer-weeks, or about 1.5 engineer-years, in eight deals. The plan doesn't account for that cost.

Root cause: We've let the loudest requests set the roadmap and support model. Enterprise asks for custom builds, SMB asks for phone support, and neither request is what drives our best economics.

Guiding policy

Where we play: Mid-market multi-location employers with hourly workforces. This is the customer we win, keep, and expand, and where our product is strongest.

What we stop: 1. Custom builds for individual deals. No bespoke engineering without a paid statement of work that covers full cost and passes a "does this generalize?" test. 2. Unlimited phone support for SMB. SMB moves to self-serve, chat, and knowledge-base support, with paid priority support as an option. 3. Dedicated enterprise hiring and outbound. Enterprise becomes inbound-only, served by the mid-market product. We don't build a headcount plan around it. 4. Single-segment edge-case features that don't help mid-market or the platform.

What we don't stop: The 34 enterprise customers keep full service. We don't reprice or abandon them. SMB remains our top-of-funnel, but it stops being a support-heavy product.

Coherent actions for next year

Build the mid-market expansion product. Ship multi-location reporting first, then shift templates, then the two payroll integrations that come up most in mid-market accounts. This gets roughly 70% of core product capacity.

Move the enterprise hiring budget to mid-market. Use it for mid-market AEs (where a 61% win rate and 9-month payback make each dollar work faster) and for expansion coverage on multi-location accounts.

Restructure SMB support. Remove unlimited phone support. Create a clear upgrade path to mid-market plans when an SMB account adds a second location. Use product changes to deflect the most common tickets. The goal is to halve support cost per SMB customer.

Productize enterprise table stakes, nothing more. Build SSO and audit logs once as standard features. Some mid-market buyers need them too, and building them once is cheaper than building them repeatedly for deals. Don't build custom workflows. Keep the 34 enterprise accounts well served through CS.

Govern the roadmap. A product council with exec sign-off reviews any request that isn't on the mid-market or platform roadmap. Every custom request must pass the segment-fit test.

Capacity split (proposed): ~70% mid-market roadmap, ~15% platform (SSO, audit logs, reliability), ~15% SMB self-serve and ticket deflection.

How we'll know it's working

Mid-market (primary): - NRR ≥ 120% (from 118%) - Logo churn ≤ 7% - Win rate vs. main competitor ≥ 61% - Payback ≤ 9 months on new mid-market CAC - Share of expansion ARR from added locations and seats, tracked quarterly

SMB: - Support cost per SMB customer down from $610 to ≤ $300 - SMB tickets per customer down year over year - Gross profit after support positive per customer on the new support model - SMB churn trending down; we accept some SMB logo loss as the price of the change

Enterprise: - Zero unplanned custom engineering weeks outside approved SOWs - Enterprise gross margin trending up toward 70%+ - No loss in renewals among the 34 existing accounts

Company: - Blended gross margin up, driven by the mix shift - Total ARR growth on the mid-market base

Revisit enterprise in 12 months if mid-market hits its targets and we see a steady flow of inbound enterprise deals (for example, three per quarter) that fit the standard product without custom work.

Risks

We'll lose some SMB and enterprise revenue. The bet is that mid-market growth and margin improvement outweigh it. We should show the exec team the expected net effect before we commit.
Mid-market buyers may demand SSO and audit logs sooner. Building them as standard features in the platform bucket covers this.
The enterprise story matters to investors and some of the team. We should explain it as focus with a defined path back, not as retreat.

I'm glad to walk through the model and the capacity assumptions with you before the exec session. My recommendation is to take this to the team as the choice between being a strong mid-market company now and a weak enterprise company later.

Claims the judge couldn’t find in the brief

Root cause: 'We've let the loudest requests set the roadmap and support model. Enterprise asks for custom builds, SMB asks for phone support, and neither request is what drives our best economics.'
SMB remains our top-of-funnel.

Grades and run details

Decision model 83 · LLM judge 77

Decision model checks

passUses the supplied evidence correctly35%
passAddresses the actual decision100%
passRespects explicit constraints48%
passIdentifies material uncertainty51%
partialAvoids unsupported claims52%
passProduces the required deliverable89%
passReads the segment economics98%
passTells the CEO plainly100%
passDiagnosis before prescription100%
partialProposes tests that could fail79%
failNames an advantage that's hard to copy46%
passSays what it gives up100%

Run

Run: #1
API response time: 36 s
Submitted: 8 Oct 2026

Gemini 3.8 FlashAPI

Needs more than a quick edit8 of 12 checks passed

The memo fails to identify material uncertainties, lacks actionable test-fail triggers, and does not define a defensible moat, so a product manager would need to rework those strategic elements.

Output

MEMORANDUM

TO: Chief Executive Officer FROM: VP of Product DATE: October 24, 2023 SUBJECT: Next Year Strategy: Diagnosis, Guiding Policy, and Execution Roadmap

---

Executive Summary: The Direct Answer

You asked: “I believe it’s enterprise. Tell me if I’m wrong.”

You are wrong. Going all-in on Enterprise next year would be an expensive strategic error.

Enterprise is currently acting as a high-friction, low-margin professional services trap. It requires 9-month sales cycles, consumes massive engineering resources (70 engineer-weeks burned on bespoke code for just five recent deals), and yields an uninspiring 104% NRR at a 26-month CAC payback. Hiring 10 enterprise AEs and a solutions team right now will accelerate cash burn while starving our real growth engine.

That growth engine is Mid-Market. Mid-market is exceptional: 81% gross margin, 9-month CAC payback, 118% NRR, 7% logo churn, and a 61% win rate against our primary competitor.

Next year, our rallying cry must be: Dominate Mid-Market, automate SMB into positive cash flow, and gate Enterprise until our core platform is ready.

---

1. Diagnosis: Where Value Is Created and Destroyed

Our current ARR is ~$23M across three starkly different businesses:

Metric	SMB Self-Serve	Mid-Market	Enterprise
:---	:---	:---	:---
Active Customers	6,200	890	34
Total ARR (approx.)	$6.82M (30%)	$12.91M (56%)	$3.26M (14%)
Avg. ARR / Customer	$1,100	$14,500	$96,000
Gross Margin	78%	81%	64%
Support Cost / Customer / Yr	$610 (55% of ARR)	$1,900 (13% of ARR)	$21,000 (22% of ARR)
CAC Payback	31 months	9 months	26 months
NRR / Logo Churn	88% / 34%	118% / 7%	104% / 9%
Competitive Win Rate	N/A	61%	Elongated (9 mo)

#### Three Key Findings:

Mid-Market is an elite SaaS business. Customers expand by adding locations and headcount (118% NRR). Our sales efficiency is world-class (9-month payback vs. industry average of 14–18 months). Most importantly, their feature requests (multi-location reporting, payroll integrations, shift templates) are repeatable, software-native, and scale across the entire cohort.
Enterprise is a bespoke distraction masquerading as scale. At 64% gross margin, 26-month payback, and only 104% NRR, our Enterprise motion does not scale. 5 of our last 8 deals required custom development—averaging 14 engineer-weeks per deal. We are acting as a subsidized software development shop for 34 accounts, pulling our best product talent away from core roadmap investments.
SMB is drowning our operations. SMB accounts for 62% of all support tickets. We charge $79/month ($948/yr) while spending $610/year in support per customer due to unlimited phone support. With 34% logo churn and 88% NRR, the 31-month CAC payback means a significant portion of SMB customers churn before breaking even on acquisition and service costs.

---

2. Guiding Policy: Where We Play, Where We Win, and What We Stop

Our guiding policy for next year is Repeatable Scale over Bespoke Revenue.

``` ┌──────────────────────────────────────────────┐ │ MID-MARKET: ACCELERATE │ │ Core growth vector; 118% NRR; 9-mo payback │ └──────────────────────┬───────────────────────┘ │ ┌─────────────────────────┴─────────────────────────┐ ▼ ▼ ┌─────────────────────────────┐ ┌─────────────────────────────────────┐ │ SMB: AUTOMATE │ │ ENTERPRISE: DISCIPLINE │ │ Eliminate phone support │ │ Freeze custom dev; sell off-the- │ │ Product-led self-service │ │ shelf MM product at premium tier │ └─────────────────────────────┘ └─────────────────────────────────────┘ ```

#### Where We Play: * The Sweet Spot: Multi-unit hourly employers with 50–500 employees (Mid-Market). These buyers have acute scheduling complexity across locations but lack the bureaucratic demands for custom software architectures.

#### What We Stop: * Stop custom engineering for individual deals. Zero custom branching or deal-contingent roadmap commits. * Stop hiring 10 Enterprise AEs. We will not deploy massive quota capacity into a 9-month sales cycle with an unstandardized product. * Stop offering phone support on sub-$5,000 ARR plans. Unlimited phone support for $79/month ends in Q1.

---

3. Coherent Actions for Next Year

To execute this guiding policy, our actions across Product, GTM, and Operations must reinforce each other.

#### A. Product & Engineering: The Repeatable Core * Deliver the Mid-Market Trifecta (H1): Allocate 70% of product capacity to build: 1. Multi-location reporting & dashboarding (enables regional managers to view compliance and labor costs across units). 2. Tier-1 payroll integrations (ADP, Gusto, Paychex, UKG) to make scheduling-to-payroll frictionless. 3. Advanced shift templates & auto-scheduling to drive rapid store-level adoption. * Enterprise Baseline, not Bespoke (H2): Build only standardized administrative features (SAML SSO, SCIM provisioning, audit logs, role-based access control). If an enterprise buyer cannot deploy our standard multi-tenant product with these enterprise rails, we walk away. * SMB Self-Serve & Deflection: Revamp onboarding flows to drive the 4% trial conversion rate upward and deploy in-app guided setup, chat deflection, and an updated knowledge center to cut incoming tickets.

#### B. Go-To-Market: Scaling the Engine * Reallocate Capital from Enterprise to Mid-Market: Instead of 10 Enterprise AEs, hire 6 Mid-Market AEs and 2 MM Sales Engineers. With our 9-month payback and 61% win rate, MM sales hires will reach cash-flow break-even within the year. * Establish an Enterprise Gatekeeper Rule: The existing enterprise sales team may close deals only under two conditions: (1) standard product only (contractually zero custom dev), and (2) minimum contract value of $100k ARR with standard pricing. * Refocus Inbound & Outbound: Concentrate marketing spend on multi-unit franchisees, regional hospitality groups, and mid-sized healthcare operators.

#### C. Operations & Support: Unit Economic Realignment * Tiered Support Model (Effective Q1): * SMB / Self-serve: In-app digital assistant, community, and email support (target: reduce support cost from $610 to <$150/year). * Mid-Market: Priority email, live chat, and scheduled onboarding calls. * Enterprise: Dedicated CSM and SLA-backed escalation. * Grandfathering/Pricing Migration for SMB: Shift low-tier customers requiring phone support to an “Assisted Support” add-on ($99/month), turning a cost center into a margin-accretive tier.

---

4. Scorecard: How We’ll Know It’s Working

We will monitor operational health and strategic traction via four quarterly benchmarks:

Pillar	Metric	Baseline	End of Next Year Target
:---	:---	:---	:---
Mid-Market Growth	Net New ARR Added	~$4.5M/yr run rate	$8.0M+
Expansion Health	Mid-Market NRR	118%	125%+
SMB Unit Economics	SMB Blended Support Cost / Year	$610	<$200
SMB Payback	SMB CAC Payback Months	31 months	<14 months
Product Discipline	Engineer-weeks spent on custom deal features	~70 weeks (last 8 deals)	0 weeks
Company Efficiency	Blended Gross Margin	76%	82%+

---

Conclusion

Enterprise is seductive because the contract sizes are large. But right now, chasing it will turn us into a bespoke agency with depressed margins and burnt-out engineers.

Mid-Market is our path to category leadership. It gives us high win rates, compounding expansion, rapid payback, and product leverage. Once we harden the platform with multi-location mid-market capabilities and standardized enterprise controls, we can attack the enterprise from a position of structural strength—not structural compromise.

Grades and run details

Decision model 71 · LLM judge 69

Decision model checks

failUses the supplied evidence correctly15%
passAddresses the actual decision64%
passRespects explicit constraints28%
failIdentifies material uncertainty80%
partialAvoids unsupported claims41%
passProduces the required deliverable83%
passReads the segment economics96%
passTells the CEO plainly100%
passDiagnosis before prescription100%
partialProposes tests that could fail54%
partialNames an advantage that's hard to copy27%
passSays what it gives up100%

Run

Run: #1
API response time: 30 s
Submitted: 6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 3

Identifies material uncertaintyRightWrong

Haiku 5.5 · API

Names the risk of SMB/enterprise revenue loss and the condition (mid-market targets + inbound enterprise flow) to revisit the enterprise decision in 12 months.

Gemini 3.8 Flash · API

The memo does not name specific unknowns that could change the decision, nor does it say how they would be resolved or what result would change the call.

Avoids unsupported claimsWrongRight

Haiku 5.5 · API

Presents 'root cause: we've let the loudest requests set the roadmap' and 'SMB remains our top-of-funnel' as facts without labelling them as hypotheses or citing evidence.

Gemini 3.8 Flash · API

Interpretations like 'professional services trap' are directly supported by the custom-work and margin data, and the memo does not present unsupported causes or forecasts as fact.

Proposes tests that could failRightWrong

Haiku 5.5 · API

Proposes numeric thresholds (NRR ≥ 120, churn ≤ 7%, support cost ≤ $300, zero custom weeks) and a 12-month revisit gate with specific conditions.

Gemini 3.8 Flash · API

The scorecard provides numeric targets and a measurement window but does not specify what action each outcome would trigger, so the tests lack a clear fail condition with a linked decision.

All got wrong 1

Names an advantage that's hard to copyWrongWrong

Haiku 5.5 · API

Does not name a benefit plus a barrier that would make the position hard for a well-funded competitor to copy; relies on features and current win rate.

Gemini 3.8 Flash · API

The strategy does not name a benefit customers value plus a barrier that would prevent a well-funded competitor from copying it; it relies on features and execution speed.

All mixed 1

Uses the supplied evidence correctlyMixedMixed

Haiku 5.5 · API

Two statements about current situation (root cause and SMB as top-of-funnel) are not from the brief nor arithmetic.

Gemini 3.8 Flash · API

All factual claims about the current situation are taken directly from the supplied context or derived by simple arithmetic from it.

All got right 7

Addresses the actual decisionRightRight

Haiku 5.5 · API

Commits early to disagreeing with the CEO, says enterprise is the wrong bet for next year, and specifies conditions to revisit.

Gemini 3.8 Flash · API

The memo commits early and unambiguously to 'You are wrong' and a mid-market-led strategy, framed for the CEO, and implies the condition for revisiting enterprise is when the platform is hardened.

Respects explicit constraintsRightRight

Haiku 5.5 · API

The memo contains diagnosis, guiding policy, actions, and measures, is under 1,200 words, and addresses the CEO.

Gemini 3.8 Flash · API

The output is a strategy memo under 1,200 words, includes diagnosis, guiding policy, coherent actions, and success metrics, and tells the CEO they are wrong.

Produces the required deliverableRightRight

Haiku 5.5 · API

Complete strategy memo in the requested form, addressed to the CEO, within word limit, and directly usable by the exec team.

Gemini 3.8 Flash · API

The memo is complete, in the right form for the CEO, within the word limit, and contains actionable sections a CEO could use.

Reads the segment economicsRightRight

Haiku 5.5 · API

Works through payback, retention, and support cost for all three segments and concludes mid-market is the strongest, SMB loses after support, enterprise is slow and costly.

Gemini 3.8 Flash · API

The memo works through payback, retention, and support cost for all three segments and correctly identifies mid-market as strongest, SMB as money-losing after support, and enterprise as slow and costly.

Tells the CEO plainlyRightRight

Haiku 5.5 · API

States plainly and respectfully that the enterprise-first plan is wrong, backed by the segment numbers.

Gemini 3.8 Flash · API

The memo states 'You are wrong' directly, supports it with segment economics, and offers a clear alternative.

Diagnosis before prescriptionRightRight

Haiku 5.5 · API

Names a crux (resource allocation driven by loudest requests) and all major actions address it.

Gemini 3.8 Flash · API

The diagnosis identifies the crux (mid-market is the engine, enterprise is a distraction, SMB is a cost drain) and each major action visibly addresses it.

Says what it gives upRightRight

Haiku 5.5 · API

Names clear stops: custom builds for deals, unlimited SMB phone support, dedicated enterprise hiring and outbound, single-segment edge-case features.

Gemini 3.8 Flash · API

The memo explicitly lists what it stops: custom engineering for deals, hiring 10 enterprise AEs, and phone support on sub-$5,000 ARR plans, with reasons.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 84% of 76 checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	GPT-6 AstrawithChatGPT	83.3	88.5	2	None
2	Haiku 5.5withAPI	77.1	80.8	2	None
3	Sonnet 5.5withAPI	77.1	80.8	2	None
4	GPT-6.1 SolwithAPI	79.2	76.9	2	None
5	GPT-6 LunawithAPI	77.1	76.9	2	None
6	Opus 5.5withClaude	83.3	65.4	2	None
7	Gemini 3.8 FlashwithAPI	68.8	57.7	2	None
8	Gemini 3.5 Flash-LitewithGemini	60.4	42.3	2	None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Nobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
GPT-6 Luna · API	74	1.0×
Haiku 5.5 · API	80	1.7×
Gemini 3.5 Flash-Lite · Gemini	49	3.1× (partly estimated)
Gemini 3.8 Flash · API	61	8.1×
GPT-6.1 Sol · API	77	12×
Sonnet 5.5 · API	78	27×
Opus 5.5 · Claude	71	68× (partly estimated)
GPT-6 Astra · ChatGPT	87	84× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Writing a strategy memo that leadership can act on.

Why it matters

Strategy that doesn't choose isn't strategy. Models write fluent strategic prose easily; making a choice the evidence supports and naming what it gives up is harder.

What good looks like

Diagnoses the one obstacle that matters before prescribing
Makes one clear choice and names what is sacrificed
Grounds the choice in the supplied evidence
Surfaces the tension or contradiction in the data
Defines how we would know it is working

Deliberately not measured

Market sizing accuracy beyond the supplied data
Financial modelling

Capability tested

Choosing where to play and what not to do, from supplied evidence

The failure we’re looking for

Strategic-sounding aspirations without a diagnosis or a choice

Grading

Decision model and LLM judge, calibrated against a blind PM review

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

Diagnosis before prescription

Names an advantage that's hard to copy

Says what it gives up

Plus our standard checks

The tasks

The outputs

Output

Diagnosis

Guiding policy

Coherent actions for next year

How we'll know it's working

Risks

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Run

Output

Executive Summary: The Direct Answer

1. Diagnosis: Where Value Is Created and Destroyed

2. Guiding Policy: Where We Play, Where We Win, and What We Stop

3. Coherent Actions for Next Year

4. Scorecard: How We’ll Know It’s Working

Conclusion

Grades and run details

Decision model checks

Run

Check by check

Where they differ 3

All got wrong 1

All mixed 1

All got right 7

Results

What you get for the cost

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured