Humanity’s Last PRD
Latest drop: Haiku 5.5 · 8 Oct
Tasks / Strategy

Develop product strategy

Can the model make a coherent choice grounded in the evidence, rather than list aspirations?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 38% were usable with at most a quick edit.

Reliably right

  1. Produces the required deliverable100% pass
    Complete strategy document in the requested form, length, and for the named reader; the board could act on it.
    GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
  2. Diagnosis before prescription100% pass
    Diagnosis: students can't find the right tutor; actions (intake, recommendations, rematch, manual matching) directly address it.
    GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
  3. Tells the CEO plainly100% pass
    It clearly states the enterprise-first plan is wrong for next year, backs it with numbers, and presents the alternative.
    GPT-6 Astra · ChatGPT · Enterprise, says the CEO

Where it slips

  1. Names an advantage that's hard to copy11% pass
    Does not name a benefit and a barrier that a well-funded competitor couldn't copy; no moat is identified.
    GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
  2. Proposes tests that could fail42% pass
    The SMB pilot lacks a numeric threshold for success; while the payback guardrail has a threshold and action, not every proposed test/gate meets the full requirement.
    GPT-6 Astra · ChatGPT · Enterprise, says the CEO
  3. Avoids unsupported claims44% pass
    Presents 'root cause: we've let the loudest requests set the roadmap' and 'SMB remains our top-of-funnel' as facts without labelling them as hypotheses or citing evidence.
    Haiku 5.5 · API · Enterprise, says the CEO

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Diagnosis before prescription

    Does the strategy name the crux, the specific obstacle or opportunity the evidence points to, and do the proposed actions follow from that diagnosis?

    Passes when States one crux grounded in the supplied evidence, and each major action visibly addresses it.

    Bangaly KabaLenny’s Podcast

  2. Names an advantage that's hard to copy

    Does the strategy say what will make its position hard for an informed, well-funded competitor to copy (a benefit customers value plus a barrier to copying it), rather than relying on being first, on features, or on funding?

    Passes when Names the benefit and the barrier, and why a capable competitor couldn't simply replicate it.

    Hamilton HelmerLenny’s Podcast

  3. Says what it gives up

    Does the strategy name what the company will stop doing, or deliberately not do, in order to pursue it?

    Passes when Names at least one real thing being dropped or declined, with the reason it loses out.

    Gokul RajaramLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Proposes tests that could fail · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the VP Product at Cadence. Our CEO wants a strategy for next year the exec team can rally behind, and believes it's enterprise (the note is below). Write the strategy memo, in no more than 1,200 words: the diagnosis, the guiding policy (where we play and what we stop), the coherent actions for next year, and how we'll know it's working. Tell the CEO if you think they're wrong. The files are attached.

What the model was given6 items: About Cadence, ceo_note.md, unit_economics_by_segment.csv, enterprise_detail.md, mid_market_detail.md, smb_detail.md
About CadenceStaff scheduling and time tracking, sold to businesses with hourly workers.
ceo_note.md1 lines · Download
From the CEO: "I want a strategy for next year that the exec team can rally behind. I believe it's enterprise. Tell me if I'm wrong."
unit_economics_by_segment.csv3 rows · Download
segment,customers,avg_arr_usd,gross_margin_pct,support_cost_per_customer_year_usd,cac_usd,payback_months,net_revenue_retention_pct,logo_churn_year_pct
SMB self-serve,6200,1100,78,610,740,31,88,34
Mid-market,890,14500,81,1900,9800,9,118,7
Enterprise,34,96000,64,21000,118000,26,104,9
enterprise_detail.md4 lines · Download
# Enterprise detail

- Average sales cycle: 9 months. 5 of the last 8 deals needed custom features (built by the core team, about 14 engineer-weeks each).
- The CEO's plan: "Win enterprise. Hire 10 enterprise AEs and a solutions team, and build SSO, audit logs and custom workflows this year."
mid_market_detail.md5 lines · Download
# Mid-market detail

- Best-retaining segment. Expansion comes from adding locations and seats.
- Top requests: multi-location reporting, payroll integrations, shift templates.
- Win rate against our main competitor: 61%.
smb_detail.md4 lines · Download
# SMB self-serve detail

- 62% of support tickets come from SMB accounts. Free trial converts at 4%.
- Most SMB customers are on the $79/month plan with unlimited phone support.
What a strong answer doesThe answer key the graders mark against

Diagnoses from the unit economics: mid-market is the engine (payback 9 months, NRR 118%, churn 7%, wins 61% against the main competitor); SMB self-serve loses money once support and churn are counted (about $1,100 a year at 78% gross margin is about $860, minus $610 of support, leaves about $250 a year against $740 to acquire, with 34% leaving each year; payback 31 months, NRR 88%) and generates 62% of tickets; enterprise is slow and expensive (payback 26 months, 64% gross margin, 9-month cycles, 5 of 8 deals needing about 14 engineer-weeks of custom work). Tells the CEO the enterprise bet is wrong for next year: it would pull the core team into custom work and stretch payback. Policy: win mid-market, expanding on locations and seats; stop subsidising SMB support (change the plan or pricing, self-serve support, raise the entry price) rather than abandon it; serve enterprise only where it buys the product as is. Coherent actions: mid-market roadmap (multi-location reporting, payroll integrations, shift templates), SMB support and pricing changes, no custom enterprise work. Names what it gives up and how it'll know (mid-market NRR and payback, SMB support cost per customer).

Critical failures (cap the score at 40)
  • Recommends hiring the enterprise team and building custom workflows as next year's main strategy
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

GPT-6 AstraChatGPT87Usable with a quick editSonnet 5.5API87Usable with a quick editGPT-6.1 SolAPI85Usable with a quick editHaiku 5.5API81Usable with a quick editGPT-6 LunaAPI75Needs more workOpus 5.5Claude71Needs more workGemini 3.8 FlashAPI67Needs more workGemini 3.5 Flash-LiteGemini56Needs more work
GPT-6.1 SolAPI

Usable with a quick edit10 of 12 checks passed

The output is a well-reasoned, complete strategy memo with correct evidence; a product manager could use it with light edits to wording or emphasis without re-checking facts or logic.

Output

Cadence: Next-Year Strategy

To: CEO and executive team From: VP Product Decision: Make multi-location mid-market customers our growth engine—not enterprise.

1. Diagnosis: Our constraint is profitable, repeatable growth

I think the enterprise-first plan is wrong for next year. Enterprise may become attractive, but our current evidence does not justify hiring ten enterprise AEs, adding a solutions team, and committing the core product team to enterprise features simultaneously. That would scale a motion we have not yet made repeatable.

Our strongest business is already mid-market:

SegmentCurrent ARR¹Gross marginCAC paybackNRRAnnual logo churn
------:---:---:---:---:
SMB self-serve$6.8M78%31 months88%34%
Mid-market$12.9M81%9 months118%7%
Enterprise$3.3M64%26 months104%9%

¹Customers multiplied by average ARR.

Mid-market contributes approximately 56% of current ARR, has the best retention and margins, and expands through additional locations and seats. Its 61% win rate against our main competitor suggests a competitive advantage worth reinforcing.

Enterprise has larger contracts, but not better economics. Sales take nine months; five of the last eight deals required custom features, consuming approximately 70 engineer-weeks. Those deals impose an opportunity cost beyond their acquisition expense: the core team cannot simultaneously build bespoke workflows and improve the product that drives our best-performing segment. Enterprise NRR of 104% provides little evidence that expansion will compensate for that burden.

SMB is also not an efficient growth engine today. Trial conversion is 4%, annual churn is 34%, and SMB generates 62% of support tickets. Unlimited phone support on a predominantly $79/month plan is a service promise poorly matched to customer value.

We should validate how support costs enter reported gross margin and payback before constructing a fully loaded profitability model. But the segment ranking is clear enough to act.

The central problem is resource allocation: we are tempted by large contracts while underinvesting in the customers where acquisition, retention, expansion, and product fit already reinforce one another.

2. Guiding policy: Win repeatable multi-location growth

Where we play: Businesses with hourly workers operating multiple locations, within our existing mid-market segment. We will refine that profile by industry, location count, and integration needs using our strongest retained and expanding cohorts—not assume every mid-market customer is equally attractive.

How we win: Make it easier to run scheduling, time tracking, and payroll handoffs consistently across locations. Land with a repeatable implementation, then grow as customers add locations and seats.

What we stop:

  • Do not execute the proposed enterprise hiring plan. No ten-AE expansion or dedicated solutions-team buildout next year.
  • Stop allowing individual prospects to commandeer the core roadmap. Custom workflows are not a strategic product category.
  • Stop optimizing SMB for signup volume while ignoring conversion, churn, and support burden.
  • Stop promising unlimited phone support to new low-price SMB customers. Change existing commitments carefully at renewal, with notice and clear alternatives.

We will retain and serve existing enterprise customers and accept selective new opportunities that fit the standard product. Enterprise remains an option to prove—not the company’s organizing strategy.

3. Coherent actions for next year

A. Build the multi-location operating backbone

Owners: Product and Engineering

Commit at least 70% of planned product capacity to the mid-market growth loop and its enabling reliability work.

Prioritize the three requests supported by current customer evidence:

  1. Multi-location reporting: Give operators a consistent view across sites.
  2. Payroll integrations: Reduce manual handoffs and implementation friction; prioritize providers used by our target customers.
  3. Shift templates: Make repeatable scheduling easier across locations.

In Q1, confirm priority and scope through customer research, usage data, and lost-deal analysis. Ship in staged releases during Q2–Q3, then use Q4 to improve adoption and remove rollout friction. Success means customers using these capabilities—not merely features released.

SSO and audit logs enter the roadmap only where they unlock repeatable demand from our target segment or address necessary security obligations. We will not bundle them with speculative custom workflows.

B. Align selling and customer success around location expansion

Owners: Sales and Customer Success

Reallocate growth investment toward mid-market rather than enterprise headcount. Increase selling capacity in stages, conditional on cohort payback remaining healthy.

Create a standard qualification and implementation playbook. Establish an explicit location-expansion plan for suitable customers, including rollout milestones, adoption checkpoints, and commercial packaging that makes expansion straightforward.

Track why we win against our main competitor and turn that evidence into positioning, demonstrations, and onboarding. Product adoption, selling, and expansion should reinforce the same promise.

C. Repair SMB economics without abandoning customers

Owners: Growth and Support

For new customers, introduce clear support tiers, reserving phone support for appropriately priced plans. Improve onboarding and self-service around the largest ticket drivers. Test packaging and onboarding changes before broad rollout.

Do not increase SMB acquisition spending until conversion and cohort payback improve. Existing customers receive a transparent transition, not a surprise withdrawal of service.

D. Put enterprise behind an investment gate

Owners: CEO, Sales, Finance, and Product

Require executive approval for exceptions, with engineering effort and opportunity cost recorded before signing. Prefer standard-product deals; do not hide custom development inside revenue targets.

Reconsider a dedicated enterprise expansion only when successive cohorts demonstrate materially improved margins, payback approaching 12–15 months, and wins that do not depend on bespoke core-team work. A handful of large bookings is insufficient evidence.

4. How we will know it is working

Finance will establish consistent cohort definitions and quarterly reporting in Q1. Our proposed operating targets are:

  • Mid-market: NRR at least 120%; annual logo churn no worse than 7%; gross margin at least 80%; CAC payback no worse than 12 months as we scale.
  • Competitive position: Maintain at least a 60% win rate against our main competitor, using a consistent opportunity definition.
  • Product impact: Improve time to activate additional locations and adoption of reporting, integrations, and templates against Q1 baselines.
  • SMB repair: Lower tickets per customer and support cost per customer, improve trial conversion, and shorten new-cohort payback without worsening retention.
  • Execution discipline: Deliver the agreed mid-market capacity allocation and report every enterprise customization exception.

These are targets, not a revenue forecast. Finance should translate the strategy into a funded operating plan.

Bottom line: Next year should prove that Cadence can compound a repeatable advantage. Mid-market already shows that pattern. Enterprise currently does not.

Grades and run details

Decision model 83 · LLM judge 85
Decision model checks
  • passUses the supplied evidence correctly26%
  • passAddresses the actual decision100%
  • passRespects explicit constraints49%
  • passIdentifies material uncertainty48%
  • partialAvoids unsupported claims43%
  • passProduces the required deliverable72%
  • partialReads the segment economics36%
  • passTells the CEO plainly100%
  • passDiagnosis before prescription100%
  • partialProposes tests that could fail78%
  • partialNames an advantage that's hard to copy70%
  • passSays what it gives up100%
Run
Run
#1
API response time
43 s
Submitted
6 Oct 2026
Gemini 3.8 FlashAPI

Needs more than a quick edit8 of 12 checks passed

The memo fails to identify material uncertainties, lacks actionable test-fail triggers, and does not define a defensible moat, so a product manager would need to rework those strategic elements.

Output

MEMORANDUM

TO: Chief Executive Officer FROM: VP of Product DATE: October 24, 2023 SUBJECT: Next Year Strategy: Diagnosis, Guiding Policy, and Execution Roadmap

---

Executive Summary: The Direct Answer

You asked: “I believe it’s enterprise. Tell me if I’m wrong.”

You are wrong. Going all-in on Enterprise next year would be an expensive strategic error.

Enterprise is currently acting as a high-friction, low-margin professional services trap. It requires 9-month sales cycles, consumes massive engineering resources (70 engineer-weeks burned on bespoke code for just five recent deals), and yields an uninspiring 104% NRR at a 26-month CAC payback. Hiring 10 enterprise AEs and a solutions team right now will accelerate cash burn while starving our real growth engine.

That growth engine is Mid-Market. Mid-market is exceptional: 81% gross margin, 9-month CAC payback, 118% NRR, 7% logo churn, and a 61% win rate against our primary competitor.

Next year, our rallying cry must be: Dominate Mid-Market, automate SMB into positive cash flow, and gate Enterprise until our core platform is ready.

---

1. Diagnosis: Where Value Is Created and Destroyed

Our current ARR is ~$23M across three starkly different businesses:

MetricSMB Self-ServeMid-MarketEnterprise
:---:---:---:---
Active Customers6,20089034
Total ARR (approx.)$6.82M (30%)$12.91M (56%)$3.26M (14%)
Avg. ARR / Customer$1,100$14,500$96,000
Gross Margin78%81%64%
Support Cost / Customer / Yr$610 (55% of ARR)$1,900 (13% of ARR)$21,000 (22% of ARR)
CAC Payback31 months9 months26 months
NRR / Logo Churn88% / 34%118% / 7%104% / 9%
Competitive Win RateN/A61%Elongated (9 mo)

#### Three Key Findings:

  1. Mid-Market is an elite SaaS business. Customers expand by adding locations and headcount (118% NRR). Our sales efficiency is world-class (9-month payback vs. industry average of 14–18 months). Most importantly, their feature requests (multi-location reporting, payroll integrations, shift templates) are repeatable, software-native, and scale across the entire cohort.
  2. Enterprise is a bespoke distraction masquerading as scale. At 64% gross margin, 26-month payback, and only 104% NRR, our Enterprise motion does not scale. 5 of our last 8 deals required custom development—averaging 14 engineer-weeks per deal. We are acting as a subsidized software development shop for 34 accounts, pulling our best product talent away from core roadmap investments.
  3. SMB is drowning our operations. SMB accounts for 62% of all support tickets. We charge $79/month ($948/yr) while spending $610/year in support per customer due to unlimited phone support. With 34% logo churn and 88% NRR, the 31-month CAC payback means a significant portion of SMB customers churn before breaking even on acquisition and service costs.

---

2. Guiding Policy: Where We Play, Where We Win, and What We Stop

Our guiding policy for next year is Repeatable Scale over Bespoke Revenue.

``` ┌──────────────────────────────────────────────┐ │ MID-MARKET: ACCELERATE │ │ Core growth vector; 118% NRR; 9-mo payback │ └──────────────────────┬───────────────────────┘ │ ┌─────────────────────────┴─────────────────────────┐ ▼ ▼ ┌─────────────────────────────┐ ┌─────────────────────────────────────┐ │ SMB: AUTOMATE │ │ ENTERPRISE: DISCIPLINE │ │ Eliminate phone support │ │ Freeze custom dev; sell off-the- │ │ Product-led self-service │ │ shelf MM product at premium tier │ └─────────────────────────────┘ └─────────────────────────────────────┘ ```

#### Where We Play: * The Sweet Spot: Multi-unit hourly employers with 50–500 employees (Mid-Market). These buyers have acute scheduling complexity across locations but lack the bureaucratic demands for custom software architectures.

#### What We Stop: * Stop custom engineering for individual deals. Zero custom branching or deal-contingent roadmap commits. * Stop hiring 10 Enterprise AEs. We will not deploy massive quota capacity into a 9-month sales cycle with an unstandardized product. * Stop offering phone support on sub-$5,000 ARR plans. Unlimited phone support for $79/month ends in Q1.

---

3. Coherent Actions for Next Year

To execute this guiding policy, our actions across Product, GTM, and Operations must reinforce each other.

#### A. Product & Engineering: The Repeatable Core * Deliver the Mid-Market Trifecta (H1): Allocate 70% of product capacity to build: 1. Multi-location reporting & dashboarding (enables regional managers to view compliance and labor costs across units). 2. Tier-1 payroll integrations (ADP, Gusto, Paychex, UKG) to make scheduling-to-payroll frictionless. 3. Advanced shift templates & auto-scheduling to drive rapid store-level adoption. * Enterprise Baseline, not Bespoke (H2): Build only standardized administrative features (SAML SSO, SCIM provisioning, audit logs, role-based access control). If an enterprise buyer cannot deploy our standard multi-tenant product with these enterprise rails, we walk away. * SMB Self-Serve & Deflection: Revamp onboarding flows to drive the 4% trial conversion rate upward and deploy in-app guided setup, chat deflection, and an updated knowledge center to cut incoming tickets.

#### B. Go-To-Market: Scaling the Engine * Reallocate Capital from Enterprise to Mid-Market: Instead of 10 Enterprise AEs, hire 6 Mid-Market AEs and 2 MM Sales Engineers. With our 9-month payback and 61% win rate, MM sales hires will reach cash-flow break-even within the year. * Establish an Enterprise Gatekeeper Rule: The existing enterprise sales team may close deals only under two conditions: (1) standard product only (contractually zero custom dev), and (2) minimum contract value of $100k ARR with standard pricing. * Refocus Inbound & Outbound: Concentrate marketing spend on multi-unit franchisees, regional hospitality groups, and mid-sized healthcare operators.

#### C. Operations & Support: Unit Economic Realignment * Tiered Support Model (Effective Q1): * SMB / Self-serve: In-app digital assistant, community, and email support (target: reduce support cost from $610 to <$150/year). * Mid-Market: Priority email, live chat, and scheduled onboarding calls. * Enterprise: Dedicated CSM and SLA-backed escalation. * Grandfathering/Pricing Migration for SMB: Shift low-tier customers requiring phone support to an “Assisted Support” add-on ($99/month), turning a cost center into a margin-accretive tier.

---

4. Scorecard: How We’ll Know It’s Working

We will monitor operational health and strategic traction via four quarterly benchmarks:

PillarMetricBaselineEnd of Next Year Target
:---:---:---:---
Mid-Market GrowthNet New ARR Added~$4.5M/yr run rate$8.0M+
Expansion HealthMid-Market NRR118%125%+
SMB Unit EconomicsSMB Blended Support Cost / Year$610<$200
SMB PaybackSMB CAC Payback Months31 months<14 months
Product DisciplineEngineer-weeks spent on custom deal features~70 weeks (last 8 deals)0 weeks
Company EfficiencyBlended Gross Margin76%82%+

---

Conclusion

Enterprise is seductive because the contract sizes are large. But right now, chasing it will turn us into a bespoke agency with depressed margins and burnt-out engineers.

Mid-Market is our path to category leadership. It gives us high win rates, compounding expansion, rapid payback, and product leverage. Once we harden the platform with multi-location mid-market capabilities and standardized enterprise controls, we can attack the enterprise from a position of structural strength—not structural compromise.

Grades and run details

Decision model 71 · LLM judge 69
Decision model checks
  • failUses the supplied evidence correctly15%
  • passAddresses the actual decision64%
  • passRespects explicit constraints28%
  • failIdentifies material uncertainty80%
  • partialAvoids unsupported claims41%
  • passProduces the required deliverable83%
  • passReads the segment economics96%
  • passTells the CEO plainly100%
  • passDiagnosis before prescription100%
  • partialProposes tests that could fail54%
  • partialNames an advantage that's hard to copy27%
  • passSays what it gives up100%
Run
Run
#1
API response time
30 s
Submitted
6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 2

Uses the supplied evidence correctlyRightMixed
GPT-6.1 Sol · API

All facts and figures about the current situation are directly from the supplied context or correct arithmetic; nothing is invented.

Gemini 3.8 Flash · API

All factual claims about the current situation are taken directly from the supplied context or derived by simple arithmetic from it.

Identifies material uncertaintyRightWrong
GPT-6.1 Sol · API

Acknowledges uncertainty in support cost accounting and sets explicit, measurable conditions for revisiting enterprise.

Gemini 3.8 Flash · API

The memo does not name specific unknowns that could change the decision, nor does it say how they would be resolved or what result would change the call.

All got wrong 2

Proposes tests that could failWrongWrong
GPT-6.1 Sol · API

The operating targets for mid-market and SMB lack explicit actions if missed; only the enterprise gate specifies what each result triggers.

Gemini 3.8 Flash · API

The scorecard provides numeric targets and a measurement window but does not specify what action each outcome would trigger, so the tests lack a clear fail condition with a linked decision.

Names an advantage that's hard to copyWrongWrong
GPT-6.1 Sol · API

Does not name a durable advantage or barrier that would prevent a well-funded competitor from copying the multi-location focus and features.

Gemini 3.8 Flash · API

The strategy does not name a benefit customers value plus a barrier that would prevent a well-funded competitor from copying it; it relies on features and execution speed.

All got right 8

Addresses the actual decisionRightRight
GPT-6.1 Sol · API

Commits early to mid-market as the growth engine, says the enterprise plan is wrong, and states conditions for reconsidering enterprise.

Gemini 3.8 Flash · API

The memo commits early and unambiguously to 'You are wrong' and a mid-market-led strategy, framed for the CEO, and implies the condition for revisiting enterprise is when the platform is hardened.

Respects explicit constraintsRightRight
GPT-6.1 Sol · API

Memo is within 1,200 words, includes all requested sections, and addresses the CEO directly.

Gemini 3.8 Flash · API

The output is a strategy memo under 1,200 words, includes diagnosis, guiding policy, coherent actions, and success metrics, and tells the CEO they are wrong.

Avoids unsupported claimsRightRight
GPT-6.1 Sol · API

Interpretations are clearly framed as diagnosis or opinion, not as established facts.

Gemini 3.8 Flash · API

Interpretations like 'professional services trap' are directly supported by the custom-work and margin data, and the memo does not present unsupported causes or forecasts as fact.

Produces the required deliverableRightRight
GPT-6.1 Sol · API

Complete strategy memo with diagnosis, policy, actions, and metrics, usable by the exec team as is.

Gemini 3.8 Flash · API

The memo is complete, in the right form for the CEO, within the word limit, and contains actionable sections a CEO could use.

Reads the segment economicsRightRight
GPT-6.1 Sol · API

Works through payback, retention, and support costs for all three segments, correctly concluding mid-market is strongest, SMB is inefficient, and enterprise is slow and costly.

Gemini 3.8 Flash · API

The memo works through payback, retention, and support cost for all three segments and correctly identifies mid-market as strongest, SMB as money-losing after support, and enterprise as slow and costly.

Tells the CEO plainlyRightRight
GPT-6.1 Sol · API

Directly tells the CEO the enterprise-first plan is wrong, with supporting numbers and a clear alternative.

Gemini 3.8 Flash · API

The memo states 'You are wrong' directly, supports it with segment economics, and offers a clear alternative.

Diagnosis before prescriptionRightRight
GPT-6.1 Sol · API

Diagnosis names resource allocation as the crux, and all major actions reallocate resources toward mid-market and away from enterprise and SMB inefficiencies.

Gemini 3.8 Flash · API

The diagnosis identifies the crux (mid-market is the engine, enterprise is a distraction, SMB is a cost drain) and each major action visibly addresses it.

Says what it gives upRightRight
GPT-6.1 Sol · API

Explicitly lists stops: enterprise hiring, custom work, SMB signup optimization, and unlimited phone support for low-price plans.

Gemini 3.8 Flash · API

The memo explicitly lists what it stops: custom engineering for deals, hiring 10 enterprise AEs, and phone support on sub-$5,000 ARR plans, with reasons.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 84% of 76 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstraChatGPT83.388.52None
2Haiku 5.5API77.180.82None
3Sonnet 5.5API77.180.82None
4GPT-6.1 SolAPI79.276.92None
5GPT-6 LunaAPI77.176.92None
6Opus 5.5Claude83.365.42None
7Gemini 3.8 FlashAPI68.857.72None
8Gemini 3.5 Flash-LiteGemini60.442.32None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.4050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 87, 84× the cheapest (1 of 2 outputs estimated)Haiku 5.5 · API: task score 80, 1.7× the cheapestSonnet 5.5 · API: task score 78, 27× the cheapestGPT-6.1 Sol · API: task score 77, 12× the cheapestGPT-6 Luna · API: task score 74, the cheapestOpus 5.5 · Claude: task score 71, 68× the cheapest (1 of 2 outputs estimated)Gemini 3.8 Flash · API: task score 61, 8.1× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 49, 3.1× the cheapest (1 of 2 outputs estimated)GPT-6 Astra · ChatGPTHaiku 5.5 · APISonnet 5.5 · APIGPT-6.1 Sol · APIGPT-6 Luna · APIOpus 5.5 · ClaudeGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Writing a strategy memo that leadership can act on.

Why it matters

Strategy that doesn't choose isn't strategy. Models write fluent strategic prose easily; making a choice the evidence supports and naming what it gives up is harder.

What good looks like

  • Diagnoses the one obstacle that matters before prescribing
  • Makes one clear choice and names what is sacrificed
  • Grounds the choice in the supplied evidence
  • Surfaces the tension or contradiction in the data
  • Defines how we would know it is working

Deliberately not measured

  • Market sizing accuracy beyond the supplied data
  • Financial modelling
Capability tested

Choosing where to play and what not to do, from supplied evidence

The failure we’re looking for

Strategic-sounding aspirations without a diagnosis or a choice

Grading

Decision model and LLM judge, calibrated against a blind PM review