Tasks / Strategy

Develop product strategy

Can the model make a coherent choice grounded in the evidence, rather than list aspirations?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 33% were usable with at most a quick edit.

Reliably right

Respects explicit constraints100% pass
One-page, ~500 words, one major bet, addressed to the CEO for the board.
GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
Produces the required deliverable100% pass
Complete strategy document in the requested form, length, and for the named reader; the board could act on it.
GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
Diagnosis before prescription100% pass
Diagnosis: students can't find the right tutor; actions (intake, recommendations, rematch, manual matching) directly address it.
GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace

Where it slips

Names an advantage that's hard to copy10% pass
Does not name a benefit and a barrier that a well-funded competitor couldn't copy; no moat is identified.
GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
Proposes tests that could fail46% pass
The operating targets for mid-market and SMB lack explicit actions if missed; only the enterprise gate specifies what each result triggers.
GPT-6.1 Sol · API · Enterprise, says the CEO
Avoids unsupported claims54% pass
Presents 'root cause: we've let the loudest requests set the roadmap' and 'SMB remains our top-of-funnel' as facts without labelling them as hypotheses or citing evidence.
Haiku 5.5 · API · Enterprise, says the CEO

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Diagnosis before prescription
Does the strategy name the crux, the specific obstacle or opportunity the evidence points to, and do the proposed actions follow from that diagnosis?
Passes when States one crux grounded in the supplied evidence, and each major action visibly addresses it.
Bangaly KabaLenny’s Podcast
Names an advantage that's hard to copy
Does the strategy say what will make its position hard for an informed, well-funded competitor to copy (a benefit customers value plus a barrier to copying it), rather than relying on being first, on features, or on funding?
Passes when Names the benefit and the barrier, and why a capable competitor couldn't simply replicate it.
Hamilton HelmerLenny’s Podcast
Says what it gives up
Does the strategy name what the company will stop doing, or deliberately not do, in order to pursue it?
Passes when Names at least one real thing being dropped or declined, with the reason it loses out.
Gokul RajaramLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Proposes tests that could fail · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the VP Product at Cadence. Our CEO wants a strategy for next year the exec team can rally behind, and believes it's enterprise (the note is below). Write the strategy memo, in no more than 1,200 words: the diagnosis, the guiding policy (where we play and what we stop), the coherent actions for next year, and how we'll know it's working. Tell the CEO if you think they're wrong. The files are attached.

What the model was given6 items: About Cadence, ceo_note.md, unit_economics_by_segment.csv, enterprise_detail.md, mid_market_detail.md, smb_detail.md

About CadenceStaff scheduling and time tracking, sold to businesses with hourly workers.

ceo_note.md1 lines · Download

From the CEO: "I want a strategy for next year that the exec team can rally behind. I believe it's enterprise. Tell me if I'm wrong."

unit_economics_by_segment.csv3 rows · Download

segment,customers,avg_arr_usd,gross_margin_pct,support_cost_per_customer_year_usd,cac_usd,payback_months,net_revenue_retention_pct,logo_churn_year_pct
SMB self-serve,6200,1100,78,610,740,31,88,34
Mid-market,890,14500,81,1900,9800,9,118,7
Enterprise,34,96000,64,21000,118000,26,104,9

enterprise_detail.md4 lines · Download

# Enterprise detail

- Average sales cycle: 9 months. 5 of the last 8 deals needed custom features (built by the core team, about 14 engineer-weeks each).
- The CEO's plan: "Win enterprise. Hire 10 enterprise AEs and a solutions team, and build SSO, audit logs and custom workflows this year."

mid_market_detail.md5 lines · Download

# Mid-market detail

- Best-retaining segment. Expansion comes from adding locations and seats.
- Top requests: multi-location reporting, payroll integrations, shift templates.
- Win rate against our main competitor: 61%.

smb_detail.md4 lines · Download

# SMB self-serve detail

- 62% of support tickets come from SMB accounts. Free trial converts at 4%.
- Most SMB customers are on the $79/month plan with unlimited phone support.

What a strong answer doesThe answer key the graders mark against

Diagnoses from the unit economics: mid-market is the engine (payback 9 months, NRR 118%, churn 7%, wins 61% against the main competitor); SMB self-serve loses money once support and churn are counted (about $1,100 a year at 78% gross margin is about $860, minus $610 of support, leaves about $250 a year against $740 to acquire, with 34% leaving each year; payback 31 months, NRR 88%) and generates 62% of tickets; enterprise is slow and expensive (payback 26 months, 64% gross margin, 9-month cycles, 5 of 8 deals needing about 14 engineer-weeks of custom work). Tells the CEO the enterprise bet is wrong for next year: it would pull the core team into custom work and stretch payback. Policy: win mid-market, expanding on locations and seats; stop subsidising SMB support (change the plan or pricing, self-serve support, raise the entry price) rather than abandon it; serve enterprise only where it buys the product as is. Coherent actions: mid-market roadmap (multi-location reporting, payroll integrations, shift templates), SMB support and pricing changes, no custom enterprise work. Names what it gives up and how it'll know (mid-market NRR and payback, SMB support cost per customer).

Critical failures (cap the score at 40)

Recommends hiring the enterprise team and building custom workflows as next year's main strategy

Task

Staff-level · v1.1

The outputs

9 setups answered this brief.

Sonnet 5.5API90Usable with a quick edit GPT-6.1 SolAPI88Usable with a quick edit GPT-6 AstraChatGPT87Usable with a quick edit Haiku 5.5API81Usable with a quick edit GPT-6 LunaAPI79Needs more work Opus 5.5Claude73Needs more work Gemini 4 ArgonAPI65Needs more work Gemini 3.8 FlashAPI62Needs more work Gemini 3.5 Flash-LiteGemini54Needs more work

Gemini 4 ArgonAPI

Needs more than a quick edit7 of 12 checks passed

The arithmetic error on engineer-years would need re-checking, and the memo lacks explicit kill criteria for its targets, requiring additions to be production-ready.

Output

MEMO

To: CEO & Executive Team From: VP of Product Subject: FY Strategy: Why We Must Bet on Mid-Market, Not Enterprise

You asked for a strategy the executive team can rally behind, and you asked me to tell you if you’re wrong about Enterprise.

Looking at our data, you are wrong about Enterprise. Pursuing an enterprise-first strategy next year will burn cash, hijack our engineering roadmap, and starve the exact part of our business that has already found product-market fit: Mid-Market.

Here is the diagnosis of where we stand, where we should play instead, and how we will execute next year.

---

1. The Diagnosis: Where Value is Created (and Destroyed)

Today, Cadence is generating ~$23M in ARR across three segments, but the underlying economics tell three completely different stories:

Mid-Market is a compounding growth engine ($12.9M ARR | 890 customers) Mid-market is our largest and healthiest segment by every metric. We have an 81% gross margin, a 9-month CAC payback, and a 61% win rate against our main competitor. Most importantly, Net Revenue Retention (NRR) is 118% with just 7% logo churn. Mid-market customers buy Cadence, stay, and expand naturally by adding locations and seats.

Enterprise is a bespoke consulting trap ($3.26M ARR | 34 customers) The $96k average ARR is seductive, but the unit economics are broken. Gross margins drop to 64% (dragged down by $21k/year in support costs per customer), CAC is $118k, and payback takes 26 months. Worse, 9% logo churn and 104% NRR prove that even when we land enterprises, they don't significantly expand, and they churn faster than mid-market.

Furthermore, we do not have a repeatable enterprise product. Five of our last eight enterprise deals required custom features built by our core engineering team, costing ~14 engineer-weeks each (nearly 1.5 engineering years spent on just five deals). If we hire 10 Enterprise AEs right now, they will spend 9 months ramping and hunting, only to pressure R&D into becoming a custom dev shop to close their pipeline. Building SSO, audit logs, and custom workflows gets us to the table, but it won't fix a broken 26-month payback model.

SMB is a support-heavy leaky bucket ($6.82M ARR | 6,200 customers) SMB looks like a 78% gross margin business on paper, but that appears to ignore the $3.78M we spend annually supporting them ($610/customer across 6,200 accounts). We are giving customers paying $948–$1,100/year unlimited phone support, allowing them to generate 62% of our support ticket volume. Combined with 34% annual churn, 88% NRR, and a 31-month payback, we are losing money on the average SMB customer before they ever pay back their CAC.

---

2. The Guiding Policy: Double Down on Mid-Market, Fix SMB, Pause Enterprise

Our strategy for next year is to dominate multi-location Mid-Market businesses while turning SMB into a true self-serve cash cow to fund our growth.

Where We Play: * Mid-Market is our primary growth engine. We will align Product, Engineering, Marketing, and Sales around multi-location businesses with hourly workers. We win here 61% of the time without custom code, and every dollar of ARR we land turns into $1.18 a year later. * SMB becomes strictly low-touch. SMB should only exist as an automated, high-margin feeder system that requires near-zero human intervention and graduates growing customers into Mid-Market.

What We Stop Doing: * Stop custom engineering for sales deals. Effective immediately, we no longer trade core engineering weeks for enterprise logos. * Stop outbound Enterprise expansion. We will not hire the 10 Enterprise AEs or the Solutions team. Existing enterprise accounts will be supported, and inbound enterprise leads can buy our off-the-shelf product if it fits, but we will not build for them next year. * Stop unlimited phone support for SMBs. We cannot afford a $610/year support cost on a $79/month plan.

---

3. Coherent Actions for Next Year

To execute this strategy, every department will align around three core initiatives:

#### A. Product & Engineering: Unlock Mid-Market Expansion & Velocity Instead of spending the year on enterprise security plumbing (SSO, audit logs) and custom workflows, 80% of R&D capacity will go to the three capabilities our Mid-Market customers are actively asking for—all of which directly drive seat/location expansion and retention: 1. Multi-Location Reporting (Q1–Q2): Make it effortless for regional managers and VP Ops to see labor costs and attendance across 5 to 50+ sites. This directly fuels our 118% NRR expansion loop. 2. Deep Payroll Integrations (Q1–Q3): Build native, bi-directional integrations with the top mid-market payroll providers. This increases stickiness, reduces support burden, and removes friction during competitive switch-outs. 3. Cross-Location Shift Templates (Q2): Allow operations teams to standardize and deploy scheduling templates across new locations in minutes, accelerating time-to-value when customers expand. 4. SMB Onboarding & Deflection (Remaining 20% R&D): Improve in-app guided setup and self-serve help tooling to lift our 4% trial conversion rate and cut ticket generation at the source.

#### B. Go-To-Market (Sales & Marketing): Pour Fuel on the 9-Month Payback 1. Reallocate Headcount to Mid-Market: Take the budget earmarked for the 10 Enterprise AEs and Solutions engineers and hire Mid-Market AEs and Expansion/Account Managers instead. With a 9-month payback and a shorter sales cycle, these reps will generate net-new ARR this year. 2. Run the Competitive Displacement Playbook: Focus marketing spend and SDR outbound exclusively on multi-location businesses currently using our main competitor, leveraging our 61% win rate and new payroll integrations. 3. Incentivize Location Expansion: Structure AM compensation around getting existing Mid-Market logos to roll Cadence out to100% of their locations.

#### C. Customer Success & Pricing: Restructure SMB Economics 1. Eliminate Phone Support on the $79/mo Plan: Transition SMB to digital-only support (AI/chat deflection, community, email) and self-serve help centers. 2. Monetize High-Touch Support: Introduce a paid "Premier Support" add-on or require upgrades to a Mid-Market tier for phone support and dedicated onboarding. Even if this causes a one-time spike in SMB logo churn, shedding unprofitable, high-ticket accounts will immediately free up millions in support capacity to service Mid-Market.

---

4. How We’ll Know It’s Working (2025 Targets)

We will track our progress quarterly against five shared company metrics:

Mid-Market ARR Growth: Grow Mid-Market ARR by >45% (from $12.9M to ~$19M+), taking it from 56% of total company revenue to over 70%.
Mid-Market Efficiency: Maintain CAC payback < 12 months, win rate > 60%, and NRR ≥ 118% as we scale the Mid-Market sales team.
Engineering Focus: Zero core engineering weeks spent on single-customer custom features (down from ~70+ engineer-weeks).
SMB Support Cost: Cut SMB support cost per customer from $610/year to < $200/year, and reduce SMB share of total support tickets from 62% to < 30%.
SMB Payback & Trial Conversion: Lift free trial conversion from 4% to > 6% via better self-serve onboarding, bringing SMB CAC payback under 18 months.

A Final Note on Enterprise: Saying "no" to Enterprise right now isn't saying "no" forever. Once we dominate Mid-Market, our product will naturally mature, and our largest Mid-Market customers will pull us upmarket from a position of strength. Right now, Mid-Market is working. Let’s pour everything we have into winning it.

Claims the judge couldn’t find in the brief

The five enterprise custom deals consumed nearly 1.5 engineering years

Grades and run details

Decision model 75 · LLM judge 62

Decision model checks

passUses the supplied evidence correctly10%
passAddresses the actual decision16%
passRespects explicit constraints35%
failIdentifies material uncertainty83%
failAvoids unsupported claims30%
passProduces the required deliverable79%
passReads the segment economics98%
passTells the CEO plainly100%
passDiagnosis before prescription100%
partialProposes tests that could fail38%
partialNames an advantage that's hard to copy38%
passSays what it gives up100%

Run

Run: #1
API response time: 21 s
Submitted: 13 Oct 2026

Check by check

Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Got wrong 3

Identifies material uncertaintyWrong

Does not name specific unknowns that could change the decision or the conditions (e.g., enterprise NRR or payback thresholds) that would flip the call.

Proposes tests that could failWrong

Proposes aspirational targets but no thresholds with explicit fail actions or measurement windows that would trigger a change.

Names an advantage that's hard to copyWrong

Does not identify a durable advantage that competitors cannot copy; relies on features and focus without specifying a barrier.

Mixed 2

Uses the supplied evidence correctlyMixed

The claim 'nearly 1.5 engineering years' from 70 weeks is inaccurate (actual ~1.35 years), so not every statement follows correctly from arithmetic.

Avoids unsupported claimsMixed

Interpretations are framed as diagnosis and conclusions based on data, not presented as established fact.

Got right 7

Addresses the actual decisionRight

Clearly states the CEO is wrong about enterprise, commits to mid-market, and says enterprise can wait.

Respects explicit constraintsRight

Memo format, addresses CEO and exec team, under 1,200 words, and directly tells CEO they are wrong.

Produces the required deliverableRight

Delivers a complete strategy memo to the CEO and exec team within the word limit, actionable as written.

Reads the segment economicsRight

Explicitly works through payback, retention, and support cost for all three segments, concluding mid-market is strongest.

Tells the CEO plainlyRight

Tells the CEO plainly and respectfully that the enterprise-first plan is wrong, using the numbers.

Diagnosis before prescriptionRight

Diagnosis identifies mid-market as engine, enterprise as cost trap, SMB as leaky bucket; proposed actions address each.

Says what it gives upRight

Names stop custom enterprise work, outbound enterprise expansion, and unlimited SMB phone support, with reasons.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 82% of 76 checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	GPT-6 AstrawithChatGPT	87.5	88.5	2	None
2	Haiku 5.5withAPI	85.4	80.8	2	None
3	Sonnet 5.5withAPI	87.5	80.8	2	None
4	GPT-6.1 SolwithAPI	89.6	76.9	2	None
5	GPT-6 LunawithAPI	83.3	76.9	2	None
6	Opus 5.5withClaude	89.6	65.4	2	None
7	Gemini 3.8 FlashwithAPI	62.5	57.7	2	None
8	Gemini 4 ArgonwithAPI	66.7	53.8	2	None
9	Gemini 3.5 Flash-LitewithGemini	66.7	42.3	2	None

What you get for the cost

Haiku 5.5 · API scores within 4 points of GPT-6 Astra · ChatGPT at a fiftieth of the cost.

Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
GPT-6 Luna · API	77	1.0×
Haiku 5.5 · API	84	1.7×
Gemini 3.5 Flash-Lite · Gemini	52	3.1× (partly estimated)
Gemini 3.8 Flash · API	58	8.1×
GPT-6.1 Sol · API	82	12×
Gemini 4 Argon · API	58	12×
Sonnet 5.5 · API	83	27×
Opus 5.5 · Claude	74	68× (partly estimated)
GPT-6 Astra · ChatGPT	88	84× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Writing a strategy memo that leadership can act on.

Why it matters

Strategy that doesn't choose isn't strategy. Models write fluent strategic prose easily; making a choice the evidence supports and naming what it gives up is harder.

What good looks like

Diagnoses the one obstacle that matters before prescribing
Makes one clear choice and names what is sacrificed
Grounds the choice in the supplied evidence
Surfaces the tension or contradiction in the data
Defines how we would know it is working

Deliberately not measured

Market sizing accuracy beyond the supplied data
Financial modelling

Capability tested

Choosing where to play and what not to do, from supplied evidence

The failure we’re looking for

Strategic-sounding aspirations without a diagnosis or a choice

Grading

Decision model and LLM judge, calibrated against a blind PM review