Tasks / Strategy

Develop product strategy

Can the model make a coherent choice grounded in the evidence, rather than list aspirations?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 38% were usable with at most a quick edit.

Reliably right

Produces the required deliverable100% pass
Complete strategy document in the requested form, length, and for the named reader; the board could act on it.
GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
Diagnosis before prescription100% pass
Diagnosis: students can't find the right tutor; actions (intake, recommendations, rematch, manual matching) directly address it.
GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
Tells the CEO plainly100% pass
It clearly states the enterprise-first plan is wrong for next year, backs it with numbers, and presents the alternative.
GPT-6 Astra · ChatGPT · Enterprise, says the CEO

Where it slips

Names an advantage that's hard to copy11% pass
Does not name a benefit and a barrier that a well-funded competitor couldn't copy; no moat is identified.
GPT-6 Astra · ChatGPT · Supply or demand for a stalled marketplace
Proposes tests that could fail42% pass
The SMB pilot lacks a numeric threshold for success; while the payback guardrail has a threshold and action, not every proposed test/gate meets the full requirement.
GPT-6 Astra · ChatGPT · Enterprise, says the CEO
Avoids unsupported claims44% pass
Presents 'root cause: we've let the loudest requests set the roadmap' and 'SMB remains our top-of-funnel' as facts without labelling them as hypotheses or citing evidence.
Haiku 5.5 · API · Enterprise, says the CEO

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Diagnosis before prescription
Does the strategy name the crux, the specific obstacle or opportunity the evidence points to, and do the proposed actions follow from that diagnosis?
Passes when States one crux grounded in the supplied evidence, and each major action visibly addresses it.
Bangaly KabaLenny’s Podcast
Names an advantage that's hard to copy
Does the strategy say what will make its position hard for an informed, well-funded competitor to copy (a benefit customers value plus a barrier to copying it), rather than relying on being first, on features, or on funding?
Passes when Names the benefit and the barrier, and why a capable competitor couldn't simply replicate it.
Hamilton HelmerLenny’s Podcast
Says what it gives up
Does the strategy name what the company will stop doing, or deliberately not do, in order to pursue it?
Passes when Names at least one real thing being dropped or declined, with the reason it loses out.
Gokul RajaramLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Proposes tests that could fail · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the VP Product at Cadence. Our CEO wants a strategy for next year the exec team can rally behind, and believes it's enterprise (the note is below). Write the strategy memo, in no more than 1,200 words: the diagnosis, the guiding policy (where we play and what we stop), the coherent actions for next year, and how we'll know it's working. Tell the CEO if you think they're wrong. The files are attached.

What the model was given6 items: About Cadence, ceo_note.md, unit_economics_by_segment.csv, enterprise_detail.md, mid_market_detail.md, smb_detail.md

About CadenceStaff scheduling and time tracking, sold to businesses with hourly workers.

ceo_note.md1 lines · Download

From the CEO: "I want a strategy for next year that the exec team can rally behind. I believe it's enterprise. Tell me if I'm wrong."

unit_economics_by_segment.csv3 rows · Download

segment,customers,avg_arr_usd,gross_margin_pct,support_cost_per_customer_year_usd,cac_usd,payback_months,net_revenue_retention_pct,logo_churn_year_pct
SMB self-serve,6200,1100,78,610,740,31,88,34
Mid-market,890,14500,81,1900,9800,9,118,7
Enterprise,34,96000,64,21000,118000,26,104,9

enterprise_detail.md4 lines · Download

# Enterprise detail

- Average sales cycle: 9 months. 5 of the last 8 deals needed custom features (built by the core team, about 14 engineer-weeks each).
- The CEO's plan: "Win enterprise. Hire 10 enterprise AEs and a solutions team, and build SSO, audit logs and custom workflows this year."

mid_market_detail.md5 lines · Download

# Mid-market detail

- Best-retaining segment. Expansion comes from adding locations and seats.
- Top requests: multi-location reporting, payroll integrations, shift templates.
- Win rate against our main competitor: 61%.

smb_detail.md4 lines · Download

# SMB self-serve detail

- 62% of support tickets come from SMB accounts. Free trial converts at 4%.
- Most SMB customers are on the $79/month plan with unlimited phone support.

What a strong answer doesThe answer key the graders mark against

Diagnoses from the unit economics: mid-market is the engine (payback 9 months, NRR 118%, churn 7%, wins 61% against the main competitor); SMB self-serve loses money once support and churn are counted (about $1,100 a year at 78% gross margin is about $860, minus $610 of support, leaves about $250 a year against $740 to acquire, with 34% leaving each year; payback 31 months, NRR 88%) and generates 62% of tickets; enterprise is slow and expensive (payback 26 months, 64% gross margin, 9-month cycles, 5 of 8 deals needing about 14 engineer-weeks of custom work). Tells the CEO the enterprise bet is wrong for next year: it would pull the core team into custom work and stretch payback. Policy: win mid-market, expanding on locations and seats; stop subsidising SMB support (change the plan or pricing, self-serve support, raise the entry price) rather than abandon it; serve enterprise only where it buys the product as is. Coherent actions: mid-market roadmap (multi-location reporting, payroll integrations, shift templates), SMB support and pricing changes, no custom enterprise work. Names what it gives up and how it'll know (mid-market NRR and payback, SMB support cost per customer).

Critical failures (cap the score at 40)

Recommends hiring the enterprise team and building custom workflows as next year's main strategy

Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

GPT-6 AstraChatGPT87Usable with a quick edit Sonnet 5.5API87Usable with a quick edit GPT-6.1 SolAPI85Usable with a quick edit Haiku 5.5API81Usable with a quick edit GPT-6 LunaAPI75Needs more work Opus 5.5Claude71Needs more work Gemini 3.8 FlashAPI67Needs more work Gemini 3.5 Flash-LiteGemini56Needs more work

Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit6 of 12 checks passed

The memo contains factual inaccuracies (incorrect SMB average price and net-negative margin claim) that a PM would need to correct before use; otherwise the reasoning is sound but the errors undermine trust in the evidence.

Output

MEMORANDUM

TO: CEO FROM: VP of Product DATE: October 24, 2023 SUBJECT: Product & Go-To-Market Strategy for Next Year

You asked for a strategy the executive team can rally behind, and stated your belief that our future lies in enterprise.

You are wrong. If we pivot our resources toward enterprise next year, we will stall our fastest-growing segment, burn cash on an unscalable product architecture, and distract the company from our true economic engine.

Here is our reality, our diagnosis, and the strategy we should actually pursue.

---

1. The Diagnosis: Where the Numbers Really Point

To evaluate your enterprise hypothesis, we have to look at the unit economics across our three segments (SMB, Mid-Market, Enterprise):

SMB Self-Serve (6,200 customers, $1,100 ARR): This segment is an anchor dragging us down. A 34% logo churn and 88% NRR means we are running to stay in place. Worse, 62% of our support tickets come from SMB accounts who pay an average of $79/month while demanding unlimited phone support. We are subsidizing their operations at a severe net-negative margin once support costs are factored in.
Enterprise (34 customers, $96,000 ARR): Enterprise looks glossy on the top line, but the mechanics are broken. Our payback period is 26 months, our gross margin has cratered to 64% (due to heavy support and custom engineering), and NRR is a mediocre 104%. As noted in the enterprise detail, 5 of our last 8 deals required custom features built by our core engineering team—averaging 14 engineer-weeks each. We are currently operating as a custom software consultancy for large accounts, not a scalable SaaS business.
Mid-Market (890 customers, $14,500 ARR): This is our crown jewel. Mid-market boasts an 81% gross margin, a lightning-fast 9-month payback period, a stellar 118% NRR, and a low 7% logo churn. We win 61% of deals against our main competitor here. Expansion happens organically as clients add locations and seats.

The Diagnosis: We are suffering from strategic schizophrenia. We are spending engineering cycles building bespoke features for 34 enterprise logos while burning support bandwidth on 6,200 low-end SMBs. Meanwhile, we are under-investing in the Mid-Market—the only segment where we have product-market fit, high retention, and clear competitive dominance.

---

2. The Guiding Policy

Where We Play: We will focus our go-to-market, product development, and company roadmap entirely on the Mid-Market (companies with roughly 100 to 1,000 hourly workers).

What We Stop: 1. We stop custom enterprise builds. We will not pull core engineers off the roadmap to build one-off workflows for enterprise deals. If a whale customer requires custom engineering, we walk away. 2. We stop optimizing for SMB acquisition. We will sunset active acquisition efforts for self-serve SMBs and strip away unscalable touchpoints (like unlimited phone support for $79/mo accounts) to stop the margin bleed. 3. We stop treating Enterprise as our primary growth vector. We will not hire 10 enterprise AEs or build a dedicated solutions team.

By pulling back from Enterprise and SMB, we concentrate all our capital on scaling what is already working brilliantly in the Mid-Market.

---

3. Coherent Actions for Next Year

To capture the mid-market opportunity, product and go-to-market will execute three synchronized moves:

#### Action 1: Productize for Mid-Market Expansion (Product & Engineering) Instead of building custom enterprise workflows (SSO, audit logs, and bespoke routing), our engineering team will build the top three requested features that drive multi-location expansion and stickiness in the Mid-Market: * Multi-location reporting: Giving regional managers roll-up visibility across disparate sites. * Robust payroll integrations: Deepening native syncs with major mid-market payroll providers to reduce administrative friction. * Advanced shift templates: Allowing complex scheduling rules to be deployed instantly across new locations.

#### Action 2: Pivot Go-To-Market to Land-and-Expand (Sales & Marketing) We will reallocate budget from SMB acquisition and enterprise pipe-dreams directly into Mid-Market account executive and customer success headcount. * Our motion will focus on landing a mid-market business at a single location or division, then leveraging our stellar NRR (118%) to expand horizontally into new locations. * We will introduce tiered packaging that naturally captures revenue as mid-market clients add locations and seats.

#### Action 3: Streamline or Harvest the SMB Tier (Operations & Support) We cannot drop our existing 6,200 SMB customers overnight, but we must stop bleeding margin on them. * Transition self-serve SMB support away from expensive phone queues to a self-service knowledge base and AI-driven chat. * Raise prices on legacy low-tier plans to match the true cost-to-serve, accepting natural churn as a feature, not a bug, which frees up support capacity for our mid-market clients.

---

4. How We'll Know It's Working

If we execute this focus on the Mid-Market, we will see specific operational shifts within the next 12 months:

Mid-Market ARR Growth: Mid-market ARR growth rate accelerates by 35% year-over-year as we direct 80% of our sales and marketing capacity here.
Payback and Margins: Our blended gross margin climbs back toward 80%+ as high-margin mid-market revenue outpaces low-margin enterprise custom work and unprofitable SMB volume. Payback periods across the active customer base remain under 10 months.
NRR Expansion: Mid-market NRR increases from 118% to 122%+ driven by faster multi-location adoption of our new reporting and payroll features.
Engineering Velocity: Feature delivery velocity increases by 40% because engineering is no longer interrupted by 14-week custom enterprise implementations.

Summary

Pivoting to enterprise feels ambitious, but the data tells us it is a trap. We would be trading a high-margin, highly defensible leadership position in the Mid-Market to become a low-margin, high-friction consultancy for a handful of giants.

Let’s double down on where we win, where retention is highest, and where our unit economics sing. Let's own the Mid-Market next year.

Grades and run details

Decision model 71 · LLM judge 46

Decision model checks

partialUses the supplied evidence correctly3%
passAddresses the actual decision45%
passRespects explicit constraints54%
failIdentifies material uncertainty99%
failAvoids unsupported claims42%
passProduces the required deliverable76%
passReads the segment economics71%
passTells the CEO plainly100%
passDiagnosis before prescription100%
partialProposes tests that could fail49%
partialNames an advantage that's hard to copy59%
passSays what it gives up100%

Run

Run: #1
Time to output: 9 s
Submitted: 5 Oct 2026

GPT-6 AstraChatGPT

Usable with a quick edit10 of 12 checks passed

The memo is evidence-based, clear, and actionable; a PM could adopt it with minor wording tweaks without re-verifying facts or redoing the analysis.

Output

Cadence: Next Year’s Strategy

To: CEO and Executive Team From: VP Product Decision: Make multi-location mid-market businesses our growth engine—not enterprise.

1. Diagnosis: Our constraint is focus, not deal size

I believe the enterprise-first plan is wrong for next year. Enterprise may become an attractive market, but our current evidence does not justify hiring ten enterprise AEs or orienting the roadmap around their deals.

Cadence has three materially different businesses:

Segment	Current ARR*	Reported CAC payback	NRR	Annual logo churn
---	---:	---:	---:	---:
SMB self-serve	$6.82M	31 months	88%	34%
Mid-market	$12.91M	9 months	118%	7%
Enterprise	$3.26M	26 months	104%	9%

Customers × average ARR.

Mid-market already generates 56% of ARR. It combines our best retention, strongest expansion, fastest acquisition payback, and highest gross margin, at 81%. A 61% win rate against our main competitor suggests a defensible product advantage. Expansion through locations and seats gives us a repeatable growth mechanism.

Enterprise offers larger contracts, but not yet a better business. Gross margin is 64%, the sales cycle averages nine months, and five of the last eight deals required custom development. Those five deals consumed approximately 70 engineer-weeks. Hiring sellers before fixing this delivery model risks scaling commitments faster than reusable product capability.

SMB is a different problem: substantial revenue with weak retention and disproportionate service demand. It generates 62% of support tickets; annual support cost averages $610 against $1,100 in ARR. Unlimited phone support on a $79 monthly plan is misaligned with self-service economics. A 4% trial conversion rate compounds the acquisition problem.

The central challenge is therefore allocating scarce product, engineering, and commercial capacity toward repeatable, retained revenue. Enterprise customization and SMB service demands are drawing capacity away from the segment where our model works best.

2. Guiding policy: Win multi-location mid-market

Where we play

Our primary customer is a mid-market business with hourly workers across multiple locations, where scheduling consistency, consolidated visibility, and payroll handoffs create recurring operational pain.

Our promise: make running the next location easier than running the first.

We will deepen that advantage and turn successful initial deployments into additional locations and seats. We will first target industries and acquisition channels where our own mid-market cohorts show the strongest retention and payback; the supplied evidence does not yet support choosing a specific vertical.

Enterprise remains a selective adjacency. SMB remains a supported, increasingly self-service business—not our primary acquisition investment.

What we stop

Do not approve the ten-enterprise-AE hiring plan or a dedicated solutions-team buildout.
Stop letting individual enterprise deals dictate core-team custom workflows.
Do not commit to the entire enterprise feature bundle. SSO and audit logs must earn priority through demonstrated, reusable demand—not their “enterprise” label.
Stop scaling SMB paid acquisition under current economics.
End unlimited phone support as the default for new low-priced plans, while honoring existing commitments and managing any migration carefully.
Stop broad roadmap work that cannot demonstrate a connection to mid-market acquisition, retention, expansion, or essential platform health.

We will continue supporting existing enterprise and SMB customers. Focus is not abandonment.

3. Coherent actions for next year

A. Build the multi-location growth loop

Product and Engineering: Give mid-market the clear majority of discretionary roadmap capacity, after reliability, security, and contractual obligations.

Sequence three connected capabilities:

Multi-location reporting: Give central operators visibility into staffing and hours across their estate.
Payroll integrations: Prioritize providers used by the largest share of our target accounts; reduce recurring payroll friction.
Shift templates: Make successful scheduling practices reusable across locations.

In Q1, validate scope with retained, expanding customers and establish adoption and workflow baselines. Release reporting and the first prioritized integration in the first half; expand integrations and template-based rollout in the second half. Measure customer outcomes and subsequent location adoption, not merely shipment.

B. Sell and expand the same repeatable product

Sales and Customer Success: Redirect incremental commercial investment toward mid-market rather than enterprise hiring.

Package a standard multi-location deployment, define onboarding milestones, and build expansion reviews around customers’ remaining locations. Align incentives with retained recurring revenue and successful activation, not bookings alone.

Increase acquisition spending in stages only while segment payback remains within our guardrail. Do not assume today’s economics survive unlimited scaling.

C. Repair SMB service economics

Support, Product, and Finance: Identify the largest ticket drivers in Q1. Improve onboarding and in-product guidance around those problems; test appropriately priced phone support and service tiers for new customers.

Pilot changes before broad rollout. Evaluate support savings alongside conversion, retention, and revenue effects. A lower ticket count caused by driving customers away is not success.

D. Keep enterprise as a bounded learning option

Sales and Product: Accept new enterprise opportunities only when they fit the standard product, require no bespoke core-team development, and meet Finance-approved deal economics.

Review enterprise quarterly. Reconsider a larger investment only after multiple customers buy and deploy substantially the same product, with improving margin and acquisition payback. One marquee logo is not proof.

E. Establish a shared economic scorecard

Finance: In Q1, reconcile acquisition costs, implementation costs, support allocation, and payback definitions across segments. Confirm whether support is already included in gross margin before subtracting it again. Track custom engineering separately so its opportunity cost remains visible.

4. How we will know it is working

These are proposed operating targets, not forecasts:

Measure	Baseline	Year-end target / guardrail
---	---:	---:
Mid-market ARR	$12.91M	At least $16.1M, approximately 25% growth
Mid-market NRR	118%	At least 120%
Mid-market annual logo churn	7%	No worse than 7%
Mid-market CAC payback	9 months	At most 12 months while scaling
Mid-market gross margin	81%	At least 80%
SMB tickets per active account	Establish in Q1	Reduce 25%, without worsening cohort revenue retention
New bespoke enterprise core-team work	Five of last eight deals	Zero

The executive team will review results monthly and resource allocation quarterly. Leading indicators are faster location activation, adoption of the new workflows, and expansion pipeline; financial outcomes remain the test.

The strategy is not “avoid enterprise.” It is “scale the repeatable business we have earned.” Mid-market gives us the strongest foundation for growth next year—and a healthier route into enterprise later.

Grades and run details

Decision model 88 · LLM judge 85

Decision model checks

passUses the supplied evidence correctly19%
passAddresses the actual decision99%
passRespects explicit constraints47%
passIdentifies material uncertainty45%
partialAvoids unsupported claims43%
passProduces the required deliverable69%
passReads the segment economics66%
passTells the CEO plainly100%
passDiagnosis before prescription100%
partialProposes tests that could fail74%
partialNames an advantage that's hard to copy76%
passSays what it gives up100%

Run

Run: #1
Time to output: 39 s
Submitted: 5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.