Humanity’s Last PRD
Latest drop: Haiku 5.5 · 8 Oct
Tasks / Growth

Diagnose a retention problem

Can the model work out from raw data what's really happening to retention, separating churn from measurement and seasonality, and find the cause?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 81% were usable with at most a quick edit.

Reliably right

  1. Produces the required deliverable100% pass
    The diagnosis is complete, actionable, and in the right format for Leo; a PM could act on it with light edits.
    GPT-6 Astra · ChatGPT · Flat, while sign-ups grow
  2. Understands before it justifies100% pass
    It tests the summer explanation against the data, finds the break aligns with the onboarding change not the paid spend increase, and reaches its own diagnosis from the evidence.
    GPT-6 Astra · ChatGPT · Flat, while sign-ups grow
  3. Splits growth into new, lost and returning100% pass
    It splits growth into new, retained, churned, and returning users, and uses the net growth equation to show which flow changed.
    GPT-6 Astra · ChatGPT · Flat, while sign-ups grow

Where it slips

  1. Gets the base of every number right61% pass
    The SMB adoption churn calculation is incorrect; 39% of ~$7.4k/month churn is ~$2.9k/month, not $34k MRR, and the annualized figure is ~$34.8k ARR, not $0.4M.
    Sonnet 5.5 · API · The collapse that wasn't
  2. Avoids unsupported claims67% pass
    The output presents causal interpretations (e.g., onboarding changes broke retention, low-intent users) as established fact without labelling them as hypotheses.
    Gemini 3.5 Flash-Lite · Gemini · Flat, while sign-ups grow
  3. Identifies material uncertainty77% pass
    The output does not name specific unknowns that could change the decision or say how they would be resolved.
    Gemini 3.5 Flash-Lite · Gemini · Flat, while sign-ups grow

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Understands before it justifies

    Does the diagnosis start from what the data shows (which users, when, at what point in their life with the product) before accepting or proposing an explanation, including the one it was handed?

    Passes when Tests the explanation it was given against the data, and reaches its own diagnosis from the evidence.

    Bangaly KabaLenny’s Podcast

  2. Splits growth into new, lost and returning

    Does the diagnosis separate new users or revenue from what was lost and what came back, rather than reading a single total?

    Passes when Uses growth accounting (new, retained, churned, returning, or new, expansion, contraction, churn) to show which flow changed.

    Naomi GleitLenny’s Podcast

  3. Reads the shape of the curves

    Does the diagnosis read how retention curves behave over time (flattening, still falling, or dipping and recovering) and what each shape means?

    Passes when Describes the curve shapes in the data and draws the right conclusion from each, such as a dip that recovers being seasonal.

    Product and growth leadersOn Lenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for activation at Stride. Leo, our Head of Growth, thinks weekly actives are flat because it's summer, and wants to double paid acquisition (his note is below). Before he does, write him a diagnosis of no more than 800 words: what's actually happening to retention, the evidence, and what we should do instead of (or before) spending more on acquisition. The files are attached.

What the model was given6 items: About Stride, leo_note.md, weekly_growth_accounting.csv (weeks run Monday to Sunday), weekly_signup_cohorts.csv (share of each week's sign-ups active in each following week), release_and_campaign_log.md, winback_campaign_results.md
About StrideA fitness habit app: short guided workouts and a weekly plan. Free, with Premium at $9.99 a month.
leo_note.md1 lines · Download
From Leo (Head of Growth): "Weekly actives have been flat for three months even though sign-ups are up 30%. It's summer: people are outdoors and on holiday. I want to double paid acquisition to push through it."
weekly_growth_accounting.csv (weeks run Monday to Sunday)week,week_starting,new_users,returning_users,retained_users,churned_users,weekly_active_users 1,2026-07-06,6034,973,44884,7116,51891 2,2026-07-13,6142,989,44529,7362,51660 3,2026-07-20,6184,1040,45074,6586,52298 4,2026-07-27,6385,991,44843,7455,52219 5,2026-08-03,6597,975,44085,8134,51657 6,2026-08-10,6689,2541,42516,9141,51746 7,2026-08-17,6779,2666,42506,9240,51951 8,2026-08-24,6955,2552,42723,9228,52230 9,2026-08-31,7156,2618,42134,10096,51908 10,2026-09-07,7230,2675,42393,9515,52298 11,2026-09-14,7564,2586,41984,10314,52134 12,2026-09-21,7550,2557,41808,10326,51915 13,2026-09-28,7847,2635,41636,10279,52118
weekly_signup_cohorts.csv (share of each week's sign-ups active in each following week)signup_week,signups,week_1_retained_pct,week_2_retained_pct,week_3_retained_pct,week_4_retained_pct,week_5_retained_pct,week_6_retained_pct,week_7_retained_pct,week_8_retained_pct 1,6000,55.4,43.2,36.4,34.2,31.9,30.9,29.8,30.1 2,6150,55.3,43.2,37.0,34.2,32.4,30.5,29.9,30.2 3,6300,55.2,43.0,37.7,33.4,32.1,30.6,29.8,30.0 4,6450,54.3,42.3,37.0,33.4,32.4,31.4,30.0,29.5 5,6600,58.4,39.4,29.6,24.5,22.1,21.1,19.8,19.9 6,6750,58.6,39.9,30.1,24.4,21.8,21.2,19.9, 7,6900,58.4,40.5,29.4,24.9,22.2,21.2,, 8,7050,58.1,40.1,29.9,25.0,21.3,,, 9,7200,57.5,40.5,29.7,25.0,,,, 10,7350,58.3,40.5,29.4,,,,, 11,7500,57.4,40.6,,,,,, 12,7650,57.5,,,,,,,
release_and_campaign_log.md6 lines · Download
# Release and campaign log

- Week 3: New workout library (40 new routines).
- Week 5: Shorter sign-up. Removed the goal-setting step ("What do you want to achieve, and how many days a week?") to cut sign-up time from 3 minutes to 1. Sign-up completion rose from 61% to 74%.
- Week 6: Win-back emails start: anyone inactive for 21 days gets "We miss you: here's a free week of Premium".
- Week 9: Paid social budget up 25%.
winback_campaign_results.md3 lines · Download
# Win-back campaign, weeks 6 to 13 (from the lifecycle team)

About 1,600 extra returning users a week compared with before the campaign. Of users who came back through a win-back email, 12% were still active two weeks later. Of users who came back on their own before the campaign, 41% were.
What a strong answer doesThe answer key the graders mark against

Shows that flat weekly actives aren't the summer: new users rose about 30% (about 6,000 to 7,800 a week) but churned users rose faster (about 7,100 to 10,300), and returning users jumped from about 1,000 to 2,600 a week only because of the win-back emails from week 6, of whom 12% stay two weeks against 41% of organic returners, so actives are being propped up. The cohort table shows why churn rose: cohorts from week 5, when the goal-setting step was removed from sign-up, retain far worse (week 4 about 25% against 34%, week 8 about 20% against 30%) even though more people finish sign-up. So more paid acquisition would pour more users into a leaky first month. Recommends restoring goal-setting (or testing it back against the short sign-up, judged on week-4 retention, not sign-up completion), measuring returning users separately so the campaign doesn't hide churn, and holding paid acquisition until new-user retention recovers.

Critical failures (cap the score at 40)
  • Recommends doubling paid acquisition now
Task

Mid-level · v1.0

The outputs

8 setups answered this brief.

GPT-6 AstraChatGPT100Usable with a quick editGPT-6 LunaAPI100Usable with a quick editGPT-6.1 SolAPI98Usable with a quick editSonnet 5.5API96Usable with a quick editHaiku 5.5API96Usable with a quick editOpus 5.5Claude92Usable with a quick editGemini 3.5 Flash-LiteGemini83Usable with a quick editGemini 3.8 FlashAPI83Usable with a quick edit
Gemini 3.5 Flash-LiteGemini

Usable with a quick edit9 of 12 checks passed

The reasoning and evidence are solid; a PM could add a condition for changing the call and soften causal language with a quick edit, without redoing any analysis.

Output

To: Leo, Head of Growth From: PM, Activation Subject: Diagnosis: Why WAU is flat (and why doubling paid acquisition right now will burn cash)

Leo, you suggested doubling paid acquisition to push through flat summer WAU. I’ve dug into the numbers, and the data tells a different story.

Weekly actives are flat (averaging ~52,000 for 13 weeks), but sign-ups are up 30% (from ~6,000 to ~7,800). The issue isn't seasonal churn—it's that our onboarding changes broke cohort retention, and our win-back campaigns are masking the leak.

Doubling down on paid acquisition right now will just pour more water into a bucket with a widening hole. Here is the diagnosis, the evidence, and what we should do instead.

---

1. What’s actually happening?

#### A. Onboarding optimization created low-intent users (Week 5) In Week 5, we removed the goal-setting step to cut sign-up time from 3 minutes to 1. Completion jumped from 61% to 74%, and sign-ups grew.

However, look at the cohort data (`weeklysignupcohorts.csv`). For Cohorts 1–4 (pre-change), Week 1 retention hovered around 55%, and Week 4 retention was ~34%. From Cohort 5 onward, Week 1 retention jumped to ~58% (thanks to frictionless sign-up), but Week 4 retention cratered from ~34% down to ~25%.

By removing goal-setting, we optimized for top-of-funnel conversion at the expense of user intent. We are acquiring users who don't know why they are here or what plan fits them, leading them to drop off heavily by week 4.

#### B. Core retention is quietly declining Look at the growth accounting data (`weeklygrowthaccounting.csv`). Total `retained_users` (users active week-over-week) dropped steadily from ~44,800 in Week 1 to ~41,600 in Week 13, a loss of over 3,200 loyal weekly actives.

#### C. Win-backs are masking the bleeding So why is WAU flat if retained users dropped by 3,200? Look at `returning_users`. From Weeks 1–5, returning users averaged ~980 per week. Starting in Week 6 (when the win-back email launched), returning users spiked to ~2,600 per week.

The win-back campaign is successfully dragging churned users back, propping up our headline WAU. But it’s a leaky bucket: lifecycle data shows that only 12% of win-back users are active two weeks later (compared to 41% of organic returning users). They churn right back out.

---

2. Why "Summer" isn't to blame

Your theory about summer (outdoors, holidays) would manifest as broad seasonal churn across all cohorts. But our data shows: 1. Older cohorts (1–4) maintained stable long-term retention. 2. The drop in retention is strictly tied to Cohort 5 onwards—the exact week we shipped the shorter sign-up flow.

It is product-led churn, not seasonal.

---

3. What we should do instead of doubling paid acquisition

Before we scale ad spend, we need to fix activation and stop relying on expensive band-aids. Let’s execute a 3-step pivot:

  1. Re-introduce intent-matching to onboarding (without hurting conversion): We need to qualify users so they receive the right weekly plan. Instead of a heavy 3-minute form, let’s test a lightweight, single-tap goal selector or defer personalization to the end of the first workout. We must restore Week 4 retention from 25% back toward 34%.
  2. Refine the win-back targeting: Pumping win-back emails to all churned users after 21 days is bringing back low-intent users who churn in 14 days. Let’s restrict win-backs to users who reached a milestone or completed at least 3 workouts before churning, matching the profile of our 41% organic returners.
  3. Optimize the Week 3–4 activation loop: We shipped 40 new routines in Week 3, but post-onboarding drop-off is neutralizing it. We need to focus activation efforts on guiding new users to these routines before they hit the Week 3 cliff.

Bottom line: Let's hold off on doubling paid acquisition for 4 weeks. If we fix the onboarding leak and improve post-signup intent, organic growth + recovered retention will naturally push us past our WAU plateau without wasting ad dollars on leaky traffic. Let's sync on this today.

Grades and run details

Decision model 88 · LLM judge 77
Decision model checks
  • passUses the supplied evidence correctly41%
  • passAddresses the actual decision91%
  • passRespects explicit constraints47%
  • partialIdentifies material uncertainty39%
  • failAvoids unsupported claims30%
  • passProduces the required deliverable93%
  • passFinds the onboarding change100%
  • passSees the win-back masking churn100%
  • passUnderstands before it justifies100%
  • passGets the base of every number right27%
  • passSplits growth into new, lost and returning99%
  • passReads the shape of the curves49%
Run
Run
#1
Time to output
6 s
Submitted
5 Oct 2026
Gemini 3.8 FlashAPI

Usable with a quick edit10 of 12 checks passed

The output is well-structured, evidence-based, and actionable. A PM could soften the causal claim and add a caveat about uncertainty with a quick edit, without redoing the analysis.

Output

To: Leo, Head of Growth From: PM for Activation Subject: Diagnosis: Why WAU is flat and why we shouldn’t double paid acquisition yet

*

Executive Summary

WAU is not flat because of summer seasonality. It is flat because of a structural collapse in early habit retention starting in Week 5, which is currently masked by two temporary factors: the Week 6 win-back email campaign and higher sign-up volume.

Our core retained user base has dropped by over 3,200 users, and weekly churn has surged by 44% (from ~7,100 to over 10,200). Doubling paid acquisition today would pour money into an onboarding funnel that currently retains a third fewer users long-term than it did two months ago.

Here is what the data shows and what we should do instead.

---

1. The Evidence: The Week 5 Onboarding Change Broke Habit Retention

If summer outdoor seasonality were the culprit, we would expect a gradual, platform-wide decay across all cohorts. Instead, the cohort data reveals an immediate step-change break between Cohort 4 and Cohort 5:

  • Cohorts 1–4 (Before Week 5): Week 4 retention was steady at 33.4%–34.2%, flattening to a healthy 30% at Week 8.
  • Cohorts 5+ (After Week 5): Week 4 retention plummeted to 24.5%–25.0%, and Week 8 dropped to 19.9%.

This represents a ~33% relative drop in long-term cohort retention.

Why did this happen? In Week 5, we removed the goal-setting step ("What do you want to achieve, and how many days a week?") to cut sign-up time from 3 minutes to 1 minute. While sign-up completion jumped from 61% to 74% and bumped Week 1 retention slightly (55% to 58%), we stripped away the user’s commitment and personal investment.

Stride is a habit product. Without clear goals and a tailored weekly schedule, users drop off rapidly after Week 1 because they lack direction and accountability.

---

2. The Illusion: Why Topline WAU Looks Flat

If retention collapsed by a third, why hasn't WAU cratered? Two factors created an artificial floor:

  1. Top-of-Funnel Inflow: Sign-ups grew ~30% (boosted by higher sign-up completion and the Week 9 paid budget increase), adding ~1,800 gross new sign-ups per week.
  2. The Week 6 Win-Back "Sugar Rush": The win-back campaign added ~1,600 returning users per week (jumping from ~1,000 to ~2,600 in growth accounting).

However, this resurrected volume is extremely low-quality: * Historically, 41% of organic returning users were still active two weeks later. * For win-back users, only 12% remain active after two weeks.

We are cycling low-intent, discount-incentivized users who churn almost immediately. Meanwhile, underlying retained users dropped from 45,074 (Week 3) to 41,636 (Week 13), while churn surged from 7,116 to 10,279 per week.

---

3. What Happens If We Double Paid Acquisition Now?

Doubling paid acquisition right now will accelerate cash burn without growing the business:

  • Impaired Unit Economics: With Week 8 retention down from 30% to 20%, LTV has dropped sharply. Premium conversion will suffer because fewer users build the 3-week habit required to see the value of paid features.
  • Higher CAC for Diminishing Returns: Scaling paid spend when the conversion engine is degraded will cause CAC to rise while payback periods stretch unsustainably.
  • A Leaky Bucket: In Week 13, we added 7,847 new users and churned 10,279. Adding more sign-ups does not fix a machine that is churning users faster than it can onboard them.

---

4. Recommended Action Plan

Before increasing ad spend, we need to plug the leak and fix activation.

1. Reinstate and Optimize Goal-Setting (Immediate): * Re-introduce the goal-setting and schedule-commitment step in onboarding, framed as building the user's "Personalized Weekly Plan." * Test progressive onboarding: let users quickly select goals without friction, but ensure their first workout calendar is anchored before Day 1. * Target: Return Week 4 retention to ≥33%. 2. Re-evaluate Win-Back & Lifecycle Spend: * Stop relying on the 21-day "free Premium week" email to prop up WAU. Shift CRM focus to Days 1–7 activation, reinforcing the weekly plan set during onboarding. 3. Hold Paid Social at Current Levels: * Keep paid spend flat (or pull back the Week 9 25% increase) until Cohort Week 3/4 retention recovers. 4. Scale Acquisition Once Activation Is Fixed: * Once Cohort Week 4 retention crosses 32% for two consecutive cohorts, we double paid acquisition. At that point, every dollar spent will generate 50% more long-term active users than it would today.

Grades and run details

Decision model 79 · LLM judge 85
Decision model checks
  • partialUses the supplied evidence correctly10%
  • passAddresses the actual decision100%
  • passRespects explicit constraints32%
  • failIdentifies material uncertainty6%
  • failAvoids unsupported claims47%
  • passProduces the required deliverable95%
  • passFinds the onboarding change100%
  • passSees the win-back masking churn100%
  • passUnderstands before it justifies100%
  • passGets the base of every number right13%
  • passSplits growth into new, lost and returning100%
  • passReads the shape of the curves78%
Run
Run
#1
API response time
29 s
Submitted
6 Oct 2026
Opus 5.5Claude

Usable with a quick edit11 of 12 checks passed

The diagnosis is complete, evidence-based, and actionable; a PM could use it with only minor wording tweaks, without rechecking data or redoing analysis.

Output

Weekly actives are flat because retention broke in week 5, not because of summer

To: Leo From: Activation PM Re: Why weekly actives are flat, and why we shouldn't double paid acquisition yet

Summary

Weekly actives aren't flat because of summer. Retention dropped sharply for every cohort that signed up from week 5 onward. That is the week we removed the goal-setting step from sign-up. The win-back campaign has hidden the damage by pulling back about 1,600 lapsed users a week, and most of them leave again within two weeks. Doubling acquisition now would pour more users into a bucket that leaks about a third faster than it did in July.

What's happening

1. Churn is up 40%, and it's offset by low-quality reactivations.

Weekly churn (users active last week but not this week, divided by last week's actives):

PeriodWeekly churnUsers churned per week
Weeks 2–412.7–14.3%~7,000
Week 617.7%~9,100
Weeks 9–1318.3–19.8%~10,300

Returning users jumped from about 1,000 to about 2,600 a week in week 6, the week win-back emails started. New users are up 30%. Together these just cancel the extra churn, which is why the headline stays near 52,000.

2. The break follows sign-up cohort, not the calendar.

CohortsWeek 1 retentionWeek 4 retentionWeek 8 retention
Weeks 1–4~55%~34%~30%
Weeks 5 onward~58%~25%~20%

The change is a clean step at the week-5 cohort, with no gradual drift.

3. The evidence argues against seasonality.

  • Cohorts behave differently in the same calendar weeks. If summer were the cause, everyone would dip together. Instead, the weeks 1–4 cohorts held at about 30% through all of August, while the cohorts that joined in August decayed fast.
  • July was summer too. Churn was at its lowest then.
  • September didn't help. Weeks 10–13 run into late September, and churn is at its worst.

4. The week-5 sign-up change is the most likely cause.

Removing "What do you want to achieve, and how many days a week?" cut sign-up from 3 minutes to 1. Completion rose from 61% to 74%. But we stopped asking users to commit to a goal and a weekly frequency, which is the core of a weekly-plan habit app.

The pattern fits: - First-week retention went up slightly (58% vs 55%), because the initial experience is fine. - From week 3 onward, users without a plan drift away.

Part of the drop may be compositional: the extra 13 points of completers may be lower intent. Either way, the cause points to the same step. Nothing else in the log lines up. The week-3 workout library shows no effect, and paid social only started in week 9.

5. The win-back campaign flatters the topline.

Only 12% of win-back returners are still active two weeks later, against 41% of users who came back on their own before the campaign. We're also giving away a free week of Premium to get them. Without the campaign, weekly actives would be roughly 1,500 lower and still falling.

Why doubling acquisition doesn't work right now

Per 1,000 sign-ups, about 300 were still active at week 8 before week 5. Now it's about 200. That means:

  • We need 50% more sign-ups just to keep the same number of retained users.
  • Sign-ups are up about 28%, which isn't keeping pace.
  • Cost per retained user is up roughly 50% at today's acquisition cost. That's before any rise in marginal cost from doubling spend, which usually happens.

Each extra cohort loses about 80% of its users within 8 weeks. Doubling spend would mostly buy churn.

What we should do instead

1. Fix sign-up first (start this week).

Run an A/B test with three arms: - (a) the current flow; - (b) the old flow with goal-setting; - (c) goal-setting moved to after account creation, in the first session.

Option (c) could keep most of the completion gain and restore the plan. Judge the test on week-3 retention, where the gap (about 37% vs 30%) shows up within three weeks, and confirm at week 4.

If we want to move faster, we could simply revert to (b) while testing (c). We'd give back some completion, but we'd stop losing about 10 points of week-8 retention on every new cohort.

2. Hold paid acquisition at its current level until retention recovers.

Revisit once new cohorts are back to about 33% at week 4. When we do scale, check whether paid-social cohorts retain as well as organic ones. We don't yet have cohort data split by channel.

3. Put the win-back campaign behind a holdout.

Withhold the email from 20% of eligible users to measure: - true incremental weekly actives; - Premium cannibalisation; - whether the free week leads to paid conversion.

Also try a version without the free-Premium offer, or one that sends people back into a plan: "Pick your days for this week".

4. Change how we report growth.

Make cohort retention at weeks 4 and 8, plus the weekly churn rate, our headline health metrics alongside weekly actives. Report reactivations separately. A flat topline hid a 40% rise in churn for two months.

Caveats

  • Causation isn't proven. The week-5 timing is strong evidence, but the A/B test is what will prove it.
  • The two data sources don't quite match. Cohort sign-up counts are slightly lower than "new users" in the growth accounting (for example, 6,000 vs 6,034 in week 1), so the two may define new users slightly differently. The gap is too small to change the conclusion.

Happy to walk through the cohort curves together. I'd recommend we agree on the sign-up test before any change to the paid budget.

Grades and run details

Decision model 83 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly13%
  • passAddresses the actual decision99%
  • partialRespects explicit constraints28%
  • passIdentifies material uncertainty97%
  • partialAvoids unsupported claims38%
  • passProduces the required deliverable93%
  • passFinds the onboarding change100%
  • passSees the win-back masking churn100%
  • passUnderstands before it justifies100%
  • failGets the base of every number right24%
  • passSplits growth into new, lost and returning100%
  • passReads the shape of the curves76%
Run
Run
#1
Time to output
43 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Addresses the actual decisionMixedRightRight
Gemini 3.5 Flash-Lite · Gemini

The output commits to holding off on paid acquisition but does not state what result or condition would change that call, as required.

Gemini 3.8 Flash · API

The output clearly commits to not doubling paid acquisition now and states the condition (Week 4 retention >32% for two cohorts) that would change the call.

Opus 5.5 · Claude

The output clearly commits to not doubling paid acquisition, recommends fixing sign-up first, and specifies that acquisition should be revisited when week-4 retention recovers to ~33%.

Identifies material uncertaintyWrongWrongRight
Gemini 3.5 Flash-Lite · Gemini

The output does not name specific unknowns that could change the decision or say how they would be resolved.

Gemini 3.8 Flash · API

The output does not name specific unknowns that could change the decision; it presents the causal link as certain and only gives a success target, not unresolved uncertainties.

Opus 5.5 · Claude

It names the unproven causation, the slight data mismatch, and the lack of channel-split cohort data, and says an A/B test will resolve the cause.

Avoids unsupported claimsWrongWrongRight
Gemini 3.5 Flash-Lite · Gemini

The output presents causal interpretations (e.g., onboarding changes broke retention, low-intent users) as established fact without labelling them as hypotheses.

Gemini 3.8 Flash · API

The output states as fact that removing goal-setting 'stripped away the user’s commitment and personal investment' and caused the drop, without labelling this as a hypothesis.

Opus 5.5 · Claude

The sign-up change is presented as the most likely cause, not as fact, and the output explicitly states causation isn't proven.

Gets the base of every number rightRightRightMixed
Gemini 3.5 Flash-Lite · Gemini

All percentages and differences are computed correctly from the supplied data, with no ambiguous bases.

Gemini 3.8 Flash · API

All derived percentages and differences (44% churn increase, 33% relative retention drop, 1,800 new users) are computed correctly from the supplied data.

Opus 5.5 · Claude

All derived figures (churn rates, retention percentages, relative changes) are computed correctly with the right denominators, and the churn rate base is explicitly stated.

All got right 8

Uses the supplied evidence correctlyRightRightRight
Gemini 3.5 Flash-Lite · Gemini

All factual claims about the current situation are taken directly from the supplied context or derived correctly by arithmetic.

Gemini 3.8 Flash · API

All factual claims about the current situation are directly from the supplied context or derived by correct arithmetic.

Opus 5.5 · Claude

All factual claims about the current situation are taken directly from the supplied data or derived correctly by arithmetic; the 40% churn increase is a reasonable rounding of the churn rate increase from ~13.5% to ~19%.

Respects explicit constraintsRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The output is a memo to Leo, well under 800 words, and respects the requested form and reader.

Gemini 3.8 Flash · API

The output is a diagnosis memo to Leo under 800 words, respecting the requested form, reader, and length.

Opus 5.5 · Claude

The output is a diagnosis memo addressed to Leo, well under 800 words, and respects the brief's request for a diagnosis and recommendation.

Produces the required deliverableRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The diagnosis is complete, in memo form, within the word limit, and usable by Leo with light edits.

Gemini 3.8 Flash · API

The diagnosis is complete, in memo form, under 800 words, and a PM could act on it with light edits.

Opus 5.5 · Claude

The output is a complete, actionable diagnosis memo for Leo, within the word limit, with clear sections and next steps.

Finds the onboarding changeRightRightRight
Gemini 3.5 Flash-Lite · Gemini

It links the retention drop to the Week 5 sign-up change using cohort numbers (Week 4 retention ~25% vs ~34%) and notes higher completion hides it.

Gemini 3.8 Flash · API

The diagnosis links the retention drop to the Week 5 goal-setting removal using cohort numbers and notes the sign-up completion increase hides it.

Opus 5.5 · Claude

It links the retention drop to the week-5 goal-setting removal using the cohort table, showing week-4 retention falling from ~34% to ~25% and noting higher sign-up completion hides the damage.

Sees the win-back masking churnRightRightRight
Gemini 3.5 Flash-Lite · Gemini

It identifies the jump in returning users from Week 6 and uses the 12% vs 41% retention to show the win-back campaign masks churn.

Gemini 3.8 Flash · API

The diagnosis uses the returning-user jump and the 12% vs 41% two-week retention to show the win-back campaign masks rising churn.

Opus 5.5 · Claude

It identifies the jump in returning users from week 6, uses the 12% vs 41% two-week retention to show low quality, and explains how this masks rising churn.

Understands before it justifiesRightRightRight
Gemini 3.5 Flash-Lite · Gemini

It tests the summer explanation against the data (stable older cohorts, drop tied to cohort 5) and reaches its own product-led diagnosis.

Gemini 3.8 Flash · API

The output tests the summer explanation against the data, finds a step-change rather than gradual decay, and reaches its own diagnosis.

Opus 5.5 · Claude

The diagnosis tests Leo's summer hypothesis against the data (cohorts in same calendar weeks, July churn, September churn) and reaches its own conclusion from the evidence.

Splits growth into new, lost and returningRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The diagnosis separates new, retained, churned, and returning users to show which flows changed.

Gemini 3.8 Flash · API

The diagnosis separates new, retained, churned, and returning users to show which flows changed.

Opus 5.5 · Claude

The diagnosis separates new, retained, churned, and returning users, showing which flows changed and how they offset each other.

Reads the shape of the curvesRightRightRight
Gemini 3.5 Flash-Lite · Gemini

It describes the retention curve change (Week 1 up, Week 4 down) and the low persistence of win-back returns, drawing correct conclusions.

Gemini 3.8 Flash · API

The output describes the cohort retention curves (flattening at 30% vs dropping to 20%) and correctly concludes a structural break, not seasonality.

Opus 5.5 · Claude

It reads the cohort curves as a clean step change at week 5 rather than a gradual seasonal dip, and notes the early retention rise followed by a steeper later drop.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstraChatGPT96.2100.02None
2GPT-6 LunaAPI96.2100.02None
3GPT-6.1 SolAPI94.1100.02None
4Haiku 5.5API92.0100.02None
5Opus 5.5Claude82.1100.02None
6Sonnet 5.5API90.189.32None
7Gemini 3.5 Flash-LiteGemini76.474.22None
8Gemini 3.8 FlashAPI72.370.92None

What you get for the cost

GPT-6 Luna · API matches GPT-6 Astra · ChatGPT at about a fiftieth of the cost.

Task score against relative cost for each setup. The table below has the same figures.607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 98, 79× the cheapestGPT-6 Luna · API: task score 98, the cheapestGPT-6.1 Sol · API: task score 97, 15× the cheapestHaiku 5.5 · API: task score 96, 3.6× the cheapestOpus 5.5 · Claude: task score 92, 77× the cheapestSonnet 5.5 · API: task score 88, 50× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 74, 3.0× the cheapestGemini 3.8 Flash · API: task score 71, 13× the cheapestGPT-6 Astra · ChatGPTGPT-6 Luna · APIGPT-6.1 Sol · APIHaiku 5.5 · APIOpus 5.5 · ClaudeSonnet 5.5 · APIGemini 3.5 Flash-Lite · GeminiGemini 3.8 Flash · API
Best value

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Working out why customers leave, and what to fix first.

Why it matters

Teams pick a story (the season, a competitor) and then pull data to justify it. Half the time the 'drop' is a definition change, and the real leak is somewhere else.

What good looks like

  • Starts from what the data shows, not a story
  • Splits growth into new, lost and returning users
  • Separates measurement and seasonality from churn
  • Sizes the real problem and fixes new-user retention first

Deliberately not measured

  • Dashboard design
Capability tested

Retention diagnosis

The failure we’re looking for

Confirming the story everyone already believes

Grading

Decision model and LLM judge, calibrated against a blind PM review