Humanity’s Last PRD
Latest drop: Haiku 5.5 · 8 Oct
Tasks / Growth

Diagnose a retention problem

Can the model work out from raw data what's really happening to retention, separating churn from measurement and seasonality, and find the cause?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 81% were usable with at most a quick edit.

Reliably right

  1. Produces the required deliverable100% pass
    The diagnosis is complete, actionable, and in the right format for Leo; a PM could act on it with light edits.
    GPT-6 Astra · ChatGPT · Flat, while sign-ups grow
  2. Understands before it justifies100% pass
    It tests the summer explanation against the data, finds the break aligns with the onboarding change not the paid spend increase, and reaches its own diagnosis from the evidence.
    GPT-6 Astra · ChatGPT · Flat, while sign-ups grow
  3. Splits growth into new, lost and returning100% pass
    It splits growth into new, retained, churned, and returning users, and uses the net growth equation to show which flow changed.
    GPT-6 Astra · ChatGPT · Flat, while sign-ups grow

Where it slips

  1. Gets the base of every number right61% pass
    The SMB adoption churn calculation is incorrect; 39% of ~$7.4k/month churn is ~$2.9k/month, not $34k MRR, and the annualized figure is ~$34.8k ARR, not $0.4M.
    Sonnet 5.5 · API · The collapse that wasn't
  2. Avoids unsupported claims67% pass
    The output presents causal interpretations (e.g., onboarding changes broke retention, low-intent users) as established fact without labelling them as hypotheses.
    Gemini 3.5 Flash-Lite · Gemini · Flat, while sign-ups grow
  3. Identifies material uncertainty77% pass
    The output does not name specific unknowns that could change the decision or say how they would be resolved.
    Gemini 3.5 Flash-Lite · Gemini · Flat, while sign-ups grow

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Understands before it justifies

    Does the diagnosis start from what the data shows (which users, when, at what point in their life with the product) before accepting or proposing an explanation, including the one it was handed?

    Passes when Tests the explanation it was given against the data, and reaches its own diagnosis from the evidence.

    Bangaly KabaLenny’s Podcast

  2. Splits growth into new, lost and returning

    Does the diagnosis separate new users or revenue from what was lost and what came back, rather than reading a single total?

    Passes when Uses growth accounting (new, retained, churned, returning, or new, expansion, contraction, churn) to show which flow changed.

    Naomi GleitLenny’s Podcast

  3. Reads the shape of the curves

    Does the diagnosis read how retention curves behave over time (flattening, still falling, or dipping and recovering) and what each shape means?

    Passes when Describes the curve shapes in the data and draws the right conclusion from each, such as a dip that recovers being seasonal.

    Product and growth leadersOn Lenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for activation at Stride. Leo, our Head of Growth, thinks weekly actives are flat because it's summer, and wants to double paid acquisition (his note is below). Before he does, write him a diagnosis of no more than 800 words: what's actually happening to retention, the evidence, and what we should do instead of (or before) spending more on acquisition. The files are attached.

What the model was given6 items: About Stride, leo_note.md, weekly_growth_accounting.csv (weeks run Monday to Sunday), weekly_signup_cohorts.csv (share of each week's sign-ups active in each following week), release_and_campaign_log.md, winback_campaign_results.md
About StrideA fitness habit app: short guided workouts and a weekly plan. Free, with Premium at $9.99 a month.
leo_note.md1 lines · Download
From Leo (Head of Growth): "Weekly actives have been flat for three months even though sign-ups are up 30%. It's summer: people are outdoors and on holiday. I want to double paid acquisition to push through it."
weekly_growth_accounting.csv (weeks run Monday to Sunday)week,week_starting,new_users,returning_users,retained_users,churned_users,weekly_active_users 1,2026-07-06,6034,973,44884,7116,51891 2,2026-07-13,6142,989,44529,7362,51660 3,2026-07-20,6184,1040,45074,6586,52298 4,2026-07-27,6385,991,44843,7455,52219 5,2026-08-03,6597,975,44085,8134,51657 6,2026-08-10,6689,2541,42516,9141,51746 7,2026-08-17,6779,2666,42506,9240,51951 8,2026-08-24,6955,2552,42723,9228,52230 9,2026-08-31,7156,2618,42134,10096,51908 10,2026-09-07,7230,2675,42393,9515,52298 11,2026-09-14,7564,2586,41984,10314,52134 12,2026-09-21,7550,2557,41808,10326,51915 13,2026-09-28,7847,2635,41636,10279,52118
weekly_signup_cohorts.csv (share of each week's sign-ups active in each following week)signup_week,signups,week_1_retained_pct,week_2_retained_pct,week_3_retained_pct,week_4_retained_pct,week_5_retained_pct,week_6_retained_pct,week_7_retained_pct,week_8_retained_pct 1,6000,55.4,43.2,36.4,34.2,31.9,30.9,29.8,30.1 2,6150,55.3,43.2,37.0,34.2,32.4,30.5,29.9,30.2 3,6300,55.2,43.0,37.7,33.4,32.1,30.6,29.8,30.0 4,6450,54.3,42.3,37.0,33.4,32.4,31.4,30.0,29.5 5,6600,58.4,39.4,29.6,24.5,22.1,21.1,19.8,19.9 6,6750,58.6,39.9,30.1,24.4,21.8,21.2,19.9, 7,6900,58.4,40.5,29.4,24.9,22.2,21.2,, 8,7050,58.1,40.1,29.9,25.0,21.3,,, 9,7200,57.5,40.5,29.7,25.0,,,, 10,7350,58.3,40.5,29.4,,,,, 11,7500,57.4,40.6,,,,,, 12,7650,57.5,,,,,,,
release_and_campaign_log.md6 lines · Download
# Release and campaign log

- Week 3: New workout library (40 new routines).
- Week 5: Shorter sign-up. Removed the goal-setting step ("What do you want to achieve, and how many days a week?") to cut sign-up time from 3 minutes to 1. Sign-up completion rose from 61% to 74%.
- Week 6: Win-back emails start: anyone inactive for 21 days gets "We miss you: here's a free week of Premium".
- Week 9: Paid social budget up 25%.
winback_campaign_results.md3 lines · Download
# Win-back campaign, weeks 6 to 13 (from the lifecycle team)

About 1,600 extra returning users a week compared with before the campaign. Of users who came back through a win-back email, 12% were still active two weeks later. Of users who came back on their own before the campaign, 41% were.
What a strong answer doesThe answer key the graders mark against

Shows that flat weekly actives aren't the summer: new users rose about 30% (about 6,000 to 7,800 a week) but churned users rose faster (about 7,100 to 10,300), and returning users jumped from about 1,000 to 2,600 a week only because of the win-back emails from week 6, of whom 12% stay two weeks against 41% of organic returners, so actives are being propped up. The cohort table shows why churn rose: cohorts from week 5, when the goal-setting step was removed from sign-up, retain far worse (week 4 about 25% against 34%, week 8 about 20% against 30%) even though more people finish sign-up. So more paid acquisition would pour more users into a leaky first month. Recommends restoring goal-setting (or testing it back against the short sign-up, judged on week-4 retention, not sign-up completion), measuring returning users separately so the campaign doesn't hide churn, and holding paid acquisition until new-user retention recovers.

Critical failures (cap the score at 40)
  • Recommends doubling paid acquisition now
Task

Mid-level · v1.0

The outputs

8 setups answered this brief.

GPT-6 AstraChatGPT100Usable with a quick editGPT-6 LunaAPI100Usable with a quick editGPT-6.1 SolAPI98Usable with a quick editSonnet 5.5API96Usable with a quick editHaiku 5.5API96Usable with a quick editOpus 5.5Claude92Usable with a quick editGemini 3.5 Flash-LiteGemini83Usable with a quick editGemini 3.8 FlashAPI83Usable with a quick edit
GPT-6 LunaAPI

Usable with a quick edit12 of 12 checks passed

The diagnosis is evidence-based, actionable, and correctly framed for the reader; a PM could use it with only minor wording edits.

Output

Leo — I wouldn’t double paid acquisition yet. The flat WAU is masking a retention problem, not showing that summer is the whole story.

  • The active base is eroding while inflows hide it. From week 1 to week 13, new users rose from 6,034 to 7,847 (+30%), but retained users fell from 44,884 to 41,636 (−7%). Weekly churn rose from 7,116 to 10,279. WAU stayed near 52,000 because more new users—and, from week 6, more returning users—offset that loss.
  • New-user retention deteriorates sharply after week 4. For the first four signup cohorts, week-4 retention was 33–34%, and week-8 retention about 30%. For cohorts in weeks 5–8, week-4 retention was 24–25%; week-5 retention was about 22%, versus about 32% for the earlier cohorts at the same age. Week-1 retention actually improved (roughly 54–55% to 58%), then users fell away faster. So sign-ups are easier to get, but fewer people build a lasting workout habit.
  • The timing points to a product change worth investigating. In week 5, we removed the goal-setting step. Sign-up completion rose from 61% to 74%, while the cohort retention pattern worsened immediately. That’s strong evidence to investigate whether the change brought in less-committed users or removed useful personalization—not proof that it caused the decline. Seasonality may contribute, but it doesn’t explain away the sharp cohort break.
  • Win-back lifts WAU more than durable retention. Returning users rose from about 1,000 to about 2,600 a week after the week-6 campaign. But only 12% of email-returners were active two weeks later, compared with 41% of people who returned on their own. The campaign may be adding short-lived activity, not repairing the underlying habit problem.

What I’d do before scaling spend:

  1. Test the onboarding change now. A/B test restoring goal-setting (or a shorter version) against the current flow. Measure signup completion alongside week-1 activation, week-4/8 retention, and workouts completed—not just registrations.
  2. Improve the first-week habit loop. Give users a clear, personalized weekly plan and a timely next-workout prompt. Track results by signup week, onboarding version, and acquisition channel.
  3. Put win-back on a holdout. Measure incremental, sustained activity and Premium conversion against users who don’t receive the offer; don’t count the initial return as success.
  4. Hold off on doubling acquisition. The week-9 paid increase came after the retention decline had already begun. Scale only when newer cohorts retain better and paid users show acceptable downstream value.

Fixing the leak first gives us a better chance of turning additional acquisition into durable actives rather than more short-lived sign-ups.

Grades and run details

Decision model 100 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly31%
  • passAddresses the actual decision98%
  • passRespects explicit constraints79%
  • passIdentifies material uncertainty83%
  • passAvoids unsupported claims64%
  • passProduces the required deliverable97%
  • passFinds the onboarding change100%
  • passSees the win-back masking churn100%
  • passUnderstands before it justifies100%
  • passGets the base of every number right39%
  • passSplits growth into new, lost and returning100%
  • passReads the shape of the curves73%
Run
Run
#1
API response time
17 s
Submitted
5 Oct 2026

Check by check

Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Got right 12

Uses the supplied evidence correctlyRight

All factual claims about the current situation are directly from the supplied data or correctly derived from it.

Addresses the actual decisionRight

The output clearly and early says not to double paid acquisition yet, and states the condition (newer cohorts retain better) that would change the call.

Respects explicit constraintsRight

The output is a diagnosis under 800 words, addressed to Leo, and respects the requested form and length.

Identifies material uncertaintyRight

It labels the onboarding change as strong evidence to investigate, not proof, acknowledges seasonality may contribute, and says how to resolve uncertainty via A/B tests and holdouts.

Avoids unsupported claimsRight

Hypotheses are clearly labelled as such (e.g., 'strong evidence to investigate', 'may be adding'), and no interpretation is presented as established fact.

Produces the required deliverableRight

The diagnosis is complete, actionable, within word limit, and written for Leo with clear next steps.

Finds the onboarding changeRight

It links the week-5 retention drop to the goal-setting removal, citing cohort numbers (week-4 24-25% vs 33-34%) and noting the sign-up completion increase hides the problem.

Sees the win-back masking churnRight

It identifies the jump in returning users from week 6 and uses the 12% vs 41% two-week retention to show the campaign inflates WAU without durable retention, masking churn.

Understands before it justifiesRight

The diagnosis tests Leo's summer hypothesis against the data, uses growth accounting and cohort curves, and reaches its own evidence-based conclusion.

Gets the base of every number rightRight

All percentages and differences are computed correctly from the supplied data, and no ambiguous base is left unstated.

Splits growth into new, lost and returningRight

The output explicitly splits the active base into new, retained, churned, and returning users to show which flow changed.

Reads the shape of the curvesRight

It describes the retention curve change: week-1 retention improved but later weeks fell faster, and earlier cohorts flattened after week 4 while later cohorts continued falling, drawing the right conclusion about habit formation.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstraChatGPT96.2100.02None
2GPT-6 LunaAPI96.2100.02None
3GPT-6.1 SolAPI94.1100.02None
4Haiku 5.5API92.0100.02None
5Opus 5.5Claude82.1100.02None
6Sonnet 5.5API90.189.32None
7Gemini 3.5 Flash-LiteGemini76.474.22None
8Gemini 3.8 FlashAPI72.370.92None

What you get for the cost

GPT-6 Luna · API matches GPT-6 Astra · ChatGPT at about a fiftieth of the cost.

Task score against relative cost for each setup. The table below has the same figures.607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 98, 79× the cheapestGPT-6 Luna · API: task score 98, the cheapestGPT-6.1 Sol · API: task score 97, 15× the cheapestHaiku 5.5 · API: task score 96, 3.6× the cheapestOpus 5.5 · Claude: task score 92, 77× the cheapestSonnet 5.5 · API: task score 88, 50× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 74, 3.0× the cheapestGemini 3.8 Flash · API: task score 71, 13× the cheapestGPT-6 Astra · ChatGPTGPT-6 Luna · APIGPT-6.1 Sol · APIHaiku 5.5 · APIOpus 5.5 · ClaudeSonnet 5.5 · APIGemini 3.5 Flash-Lite · GeminiGemini 3.8 Flash · API
Best value

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Working out why customers leave, and what to fix first.

Why it matters

Teams pick a story (the season, a competitor) and then pull data to justify it. Half the time the 'drop' is a definition change, and the real leak is somewhere else.

What good looks like

  • Starts from what the data shows, not a story
  • Splits growth into new, lost and returning users
  • Separates measurement and seasonality from churn
  • Sizes the real problem and fixes new-user retention first

Deliberately not measured

  • Dashboard design
Capability tested

Retention diagnosis

The failure we’re looking for

Confirming the story everyone already believes

Grading

Decision model and LLM judge, calibrated against a blind PM review