Humanity’s Last PRD
Latest drop: Gemini 3.8 Flash · 1 Oct
Tasks / Design

Activation & onboarding review

Can the model find the friction that matters most and prioritise the fixes?

Measures the modelTask v1.1 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 14 graded outputs by 7 models. 50% were usable with at most a quick edit.

Reliably right

  1. Activation defined by what predicts retention100% pass
    Defines activation as three workouts linked to day-60 retention, and evaluates fixes by effect on that event.
    GPT-6 Astra · ChatGPT · Fitness app first week
  2. Prioritises by impact98% pass
    Ranks changes by likely impact on achieving three workouts, aligned with the identified activation threshold.
    GPT-6 Astra · ChatGPT · Fitness app first week
  3. Produces the required deliverable93% pass
    Provides a complete, actionable memo that a growth PM could implement without major gaps.
    GPT-6 Astra · ChatGPT · Fitness app first week

Where it slips

  1. Avoids unsupported claims54% pass
    It presents causal or evaluative claims such as 'successful connection is the gateway' and 'a required technical connection is an especially poor first step' as established fact rather than hypothesis.
    GPT-6 Luna · API · Analytics tool losing users at setup
  2. Uses the supplied evidence correctly54% pass
    It includes an unsupported motivation claim and a miscomputed '4% of trial starters' figure, so not every current-situation statement is supported by the brief or arithmetic.
    Sonnet 5.5 · API · Fitness app first week
  3. Identifies material uncertainty59% pass
    Identifies the lack of step-level data but does not state what result would change the call or how it would affect the ranking.
    Opus 5.5 · Claude · Analytics tool losing users at setup

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

Review the first-week experience of our fitness app, described below, and recommend the changes you would make, in priority order, as a memo to the growth PM. Keep it under 500 words.

What the model was given4 items: Scenario, First week, step by step, Funnel, Team plan
ScenarioStride is a home-workout app. It costs £9.99 a month after a seven-day free trial, and the trial needs a card up front.
First week, step by stepDay 0: nine onboarding questions (goal, age, height, weight, fitness level, equipment, injuries, preferred days, preferred time), then the paywall to start the trial, then a plan with the first workout scheduled for the next morning. Every workout is 30–45 minutes. Day 1: a 7am push notification, “Your workout is ready.” Days 2–7: one push a day at the preferred time. Missing a workout moves the plan on to the next one; there is no shorter or catch-up option. The streak counter first appears after seven days.
FunnelInstall → finished onboarding 71% → started trial 52% → first workout 48% → three workouts in week one 19%. Users who complete three workouts in week one are four times as likely to still be subscribed at day 60.
Team planThe design team wants to spend this quarter redesigning the onboarding questionnaire and the paywall screen.
What a strong answer doesThe answer key the graders mark against

Focus on getting new users to three workouts in week one (shorter first workouts, a same-day first session, catch-up options, earlier progress feedback) rather than onboarding and paywall polish.

Case

Warm-up · v1.8 · synthetic · consumer, mobile

The outputs

7 setups answered this brief. Pick up to three to read side by side.

GPT-6 AstraChatGPT100Usable with a quick editGPT-6 LunaAPI100Usable with a quick editGPT-6.1 SolAPI100Usable with a quick editSonnet 5.5API63Needs more workOpus 5.5Claude60Needs more workGemini 3.8 FlashAPI60Needs more workGemini 3.5 Flash-LiteGemini55Needs more work
Gemini 3.8 FlashAPI

Needs more than a quick edit6 of 9 checks passed

A PM would need to re-check and soften unsupported causal claims before acting. The core recommendation is usable, but the evidence framing is not safe as written.

Output

MEMO

TO: Growth PM FROM: Product & Growth Review SUBJECT: Week 1 Retention & Experience Optimization

Executive Summary

Push back on the design team’s plan to redesign onboarding and the paywall. Converting 71% through onboarding and 52% into a card-upfront trial is exceptionally strong.

Our growth bottleneck is Week 1 activation: while 48% log a first workout, only 19% reach three workouts—the critical threshold that drives 4x Day-60 retention. We are losing trialists during their first week because of delayed gratification, rigid scheduling, and high initial friction.

Here are the four prioritized changes to drive 3-workout completion.

---

Recommended Changes (Priority Order)

#### 1. Instant Gratification on Day 0 (“Start Now” Option) * Problem: Scheduling the first session for the next morning wastes peak user motivation right after purchase. * Fix: Offer an immediate, optional 5–10 minute "Quick Start / Assessment" on Day 0. Completing this counts as Workout #1, immediately advancing users toward the 3-workout milestone.

#### 2. Introduce "Micro-Workouts" & Flexible Session Lengths * Problem: A rigid 30–45 minute commitment is daunting for home workouts, leading to missed days. When a day is missed, the plan simply skips ahead without recourse, inducing guilt and churn. * Fix: Introduce 10- and 20-minute “time-crunched” alternatives when a user opens a session or is at risk of missing one. A 10-minute workout must count toward the 3-workout goal. Never skip a workout automatically; let users swap, reschedule, or condense.

#### 3. Surface Streaks & Milestones on Day 1 (Not Day 7) * Problem: Hiding the streak counter until Day 7 removes the most effective psychological hook during the make-or-break trial period. * Fix: * Show the streak counter immediately upon completing Workout #1. * Explicitly gamify the activation metric: show a progress bar toward the "Week 1 Foundation Badge" (1/3, 2/3, 3/3 workouts completed).

#### 4. Action-Oriented, Personalized Push Notifications * Problem: A generic 7:00 AM push ("Your workout is ready") is transactional and easily dismissed. * Fix: Leverage the rich onboarding data (goals, fitness level) to trigger personalized, high-context pushes: * Copy: “Crush your [Goal] today: 15 mins to hit 2/3 of your Week 1 target.” * Timing: Align strictly to their chosen time, with an evening "last chance" nudge offering an express 10-minute session if uncompleted.

---

Next Steps

Pause the onboarding/paywall redesign. Run a two-week sprint to implement immediate Day 0 activation, 15-minute workout variants, and Day 1 streak mechanics. Our target metric is moving Week 1 (3+ workouts) from 19% to 30%.

Claims the judge couldn’t find in the brief

  • The first week has high initial friction.

Grades and run details

Decision model 67 · LLM judge 6 of 10 checks
Decision model checks
  • failUses the supplied evidence correctly40%
  • passAddresses the actual decision98%
  • passRespects explicit constraints54%
  • failIdentifies material uncertainty88%
  • failAvoids unsupported claims33%
  • passProduces the required deliverable98%
  • passPrioritises by impact91%
  • passActivation defined by what predicts retention100%
  • passGets the base of every number right93%
Run
Run
#1
API response time
15 s
Submitted
1 Oct 2026
Sonnet 5.5API

Needs more than a quick edit6 of 9 checks passed

It contains a miscomputed funnel rate and an unsupported causal claim, so an editor would need to re-check the evidence before using it.

Output

To: Growth PM Re: First-week experience: recommended changes, in priority order

Summary: The design team's plan targets the wrong part of the funnel. The biggest leak that matters is after the trial starts. Only 40% of users who do a first workout go on to do three (48% → 19%), and three workouts is the behavior linked to 4x day-60 retention. Onboarding and the paywall do lose people, but the week-one experience is where I'd invest first.

Priorities

  1. Let users work out on Day 0. Today the first workout is the next morning, so motivation peaks at the paywall and then goes unused. After trial start, offer "Start your first workout now" with a 10–15 minute option. Also, 4% of trial starters (52% → 48%) never do a first workout, and this fixes that too.
  1. Add short and catch-up workouts. Every session is 30–45 minutes, and a missed one is simply skipped. That punishes exactly the users at risk. Add 10–15 minute versions and a "catch up" option after a miss, so the plan flexes instead of moving on. This is likely the biggest lever on the 48% → 19% drop.
  1. Fix the notifications. Day 1 goes out at 7am, not the time users told us they prefer, which looks like a bug and wastes the most important nudge. Beyond fixing that, make the messages specific ("15 minutes, no equipment, ready when you are"). Add a gentle recovery message after a missed day instead of continuing the normal schedule.
  1. Show the streak from Day 1. It currently appears after seven days, when the habit has already formed or failed. Show it after the first workout. Make it forgiving (e.g. weekly goals or a freeze) so one missed day doesn't feel like failure.
  1. Trim onboarding, don't redesign it. Nine questions likely contribute to the 29% who drop before finishing. Keep what personalization needs (goal, level, equipment, injuries, preferred days and time) and defer height, weight and age, or make them optional. This is a small change and can ship alongside the items above.
  1. Test the paywall last. Only 27% of onboarded users drop there, and a card-up-front trial is already a strong filter. Once the trial experience is better, we can test the paywall with better evidence. Consider a trial-end reminder on day 5 to build trust.

Suggested redirect for design: Rather than redesigning the questionnaire and paywall, spend the quarter on the Day 0–7 experience: the immediate first workout, short sessions, catch-up flow, and streak.

Caveats and measurement - The 4x retention figure is correlational. Motivated users may simply do more workouts. Run these changes as A/B tests with day-60 retention as the outcome, not just workout counts. - Primary metric: share of trial starters completing three workouts in week one (currently about 37%, or 19/52). - Guardrails: trial start rate, and whether shorter workouts reduce perceived value or conversion to paid.

Sequencing: items 1, 3 and 5 are quick to ship, and items 2 and 4 need more design and content work. I'd begin experiments on 1 and 3 within the first few weeks.

Claims the judge couldn’t find in the brief

  • 4% of trial starters never do a first workout.
  • Motivation peaks at the paywall and then goes unused by the next morning.

Grades and run details

Decision model 72 · LLM judge 6 of 10 checks
Decision model checks
  • failUses the supplied evidence correctly21%
  • passAddresses the actual decision100%
  • passRespects explicit constraints28%
  • passIdentifies material uncertainty76%
  • partialAvoids unsupported claims56%
  • passProduces the required deliverable94%
  • passPrioritises by impact94%
  • passActivation defined by what predicts retention100%
  • failGets the base of every number right75%
Run
Run
#1
API response time
19 s
Submitted
29 Sept 2026
Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit5 of 9 checks passed

Unsupported claim about user sentiment and missing conditions for the recommendation mean the audience would need to re-check evidence and add decision criteria before using it.

Output

MEMORANDUM

To: Growth Product Manager From: Growth & Product Strategy Date: September 25, 2026 Subject: First-Week Experience Review & Recommendations

---

Executive Summary

Our funnel reveals a critical bottleneck: while 52% start the trial, only 19% complete three workouts in week one1—our primary retention driver (users hitting this milestone are 4X more likely to subscribe at day 60). Furthermore, the design team's current focus on redesigning onboarding and the paywall is misaligned; onboarding completion is already healthy at 71%2, and trial conversion sits at 52%. Instead, optimization efforts must shift to activation, early habit formation, and flexibility.

Below are our recommended changes, ordered by priority.

---

Priority Recommendations

#### 1. Introduce Flexible "Catch-Up" & Micro-Workouts (High Impact) * The Problem: Currently, missing a scheduled 30–45 minute workout simply shuffles the plan forward without offering alternative options, leaving users overwhelmed and leading to immediate drop-off4 before reaching the vital three-workout milestone. * The Fix: Implement a "Catch-Up" mode offering shorter 10–15 minute micro-workouts or scaled options when a user misses a session. This lowers the barrier to entry and protects the Day 7 retention streak.

#### 2. Shift the Streak Counter and Early Milestone Rewards to Day 1 (High Impact) * The Problem: The streak counter currently remains hidden until Day 7. Users miss out on crucial early psychological reinforcement during their most vulnerable window. * The Fix: Surface the streak counter immediately on Day 1 upon completing the first workout. Introduce immediate celebratory micro-milestones (e.g., badges or progress indicators) after the very first completed session to build early momentum.

#### 3. Personalize and Optimize Push Notification Triggers (Medium Impact) * The Problem: Rigid 7 AM notifications or static reminders often feel intrusive or misaligned with user behavior3, contributing to friction during the trial week. * The Fix: Dynamically adapt push notifications based on user behavior. If a user misses their preferred time, send a gentle, encouraging nudge offering a shorter alternative workout rather than a rigid command.

#### 4. Deprioritize Onboarding and Paywall Redesigns (Strategic Pivot) * The Problem: The design team’s current plan to spend the quarter redesigning a 71% completion onboarding flow and a 52% trial-conversion paywall addresses healthy metrics. * The Fix: Reallocate design and engineering resources away from onboarding/paywalls and toward activation features (such as the micro-workout player, catch-up flows, and early streak tracking).

What a PM had to fix

  1. 1Numbers wrongRedo the arithmeticStart again

    Source: Funnelstarted trial 52% → first workout 48% → three workouts in week one 19%

    What we’d changeRead each step against the one before: 92% of trial starters do a first workout, and 40% of those reach three. That second drop is the leak. Add a test plan and success metrics.

  2. 2Invented evidenceVerify or remove the claimSubstantial rework

    What we’d changeRemove 'healthy': nothing in the brief benchmarks it. And note that a quarter of finishers still drop at the paywall.

  3. 3Invented evidenceVerify or remove the claimQuick edit

    Source: First week, step by stepDays 2–7: one push a day at the preferred time.

    What we’d changeOnly the day-1 push is at 7am. The rest already go at the user's preferred time. The fix is the day-1 push, not all reminders.

  4. 4Hypothesis stated as factReframe it as a hypothesisQuick edit

    What we’d changePresent it as a likely reason to test. The brief shows the drop-off, not how users felt.

Claims the judge couldn’t find in the brief

  • Rigid 7 AM notifications or static reminders often feel intrusive or misaligned with user behavior, contributing to friction during the trial week

Grades and run details

Decision model 67 · LLM judge 5 of 10 checks
Decision model checks
  • failUses the supplied evidence correctly47%
  • passAddresses the actual decision93%
  • passRespects explicit constraints45%
  • failIdentifies material uncertainty99%
  • partialAvoids unsupported claims31%
  • passProduces the required deliverable81%
  • partialPrioritises by impact38%
  • passActivation defined by what predicts retention100%
  • passGets the base of every number right96%
Run
Run
#1
Time to output
6 s
Submitted
25 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 3

Addresses the actual decisionRightRightMixed
Gemini 3.8 Flash · API

It clearly commits to prioritising week-one activation over onboarding/paywall redesign and gives a ranked set of changes for the growth PM.

Sonnet 5.5 · API

It commits early to a clear ranked list for the Growth PM and notes A/B tests and guardrails that would change the call.

Gemini 3.5 Flash-Lite · Gemini

Does not indicate what result or condition would change the recommended priority or actions.

Identifies material uncertaintyWrongRightWrong
Gemini 3.8 Flash · API

It does not name material unknowns or conditions that would change the call, such as whether shorter workouts preserve habit formation or whether the 4x relationship is causal.

Sonnet 5.5 · API

It names the causal uncertainty in the 4x retention figure and the risk that shorter workouts reduce perceived value, and says how to resolve them with A/B tests and guardrails.

Gemini 3.5 Flash-Lite · Gemini

Does not name any unknowns that could change the decision or say how they would be resolved.

Gets the base of every number rightRightWrongRight
Gemini 3.8 Flash · API

The percentages and 4x relationship are used against the correct funnel steps and denominators.

Sonnet 5.5 · API

The 52% to 48% drop is misstated as 4% of trial starters; the correct share of trial starters is about 7.7%, or 4 percentage points of installs.

Gemini 3.5 Flash-Lite · Gemini

All numbers are directly from the supplied funnel; no derived figures require base checks.

All got wrong 2

Uses the supplied evidence correctlyWrongWrongWrong
Gemini 3.8 Flash · API

It states unsupported current-situation claims such as high initial friction, missed days, guilt, churn, and a psychological hook as facts rather than labelled hypotheses.

Sonnet 5.5 · API

It includes an unsupported motivation claim and a miscomputed '4% of trial starters' figure, so not every current-situation statement is supported by the brief or arithmetic.

Gemini 3.5 Flash-Lite · Gemini

States that notifications often feel intrusive or misaligned, which is not in the supplied context.

Avoids unsupported claimsWrongWrongWrong
Gemini 3.8 Flash · API

It presents several causes and psychological effects as established facts when the supplied evidence only supports correlations or funnel drop-offs.

Sonnet 5.5 · API

The motivation-peak and punishment interpretations are presented as fact rather than as hypotheses the evidence only suggests.

Gemini 3.5 Flash-Lite · Gemini

Presents the claim that notifications feel intrusive as established fact without labelling it as a hypothesis.

All got right 4

Respects explicit constraintsRightRightRight
Gemini 3.8 Flash · API

It is a memo to the growth PM, under 500 words, and recommends changes in priority order.

Sonnet 5.5 · API

It is a memo to the Growth PM, prioritized, and under the 500-word limit.

Gemini 3.5 Flash-Lite · Gemini

Delivered as a memo to the growth PM under 500 words.

Produces the required deliverableRightRightRight
Gemini 3.8 Flash · API

The requested memo is present, complete, addressed to the growth PM, and actionable within the length limit.

Sonnet 5.5 · API

The requested memo is present, complete, actionable, and within the requested length.

Gemini 3.5 Flash-Lite · Gemini

Memo is present, complete, within length, and gives actionable prioritized recommendations.

Prioritises by impactRightRightRight
Gemini 3.8 Flash · API

The fixes are ranked in a plausible impact order centred on reaching three workouts in week one.

Sonnet 5.5 · API

Fixes are ranked by likely impact on reaching three workouts and retention.

Gemini 3.5 Flash-Lite · Gemini

Fixes are ranked by priority with impact levels (high/medium), targeting activation.

Activation defined by what predicts retentionRightRightRight
Gemini 3.8 Flash · API

It defines activation as three workouts in week one, links that to day-60 retention, and ranks fixes by their effect on that activation event.

Sonnet 5.5 · API

It defines activation as three workouts in week one and judges fixes by their effect on that retention-predictive behavior.

Gemini 3.5 Flash-Lite · Gemini

Activation is defined as three workouts, linked to retention (4x), and ranking is based on effect on this event.

Results

Every setup we’ve tested on this task, across all cases and repeats, graded on the current checklist. Calibrated: the graders match our PM on 82% of checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6.1 SolAPI100.0100.02None
2GPT-6 AstraChatGPT94.4100.02None
3GPT-6 LunaAPI94.485.02None
4Opus 5.5Claude80.665.02None
5Gemini 3.5 Flash-LiteGemini69.455.02None
6Sonnet 5.5API77.845.02None
7Gemini 3.8 FlashAPI69.450.02None

The PM job

Reviewing a signup and onboarding flow that is losing users.

Why it matters

Anyone can list fifty UX nits. The job is finding the two that explain the drop-off, backed by the funnel data supplied.

What good looks like

  • Ties each issue to the funnel data
  • Prioritises by likely impact
  • Defines activation as the behaviour that predicts retention, not finishing onboarding

Deliberately not measured

  • Accessibility audit completeness
  • Visual redesign
Capability tested

Consequential critique

The failure we’re looking for

A generic UX checklist

Grading

Decision model and LLM judge, calibrated against a blind PM review