Tasks / Design

Activation & onboarding review

Can the model find the friction that matters most and prioritise the fixes?

Measures the modelTask v1.1 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 14 graded outputs by 7 models. 50% were usable with at most a quick edit.

Reliably right

Activation defined by what predicts retention100% pass
Defines activation as three workouts linked to day-60 retention, and evaluates fixes by effect on that event.
GPT-6 Astra · ChatGPT · Fitness app first week
Prioritises by impact98% pass
Ranks changes by likely impact on achieving three workouts, aligned with the identified activation threshold.
GPT-6 Astra · ChatGPT · Fitness app first week
Produces the required deliverable93% pass
Provides a complete, actionable memo that a growth PM could implement without major gaps.
GPT-6 Astra · ChatGPT · Fitness app first week

Where it slips

Avoids unsupported claims54% pass
It presents causal or evaluative claims such as 'successful connection is the gateway' and 'a required technical connection is an especially poor first step' as established fact rather than hypothesis.
GPT-6 Luna · API · Analytics tool losing users at setup
Uses the supplied evidence correctly54% pass
It includes an unsupported motivation claim and a miscomputed '4% of trial starters' figure, so not every current-situation statement is supported by the brief or arithmetic.
Sonnet 5.5 · API · Fitness app first week
Identifies material uncertainty59% pass
Identifies the lack of step-level data but does not state what result would change the call or how it would affect the ranking.
Opus 5.5 · Claude · Analytics tool losing users at setup

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

Review the first-week experience of our fitness app, described below, and recommend the changes you would make, in priority order, as a memo to the growth PM. Keep it under 500 words.

What the model was given4 items: Scenario, First week, step by step, Funnel, Team plan

ScenarioStride is a home-workout app. It costs £9.99 a month after a seven-day free trial, and the trial needs a card up front.

First week, step by stepDay 0: nine onboarding questions (goal, age, height, weight, fitness level, equipment, injuries, preferred days, preferred time), then the paywall to start the trial, then a plan with the first workout scheduled for the next morning. Every workout is 30–45 minutes. Day 1: a 7am push notification, “Your workout is ready.” Days 2–7: one push a day at the preferred time. Missing a workout moves the plan on to the next one; there is no shorter or catch-up option. The streak counter first appears after seven days.

FunnelInstall → finished onboarding 71% → started trial 52% → first workout 48% → three workouts in week one 19%. Users who complete three workouts in week one are four times as likely to still be subscribed at day 60.

Team planThe design team wants to spend this quarter redesigning the onboarding questionnaire and the paywall screen.

What a strong answer doesThe answer key the graders mark against

Focus on getting new users to three workouts in week one (shorter first workouts, a same-day first session, catch-up options, earlier progress feedback) rather than onboarding and paywall polish.

Case

Warm-up · v1.8 · synthetic · consumer, mobile

The outputs

7 setups answered this brief. Pick up to three to read side by side.

GPT-6 AstraChatGPT100Usable with a quick edit GPT-6 LunaAPI100Usable with a quick edit GPT-6.1 SolAPI100Usable with a quick edit Sonnet 5.5API63Needs more work Opus 5.5Claude60Needs more work Gemini 3.8 FlashAPI60Needs more work Gemini 3.5 Flash-LiteGemini55Needs more work

Gemini 3.8 FlashAPI

Needs more than a quick edit6 of 9 checks passed

A PM would need to re-check and soften unsupported causal claims before acting. The core recommendation is usable, but the evidence framing is not safe as written.

Output

MEMO

TO: Growth PM FROM: Product & Growth Review SUBJECT: Week 1 Retention & Experience Optimization

Executive Summary

Push back on the design team’s plan to redesign onboarding and the paywall. Converting 71% through onboarding and 52% into a card-upfront trial is exceptionally strong.

Our growth bottleneck is Week 1 activation: while 48% log a first workout, only 19% reach three workouts—the critical threshold that drives 4x Day-60 retention. We are losing trialists during their first week because of delayed gratification, rigid scheduling, and high initial friction.

Here are the four prioritized changes to drive 3-workout completion.

---

Recommended Changes (Priority Order)

#### 1. Instant Gratification on Day 0 (“Start Now” Option) * Problem: Scheduling the first session for the next morning wastes peak user motivation right after purchase. * Fix: Offer an immediate, optional 5–10 minute "Quick Start / Assessment" on Day 0. Completing this counts as Workout #1, immediately advancing users toward the 3-workout milestone.

#### 2. Introduce "Micro-Workouts" & Flexible Session Lengths * Problem: A rigid 30–45 minute commitment is daunting for home workouts, leading to missed days. When a day is missed, the plan simply skips ahead without recourse, inducing guilt and churn. * Fix: Introduce 10- and 20-minute “time-crunched” alternatives when a user opens a session or is at risk of missing one. A 10-minute workout must count toward the 3-workout goal. Never skip a workout automatically; let users swap, reschedule, or condense.

#### 3. Surface Streaks & Milestones on Day 1 (Not Day 7) * Problem: Hiding the streak counter until Day 7 removes the most effective psychological hook during the make-or-break trial period. * Fix: * Show the streak counter immediately upon completing Workout #1. * Explicitly gamify the activation metric: show a progress bar toward the "Week 1 Foundation Badge" (1/3, 2/3, 3/3 workouts completed).

#### 4. Action-Oriented, Personalized Push Notifications * Problem: A generic 7:00 AM push ("Your workout is ready") is transactional and easily dismissed. * Fix: Leverage the rich onboarding data (goals, fitness level) to trigger personalized, high-context pushes: * Copy: “Crush your [Goal] today: 15 mins to hit 2/3 of your Week 1 target.” * Timing: Align strictly to their chosen time, with an evening "last chance" nudge offering an express 10-minute session if uncompleted.

---

Next Steps

Pause the onboarding/paywall redesign. Run a two-week sprint to implement immediate Day 0 activation, 15-minute workout variants, and Day 1 streak mechanics. Our target metric is moving Week 1 (3+ workouts) from 19% to 30%.

Claims the judge couldn’t find in the brief

The first week has high initial friction.

Grades and run details

Decision model 67 · LLM judge 6 of 10 checks

Decision model checks

failUses the supplied evidence correctly40%
passAddresses the actual decision98%
passRespects explicit constraints54%
failIdentifies material uncertainty88%
failAvoids unsupported claims33%
passProduces the required deliverable98%
passPrioritises by impact91%
passActivation defined by what predicts retention100%
passGets the base of every number right93%

Run

Run: #1
API response time: 15 s
Submitted: 1 Oct 2026

Check by check

Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Got wrong 3

Uses the supplied evidence correctlyWrong

It states unsupported current-situation claims such as high initial friction, missed days, guilt, churn, and a psychological hook as facts rather than labelled hypotheses.

Identifies material uncertaintyWrong

It does not name material unknowns or conditions that would change the call, such as whether shorter workouts preserve habit formation or whether the 4x relationship is causal.

Avoids unsupported claimsWrong

It presents several causes and psychological effects as established facts when the supplied evidence only supports correlations or funnel drop-offs.

Got right 6

Addresses the actual decisionRight

It clearly commits to prioritising week-one activation over onboarding/paywall redesign and gives a ranked set of changes for the growth PM.

Respects explicit constraintsRight

It is a memo to the growth PM, under 500 words, and recommends changes in priority order.

Produces the required deliverableRight

The requested memo is present, complete, addressed to the growth PM, and actionable within the length limit.

Prioritises by impactRight

The fixes are ranked in a plausible impact order centred on reaching three workouts in week one.

Activation defined by what predicts retentionRight

It defines activation as three workouts in week one, links that to day-60 retention, and ranks fixes by their effect on that activation event.

Gets the base of every number rightRight

The percentages and 4x relationship are used against the correct funnel steps and denominators.

Results

Every setup we’ve tested on this task, across all cases and repeats, graded on the current checklist. Calibrated: the graders match our PM on 82% of checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	GPT-6.1 SolwithAPI	100.0	100.0	2	None
2	GPT-6 AstrawithChatGPT	94.4	100.0	2	None
3	GPT-6 LunawithAPI	94.4	85.0	2	None
4	Opus 5.5withClaude	80.6	65.0	2	None
5	Gemini 3.5 Flash-LitewithGemini	69.4	55.0	2	None
6	Sonnet 5.5withAPI	77.8	45.0	2	None
7	Gemini 3.8 FlashwithAPI	69.4	50.0	2	None

The PM job

Reviewing a signup and onboarding flow that is losing users.

Why it matters

Anyone can list fifty UX nits. The job is finding the two that explain the drop-off, backed by the funnel data supplied.

What good looks like

Ties each issue to the funnel data
Prioritises by likely impact
Defines activation as the behaviour that predicts retention, not finishing onboarding

Deliberately not measured

Accessibility audit completeness
Visual redesign

Capability tested

Consequential critique

The failure we’re looking for

A generic UX checklist

Grading

Decision model and LLM judge, calibrated against a blind PM review