Tasks / Growth

Activation & onboarding review

Can the model find the friction that matters most and prioritise the fixes?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 69% were usable with at most a quick edit.

Reliably right

Respects explicit constraints100% pass
The output is a review addressed to Omar, well under 800 words, and respects the requested form.
GPT-6 Luna · API · The checklist isn't the moment
Produces the required deliverable100% pass
The review covers activation definition, evidence, changes, design proposal, and goal, is addressed to Omar, within length, and is actionable.
GPT-6 Luna · API · The checklist isn't the moment
Activation defined by what predicts retention100% pass
Activation is defined by the behaviour that predicts retention (bank+cat 10+), and fixes are judged by their effect on that event.
GPT-6 Luna · API · The checklist isn't the moment

Where it slips

Uses the supplied evidence correctly44% pass
The output states 'we have no step-level data for screens 2–8', 'a blank builder is hard for non-analysts', and 'people invite colleagues to something useful, not to a setup screen' as facts, none of which appear in the brief or follow from arithmetic.
Opus 5.5 · Claude · Analytics tool losing users at setup
Avoids unsupported claims55% pass
It presents 'a blank builder is hard for non-analysts' and 'people invite colleagues to something useful, not to a setup screen' as established facts without labelling them as hypotheses.
Opus 5.5 · Claude · Analytics tool losing users at setup
Addresses the actual decision56% pass
The output commits to a ranked order but does not state what result or condition would change that order, as required.
GPT-6 Astra · ChatGPT · Analytics tool losing users at setup

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Activation defined by what predicts retention
Does the review define (or question) the activation event by whether it predicts retention, and judge fixes by their effect on that event rather than on step completion?
Passes when Names the activation behaviour and its link to retention (from the supplied data, or as an explicit assumption to verify), and ranks fixes by their effect on it.
Bangaly KabaLenny’s Podcast Lauryn IsfordLenny’s Podcast
Gets people to value sooner
Do the recommended fixes shorten the path to the first moment of value, by removing or deferring steps before it or starting people with templates or sample data, rather than adding tours, tooltips or extra steps?
Passes when The main fixes cut or defer steps before first value, or give a warm start, and say which steps go.
Hila QuLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for onboarding at Tidepool. Omar, our Head of Product, has asked for a review of onboarding (his note is below). Write it in no more than 800 words: what activation should mean for us, the evidence, what to change in onboarding, and what to make of the design team's proposal and next quarter's goal. The files are attached.

What the model was given4 items: About Tidepool, omar_note.md, onboarding_today.md, first_week_behaviour.csv (August sign-ups: how many who did each thing in their first week were still active at day 60)

About TidepoolA personal finance app: connect your bank, see where your money goes, and budget.

omar_note.md1 lines · Download

From Omar (Head of Product): "Activation is flat at 61%. Review our onboarding and tell me what to fix. The design team has proposed a celebration screen and a progress bar for the checklist."

onboarding_today.md8 lines · Download

# Onboarding today (for 4,000 sign-ups in August)

1. Sign up (100%)
2. Setup checklist: name your account, pick a theme, set a savings goal, enable notifications (61% complete it)
3. Connect a bank (50%)
4. Categorise transactions (34% reach 10)

…

first_week_behaviour.csv (August sign-ups: how many who did each thing in their first week were still active at day 60)first_week_behaviour,retained_at_day_60,churned_by_day_60 All sign-ups,1120,2880 Completed the setup checklist,742,1696 Connected a bank account,964,1036 Categorised 10+ transactions,891,489 Connected a bank AND categorised 10+ transactions,842,238 Created a budget,410,820 Opened the app 4+ days,870,1310

What a strong answer doesThe answer key the graders mark against

Works out retention for each first-week behaviour from the file: all sign-ups 28% (1,120 of 4,000); completed the checklist about 30% (barely better than average); connected a bank about 48%; categorised 10+ transactions about 65%; connected a bank and categorised 10+ about 78%, and that group is about three-quarters of everyone retained (842 of 1,120). So the checklist isn't activation and raising its completion to 75% would do little; activation should be connecting a bank and categorising ten transactions in week one. Recommends cutting or deferring checklist steps that sit before that (theme, notifications, savings goal) and getting people to connect a bank and categorise sooner (for example suggested categories to confirm), and resets the goal to the new activation rate (about 27% today, 1,080 of 4,000). Treats the celebration screen and progress bar as polishing the wrong step. Notes that the data shows correlation and proposes a test to confirm that driving the behaviour improves retention.

Critical failures (cap the score at 40)

Recommends raising checklist completion as the main goal

Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

GPT-6 LunaAPI100Usable with a quick edit GPT-6 AstraChatGPT92Usable with a quick edit GPT-6.1 SolAPI92Usable with a quick edit Sonnet 5.5API92Usable with a quick edit Haiku 5.5API88Usable with a quick edit Opus 5.5Claude75Needs more work Gemini 3.8 FlashAPI60Needs more work Gemini 3.5 Flash-LiteGemini40Needs more work

GPT-6 LunaAPI

Usable with a quick edit11 of 11 checks passed

The output is clear, evidence-based, and actionable; a PM could use it with minor edits without rechecking evidence or redoing reasoning.

Output

To Omar — onboarding review

What activation should mean

Activation should mean a new user has reached Tidepool’s core value: connected a bank account and categorized at least 10 transactions within their first seven days. Completing setup tasks is not the same as understanding where your money goes.

What the evidence says

Of 4,000 August sign-ups, 61% completed the checklist, 50% connected a bank, and 34% categorized 10+ transactions. The behavioral data points to which events are most associated with retention at day 60:

All sign-ups: 28% retained (1,120/4,000)
Completed the checklist: 30% (742/2,438)
Connected a bank: 48% (964/2,000)
Categorized 10+ transactions: 65% (891/1,380)
Connected a bank and categorized 10+: 78% (842/1,080)

The combined behavior is the strongest signal and represents 27% of all sign-ups today. By contrast, checklist completion is barely above the overall retention rate. Creating a budget is associated with 33% retention, so it is not a better activation milestone on this evidence.

These are associations, not proof that onboarding changes will cause retention to rise: users who connect accounts or categorize transactions may already be more motivated. We should treat the numbers as a strong direction for what value to help users reach, then test changes and track outcomes.

What to change

Move bank connection earlier. Defer account naming, theme choice, savings goals, and notifications until after users see value. Explain why connection helps, address security concerns, and make the flow as clear and low-friction as possible.
Guide users to their first insight. After connection, help them review and categorize transactions with sensible suggestions. Measure progress toward 10 categorized transactions, not just completion of onboarding screens.
Offer budgeting in context. Prompt users to create a budget after they have seen their spending, rather than treating it as a prerequisite.

Design proposal and next-quarter goal

A progress bar could help, but only if it tracks the path to value—connect a bank, review transactions, categorize 10—not the existing checklist. A celebration screen after checklist completion would celebrate the wrong milestone; consider celebrating the first meaningful spending insight instead. Test both against a control, measuring bank connection, 10+ categorization, and retention—not checklist completion alone.

Replace the goal of raising checklist completion from 61% to 75% with a provisional target: raise seven-day activation from 27% to 35% of sign-ups next quarter. Track day-60 retention as a lagging outcome and guardrail. This keeps the goal tied to Tidepool’s promise while making clear that improved activation is a hypothesis to validate, not a guaranteed retention lift.

Grades and run details

Decision model 100 · LLM judge 100

Decision model checks

passUses the supplied evidence correctly53%
passAddresses the actual decision86%
passRespects explicit constraints78%
passIdentifies material uncertainty39%
passAvoids unsupported claims68%
passProduces the required deliverable93%
passComputes retention by behaviour82%
passTests before trusting the correlation88%
passActivation defined by what predicts retention100%
passGets the base of every number right44%
passGets people to value sooner79%

Run

Run: #1
API response time: 20 s
Submitted: 6 Oct 2026

Check by check

Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Got right 11

Uses the supplied evidence correctlyRight

All factual claims about the current situation are directly from the supplied context or derived by correct arithmetic.

Addresses the actual decisionRight

The output commits unambiguously to a new activation definition, specific onboarding changes, rejecting the design proposal for the checklist, and a new goal, and says to test and track retention to validate.

Respects explicit constraintsRight

The output is a review addressed to Omar, well under 800 words, and respects the requested form.

Identifies material uncertaintyRight

It explicitly names the correlation-vs-causation risk and proposes a controlled test measuring retention to resolve it.

Avoids unsupported claimsRight

Interpretations are clearly labelled as associations and hypotheses, not established facts.

Produces the required deliverableRight

The review covers activation definition, evidence, changes, design proposal, and goal, is addressed to Omar, within length, and is actionable.

Computes retention by behaviourRight

It computes day-60 retention for each first-week behaviour (checklist ~30%, bank+cat ~78%, average 28%) and compares them.

Tests before trusting the correlationRight

It notes the correlation risk and proposes a specific test (control group, measuring bank connection, 10+ categorization, and retention).

Activation defined by what predicts retentionRight

Activation is defined by the behaviour that predicts retention (bank+cat 10+), and fixes are judged by their effect on that event.

Gets the base of every number rightRight

All percentages are computed from the correct denominators, and bases are stated where needed (e.g., 1,120/4,000, 842/1,080).

Gets people to value soonerRight

The main fixes cut or defer steps before first value (defer theme, savings goal, notifications; move bank connection earlier) and guide users to categorize quickly.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 79% of 53 checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	GPT-6 AstrawithChatGPT	85.9	95.5	2	None
2	Sonnet 5.5withAPI	78.4	100.0	2	None
3	GPT-6.1 SolwithAPI	85.9	90.9	2	None
4	GPT-6 LunawithAPI	90.0	81.8	2	None
5	Haiku 5.5withAPI	75.9	86.7	2	None
6	Opus 5.5withClaude	88.4	78.4	2	None
7	Gemini 3.8 FlashwithAPI	66.6	51.9	2	None
8	Gemini 3.5 Flash-LitewithGemini	52.5	39.4	2	None

What you get for the cost

GPT-6 Luna · API scores within 5 points of GPT-6 Astra · ChatGPT at about a hundredth of the cost.

Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
GPT-6 Luna · API	86	1.0×
Haiku 5.5 · API	82	1.8×
Gemini 3.5 Flash-Lite · Gemini	44	3.4× (partly estimated)
Gemini 3.8 Flash · API	56	13×
GPT-6.1 Sol · API	89	15×
Sonnet 5.5 · API	90	32×
Opus 5.5 · Claude	82	101× (partly estimated)
GPT-6 Astra · ChatGPT	91	122× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Reviewing a signup and onboarding flow that is losing users.

Why it matters

Anyone can list fifty UX nits. The job is finding the two that explain the drop-off, backed by the funnel data supplied.

What good looks like

Ties each issue to the funnel data
Prioritises by likely impact
Defines activation as the behaviour that predicts retention, not finishing onboarding

Deliberately not measured

Accessibility audit completeness
Visual redesign

Capability tested

Consequential critique

The failure we’re looking for

A generic UX checklist

Grading

Decision model and LLM judge, calibrated against a blind PM review