Tasks / Growth

Activation & onboarding review

Can the model find the friction that matters most and prioritise the fixes?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 69% were usable with at most a quick edit.

Reliably right

Respects explicit constraints100% pass
The output is a review addressed to Omar, well under 800 words, and respects the requested form.
GPT-6 Luna · API · The checklist isn't the moment
Produces the required deliverable100% pass
The review covers activation definition, evidence, changes, design proposal, and goal, is addressed to Omar, within length, and is actionable.
GPT-6 Luna · API · The checklist isn't the moment
Activation defined by what predicts retention100% pass
Activation is defined by the behaviour that predicts retention (bank+cat 10+), and fixes are judged by their effect on that event.
GPT-6 Luna · API · The checklist isn't the moment

Where it slips

Uses the supplied evidence correctly44% pass
The output states 'we have no step-level data for screens 2–8', 'a blank builder is hard for non-analysts', and 'people invite colleagues to something useful, not to a setup screen' as facts, none of which appear in the brief or follow from arithmetic.
Opus 5.5 · Claude · Analytics tool losing users at setup
Avoids unsupported claims55% pass
It presents 'a blank builder is hard for non-analysts' and 'people invite colleagues to something useful, not to a setup screen' as established facts without labelling them as hypotheses.
Opus 5.5 · Claude · Analytics tool losing users at setup
Addresses the actual decision56% pass
The output commits to a ranked order but does not state what result or condition would change that order, as required.
GPT-6 Astra · ChatGPT · Analytics tool losing users at setup

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Activation defined by what predicts retention
Does the review define (or question) the activation event by whether it predicts retention, and judge fixes by their effect on that event rather than on step completion?
Passes when Names the activation behaviour and its link to retention (from the supplied data, or as an explicit assumption to verify), and ranks fixes by their effect on it.
Bangaly KabaLenny’s Podcast Lauryn IsfordLenny’s Podcast
Gets people to value sooner
Do the recommended fixes shorten the path to the first moment of value, by removing or deferring steps before it or starting people with templates or sample data, rather than adding tours, tooltips or extra steps?
Passes when The main fixes cut or defer steps before first value, or give a warm start, and say which steps go.
Hila QuLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for onboarding at Tidepool. Omar, our Head of Product, has asked for a review of onboarding (his note is below). Write it in no more than 800 words: what activation should mean for us, the evidence, what to change in onboarding, and what to make of the design team's proposal and next quarter's goal. The files are attached.

What the model was given4 items: About Tidepool, omar_note.md, onboarding_today.md, first_week_behaviour.csv (August sign-ups: how many who did each thing in their first week were still active at day 60)

About TidepoolA personal finance app: connect your bank, see where your money goes, and budget.

omar_note.md1 lines · Download

From Omar (Head of Product): "Activation is flat at 61%. Review our onboarding and tell me what to fix. The design team has proposed a celebration screen and a progress bar for the checklist."

onboarding_today.md8 lines · Download

# Onboarding today (for 4,000 sign-ups in August)

1. Sign up (100%)
2. Setup checklist: name your account, pick a theme, set a savings goal, enable notifications (61% complete it)
3. Connect a bank (50%)
4. Categorise transactions (34% reach 10)

…

first_week_behaviour.csv (August sign-ups: how many who did each thing in their first week were still active at day 60)first_week_behaviour,retained_at_day_60,churned_by_day_60 All sign-ups,1120,2880 Completed the setup checklist,742,1696 Connected a bank account,964,1036 Categorised 10+ transactions,891,489 Connected a bank AND categorised 10+ transactions,842,238 Created a budget,410,820 Opened the app 4+ days,870,1310

What a strong answer doesThe answer key the graders mark against

Works out retention for each first-week behaviour from the file: all sign-ups 28% (1,120 of 4,000); completed the checklist about 30% (barely better than average); connected a bank about 48%; categorised 10+ transactions about 65%; connected a bank and categorised 10+ about 78%, and that group is about three-quarters of everyone retained (842 of 1,120). So the checklist isn't activation and raising its completion to 75% would do little; activation should be connecting a bank and categorising ten transactions in week one. Recommends cutting or deferring checklist steps that sit before that (theme, notifications, savings goal) and getting people to connect a bank and categorise sooner (for example suggested categories to confirm), and resets the goal to the new activation rate (about 27% today, 1,080 of 4,000). Treats the celebration screen and progress bar as polishing the wrong step. Notes that the data shows correlation and proposes a test to confirm that driving the behaviour improves retention.

Critical failures (cap the score at 40)

Recommends raising checklist completion as the main goal

Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

GPT-6 LunaAPI100Usable with a quick edit GPT-6 AstraChatGPT92Usable with a quick edit GPT-6.1 SolAPI92Usable with a quick edit Sonnet 5.5API92Usable with a quick edit Haiku 5.5API88Usable with a quick edit Opus 5.5Claude75Needs more work Gemini 3.8 FlashAPI60Needs more work Gemini 3.5 Flash-LiteGemini40Needs more work

Gemini 3.8 FlashAPI

Needs more than a quick edit6 of 11 checks passed

The output lacks a causation caveat and test proposal, and makes unsupported causal claims; a PM would need to add those before using it.

Output

To: Omar, Head of Product From: PM, Onboarding Subject: Onboarding Review & Activation Strategy

---

Executive Summary

Our current activation rate (61%) is flat because we are measuring and optimizing the wrong milestone. Completing our setup checklist barely moves the needle on long-term retention.

To drive retention, we must redefine activation around core product value—connecting a bank and categorising transactions—and reorient our onboarding funnel and quarterly goals to support it.

---

1. What Activation Should Mean (and the Evidence)

Activation should represent the “Aha!” moment: the point where a user experiences Tidepool’s core value proposition and forms a lasting habit.

Currently, we define activation as completing the setup checklist (name account, pick theme, set savings goal, enable notifications). The data shows this is a vanity metric:

Baseline Day 60 Retention: 28.0% (1,120 / 4,000 sign-ups)
Checklist Completion: 30.4% Day 60 retention (742 / 2,438) — a negligible +2.4 percentage point lift over baseline.

By contrast, actions tied to our core utility (seeing where money goes) dramatically predict retention: * Connected a bank account: 48.2% Day 60 retention (964 / 2,000) * Categorised 10+ transactions: 64.6% Day 60 retention (891 / 1,380) * Connected bank AND categorised 10+ transactions: 78.0% Day 60 retention (842 / 1,080)

Users who complete both core actions retain at nearly triple the baseline rate. In fact, this group accounts for 75.2% of all retained users at Day 60 (842 of 1,120), despite representing only 27% of initial sign-ups.

New Activation Metric: > A user connects a bank account and categorises at least 10 transactions within their first 7 days.

---

2. Design Proposal & Next Quarter’s Goal

#### The Design Proposal (Celebration Screen & Progress Bar) The design team’s proposal is solving the wrong problem. Adding progress bars and celebration screens to cosmetic tasks (picking themes, naming accounts) creates artificial friction before users experience any real value. We are gamifying administrative overhead instead of utility.

Recommendation: Repurpose the design team’s work, but attach it to the real value milestone (e.g., celebrating when their first 10 transactions are imported and categorized).

#### Next Quarter’s Goal (Raise Checklist Completion from 61% to 75%) We should retire this goal immediately.

Even if we hit 75% checklist completion, model projections indicate Day 60 retention would increase by less than 1%. Optimizing an arbitrary checklist delivers zero business impact.

Proposed New Goal for Next Quarter: > Increase the core Activation Rate (Bank Connected + 10 Categorised Transactions) from 27% to 40% of sign-ups.

---

3. What to Change in Onboarding

Our onboarding funnel currently bleeds users before they experience core value: 50% drop off before connecting a bank, and only 34% categorize 10 transactions. We must clear the runway to value:

1. Eliminate the Cosmetic Checklist from First-Run Flow * Move “pick a theme” and “name your account” to Settings/Profile. * Defer “enable notifications” and “set a savings goal” until after the user sees their financial data.

2. Lead with Bank Connection * Prompt bank linking immediately post-registration. * Pre-frame the security and the payoff clearly: “Connect your account to see your last 30 days of spending instantly.”

3. Accelerate Time-to-Value (TTV) on Categorisation * Upon successful bank connection, immediately import transactions and guide the user through a rapid, swipeable 10-transaction categorisation flow (or auto-categorise and prompt a one-tap review). * Conclude with an instant visual breakdown: “Here’s where your money went this month.” This is Tidepool’s true “Aha!” moment.

4. Contextual Budgeting * Only prompt users to create a budget after categorisation. Budgeting retention is low in week one (33.3%) because users cannot budget accurately without understanding their baseline spend.

Next Steps

Align leadership on the new Activation definition.
Re-scope the design sprint from checklist gamification to a streamlined "Connect & Review 10 Transactions" flow.
Update the team OKR dashboard for next quarter.

Grades and run details

Decision model 68 · LLM judge 58

Decision model checks

failUses the supplied evidence correctly24%
passAddresses the actual decision56%
passRespects explicit constraints49%
failIdentifies material uncertainty71%
partialAvoids unsupported claims31%
passProduces the required deliverable68%
passComputes retention by behaviour71%
failTests before trusting the correlation95%
passActivation defined by what predicts retention100%
passGets the base of every number right19%
passGets people to value sooner84%

Run

Run: #1
API response time: 20 s
Submitted: 6 Oct 2026

Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit4 of 11 checks passed

The output misstates the baseline activation rate (34% instead of 27%) and lacks a causation test, so a PM would need to re-check the numbers and add a test plan.

Output

To: Omar, Head of Product From: PM, Onboarding Date: September 2024 Subject: Onboarding Review & Activation Strategy

---

1. What Activation Should Mean for Us

Currently, our definition of activation—completing a checklist of cosmetic preferences (name, theme, savings goal, notifications)—is flawed. It measures vanity completion rather than value realization.

As a personal finance app, Tidepool delivers core value when users connect their financial reality and engage with it. Looking at our August cohort (4,000 sign-ups), the data tells a clear story about Day 60 retention:

Checklist completers retain at 30.4% (742/2,438).
Bank connectors retain at 48.2% (964/2,000).
Users who categorise 10+ transactions retain at 64.5% (891/1,380).
Users who do BOTH (Connect + Categorise) retain at 78.0% (842/1,080).

Recommendation: We must redefine "Activation" from “completed the setup checklist” to “Connected a bank AND categorised at least 10 transactions within their first week.” This anchors our success metric to actual product utility, shifting our focus from cosmetic setup to establishing long-term financial habits.

---

2. The Evidence: Where We Are Losing Users

Our current funnel drops off steeply: 1. Sign up: 4,000 (100%) 2. Setup checklist: 2,440 (61%) 3. Connect a bank: 2,000 (50%) 4. Categorise 10 transactions: 1,360 (34%)

The biggest leak is bank connection (dropping 11 percentage points right after the checklist) and transaction categorisation (dropping another 16 points).

Interestingly, the checklist itself acts as a barrier. Users are spending cognitive load on naming accounts and picking themes before experiencing the core product. Meanwhile, power features like "Created a budget" (33% retention) and opening the app 4+ days are strong secondary indicators, but bank linking and categorisation are the undisputed gateways to retention.

---

3. What to Change in Onboarding

To drive our new definition of activation, we need to restructure the first-time user experience (FTUX):

Kill the upfront setup checklist: Strip out non-essential steps (naming accounts, picking themes, notifications) from the critical path. Move them to profile settings or prompt for them after value has been proven.
Accelerate time-to-bank: Move bank connection immediately following sign-up. Users downloaded Tidepool to see their money; let them see it instantly via Plaid/Aggregator integration.
Gamify the "aha" moment: Guide users smoothly from bank connection directly into categorising their first 10 transactions. Use smart auto-categorisation with a clear, frictionless bulk-approval or swipe UI to make reaching the 10-transaction milestone effortless.

---

4. Verdict on the Design Proposal and Next Quarter's Goal

The Design Team’s Proposal (Celebration screen + Progress bar): * Verdict: Reject in its current form. Adding a progress bar and celebration screen for the existing checklist optimizes the wrong metric. It puts lipstick on a superficial activation loop. * Pivot: If we redesign onboarding to focus on connecting a bank and categorising transactions, then we can use a celebration screen and progress bar—not to celebrate picking a theme, but to celebrate linking an account and hitting that crucial 10-transaction milestone.

Next Quarter's Goal (Raise checklist completion from 61% to 75%): * Verdict: Abandon this goal. Optimizing checklist completion is a local maximum that misaligns product and user success. Chasing 75% checklist completion will likely result in higher drop-offs further down the funnel and flat-to-declining Day 60 retention. * New Goal: We should replace this with a target to raise our true activation rate (Bank Connected + 10 Transactions Categorised) from the current baseline (~34%) to 50% next quarter. This directly correlates with our actual business health and user retention.

Claims the judge couldn’t find in the brief

The checklist itself acts as a barrier.
Users are spending cognitive load on naming accounts and picking themes before experiencing the core product.
Bank linking and categorisation are the undisputed gateways to retention.

Grades and run details

Decision model 50 · LLM judge 33

Decision model checks

failUses the supplied evidence correctly41%
passAddresses the actual decision53%
passRespects explicit constraints33%
failIdentifies material uncertainty87%
failAvoids unsupported claims33%
passProduces the required deliverable48%
partialComputes retention by behaviour25%
failTests before trusting the correlation98%
passActivation defined by what predicts retention99%
failGets the base of every number right55%
passGets people to value sooner93%

Run

Run: #1
Time to output: 5 s
Submitted: 5 Oct 2026

Sonnet 5.5API

Usable with a quick edit9 of 11 checks passed

The output is a complete, data-driven review with clear recommendations and next steps, requiring no re-checking of evidence or reasoning.

Output

To: Omar | Re: Onboarding review

Recommendation

Activation should mean connecting a bank and categorising 10+ transactions in the first week. Our current definition, completing the setup checklist, barely predicts retention, so the 61% is flat for a reason: we're measuring the wrong thing. I'd fix the path to the real activation event, use the design team's ideas only on that path, and replace next quarter's goal.

Evidence (4,000 August sign-ups, day-60 retention)

First-week behaviour	Users	Retained at day 60
All sign-ups	4,000	28%
Completed checklist	2,438	30%
Did not complete checklist	1,562	24%
Connected bank	2,000	48%
Did not connect bank	2,000	8%
Categorised 10+	1,380	65%
Budget created	1,230	33%
Opened app 4+ days	2,180	40%
Bank AND 10+ categorised	1,080	78%

Three things stand out:

The checklist is a weak signal. Checklist completers retain at 30%, against 28% for everyone. The checklist is theme, notifications, account name and goal, so it's mostly cosmetic setup.
The combination matters, not either step alone. Backing out the overlaps from the CSV:
Bank only (no categorising): about 920 users, 13% retained.
Categorised only (no bank): about 300 users, 16% retained.
Neither: about 1,700 users, 6% retained.
Both: 78% retained.

Users who connect but never see their spending sorted do worse than average. The value moment is seeing where the money goes, which needs both steps. - Budgets and app opens aren't the answer. Budget creators retain at 33%, close to baseline. Opening the app 4+ days (40%) is engagement, not a lever we can pull in onboarding.

Caveat: this is observational. Motivated users probably do everything, so these numbers show who retains, not what causes it. The recommendations below are therefore tests, not guarantees.

What to change

The funnel is 100% → 50% connect → 27% connect and categorise. There are two leaks:

Half of users never connect a bank, and they retain at 8%. Put bank connection first, straight after sign-up. Move theme, notifications and account name out of the critical path (defer or drop them). Look at where the connect flow fails and whether trust copy (read-only access, security) is needed. I can't tell from this data whether the checklist is causing bank drop-off, so I'd A/B test bank-first against the current order.
46% of those who connect (920 of 2,000) never reach 10 categorised transactions. This is the biggest fixable gap, since 78% of those who finish retain. Auto-categorise on connect, pre-fill best guesses, and walk users through their first 10 with one-tap confirmations rather than leaving them to find the feature.

I'd also instrument the funnel by step and by time, so we can see where in the first week people stall.

The design team's proposal

A celebration screen and a progress bar for the checklist would probably raise checklist completion, but that metric barely relates to retention, so we'd likely see a "win" with no change in retained users. I wouldn't ship them as proposed. Both ideas are cheap and sound, though, so I'd redirect them:

Make the progress bar track the real path (connect bank → categorise 10).
Put the celebration at the moment of value, such as the 10th categorised transaction, ideally with a first insight ("here's where your money went").

This is a good way to involve design in the work above rather than rejecting their proposal.

Next quarter's goal

Drop "checklist completion 61% → 75%". Pushing it up could even pull effort toward low-value steps.

Replace it with: percentage of sign-ups who connect a bank and categorise 10+ transactions within 7 days. Baseline is 27%. I'd propose a target around 35%, but that is a judgment call until we've sized the effect of the first experiments, so let's set it after a few weeks of data. Guardrail: day-60 retention among those users should hold near its current 78% (if we widen the funnel with less-motivated users, it may dip, and we should watch that). Track the real outcome, overall day-60 retention (28% today), alongside it.

Next steps

Run the bank-first vs current-order A/B.
Prototype auto-categorisation and a guided first-10 flow.
Work with design on the redirected progress bar and celebration.
Agree the new activation definition and dashboard before the quarter starts.

I'm happy to walk through the numbers if useful.

Grades and run details

Decision model 82 · LLM judge 100

Decision model checks

failUses the supplied evidence correctly77%
passAddresses the actual decision89%
passRespects explicit constraints51%
passIdentifies material uncertainty91%
passAvoids unsupported claims35%
passProduces the required deliverable65%
passComputes retention by behaviour46%
passTests before trusting the correlation95%
passActivation defined by what predicts retention100%
failGets the base of every number right56%
passGets people to value sooner71%

Run

Run: #1
API response time: 26 s
Submitted: 6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 7

Uses the supplied evidence correctlyMixedWrongMixed

Gemini 3.8 Flash · API

All statements about the current situation are directly from the supplied context or correctly derived from it.

Gemini 3.5 Flash-Lite · Gemini

The output states 'the checklist itself acts as a barrier' and similar causal claims as facts, which are not in the supplied context and do not follow by arithmetic.

Sonnet 5.5 · API

All statements about the current situation are taken directly from the supplied context or derived correctly by arithmetic.

Addresses the actual decisionMixedMixedRight

Gemini 3.8 Flash · API

The output commits to a clear answer but does not state what result or condition would change that answer, as required.

Gemini 3.5 Flash-Lite · Gemini

The output does not state what result or condition would change its recommendation, so it fails to commit with a clear change condition.

Sonnet 5.5 · API

The output commits early to a clear activation definition (connect bank and categorise 10+), gives a ranked set of changes, and says what would change the call (A/B test results).

Identifies material uncertaintyWrongWrongRight

Gemini 3.8 Flash · API

The output does not name any unknowns that could change the decision or say how they would be resolved.

Gemini 3.5 Flash-Lite · Gemini

The output does not name any unknowns that could change the decision or how they would be resolved.

Sonnet 5.5 · API

It explicitly names the correlation vs. causation risk, notes the data is observational, and proposes specific tests (bank-first A/B, auto-categorisation) with retention as the measure.

Avoids unsupported claimsWrongWrongRight

Gemini 3.8 Flash · API

It presents causal claims (e.g., checklist steps create artificial friction, low budgeting retention is because users lack baseline spend) as fact without labelling them as hypotheses.

Gemini 3.5 Flash-Lite · Gemini

Interpretations like 'the checklist itself acts as a barrier' and 'undisputed gateways' are presented as established fact without being labelled as hypotheses.

Sonnet 5.5 · API

Interpretations like 'the value moment is seeing where the money goes' are presented as conclusions from the data, not as established fact, and the overall review is cautious about causation.

Computes retention by behaviourRightWrongRight

Gemini 3.8 Flash · API

Day-60 retention rates are correctly computed: checklist ~30%, bank+categorise ~78%, average 28%.

Gemini 3.5 Flash-Lite · Gemini

The output does not compute the overall average retention (28%) and does not compare the subgroup rates to it.

Sonnet 5.5 · API

It computes day-60 retention for each first-week behaviour (checklist ~30%, bank+cat ~78%, average 28%) and compares them correctly.

Tests before trusting the correlationWrongWrongRight

Gemini 3.8 Flash · API

The output does not note that the observed relationships are correlations and does not propose a test to drive the behaviour and measure retention.

Gemini 3.5 Flash-Lite · Gemini

The output does not note that the retention differences are correlations and does not propose a test to confirm causation.

Sonnet 5.5 · API

It notes the correlation risk and proposes a specific A/B test (bank-first vs. current order) with retention as the outcome, plus tracking overall retention.

Gets the base of every number rightRightWrongMixed

Gemini 3.8 Flash · API

All percentages and rates use the correct denominators from the data, and the bases are clear.

Gemini 3.5 Flash-Lite · Gemini

The baseline for the new activation rate is given as ~34%, but the correct rate for 'connected bank AND categorised 10+' is 1,080/4,000 = 27%.

Sonnet 5.5 · API

All derived percentages and step-to-step changes use the correct denominators (e.g., 27% from 1,080/4,000, 78% from 842/1,080), and ambiguous bases are stated.

All got right 4

Respects explicit constraintsRightRightRight

Gemini 3.8 Flash · API

The output is a memo to Omar, under 800 words, and addresses all requested elements.

Gemini 3.5 Flash-Lite · Gemini

The output is a memo to Omar, under 800 words, and addresses the requested topics.

Sonnet 5.5 · API

The output is a review memo addressed to Omar, covers all requested topics, and is well under 800 words.

Produces the required deliverableRightRightRight

Gemini 3.8 Flash · API

The review is complete, in memo form, within the word limit, and a PM could act on it with light edits.

Gemini 3.5 Flash-Lite · Gemini

The review is complete, in memo form, within the word limit, and a PM could act on it with light edits.

Sonnet 5.5 · API

The output is a complete, actionable review in memo form, within the word limit, that Omar could act on without filling major gaps.

Activation defined by what predicts retentionRightRightRight

Gemini 3.8 Flash · API

Activation is defined by the behaviour that predicts retention (bank connect + 10 categorised), and fixes are ranked by their effect on reaching that event.

Gemini 3.5 Flash-Lite · Gemini

Activation is redefined based on retention prediction (bank + categorise), and fixes are judged by their effect on that event.

Sonnet 5.5 · API

Activation is defined by the behaviour that predicts retention (bank + 10 categorised), and fixes are judged by their effect on that event, not on checklist completion.

Gets people to value soonerRightRightRight

Gemini 3.8 Flash · API

The main fixes cut or defer steps before first value (cosmetic checklist, notifications, savings goal) and lead directly to bank connection and categorisation.

Gemini 3.5 Flash-Lite · Gemini

The main fixes cut or defer checklist steps before value and move bank connection and categorisation earlier.

Sonnet 5.5 · API

The main fixes cut or defer steps before value (move bank connection first, defer theme/notifications/name) and give a warm start (auto-categorise, one-tap confirmations).

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 79% of 53 checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	GPT-6 AstrawithChatGPT	85.9	95.5	2	None
2	Sonnet 5.5withAPI	78.4	100.0	2	None
3	GPT-6.1 SolwithAPI	85.9	90.9	2	None
4	GPT-6 LunawithAPI	90.0	81.8	2	None
5	Haiku 5.5withAPI	75.9	86.7	2	None
6	Opus 5.5withClaude	88.4	78.4	2	None
7	Gemini 3.8 FlashwithAPI	66.6	51.9	2	None
8	Gemini 3.5 Flash-LitewithGemini	52.5	39.4	2	None

What you get for the cost

GPT-6 Luna · API scores within 5 points of GPT-6 Astra · ChatGPT at about a hundredth of the cost.

Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
GPT-6 Luna · API	86	1.0×
Haiku 5.5 · API	82	1.8×
Gemini 3.5 Flash-Lite · Gemini	44	3.4× (partly estimated)
Gemini 3.8 Flash · API	56	13×
GPT-6.1 Sol · API	89	15×
Sonnet 5.5 · API	90	32×
Opus 5.5 · Claude	82	101× (partly estimated)
GPT-6 Astra · ChatGPT	91	122× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Reviewing a signup and onboarding flow that is losing users.

Why it matters

Anyone can list fifty UX nits. The job is finding the two that explain the drop-off, backed by the funnel data supplied.

What good looks like

Ties each issue to the funnel data
Prioritises by likely impact
Defines activation as the behaviour that predicts retention, not finishing onboarding

Deliberately not measured

Accessibility audit completeness
Visual redesign

Capability tested

Consequential critique

The failure we’re looking for

A generic UX checklist

Grading

Decision model and LLM judge, calibrated against a blind PM review

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

Activation defined by what predicts retention

Gets people to value sooner

Plus our standard checks

The tasks

The outputs

Output

Executive Summary

1. What Activation Should Mean (and the Evidence)

2. Design Proposal & Next Quarter’s Goal

3. What to Change in Onboarding

Next Steps

Grades and run details

Decision model checks

Run

Output

1. What Activation Should Mean for Us

2. The Evidence: Where We Are Losing Users

3. What to Change in Onboarding

4. Verdict on the Design Proposal and Next Quarter's Goal

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Run

Output

Recommendation

Evidence (4,000 August sign-ups, day-60 retention)

What to change

The design team's proposal

Next quarter's goal

Next steps

Grades and run details

Decision model checks

Run

Check by check

Where they differ 7

All got right 4

Results

What you get for the cost

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured