Tasks / Growth

Activation & onboarding review

Can the model find the friction that matters most and prioritise the fixes?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 69% were usable with at most a quick edit.

Reliably right

Respects explicit constraints100% pass
The output is a review addressed to Omar, well under 800 words, and respects the requested form.
GPT-6 Luna · API · The checklist isn't the moment
Produces the required deliverable100% pass
The review covers activation definition, evidence, changes, design proposal, and goal, is addressed to Omar, within length, and is actionable.
GPT-6 Luna · API · The checklist isn't the moment
Activation defined by what predicts retention100% pass
Activation is defined by the behaviour that predicts retention (bank+cat 10+), and fixes are judged by their effect on that event.
GPT-6 Luna · API · The checklist isn't the moment

Where it slips

Uses the supplied evidence correctly44% pass
The output states 'we have no step-level data for screens 2–8', 'a blank builder is hard for non-analysts', and 'people invite colleagues to something useful, not to a setup screen' as facts, none of which appear in the brief or follow from arithmetic.
Opus 5.5 · Claude · Analytics tool losing users at setup
Avoids unsupported claims55% pass
It presents 'a blank builder is hard for non-analysts' and 'people invite colleagues to something useful, not to a setup screen' as established facts without labelling them as hypotheses.
Opus 5.5 · Claude · Analytics tool losing users at setup
Addresses the actual decision56% pass
The output commits to a ranked order but does not state what result or condition would change that order, as required.
GPT-6 Astra · ChatGPT · Analytics tool losing users at setup

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

Activation defined by what predicts retention
Does the review define (or question) the activation event by whether it predicts retention, and judge fixes by their effect on that event rather than on step completion?
Passes when Names the activation behaviour and its link to retention (from the supplied data, or as an explicit assumption to verify), and ranks fixes by their effect on it.
Bangaly KabaLenny’s Podcast Lauryn IsfordLenny’s Podcast
Gets people to value sooner
Do the recommended fixes shorten the path to the first moment of value, by removing or deferring steps before it or starting people with templates or sample data, rather than adding tours, tooltips or extra steps?
Passes when The main fixes cut or defer steps before first value, or give a warm start, and say which steps go.
Hila QuLenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for onboarding at Tidepool. Omar, our Head of Product, has asked for a review of onboarding (his note is below). Write it in no more than 800 words: what activation should mean for us, the evidence, what to change in onboarding, and what to make of the design team's proposal and next quarter's goal. The files are attached.

What the model was given4 items: About Tidepool, omar_note.md, onboarding_today.md, first_week_behaviour.csv (August sign-ups: how many who did each thing in their first week were still active at day 60)

About TidepoolA personal finance app: connect your bank, see where your money goes, and budget.

omar_note.md1 lines · Download

From Omar (Head of Product): "Activation is flat at 61%. Review our onboarding and tell me what to fix. The design team has proposed a celebration screen and a progress bar for the checklist."

onboarding_today.md8 lines · Download

# Onboarding today (for 4,000 sign-ups in August)

1. Sign up (100%)
2. Setup checklist: name your account, pick a theme, set a savings goal, enable notifications (61% complete it)
3. Connect a bank (50%)
4. Categorise transactions (34% reach 10)

…

first_week_behaviour.csv (August sign-ups: how many who did each thing in their first week were still active at day 60)first_week_behaviour,retained_at_day_60,churned_by_day_60 All sign-ups,1120,2880 Completed the setup checklist,742,1696 Connected a bank account,964,1036 Categorised 10+ transactions,891,489 Connected a bank AND categorised 10+ transactions,842,238 Created a budget,410,820 Opened the app 4+ days,870,1310

What a strong answer doesThe answer key the graders mark against

Works out retention for each first-week behaviour from the file: all sign-ups 28% (1,120 of 4,000); completed the checklist about 30% (barely better than average); connected a bank about 48%; categorised 10+ transactions about 65%; connected a bank and categorised 10+ about 78%, and that group is about three-quarters of everyone retained (842 of 1,120). So the checklist isn't activation and raising its completion to 75% would do little; activation should be connecting a bank and categorising ten transactions in week one. Recommends cutting or deferring checklist steps that sit before that (theme, notifications, savings goal) and getting people to connect a bank and categorise sooner (for example suggested categories to confirm), and resets the goal to the new activation rate (about 27% today, 1,080 of 4,000). Treats the celebration screen and progress bar as polishing the wrong step. Notes that the data shows correlation and proposes a test to confirm that driving the behaviour improves retention.

Critical failures (cap the score at 40)

Recommends raising checklist completion as the main goal

Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

GPT-6 LunaAPI100Usable with a quick edit GPT-6 AstraChatGPT92Usable with a quick edit GPT-6.1 SolAPI92Usable with a quick edit Sonnet 5.5API92Usable with a quick edit Haiku 5.5API88Usable with a quick edit Opus 5.5Claude75Needs more work Gemini 3.8 FlashAPI60Needs more work Gemini 3.5 Flash-LiteGemini40Needs more work

GPT-6 AstraChatGPT

Usable with a quick edit9 of 11 checks passed

The review is clear, evidence-based, and covers all required points; a PM could act on it with only minor wording tweaks.

Output

Onboarding review: activate on financial value, not setup

Recommendation: Replace checklist completion as our activation metric with connecting a bank and categorising at least 10 transactions within seven days of signup. Reorient onboarding around that outcome. The celebration screen and progress bar are secondary; making the current checklist more engaging risks improving the wrong metric.

What activation should mean

For Tidepool, activation should mean a user has taken the first meaningful steps toward understanding their spending—not finished configuring the app.

Bank connection brings their financial data into Tidepool; categorisation makes that data useful. Together, these are our strongest available proxy for experiencing core value. This is a working definition, not proof that exactly 10 categorisations creates a lasting habit. We should validate the threshold and whether users actually understand something useful about their money.

Budget creation can follow that first insight. The evidence does not support making it a mandatory activation requirement today.

What the evidence says

Among August’s 4,000 signups, overall day-60 retention was 28%.

First-week behaviour	Users	Day-60 retention
---	---:	---:
Completed setup checklist	2,438	30.4%
Connected a bank	2,000	48.2%
Categorised 10+ transactions	1,380	64.6%
Connected a bank and categorised 10+	1,080	78.0%
Created a budget	1,230	33.3%
Opened the app on 4+ days	2,180	39.9%

The checklist has weak separation: completers retained at 30.4%, versus 24.2% for non-completers. Its 61% completion rate is therefore a poor headline measure of delivered value.

By contrast, only 27% of signups connected a bank and categorised 10+ transactions, but they accounted for 842 of 1,120 retained users—75% of everyone retained. Users who did not achieve both retained at 9.5%.

These are observational associations. More motivated users may both complete these actions and remain active; we cannot promise 78% retention simply by pushing people through them. Behaviour groups also overlap, so the reported steps should not be treated as a strictly sequential funnel. Before testing, reconcile the rounded onboarding figures with event counts and standardise the day-60 activity definition.

What to change

Remove configuration as a gate to value. Defer theme selection and account naming; make savings goals optional. Request notification permission when there is a clear benefit, such as a relevant budget alert—not as checklist work.

Bring bank connection forward. Explain why access is needed, what data is used, and how users retain control. Instrument connection starts, provider errors, unsupported banks, abandonment and successful imports. Preserve a skip/retry path rather than forcing consent.

Guide users from imported transactions to spending insight. Offer a short, low-effort review of suggested categories. Let users confirm or correct them, then show an understandable spending summary. Do not require ten arbitrary manual edits merely to satisfy the metric.

Offer a budget as the next useful action. Use the spending summary to suggest a realistic starting budget, rather than asking users to invent one before seeing their finances.

Test the redesigned flow against today’s onboarding. Measure seven-day activation first, then downstream retention; monitor connection failures, abandonment and user-reported trust and usefulness.

Design proposal and next quarter’s goal

A progress bar is worth testing if it tracks meaningful steps toward understanding spending. It should not encourage completion of low-value setup tasks. A celebration screen belongs after a genuine milestone and should reinforce the insight gained, not interrupt access to it. Neither should take priority over bank-connection and categorisation friction.

Replace the 61% → 75% checklist goal with a goal for the new activation definition. I propose 27% → 35% within seven days, provisionally, subject to instrumentation validation and early experiment results. That is a planning target—not a forecast supported by this dataset.

Keep checklist completion as a diagnostic. Judge success by increased meaningful activation without sacrificing trust or retention, and confirm the retention effect once next quarter’s cohorts mature.

Grades and run details

Decision model 82 · LLM judge 100

Decision model checks

failUses the supplied evidence correctly90%
passAddresses the actual decision86%
passRespects explicit constraints30%
passIdentifies material uncertainty83%
passAvoids unsupported claims43%
passProduces the required deliverable42%
passComputes retention by behaviour32%
passTests before trusting the correlation97%
passActivation defined by what predicts retention99%
failGets the base of every number right82%
passGets people to value sooner78%

Run

Run: #1
Time to output: 30 s
Submitted: 5 Oct 2026

Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit4 of 11 checks passed

The output misstates the baseline activation rate (34% instead of 27%) and lacks a causation test, so a PM would need to re-check the numbers and add a test plan.

Output

To: Omar, Head of Product From: PM, Onboarding Date: September 2024 Subject: Onboarding Review & Activation Strategy

---

1. What Activation Should Mean for Us

Currently, our definition of activation—completing a checklist of cosmetic preferences (name, theme, savings goal, notifications)—is flawed. It measures vanity completion rather than value realization.

As a personal finance app, Tidepool delivers core value when users connect their financial reality and engage with it. Looking at our August cohort (4,000 sign-ups), the data tells a clear story about Day 60 retention:

Checklist completers retain at 30.4% (742/2,438).
Bank connectors retain at 48.2% (964/2,000).
Users who categorise 10+ transactions retain at 64.5% (891/1,380).
Users who do BOTH (Connect + Categorise) retain at 78.0% (842/1,080).

Recommendation: We must redefine "Activation" from “completed the setup checklist” to “Connected a bank AND categorised at least 10 transactions within their first week.” This anchors our success metric to actual product utility, shifting our focus from cosmetic setup to establishing long-term financial habits.

---

2. The Evidence: Where We Are Losing Users

Our current funnel drops off steeply: 1. Sign up: 4,000 (100%) 2. Setup checklist: 2,440 (61%) 3. Connect a bank: 2,000 (50%) 4. Categorise 10 transactions: 1,360 (34%)

The biggest leak is bank connection (dropping 11 percentage points right after the checklist) and transaction categorisation (dropping another 16 points).

Interestingly, the checklist itself acts as a barrier. Users are spending cognitive load on naming accounts and picking themes before experiencing the core product. Meanwhile, power features like "Created a budget" (33% retention) and opening the app 4+ days are strong secondary indicators, but bank linking and categorisation are the undisputed gateways to retention.

---

3. What to Change in Onboarding

To drive our new definition of activation, we need to restructure the first-time user experience (FTUX):

Kill the upfront setup checklist: Strip out non-essential steps (naming accounts, picking themes, notifications) from the critical path. Move them to profile settings or prompt for them after value has been proven.
Accelerate time-to-bank: Move bank connection immediately following sign-up. Users downloaded Tidepool to see their money; let them see it instantly via Plaid/Aggregator integration.
Gamify the "aha" moment: Guide users smoothly from bank connection directly into categorising their first 10 transactions. Use smart auto-categorisation with a clear, frictionless bulk-approval or swipe UI to make reaching the 10-transaction milestone effortless.

---

4. Verdict on the Design Proposal and Next Quarter's Goal

The Design Team’s Proposal (Celebration screen + Progress bar): * Verdict: Reject in its current form. Adding a progress bar and celebration screen for the existing checklist optimizes the wrong metric. It puts lipstick on a superficial activation loop. * Pivot: If we redesign onboarding to focus on connecting a bank and categorising transactions, then we can use a celebration screen and progress bar—not to celebrate picking a theme, but to celebrate linking an account and hitting that crucial 10-transaction milestone.

Next Quarter's Goal (Raise checklist completion from 61% to 75%): * Verdict: Abandon this goal. Optimizing checklist completion is a local maximum that misaligns product and user success. Chasing 75% checklist completion will likely result in higher drop-offs further down the funnel and flat-to-declining Day 60 retention. * New Goal: We should replace this with a target to raise our true activation rate (Bank Connected + 10 Transactions Categorised) from the current baseline (~34%) to 50% next quarter. This directly correlates with our actual business health and user retention.

Claims the judge couldn’t find in the brief

The checklist itself acts as a barrier.
Users are spending cognitive load on naming accounts and picking themes before experiencing the core product.
Bank linking and categorisation are the undisputed gateways to retention.

Grades and run details

Decision model 50 · LLM judge 33

Decision model checks

failUses the supplied evidence correctly41%
passAddresses the actual decision53%
passRespects explicit constraints33%
failIdentifies material uncertainty87%
failAvoids unsupported claims33%
passProduces the required deliverable48%
partialComputes retention by behaviour25%
failTests before trusting the correlation98%
passActivation defined by what predicts retention99%
failGets the base of every number right55%
passGets people to value sooner93%

Run

Run: #1
Time to output: 5 s
Submitted: 5 Oct 2026

Opus 5.5Claude

Needs more than a quick edit8 of 11 checks passed

Unsupported claims about the value of naming/theme and the visibility of bank-connection drop-off would need correction or removal before use.

Output

Onboarding review: what activation should mean

To: Omar | From: Onboarding PM | Data: August cohort (4,000 sign-ups), day-60 retention

Summary

Activation is flat because we are measuring the wrong thing. Finishing the setup checklist barely predicts whether someone stays. Connecting a bank and categorising 10+ transactions in week one does predict it strongly. I recommend we redefine activation around that, rebuild onboarding to get people there faster, and replace next quarter's checklist goal.

What activation should mean

Activated = connected a bank AND categorised 10+ transactions in the first week.

This is the moment Tidepool delivers its core promise: seeing where your money goes. Only 27% of August sign-ups reached it, so our real activation rate is less than half the 61% we report.

The evidence

Day-60 retention by first-week behaviour (baseline: 28% of all sign-ups retained):

First-week behaviour	Users	Retained if did it	Retained if didn't
Completed setup checklist	2,438 (61%)	30%	24%
Created a budget	1,230 (31%)	33%	26%
Opened app 4+ days	2,180 (55%)	40%	14%
Connected a bank	2,000 (50%)	48%	8%
Categorised 10+ transactions	1,380 (35%)	65%	9%
Bank + 10+ categorised	1,080 (27%)	78%	10%

Five things stand out:

The checklist is a weak signal. Completers retain at 30% versus 24% for non-completers, close to the 28% baseline. 1,696 people completed it and still churned.
Bank plus categorisation is the strongest signal. Users who did both retain at 78%. They make up 27% of sign-ups but 75% of everyone retained at day 60.
Each half is weak on its own. About 920 people connected a bank without categorising, and they retained at 13%. About 300 categorised without a bank, and they retained at 16%. The value comes from real transactions that the user has made sense of.
The biggest leaks are at the steps that matter. Half of sign-ups never connect a bank. Of those who do, 46% never reach 10 categorised transactions.
Budget creation isn't a good activation metric. Its retention lift is small and only 31% do it. That is worth investigating separately for a budgeting app.

Caveats: this is one cohort and the data is correlational. Motivated users may simply do more of everything. The gaps are large enough to act on, but we should confirm cause with experiments rather than assume it.

What to change in onboarding

Put bank connection first. Today it comes after a four-item checklist that we lose 39% of users on. Naming the account and picking a theme add no value at signup and can move to settings. Ask for notifications later, once there is something worth notifying about, such as a first spending summary. The savings goal becomes optional after categorisation.
Make connecting a bank feel safe and fast. Address security concerns at the moment of asking, with plain-language reassurance about read-only access. Offer the most popular banks up front. Instrument the drop-off between starting and finishing the connection, which we can't currently see.
Guide the first 10 categorisations straight after connecting. Show recent transactions with suggested categories so each one is a one-tap confirm. End with an immediate payoff: "Here's where your money went this month."
Recover people who stall. Send a reminder to anyone who connected but hasn't categorised within 48 hours. That group is about 920 people a month retaining at 13%.

Each change should ship as an A/B test with day-60 retention as the outcome. That is how we confirm the relationship is causal.

The design team's proposal

A celebration screen and a progress bar are reasonable patterns, but aimed at the checklist they would polish a step that doesn't drive retention. I'd ask design to point the same ideas at the new milestone:

A progress bar for "connect bank, then categorise 10 transactions."
A celebration on the 10th categorisation, paired with the first spending insight.

That is a small redirect of their work, not a rejection of it.

Next quarter's goal

I recommend we retire the 61% → 75% checklist goal. Even if we hit it, and every extra completer gained the full 6-point retention gap (which is generous), that would mean roughly 30–35 more retained users per 4,000 sign-ups.

Compare moving just 10% of sign-ups (400 people) into the bank-plus-categorised group. If even a fraction of that 68-point retention gap is causal, the payoff is several times larger.

Proposed goal: raise week-one bank connection plus 10+ categorised transactions from 27% to 35%, with day-60 retention as the guardrail metric. We should set the final target once the first experiment shows how much the metric moves.

Next steps

This week: confirm the new activation definition with you and analytics, and backfill it for June and July to check it holds across cohorts.
Weeks 2–4: run an A/B test of bank-first onboarding with the checklist deferred, and add instrumentation to the bank connection flow.
Weeks 4–8: test the guided categorisation flow and the redirected progress bar and celebration screen.

Claims the judge couldn’t find in the brief

Naming the account and picking a theme add no value at signup.
We can't currently see drop-off between starting and finishing the bank connection.

Grades and run details

Decision model 82 · LLM judge 75

Decision model checks

failUses the supplied evidence correctly74%
passAddresses the actual decision93%
passRespects explicit constraints39%
passIdentifies material uncertainty80%
passAvoids unsupported claims30%
passProduces the required deliverable62%
passComputes retention by behaviour36%
passTests before trusting the correlation100%
passActivation defined by what predicts retention100%
failGets the base of every number right70%
passGets people to value sooner81%

Run

Run: #1
Time to output: 32 s
Submitted: 5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 7

Uses the supplied evidence correctlyMixedWrongWrong

GPT-6 Astra · ChatGPT

All factual claims about the current situation are directly supported by the supplied context or correct arithmetic.

Gemini 3.5 Flash-Lite · Gemini

The output states 'the checklist itself acts as a barrier' and similar causal claims as facts, which are not in the supplied context and do not follow by arithmetic.

Opus 5.5 · Claude

Two claims about the current situation ('naming and theme add no value at signup' and 'we can't currently see drop-off') are not in the supplied context and are not arithmetic derivations.

Addresses the actual decisionRightMixedRight

GPT-6 Astra · ChatGPT

Commits early to a clear activation definition and changes, and states that validation and experiment results would change the call.

Gemini 3.5 Flash-Lite · Gemini

The output does not state what result or condition would change its recommendation, so it fails to commit with a clear change condition.

Opus 5.5 · Claude

Commits early to redefining activation to bank + categorised, changing onboarding, and retiring the checklist goal, framed for Omar, with conditions (experiments) that would change the call.

Identifies material uncertaintyRightWrongRight

GPT-6 Astra · ChatGPT

Names the observational nature, threshold uncertainty, and need for validation; says how to resolve them.

Gemini 3.5 Flash-Lite · Gemini

The output does not name any unknowns that could change the decision or how they would be resolved.

Opus 5.5 · Claude

Names the correlation vs causation risk, bounds it with the data, and proposes A/B tests with day-60 retention to resolve it.

Avoids unsupported claimsRightWrongMixed

GPT-6 Astra · ChatGPT

Hypotheses and forecasts are clearly labelled as such, and confident claims are supported by the evidence.

Gemini 3.5 Flash-Lite · Gemini

Interpretations like 'the checklist itself acts as a barrier' and 'undisputed gateways' are presented as established fact without being labelled as hypotheses.

Opus 5.5 · Claude

Presents 'naming and theme add no value' and 'we can't see drop-off' as established facts without labelling them as hypotheses or assumptions.

Computes retention by behaviourRightWrongRight

GPT-6 Astra · ChatGPT

Computes day-60 retention for each first-week behaviour and compares with the 28% average.

Gemini 3.5 Flash-Lite · Gemini

The output does not compute the overall average retention (28%) and does not compare the subgroup rates to it.

Opus 5.5 · Claude

Computes day-60 retention for each behaviour: checklist ~30%, bank+categorised ~78%, average 28%, and uses them.

Tests before trusting the correlationRightWrongRight

GPT-6 Astra · ChatGPT

Notes the correlation risk and proposes a specific test (redesigned flow vs. today, measuring retention) to check causation.

Gemini 3.5 Flash-Lite · Gemini

The output does not note that the retention differences are correlations and does not propose a test to confirm causation.

Opus 5.5 · Claude

Notes the correlation risk and proposes a specific A/B test driving the behaviour with retention as the measure.

Gets the base of every number rightMixedWrongMixed

GPT-6 Astra · ChatGPT

All percentages are computed from the correct denominators, and bases are stated or clear from context.

Gemini 3.5 Flash-Lite · Gemini

The baseline for the new activation rate is given as ~34%, but the correct rate for 'connected bank AND categorised 10+' is 1,080/4,000 = 27%.

Opus 5.5 · Claude

All derived percentages (30%, 78%, 27%, 13%, 16%, etc.) check out against the supplied data, and bases are stated or clear.

All got right 4

Respects explicit constraintsRightRightRight

GPT-6 Astra · ChatGPT

The review is under 800 words, addresses Omar, and covers all requested elements.

Gemini 3.5 Flash-Lite · Gemini

The output is a memo to Omar, under 800 words, and addresses the requested topics.

Opus 5.5 · Claude

The output is a review addressed to Omar, within 800 words, and the proposed changes would enforce the new activation definition in practice.

Produces the required deliverableRightRightRight

GPT-6 Astra · ChatGPT

The output is a complete review with recommendation, evidence, changes, and view on proposal and goal, usable as is.

Gemini 3.5 Flash-Lite · Gemini

The review is complete, in memo form, within the word limit, and a PM could act on it with light edits.

Opus 5.5 · Claude

Delivers a complete onboarding review memo to Omar, within the word limit, with actionable recommendations and next steps.

Activation defined by what predicts retentionRightRightRight

GPT-6 Astra · ChatGPT

Defines activation by retention (bank + categorise 10+) and ranks fixes by their effect on that event.

Gemini 3.5 Flash-Lite · Gemini

Activation is redefined based on retention prediction (bank + categorise), and fixes are judged by their effect on that event.

Opus 5.5 · Claude

Defines activation by retention prediction (bank + categorised) and ranks fixes by their effect on that event, not on checklist completion.

Gets people to value soonerRightRightRight

GPT-6 Astra · ChatGPT

Main fixes cut or defer configuration steps before value and bring bank connection and categorisation forward.

Gemini 3.5 Flash-Lite · Gemini

The main fixes cut or defer checklist steps before value and move bank connection and categorisation earlier.

Opus 5.5 · Claude

Main fixes cut or defer steps before first value (move naming/theme to settings, defer notifications, make savings goal optional) and guide categorisation right after bank connection.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 79% of 53 checks.

#	Model · Harness	Decision model	LLM judge	Runs	Critical failures
1	GPT-6 AstrawithChatGPT	85.9	95.5	2	None
2	Sonnet 5.5withAPI	78.4	100.0	2	None
3	GPT-6.1 SolwithAPI	85.9	90.9	2	None
4	GPT-6 LunawithAPI	90.0	81.8	2	None
5	Haiku 5.5withAPI	75.9	86.7	2	None
6	Opus 5.5withClaude	88.4	78.4	2	None
7	Gemini 3.8 FlashwithAPI	66.6	51.9	2	None
8	Gemini 3.5 Flash-LitewithGemini	52.5	39.4	2	None

What you get for the cost

GPT-6 Luna · API scores within 5 points of GPT-6 Astra · ChatGPT at about a hundredth of the cost.

Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)

Task score and relative cost by setup
Setup	Task score	Relative cost (cheapest = 1)
GPT-6 Luna · API	86	1.0×
Haiku 5.5 · API	82	1.8×
Gemini 3.5 Flash-Lite · Gemini	44	3.4× (partly estimated)
Gemini 3.8 Flash · API	56	13×
GPT-6.1 Sol · API	89	15×
Sonnet 5.5 · API	90	32×
Opus 5.5 · Claude	82	101× (partly estimated)
GPT-6 Astra · ChatGPT	91	122× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Reviewing a signup and onboarding flow that is losing users.

Why it matters

Anyone can list fifty UX nits. The job is finding the two that explain the drop-off, backed by the funnel data supplied.

What good looks like

Ties each issue to the funnel data
Prioritises by likely impact
Defines activation as the behaviour that predicts retention, not finishing onboarding

Deliberately not measured

Accessibility audit completeness
Visual redesign

Capability tested

Consequential critique

The failure we’re looking for

A generic UX checklist

Grading

Decision model and LLM judge, calibrated against a blind PM review

What AI gets right here, and what you’ll still have to catch

Reliably right

Where it slips

How it’s graded

Activation defined by what predicts retention

Gets people to value sooner

Plus our standard checks

The tasks

The outputs

Output

Onboarding review: activate on financial value, not setup

What activation should mean

What the evidence says

What to change

Design proposal and next quarter’s goal

Grades and run details

Decision model checks

Run

Output

1. What Activation Should Mean for Us

2. The Evidence: Where We Are Losing Users

3. What to Change in Onboarding

4. Verdict on the Design Proposal and Next Quarter's Goal

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Run

Output

Onboarding review: what activation should mean

Summary

What activation should mean

The evidence

What to change in onboarding

The design team's proposal

Next quarter's goal

Next steps

Claims the judge couldn’t find in the brief

Grades and run details

Decision model checks

Run

Check by check

Where they differ 7

All got right 4

Results

What you get for the cost

About the task

The PM job

Why it matters

What good looks like

Deliberately not measured