Tasks / Growth

Activation & onboarding review

Can the model find the friction that matters most and prioritise the fixes?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 69% were usable with at most a quick edit.

Reliably right

  1. Respects explicit constraints100% pass
    The output is a review addressed to Omar, well under 800 words, and respects the requested form.
    GPT-6 Luna · API · The checklist isn't the moment
  2. Produces the required deliverable100% pass
    The review covers activation definition, evidence, changes, design proposal, and goal, is addressed to Omar, within length, and is actionable.
    GPT-6 Luna · API · The checklist isn't the moment
  3. Activation defined by what predicts retention100% pass
    Activation is defined by the behaviour that predicts retention (bank+cat 10+), and fixes are judged by their effect on that event.
    GPT-6 Luna · API · The checklist isn't the moment

Where it slips

  1. Uses the supplied evidence correctly44% pass
    The output states 'we have no step-level data for screens 2–8', 'a blank builder is hard for non-analysts', and 'people invite colleagues to something useful, not to a setup screen' as facts, none of which appear in the brief or follow from arithmetic.
    Opus 5.5 · Claude · Analytics tool losing users at setup
  2. Avoids unsupported claims55% pass
    It presents 'a blank builder is hard for non-analysts' and 'people invite colleagues to something useful, not to a setup screen' as established facts without labelling them as hypotheses.
    Opus 5.5 · Claude · Analytics tool losing users at setup
  3. Addresses the actual decision56% pass
    The output commits to a ranked order but does not state what result or condition would change that order, as required.
    GPT-6 Astra · ChatGPT · Analytics tool losing users at setup

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Activation defined by what predicts retention

    Does the review define (or question) the activation event by whether it predicts retention, and judge fixes by their effect on that event rather than on step completion?

    Passes when Names the activation behaviour and its link to retention (from the supplied data, or as an explicit assumption to verify), and ranks fixes by their effect on it.

  2. Gets people to value sooner

    Do the recommended fixes shorten the path to the first moment of value, by removing or deferring steps before it or starting people with templates or sample data, rather than adding tours, tooltips or extra steps?

    Passes when The main fixes cut or defer steps before first value, or give a warm start, and say which steps go.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for onboarding at Tidepool. Omar, our Head of Product, has asked for a review of onboarding (his note is below). Write it in no more than 800 words: what activation should mean for us, the evidence, what to change in onboarding, and what to make of the design team's proposal and next quarter's goal. The files are attached.

What the model was given4 items: About Tidepool, omar_note.md, onboarding_today.md, first_week_behaviour.csv (August sign-ups: how many who did each thing in their first week were still active at day 60)
About TidepoolA personal finance app: connect your bank, see where your money goes, and budget.
omar_note.md1 lines · Download
From Omar (Head of Product): "Activation is flat at 61%. Review our onboarding and tell me what to fix. The design team has proposed a celebration screen and a progress bar for the checklist."
onboarding_today.md8 lines · Download
# Onboarding today (for 4,000 sign-ups in August)

1. Sign up (100%)
2. Setup checklist: name your account, pick a theme, set a savings goal, enable notifications (61% complete it)
3. Connect a bank (50%)
4. Categorise transactions (34% reach 10)

…
first_week_behaviour.csv (August sign-ups: how many who did each thing in their first week were still active at day 60)first_week_behaviour,retained_at_day_60,churned_by_day_60 All sign-ups,1120,2880 Completed the setup checklist,742,1696 Connected a bank account,964,1036 Categorised 10+ transactions,891,489 Connected a bank AND categorised 10+ transactions,842,238 Created a budget,410,820 Opened the app 4+ days,870,1310
What a strong answer doesThe answer key the graders mark against

Works out retention for each first-week behaviour from the file: all sign-ups 28% (1,120 of 4,000); completed the checklist about 30% (barely better than average); connected a bank about 48%; categorised 10+ transactions about 65%; connected a bank and categorised 10+ about 78%, and that group is about three-quarters of everyone retained (842 of 1,120). So the checklist isn't activation and raising its completion to 75% would do little; activation should be connecting a bank and categorising ten transactions in week one. Recommends cutting or deferring checklist steps that sit before that (theme, notifications, savings goal) and getting people to connect a bank and categorise sooner (for example suggested categories to confirm), and resets the goal to the new activation rate (about 27% today, 1,080 of 4,000). Treats the celebration screen and progress bar as polishing the wrong step. Notes that the data shows correlation and proposes a test to confirm that driving the behaviour improves retention.

Critical failures (cap the score at 40)
  • Recommends raising checklist completion as the main goal
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Usable with a quick edit11 of 11 checks passed

The output is clear, evidence-based, and actionable; a PM could use it with minor edits without rechecking evidence or redoing reasoning.

Output

To Omar — onboarding review

What activation should mean

Activation should mean a new user has reached Tidepool’s core value: connected a bank account and categorized at least 10 transactions within their first seven days. Completing setup tasks is not the same as understanding where your money goes.

What the evidence says

Of 4,000 August sign-ups, 61% completed the checklist, 50% connected a bank, and 34% categorized 10+ transactions. The behavioral data points to which events are most associated with retention at day 60:

  • All sign-ups: 28% retained (1,120/4,000)
  • Completed the checklist: 30% (742/2,438)
  • Connected a bank: 48% (964/2,000)
  • Categorized 10+ transactions: 65% (891/1,380)
  • Connected a bank and categorized 10+: 78% (842/1,080)

The combined behavior is the strongest signal and represents 27% of all sign-ups today. By contrast, checklist completion is barely above the overall retention rate. Creating a budget is associated with 33% retention, so it is not a better activation milestone on this evidence.

These are associations, not proof that onboarding changes will cause retention to rise: users who connect accounts or categorize transactions may already be more motivated. We should treat the numbers as a strong direction for what value to help users reach, then test changes and track outcomes.

What to change

  • Move bank connection earlier. Defer account naming, theme choice, savings goals, and notifications until after users see value. Explain why connection helps, address security concerns, and make the flow as clear and low-friction as possible.
  • Guide users to their first insight. After connection, help them review and categorize transactions with sensible suggestions. Measure progress toward 10 categorized transactions, not just completion of onboarding screens.
  • Offer budgeting in context. Prompt users to create a budget after they have seen their spending, rather than treating it as a prerequisite.

Design proposal and next-quarter goal

A progress bar could help, but only if it tracks the path to value—connect a bank, review transactions, categorize 10—not the existing checklist. A celebration screen after checklist completion would celebrate the wrong milestone; consider celebrating the first meaningful spending insight instead. Test both against a control, measuring bank connection, 10+ categorization, and retention—not checklist completion alone.

Replace the goal of raising checklist completion from 61% to 75% with a provisional target: raise seven-day activation from 27% to 35% of sign-ups next quarter. Track day-60 retention as a lagging outcome and guardrail. This keeps the goal tied to Tidepool’s promise while making clear that improved activation is a hypothesis to validate, not a guaranteed retention lift.

Grades and run details

Decision model 100 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly53%
  • passAddresses the actual decision86%
  • passRespects explicit constraints78%
  • passIdentifies material uncertainty39%
  • passAvoids unsupported claims68%
  • passProduces the required deliverable93%
  • passComputes retention by behaviour82%
  • passTests before trusting the correlation88%
  • passActivation defined by what predicts retention100%
  • passGets the base of every number right44%
  • passGets people to value sooner79%
Run
Run
#1
API response time
20 s
Submitted
6 Oct 2026

Usable with a quick edit9 of 11 checks passed

The output is a complete, data-driven review with clear recommendations and next steps, requiring no re-checking of evidence or reasoning.

Output

To: Omar | Re: Onboarding review

Recommendation

Activation should mean connecting a bank and categorising 10+ transactions in the first week. Our current definition, completing the setup checklist, barely predicts retention, so the 61% is flat for a reason: we're measuring the wrong thing. I'd fix the path to the real activation event, use the design team's ideas only on that path, and replace next quarter's goal.

Evidence (4,000 August sign-ups, day-60 retention)

First-week behaviourUsersRetained at day 60
All sign-ups4,00028%
Completed checklist2,43830%
Did not complete checklist1,56224%
Connected bank2,00048%
Did not connect bank2,0008%
Categorised 10+1,38065%
Budget created1,23033%
Opened app 4+ days2,18040%
Bank AND 10+ categorised1,08078%

Three things stand out:

  • The checklist is a weak signal. Checklist completers retain at 30%, against 28% for everyone. The checklist is theme, notifications, account name and goal, so it's mostly cosmetic setup.
  • The combination matters, not either step alone. Backing out the overlaps from the CSV:
  • Bank only (no categorising): about 920 users, 13% retained.
  • Categorised only (no bank): about 300 users, 16% retained.
  • Neither: about 1,700 users, 6% retained.
  • Both: 78% retained.

Users who connect but never see their spending sorted do worse than average. The value moment is seeing where the money goes, which needs both steps. - Budgets and app opens aren't the answer. Budget creators retain at 33%, close to baseline. Opening the app 4+ days (40%) is engagement, not a lever we can pull in onboarding.

Caveat: this is observational. Motivated users probably do everything, so these numbers show who retains, not what causes it. The recommendations below are therefore tests, not guarantees.

What to change

The funnel is 100% → 50% connect → 27% connect and categorise. There are two leaks:

  1. Half of users never connect a bank, and they retain at 8%. Put bank connection first, straight after sign-up. Move theme, notifications and account name out of the critical path (defer or drop them). Look at where the connect flow fails and whether trust copy (read-only access, security) is needed. I can't tell from this data whether the checklist is causing bank drop-off, so I'd A/B test bank-first against the current order.
  2. 46% of those who connect (920 of 2,000) never reach 10 categorised transactions. This is the biggest fixable gap, since 78% of those who finish retain. Auto-categorise on connect, pre-fill best guesses, and walk users through their first 10 with one-tap confirmations rather than leaving them to find the feature.

I'd also instrument the funnel by step and by time, so we can see where in the first week people stall.

The design team's proposal

A celebration screen and a progress bar for the checklist would probably raise checklist completion, but that metric barely relates to retention, so we'd likely see a "win" with no change in retained users. I wouldn't ship them as proposed. Both ideas are cheap and sound, though, so I'd redirect them:

  • Make the progress bar track the real path (connect bank → categorise 10).
  • Put the celebration at the moment of value, such as the 10th categorised transaction, ideally with a first insight ("here's where your money went").

This is a good way to involve design in the work above rather than rejecting their proposal.

Next quarter's goal

Drop "checklist completion 61% → 75%". Pushing it up could even pull effort toward low-value steps.

Replace it with: percentage of sign-ups who connect a bank and categorise 10+ transactions within 7 days. Baseline is 27%. I'd propose a target around 35%, but that is a judgment call until we've sized the effect of the first experiments, so let's set it after a few weeks of data. Guardrail: day-60 retention among those users should hold near its current 78% (if we widen the funnel with less-motivated users, it may dip, and we should watch that). Track the real outcome, overall day-60 retention (28% today), alongside it.

Next steps

  • Run the bank-first vs current-order A/B.
  • Prototype auto-categorisation and a guided first-10 flow.
  • Work with design on the redirected progress bar and celebration.
  • Agree the new activation definition and dashboard before the quarter starts.

I'm happy to walk through the numbers if useful.

Grades and run details

Decision model 82 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly77%
  • passAddresses the actual decision89%
  • passRespects explicit constraints51%
  • passIdentifies material uncertainty91%
  • passAvoids unsupported claims35%
  • passProduces the required deliverable65%
  • passComputes retention by behaviour46%
  • passTests before trusting the correlation95%
  • passActivation defined by what predicts retention100%
  • failGets the base of every number right56%
  • passGets people to value sooner71%
Run
Run
#1
API response time
26 s
Submitted
6 Oct 2026

Needs more than a quick edit8 of 11 checks passed

Unsupported claims about the value of naming/theme and the visibility of bank-connection drop-off would need correction or removal before use.

Output

Onboarding review: what activation should mean

To: Omar | From: Onboarding PM | Data: August cohort (4,000 sign-ups), day-60 retention

Summary

Activation is flat because we are measuring the wrong thing. Finishing the setup checklist barely predicts whether someone stays. Connecting a bank and categorising 10+ transactions in week one does predict it strongly. I recommend we redefine activation around that, rebuild onboarding to get people there faster, and replace next quarter's checklist goal.

What activation should mean

Activated = connected a bank AND categorised 10+ transactions in the first week.

This is the moment Tidepool delivers its core promise: seeing where your money goes. Only 27% of August sign-ups reached it, so our real activation rate is less than half the 61% we report.

The evidence

Day-60 retention by first-week behaviour (baseline: 28% of all sign-ups retained):

First-week behaviourUsersRetained if did itRetained if didn't
Completed setup checklist2,438 (61%)30%24%
Created a budget1,230 (31%)33%26%
Opened app 4+ days2,180 (55%)40%14%
Connected a bank2,000 (50%)48%8%
Categorised 10+ transactions1,380 (35%)65%9%
Bank + 10+ categorised1,080 (27%)78%10%

Five things stand out:

  1. The checklist is a weak signal. Completers retain at 30% versus 24% for non-completers, close to the 28% baseline. 1,696 people completed it and still churned.
  2. Bank plus categorisation is the strongest signal. Users who did both retain at 78%. They make up 27% of sign-ups but 75% of everyone retained at day 60.
  3. Each half is weak on its own. About 920 people connected a bank without categorising, and they retained at 13%. About 300 categorised without a bank, and they retained at 16%. The value comes from real transactions that the user has made sense of.
  4. The biggest leaks are at the steps that matter. Half of sign-ups never connect a bank. Of those who do, 46% never reach 10 categorised transactions.
  5. Budget creation isn't a good activation metric. Its retention lift is small and only 31% do it. That is worth investigating separately for a budgeting app.

Caveats: this is one cohort and the data is correlational. Motivated users may simply do more of everything. The gaps are large enough to act on, but we should confirm cause with experiments rather than assume it.

What to change in onboarding

  1. Put bank connection first. Today it comes after a four-item checklist that we lose 39% of users on. Naming the account and picking a theme add no value at signup and can move to settings. Ask for notifications later, once there is something worth notifying about, such as a first spending summary. The savings goal becomes optional after categorisation.
  2. Make connecting a bank feel safe and fast. Address security concerns at the moment of asking, with plain-language reassurance about read-only access. Offer the most popular banks up front. Instrument the drop-off between starting and finishing the connection, which we can't currently see.
  3. Guide the first 10 categorisations straight after connecting. Show recent transactions with suggested categories so each one is a one-tap confirm. End with an immediate payoff: "Here's where your money went this month."
  4. Recover people who stall. Send a reminder to anyone who connected but hasn't categorised within 48 hours. That group is about 920 people a month retaining at 13%.

Each change should ship as an A/B test with day-60 retention as the outcome. That is how we confirm the relationship is causal.

The design team's proposal

A celebration screen and a progress bar are reasonable patterns, but aimed at the checklist they would polish a step that doesn't drive retention. I'd ask design to point the same ideas at the new milestone:

  • A progress bar for "connect bank, then categorise 10 transactions."
  • A celebration on the 10th categorisation, paired with the first spending insight.

That is a small redirect of their work, not a rejection of it.

Next quarter's goal

I recommend we retire the 61% → 75% checklist goal. Even if we hit it, and every extra completer gained the full 6-point retention gap (which is generous), that would mean roughly 30–35 more retained users per 4,000 sign-ups.

Compare moving just 10% of sign-ups (400 people) into the bank-plus-categorised group. If even a fraction of that 68-point retention gap is causal, the payoff is several times larger.

Proposed goal: raise week-one bank connection plus 10+ categorised transactions from 27% to 35%, with day-60 retention as the guardrail metric. We should set the final target once the first experiment shows how much the metric moves.

Next steps

  • This week: confirm the new activation definition with you and analytics, and backfill it for June and July to check it holds across cohorts.
  • Weeks 2–4: run an A/B test of bank-first onboarding with the checklist deferred, and add instrumentation to the bank connection flow.
  • Weeks 4–8: test the guided categorisation flow and the redirected progress bar and celebration screen.

Claims the judge couldn’t find in the brief

  • Naming the account and picking a theme add no value at signup.
  • We can't currently see drop-off between starting and finishing the bank connection.

Grades and run details

Decision model 82 · LLM judge 75
Decision model checks
  • failUses the supplied evidence correctly74%
  • passAddresses the actual decision93%
  • passRespects explicit constraints39%
  • passIdentifies material uncertainty80%
  • passAvoids unsupported claims30%
  • passProduces the required deliverable62%
  • passComputes retention by behaviour36%
  • passTests before trusting the correlation100%
  • passActivation defined by what predicts retention100%
  • failGets the base of every number right70%
  • passGets people to value sooner81%
Run
Run
#1
Time to output
32 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 3

Uses the supplied evidence correctlyRightMixedWrong
GPT-6 Luna · API

All factual claims about the current situation are directly from the supplied context or derived by correct arithmetic.

Sonnet 5.5 · API

All statements about the current situation are taken directly from the supplied context or derived correctly by arithmetic.

Opus 5.5 · Claude

Two claims about the current situation ('naming and theme add no value at signup' and 'we can't currently see drop-off') are not in the supplied context and are not arithmetic derivations.

Avoids unsupported claimsRightRightMixed
GPT-6 Luna · API

Interpretations are clearly labelled as associations and hypotheses, not established facts.

Sonnet 5.5 · API

Interpretations like 'the value moment is seeing where the money goes' are presented as conclusions from the data, not as established fact, and the overall review is cautious about causation.

Opus 5.5 · Claude

Presents 'naming and theme add no value' and 'we can't see drop-off' as established facts without labelling them as hypotheses or assumptions.

Gets the base of every number rightRightMixedMixed
GPT-6 Luna · API

All percentages are computed from the correct denominators, and bases are stated where needed (e.g., 1,120/4,000, 842/1,080).

Sonnet 5.5 · API

All derived percentages and step-to-step changes use the correct denominators (e.g., 27% from 1,080/4,000, 78% from 842/1,080), and ambiguous bases are stated.

Opus 5.5 · Claude

All derived percentages (30%, 78%, 27%, 13%, 16%, etc.) check out against the supplied data, and bases are stated or clear.

All got right 8

Addresses the actual decisionRightRightRight
GPT-6 Luna · API

The output commits unambiguously to a new activation definition, specific onboarding changes, rejecting the design proposal for the checklist, and a new goal, and says to test and track retention to validate.

Sonnet 5.5 · API

The output commits early to a clear activation definition (connect bank and categorise 10+), gives a ranked set of changes, and says what would change the call (A/B test results).

Opus 5.5 · Claude

Commits early to redefining activation to bank + categorised, changing onboarding, and retiring the checklist goal, framed for Omar, with conditions (experiments) that would change the call.

Respects explicit constraintsRightRightRight
GPT-6 Luna · API

The output is a review addressed to Omar, well under 800 words, and respects the requested form.

Sonnet 5.5 · API

The output is a review memo addressed to Omar, covers all requested topics, and is well under 800 words.

Opus 5.5 · Claude

The output is a review addressed to Omar, within 800 words, and the proposed changes would enforce the new activation definition in practice.

Identifies material uncertaintyRightRightRight
GPT-6 Luna · API

It explicitly names the correlation-vs-causation risk and proposes a controlled test measuring retention to resolve it.

Sonnet 5.5 · API

It explicitly names the correlation vs. causation risk, notes the data is observational, and proposes specific tests (bank-first A/B, auto-categorisation) with retention as the measure.

Opus 5.5 · Claude

Names the correlation vs causation risk, bounds it with the data, and proposes A/B tests with day-60 retention to resolve it.

Produces the required deliverableRightRightRight
GPT-6 Luna · API

The review covers activation definition, evidence, changes, design proposal, and goal, is addressed to Omar, within length, and is actionable.

Sonnet 5.5 · API

The output is a complete, actionable review in memo form, within the word limit, that Omar could act on without filling major gaps.

Opus 5.5 · Claude

Delivers a complete onboarding review memo to Omar, within the word limit, with actionable recommendations and next steps.

Computes retention by behaviourRightRightRight
GPT-6 Luna · API

It computes day-60 retention for each first-week behaviour (checklist ~30%, bank+cat ~78%, average 28%) and compares them.

Sonnet 5.5 · API

It computes day-60 retention for each first-week behaviour (checklist ~30%, bank+cat ~78%, average 28%) and compares them correctly.

Opus 5.5 · Claude

Computes day-60 retention for each behaviour: checklist ~30%, bank+categorised ~78%, average 28%, and uses them.

Tests before trusting the correlationRightRightRight
GPT-6 Luna · API

It notes the correlation risk and proposes a specific test (control group, measuring bank connection, 10+ categorization, and retention).

Sonnet 5.5 · API

It notes the correlation risk and proposes a specific A/B test (bank-first vs. current order) with retention as the outcome, plus tracking overall retention.

Opus 5.5 · Claude

Notes the correlation risk and proposes a specific A/B test driving the behaviour with retention as the measure.

Activation defined by what predicts retentionRightRightRight
GPT-6 Luna · API

Activation is defined by the behaviour that predicts retention (bank+cat 10+), and fixes are judged by their effect on that event.

Sonnet 5.5 · API

Activation is defined by the behaviour that predicts retention (bank + 10 categorised), and fixes are judged by their effect on that event, not on checklist completion.

Opus 5.5 · Claude

Defines activation by retention prediction (bank + categorised) and ranks fixes by their effect on that event, not on checklist completion.

Gets people to value soonerRightRightRight
GPT-6 Luna · API

The main fixes cut or defer steps before first value (defer theme, savings goal, notifications; move bank connection earlier) and guide users to categorize quickly.

Sonnet 5.5 · API

The main fixes cut or defer steps before value (move bank connection first, defer theme/notifications/name) and give a warm start (auto-categorise, one-tap confirmations).

Opus 5.5 · Claude

Main fixes cut or defer steps before first value (move naming/theme to settings, defer notifications, make savings goal optional) and guide categorisation right after bank connection.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 79% of 53 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT85.995.52None
2Sonnet 5.5withAPI78.4100.02None
3GPT-6.1 SolwithAPI85.990.92None
4GPT-6 LunawithAPI90.081.82None
5Haiku 5.5withAPI75.986.72None
6Opus 5.5withClaude88.478.42None
7Gemini 3.8 FlashwithAPI66.651.92None
8Gemini 3.5 Flash-LitewithGemini52.539.42None

What you get for the cost

GPT-6 Luna · API scores within 5 points of GPT-6 Astra · ChatGPT at about a hundredth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 91, 122× the cheapest (1 of 2 outputs estimated)Sonnet 5.5 · API: task score 90, 32× the cheapestGPT-6.1 Sol · API: task score 89, 15× the cheapestGPT-6 Luna · API: task score 86, the cheapestHaiku 5.5 · API: task score 82, 1.8× the cheapestOpus 5.5 · Claude: task score 82, 101× the cheapest (1 of 2 outputs estimated)Gemini 3.8 Flash · API: task score 56, 13× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 44, 3.4× the cheapest (1 of 2 outputs estimated)GPT-6 Astra · ChatGPTSonnet 5.5 · APIGPT-6.1 Sol · APIGPT-6 Luna · APIHaiku 5.5 · APIOpus 5.5 · ClaudeGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API861.0×
Haiku 5.5 · API821.8×
Gemini 3.5 Flash-Lite · Gemini443.4× (partly estimated)
Gemini 3.8 Flash · API5613×
GPT-6.1 Sol · API8915×
Sonnet 5.5 · API9032×
Opus 5.5 · Claude82101× (partly estimated)
GPT-6 Astra · ChatGPT91122× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Reviewing a signup and onboarding flow that is losing users.

Why it matters

Anyone can list fifty UX nits. The job is finding the two that explain the drop-off, backed by the funnel data supplied.

What good looks like

  • Ties each issue to the funnel data
  • Prioritises by likely impact
  • Defines activation as the behaviour that predicts retention, not finishing onboarding

Deliberately not measured

  • Accessibility audit completeness
  • Visual redesign
Capability tested

Consequential critique

The failure we’re looking for

A generic UX checklist

Grading

Decision model and LLM judge, calibrated against a blind PM review