Tasks / Growth

Activation & onboarding review

Can the model find the friction that matters most and prioritise the fixes?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 67% were usable with at most a quick edit.

Reliably right

  1. Respects explicit constraints100% pass
    The output is a review addressed to Omar, well under 800 words, and respects the requested form.
    GPT-6 Luna · API · The checklist isn't the moment
  2. Produces the required deliverable100% pass
    The review covers activation definition, evidence, changes, design proposal, and goal, is addressed to Omar, within length, and is actionable.
    GPT-6 Luna · API · The checklist isn't the moment
  3. Activation defined by what predicts retention100% pass
    Activation is defined by the behaviour that predicts retention (bank+cat 10+), and fixes are judged by their effect on that event.
    GPT-6 Luna · API · The checklist isn't the moment

Where it slips

  1. Avoids unsupported claims26% pass
    States 'App-open frequency is an outcome, not a cause' as an established fact, but only correlation is shown in the data; this causal direction is an unsupported interpretation.
    Haiku 5.5 · API · The checklist isn't the moment
  2. Identifies material uncertainty61% pass
    Does not name specific unknowns that could change the decision, only suggests tracking metrics before expanding.
    GPT-6 Luna · API · Analytics tool losing users at setup
  3. Uses the supplied evidence correctly64% pass
    The output states 'we have no step-level data for screens 2–8', 'a blank builder is hard for non-analysts', and 'people invite colleagues to something useful, not to a setup screen' as facts, none of which appear in the brief or follow from arithmetic.
    Opus 5.5 · Claude · Analytics tool losing users at setup

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Activation defined by what predicts retention

    Does the review define (or question) the activation event by whether it predicts retention, and judge fixes by their effect on that event rather than on step completion?

    Passes when Names the activation behaviour and its link to retention (from the supplied data, or as an explicit assumption to verify), and ranks fixes by their effect on it.

  2. Gets people to value sooner

    Do the recommended fixes shorten the path to the first moment of value, by removing or deferring steps before it or starting people with templates or sample data, rather than adding tours, tooltips or extra steps?

    Passes when The main fixes cut or defer steps before first value, or give a warm start, and say which steps go.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the PM for onboarding at Tidepool. Omar, our Head of Product, has asked for a review of onboarding (his note is below). Write it in no more than 800 words: what activation should mean for us, the evidence, what to change in onboarding, and what to make of the design team's proposal and next quarter's goal. The files are attached.

What the model was given4 items: About Tidepool, omar_note.md, onboarding_today.md, first_week_behaviour.csv (August sign-ups: how many who did each thing in their first week were still active at day 60)
About TidepoolA personal finance app: connect your bank, see where your money goes, and budget.
omar_note.md1 lines · Download
From Omar (Head of Product): "Activation is flat at 61%. Review our onboarding and tell me what to fix. The design team has proposed a celebration screen and a progress bar for the checklist."
onboarding_today.md8 lines · Download
# Onboarding today (for 4,000 sign-ups in August)

1. Sign up (100%)
2. Setup checklist: name your account, pick a theme, set a savings goal, enable notifications (61% complete it)
3. Connect a bank (50%)
4. Categorise transactions (34% reach 10)

…
first_week_behaviour.csv (August sign-ups: how many who did each thing in their first week were still active at day 60)first_week_behaviour,retained_at_day_60,churned_by_day_60 All sign-ups,1120,2880 Completed the setup checklist,742,1696 Connected a bank account,964,1036 Categorised 10+ transactions,891,489 Connected a bank AND categorised 10+ transactions,842,238 Created a budget,410,820 Opened the app 4+ days,870,1310
What a strong answer doesThe answer key the graders mark against

Works out retention for each first-week behaviour from the file: all sign-ups 28% (1,120 of 4,000); completed the checklist about 30% (barely better than average); connected a bank about 48%; categorised 10+ transactions about 65%; connected a bank and categorised 10+ about 78%, and that group is about three-quarters of everyone retained (842 of 1,120). So the checklist isn't activation and raising its completion to 75% would do little; activation should be connecting a bank and categorising ten transactions in week one. Recommends cutting or deferring checklist steps that sit before that (theme, notifications, savings goal) and getting people to connect a bank and categorise sooner (for example suggested categories to confirm), and resets the goal to the new activation rate (about 27% today, 1,080 of 4,000). Treats the celebration screen and progress bar as polishing the wrong step. Notes that the data shows correlation and proposes a test to confirm that driving the behaviour improves retention.

Critical failures (cap the score at 40)
  • Recommends raising checklist completion as the main goal
Task

Staff-level · v1.1

The outputs

9 setups answered this brief.

Needs more than a quick edit8 of 11 checks passed

The output lacks an explicit correlation-causation caveat and a proposed test, which a PM would need to add before it could be acted upon confidently.

Output

To: Omar (Head of Product) From: PM, Onboarding Subject: Onboarding Review: Redefining Activation and Q4 Strategy

Executive Summary

Our current activation metric—completing the setup checklist (61%)—is a vanity metric. Analysis of August’s 4,000 sign-ups shows that users who complete the checklist retain at Day 60 at roughly the same rate as the overall average (30.4% vs. 28.0%). Worse, the checklist sits before our core value proposition, causing 39% of users to drop off before we even ask them to connect a bank.

We need to redefine activation around real product value, scrap the current Q4 checklist goal, and redesign onboarding to get users straight to their financial data.

---

1. The Evidence: What Actually Drives Retention?

Activation should measure the moment a user experiences Tidepool’s core value ("see where your money goes") in a way that predicts long-term habit formation. Looking at Day 60 retention from August’s cohort (4,000 total sign-ups), the data is clear:

  • All sign-ups: 28.0% D60 retention (1,120 / 4,000)
  • Completed setup checklist (61% of users): 30.4% retention (742 / 2,438)
  • Created a budget (31% of users): 33.3% retention (410 / 1,230)
  • Opened the app 4+ days (55% of users): 39.9% retention (870 / 2,180)
  • Connected a bank account (50% of users): 48.2% retention (964 / 2,000)
  • Categorised 10+ transactions (34% of users): 64.6% retention (891 / 1,380)
  • Connected a bank AND categorised 10+ transactions (27% of users): 78.0% retention (842 / 1,080)

Key Takeaways from the Data: 1. The setup checklist does not predict retention. Doing administrative chores (naming an account, picking a color theme) does not deliver financial clarity. Users who skip or abandon the checklist aren't failing to activate; they are likely just bored. 2. Core value is delivered through bank connection and transaction categorisation. When a user connects their bank and categorises 10+ transactions in week one, their Day 60 retention jumps to 78.0%. Notably, this 27% of users accounts for 75% of all retained users at Day 60 (842 of 1,120). 3. Crucial caveat on Bank vs. Manual: Notice that 1,380 users categorised 10+ transactions, but only 1,080 did so and connected a bank. For the 300 users who categorised 10+ transactions without connecting a bank (presumably via manual entry or CSV), D60 retention was only 16.3% (49 / 300). Categorising alone isn’t enough; automated bank syncing is essential.

---

2. Redefining Activation & Next Quarter’s Goal

  • New Activation Definition: "Connected a bank account AND categorised 10+ transactions within 7 days of sign-up."
  • Verdict on Q4 Goal: We must cancel the goal of raising checklist completion from 61% to 75%. Even if we hit 75%, D60 retention wouldn't budge.
  • New Q4 Goal: Increase true activation (Bank + 10 Categorised Transactions) from 27.0% to 38.0%. Because this cohort retains at 78% (though we should monitor for slight dilution as we push less-motivated users through the funnel), moving this needle is the highest-leverage way to lift overall D60 retention above 28%.

---

3. Assessment of the Design Team's Proposal

The design team proposed adding a progress bar and a celebration screen to the setup checklist. We should not build this for the current checklist.

Optimising a step that adds friction without adding value is wasted effort. However, we should repurpose the design team’s UI patterns for the Categorisation step. A progress bar ("4 of 10 transactions categorised to unlock your spending breakdown") and a celebration screen that immediately reveals their visualised spending insights will directly drive our new activation metric.

---

4. What to Change in Onboarding

Right now, our funnel is backwards: we ask for customization before showing value. We should restructure the flow as follows:

  1. Kill the upfront setup checklist: Remove "name your account" and "pick a theme" from onboarding entirely (default these and move them to Settings). Delay "set a savings goal" and "enable notifications" until after the user has seen their spending data, when those prompts actually have context.
  2. Make "Connect a Bank" Step 1: Immediately after sign-up, prompt bank connection. Currently, we lose 50% of users here (many because they dropped off at the checklist first). Moving this to Step 1—and adding trust signals around security and read-only access—will immediately increase the top of our true value funnel.
  3. Guided "First 10" Categorisation Flow: Once the bank syncs, drop users into a focused, swipe/tap experience (using Design's progress bar) to categorise their 10 most recent transactions, culminating in an immediate "Aha!" dashboard showing where their money went this month.

Grades and run details

Decision model 77 · LLM judge 67
Decision model checks
  • passUses the supplied evidence correctly25%
  • passAddresses the actual decision49%
  • passRespects explicit constraints55%
  • partialIdentifies material uncertainty80%
  • failAvoids unsupported claims82%
  • passProduces the required deliverable77%
  • passComputes retention by behaviour72%
  • failTests before trusting the correlation82%
  • passActivation defined by what predicts retention100%
  • passGets the base of every number right20%
  • passGets people to value sooner91%
Run
Run
#1
API response time
12 s
Submitted
13 Oct 2026

Usable with a quick edit10 of 11 checks passed

The memo is structured, evidence-based, and a PM could act on it with a quick edit to soften the unsupported causation claim about app opens.

Output

To: Omar From: PM, Onboarding Re: Onboarding review

Summary

Our activation metric is measuring the wrong thing. Completing the setup checklist is 61% of sign-ups, but it barely separates users who stay from those who leave: 30% of completers are retained at day 60, against 24% of non-completers. The behaviours that matter are connecting a bank account and categorising transactions. Users who do both are retained at 78%. Users who do neither are retained at 6%. I recommend redefining activation, putting bank connection at the centre of onboarding, and reshaping the design proposal around it.

What activation should mean

Activation = connected a bank account and categorised 10+ transactions in the first 7 days.

Baseline: 27% of August sign-ups (1,080 of 4,000). This definition reflects the core value (seeing where money goes), is the only combination with a large retention gap, and is measurable from event data.

Evidence

First-week behaviourShare of sign-upsRetained at day 60
Completed checklist61%30%
Did not complete checklist39%24%
Connected a bank50%48%
Did not connect a bank50%8%
Categorised 10+35%65%
Did not categorise 10+65%9%
Bank and categorised27%78%
Bank only23%13%
Categorised only8%16%
Neither43%6%
Opened app on 4+ days55%40%
Created a budget31%33%

Four things stand out:

  • Bank connection and categorising are complementary. Each alone yields 13–16% retention; together they yield 78%. The value appears only when both happen.
  • The biggest fixable leak is after bank connection. 46% of users who connect a bank never categorise 10 transactions.
  • Budget creation is weak in week one. Its retention (33%) is barely above the 28% baseline, so it shouldn't define activation yet.
  • App-open frequency is an outcome, not a cause. It's a useful engagement metric, but users who are already activated open the app more.

These are correlations. Users who do more are probably more motivated to begin with, so the 78% overstates the causal effect. We need experiments to confirm the lift, but the gap is large enough to act on now.

What to change in onboarding

  1. Make bank connection the first real step. Only half of sign-ups connect, and the cosmetic steps (name, theme) come first. Move bank linking up, explain the payoff ("see where last month went"), and reduce friction: clear read-only reassurance, and recovery paths for failed connections.
  2. Help users categorise 10 transactions quickly. Pre-suggest categories, allow one-tap confirmation, and show progress ("3 of 10: your spending is taking shape"). Target the 46% who stop short.
  3. Demote checklist items that don't carry the value. Theme, name, and notifications should be optional. The savings goal is untested in our data, so test it before cutting it.
  4. Move budget creation later. Prompt it after the user has categorised 10 transactions, and measure whether that lifts budget creation and retention in weeks 2–4.

The design team's proposal

The celebration screen and progress bar are good craft aimed at the wrong target. A progress bar on the current checklist will push users through theme and notifications faster. That raises the 61% number without touching the steps that predict retention, and may do nothing for bank connection.

The celebration is valuable, but only at a milestone that matters. I'd ask the design team to rebuild the progress bar around the two activation steps (bank connected, 10 transactions categorised), with the celebration triggered when both are done. We test it against the current flow and ship only if activation moves.

Next quarter's goal

Replace "checklist completion 61% → 75%" with "activation (bank connected and 10+ categorised in week one) 27% → 35%."

Sizing: in a 4,000-user cohort, 27% → 35% is about 320 more activated users. At 78% retention, that's roughly 250 more retained at day 60. A checklist point, by contrast, is worth about 10 retained users at current completer retention. Marginal users will retain worse than average, so treat 250 as an upper bound.

35% is ambitious but reachable: it requires bank connection to rise from 50% to about 60%, and categorising conversion from 54% to about 58%. The checklist stays as a leading indicator, not the goal.

Next steps

  1. Adopt the new activation definition this week and add it to the onboarding dashboard.
  2. Run an A/B test of bank-first onboarding against the current flow.
  3. Ask design to rebuild the progress bar and celebration around the activation steps, and test before shipping.
  4. Confirm the retention lift with a holdout or experiment, since current figures are correlational.

Grades and run details

Decision model 91 · LLM judge 92
Decision model checks
  • passUses the supplied evidence correctly14%
  • passAddresses the actual decision92%
  • passRespects explicit constraints62%
  • passIdentifies material uncertainty69%
  • failAvoids unsupported claims30%
  • passProduces the required deliverable92%
  • passComputes retention by behaviour62%
  • passTests before trusting the correlation100%
  • passActivation defined by what predicts retention100%
  • passGets the base of every number right18%
  • passGets people to value sooner86%
Run
Run
#1
API response time
29 s
Submitted
8 Oct 2026

Usable with a quick edit9 of 11 checks passed

The review is data-driven, clearly structured, and makes a strong, actionable recommendation. A PM could use it with minor wording tweaks and no need to re-verify evidence or reasoning.

Output

Onboarding review for Omar

Recommendation

We are measuring setup, not activation. Completing a checklist of preferences does not establish that someone has experienced Tidepool’s value: seeing where their money goes and taking control of it.

Redesign onboarding around that first useful financial insight. Treat the celebration screen and progress bar as supporting elements—not the main fix—and replace next quarter’s checklist-completion goal.

What activation should mean

Use “connected a bank and categorised at least 10 transactions within the first week” as our provisional activation definition.

This combines access to real financial data with an action that helps users understand their spending. It is a stronger approximation of delivered value than naming an account or picking a theme.

The threshold of 10 is a starting hypothesis, not a proven magic number. We should validate whether users actually gain an insight at this point and whether a different threshold or milestone—such as reviewing a spending summary—better predicts durable use. Creating a budget may become a subsequent milestone, but the current evidence does not support making it mandatory for initial activation.

What the evidence says

Among August’s 4,000 sign-ups, overall day-60 retention was 28%.

First-week behaviourUsersDay-60 retention
------:---:
Completed setup checklist2,43830.4%
Connected a bank2,00048.2%
Categorised 10+ transactions1,38064.6%
Connected a bank and categorised 10+1,08078.0%
Created a budget1,23033.3%
Opened the app on 4+ days2,18039.9%

The strongest signal is the combined bank-and-categorisation milestone: 27% of sign-ups reached it, and those users accounted for 842 of our 1,120 retained users. Retention among everyone else was approximately 9.5%.

Checklist completion, by contrast, is associated with retention only slightly above the cohort average. Raising it could improve the reported activation rate without improving users’ outcomes.

These are observational, overlapping groups. Motivated users may both complete meaningful actions and remain active; the data does not prove that pushing everyone through those actions will cause retention. Nor can we infer exact step-by-step drop-off from these aggregates. The CSV also differs slightly from the funnel summary on checklist and categorisation counts; reconcile reporting before setting the baseline.

What to change in onboarding

  1. Lead with the value proposition, then bank connection. Explain what users will get from connecting, how their data is handled, and what happens next. Instrument connection starts, failures, abandonment and successful imports; address reliability and trust barriers.
  1. Replace the upfront preference checklist with a guided first-value journey. Move theme selection and account naming out of the critical path. Ask for notification permission when there is a clear benefit. Keep savings goals optional or introduce them after users understand their finances.
  1. Guide users through real transactions. Provide sensible category suggestions, easy corrections and a short review flow. Then show a useful spending summary—not merely confirmation that ten items were completed.
  1. Offer budgeting as the next step. Help users turn their spending insight into a realistic budget, without making budget creation another prerequisite that delays first value.

Measure the full journey and test changes against the existing experience, including first-week milestone attainment and day-60 retention.

Design proposal and next quarter’s goal

A progress bar is worth testing if it tracks meaningful value steps, not cosmetic setup tasks. A celebration screen belongs after the first useful insight, should be brief, and should not interrupt it. Neither proposal addresses the core problem on its own.

Replace “raise checklist completion from 61% to 75%” with “increase first-week bank-plus-10-transaction activation.” A provisional target of 27% to 35% is reasonable to plan against, subject to baseline reconciliation and experiment sizing—not a forecast supported by this dataset.

Success must also include improved day-60 retention, with bank-connection reliability, privacy complaints and abandonment monitored as guardrails. Keep checklist completion as a diagnostic metric, not the team’s definition of success.

Grades and run details

Decision model 77 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly77%
  • passAddresses the actual decision62%
  • passRespects explicit constraints37%
  • passIdentifies material uncertainty80%
  • partialAvoids unsupported claims40%
  • passProduces the required deliverable40%
  • passComputes retention by behaviour23%
  • passTests before trusting the correlation76%
  • passActivation defined by what predicts retention100%
  • failGets the base of every number right80%
  • passGets people to value sooner76%
Run
Run
#1
API response time
26 s
Submitted
6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 5

Uses the supplied evidence correctlyRightRightMixed
Gemini 4 Argon · API

All factual claims about current numbers and performance are taken correctly from the supplied context or follow from arithmetic.

Haiku 5.5 · API

Every factual statement about the current situation is taken from the supplied context or follows from correct arithmetic.

GPT-6.1 Sol · API

All claims about the current situation are directly from the supplied context or derived by correct arithmetic.

Identifies material uncertaintyWrongRightRight
Gemini 4 Argon · API

Does not name the key uncertainty (correlation vs causation) or say how it would be resolved; only mentions dilution risk without a resolution plan.

Haiku 5.5 · API

Explicitly notes that retention figures are correlations and proposes A/B tests and a holdout to resolve the causal uncertainty.

GPT-6.1 Sol · API

Identifies the unproven threshold, the correlation-causation risk, and data discrepancies, and says how to resolve each (validate with users, run a test, reconcile reporting).

Avoids unsupported claimsWrongWrongRight
Gemini 4 Argon · API

Presents cause-effect interpretations ('causing 39% to drop off', 'users are likely just bored') as fact, without labelling them as hypotheses.

Haiku 5.5 · API

States 'App-open frequency is an outcome, not a cause' as an established fact, but only correlation is shown in the data; this causal direction is an unsupported interpretation.

GPT-6.1 Sol · API

Hypotheses and limitations are clearly labelled; no interpretation is presented as established fact without support.

Tests before trusting the correlationWrongRightRight
Gemini 4 Argon · API

Does not note that the retention links are correlations, nor propose a test to drive the activation behaviour and check retention.

Haiku 5.5 · API

Notes correlation risk and proposes a specific A/B test of bank-first onboarding and a holdout experiment with retention as the measure.

GPT-6.1 Sol · API

Explicitly notes the observational nature and correlation risk, and proposes testing onboarding changes with first-week milestone attainment and day-60 retention as measures.

Gets the base of every number rightRightRightMixed
Gemini 4 Argon · API

All percentages and rates are correctly derived from the supplied data, and denominators are clear (e.g., retention rates use the behaviour cohort as base).

Haiku 5.5 · API

All derived percentages (24%, 30%, 78%, 6%, 48%, 13%, 16%, etc.) use the correct denominators, with no ambiguous bases.

GPT-6.1 Sol · API

All percentages use the correct denominators (e.g., 1080/4000 for 27%, 842/1080 for 78.0%, 278/2920 for 9.5%) and the base is clear from context.

All got right 6

Addresses the actual decisionRightRightRight
Gemini 4 Argon · API

Output commits early and unambiguously to redefining activation as bank + 10 categorised and abandoning the checklist goal.

Haiku 5.5 · API

Commits to a clear activation definition, specific onboarding changes, and a verdict on the design proposal and next quarter's goal, with conditions for change.

GPT-6.1 Sol · API

Commits early to a new activation definition and onboarding changes, framed for Omar, and states that validation of the threshold and a test would change the call.

Respects explicit constraintsRightRightRight
Gemini 4 Argon · API

Output is a memo to Omar, under 800 words, covers the requested components, and respects the form and reader.

Haiku 5.5 · API

Output is a memo to Omar, addresses all parts of the brief, and is well within the 800-word limit.

GPT-6.1 Sol · API

The output is a review under 800 words, addresses all requested parts, and respects the reader and the brief's form.

Produces the required deliverableRightRightRight
Gemini 4 Argon · API

Produced a complete, usable memo within the length limit, addressed to Omar as a review.

Haiku 5.5 · API

Memo format, addressed to Omar, includes activation definition, evidence, changes, design proposal view, and next quarter goal, all under the word limit.

GPT-6.1 Sol · API

The review is complete, in the right form for Omar, within the word limit, and actionable with light edits.

Computes retention by behaviourRightRightRight
Gemini 4 Argon · API

Correctly computed Day-60 retention for each behaviour, including 28% overall, 30.4% checklist, 48.2% bank, 64.6% categorise, 78.0% both.

Haiku 5.5 · API

Computes retention for each first-week behaviour including checklist (≈30%), bank+cat (78%), and baseline (28%), and compares them.

GPT-6.1 Sol · API

Day-60 retention rates are computed for each first-week behaviour (30.4%, 48.2%, 64.6%, 78.0%, etc.) and compared with the 28% average.

Activation defined by what predicts retentionRightRightRight
Gemini 4 Argon · API

Defines activation by retention prediction (bank + 10 categorised) and ranks fixes by their effect on that event.

Haiku 5.5 · API

Defines activation by retention (bank+cat 78%), not checklist; ranks fixes by their effect on that activation behaviour.

GPT-6.1 Sol · API

Defines activation as bank-and-categorise based on its strong retention signal, and judges fixes by their ability to drive that event.

Gets people to value soonerRightRightRight
Gemini 4 Argon · API

Recommends removing upfront checklist steps and making bank connection step 1, directly shortening the path to first value.

Haiku 5.5 · API

Recommended fixes move bank connection earlier, pre-suggest categories, and demote or defer theme/name/notifications, cutting steps before value.

GPT-6.1 Sol · API

Recommends cutting or deferring theme, account naming, notifications, and savings goals, and guiding users through transactions with suggestions, shortening the path to first value.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 72% of 53 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5withAPI83.6100.02None
2GPT-6 AstrawithChatGPT86.195.52None
3GPT-6.1 SolwithAPI83.690.92None
4Haiku 5.5withAPI88.086.72None
5GPT-6 LunawithAPI92.781.82None
6Opus 5.5withClaude81.478.42None
7Gemini 4 ArgonwithAPI73.665.22None
8Gemini 3.8 FlashwithAPI71.451.92None
9Gemini 3.5 Flash-LitewithGemini60.039.42None

What you get for the cost

GPT-6 Luna · API scores within 5 points of Sonnet 5.5 · API at about a thirtieth of the cost.

Task score against relative cost for each setup. The table below has the same figures.4050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreSonnet 5.5 · API: task score 93, 32× the cheapestGPT-6 Astra · ChatGPT: task score 91, 122× the cheapest (1 of 2 outputs estimated)GPT-6.1 Sol · API: task score 88, 15× the cheapestHaiku 5.5 · API: task score 88, 1.8× the cheapestGPT-6 Luna · API: task score 88, the cheapestOpus 5.5 · Claude: task score 79, 101× the cheapest (1 of 2 outputs estimated)Gemini 4 Argon · API: task score 68, 15× the cheapestGemini 3.8 Flash · API: task score 59, 13× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 47, 3.4× the cheapest (1 of 2 outputs estimated)Sonnet 5.5 · APIGPT-6 Astra · ChatGPTGPT-6.1 Sol · APIHaiku 5.5 · APIGPT-6 Luna · APIOpus 5.5 · ClaudeGemini 4 Argon · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API881.0×
Haiku 5.5 · API881.8×
Gemini 3.5 Flash-Lite · Gemini473.4× (partly estimated)
Gemini 3.8 Flash · API5913×
GPT-6.1 Sol · API8815×
Gemini 4 Argon · API6815×
Sonnet 5.5 · API9332×
Opus 5.5 · Claude79101× (partly estimated)
GPT-6 Astra · ChatGPT91122× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Reviewing a signup and onboarding flow that is losing users.

Why it matters

Anyone can list fifty UX nits. The job is finding the two that explain the drop-off, backed by the funnel data supplied.

What good looks like

  • Ties each issue to the funnel data
  • Prioritises by likely impact
  • Defines activation as the behaviour that predicts retention, not finishing onboarding

Deliberately not measured

  • Accessibility audit completeness
  • Visual redesign
Capability tested

Consequential critique

The failure we’re looking for

A generic UX checklist

Grading

Decision model and LLM judge, calibrated against a blind PM review