Tasks / Growth

Activation & onboarding review

Can the model find the friction that matters most and prioritise the fixes?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 67% were usable with at most a quick edit.

Reliably right

  1. Respects explicit constraints100% pass
    The output is a review addressed to Omar, well under 800 words, and respects the requested form.
    GPT-6 Luna · API · The checklist isn't the moment
  2. Produces the required deliverable100% pass
    The review covers activation definition, evidence, changes, design proposal, and goal, is addressed to Omar, within length, and is actionable.
    GPT-6 Luna · API · The checklist isn't the moment
  3. Activation defined by what predicts retention100% pass
    Activation is defined by the behaviour that predicts retention (bank+cat 10+), and fixes are judged by their effect on that event.
    GPT-6 Luna · API · The checklist isn't the moment

Where it slips

  1. Avoids unsupported claims26% pass
    States 'App-open frequency is an outcome, not a cause' as an established fact, but only correlation is shown in the data; this causal direction is an unsupported interpretation.
    Haiku 5.5 · API · The checklist isn't the moment
  2. Identifies material uncertainty61% pass
    Does not name specific unknowns that could change the decision, only suggests tracking metrics before expanding.
    GPT-6 Luna · API · Analytics tool losing users at setup
  3. Uses the supplied evidence correctly64% pass
    The output states 'we have no step-level data for screens 2–8', 'a blank builder is hard for non-analysts', and 'people invite colleagues to something useful, not to a setup screen' as facts, none of which appear in the brief or follow from arithmetic.
    Opus 5.5 · Claude · Analytics tool losing users at setup

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Activation defined by what predicts retention

    Does the review define (or question) the activation event by whether it predicts retention, and judge fixes by their effect on that event rather than on step completion?

    Passes when Names the activation behaviour and its link to retention (from the supplied data, or as an explicit assumption to verify), and ranks fixes by their effect on it.

  2. Gets people to value sooner

    Do the recommended fixes shorten the path to the first moment of value, by removing or deferring steps before it or starting people with templates or sample data, rather than adding tours, tooltips or extra steps?

    Passes when The main fixes cut or defer steps before first value, or give a warm start, and say which steps go.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 1 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

Review our onboarding flow and funnel below. Write a memo to the onboarding squad saying where we are losing people and what to fix, in the order you would fix it, with the reason for each. Keep it under 500 words.

What the model was given4 items: Scenario, Funnel (last 90 days, 12,400 signups), Onboarding flow, screen by screen, Support
ScenarioChartwise is a self-serve product-analytics tool. Most people who sign up are product managers or marketers, not data engineers. Pricing starts at $49 a month after a 14-day free trial.
Funnel (last 90 days, 12,400 signups)Signup 100% → Connect data source 41% → First chart 33% → Invite teammate 9%. Week-2 retention: 62% for users who reach a first chart, 11% otherwise.
Onboarding flow, screen by screen1. Sign up: email, password, company name. 2. Verify email: the flow pauses until the link is clicked. 3. “What's your role?” (seven options). 4. “How big is your team?” (five options). 5. Connect your data: choose Postgres, BigQuery, Snowflake or CSV upload. There is no sample data and no skip. 6. Connection form: host, port, database, username, password and SSL mode. The help link opens a 14-page setup guide. 7. Choose tables: a list of every table in the database. 8. Syncing: a progress bar that can take up to 20 minutes; leaving the page cancels the sync. 9. Pick a dashboard template (twelve options). 10. First chart: a chart builder with an empty canvas. 11. Invite teammates: email fields, with a “Skip for now” link.
SupportThe most common onboarding ticket (41% of them) is some version of “I don't have the database credentials.” The next is “my sync failed” (18%).
What a strong answer doesThe answer key the graders mark against

The data-source connection step (screens 5–8) is the activation bottleneck; prioritise a sample-data or CSV-first path to a first chart, then non-admin credential handoff and a sync that survives leaving the page, ahead of cosmetic fixes.

Task

Mid-level · v1.13

The outputs

9 setups answered this brief.

Usable with a quick edit6 of 10 checks passed

The memo is well-structured, evidence-based, and actionable; a quick edit to rephrase the unsupported SSL-mode claim and add a condition-note would make it fully usable without redoing research.

Output

MEMO To: Onboarding Squad From: Product Leadership Subject: Fixing the Onboarding Funnel

Team,

Looking at the last 90 days, the story is clear: getting users to their first chart is everything. Week-2 retention jumps from 11% to 62% once a user builds a chart.

Right now, we are losing 59% of signups before they ever connect data, and another 8% before they build that first chart. Why? Because our signups are PMs and marketers, yet we are forcing them through a data-engineer onboarding flow.

Here is where we are losing people and what we need to fix, in priority order:

1. Add a “Send to an Engineer” flow & Sample Data (Screens 5–6) * The Problem: 59% of users drop off before connecting data, and 41% of support tickets are “I don’t have database credentials.” PMs and marketers don't have Postgres passwords or understand SSL modes, and a 14-page guide won't help them. Because there is no skip option, they hit a hard wall. * The Fix: Add two buttons to Screen 5: “Explore with Sample Data” and “Invite an Engineer to connect data.” Let the PM experience the "aha" moment immediately with a sandbox dataset while their engineer handles the credentials in parallel. This also instantly fixes our low 9% teammate-invite rate by giving users a real reason to invite a colleague early.

2. Make data syncing asynchronous (Screen 8) * The Problem: We lose 8% of our total signups (nearly 20% of those who actually manage to connect data) between connecting a source and building a chart. A 20-minute progress bar that cancels if you close the tab is a conversion killer, and it's driving 18% of our support tickets. * The Fix: Move syncing to the background. Let users close the tab (or play with sample data) and email them the moment their data is ready. Also, replace the raw list of every database table (Screen 7) with smart defaults so non-technical users aren't overwhelmed.

3. Eliminate the "Empty Canvas" (Screens 9–10) * The Problem: A user picks a dashboard template on Screen 9, only to be dropped into a blank chart builder on Screen 10. Blank canvases cause analysis paralysis. * The Fix: Make the templates on Screen 9 actually populate pre-built charts on Screen 10 using their data (or the sample data). Give them a working chart immediately that they can tweak, getting them across that 62% retention threshold automatically.

4. Remove pre-value friction (Screens 2–4) * The Problem: We pause the flow for email verification and ask two CRM-style profiling questions before the user has seen the product. Every step here costs us momentum. * The Fix: Let users into the product immediately and require email verification later (e.g., before they can share a chart). Cut the “team size” question entirely, and only keep “role” if we use it to auto-select their dashboard template in Step 9.

Let's scope Item #1 for this sprint.

Claims the judge couldn’t find in the brief

  • PMs and marketers don't have Postgres passwords or understand SSL modes, and a 14-page guide won't help them.

Grades and run details

Decision model 70 · LLM judge 64
Decision model checks
  • passUses the supplied evidence correctly69%
  • failAddresses the actual decision49%
  • passRespects explicit constraints61%
  • failIdentifies material uncertainty100%
  • failAvoids unsupported claims86%
  • passProduces the required deliverable95%
  • passPrioritises by impact97%
  • passActivation defined by what predicts retention100%
  • passGets the base of every number right45%
  • passGets people to value sooner97%
Run
Run
#1
API response time
8 s
Submitted
13 Oct 2026

Needs more than a quick edit5 of 10 checks passed

The output misstates support ticket percentages as user percentages, which would require re-checking the evidence before use. A PM would need to correct this factual error.

Output

MEMORANDUM

To: Onboarding Squad From: Product Team Subject: Onboarding Funnel Analysis & Prioritized Fixes

Our last 90 days show 12,400 signups, but we face a steep 59% drop-off immediately after signup (reaching only 41% at data source connection), followed by a massive drop at team invites (down to 9%). Because week-2 retention jumps from 11% to 62% for users who build a first chart, removing friction before that milestone is critical.

Here is where we are losing people and the fixes, ordered by priority:

1. The Data Connection Blockade (Fix First)

  • Where we lose people: We drop 59% of users right at the data connection step.2 Support tickets confirm why: 41% lack database credentials, and 18% experience sync failures. Forcing product managers and marketers—who are not data engineers—to input host, port, SSL modes, and select from raw database tables while blocking the UI for up to 20 minutes is a conversion killer.
  • What to fix:
  • Add a pre-loaded Sample Data / Demo Workspace option so users can experience the product immediately without connecting a live database.
  • Make data connection skippable or move it after the first chart experience.
  • Simplify the CSV upload option or provide pre-built connectors with clear, inline tooltips instead of a 14-page setup guide.

2. The Premature Team Invite Wall (Fix Second)

  • Where we lose people: Reaching a first chart drops to 33%, but inviting teammates plummets to 9%—a catastrophic 24-percentage-point drop right before value is fully realized.1 Users are being asked to advocate for a tool before they’ve even proven its worth to themselves.
  • What to fix:
  • Remove the team invite step from the core linear onboarding path entirely.
  • Relocate the invite prompt to the dashboard or settings menu once the user has successfully built and saved their first chart.

3. Redundant Survey Friction (Fix Third)

  • Where we lose people: Screens 3 ("What's your role?") and 4 ("How big is your team?") add unnecessary micro-friction right after email verification.
  • What to fix:
  • Combine these into a single, clean optional screen or infer them from workspace setup later to streamline time-to-value.

Fixing data connection roadblocks and removing premature team invites will directly protect our week-2 retention (which hinges on reaching that first chart).

What a PM had to fix

  1. 1Numbers wrongRedo the arithmeticTargeted repair

    Source: Funnel (last 90 days, 12,400 signups)Connect data source 41% → First chart 33% → Invite teammate 9%.

    What we’d changeRank the 41% to 33% drop between connecting and a first chart second. The invite comes after the first chart, so it isn't what stands between users and value.

  2. 2Hypothesis stated as factReframe it as a hypothesisQuick edit

    What we’d changeThe 59% is lost somewhere between screens 2 and 8. Say the tickets point to the connection form, and instrument the screens to confirm it.

Claims the judge couldn’t find in the brief

  • 41% lack database credentials
  • 18% experience sync failures

Grades and run details

Decision model 70 · LLM judge 45
Decision model checks
  • passUses the supplied evidence correctly64%
  • failAddresses the actual decision66%
  • passRespects explicit constraints73%
  • failIdentifies material uncertainty100%
  • failAvoids unsupported claims66%
  • passProduces the required deliverable83%
  • passPrioritises by impact69%
  • passActivation defined by what predicts retention98%
  • passGets the base of every number right38%
  • passGets people to value sooner95%
Run
Run
#1
Time to output
8 s
Submitted
25 Sept 2026

Usable with a quick edit8 of 10 checks passed

The memo is well-structured, uses the evidence correctly for the core argument, and the few unsupported claims are minor and can be edited out without redoing the reasoning.

Output

Onboarding: where we lose people and what to fix first

Sep 25, 2026 · @Martin Slaney

To: Onboarding squad

Bottom line

Onboarding works for people who reach a first chart: 62% of them are still here in week 2, against 11% of everyone else. Two-thirds of signups never get there, and most of those stop at "Connect your data", where we ask product managers and marketers for database credentials they don't have.

Where we lose people (last 90 days, 12,400 signups)

  • Signup → connect data: 59% lost (\~7,300 people). The biggest leak by far. The top support ticket (41%) is "I don't have the database credentials."
  • Connect → first chart: \~990 more lost, a fifth of those who connected. Syncs take up to 20 minutes, die if you leave the page, and "my sync failed" is the second-biggest ticket (18%).
  • First chart → invite: 33% → 9%. Real, but downstream of the first two.

Caveat: we have no step-level data for screens 2–8, so we can't split the 59% between verification, the two questions and the connection form. The ticket data points hard at the form.

What to fix, in order

  1. Let people reach a first chart without their own data. Add a sample dataset and a skip on screen 5, and land them on a ready-made chart. Why: it goes around the step that loses most people, and the first chart is the moment that predicts retention.
  2. Let them hand off the credentials step. "Ask a teammate to connect" sends an engineer a secure link to fill in the connection form. Why: it answers the 41% ticket directly and turns the blocker into an invite.
  3. Make syncing forgiving. Run it in the background, keep it going when they leave, email them when it's ready, and load a small sample of rows first so a chart appears in minutes. Show plain-language errors with a retry. Why: a 20-minute wait that punishes leaving the page is where connected users die, and it's the 18% ticket.
  4. Don't start from an empty canvas. Use the role answer from screen 3 to suggest two or three templates instead of twelve, and draw the first chart automatically. Why: we already ask the role and do nothing with it; a blank builder is hard for non-analysts.
  5. Trim the steps before connecting. Let people in before they verify their email (verify within a day), and drop or defer the team-size question. Why: cheap to do, but unmeasured, so instrument first rather than assume it's a big win.
  6. Move the invite prompt to after value is shown, such as when someone saves or shares a chart. Why: people invite colleagues to something useful, not to a setup screen.

How we'll know it worked

  • Instrument every screen, 1 to 11, before shipping anything.1
  • Judge fix 1 by how many sample-data users go on to connect real data and are retained in week 2, not by first-chart rate. A chart of sample data can lift that metric without anyone sticking around, so don't assume the 62% carries over.

What a PM had to fix

  1. 1OtherFix itQuick edit

    What we’d changeInstrument alongside fix 1 rather than before it, and merge fixes 1 and 4, which both land the user on an auto-drawn chart.

Claims the judge couldn’t find in the brief

  • We have no step-level data for screens 2–8.
  • A blank builder is hard for non-analysts.
  • People invite colleagues to something useful, not to a setup screen.

Grades and run details

Decision model 90 · LLM judge 82
Decision model checks
  • passUses the supplied evidence correctly56%
  • passAddresses the actual decision73%
  • passRespects explicit constraints30%
  • passIdentifies material uncertainty95%
  • failAvoids unsupported claims30%
  • passProduces the required deliverable95%
  • passPrioritises by impact98%
  • passActivation defined by what predicts retention100%
  • passGets the base of every number right55%
  • passGets people to value sooner100%
Run
Run
#1
Time to output
40 s
Submitted
25 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 3

Addresses the actual decisionWrongWrongRight
Gemini 4 Argon · API

The output does not state any condition or result that would change the prioritized answer, as required by the criteria.

Gemini 3.5 Flash-Lite · Gemini

The output does not say what result or condition would change the prioritised order, which is required for a clear decision.

Opus 5.5 · Claude

The memo commits to a clear ranked order of fixes early, and says what result would change the call (instrumentation, retention of sample-data users).

Identifies material uncertaintyWrongWrongRight
Gemini 4 Argon · API

It does not name any unknowns that could change the decision or say how they would be resolved.

Gemini 3.5 Flash-Lite · Gemini

The output does not name any unknowns that could change the decision or say how they would be resolved.

Opus 5.5 · Claude

It names the lack of step-level data as a caveat, says to instrument before assuming wins, and specifies how to judge fix 1 to avoid a vanity metric.

Gets the base of every number rightRightMixedRight
Gemini 4 Argon · API

All derived percentages (59%, 8%) are correctly computed from the funnel steps, and bases like support tickets are stated where ambiguous.

Gemini 3.5 Flash-Lite · Gemini

The 41% and 18% figures are not computed from the right denominator (they are ticket shares, not user shares), and the base is not stated, making them misleading.

Opus 5.5 · Claude

All derived figures (59% loss, ~7,300, ~990, a fifth) are computed from the correct denominators and match the supplied data.

All got wrong 1

Avoids unsupported claimsWrongWrongWrong
Gemini 4 Argon · API

It presents interpretations like 'PMs don't understand SSL modes' and 'blank canvases cause analysis paralysis' as established facts without labelling them as hypotheses.

Gemini 3.5 Flash-Lite · Gemini

The output presents '41% lack database credentials' and '18% experience sync failures' as established facts, and states the data connection flow 'is a conversion killer' as a confident causal claim, without labelling them as hypotheses.

Opus 5.5 · Claude

It presents 'a blank builder is hard for non-analysts' and 'people invite colleagues to something useful, not to a setup screen' as established facts without labelling them as hypotheses.

All mixed 1

Uses the supplied evidence correctlyMixedMixedMixed
Gemini 4 Argon · API

The claim 'PMs and marketers don't understand SSL modes' is not supported by the brief or supplied context; it is invented.

Gemini 3.5 Flash-Lite · Gemini

The output states '41% lack database credentials' and '18% experience sync failures' as if they are user percentages, but the source only gives ticket percentages; this misrepresents the evidence.

Opus 5.5 · Claude

The output states 'we have no step-level data for screens 2–8', 'a blank builder is hard for non-analysts', and 'people invite colleagues to something useful, not to a setup screen' as facts, none of which appear in the brief or follow from arithmetic.

All got right 5

Respects explicit constraintsRightRightRight
Gemini 4 Argon · API

The memo is under 500 words, addressed to the onboarding squad, and proposes fixes that respect the brief's constraints.

Gemini 3.5 Flash-Lite · Gemini

The output is a memo to the onboarding squad, under 500 words, and respects the requested form and length.

Opus 5.5 · Claude

The output is a memo to the onboarding squad, under 500 words, and respects the requested form.

Produces the required deliverableRightRightRight
Gemini 4 Argon · API

The deliverable is a complete memo with a prioritized list of fixes and reasons, under 500 words, ready for the onboarding squad.

Gemini 3.5 Flash-Lite · Gemini

The memo is present, addressed to the onboarding squad, within the word limit, and provides a prioritised list of fixes with reasons.

Opus 5.5 · Claude

The memo is complete, in the right form, for the named reader, and under 500 words; a PM could act on it with light edits.

Prioritises by impactRightRightRight
Gemini 4 Argon · API

Fixes are ordered by impact on the activation bottleneck, starting with the biggest drop-off before connecting data.

Gemini 3.5 Flash-Lite · Gemini

Fixes are ranked by likely impact on activation, with data connection first, then team invite, then surveys.

Opus 5.5 · Claude

Fixes are ranked 1–6 by likely impact on activation, with reasoning tied to the bottleneck and retention data.

Activation defined by what predicts retentionRightRightRight
Gemini 4 Argon · API

It explicitly defines activation as building a first chart because of its retention lift (62% vs 11%) and ranks fixes by their effect on reaching that event.

Gemini 3.5 Flash-Lite · Gemini

The output identifies reaching a first chart as the activation event, links it to the retention jump (62% vs 11%), and ranks fixes by their effect on that event.

Opus 5.5 · Claude

It defines activation as reaching a first chart because it predicts retention (62% vs 11%), and ranks fixes by their effect on that event.

Gets people to value soonerRightRightRight
Gemini 4 Argon · API

Main fixes cut or defer steps before first value (sample data, invite engineer, defer email verification) and give a warm start with pre-populated charts.

Gemini 3.5 Flash-Lite · Gemini

The main fixes (sample data, skippable connection, deferring team invite) cut or defer steps before the first chart, shortening the path to value.

Opus 5.5 · Claude

The main fixes cut or defer steps before first value (sample data, skip connection, handoff, forgiving sync, trim pre-connect steps) and give a warm start (role-based templates).

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 72% of 53 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Sonnet 5.5withAPI83.6100.02None
2GPT-6 AstrawithChatGPT86.195.52None
3GPT-6.1 SolwithAPI83.690.92None
4Haiku 5.5withAPI88.086.72None
5GPT-6 LunawithAPI92.781.82None
6Opus 5.5withClaude81.478.42None
7Gemini 4 ArgonwithAPI73.665.22None
8Gemini 3.8 FlashwithAPI71.451.92None
9Gemini 3.5 Flash-LitewithGemini60.039.42None

What you get for the cost

GPT-6 Luna · API scores within 5 points of Sonnet 5.5 · API at about a thirtieth of the cost.

Task score against relative cost for each setup. The table below has the same figures.4050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreSonnet 5.5 · API: task score 93, 32× the cheapestGPT-6 Astra · ChatGPT: task score 91, 122× the cheapest (1 of 2 outputs estimated)GPT-6.1 Sol · API: task score 88, 15× the cheapestHaiku 5.5 · API: task score 88, 1.8× the cheapestGPT-6 Luna · API: task score 88, the cheapestOpus 5.5 · Claude: task score 79, 101× the cheapest (1 of 2 outputs estimated)Gemini 4 Argon · API: task score 68, 15× the cheapestGemini 3.8 Flash · API: task score 59, 13× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 47, 3.4× the cheapest (1 of 2 outputs estimated)Sonnet 5.5 · APIGPT-6 Astra · ChatGPTGPT-6.1 Sol · APIHaiku 5.5 · APIGPT-6 Luna · APIOpus 5.5 · ClaudeGemini 4 Argon · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API881.0×
Haiku 5.5 · API881.8×
Gemini 3.5 Flash-Lite · Gemini473.4× (partly estimated)
Gemini 3.8 Flash · API5913×
GPT-6.1 Sol · API8815×
Gemini 4 Argon · API6815×
Sonnet 5.5 · API9332×
Opus 5.5 · Claude79101× (partly estimated)
GPT-6 Astra · ChatGPT91122× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Reviewing a signup and onboarding flow that is losing users.

Why it matters

Anyone can list fifty UX nits. The job is finding the two that explain the drop-off, backed by the funnel data supplied.

What good looks like

  • Ties each issue to the funnel data
  • Prioritises by likely impact
  • Defines activation as the behaviour that predicts retention, not finishing onboarding

Deliberately not measured

  • Accessibility audit completeness
  • Visual redesign
Capability tested

Consequential critique

The failure we’re looking for

A generic UX checklist

Grading

Decision model and LLM judge, calibrated against a blind PM review