Tasks / Growth

Activation & onboarding review

Can the model find the friction that matters most and prioritise the fixes?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 69% were usable with at most a quick edit.

Reliably right

  1. Respects explicit constraints100% pass
    The output is a review addressed to Omar, well under 800 words, and respects the requested form.
    GPT-6 Luna · API · The checklist isn't the moment
  2. Produces the required deliverable100% pass
    The review covers activation definition, evidence, changes, design proposal, and goal, is addressed to Omar, within length, and is actionable.
    GPT-6 Luna · API · The checklist isn't the moment
  3. Activation defined by what predicts retention100% pass
    Activation is defined by the behaviour that predicts retention (bank+cat 10+), and fixes are judged by their effect on that event.
    GPT-6 Luna · API · The checklist isn't the moment

Where it slips

  1. Uses the supplied evidence correctly44% pass
    The output states 'we have no step-level data for screens 2–8', 'a blank builder is hard for non-analysts', and 'people invite colleagues to something useful, not to a setup screen' as facts, none of which appear in the brief or follow from arithmetic.
    Opus 5.5 · Claude · Analytics tool losing users at setup
  2. Avoids unsupported claims55% pass
    It presents 'a blank builder is hard for non-analysts' and 'people invite colleagues to something useful, not to a setup screen' as established facts without labelling them as hypotheses.
    Opus 5.5 · Claude · Analytics tool losing users at setup
  3. Addresses the actual decision56% pass
    The output commits to a ranked order but does not state what result or condition would change that order, as required.
    GPT-6 Astra · ChatGPT · Analytics tool losing users at setup

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Activation defined by what predicts retention

    Does the review define (or question) the activation event by whether it predicts retention, and judge fixes by their effect on that event rather than on step completion?

    Passes when Names the activation behaviour and its link to retention (from the supplied data, or as an explicit assumption to verify), and ranks fixes by their effect on it.

  2. Gets people to value sooner

    Do the recommended fixes shorten the path to the first moment of value, by removing or deferring steps before it or starting people with templates or sample data, rather than adding tours, tooltips or extra steps?

    Passes when The main fixes cut or defer steps before first value, or give a warm start, and say which steps go.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 1 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

Review our onboarding flow and funnel below. Write a memo to the onboarding squad saying where we are losing people and what to fix, in the order you would fix it, with the reason for each. Keep it under 500 words.

What the model was given4 items: Scenario, Funnel (last 90 days, 12,400 signups), Onboarding flow, screen by screen, Support
ScenarioChartwise is a self-serve product-analytics tool. Most people who sign up are product managers or marketers, not data engineers. Pricing starts at $49 a month after a 14-day free trial.
Funnel (last 90 days, 12,400 signups)Signup 100% → Connect data source 41% → First chart 33% → Invite teammate 9%. Week-2 retention: 62% for users who reach a first chart, 11% otherwise.
Onboarding flow, screen by screen1. Sign up: email, password, company name. 2. Verify email: the flow pauses until the link is clicked. 3. “What's your role?” (seven options). 4. “How big is your team?” (five options). 5. Connect your data: choose Postgres, BigQuery, Snowflake or CSV upload. There is no sample data and no skip. 6. Connection form: host, port, database, username, password and SSL mode. The help link opens a 14-page setup guide. 7. Choose tables: a list of every table in the database. 8. Syncing: a progress bar that can take up to 20 minutes; leaving the page cancels the sync. 9. Pick a dashboard template (twelve options). 10. First chart: a chart builder with an empty canvas. 11. Invite teammates: email fields, with a “Skip for now” link.
SupportThe most common onboarding ticket (41% of them) is some version of “I don't have the database credentials.” The next is “my sync failed” (18%).
What a strong answer doesThe answer key the graders mark against

The data-source connection step (screens 5–8) is the activation bottleneck; prioritise a sample-data or CSV-first path to a first chart, then non-admin credential handoff and a sync that survives leaving the page, ahead of cosmetic fixes.

Task

Mid-level · v1.12

The outputs

8 setups answered this brief.

Usable with a quick edit7 of 10 checks passed

The invite placement error is a minor factual slip that can be corrected quickly; the rest is structured and actionable.

Output

MEMO

To: Onboarding squad Re: Where we lose people, and what to fix first

Where we lose people

  • Signup to connect (100% to 41%): The largest loss. Most of our users are PMs and marketers who don't have database credentials, and the flow gives them no way forward without them.
  • Connect to first chart (41% to 33%): About a fifth of people who connect never build a chart. Sync and the empty canvas are the likely causes.
  • First chart to invite (33% to 9%): Expected, since invites come after value. Lower priority for now.

Retention is the reason to care about the first chart: 62% of users who reach one are active in week 2, versus 11% of those who don't. This is correlational, since engaged users may be the ones who build charts, but it is the strongest signal we have.

What to fix, in order

  1. Add sample data and a skip at step 5. Users without credentials currently hit a dead end. Sample data gets them to a chart in minutes, which is the milestone that predicts retention. This also targets the top support ticket (41%). Reason: highest expected impact on the number that matters.
  1. Let people in before email verification, and drop the role and team screens from the critical path. Step 2 blocks all progress, and steps 3–4 ask questions before the user has seen any value. Move role and team to after the first chart, where they can personalize templates. Reason: removes the earliest friction, when drop-off is highest.
  1. Rebuild the credentials path. Replace the 14-page guide with short, per-warehouse steps for Postgres, BigQuery and Snowflake, and add a "send these instructions to your data engineer" link that delivers a connect-only invite. The person who has the credentials is often not the person signing up. Reason: addresses the remaining credential tickets and the step 6 drop-off.
  1. Fix sync. Let leaving the page not cancel the sync. Show partial results as tables finish. Pre-select likely tables and add search to the step 7 list. Make CSV upload the fastest route to a first chart. Reason: sync failures are the second-largest ticket category (18%), and cancellation on exit loses users mid-setup.
  1. Make the first chart appear without effort. Templates (step 9) should render a live chart immediately. Replace the empty canvas at step 10 with a suggested chart the user can edit. Reason: a blank canvas is the last hurdle before the retention milestone.
  1. Move team invites after the first chart, and prompt for them there. Keep the skip link. Reason: invites are useful but come after value; test this once the upstream fixes land.

Measurement

Track signup-to-first-chart as the primary metric, with week-2 retention as the outcome. Run fixes 1 and 2 as separate experiments so we can see which one moves the funnel.

Claims the judge couldn’t find in the brief

  • Most of our users are PMs and marketers who don't have database credentials.
  • Sync and the empty canvas are the likely causes (of drop-off from connect to first chart).
  • The person who has the credentials is often not the person signing up.
  • There is a 'step 6 drop-off' that the credentials path fix addresses.
  • Invite teammates is currently before the first chart (implied by 'Move team invites after the first chart').

Grades and run details

Decision model 70 · LLM judge 82
Decision model checks
  • failUses the supplied evidence correctly33%
  • failAddresses the actual decision46%
  • passRespects explicit constraints38%
  • partialIdentifies material uncertainty30%
  • partialAvoids unsupported claims38%
  • passProduces the required deliverable97%
  • passPrioritises by impact73%
  • passActivation defined by what predicts retention100%
  • passGets the base of every number right45%
  • passGets people to value sooner100%
Run
Run
#1
API response time
11 s
Submitted
8 Oct 2026

Usable with a quick edit8 of 10 checks passed

The memo is well-structured, uses the evidence correctly for the core argument, and the few unsupported claims are minor and can be edited out without redoing the reasoning.

Output

Onboarding: where we lose people and what to fix first

Sep 25, 2026 · @Martin Slaney

To: Onboarding squad

Bottom line

Onboarding works for people who reach a first chart: 62% of them are still here in week 2, against 11% of everyone else. Two-thirds of signups never get there, and most of those stop at "Connect your data", where we ask product managers and marketers for database credentials they don't have.

Where we lose people (last 90 days, 12,400 signups)

  • Signup → connect data: 59% lost (\~7,300 people). The biggest leak by far. The top support ticket (41%) is "I don't have the database credentials."
  • Connect → first chart: \~990 more lost, a fifth of those who connected. Syncs take up to 20 minutes, die if you leave the page, and "my sync failed" is the second-biggest ticket (18%).
  • First chart → invite: 33% → 9%. Real, but downstream of the first two.

Caveat: we have no step-level data for screens 2–8, so we can't split the 59% between verification, the two questions and the connection form. The ticket data points hard at the form.

What to fix, in order

  1. Let people reach a first chart without their own data. Add a sample dataset and a skip on screen 5, and land them on a ready-made chart. Why: it goes around the step that loses most people, and the first chart is the moment that predicts retention.
  2. Let them hand off the credentials step. "Ask a teammate to connect" sends an engineer a secure link to fill in the connection form. Why: it answers the 41% ticket directly and turns the blocker into an invite.
  3. Make syncing forgiving. Run it in the background, keep it going when they leave, email them when it's ready, and load a small sample of rows first so a chart appears in minutes. Show plain-language errors with a retry. Why: a 20-minute wait that punishes leaving the page is where connected users die, and it's the 18% ticket.
  4. Don't start from an empty canvas. Use the role answer from screen 3 to suggest two or three templates instead of twelve, and draw the first chart automatically. Why: we already ask the role and do nothing with it; a blank builder is hard for non-analysts.
  5. Trim the steps before connecting. Let people in before they verify their email (verify within a day), and drop or defer the team-size question. Why: cheap to do, but unmeasured, so instrument first rather than assume it's a big win.
  6. Move the invite prompt to after value is shown, such as when someone saves or shares a chart. Why: people invite colleagues to something useful, not to a setup screen.

How we'll know it worked

  • Instrument every screen, 1 to 11, before shipping anything.1
  • Judge fix 1 by how many sample-data users go on to connect real data and are retained in week 2, not by first-chart rate. A chart of sample data can lift that metric without anyone sticking around, so don't assume the 62% carries over.

What a PM had to fix

  1. 1OtherFix itQuick edit

    What we’d changeInstrument alongside fix 1 rather than before it, and merge fixes 1 and 4, which both land the user on an auto-drawn chart.

Claims the judge couldn’t find in the brief

  • We have no step-level data for screens 2–8.
  • A blank builder is hard for non-analysts.
  • People invite colleagues to something useful, not to a setup screen.

Grades and run details

Decision model 95 · LLM judge 82
Decision model checks
  • passUses the supplied evidence correctly27%
  • passAddresses the actual decision16%
  • passRespects explicit constraints30%
  • passIdentifies material uncertainty94%
  • partialAvoids unsupported claims38%
  • passProduces the required deliverable93%
  • passPrioritises by impact97%
  • passActivation defined by what predicts retention100%
  • passGets the base of every number right39%
  • passGets people to value sooner100%
Run
Run
#1
Time to output
40 s
Submitted
25 Sept 2026

Needs more than a quick edit5 of 10 checks passed

The output contains a factual error (treating 41% of support tickets as a user share) and unsupported claims, so a PM would need to re-check the evidence and correct the reasoning before using it.

Output

To: Onboarding Squad Subject: Fixing Our Activation Funnel: Priorities & Action Plan

Where We Are Losing People

Our biggest drop-off occurs at Step 5–8 (Connect Data): 59% of signups leave without connecting a data source.

We are asking PMs and marketers for database ports, SSL modes, and passwords they do not possess. 41% of our support tickets are “I don't have credentials,” followed by 18% for failed syncs. Furthermore, requiring a 20-minute, open-tab sync kills momentum.

Because Week-2 retention jumps from 11% to 62% once a user builds their first chart, our sole objective is getting non-technical users to that "aha!" moment in under two minutes.

---

What to Fix (In Priority Order)

#### 1. Add “Explore with Sample Data” & a Clear “Skip” Button (Step 5) * What: Let users bypass database connection entirely using a fully populated e-commerce or SaaS demo dataset. * Why: This is the highest-leverage fix. It immediately unblocks the 59% who drop off at Step 5 and the 41% who lack DB credentials, giving them an instant path to create a first chart and experience the product’s value on Day 1.

#### 2. Introduce “Invite a Data Teammate” at Connection (Step 6) * What: Add a 1-click option: “Don't have credentials? Send an invite link to your data lead to connect your warehouse.” * Why: Replaces a dead end with collaboration. The PM stays in the app using sample data while the engineer handles credentials asynchronously.

#### 3. Make Syncing Asynchronous (Step 8) * What: Run syncing in the background. Notify users via email/in-app alert when ready instead of forcing them to keep a 20-minute tab open. * Why: Directly addresses our second-biggest support issue (18% sync failures/cancellations) and eliminates the drop-off caused by tab abandonment.

#### 4. Pre-Populate the Dashboard; Kill the Blank Canvas (Steps 9 & 10) * What: When a user selects a template, generate it automatically with 3–4 standard charts instead of dropping them onto an empty canvas. Prompt them to tweak one existing chart. * Why: Bridges the 8% drop-off between connecting data (41%) and creating a first chart (33%). Editing an existing chart is far lower friction than building from scratch.

#### 5. Defer Email Verification and Profiling Questions (Steps 2, 3, & 4) * What: Move verification to a non-blocking banner at the top of the app, and cut or defer the role/team-size questions until after the user creates their first chart. * Why: Every barrier before core value reduces trial-to-paid conversion. Let users reach the product before demanding administrative tasks.

---

Next Step

Let’s implement Priority 1 (Sample Data) this sprint. It requires zero infrastructure changes to our warehouse sync logic and directly targets our biggest activation leak.

Claims the judge couldn’t find in the brief

  • requiring a 20-minute, open-tab sync kills momentum.
  • the 41% who lack DB credentials (as a proportion of users).
  • It requires zero infrastructure changes to our warehouse sync logic.

Grades and run details

Decision model 65 · LLM judge 45
Decision model checks
  • failUses the supplied evidence correctly20%
  • failAddresses the actual decision96%
  • passRespects explicit constraints52%
  • failIdentifies material uncertainty99%
  • partialAvoids unsupported claims38%
  • passProduces the required deliverable96%
  • passPrioritises by impact98%
  • passActivation defined by what predicts retention100%
  • passGets the base of every number right50%
  • passGets people to value sooner100%
Run
Run
#1
API response time
21 s
Submitted
1 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Uses the supplied evidence correctlyWrongMixedWrong
Haiku 5.5 · API

Several statements about the current situation are unsupported or contradicted, such as inviting teammates being before the first chart and the existence of a 'step 6 drop-off' with no data.

Opus 5.5 · Claude

The output states 'we have no step-level data for screens 2–8', 'a blank builder is hard for non-analysts', and 'people invite colleagues to something useful, not to a setup screen' as facts, none of which appear in the brief or follow from arithmetic.

Gemini 3.8 Flash · API

The output misuses the 41% support-ticket share as a user percentage ('the 41% who lack DB credentials') and presents unsupported interpretations as fact.

Addresses the actual decisionMixedRightWrong
Haiku 5.5 · API

Commits unambiguously to a ranked list of fixes in order and says what to measure.

Opus 5.5 · Claude

The memo commits to a clear ranked order of fixes early, and says what result would change the call (instrumentation, retention of sample-data users).

Gemini 3.8 Flash · API

The output does not state what result or condition would change the recommended priority order.

Identifies material uncertaintyRightRightWrong
Haiku 5.5 · API

Acknowledges the retention correlation may not be causal and proposes experiments to resolve which fixes move the funnel.

Opus 5.5 · Claude

It names the lack of step-level data as a caveat, says to instrument before assuming wins, and specifies how to judge fix 1 to avoid a vanity metric.

Gemini 3.8 Flash · API

No unknowns that could change the decision are named, nor how they would be resolved.

Gets the base of every number rightRightRightMixed
Haiku 5.5 · API

Percentages are computed from the correct steps, and ticket shares are correctly sourced.

Opus 5.5 · Claude

All derived figures (59% loss, ~7,300, ~990, a fifth) are computed from the correct denominators and match the supplied data.

Gemini 3.8 Flash · API

The phrase 'the 41% who lack DB credentials' is ambiguous and could be read as a user share; the base (support tickets) is not restated, leading to a misrepresentation.

All got wrong 1

Avoids unsupported claimsWrongWrongWrong
Haiku 5.5 · API

Presents several causal claims and inferences (e.g., who holds credentials, step 6 drop-off) as facts without labelling them as hypotheses.

Opus 5.5 · Claude

It presents 'a blank builder is hard for non-analysts' and 'people invite colleagues to something useful, not to a setup screen' as established facts without labelling them as hypotheses.

Gemini 3.8 Flash · API

It presents 'kills momentum' and 'zero infrastructure changes' as established facts without evidence, and mislabels the 41% ticket share as a user proportion.

All got right 5

Respects explicit constraintsRightRightRight
Haiku 5.5 · API

Delivers a memo to the onboarding squad, under 500 words.

Opus 5.5 · Claude

The output is a memo to the onboarding squad, under 500 words, and respects the requested form.

Gemini 3.8 Flash · API

The output is a memo to the onboarding squad, under 500 words, and respects the requested form.

Produces the required deliverableRightRightRight
Haiku 5.5 · API

The memo is a complete, actionable set of recommendations that can be used with light editing.

Opus 5.5 · Claude

The memo is complete, in the right form, for the named reader, and under 500 words; a PM could act on it with light edits.

Gemini 3.8 Flash · API

The memo is complete, correctly addressed, within the word limit, and actionable with light edits.

Prioritises by impactRightRightRight
Haiku 5.5 · API

Fixes are ranked by expected impact on reaching the first chart and retention.

Opus 5.5 · Claude

Fixes are ranked 1–6 by likely impact on activation, with reasoning tied to the bottleneck and retention data.

Gemini 3.8 Flash · API

Fixes are ranked by likely impact on activation, starting with the biggest drop-off at data connection.

Activation defined by what predicts retentionRightRightRight
Haiku 5.5 · API

Defines activation by the first chart and its link to retention, and judges fixes by their effect on that milestone.

Opus 5.5 · Claude

It defines activation as reaching a first chart because it predicts retention (62% vs 11%), and ranks fixes by their effect on that event.

Gemini 3.8 Flash · API

It defines activation as building a first chart, links it to retention (11% vs 62%), and ranks fixes by their effect on reaching that event.

Gets people to value soonerRightRightRight
Haiku 5.5 · API

Main fixes cut or defer steps before first value (sample data, skip, deferring role/team) and provide a warm start (live templates).

Opus 5.5 · Claude

The main fixes cut or defer steps before first value (sample data, skip connection, handoff, forgiving sync, trim pre-connect steps) and give a warm start (role-based templates).

Gemini 3.8 Flash · API

The main fixes cut or defer steps before first value (sample data bypasses connection, invite defers credentials, async sync removes wait, pre-populated dashboard gives warm start, verification/questions deferred).

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated. Where our PM checked the graders by hand, they agreed on 79% of 53 checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT85.995.52None
2Sonnet 5.5withAPI78.4100.02None
3GPT-6.1 SolwithAPI85.990.92None
4GPT-6 LunawithAPI90.081.82None
5Haiku 5.5withAPI75.986.72None
6Opus 5.5withClaude88.478.42None
7Gemini 3.8 FlashwithAPI66.651.92None
8Gemini 3.5 Flash-LitewithGemini52.539.42None

What you get for the cost

GPT-6 Luna · API scores within 5 points of GPT-6 Astra · ChatGPT at about a hundredth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 91, 122× the cheapest (1 of 2 outputs estimated)Sonnet 5.5 · API: task score 90, 32× the cheapestGPT-6.1 Sol · API: task score 89, 15× the cheapestGPT-6 Luna · API: task score 86, the cheapestHaiku 5.5 · API: task score 82, 1.8× the cheapestOpus 5.5 · Claude: task score 82, 101× the cheapest (1 of 2 outputs estimated)Gemini 3.8 Flash · API: task score 56, 13× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 44, 3.4× the cheapest (1 of 2 outputs estimated)GPT-6 Astra · ChatGPTSonnet 5.5 · APIGPT-6.1 Sol · APIGPT-6 Luna · APIHaiku 5.5 · APIOpus 5.5 · ClaudeGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and costCost partly estimated (app runs with no usage meter)
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API861.0×
Haiku 5.5 · API821.8×
Gemini 3.5 Flash-Lite · Gemini443.4× (partly estimated)
Gemini 3.8 Flash · API5613×
GPT-6.1 Sol · API8915×
Sonnet 5.5 · API9032×
Opus 5.5 · Claude82101× (partly estimated)
GPT-6 Astra · ChatGPT91122× (partly estimated)

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Reviewing a signup and onboarding flow that is losing users.

Why it matters

Anyone can list fifty UX nits. The job is finding the two that explain the drop-off, backed by the funnel data supplied.

What good looks like

  • Ties each issue to the funnel data
  • Prioritises by likely impact
  • Defines activation as the behaviour that predicts retention, not finishing onboarding

Deliberately not measured

  • Accessibility audit completeness
  • Visual redesign
Capability tested

Consequential critique

The failure we’re looking for

A generic UX checklist

Grading

Decision model and LLM judge, calibrated against a blind PM review