Tasks / Design

Activation & onboarding review

Can the model find the friction that matters most and prioritise the fixes?

Measures the modelTask v1.1 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 12 graded outputs by 6 models. 58% were usable with at most a quick edit.

Reliably right

  1. Activation defined by what predicts retention100% pass
    Defines activation as three workouts linked to day-60 retention, and evaluates fixes by effect on that event.
    GPT-6 Astra · ChatGPT · Fitness app first week
  2. Prioritises by impact98% pass
    Ranks changes by likely impact on achieving three workouts, aligned with the identified activation threshold.
    GPT-6 Astra · ChatGPT · Fitness app first week
  3. Produces the required deliverable92% pass
    Provides a complete, actionable memo that a growth PM could implement without major gaps.
    GPT-6 Astra · ChatGPT · Fitness app first week

Where it slips

  1. Avoids unsupported claims60% pass
    It presents causal or evaluative claims such as 'successful connection is the gateway' and 'a required technical connection is an especially poor first step' as established fact rather than hypothesis.
    GPT-6 Luna · API · Analytics tool losing users at setup
  2. Uses the supplied evidence correctly63% pass
    The output claims that users charged on day 7 “churn or ask for refunds”, which is not in the supplied context and is not supported by the data.
    Opus 5.5 · Claude · Fitness app first week
  3. Identifies material uncertainty69% pass
    Identifies the lack of step-level data but does not state what result would change the call or how it would affect the ranking.
    Opus 5.5 · Claude · Analytics tool losing users at setup

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

Review our onboarding flow and funnel below. Write a memo to the onboarding squad saying where we are losing people and what to fix, in the order you would fix it, with the reason for each. Keep it under 500 words.

ScenarioChartwise is a self-serve product-analytics tool. Most people who sign up are product managers or marketers, not data engineers. Pricing starts at $49 a month after a 14-day free trial.
Funnel (last 90 days, 12,400 signups)Signup 100% → Connect data source 41% → First chart 33% → Invite teammate 9%. Week-2 retention: 62% for users who reach a first chart, 11% otherwise.
Onboarding flow, screen by screen1. Sign up: email, password, company name. 2. Verify email: the flow pauses until the link is clicked. 3. “What's your role?” (seven options). 4. “How big is your team?” (five options). 5. Connect your data: choose Postgres, BigQuery, Snowflake or CSV upload. There is no sample data and no skip. 6. Connection form: host, port, database, username, password and SSL mode. The help link opens a 14-page setup guide. 7. Choose tables: a list of every table in the database. 8. Syncing: a progress bar that can take up to 20 minutes; leaving the page cancels the sync. 9. Pick a dashboard template (twelve options). 10. First chart: a chart builder with an empty canvas. 11. Invite teammates: email fields, with a “Skip for now” link.
SupportThe most common onboarding ticket (41% of them) is some version of “I don't have the database credentials.” The next is “my sync failed” (18%).
What a strong answer does

The data-source connection step (screens 5–8) is the activation bottleneck; prioritise a sample-data or CSV-first path to a first chart, then non-admin credential handoff and a sync that survives leaving the page, ahead of cosmetic fixes.

Case

v1.6 · anonymised real · B2B SaaS, data

Usable with a quick edit6 of 9 checks passed

The memo is actionable and fully reasoned; a PM could trim a few words to meet the word limit without redoing any analysis.

Output

Onboarding: where we lose people and what to fix first

Sep 25, 2026 · @Martin Slaney

To: Onboarding squad

Bottom line

Onboarding works for people who reach a first chart: 62% of them are still here in week 2, against 11% of everyone else. Two-thirds of signups never get there, and most of those stop at "Connect your data", where we ask product managers and marketers for database credentials they don't have.

Where we lose people (last 90 days, 12,400 signups)

  • Signup → connect data: 59% lost (\~7,300 people). The biggest leak by far. The top support ticket (41%) is "I don't have the database credentials."
  • Connect → first chart: \~990 more lost, a fifth of those who connected. Syncs take up to 20 minutes, die if you leave the page, and "my sync failed" is the second-biggest ticket (18%).
  • First chart → invite: 33% → 9%. Real, but downstream of the first two.

Caveat: we have no step-level data for screens 2–8, so we can't split the 59% between verification, the two questions and the connection form. The ticket data points hard at the form.

What to fix, in order

  1. Let people reach a first chart without their own data. Add a sample dataset and a skip on screen 5, and land them on a ready-made chart. Why: it goes around the step that loses most people, and the first chart is the moment that predicts retention.
  2. Let them hand off the credentials step. "Ask a teammate to connect" sends an engineer a secure link to fill in the connection form. Why: it answers the 41% ticket directly and turns the blocker into an invite.
  3. Make syncing forgiving. Run it in the background, keep it going when they leave, email them when it's ready, and load a small sample of rows first so a chart appears in minutes. Show plain-language errors with a retry. Why: a 20-minute wait that punishes leaving the page is where connected users die, and it's the 18% ticket.
  4. Don't start from an empty canvas. Use the role answer from screen 3 to suggest two or three templates instead of twelve, and draw the first chart automatically. Why: we already ask the role and do nothing with it; a blank builder is hard for non-analysts.
  5. Trim the steps before connecting. Let people in before they verify their email (verify within a day), and drop or defer the team-size question. Why: cheap to do, but unmeasured, so instrument first rather than assume it's a big win.
  6. Move the invite prompt to after value is shown, such as when someone saves or shares a chart. Why: people invite colleagues to something useful, not to a setup screen.

How we'll know it worked

  • Instrument every screen, 1 to 11, before shipping anything.1
  • Judge fix 1 by how many sample-data users go on to connect real data and are retained in week 2, not by first-chart rate. A chart of sample data can lift that metric without anyone sticking around, so don't assume the 62% carries over.

What a PM had to fix

  1. 1OtherFix itQuick edit

    What we’d changeInstrument alongside fix 1 rather than before it, and merge fixes 1 and 4, which both land the user on an auto-drawn chart.

Check by check

Mixed · 3
  • Respects explicit constraintsThe output is exactly 500 words; the brief requires under 500 words.The two graders disagreed on this one.
  • Identifies material uncertaintyIdentifies the lack of step-level data but does not state what result would change the call or how it would affect the ranking.The two graders disagreed on this one.
  • Produces the required deliverableThe memo is complete and usable but exceeds the word limit: it is 500 words, not under 500.The two graders disagreed on this one.
Got right · 6
  • Uses the supplied evidence correctlyAll factual statements about the current situation are directly from the supplied context or derived by correct arithmetic.
  • Addresses the actual decisionCommits to a clear, ranked order of fixes early in the memo, framed for the onboarding squad.
  • Avoids unsupported claimsInterpretations and causes are presented as reasoning (often within 'Why' sections) and not as established facts.
  • Prioritises by impactFixes are ranked by impact on activation, with the largest drop addressed first.
  • Activation defined by what predicts retentionIdentifies the first chart as the activation event that predicts retention (62% vs 11%) and ranks fixes by their effect on reaching it.
  • Gets the base of every number rightAll derived figures (59%, ~7,300, ~990, a fifth) are computed from the correct steps and denominators, with bases stated where needed.

Grades and run details

Decision model 94 · LLM judge 7 of 10 checks
Decision model checks
  • passUses the supplied evidence correctly19%
  • passAddresses the actual decision92%
  • passRespects explicit constraints28%
  • passIdentifies material uncertainty95%
  • partialAvoids unsupported claims37%
  • passProduces the required deliverable94%
  • passPrioritises by impact98%
  • passActivation defined by what predicts retention100%
  • passGets the base of every number right51%
Run
Run
#1
Time to output
40 s
Submitted
25 Sept 2026

Usable with a quick edit9 of 9 checks passed

The memo is a clear, prioritized, evidence-based list with a straightforward measurement plan; a PM could use it with minimal edits.

Output

To: Onboarding squad Subject: Fix access and sync before optimising chart creation

Our biggest measured loss is before data connection: 59% of signups, roughly 7,300 people, never connect a source. We ask PMs and marketers to complete a database setup that many cannot do themselves, then make a lengthy sync depend on keeping a page open.

Another 8 percentage points disappear between connection and first chart: about 20% of those who connect. Teammate invitations fall to 9% of signups, but we should establish whether collaboration is necessary for value before treating that as our next bottleneck.

First-chart users have much higher week-2 retention, 62% versus 11%. That makes chart creation a promising activation milestone, but does not prove that pushing everyone through a chart will cause better retention.

Fix in this order

1. Give users a route forward without database credentials. Offer a clearly labelled sample-data experience and make CSV upload a prominent alternative.1 Add a setup hand-off that lets users ask a colleague to connect the source, then resume when it is ready. Replace the generic setup guide with short, source-specific instructions.

This addresses the largest measured loss and the strongest support signal: missing credentials account for 41% of onboarding tickets. That is a share of tickets, not signups, so we cannot yet quantify how much drop-off it explains. Sample data should demonstrate value while preserving a clear path to the user’s own data.

2. Make syncing durable and recoverable. Run syncs in the background so closing the page does not cancel them. Show meaningful status, explain failures and allow retries without repeating setup. Notify users when their data is ready.

A 20-minute foreground wait is a fragile requirement. Sync failures generate 18% of onboarding tickets and can waste the effort users have already invested.

3. Turn connected data into a useful first chart. Help users identify relevant tables instead of presenting every table equally. Use their chosen template to produce an editable chart from compatible data, with guided choices where mapping is needed. Do not finish template selection with an empty canvas.

This targets the roughly one in five connected users who never reach a chart. Measure whether they produce something useful, not merely whether a chart renders.

4. Remove avoidable preliminary work. Defer role and team-size questions unless their answers immediately improve setup. Review whether email verification must block exploration, retaining verification where required for account security.

These steps add friction before value, but we lack evidence that they drive the main loss.

Measure alongside delivery: instrument each screen, connection method, error and return visit. Test changes against first charts using customers’ own data, week-2 retention and trial-to-paid conversion. Track sample-data charts separately. Revisit invitations once we understand whether users need collaboration to succeed. To: Onboarding squad Subject: Fix access and sync before optimising chart creation

Our biggest measured loss is before data connection: 59% of signups, roughly 7,300 people, never connect a source. We ask PMs and marketers to complete a database setup that many cannot do themselves, then make a lengthy sync depend on keeping a page open.

Another 8 percentage points disappear between connection and first chart: about 20% of those who connect. Teammate invitations fall to 9% of signups, but we should establish whether collaboration is necessary for value before treating that as our next bottleneck.

First-chart users have much higher week-2 retention, 62% versus 11%. That makes chart creation a promising activation milestone, but does not prove that pushing everyone through a chart will cause better retention.

Fix in this order

1. Give users a route forward without database credentials. Offer a clearly labelled sample-data experience and make CSV upload a prominent alternative. Add a setup hand-off that lets users ask a colleague to connect the source, then resume when it is ready. Replace the generic setup guide with short, source-specific instructions.

This addresses the largest measured loss and the strongest support signal: missing credentials account for 41% of onboarding tickets. That is a share of tickets, not signups, so we cannot yet quantify how much drop-off it explains. Sample data should demonstrate value while preserving a clear path to the user’s own data.

2. Make syncing durable and recoverable. Run syncs in the background so closing the page does not cancel them. Show meaningful status, explain failures and allow retries without repeating setup. Notify users when their data is ready.

A 20-minute foreground wait is a fragile requirement. Sync failures generate 18% of onboarding tickets and can waste the effort users have already invested.

3. Turn connected data into a useful first chart. Help users identify relevant tables instead of presenting every table equally. Use their chosen template to produce an editable chart from compatible data, with guided choices where mapping is needed. Do not finish template selection with an empty canvas.

This targets the roughly one in five connected users who never reach a chart. Measure whether they produce something useful, not merely whether a chart renders.

4. Remove avoidable preliminary work. Defer role and team-size questions unless their answers immediately improve setup. Review whether email verification must block exploration, retaining verification where required for account security.

These steps add friction before value, but we lack evidence that they drive the main loss.

Measure alongside delivery: instrument each screen, connection method, error and return visit. Test changes against first charts using customers’ own data, week-2 retention and trial-to-paid conversion. Track sample-data charts separately. Revisit invitations once we understand whether users need collaboration to succeed.

What a PM had to fix

  1. 1Decision deferredMake the callQuick edit

    What we’d changeRank the four parts of fix 1 so the squad knows what ships first, and make the calls fix 4 leaves open.

Check by check

Got right · 9
  • Uses the supplied evidence correctlyAll statements about the current situation are taken correctly from the brief or derived arithmetically; nothing is invented.
  • Addresses the actual decisionCommits early to a ranked order of fixes, addressed to the onboarding squad, and says team invitations will be revisited after understanding if collaboration is needed.
  • Respects explicit constraintsMemo format, under 500 words, directed to the onboarding squad, respecting all explicit constraints.
  • Identifies material uncertaintyIdentifies that first-chart correlation does not prove causation, ticket share is not user drop-off, and role/team questions lack evidence; proposes measurement to resolve.
  • Avoids unsupported claimsInterpretations are grounded in the data and qualified where evidence is suggestive, so no factual claims are presented as established without support.
  • Produces the required deliverableDelivers a complete memo with prioritized fixes, reasons, and measurement plan that the squad can act on with light edits.
  • Prioritises by impactFixes are ranked by likely impact on activation: credential-free path first (largest drop), then sync durability, then first-chart creation, then friction removal.
  • Activation defined by what predicts retentionActivation is defined as first chart, linked to week-2 retention (62% vs 11%), and fixes are judged by their effect on reaching that milestone.
  • Gets the base of every number rightAll percentages are computed from the correct funnel steps, and ticket shares are explicitly noted as share of tickets, not users.

Grades and run details

Decision model 94 · LLM judge 10 of 10 checks
Decision model checks
  • passUses the supplied evidence correctly31%
  • passAddresses the actual decision85%
  • partialRespects explicit constraints17%
  • passIdentifies material uncertainty85%
  • passAvoids unsupported claims30%
  • passProduces the required deliverable63%
  • passPrioritises by impact98%
  • passActivation defined by what predicts retention94%
  • passGets the base of every number right67%
Run
Run
#1
Time to output
40 s
Submitted
25 Sept 2026

Results

Every setup we’ve tested on this task, across all cases and repeats, graded on the current checklist. Calibrated: the graders match our PM on 84% of checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6.1 SolwithAPI100.0100.02None
2GPT-6 AstrawithChatGPT97.2100.02None
3GPT-6 LunawithAPI94.485.02None
4Opus 5.5withClaude86.165.02None
5Sonnet 5.5withAPI77.845.02None
6Gemini 3.5 Flash-LitewithGemini61.155.02None

About the task

The PM job

Reviewing a signup and onboarding flow that is losing users.

Why it matters

Anyone can list fifty UX nits. The job is finding the two that explain the drop-off, backed by the funnel data supplied.

What good looks like

  • Ties each issue to the funnel data
  • Prioritises by likely impact
  • Defines activation as the behaviour that predicts retention, not finishing onboarding

Deliberately not measured

  • Accessibility audit completeness
  • Visual redesign
Capability tested

Consequential critique

The failure we’re looking for

A generic UX checklist

Grading

Decision model and LLM judge, calibrated against a blind PM review