Tasks / Design

Activation & onboarding review

Can the model find the friction that matters most and prioritise the fixes?

Measures the modelTask v1.1 · 2 casesDifficulty

What AI gets right here, and what you’ll still have to catch

From 14 graded outputs by 7 models. 50% were usable with at most a quick edit.

Reliably right

  1. Activation defined by what predicts retention100% pass
    Defines activation as three workouts linked to day-60 retention, and evaluates fixes by effect on that event.
    GPT-6 Astra · ChatGPT · Fitness app first week
  2. Prioritises by impact98% pass
    Ranks changes by likely impact on achieving three workouts, aligned with the identified activation threshold.
    GPT-6 Astra · ChatGPT · Fitness app first week
  3. Produces the required deliverable93% pass
    Provides a complete, actionable memo that a growth PM could implement without major gaps.
    GPT-6 Astra · ChatGPT · Fitness app first week

Where it slips

  1. Avoids unsupported claims54% pass
    It presents causal or evaluative claims such as 'successful connection is the gateway' and 'a required technical connection is an especially poor first step' as established fact rather than hypothesis.
    GPT-6 Luna · API · Analytics tool losing users at setup
  2. Uses the supplied evidence correctly54% pass
    It includes an unsupported motivation claim and a miscomputed '4% of trial starters' figure, so not every current-situation statement is supported by the brief or arithmetic.
    Sonnet 5.5 · API · Fitness app first week
  3. Identifies material uncertainty59% pass
    Identifies the lack of step-level data but does not state what result would change the call or how it would affect the ranking.
    Opus 5.5 · Claude · Analytics tool losing users at setup

Case viewer

Read the brief, then put up to three outputs side by side, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

Review our onboarding flow and funnel below. Write a memo to the onboarding squad saying where we are losing people and what to fix, in the order you would fix it, with the reason for each. Keep it under 500 words.

What the model was given4 items: Scenario, Funnel (last 90 days, 12,400 signups), Onboarding flow, screen by screen, Support
ScenarioChartwise is a self-serve product-analytics tool. Most people who sign up are product managers or marketers, not data engineers. Pricing starts at $49 a month after a 14-day free trial.
Funnel (last 90 days, 12,400 signups)Signup 100% → Connect data source 41% → First chart 33% → Invite teammate 9%. Week-2 retention: 62% for users who reach a first chart, 11% otherwise.
Onboarding flow, screen by screen1. Sign up: email, password, company name. 2. Verify email: the flow pauses until the link is clicked. 3. “What's your role?” (seven options). 4. “How big is your team?” (five options). 5. Connect your data: choose Postgres, BigQuery, Snowflake or CSV upload. There is no sample data and no skip. 6. Connection form: host, port, database, username, password and SSL mode. The help link opens a 14-page setup guide. 7. Choose tables: a list of every table in the database. 8. Syncing: a progress bar that can take up to 20 minutes; leaving the page cancels the sync. 9. Pick a dashboard template (twelve options). 10. First chart: a chart builder with an empty canvas. 11. Invite teammates: email fields, with a “Skip for now” link.
SupportThe most common onboarding ticket (41% of them) is some version of “I don't have the database credentials.” The next is “my sync failed” (18%).
What a strong answer doesThe answer key the graders mark against

The data-source connection step (screens 5–8) is the activation bottleneck; prioritise a sample-data or CSV-first path to a first chart, then non-admin credential handoff and a sync that survives leaving the page, ahead of cosmetic fixes.

Case

Mid-level · v1.8 · anonymised real · B2B SaaS, data

The outputs

7 setups answered this brief. Pick up to three to read side by side.

Needs more than a quick edit5 of 9 checks passed

The output presents causal interpretations as certain and omits conditions that could change the priority, so a product manager would need to add caveats and uncertainty before sending. The underlying evidence and reasoning are sound, but it requires more than just a quick edit to align with a demanding senior PM's standard for acknowledging uncertainty.

Output

MEMORANDUM

To: Onboarding Squad From: Product Team Subject: Onboarding Funnel Analysis & Prioritized Fixes

Our last 90 days show 12,400 signups, but we face a steep 59% drop-off immediately after signup (reaching only 41% at data source connection), followed by a massive drop at team invites (down to 9%). Because week-2 retention jumps from 11% to 62% for users who build a first chart, removing friction before that milestone is critical.

Here is where we are losing people and the fixes, ordered by priority:

1. The Data Connection Blockade (Fix First)

  • Where we lose people: We drop 59% of users right at the data connection step.2 Support tickets confirm why: 41% lack database credentials, and 18% experience sync failures. Forcing product managers and marketers—who are not data engineers—to input host, port, SSL modes, and select from raw database tables while blocking the UI for up to 20 minutes is a conversion killer.
  • What to fix:
  • Add a pre-loaded Sample Data / Demo Workspace option so users can experience the product immediately without connecting a live database.
  • Make data connection skippable or move it after the first chart experience.
  • Simplify the CSV upload option or provide pre-built connectors with clear, inline tooltips instead of a 14-page setup guide.

2. The Premature Team Invite Wall (Fix Second)

  • Where we lose people: Reaching a first chart drops to 33%, but inviting teammates plummets to 9%—a catastrophic 24-percentage-point drop right before value is fully realized.1 Users are being asked to advocate for a tool before they’ve even proven its worth to themselves.
  • What to fix:
  • Remove the team invite step from the core linear onboarding path entirely.
  • Relocate the invite prompt to the dashboard or settings menu once the user has successfully built and saved their first chart.

3. Redundant Survey Friction (Fix Third)

  • Where we lose people: Screens 3 ("What's your role?") and 4 ("How big is your team?") add unnecessary micro-friction right after email verification.
  • What to fix:
  • Combine these into a single, clean optional screen or infer them from workspace setup later to streamline time-to-value.

Fixing data connection roadblocks and removing premature team invites will directly protect our week-2 retention (which hinges on reaching that first chart).

What a PM had to fix

  1. 1Numbers wrongRedo the arithmeticTargeted repair

    Source: Funnel (last 90 days, 12,400 signups)Connect data source 41% → First chart 33% → Invite teammate 9%.

    What we’d changeRank the 41% to 33% drop between connecting and a first chart second. The invite comes after the first chart, so it isn't what stands between users and value.

  2. 2Hypothesis stated as factReframe it as a hypothesisQuick edit

    What we’d changeThe 59% is lost somewhere between screens 2 and 8. Say the tickets point to the connection form, and instrument the screens to confirm it.

Grades and run details

Decision model 72 · LLM judge 6 of 10 checks
Decision model checks
  • failUses the supplied evidence correctly20%
  • passAddresses the actual decision87%
  • passRespects explicit constraints68%
  • failIdentifies material uncertainty99%
  • partialAvoids unsupported claims38%
  • passProduces the required deliverable82%
  • passPrioritises by impact64%
  • passActivation defined by what predicts retention98%
  • passGets the base of every number right37%
Run
Run
#1
Time to output
8 s
Submitted
25 Sept 2026

Usable with a quick edit6 of 9 checks passed

The evidence and priorities are usable, but it needs a sentence stating what would change the order and softer labelling of causal claims; no re-checking or redoing of the reasoning is needed.

Output

To: Onboarding Squad Subject: Where onboarding is leaking—and what to fix first

The biggest loss is before a data source is connected: only 41% of signups get that far. Another 8 percentage points drop before a first chart. Just 9% of signups invite a teammate. The strongest activation signal is the chart: week-2 retention is 62% for users who reach one, versus 11% for those who don’t. Prioritize getting people to a useful chart; treat invites as a later step.

Fixes, in priority order

  1. Offer a sample-data path that reaches a real first chart. Let users skip connecting their own data and explore a preloaded dataset, with a clear route to connect later. Many signups are PMs and marketers, and the most common onboarding ticket is missing database credentials (41%). A required technical connection is an especially poor first step for them. This also gives users value while they wait for access.
  1. Make connecting data simpler and recoverable. Replace or supplement the 14-page guide with short, connector-specific instructions and troubleshooting. Validate credentials early, explain the required access, and offer a clear retry path. Don’t cancel a sync when someone leaves the page; send progress and failure notices and let users resume. The credential and sync tickets point to concrete blockers, and successful connection is the gateway to the high-retention chart experience.
  1. Get users to a chart with fewer decisions. Replace the database-wide table list with suggested tables and plain-language descriptions; cut the 12 dashboard templates to a few recommended starting points. Provide a prebuilt chart or guided first-chart flow rather than an empty canvas. The 8-point drop between connection and first chart, plus the sync burden, suggests too much work remains after connection.
  1. Remove early friction that doesn’t create value. Test deferring email verification until after users enter the product, and move role/team questions until they can personalize something useful. These steps currently delay access before users see value.
  1. Make inviting a teammate contextual, not a hurdle. Keep it optional; prompt after a user has made or shared a chart, and explain the benefit. Only 9% of signups invite someone, but chart reach—not inviting—is the clearest retention signal.

Track step conversion and time-to-first-chart by path, especially sample-data versus live connection, before expanding the changes.

Grades and run details

Decision model 89 · LLM judge 7 of 10 checks
Decision model checks
  • passUses the supplied evidence correctly33%
  • passAddresses the actual decision98%
  • passRespects explicit constraints71%
  • partialIdentifies material uncertainty65%
  • partialAvoids unsupported claims48%
  • passProduces the required deliverable97%
  • passPrioritises by impact99%
  • passActivation defined by what predicts retention100%
  • passGets the base of every number right79%
Run
Run
#1
API response time
12 s
Submitted
29 Sept 2026

Needs more than a quick edit3 of 9 checks passed

It exceeds the word limit and contains an unsupported user-capability claim. A PM would need to recheck evidence and trim before use.

Output

To: Onboarding squad Re: Where we lose people and what to fix, in order

Summary: Reaching a first chart is what predicts retention (62% at week 2 versus 11% otherwise). Only 33% of signups get there, and the biggest loss, 59% of signups, happens before they connect data. Most of our users are PMs and marketers who can't complete a database connection alone.

Fixes, in priority order

1. Add sample data and a skip option on the "Connect your data" screen. The most common ticket (41%) is "I don't have the database credentials." We ask non-engineers for host, port and SSL mode before they've seen any value. Sample data lets them reach a chart in minutes, and they can connect real data later. Caveat: sample-data users may retain worse than the 62% figure, which likely reflects real data. Track them as a separate cohort and prompt them to connect real data after their first chart.

2. Make it easy to get credentials from someone else. Add a "Send this to your engineer" option: a shareable link or prefilled email containing the connection form and a short setup checklist. Replace the 14-page guide with inline, per-field help. Promote CSV upload as the no-engineer path, since it's currently one of four equal options.

3. Fix the sync experience and its failures. Syncs run up to 20 minutes and cancel if the user leaves. Run them in the background, send an email when they finish, and let users start exploring once the first tables land. Then look at the 18% of tickets that say "my sync failed" and surface specific error messages. In the table-selection step, preselect a sensible default instead of listing every table.

4. Get people to a first chart without an empty canvas. We show twelve templates, then a blank chart builder. Cut the choice to two or three based on role, and auto-generate a first chart from the synced data (or sample data) that the user can edit. This targets the 41% → 33% drop and shortens time to value.

5. Remove friction at the front. Let users into the product before email verification, and enforce it later (e.g., before inviting teammates). Move the role question ahead of template selection, where it drives the default, and drop or defer team size unless something uses it. These are cheap fixes, but I'm ranking them below the others because I can't yet show they're a big loss.

6. Move the invite prompt to after the first chart. The 9% invite rate is 27% of users who reach a chart, so it's mostly a downstream symptom. Fix activation first. Then trigger the invite prompt right after a first chart is created, with a message like "Share this chart," rather than as a generic final step.

Measure first The funnel jumps from signup straight to data connection, so we can't see how many people drop at email verification, role, team size, or the connection form. Add per-screen events this sprint, before changing anything, so we can confirm the ordering above and measure each fix.

Success metrics: signup → first chart (33% today), share of users who connect real data within 7 days, sync failure rate, and week-2 retention split by sample-data versus real-data users.

Claims the judge couldn’t find in the brief

  • Most users are PMs and marketers who can't complete a database connection alone.

Grades and run details

Decision model 83 · LLM judge 3 of 10 checks
Decision model checks
  • failUses the supplied evidence correctly7%
  • passAddresses the actual decision99%
  • passRespects explicit constraints24%
  • passIdentifies material uncertainty90%
  • partialAvoids unsupported claims43%
  • passProduces the required deliverable93%
  • passPrioritises by impact95%
  • passActivation defined by what predicts retention100%
  • passGets the base of every number right42%
Run
Run
#1
API response time
15 s
Submitted
29 Sept 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 4

Uses the supplied evidence correctlyMixedRightWrong
Gemini 3.5 Flash-Lite · Gemini

All factual claims about the current situation are directly from the brief or context, or derived correctly by arithmetic.

GPT-6 Luna · API

All factual statements about the current situation are taken correctly from the brief and supplied context, with no invented facts.

Sonnet 5.5 · API

It presents as fact that PMs and marketers cannot complete a database connection alone, which the supplied context does not establish.

Respects explicit constraintsRightRightMixed
Gemini 3.5 Flash-Lite · Gemini

The output is a memo to the onboarding squad, well under 500 words, and addresses the requested points.

GPT-6 Luna · API

It is a memo to the onboarding squad, well under 500 words, and respects the requested form and reader.

Sonnet 5.5 · API

The memo is approximately 541 words, exceeding the explicit 500-word limit.

Identifies material uncertaintyWrongWrongMixed
Gemini 3.5 Flash-Lite · Gemini

The output does not identify unknowns that could change the decision (e.g., whether sample data actually drives first-chart creation) or say how they could be resolved.

GPT-6 Luna · API

It does not name specific unknowns that could change the priorities or say how resolving them would alter the call.

Sonnet 5.5 · API

It names sample-data retention and missing funnel events but does not specify what data result would change the prioritisation.

Produces the required deliverableRightRightMixed
Gemini 3.5 Flash-Lite · Gemini

The memo is complete, in the right form, under the word limit, and the squad could act on it with minimal edits.

GPT-6 Luna · API

The required memo is complete, actionable, and usable by the onboarding squad as written or with light edits.

Sonnet 5.5 · API

It is a usable memo in form and audience, but it does not meet the required length constraint.

All got wrong 1

Avoids unsupported claimsWrongWrongWrong
Gemini 3.5 Flash-Lite · Gemini

Causal claims like 'conversion killer', 'premature team invite wall', and 'unnecessary micro-friction' are stated as facts rather than labelled as hypotheses or inferences.

GPT-6 Luna · API

It presents causal or evaluative claims such as 'successful connection is the gateway' and 'a required technical connection is an especially poor first step' as established fact rather than hypothesis.

Sonnet 5.5 · API

The user-capability claim and 'mostly downstream symptom' are stated as established fact rather than labelled interpretations.

All mixed 1

Addresses the actual decisionMixedMixedMixed
Gemini 3.5 Flash-Lite · Gemini

The output commits to a ranked order but does not state what result or condition would change that answer, as required by the criterion.

GPT-6 Luna · API

The memo commits to a clear ranked order but never states what result or condition would change that order; the closing sentence is only about expanding changes.

Sonnet 5.5 · API

It commits to a ranked order but does not state a result or condition that would change the answer, only that per-screen events would confirm it.

All got right 3

Prioritises by impactRightRightRight
Gemini 3.5 Flash-Lite · Gemini

Fixes are ranked by likely impact on reaching the first-chart activation milestone that predicts retention.

GPT-6 Luna · API

Fixes are ranked by likely impact on reaching a first chart, tied to the retention signal.

Sonnet 5.5 · API

Fixes are explicitly ranked by likely impact on reaching first chart, with reasons for each.

Activation defined by what predicts retentionRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The memo explicitly names reaching a first chart as the activation event, links it to the retention jump, and uses that to prioritize fixes.

GPT-6 Luna · API

It names first chart as the activation behaviour based on 62% vs 11% retention and ranks fixes by their effect on it.

Sonnet 5.5 · API

It defines first chart as the activation behaviour using the retention split and ranks fixes by effect on that event.

Gets the base of every number rightRightRightRight
Gemini 3.5 Flash-Lite · Gemini

All percentages and differences are correctly calculated from the supplied data, and the base for support-ticket figures is clearly ticket counts.

GPT-6 Luna · API

All percentages use the correct denominators, and the 41% ticket figure is understood as a share of onboarding tickets rather than users.

Sonnet 5.5 · API

Derived percentages use the correct denominators, including 59% before data, 27% invite of first-chart users, and the 41% to 33% step.

Results

Every setup we’ve tested on this task, across all cases and repeats, graded on the current checklist. Calibrated: the graders match our PM on 82% of checks.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6.1 SolwithAPI100.0100.02None
2GPT-6 AstrawithChatGPT94.4100.02None
3GPT-6 LunawithAPI94.485.02None
4Opus 5.5withClaude80.665.02None
5Gemini 3.5 Flash-LitewithGemini69.455.02None
6Sonnet 5.5withAPI77.845.02None
7Gemini 3.8 FlashwithAPI69.450.02None

About the task

The PM job

Reviewing a signup and onboarding flow that is losing users.

Why it matters

Anyone can list fifty UX nits. The job is finding the two that explain the drop-off, backed by the funnel data supplied.

What good looks like

  • Ties each issue to the funnel data
  • Prioritises by likely impact
  • Defines activation as the behaviour that predicts retention, not finishing onboarding

Deliberately not measured

  • Accessibility audit completeness
  • Visual redesign
Capability tested

Consequential critique

The failure we’re looking for

A generic UX checklist

Grading

Decision model and LLM judge, calibrated against a blind PM review