Usable with a quick edit6 of 9 checks passed
The memo is actionable and fully reasoned; a PM could trim a few words to meet the word limit without redoing any analysis.
Output
Onboarding: where we lose people and what to fix first
Sep 25, 2026 · @Martin Slaney
To: Onboarding squad
Bottom line
Onboarding works for people who reach a first chart: 62% of them are still here in week 2, against 11% of everyone else. Two-thirds of signups never get there, and most of those stop at "Connect your data", where we ask product managers and marketers for database credentials they don't have.
Where we lose people (last 90 days, 12,400 signups)
- Signup → connect data: 59% lost (\~7,300 people). The biggest leak by far. The top support ticket (41%) is "I don't have the database credentials."
- Connect → first chart: \~990 more lost, a fifth of those who connected. Syncs take up to 20 minutes, die if you leave the page, and "my sync failed" is the second-biggest ticket (18%).
- First chart → invite: 33% → 9%. Real, but downstream of the first two.
Caveat: we have no step-level data for screens 2–8, so we can't split the 59% between verification, the two questions and the connection form. The ticket data points hard at the form.
What to fix, in order
- Let people reach a first chart without their own data. Add a sample dataset and a skip on screen 5, and land them on a ready-made chart. Why: it goes around the step that loses most people, and the first chart is the moment that predicts retention.
- Let them hand off the credentials step. "Ask a teammate to connect" sends an engineer a secure link to fill in the connection form. Why: it answers the 41% ticket directly and turns the blocker into an invite.
- Make syncing forgiving. Run it in the background, keep it going when they leave, email them when it's ready, and load a small sample of rows first so a chart appears in minutes. Show plain-language errors with a retry. Why: a 20-minute wait that punishes leaving the page is where connected users die, and it's the 18% ticket.
- Don't start from an empty canvas. Use the role answer from screen 3 to suggest two or three templates instead of twelve, and draw the first chart automatically. Why: we already ask the role and do nothing with it; a blank builder is hard for non-analysts.
- Trim the steps before connecting. Let people in before they verify their email (verify within a day), and drop or defer the team-size question. Why: cheap to do, but unmeasured, so instrument first rather than assume it's a big win.
- Move the invite prompt to after value is shown, such as when someone saves or shares a chart. Why: people invite colleagues to something useful, not to a setup screen.
How we'll know it worked
- Instrument every screen, 1 to 11, before shipping anything.1
- Judge fix 1 by how many sample-data users go on to connect real data and are retained in week 2, not by first-chart rate. A chart of sample data can lift that metric without anyone sticking around, so don't assume the 62% carries over.
What a PM had to fix
1OtherFix itQuick edit
What we’d changeInstrument alongside fix 1 rather than before it, and merge fixes 1 and 4, which both land the user on an auto-drawn chart.
Check by check
Mixed · 3
- Respects explicit constraintsThe output is exactly 500 words; the brief requires under 500 words.The two graders disagreed on this one.
- Identifies material uncertaintyIdentifies the lack of step-level data but does not state what result would change the call or how it would affect the ranking.The two graders disagreed on this one.
- Produces the required deliverableThe memo is complete and usable but exceeds the word limit: it is 500 words, not under 500.The two graders disagreed on this one.
Got right · 6
- Uses the supplied evidence correctlyAll factual statements about the current situation are directly from the supplied context or derived by correct arithmetic.
- Addresses the actual decisionCommits to a clear, ranked order of fixes early in the memo, framed for the onboarding squad.
- Avoids unsupported claimsInterpretations and causes are presented as reasoning (often within 'Why' sections) and not as established facts.
- Prioritises by impactFixes are ranked by impact on activation, with the largest drop addressed first.
- Activation defined by what predicts retentionIdentifies the first chart as the activation event that predicts retention (62% vs 11%) and ranks fixes by their effect on reaching it.
- Gets the base of every number rightAll derived figures (59%, ~7,300, ~990, a fifth) are computed from the correct steps and denominators, with bases stated where needed.
Grades and run details
Decision model 94 · LLM judge 7 of 10 checks
Decision model checks
- passUses the supplied evidence correctly19%
- passAddresses the actual decision92%
- passRespects explicit constraints28%
- passIdentifies material uncertainty95%
- partialAvoids unsupported claims37%
- passProduces the required deliverable94%
- passPrioritises by impact98%
- passActivation defined by what predicts retention100%
- passGets the base of every number right51%
Run
- Run
- #1
- Time to output
- 40 s
- Submitted
- 25 Sept 2026