Diagnose a retention problem

Weekly actives are flat because retention broke in week 5, not because of summer

To: Leo From: Activation PM Re: Why weekly actives are flat, and why we shouldn't double paid acquisition yet

Summary

Weekly actives aren't flat because of summer. Retention dropped sharply for every cohort that signed up from week 5 onward. That is the week we removed the goal-setting step from sign-up. The win-back campaign has hidden the damage by pulling back about 1,600 lapsed users a week, and most of them leave again within two weeks. Doubling acquisition now would pour more users into a bucket that leaks about a third faster than it did in July.

What's happening

1. Churn is up 40%, and it's offset by low-quality reactivations.

Weekly churn (users active last week but not this week, divided by last week's actives):

Returning users jumped from about 1,000 to about 2,600 a week in week 6, the week win-back emails started. New users are up 30%. Together these just cancel the extra churn, which is why the headline stays near 52,000.

2. The break follows sign-up cohort, not the calendar.

The change is a clean step at the week-5 cohort, with no gradual drift.

3. The evidence argues against seasonality.

• Cohorts behave differently in the same calendar weeks. If summer were the cause, everyone would dip together. Instead, the weeks 1–4 cohorts held at about 30% through all of August, while the cohorts that joined in August decayed fast.
• July was summer too. Churn was at its lowest then.
• September didn't help. Weeks 10–13 run into late September, and churn is at its worst.

4. The week-5 sign-up change is the most likely cause.

Removing "What do you want to achieve, and how many days a week?" cut sign-up from 3 minutes to 1. Completion rose from 61% to 74%. But we stopped asking users to commit to a goal and a weekly frequency, which is the core of a weekly-plan habit app.

The pattern fits: - First-week retention went up slightly (58% vs 55%), because the initial experience is fine. - From week 3 onward, users without a plan drift away.

Part of the drop may be compositional: the extra 13 points of completers may be lower intent. Either way, the cause points to the same step. Nothing else in the log lines up. The week-3 workout library shows no effect, and paid social only started in week 9.

5. The win-back campaign flatters the topline.

Only 12% of win-back returners are still active two weeks later, against 41% of users who came back on their own before the campaign. We're also giving away a free week of Premium to get them. Without the campaign, weekly actives would be roughly 1,500 lower and still falling.

Why doubling acquisition doesn't work right now

Per 1,000 sign-ups, about 300 were still active at week 8 before week 5. Now it's about 200. That means:

• We need 50% more sign-ups just to keep the same number of retained users.
• Sign-ups are up about 28%, which isn't keeping pace.
• Cost per retained user is up roughly 50% at today's acquisition cost. That's before any rise in marginal cost from doubling spend, which usually happens.

Each extra cohort loses about 80% of its users within 8 weeks. Doubling spend would mostly buy churn.

What we should do instead

1. Fix sign-up first (start this week).

Run an A/B test with three arms: - (a) the current flow; - (b) the old flow with goal-setting; - (c) goal-setting moved to after account creation, in the first session.

Option (c) could keep most of the completion gain and restore the plan. Judge the test on week-3 retention, where the gap (about 37% vs 30%) shows up within three weeks, and confirm at week 4.

If we want to move faster, we could simply revert to (b) while testing (c). We'd give back some completion, but we'd stop losing about 10 points of week-8 retention on every new cohort.

2. Hold paid acquisition at its current level until retention recovers.

Revisit once new cohorts are back to about 33% at week 4. When we do scale, check whether paid-social cohorts retain as well as organic ones. We don't yet have cohort data split by channel.

3. Put the win-back campaign behind a holdout.

Withhold the email from 20% of eligible users to measure: - true incremental weekly actives; - Premium cannibalisation; - whether the free week leads to paid conversion.

Also try a version without the free-Premium offer, or one that sends people back into a plan: "Pick your days for this week".

4. Change how we report growth.

Make cohort retention at weeks 4 and 8, plus the weekly churn rate, our headline health metrics alongside weekly actives. Report reactivations separately. A flat topline hid a 40% rise in churn for two months.

Caveats

• Causation isn't proven. The week-5 timing is strong evidence, but the A/B test is what will prove it.
• The two data sources don't quite match. Cohort sign-up counts are slightly lower than "new users" in the growth accounting (for example, 6,000 vs 6,034 in week 1), so the two may define new users slightly differently. The gap is too small to change the conclusion.

Happy to walk through the cohort curves together. I'd recommend we agree on the sign-up test before any change to the paid budget.

Leo — I wouldn’t double paid acquisition yet. We have a retention leak, not just a summer growth slowdown. More new users and campaign-driven returns are masking deteriorating retention. The clearest break coincides with the week-5 onboarding change, though we need an experiment to establish causality.

What’s actually happening

Weekly actives remain around 52,000 because inflows are replacing increasingly large losses:

New users rose 30%, but weekly churn rose 44%. The share of the prior week’s actives retained fell from approximately 86.3% to 80.2%. In week 13, 10,482 new and returning users barely exceeded 10,279 churned users: net growth was just 203.

Flat actives are therefore concealing a less durable user base.

The evidence points to an activation problem

1. New cohorts look better initially, then retain substantially worse. For signup cohorts 1–4, first-following-week retention averaged 55.1%; for cohorts 5–12, it averaged 58.0%. But by the fourth following week, retention falls from 33.8% for cohorts 1–4 to 24.8% for cohorts 5–9. At eight weeks, the earlier cohorts retain roughly 30%, versus 19.9% for cohort 5.

That is not simply fewer people trying Stride. More are getting through the front door, but fewer form a lasting habit.

2. The break aligns with removing goal-setting in week 5. Shorter signup raised completion from 61% to 74%, but removed the step asking users what they wanted to achieve and how often they would exercise. For a habit app built around a weekly plan, that step may have helped users commit to a relevant routine.

The timing makes this our leading hypothesis—not proven causality. We should also check acquisition mix and other changes. However, the paid-budget increase occurred in week 9, after the cohort deterioration began. “Summer” alone does not explain this sharp onboarding-aligned break.

3. Win-back is propping up the headline metric, not repairing retention. Returning users jumped from about 1,000 to 2,600 a week when emails launched in week 6. Yet only 12% of email-driven returners remained active two weeks later, versus 41% of earlier organic returners. These are different populations, so this is not a causal campaign comparison, but it clearly warns against treating every return as durable recovery.

What we should do before spending more

1. Hold acquisition spend steady while testing onboarding. Randomize the current short signup against a lightweight goal-setting step that produces a personalized weekly plan. Preserve the speed improvement where possible.

1. Optimize for durable activation, not signup completion. Make four-week retention a primary outcome, alongside first workout, plan adoption and repeat workout behavior. Measure retained users per signup and per signup starter so a longer flow cannot “win” merely by filtering people out. Track Premium conversion as a guardrail.

1. Locate the failure point. Segment cohorts by acquisition channel, campaign and onboarding exposure. Compare where users stop: before their first workout, after one workout, or when beginning the next weekly plan. Reconcile the signup and new-active counts before calculating acquisition efficiency.

1. Judge win-back on incremental lasting activity. Use a randomized holdout and measure two- and four-week retention, not email-driven returns alone. Test messages that restore a realistic plan rather than relying only on free Premium.

Decision: scale acquisition once we demonstrate improved multiweek retention and acceptable acquisition economics. Doubling spend now risks buying more users into a leaking habit loop—and making flat actives look healthier without fixing the underlying problem.

Leo — I wouldn’t double paid acquisition yet. The flat WAU is masking a retention problem, not showing that summer is the whole story.

• The active base is eroding while inflows hide it. From week 1 to week 13, new users rose from 6,034 to 7,847 (+30%), but retained users fell from 44,884 to 41,636 (−7%). Weekly churn rose from 7,116 to 10,279. WAU stayed near 52,000 because more new users—and, from week 6, more returning users—offset that loss.
• New-user retention deteriorates sharply after week 4. For the first four signup cohorts, week-4 retention was 33–34%, and week-8 retention about 30%. For cohorts in weeks 5–8, week-4 retention was 24–25%; week-5 retention was about 22%, versus about 32% for the earlier cohorts at the same age. Week-1 retention actually improved (roughly 54–55% to 58%), then users fell away faster. So sign-ups are easier to get, but fewer people build a lasting workout habit.
• The timing points to a product change worth investigating. In week 5, we removed the goal-setting step. Sign-up completion rose from 61% to 74%, while the cohort retention pattern worsened immediately. That’s strong evidence to investigate whether the change brought in less-committed users or removed useful personalization—not proof that it caused the decline. Seasonality may contribute, but it doesn’t explain away the sharp cohort break.
• Win-back lifts WAU more than durable retention. Returning users rose from about 1,000 to about 2,600 a week after the week-6 campaign. But only 12% of email-returners were active two weeks later, compared with 41% of people who returned on their own. The campaign may be adding short-lived activity, not repairing the underlying habit problem.

What I’d do before scaling spend:

1. Test the onboarding change now. A/B test restoring goal-setting (or a shorter version) against the current flow. Measure signup completion alongside week-1 activation, week-4/8 retention, and workouts completed—not just registrations.
2. Improve the first-week habit loop. Give users a clear, personalized weekly plan and a timely next-workout prompt. Track results by signup week, onboarding version, and acquisition channel.
3. Put win-back on a holdout. Measure incremental, sustained activity and Premium conversion against users who don’t receive the offer; don’t count the initial return as success.
4. Hold off on doubling acquisition. The week-9 paid increase came after the retention decline had already begun. Scale only when newer cohorts retain better and paid users show acceptable downstream value.

Fixing the leak first gives us a better chance of turning additional acquisition into durable actives rather than more short-lived sign-ups.