Usable with a quick edit11 of 12 checks passed
The diagnosis is complete, evidence-based, and actionable; a PM could use it with only minor wording tweaks, without rechecking data or redoing analysis.
Output
Weekly actives are flat because retention broke in week 5, not because of summer
To: Leo From: Activation PM Re: Why weekly actives are flat, and why we shouldn't double paid acquisition yet
Summary
Weekly actives aren't flat because of summer. Retention dropped sharply for every cohort that signed up from week 5 onward. That is the week we removed the goal-setting step from sign-up. The win-back campaign has hidden the damage by pulling back about 1,600 lapsed users a week, and most of them leave again within two weeks. Doubling acquisition now would pour more users into a bucket that leaks about a third faster than it did in July.
What's happening
1. Churn is up 40%, and it's offset by low-quality reactivations.
Weekly churn (users active last week but not this week, divided by last week's actives):
| Period | Weekly churn | Users churned per week |
|---|---|---|
| Weeks 2–4 | 12.7–14.3% | ~7,000 |
| Week 6 | 17.7% | ~9,100 |
| Weeks 9–13 | 18.3–19.8% | ~10,300 |
Returning users jumped from about 1,000 to about 2,600 a week in week 6, the week win-back emails started. New users are up 30%. Together these just cancel the extra churn, which is why the headline stays near 52,000.
2. The break follows sign-up cohort, not the calendar.
| Cohorts | Week 1 retention | Week 4 retention | Week 8 retention |
|---|---|---|---|
| Weeks 1–4 | ~55% | ~34% | ~30% |
| Weeks 5 onward | ~58% | ~25% | ~20% |
The change is a clean step at the week-5 cohort, with no gradual drift.
3. The evidence argues against seasonality.
- Cohorts behave differently in the same calendar weeks. If summer were the cause, everyone would dip together. Instead, the weeks 1–4 cohorts held at about 30% through all of August, while the cohorts that joined in August decayed fast.
- July was summer too. Churn was at its lowest then.
- September didn't help. Weeks 10–13 run into late September, and churn is at its worst.
4. The week-5 sign-up change is the most likely cause.
Removing "What do you want to achieve, and how many days a week?" cut sign-up from 3 minutes to 1. Completion rose from 61% to 74%. But we stopped asking users to commit to a goal and a weekly frequency, which is the core of a weekly-plan habit app.
The pattern fits: - First-week retention went up slightly (58% vs 55%), because the initial experience is fine. - From week 3 onward, users without a plan drift away.
Part of the drop may be compositional: the extra 13 points of completers may be lower intent. Either way, the cause points to the same step. Nothing else in the log lines up. The week-3 workout library shows no effect, and paid social only started in week 9.
5. The win-back campaign flatters the topline.
Only 12% of win-back returners are still active two weeks later, against 41% of users who came back on their own before the campaign. We're also giving away a free week of Premium to get them. Without the campaign, weekly actives would be roughly 1,500 lower and still falling.
Why doubling acquisition doesn't work right now
Per 1,000 sign-ups, about 300 were still active at week 8 before week 5. Now it's about 200. That means:
- We need 50% more sign-ups just to keep the same number of retained users.
- Sign-ups are up about 28%, which isn't keeping pace.
- Cost per retained user is up roughly 50% at today's acquisition cost. That's before any rise in marginal cost from doubling spend, which usually happens.
Each extra cohort loses about 80% of its users within 8 weeks. Doubling spend would mostly buy churn.
What we should do instead
1. Fix sign-up first (start this week).
Run an A/B test with three arms: - (a) the current flow; - (b) the old flow with goal-setting; - (c) goal-setting moved to after account creation, in the first session.
Option (c) could keep most of the completion gain and restore the plan. Judge the test on week-3 retention, where the gap (about 37% vs 30%) shows up within three weeks, and confirm at week 4.
If we want to move faster, we could simply revert to (b) while testing (c). We'd give back some completion, but we'd stop losing about 10 points of week-8 retention on every new cohort.
2. Hold paid acquisition at its current level until retention recovers.
Revisit once new cohorts are back to about 33% at week 4. When we do scale, check whether paid-social cohorts retain as well as organic ones. We don't yet have cohort data split by channel.
3. Put the win-back campaign behind a holdout.
Withhold the email from 20% of eligible users to measure: - true incremental weekly actives; - Premium cannibalisation; - whether the free week leads to paid conversion.
Also try a version without the free-Premium offer, or one that sends people back into a plan: "Pick your days for this week".
4. Change how we report growth.
Make cohort retention at weeks 4 and 8, plus the weekly churn rate, our headline health metrics alongside weekly actives. Report reactivations separately. A flat topline hid a 40% rise in churn for two months.
Caveats
- Causation isn't proven. The week-5 timing is strong evidence, but the A/B test is what will prove it.
- The two data sources don't quite match. Cohort sign-up counts are slightly lower than "new users" in the growth accounting (for example, 6,000 vs 6,034 in week 1), so the two may define new users slightly differently. The gap is too small to change the conclusion.
Happy to walk through the cohort curves together. I'd recommend we agree on the sign-up test before any change to the paid budget.
Grades and run details
Decision model 83 · LLM judge 100
Decision model checks
- passUses the supplied evidence correctly13%
- passAddresses the actual decision99%
- partialRespects explicit constraints28%
- passIdentifies material uncertainty97%
- partialAvoids unsupported claims38%
- passProduces the required deliverable93%
- passFinds the onboarding change100%
- passSees the win-back masking churn100%
- passUnderstands before it justifies100%
- failGets the base of every number right24%
- passSplits growth into new, lost and returning100%
- passReads the shape of the curves76%
Run
- Run
- #1
- Time to output
- 43 s
- Submitted
- 5 Oct 2026