Diagnose a retention problem

Weekly actives are flat because retention broke in week 5, not because of summer

To: Leo From: Activation PM Re: Why weekly actives are flat, and why we shouldn't double paid acquisition yet

Summary

Weekly actives aren't flat because of summer. Retention dropped sharply for every cohort that signed up from week 5 onward. That is the week we removed the goal-setting step from sign-up. The win-back campaign has hidden the damage by pulling back about 1,600 lapsed users a week, and most of them leave again within two weeks. Doubling acquisition now would pour more users into a bucket that leaks about a third faster than it did in July.

What's happening

1. Churn is up 40%, and it's offset by low-quality reactivations.

Weekly churn (users active last week but not this week, divided by last week's actives):

Returning users jumped from about 1,000 to about 2,600 a week in week 6, the week win-back emails started. New users are up 30%. Together these just cancel the extra churn, which is why the headline stays near 52,000.

2. The break follows sign-up cohort, not the calendar.

The change is a clean step at the week-5 cohort, with no gradual drift.

3. The evidence argues against seasonality.

• Cohorts behave differently in the same calendar weeks. If summer were the cause, everyone would dip together. Instead, the weeks 1–4 cohorts held at about 30% through all of August, while the cohorts that joined in August decayed fast.
• July was summer too. Churn was at its lowest then.
• September didn't help. Weeks 10–13 run into late September, and churn is at its worst.

4. The week-5 sign-up change is the most likely cause.

Removing "What do you want to achieve, and how many days a week?" cut sign-up from 3 minutes to 1. Completion rose from 61% to 74%. But we stopped asking users to commit to a goal and a weekly frequency, which is the core of a weekly-plan habit app.

The pattern fits: - First-week retention went up slightly (58% vs 55%), because the initial experience is fine. - From week 3 onward, users without a plan drift away.

Part of the drop may be compositional: the extra 13 points of completers may be lower intent. Either way, the cause points to the same step. Nothing else in the log lines up. The week-3 workout library shows no effect, and paid social only started in week 9.

5. The win-back campaign flatters the topline.

Only 12% of win-back returners are still active two weeks later, against 41% of users who came back on their own before the campaign. We're also giving away a free week of Premium to get them. Without the campaign, weekly actives would be roughly 1,500 lower and still falling.

Why doubling acquisition doesn't work right now

Per 1,000 sign-ups, about 300 were still active at week 8 before week 5. Now it's about 200. That means:

• We need 50% more sign-ups just to keep the same number of retained users.
• Sign-ups are up about 28%, which isn't keeping pace.
• Cost per retained user is up roughly 50% at today's acquisition cost. That's before any rise in marginal cost from doubling spend, which usually happens.

Each extra cohort loses about 80% of its users within 8 weeks. Doubling spend would mostly buy churn.

What we should do instead

1. Fix sign-up first (start this week).

Run an A/B test with three arms: - (a) the current flow; - (b) the old flow with goal-setting; - (c) goal-setting moved to after account creation, in the first session.

Option (c) could keep most of the completion gain and restore the plan. Judge the test on week-3 retention, where the gap (about 37% vs 30%) shows up within three weeks, and confirm at week 4.

If we want to move faster, we could simply revert to (b) while testing (c). We'd give back some completion, but we'd stop losing about 10 points of week-8 retention on every new cohort.

2. Hold paid acquisition at its current level until retention recovers.

Revisit once new cohorts are back to about 33% at week 4. When we do scale, check whether paid-social cohorts retain as well as organic ones. We don't yet have cohort data split by channel.

3. Put the win-back campaign behind a holdout.

Withhold the email from 20% of eligible users to measure: - true incremental weekly actives; - Premium cannibalisation; - whether the free week leads to paid conversion.

Also try a version without the free-Premium offer, or one that sends people back into a plan: "Pick your days for this week".

4. Change how we report growth.

Make cohort retention at weeks 4 and 8, plus the weekly churn rate, our headline health metrics alongside weekly actives. Report reactivations separately. A flat topline hid a 40% rise in churn for two months.

Caveats

• Causation isn't proven. The week-5 timing is strong evidence, but the A/B test is what will prove it.
• The two data sources don't quite match. Cohort sign-up counts are slightly lower than "new users" in the growth accounting (for example, 6,000 vs 6,034 in week 1), so the two may define new users slightly differently. The gap is too small to change the conclusion.

Happy to walk through the cohort curves together. I'd recommend we agree on the sign-up test before any change to the paid budget.

To: Leo | From: Activation PM | Re: Why WAU is flat, and why I'd hold off on doubling paid

Bottom line: This isn't a summer dip. Retention has been deteriorating since early August, and new sign-ups and win-back emails are hiding it. Doubling paid now would mostly buy users who leave faster.

1. What's happening

WAU = new + returning + retained users, which reconciles exactly every week. Across weeks 1–13, WAU moved only +227 (51,891 → 52,118). That flatness is three movements netting out:

• New users: +1,813 (up 30%, as you said)
• Returning users: +1,662, nearly all from win-back
• Retained users (active last week and this week): −3,248

Retained users held near 44.5–45k through week 4, then stepped down from week 5 and kept sliding to 41.6k. Churn (last week's actives who didn't return) rose from about 14% of prior-week actives in weeks 2–4 to about 20% in weeks 11–13.

2. Why it isn't summer

Seasonality would hit all cohorts alike. It doesn't. Cohorts 1–4 (signed up July to late July) had week-4 retention of 33–34%. Cohorts 5 onward (from the week of 3 August, the first week of the shorter sign-up) drop to 24–25% and stay there, even though cohorts 2–4 went through the same weather. The break lines up with the week-5 release, which removed the goal-setting step.

The funnel math shows the change is a net loss. Per 100 visitors:

We're converting more visitors into accounts, but by week 8 each signup is about 20% less likely to still be active. We're getting more users who don't build the habit.

3. Win-back is flattering the numbers

Win-back adds about 1,600 returning users a week. Of users who came back through a win-back email, 12% were still active two weeks later. Of users who came back on their own, 41% were. At 1,600 a week, that's roughly 190 durable returners instead of about 650 if they'd come back organically. We're also giving away a free week of Premium. "Returning users" now looks like a health signal, but most of it is low-quality.

The week-9 paid social increase (+25%) isn't visible as a break in the new-user trend, so I can't yet say what paid is adding.

What I'd do instead

1. Hold the paid increase. Keep current spend. Revisit when cohort week-4 retention is back above 30%.

1. Restore the goal-setting step, or move it into the app after the first workout. Run it as a holdout (50/50, old flow vs new), and judge on week-4 and week-8 retention per signup, not completion. Using the cohort tables, bringing cohort retention back to about 30% at week 4 is worth roughly 3–4k weekly actives at steady state. That's about the size of the whole retained-user decline. This is my rough estimate and needs confirming with the holdout.

1. Find the churn in tenured users. Most of the 41k retained base isn't new. Segment weeks 4–13 churn by tenure, plan usage, workout type, and platform. Also check whether win-back returners churn quickly the following week, since they add to churn in the same weeks.

1. Rework win-back. Test a smaller, conditional offer, or a guided re-entry workout instead of a free Premium week. Measure two-week retention against a holdout. Keep it only if the incremental retained users justify the cost.

1. Change what we report. Show the WAU decomposition (new, returning, retained) and cohort retention by signup week every week. Stop counting returners as growth.

On the summer theory: it's testable. If we have last year's data for the same weeks, compare churn rates. If summer were the cause, we'd expect last year's churn to rise over these weeks too, and we'd expect older cohorts to show the same pattern as new ones. The data so far points to the week-5 change.

Ask: Can we set up the sign-up holdout this week and pull the tenure-level churn cut before any paid budget decision?

To: Leo, Head of Growth From: PM, Activation Subject: Diagnosis: Why WAU is flat (and why doubling paid acquisition right now will burn cash)

Leo, you suggested doubling paid acquisition to push through flat summer WAU. I’ve dug into the numbers, and the data tells a different story.

Weekly actives are flat (averaging ~52,000 for 13 weeks), but sign-ups are up 30% (from ~6,000 to ~7,800). The issue isn't seasonal churn—it's that our onboarding changes broke cohort retention, and our win-back campaigns are masking the leak.

Doubling down on paid acquisition right now will just pour more water into a bucket with a widening hole. Here is the diagnosis, the evidence, and what we should do instead.

---

1. What’s actually happening?

#### A. Onboarding optimization created low-intent users (Week 5) In Week 5, we removed the goal-setting step to cut sign-up time from 3 minutes to 1. Completion jumped from 61% to 74%, and sign-ups grew.

However, look at the cohort data (`weeklysignupcohorts.csv`). For Cohorts 1–4 (pre-change), Week 1 retention hovered around 55%, and Week 4 retention was ~34%. From Cohort 5 onward, Week 1 retention jumped to ~58% (thanks to frictionless sign-up), but Week 4 retention cratered from ~34% down to ~25%.

By removing goal-setting, we optimized for top-of-funnel conversion at the expense of user intent. We are acquiring users who don't know why they are here or what plan fits them, leading them to drop off heavily by week 4.

#### B. Core retention is quietly declining Look at the growth accounting data (`weeklygrowthaccounting.csv`). Total `retained_users` (users active week-over-week) dropped steadily from ~44,800 in Week 1 to ~41,600 in Week 13, a loss of over 3,200 loyal weekly actives.

#### C. Win-backs are masking the bleeding So why is WAU flat if retained users dropped by 3,200? Look at `returning_users`. From Weeks 1–5, returning users averaged ~980 per week. Starting in Week 6 (when the win-back email launched), returning users spiked to ~2,600 per week.

The win-back campaign is successfully dragging churned users back, propping up our headline WAU. But it’s a leaky bucket: lifecycle data shows that only 12% of win-back users are active two weeks later (compared to 41% of organic returning users). They churn right back out.

---

2. Why "Summer" isn't to blame

Your theory about summer (outdoors, holidays) would manifest as broad seasonal churn across all cohorts. But our data shows: 1. Older cohorts (1–4) maintained stable long-term retention. 2. The drop in retention is strictly tied to Cohort 5 onwards—the exact week we shipped the shorter sign-up flow.

It is product-led churn, not seasonal.

---

3. What we should do instead of doubling paid acquisition

Before we scale ad spend, we need to fix activation and stop relying on expensive band-aids. Let’s execute a 3-step pivot:

1. Re-introduce intent-matching to onboarding (without hurting conversion): We need to qualify users so they receive the right weekly plan. Instead of a heavy 3-minute form, let’s test a lightweight, single-tap goal selector or defer personalization to the end of the first workout. We must restore Week 4 retention from 25% back toward 34%.
2. Refine the win-back targeting: Pumping win-back emails to all churned users after 21 days is bringing back low-intent users who churn in 14 days. Let’s restrict win-backs to users who reached a milestone or completed at least 3 workouts before churning, matching the profile of our 41% organic returners.
3. Optimize the Week 3–4 activation loop: We shipped 40 new routines in Week 3, but post-onboarding drop-off is neutralizing it. We need to focus activation efforts on guiding new users to these routines before they hit the Week 3 cliff.

Bottom line: Let's hold off on doubling paid acquisition for 4 weeks. If we fix the onboarding leak and improve post-signup intent, organic growth + recovered retention will naturally push us past our WAU plateau without wasting ad dollars on leaky traffic. Let's sync on this today.