Tasks / Growth

Diagnose a retention problem

Can the model work out from raw data what's really happening to retention, separating churn from measurement and seasonality, and find the cause?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 81% were usable with at most a quick edit.

Reliably right

  1. Produces the required deliverable100% pass
    The diagnosis is complete, actionable, and in the right format for Leo; a PM could act on it with light edits.
    GPT-6 Astra · ChatGPT · Flat, while sign-ups grow
  2. Understands before it justifies100% pass
    It tests the summer explanation against the data, finds the break aligns with the onboarding change not the paid spend increase, and reaches its own diagnosis from the evidence.
    GPT-6 Astra · ChatGPT · Flat, while sign-ups grow
  3. Splits growth into new, lost and returning100% pass
    It splits growth into new, retained, churned, and returning users, and uses the net growth equation to show which flow changed.
    GPT-6 Astra · ChatGPT · Flat, while sign-ups grow

Where it slips

  1. Gets the base of every number right61% pass
    The SMB adoption churn calculation is incorrect; 39% of ~$7.4k/month churn is ~$2.9k/month, not $34k MRR, and the annualized figure is ~$34.8k ARR, not $0.4M.
    Sonnet 5.5 · API · The collapse that wasn't
  2. Avoids unsupported claims67% pass
    The output presents causal interpretations (e.g., onboarding changes broke retention, low-intent users) as established fact without labelling them as hypotheses.
    Gemini 3.5 Flash-Lite · Gemini · Flat, while sign-ups grow
  3. Identifies material uncertainty77% pass
    The output does not name specific unknowns that could change the decision or say how they would be resolved.
    Gemini 3.5 Flash-Lite · Gemini · Flat, while sign-ups grow

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Understands before it justifies

    Does the diagnosis start from what the data shows (which users, when, at what point in their life with the product) before accepting or proposing an explanation, including the one it was handed?

    Passes when Tests the explanation it was given against the data, and reaches its own diagnosis from the evidence.

  2. Splits growth into new, lost and returning

    Does the diagnosis separate new users or revenue from what was lost and what came back, rather than reading a single total?

    Passes when Uses growth accounting (new, retained, churned, returning, or new, expansion, contraction, churn) to show which flow changed.

  3. Reads the shape of the curves

    Does the diagnosis read how retention curves behave over time (flattening, still falling, or dipping and recovering) and what each shape means?

    Passes when Describes the curve shapes in the data and draws the right conclusion from each, such as a dip that recovers being seasonal.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 3 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a Staff PM at Tandem. Our CEO, Maya Lind, believes retention collapsed in March and wants two squads on an AI assistant next quarter (her note is below). She's asked you to confirm the diagnosis and find where else we're losing customers. Write her a memo of no more than 1,200 words: what is and isn't happening to retention, the evidence for each, the size of the real problem in revenue, and what to do about it. The files are attached, as exported.

What the model was given6 items: About Tandem, ceo_note.md, active_accounts_by_month.csv (from the main dashboard), mrr_movements_by_month.csv, exit_survey_2026.csv (accounts that cancelled), product_and_pricing_changes.md
About TandemA shared workspace for teams: docs, wikis and project pages. Sold per seat per month. Three segments: education (schools and universities, mostly annual contracts renewing in August), SMB, and mid-market.
ceo_note.md1 lines · Download
From Maya (CEO): "Retention collapsed in March and hasn't recovered: active accounts are down by a fifth and education is in freefall. Notion AI is eating us. I want two squads on an AI assistant next quarter. Can you confirm the diagnosis and tell me where else we're bleeding?"
active_accounts_by_month.csv (from the main dashboard)month,segment,paying_accounts,active_accounts 2026-01,Education,610,525 2026-01,SMB,2140,1758 2026-01,Mid-market,420,378 2026-02,Education,617,531 2026-02,SMB,2166,1785 2026-02,Mid-market,425,381 2026-03,Education,625,428 2026-03,SMB,2191,1434 2026-03,Mid-market,430,314 2026-04,Education,632,439 2026-04,SMB,2217,1447 2026-04,Mid-market,435,317 2026-05,Education,639,443 2026-05,SMB,2243,1448 2026-05,Mid-market,440,322 2026-06,Education,647,293 2026-06,SMB,2268,1476 2026-06,Mid-market,445,325 2026-07,Education,654,201 2026-07,SMB,2294,1497 2026-07,Mid-market,450,327 2026-08,Education,661,296 2026-08,SMB,2320,1514 2026-08,Mid-market,455,330 2026-09,Education,669,461 2026-09,SMB,2345,1524 2026-09,Mid-market,460,335
mrr_movements_by_month.csv27 rows · Download
month,segment,starting_mrr_usd,new_mrr_usd,expansion_mrr_usd,contraction_mrr_usd,churned_mrr_usd,ending_mrr_usd
2026-01,Education,61000,2237,603,311,874,62655
2026-01,SMB,214000,9047,3152,1616,7154,217429
2026-01,Mid-market,268000,7253,8246,2374,2613,278512
2026-02,Education,62655,2092,584,300,934,64097
2026-02,SMB,217429,9117,3227,1592,7555,220626
2026-02,Mid-market,278512,6797,8384,2324,2769,288600
…
exit_survey_2026.csv (accounts that cancelled)response_id,segment,main_reason X501,Mid-market,Too expensive per seat after the price change X502,Mid-market,Too expensive per seat after the price change X503,Mid-market,Too expensive per seat after the price change X504,Mid-market,Too expensive per seat after the price change X505,Mid-market,Too expensive per seat after the price change X506,Mid-market,Too expensive per seat after the price change X507,Mid-market,Too expensive per seat after the price change X508,Mid-market,Too expensive per seat after the price change X509,Mid-market,Too expensive per seat after the price change X510,Mid-market,Too expensive per seat after the price change X511,Mid-market,Too expensive per seat after the price change X512,Mid-market,Consolidating tools X513,Mid-market,Consolidating tools X514,Mid-market,Consolidating tools X515,Mid-market,Switched to Notion X516,SMB,Company closed or downsized X517,SMB,Company closed or downsized X518,SMB,Company closed or downsized X519,SMB,Company closed or downsized X520,SMB,Company closed or downsized X521,SMB,Company closed or downsized X522,SMB,Company closed or downsized X523,SMB,Company closed or downsized X524,SMB,Company closed or downsized X525,SMB,Company closed or downsized X526,SMB,Company closed or downsized X527,SMB,Company closed or downsized X528,SMB,Company closed or downsized X529,SMB,Company closed or downsized X530,SMB,Too expensive X531,SMB,Too expensive X532,SMB,Too expensive X533,SMB,Too expensive X534,SMB,Too expensive X535,SMB,Too expensive X536,SMB,Too expensive X537,SMB,Too expensive X538,SMB,Too expensive X539,SMB,Switched to Notion X540,SMB,Switched to Notion X541,SMB,Switched to Notion X542,SMB,Switched to Notion X543,SMB,Didn't get the team to adopt it X544,SMB,Didn't get the team to adopt it X545,SMB,Didn't get the team to adopt it X546,SMB,Didn't get the team to adopt it X547,SMB,Didn't get the team to adopt it X548,SMB,Didn't get the team to adopt it X549,SMB,Didn't get the team to adopt it X550,SMB,Didn't get the team to adopt it X551,SMB,Didn't get the team to adopt it X552,SMB,Didn't get the team to adopt it X553,SMB,Didn't get the team to adopt it X554,SMB,Didn't get the team to adopt it X555,SMB,Didn't get the team to adopt it X556,SMB,Didn't get the team to adopt it X557,SMB,Didn't get the team to adopt it X558,SMB,Didn't get the team to adopt it X559,SMB,Didn't get the team to adopt it X560,Education,"End of academic year, will review" X561,Education,"End of academic year, will review" X562,Education,"End of academic year, will review" X563,Education,"End of academic year, will review" X564,Education,"End of academic year, will review" X565,Education,"End of academic year, will review" X566,Education,"End of academic year, will review" X567,Education,"End of academic year, will review" X568,Education,"End of academic year, will review" X569,Education,"End of academic year, will review" X570,Education,"End of academic year, will review" X571,Education,"End of academic year, will review" X572,Education,Budget cut X573,Education,Budget cut X574,Education,Budget cut X575,Education,Budget cut X576,Education,Budget cut X577,Education,Budget cut X578,Education,Switched to Google Workspace X579,Education,Switched to Google Workspace
product_and_pricing_changes.md6 lines · Download
# Product and pricing changes, 2026

- 2 March: Dashboard metric "active account" changed from "anyone logged in within 30 days" to "anyone created or edited a page within 30 days", to measure real use.
- 15 May: Mid-market plan from $12 to $15 per seat per month, at each account's next monthly bill.
- 15 May: Seat audit launched for admins: a page listing seats with no activity in 60 days, with a one-click "remove seat".
- 3 August: Notion launched its AI assistant on every paid plan.
What a strong answer doesThe answer key the graders mark against

Separates what isn't churn from what is. (1) March: active accounts fall about 17 points in every segment at once while paying accounts keep growing, because on 2 March 'active' changed from logging in to editing: a definition change, not a collapse. (2) Education: active accounts dip June to August and recover in September, the summer break; education MRR grows, with a modest churn bump at August renewals. (3) Logo churn is roughly flat throughout, and Notion is the main reason in only 5 of 79 exit responses, and Notion's AI launched in August, months after the 'collapse'. The real loss is mid-market contraction: after the 15 May rise from $12 to $15 per seat, shipped the same day as the seat audit, contraction jumps from about $2,400 a month to $12,000 to $22,000 a month (about $75,000 in May to September against about $9,600 in January to April), and mid-market MRR stops growing (flat at about $322,000 from June after rising about $10,000 a month), while 11 of 15 mid-market exits cite price per seat. A secondary issue is SMB adoption ('didn't get the team to adopt it' is the top SMB reason). Recommends fixing the mid-market seat economics (such as pricing active seats or a seat-band discount, and rethinking how the seat audit is presented), restating the dashboard on a consistent definition, and not putting two squads on an AI assistant as a retention fix without evidence it would help.

Critical failures (cap the score at 40)
  • Confirms that retention collapsed in March
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Needs more than a quick edit11 of 13 checks passed

The SMB adoption churn calculation is incorrect, so a PM would need to redo that analysis before using the memo.

Output

To: Maya Lind From: Staff PM Re: What is happening to retention, and what we should do about it

Bottom line

Retention did not collapse in March, and education is not in freefall. The "fifth" drop in active accounts comes from a metric redefinition. The real leak is in mid-market, where our May price increase and seat audit launched together and revenue has stalled since. I recommend we don't staff two squads on an AI assistant yet. Put one squad on mid-market pricing and seats and one on SMB activation.

1. March was a metric change, not churn

On 2 March the dashboard changed "active account" from "logged in within 30 days" to "created or edited a page within 30 days." Active accounts fell from 2,697 in February to 2,176 in March (−19%), which is the drop you saw. Everything that measures money or contracts says nothing happened:

  • Paying accounts rose in every segment in March (total 3,208 → 3,246).
  • Total MRR rose 2.6% in March ($573k → $588k), and every segment grew.
  • Churned MRR in March was in line with January and February in every segment.

The new definition simply counts fewer accounts as active. The active-to-paying ratio stepped down once and then held: education 86% → 69%, SMB 82% → 65%, mid-market 90% → 73%. The January–February numbers need restating on the new definition. Until then, no one should compare across March.

2. Education is seasonal, with one modest renewal bump

Education "active" fell to 293 in June and 201 in July, then recovered to 296 in August and 461 in September (69% of paying, against 68–70% in March–May). That's a school-year pattern. We have no prior-year data to confirm it, so I'm inferring it from the academic calendar. Education paying accounts and MRR never fell off a cliff.

The one real event is August, the renewal month. Churned MRR was $3,925 against a normal ~$910, so about $3,000 MRR (~$36k ARR) of excess churn. September MRR ($72.2k) is already above July's. Exit reasons are end of academic year (12 of 20), budget cuts (6), and Google Workspace (2). None cite Notion.

One thing to reconcile: education paying accounts rose in August even as churned MRR quadrupled. Either the churn was seat reductions rather than lost accounts, or the account count lags.

3. Notion AI: no evidence yet

Notion launched its assistant on 3 August, five months after the "collapse," so it can't explain March. Since then:

  • SMB and mid-market churned MRR in August and September is flat against the prior months.
  • Only 5 of 79 exit responses (6%) say "Switched to Notion": 4 SMB, 1 mid-market, no education.

Two months of data is early, and the survey is undated, so I'd watch this rather than rule it out. But it doesn't support a two-squad bet today.

4. The real problem: mid-market, since 15 May

Two changes shipped on 15 May. The plan price rose from $12 to $15 per seat (+25%), and admins got a seat-audit page with one-click removal of seats inactive for 60 days.

Mid-marketJan–Apr avgMayJunJulAugSep
Contraction MRR$2.4k$10.2k$22.4k$16.5k$12.5k$13.2k
Net MRR change+$10.3k+$25.8k−$9.5k−$3.8k+$0.5k−$0.1k

May's net figure is inflated by a $31.6k expansion spike, which is the price increase hitting bills. After that, contraction ran 5–9× normal for five straight months. Churned MRR did not move (~$2.7k a month throughout), and mid-market paying accounts kept growing (435 → 460). So we're not losing logos. We're losing seats within accounts that stay. That is my inference from the contraction pattern, and the account-level data should confirm it.

Mid-market grew about $10.3k MRR a month before May. It has been flat since. Against that trend, September MRR should have been ~$360.6k and was $321.9k:

  • Gap: ~$38.6k MRR, or ~$0.46M ARR (about 6% of our $640k MRR).
  • It grows by roughly $10k MRR every month the pattern continues.

The exit survey agrees: 11 of 15 mid-market cancellations cite "too expensive per seat after the price change." The survey only covers cancelled accounts, so it misses the seat removals, which are the larger leak.

I suspect customers were paying for shelfware. The price rise made them look, and the audit tool made trimming a single click. Some of that contraction is a correction we'd have faced eventually. But we shipped both changes on the same day, so we can't separate the two effects today.

Two open questions: (a) the price increase booked only ~$23k of excess expansion, far less than 25% of a ~$309k base, so I want to confirm whether all accounts have been repriced or some are grandfathered; and (b) how much of the contraction is seats with zero activity versus active seats cut for cost.

5. Second leak: SMB activation

SMB loses the most revenue in absolute terms. Churned MRR runs ~$7.4k a month, about 3.2% of segment MRR monthly, or roughly $1.07M ARR a year. It's masked because new business (~$9–10k a month) covers it. Exit reasons (44 responses):

  • Didn't get the team to adopt it: 17 (39%)
  • Company closed or downsized: 14 (32%, not addressable)
  • Too expensive: 9 (20%)
  • Switched to Notion: 4 (9%)

SMB also has the lowest active ratio (65%). Applying the survey mix to churned dollars (a rough assumption from a small sample), adoption failures cost about $34k MRR, or ~$0.4M ARR, a year. That's roughly the same size as the mid-market gap, and it has nothing to do with AI.

Size of the problem

IssueRevenue at stake
March "collapse"$0 (metric artifact)
Education August bump~$36k ARR, largely recovered
Mid-market seats/pricing~$0.46M ARR gap so far, growing ~$120k ARR per month of delay
SMB adoption churn~$0.4M ARR per year
Notion switching (survey)Small, unproven

Recommendations

  1. Fix the metric this week. Restate January–February on the new definition, show active rate (active ÷ paying) alongside the count, and stop reporting the March figure as churn. Owner: Data/Analytics.

2. Put one squad on mid-market pricing and seats. In two weeks, pull account-level data to split contraction into zero-activity seats versus active seats, by account size. Then run a controlled test across renewal cohorts. Options include: - a lower price (such as $13.50), or grandfathering existing accounts for 12 months in exchange for annual commitments; - a seat-band discount for larger accounts; - a "reactivate seat" flow in the audit page.

I'd keep the audit. Hiding unused seats would cost us trust, and the fix is to make the value of each seat clearer. I wouldn't revert the price blindly until we know whether the lost seats were active.

  1. Put one squad on SMB activation. Focus on first-30-day adoption: invite flows, starter templates, and admin nudges when fewer than a set share of seats are active. A target of cutting adoption churn by a third is worth ~$0.13M ARR a year, and we'd see leading indicators in weeks.
  1. Education: no product response. Customer Success should confirm the August cancellations and offer budget-constrained schools a renewal option. Also reconcile the account counts.
  1. Hold the AI assistant decision until November. Add cancellation date, competitor, and free-text fields to the exit survey. If Notion-attributed churn or win/loss data shows a meaningful rise after August, we revisit with real evidence. A small discovery effort is fine, but not two squads.

Claims the judge couldn’t find in the brief

  • Adoption failures cost about $34k MRR, or ~$0.4M ARR, a year (applying survey mix to churned dollars).

Grades and run details

Decision model 88 · LLM judge 79
Decision model checks
  • passUses the supplied evidence correctly13%
  • passAddresses the actual decision99%
  • passRespects explicit constraints42%
  • passIdentifies material uncertainty99%
  • partialAvoids unsupported claims28%
  • passProduces the required deliverable86%
  • passSeparates what isn't churn98%
  • passFinds and sizes the real loss100%
  • passAnswers the AI plan with evidence100%
  • passUnderstands before it justifies100%
  • failGets the base of every number right39%
  • passSplits growth into new, lost and returning99%
  • passReads the shape of the curves100%
Run
Run
#1
API response time
65 s
Submitted
6 Oct 2026

Usable with a quick edit12 of 13 checks passed

The memo is well-structured, evidence-based, and directly actionable; a PM could use it with minor wording edits without re-checking evidence or redoing reasoning.

Output

To: Maya Lind, CEO From: Staff PM Subject: Retention diagnosis: fix mid-market seat losses and SMB retention before committing two squads to AI

Recommendation

I would not commit two squads to an AI assistant on this evidence. March’s apparent collapse coincides with a dashboard-definition change, education’s summer activity decline largely reverses, and the clearest new revenue problem begins in mid-market after May’s pricing and seat-audit changes—not after Notion’s August AI launch.

There are real losses: persistent SMB churn, a sharp increase in mid-market seat contraction, and an August education renewal spike. Prioritize those, while running a bounded investigation into AI-related competitive losses.

What is—and is not—happening

1. March’s activity cliff is not evidence of a retention collapse

Total active accounts fell from 2,697 in February to 2,176 in March: −19.3%, matching your observation. But on March 2, “active” changed from any login in 30 days to creating or editing a page in 30 days. Those counts are not comparable.

Financial data does not show a corresponding March shock:

  • Total MRR rose from $573,323 to $588,463, up 2.6%.
  • Paying accounts rose from 3,208 to 3,246.
  • Churn and contraction remained close to January–February levels in all three segments.

Under the new definition, total active accounts increased 6.6% from March to September. SMB’s active share of paying accounts was approximately 65% in both months, and mid-market’s approximately 73%.

This does not prove customer retention is healthy: acquisition can conceal customer losses, and account activity can conceal seat reductions. It does mean we cannot infer a March retention collapse—or failure to recover—from this dashboard. We need comparable activity history and cohort retention.

2. Education is seasonal, with a real but bounded renewal issue

Education activity fell from 428 in March to 201 in July, then recovered to 461 in September. Its active share recovered from 31% in July to 69% in September, essentially March’s level. That pattern is consistent with the academic calendar, not sustained freefall.

Financially, education MRR increased from $61,000 entering January to $72,211 exiting September. Nevertheless, August churned MRR rose to $3,925, versus roughly $900 per month otherwise. August gross revenue retention was 94.1%; September returned to 98.3%.

The timing fits annual renewals. Of 20 education cancellation-survey responses, 12 cite the academic year ending, six budget cuts, and two Google Workspace. These are actual cancellations, not merely inactive accounts. We need the August renewal-eligible revenue denominator and prior-year comparisons before judging whether this renewal season was unusually bad.

3. Mid-market’s problem is seat contraction, beginning after May

Mid-market contraction averaged $2,397 MRR/month in January–April. It then rose to:

MonthContraction MRR
------:
May$10,239
June$22,351
July$16,514
August$12,486
September$13,212

Account churn remained approximately $2,600–$2,800 MRR/month. We are predominantly losing revenue within surviving accounts, which growing account counts obscure.

Two changes launched together on May 15: a 25% per-seat price increase and an admin tool identifying unused seats with one-click removal. May’s $31,585 expansion spike is consistent with repricing; it is not evidence of stronger adoption. Subsequent seat removal could reflect price sensitivity, newly visible unused seats, or both.

Cancellation feedback supports investigating price: 11 of 15 mid-market respondents cite the price change. But these surveys exclude customers who only reduced seats, and cannot separate pricing from the audit’s effect.

4. SMB has substantial ongoing losses—not a new March or August cliff

SMB loses approximately $7,100–$7,700 MRR/month to cancellations, plus $1,600–$1,700 to contraction. Monthly gross revenue retention is roughly 96%, and net revenue retention approximately 97.3%–97.9%. New sales keep total MRR growing despite losses from existing customers.

Among 44 cancellation respondents, 17 cite failed team adoption, 14 closure/downsizing, nine price, and four Notion. Adoption is a meaningful product opportunity; business closure is less directly addressable.

5. “Notion AI is eating us” remains unproven

Only five of 79 survey respondents cite Notion; none explicitly cite AI. These are unweighted, cancellation-only responses with no dates, not a representative estimate of lost revenue.

Notion’s AI launch was August 3. It cannot explain March’s dashboard discontinuity or June’s mid-market contraction spike. There is no clear August break in SMB losses. Competitive risk may be real, but these exports do not establish it.

Size of the revenue problem

Across January–September, gross churn plus contraction removed $204,460 of MRR movements:

SegmentChurn + contraction
------:
Mid-market$108,814
SMB$81,714
Education$13,932

These are summed recurring-revenue reductions—not cash revenue lost during the period, and not wholly recoverable.

The most actionable exposures are:

  • Mid-market deterioration: June–September contraction totaled $64,563, about $55,000 above four months at the January–April dollar baseline. That is roughly $660,000 in annualized run-rate equivalent, not a causal estimate or recovery forecast. After expansion, existing-account MRR declined $40,585 over those four months; new sales offset much of it.
  • Persistent SMB leakage: expansion offset only part of gross losses, leaving $52,837 in net existing-account MRR reductions across nine months—about $634,000 annualized equivalent.
  • Education renewal spike: August churn was approximately $3,000 MRR above the usual monthly level, about $36,000 annualized equivalent, subject to renewal-cohort analysis.

Total company MRR still grew 17.9% from January’s opening balance to September’s close. Growth is masking retention losses.

What to do next quarter

  1. Squad one: mid-market revenue retention. With Finance and CS, decompose revenue changes into price, seats, cancellations, and reactivations; compare billing dates and audit usage. Interview downsizers, not just cancellers. Test targeted renewal concessions, right-sized packages, and clearer paid-seat value. Preserve customers’ ability to remove unused seats. Measure cohort seat retention and gross/net revenue retention.
  1. Squad two: SMB team adoption. Test onboarding and collaboration workflows that get multiple teammates creating and editing early. Measure activation-to-paid-retention and retained MRR against a control group—not logins alone.
  1. CS-led education renewal program. Review August losses by contract value and renewal eligibility; pursue budget-sensitive saves and reactivation before the next academic cycle.
  1. Repair measurement; validate AI separately. Backfill comparable activity definitions, publish logo and revenue-retention cohorts, and date/revenue-weight cancellation and downsizing feedback. Run focused competitive interviews and a small AI prototype before making a two-squad commitment.

Grades and run details

Decision model 92 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly19%
  • passAddresses the actual decision74%
  • passRespects explicit constraints48%
  • passIdentifies material uncertainty89%
  • passAvoids unsupported claims50%
  • passProduces the required deliverable77%
  • passSeparates what isn't churn99%
  • passFinds and sizes the real loss100%
  • passAnswers the AI plan with evidence100%
  • passUnderstands before it justifies100%
  • failGets the base of every number right33%
  • passSplits growth into new, lost and returning98%
  • passReads the shape of the curves100%
Run
Run
#1
API response time
68 s
Submitted
5 Oct 2026

Usable with a quick edit12 of 13 checks passed

The output is clear, evidence-based, and actionable; a PM could use it with minor wording tweaks without rechecking data or reasoning.

Output

To: Maya Lind Subject: March retention diagnosis and where revenue is actually at risk

Bottom line: The March dashboard cliff is real as a reported activity change, but it is not evidence that customer retention collapsed. The metric changed on March 2, and paying accounts and MRR continued to grow. There are real, more specific risks: SMB adoption, mid-market seat contraction after pricing and seat-audit changes, and education’s seasonal renewal cycle. The data does not support assigning two squads to an AI assistant on the premise that Notion AI caused a broad retention collapse.

What is—and isn’t—happening

Active accounts fell from 2,697 in February to 2,176 in March, down 19.3%. But on March 2, “active” changed from anyone logged in within 30 days to anyone who created or edited a page within 30 days. That stricter definition plausibly explains the abrupt drop: activity fell by roughly 18–20% in all three segments at once.

The more comparable evidence is what happened after the change. From March to September, active accounts rose from 2,176 to 2,320 (+6.6%): SMB rose 6%, mid-market 7%, and education 8%. SMB and mid-market activity rates were broadly stable over that period. These figures do not show an ongoing collapse—though we cannot make a clean February-to-March retention comparison without recalculating historical activity on one definition.

Education’s summer decline is pronounced: active accounts fell from 428 in March to 201 in July, then recovered to 461 in September. That pattern fits the academic calendar and renewal timing better than a continuous decline. Paying education accounts rose every month, from 625 in March to 669 in September. The activity metric is not the same as paid-account retention, so this is not proof that every education account is secure; it is evidence against calling education “in freefall.”

Revenue also contradicts the broad-collapse diagnosis. Ending MRR rose from $558.6K in January to $640.3K in September, up $81.7K (14.6%). Monthly churned MRR was generally about $10.6K–$11.4K from January through July, rose to $14.1K in August, then returned to $11.1K in September. There is no sustained March deterioration in these figures.

Where the real risks are

  • Mid-market pricing and seat contraction: The mid-market plan price rose 25%, from $12 to $15 per seat, and a one-click seat-removal audit launched in May. Mid-market contraction MRR then reached $10.2K in May, $22.4K in June, $16.5K in July, and $12.5K in August, versus roughly $2.4K per month in the surrounding months. That is about $52K more contraction MRR over May–August than the surrounding run rate would suggest. It is a meaningful warning, but not $52K of confirmed churn: contraction may reflect customers removing idle seats, and the report does not isolate price effects from seat cleanup. Mid-market ending MRR was still $12.8K above April’s by September, after the price change and other movements.
  • SMB adoption: In the exit survey, 17 of 44 SMB respondents cited failure to get their team to adopt Tandem—the largest single SMB reason. Another 14 cited company closure or downsizing, nine cited price, and four said they switched to Notion. This points to an adoption problem as well as some external business distress.
  • Education renewals: Twelve of 20 education respondents said they would review at the end of the academic year; six cited budget cuts. Education churned MRR spiked to $3.9K in August, versus roughly $0.9K in most other months, then returned to $0.9K in September. Prepare for the renewal window rather than treating the summer activity trough as evidence of a year-round collapse.

The exit survey is a sample of 79 cancellations, not a representative churn-rate study. Only five respondents said they switched to Notion (four SMB, one mid-market), and the survey does not say they left because of AI. Notion’s August AI launch is too recent to explain the March dashboard break, and these data do not establish that it is driving the current losses.

Recommended actions

  1. Fix the measurement first. Backfill the new activity definition, annotate the dashboard break, and report paid-account retention, logo churn, revenue churn, and usage separately by segment and renewal cohort.
  2. Investigate mid-market changes account by account. Separate price increases, seat removals, and cancellations; quantify retained seats and renewal outcomes. Test targeted pricing or seat flexibility where value is at risk before considering a broad rollback.
  3. Improve SMB activation. Focus on onboarding, team rollout, and early adoption signals; distinguish preventable adoption failures from business closures.
  4. Run an education renewal plan ahead of the academic-year cycle, with early outreach and budget-sensitive renewal options.
  5. Do not staff two AI squads on this diagnosis. If AI is strategically important, run a bounded customer discovery and prototype test, with evidence of demand and retention impact as a gate for further investment.

Grades and run details

Decision model 92 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly12%
  • passAddresses the actual decision79%
  • passRespects explicit constraints56%
  • passIdentifies material uncertainty78%
  • passAvoids unsupported claims43%
  • passProduces the required deliverable69%
  • passSeparates what isn't churn99%
  • passFinds and sizes the real loss72%
  • passAnswers the AI plan with evidence100%
  • passUnderstands before it justifies100%
  • failGets the base of every number right17%
  • passSplits growth into new, lost and returning55%
  • passReads the shape of the curves99%
Run
Run
#1
API response time
34 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 2

Uses the supplied evidence correctlyMixedRightRight
Sonnet 5.5 · API

The SMB adoption churn calculation is incorrect; 39% of ~$7.4k/month churn is ~$2.9k/month, not $34k MRR, and the annualized figure is ~$34.8k ARR, not $0.4M.

GPT-6.1 Sol · API

Every factual statement about the current situation is taken directly from the supplied context or follows from arithmetic on that data.

GPT-6 Luna · API

All factual claims about the current situation are directly supported by the supplied data or arithmetic.

Gets the base of every number rightWrongMixedMixed
Sonnet 5.5 · API

The SMB adoption churn calculation is incorrect; 39% of ~$7.4k/month churn is ~$2.9k/month, not $34k MRR, and the annualized figure is ~$34.8k ARR, not $0.4M.

GPT-6.1 Sol · API

All percentages and rates are computed correctly from the supplied data, and bases are clear (e.g., active share of paying accounts, gross retention from starting MRR).

GPT-6 Luna · API

All percentages and differences are computed correctly from the supplied data, with bases stated or clear.

All got right 11

Addresses the actual decisionRightRightRight
Sonnet 5.5 · API

Commits to not staffing two AI squads, recommends mid-market and SMB squads, and says what would change the AI decision.

GPT-6.1 Sol · API

The memo unambiguously advises against committing two squads to an AI assistant now and states that focused competitive interviews and a small AI prototype would be needed to change that call.

GPT-6 Luna · API

The memo unambiguously advises against staffing two AI squads on this diagnosis and states what evidence would change that call.

Respects explicit constraintsRightRightRight
Sonnet 5.5 · API

Memo is within 1,200 words, addressed to the CEO, and respects all explicit constraints.

GPT-6.1 Sol · API

The output is a memo to the CEO, under 1,200 words, and addresses the requested diagnosis, evidence, revenue sizing, and actions.

GPT-6 Luna · API

The output is a memo to Maya, well under 1,200 words, and respects the requested form and reader.

Identifies material uncertaintyRightRightRight
Sonnet 5.5 · API

Identifies unknowns about repricing and seat activity, and says how to resolve them.

GPT-6.1 Sol · API

It names specific unknowns (comparable activity history, renewal-eligible revenue denominator, separation of price and audit effects) and says how to resolve them.

GPT-6 Luna · API

It names specific unknowns (definition change prevents clean comparison, contraction may be seat cleanup not churn, exit survey is a sample) and says how to resolve them.

Avoids unsupported claimsRightRightRight
Sonnet 5.5 · API

Hypotheses are labelled, and the SMB adoption figure is presented as a rough assumption, not established fact.

GPT-6.1 Sol · API

Interpretations and causes are labelled as hypotheses (e.g., 'consistent with', 'could reflect', 'may be real'), and confident claims are backed by the data.

GPT-6 Luna · API

Interpretations are hedged ('plausibly', 'points to', 'may reflect') and not presented as established fact.

Produces the required deliverableRightRightRight
Sonnet 5.5 · API

Memo is complete, actionable, and within length.

GPT-6.1 Sol · API

The memo is complete, in the right form for the CEO, within the word limit, and actionable with clear next steps.

GPT-6 Luna · API

The memo is complete, in the right form, within length, and a PM could act on it with light edits.

Separates what isn't churnRightRightRight
Sonnet 5.5 · API

Shows March drop is definition change and education dip is seasonal with September recovery.

GPT-6.1 Sol · API

It links the March drop across all segments to the 2 March definition change and shows education's summer dip recovering in September, using the files.

GPT-6 Luna · API

It links the March drop to the 2 March definition change and shows education's summer dip recovering in September.

Finds and sizes the real lossRightRightRight
Sonnet 5.5 · API

Identifies mid-market contraction jump after 15 May, links to price rise and seat audit, and sizes gap at ~$38.6k MRR.

GPT-6.1 Sol · API

It identifies the mid-market contraction jump after 15 May, ties it to the price rise and seat audit, and sizes it in MRR (e.g., $64,563 June–September, ~$55k above baseline).

GPT-6 Luna · API

It identifies the mid-market contraction jump after 15 May, links it to the price rise and seat audit, and sizes it at ~$52K excess contraction MRR.

Answers the AI plan with evidenceRightRightRight
Sonnet 5.5 · API

Uses exit data and Notion launch timeline to advise against AI assistant as retention fix.

GPT-6.1 Sol · API

It uses exit data (5 of 79 cite Notion) and the August launch timeline to show the AI assistant is not the cause of the retention issues and advises against it as a retention fix.

GPT-6 Luna · API

It uses exit data (only 5 Notion switchers, no AI mention) and the August launch timeline to advise against the AI assistant as a retention fix.

Understands before it justifiesRightRightRight
Sonnet 5.5 · API

Tests CEO's diagnosis against data and reaches own diagnosis.

GPT-6.1 Sol · API

The diagnosis starts from the data (definition change, seasonality, contraction spike) and tests the CEO's hypothesis before reaching its own conclusions.

GPT-6 Luna · API

The diagnosis starts from the data, tests the CEO's Notion AI explanation, and reaches its own evidence-based conclusions.

Splits growth into new, lost and returningRightRightRight
Sonnet 5.5 · API

Splits MRR into new, expansion, contraction, churn to identify the leak.

GPT-6.1 Sol · API

It splits MRR movements into new, expansion, contraction, and churn, and shows net existing-account reductions separate from new sales.

GPT-6 Luna · API

It uses MRR movements (new, expansion, contraction, churn) to show which flow changed, particularly the contraction spike.

Reads the shape of the curvesRightRightRight
Sonnet 5.5 · API

Describes education dip and recovery as seasonal, and mid-market contraction as a step change.

GPT-6.1 Sol · API

It describes the education dip-and-recovery as seasonal, the mid-market contraction spike, and steady SMB losses, drawing correct conclusions from each shape.

GPT-6 Luna · API

It reads the education dip and recovery as seasonal and notes the post-March active account rise, not a continued decline.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT96.2100.02None
2GPT-6 LunawithAPI96.2100.02None
3GPT-6.1 SolwithAPI94.1100.02None
4Haiku 5.5withAPI92.0100.02None
5Opus 5.5withClaude82.1100.02None
6Sonnet 5.5withAPI90.189.32None
7Gemini 3.5 Flash-LitewithGemini76.474.22None
8Gemini 3.8 FlashwithAPI72.370.92None

What you get for the cost

GPT-6 Luna · API matches GPT-6 Astra · ChatGPT at about a fiftieth of the cost.

Task score against relative cost for each setup. The table below has the same figures.607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 98, 79× the cheapestGPT-6 Luna · API: task score 98, the cheapestGPT-6.1 Sol · API: task score 97, 15× the cheapestHaiku 5.5 · API: task score 96, 3.6× the cheapestOpus 5.5 · Claude: task score 92, 77× the cheapestSonnet 5.5 · API: task score 88, 50× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 74, 3.0× the cheapestGemini 3.8 Flash · API: task score 71, 13× the cheapestGPT-6 Astra · ChatGPTGPT-6 Luna · APIGPT-6.1 Sol · APIHaiku 5.5 · APIOpus 5.5 · ClaudeSonnet 5.5 · APIGemini 3.5 Flash-Lite · GeminiGemini 3.8 Flash · API
Best value
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API981.0×
Gemini 3.5 Flash-Lite · Gemini743.0×
Haiku 5.5 · API963.6×
Gemini 3.8 Flash · API7113×
GPT-6.1 Sol · API9715×
Sonnet 5.5 · API8850×
Opus 5.5 · Claude9277×
GPT-6 Astra · ChatGPT9879×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Working out why customers leave, and what to fix first.

Why it matters

Teams pick a story (the season, a competitor) and then pull data to justify it. Half the time the 'drop' is a definition change, and the real leak is somewhere else.

What good looks like

  • Starts from what the data shows, not a story
  • Splits growth into new, lost and returning users
  • Separates measurement and seasonality from churn
  • Sizes the real problem and fixes new-user retention first

Deliberately not measured

  • Dashboard design
Capability tested

Retention diagnosis

The failure we’re looking for

Confirming the story everyone already believes

Grading

Decision model and LLM judge, calibrated against a blind PM review