Tasks / Growth

Diagnose a retention problem

Can the model work out from raw data what's really happening to retention, separating churn from measurement and seasonality, and find the cause?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 81% were usable with at most a quick edit.

Reliably right

  1. Produces the required deliverable100% pass
    The diagnosis is complete, actionable, and in the right format for Leo; a PM could act on it with light edits.
    GPT-6 Astra · ChatGPT · Flat, while sign-ups grow
  2. Understands before it justifies100% pass
    It tests the summer explanation against the data, finds the break aligns with the onboarding change not the paid spend increase, and reaches its own diagnosis from the evidence.
    GPT-6 Astra · ChatGPT · Flat, while sign-ups grow
  3. Splits growth into new, lost and returning100% pass
    It splits growth into new, retained, churned, and returning users, and uses the net growth equation to show which flow changed.
    GPT-6 Astra · ChatGPT · Flat, while sign-ups grow

Where it slips

  1. Gets the base of every number right61% pass
    The SMB adoption churn calculation is incorrect; 39% of ~$7.4k/month churn is ~$2.9k/month, not $34k MRR, and the annualized figure is ~$34.8k ARR, not $0.4M.
    Sonnet 5.5 · API · The collapse that wasn't
  2. Avoids unsupported claims67% pass
    The output presents causal interpretations (e.g., onboarding changes broke retention, low-intent users) as established fact without labelling them as hypotheses.
    Gemini 3.5 Flash-Lite · Gemini · Flat, while sign-ups grow
  3. Identifies material uncertainty77% pass
    The output does not name specific unknowns that could change the decision or say how they would be resolved.
    Gemini 3.5 Flash-Lite · Gemini · Flat, while sign-ups grow

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Understands before it justifies

    Does the diagnosis start from what the data shows (which users, when, at what point in their life with the product) before accepting or proposing an explanation, including the one it was handed?

    Passes when Tests the explanation it was given against the data, and reaches its own diagnosis from the evidence.

  2. Splits growth into new, lost and returning

    Does the diagnosis separate new users or revenue from what was lost and what came back, rather than reading a single total?

    Passes when Uses growth accounting (new, retained, churned, returning, or new, expansion, contraction, churn) to show which flow changed.

  3. Reads the shape of the curves

    Does the diagnosis read how retention curves behave over time (flattening, still falling, or dipping and recovering) and what each shape means?

    Passes when Describes the curve shapes in the data and draws the right conclusion from each, such as a dip that recovers being seasonal.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 3 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a Staff PM at Tandem. Our CEO, Maya Lind, believes retention collapsed in March and wants two squads on an AI assistant next quarter (her note is below). She's asked you to confirm the diagnosis and find where else we're losing customers. Write her a memo of no more than 1,200 words: what is and isn't happening to retention, the evidence for each, the size of the real problem in revenue, and what to do about it. The files are attached, as exported.

What the model was given6 items: About Tandem, ceo_note.md, active_accounts_by_month.csv (from the main dashboard), mrr_movements_by_month.csv, exit_survey_2026.csv (accounts that cancelled), product_and_pricing_changes.md
About TandemA shared workspace for teams: docs, wikis and project pages. Sold per seat per month. Three segments: education (schools and universities, mostly annual contracts renewing in August), SMB, and mid-market.
ceo_note.md1 lines · Download
From Maya (CEO): "Retention collapsed in March and hasn't recovered: active accounts are down by a fifth and education is in freefall. Notion AI is eating us. I want two squads on an AI assistant next quarter. Can you confirm the diagnosis and tell me where else we're bleeding?"
active_accounts_by_month.csv (from the main dashboard)month,segment,paying_accounts,active_accounts 2026-01,Education,610,525 2026-01,SMB,2140,1758 2026-01,Mid-market,420,378 2026-02,Education,617,531 2026-02,SMB,2166,1785 2026-02,Mid-market,425,381 2026-03,Education,625,428 2026-03,SMB,2191,1434 2026-03,Mid-market,430,314 2026-04,Education,632,439 2026-04,SMB,2217,1447 2026-04,Mid-market,435,317 2026-05,Education,639,443 2026-05,SMB,2243,1448 2026-05,Mid-market,440,322 2026-06,Education,647,293 2026-06,SMB,2268,1476 2026-06,Mid-market,445,325 2026-07,Education,654,201 2026-07,SMB,2294,1497 2026-07,Mid-market,450,327 2026-08,Education,661,296 2026-08,SMB,2320,1514 2026-08,Mid-market,455,330 2026-09,Education,669,461 2026-09,SMB,2345,1524 2026-09,Mid-market,460,335
mrr_movements_by_month.csv27 rows · Download
month,segment,starting_mrr_usd,new_mrr_usd,expansion_mrr_usd,contraction_mrr_usd,churned_mrr_usd,ending_mrr_usd
2026-01,Education,61000,2237,603,311,874,62655
2026-01,SMB,214000,9047,3152,1616,7154,217429
2026-01,Mid-market,268000,7253,8246,2374,2613,278512
2026-02,Education,62655,2092,584,300,934,64097
2026-02,SMB,217429,9117,3227,1592,7555,220626
2026-02,Mid-market,278512,6797,8384,2324,2769,288600
…
exit_survey_2026.csv (accounts that cancelled)response_id,segment,main_reason X501,Mid-market,Too expensive per seat after the price change X502,Mid-market,Too expensive per seat after the price change X503,Mid-market,Too expensive per seat after the price change X504,Mid-market,Too expensive per seat after the price change X505,Mid-market,Too expensive per seat after the price change X506,Mid-market,Too expensive per seat after the price change X507,Mid-market,Too expensive per seat after the price change X508,Mid-market,Too expensive per seat after the price change X509,Mid-market,Too expensive per seat after the price change X510,Mid-market,Too expensive per seat after the price change X511,Mid-market,Too expensive per seat after the price change X512,Mid-market,Consolidating tools X513,Mid-market,Consolidating tools X514,Mid-market,Consolidating tools X515,Mid-market,Switched to Notion X516,SMB,Company closed or downsized X517,SMB,Company closed or downsized X518,SMB,Company closed or downsized X519,SMB,Company closed or downsized X520,SMB,Company closed or downsized X521,SMB,Company closed or downsized X522,SMB,Company closed or downsized X523,SMB,Company closed or downsized X524,SMB,Company closed or downsized X525,SMB,Company closed or downsized X526,SMB,Company closed or downsized X527,SMB,Company closed or downsized X528,SMB,Company closed or downsized X529,SMB,Company closed or downsized X530,SMB,Too expensive X531,SMB,Too expensive X532,SMB,Too expensive X533,SMB,Too expensive X534,SMB,Too expensive X535,SMB,Too expensive X536,SMB,Too expensive X537,SMB,Too expensive X538,SMB,Too expensive X539,SMB,Switched to Notion X540,SMB,Switched to Notion X541,SMB,Switched to Notion X542,SMB,Switched to Notion X543,SMB,Didn't get the team to adopt it X544,SMB,Didn't get the team to adopt it X545,SMB,Didn't get the team to adopt it X546,SMB,Didn't get the team to adopt it X547,SMB,Didn't get the team to adopt it X548,SMB,Didn't get the team to adopt it X549,SMB,Didn't get the team to adopt it X550,SMB,Didn't get the team to adopt it X551,SMB,Didn't get the team to adopt it X552,SMB,Didn't get the team to adopt it X553,SMB,Didn't get the team to adopt it X554,SMB,Didn't get the team to adopt it X555,SMB,Didn't get the team to adopt it X556,SMB,Didn't get the team to adopt it X557,SMB,Didn't get the team to adopt it X558,SMB,Didn't get the team to adopt it X559,SMB,Didn't get the team to adopt it X560,Education,"End of academic year, will review" X561,Education,"End of academic year, will review" X562,Education,"End of academic year, will review" X563,Education,"End of academic year, will review" X564,Education,"End of academic year, will review" X565,Education,"End of academic year, will review" X566,Education,"End of academic year, will review" X567,Education,"End of academic year, will review" X568,Education,"End of academic year, will review" X569,Education,"End of academic year, will review" X570,Education,"End of academic year, will review" X571,Education,"End of academic year, will review" X572,Education,Budget cut X573,Education,Budget cut X574,Education,Budget cut X575,Education,Budget cut X576,Education,Budget cut X577,Education,Budget cut X578,Education,Switched to Google Workspace X579,Education,Switched to Google Workspace
product_and_pricing_changes.md6 lines · Download
# Product and pricing changes, 2026

- 2 March: Dashboard metric "active account" changed from "anyone logged in within 30 days" to "anyone created or edited a page within 30 days", to measure real use.
- 15 May: Mid-market plan from $12 to $15 per seat per month, at each account's next monthly bill.
- 15 May: Seat audit launched for admins: a page listing seats with no activity in 60 days, with a one-click "remove seat".
- 3 August: Notion launched its AI assistant on every paid plan.
What a strong answer doesThe answer key the graders mark against

Separates what isn't churn from what is. (1) March: active accounts fall about 17 points in every segment at once while paying accounts keep growing, because on 2 March 'active' changed from logging in to editing: a definition change, not a collapse. (2) Education: active accounts dip June to August and recover in September, the summer break; education MRR grows, with a modest churn bump at August renewals. (3) Logo churn is roughly flat throughout, and Notion is the main reason in only 5 of 79 exit responses, and Notion's AI launched in August, months after the 'collapse'. The real loss is mid-market contraction: after the 15 May rise from $12 to $15 per seat, shipped the same day as the seat audit, contraction jumps from about $2,400 a month to $12,000 to $22,000 a month (about $75,000 in May to September against about $9,600 in January to April), and mid-market MRR stops growing (flat at about $322,000 from June after rising about $10,000 a month), while 11 of 15 mid-market exits cite price per seat. A secondary issue is SMB adoption ('didn't get the team to adopt it' is the top SMB reason). Recommends fixing the mid-market seat economics (such as pricing active seats or a seat-band discount, and rethinking how the seat audit is presented), restating the dashboard on a consistent definition, and not putting two squads on an AI assistant as a retention fix without evidence it would help.

Critical failures (cap the score at 40)
  • Confirms that retention collapsed in March
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Needs more than a quick edit8 of 13 checks passed

The memo contains a miscalculation of SMB churn average and an unsupported claim about Notion revenue, requiring re-checking before use.

Output

MEMORANDUM

TO: Maya Lind, CEO FROM: Staff Product Manager DATE: October 2, 2026 SUBJECT: Diagnostic: 2026 Retention Dynamics, Revenue Bleed, and Product Priorities

---

Executive Summary

Our customer and revenue retention did not collapse in March, Education is not in freefall, and Notion AI is not causing our churn. Allocating two squads to an AI assistant next quarter misdiagnoses our core retention drivers and will not stem our real revenue losses.

1. The March "Collapse" is a Telemetry Artifact: On March 2, we updated the definition of an “active account” on the main dashboard from any login within 30 days to creating/editing a page within 30 days. Paying accounts and monthly recurring revenue (MRR) actually grew in March. 2. Education is Cyclical, Not Collapsing: The summer dip in active accounts was normal academic seasonality. Active accounts rebounded by +129% in September as schools resumed, and paying Education accounts reached an all-time high (669). 3. Notion AI is Not Displacing Us: Notion launched AI in August—five months after the perceived March drop. Only 6% of churned accounts cited Notion in exit surveys. 4. Where We Are Actually Bleeding ($75k+ MRR Impact): * Mid-Market Contraction: The May 15 price hike ($12 to $15/seat) combined with the newly launched "Seat Audit" tool prompted enterprise admins to aggressively purge inactive seats. Contraction MRR quintupled from ~$2.4k/month to an average of ~$15k/month, erasing $62.8k in cumulative MRR since May and flatlining Mid-Market growth. * SMB Onboarding Failure: 39% of SMB churn is driven by failed team adoption, draining ~$7.5k in MRR every month.

---

Part 1: What Is NOT Happening (The False Alarms)

#### 1. Retention Did Not Collapse in March * What the dashboard showed: Total active accounts dropped 19.3% from February (2,697) to March (2,176). * The reality: On March 2, the data team modified the “active account” definition from passive logins to active page creation/editing. * The evidence: Paying accounts grew across every segment in March (total paying accounts rose from 3,208 to 3,246). Total MRR grew from $573.3k to $588.5k (+2.6% MoM). Real account and revenue retention remained entirely stable.

#### 2. Education is Following Predictable Academic Seasonality * What the dashboard showed: Active accounts fell from 443 in May to 201 in July (-55%). * The reality: K-12 and university campuses close for summer break. As classes resumed in September, active Education accounts surged back to 461. * The evidence: Paying accounts grew every single month without interruption (from 610 in January to 669 in September). The one-time churn spike in August ($3,925 churned MRR vs. ~$900 monthly baseline) is the annual renewal window for school districts, where 60% of exiting accounts cited "end of academic year review" and 30% cited "budget cuts"—not product dissatisfaction.

#### 3. Notion AI is a Phantom Culprit * Notion launched its AI assistant on August 3, 2026. It cannot explain a trend that supposedly started in March. * Across all 79 exit survey responses in 2026, only 5 customers (6.3%) cited switching to Notion (4 SMBs, 1 Mid-Market, 0 Education). In contrast, 0 Education accounts left for Notion (2 left for Google Workspace).

---

Part 2: What IS Happening (The Real Revenue Leaks)

``` Mid-Market Contraction MRR (2026) Jan - Apr Baseline: ~$2,400/mo May: ██████████ $10,239 <-- May 15: Price hike to $15 + Seat Audit tool Jun: ██████████████████████ $22,351 Jul: ████████████████ $16,514 Aug: ████████████ $12,486 Sep: █████████████ $13,212 ```

#### Leak #1: Mid-Market Seat Slashing (The Primary Financial Threat) On May 15, we paired a 25% price increase ($12 to $15/seat) with an admin "Seat Audit" tool that surfaces inactive 60-day seats with a one-click removal button.

  • The Reaction: While May expansion MRR spiked to $31.6k as existing seats re-billed at $15, customers responded by aggressively cutting seats to offset the price hike.
  • The Impact:
  • Mid-Market contraction MRR jumped from a baseline of $2,397/month (Jan–Apr average) to $22,351 in June.
  • Contraction has remained elevated through September ($13,212/month).
  • In total, Mid-Market experienced $74,802 in contraction MRR between May and September—$62,817 above historical baseline.
  • Mid-Market ending MRR peaked in May at $334.9k and has since declined to $321.9k.
  • Exit survey validation: 11 out of 15 cancelled Mid-Market accounts (73%) explicitly cited: "Too expensive per seat after the price change."

#### Leak #2: SMB Team Adoption Failure * SMB churn is relentless, averaging $7,466 in churned MRR each month (~$90k annual run-rate). * Exit survey validation: The #1 reason for SMB churn was "Didn't get the team to adopt it" (17/44 responses, or 38.6%), outranking company failure/downsizing (31.8%). * The March metric change exposed that ~35% of paying SMB accounts lack active editors. When a tool is not operationalized across the team, small businesses churn at the first sign of cost scrutiny.

---

Part 3: Sizing the Revenue Impact

Leak SourceMechanismMonthly Revenue DragCumulative 2026 Impact
:---:---:---:---
Mid-Market Excess ContractionPrice increase shock + 1-click seat audit tool~$10,800/mo above baseline$62,817 MRR lost since May (~$754k ARR run-rate)
SMB Adoption ChurnPoor onboarding / team activation failure~$7,500/mo churned MRR$67,192 MRR lost YTD
Education August Lapsed RenewalsUnmanaged annual renewal cliff & budget cutsOne-time ~$3,000 above baseline$3,000 MRR recurring annual hit

Note on Notion AI: Revenue lost to Notion across all exit survey respondents totals less than $1,200 MRR. Chasing an AI assistant does not address our $750k+ ARR seat-contraction problem.

---

Part 4: Recommendations & Squad Allocation

Rather than committing two squads to build an AI assistant to fight a competitor who is not taking our customers, we should realign our product and go-to-market efforts to protect existing contract values and solve team activation.

#### 1. Reallocate Squads to High-ROI Retention Drivers

  • Squad 1: Mid-Market Packaging & Admin Value (Immediate Priority)
  • Rethink Seat Pruning: Modify the Seat Audit tool. Instead of a direct "remove seat" button that prompts immediate seat downgrades, guide admins to reassign unused licenses to other departments or switch inactive users to free "Viewer/Commenter" roles without reducing tier commitments.
  • Volume & Tier Structuring: Implement tiered seat thresholds (e.g., bundles of 25/50/100 seats) so mid-market accounts are not incentivized to trim single individual seats every billing cycle.
  • Success Target: Reduce Mid-Market contraction from ~$13k/month back toward the $5k/month range, preserving ~$95k in ARR each quarter.
  • Squad 2: SMB Activation & Collaborative Onboarding
  • Target Adoption Failure: Build guided team-onboarding templates, Slack/Google Workspace identity integrations, and automated "first shared document" workflows.
  • Early Warning Telemetry: Flag accounts where <30% of paid seats have edited a doc within 14 days of purchase, triggering targeted in-app adoption guides or CS intervention.
  • Success Target: Decrease SMB "failure to adopt" churn by 25%, saving ~$22k in annual churned MRR.

#### 2. Commercial & Operational Interventions

  • Customer Success Pricing Outreach (Mid-Market): Have Account Managers proactively reach out to Mid-Market accounts experiencing contraction with annual lock-in discounts (e.g., offer $12/seat on annual prepaid agreements instead of $15/seat monthly).
  • Education Annual Renewal Playbook: Education accounts cannot be left unattended until August. Account Managers must run an "Academic Review" campaign in April/May to lock in fall renewals before university administrative teams depart for the summer.
  • Dashboard Governance: Establish data governance protocols. Major metric definition shifts must be annotated on executive dashboards to prevent false alarms and reactive product strategy shifts.

Claims the judge couldn’t find in the brief

  • The May 15 price hike combined with the newly launched 'Seat Audit' tool prompted enterprise admins to aggressively purge inactive seats.
  • SMB churn is relentless, averaging $7,466 in churned MRR each month.
  • Revenue lost to Notion across all exit survey respondents totals less than $1,200 MRR.

Grades and run details

Decision model 65 · LLM judge 57
Decision model checks
  • failUses the supplied evidence correctly62%
  • passAddresses the actual decision61%
  • partialRespects explicit constraints22%
  • failIdentifies material uncertainty89%
  • failAvoids unsupported claims30%
  • passProduces the required deliverable75%
  • passSeparates what isn't churn99%
  • passFinds and sizes the real loss98%
  • passAnswers the AI plan with evidence100%
  • passUnderstands before it justifies100%
  • failGets the base of every number right59%
  • passSplits growth into new, lost and returning86%
  • passReads the shape of the curves99%
Run
Run
#1
API response time
43 s
Submitted
6 Oct 2026

Usable with a quick edit12 of 13 checks passed

The memo is evidence-based, correctly diagnoses the issues, and provides a clear, actionable plan. A PM could use it with minor wording tweaks, without rechecking data or redoing reasoning.

Output

To: Maya Lind, CEO From: Staff PM Subject: Retention diagnosis: a March measurement break, persistent SMB leakage, and a mid-market contraction problem

Recommendation

Do not commit two squads to an AI assistant on this evidence. March’s apparent collapse coincides with a changed activity definition, not a revenue-retention collapse. Education’s summer inactivity largely reverses, although August has a real renewal-season revenue loss. The actionable problems are persistent SMB attrition and a sharp increase in mid-market seat contraction after May’s pricing and seat-audit changes.

I recommend one squad on mid-market pricing/value retention and one on SMB team adoption, with a two-week diagnostic phase before selecting interventions. Keep AI to bounded discovery until we establish that it would address material, recoverable revenue loss.

What is—and isn’t—happening

1. March’s 19% activity drop is not evidence of collapsed retention

Total active accounts fell from 2,697 in February to 2,176 in March, down 19.3%. But on March 2, “active” changed from anyone logging in to anyone creating or editing a page. The dashboard crosses a measurement discontinuity; the persistently lower level does not establish persistently worse retention.

The financial evidence contradicts a March revenue-retention collapse:

  • Paying accounts increased from 3,208 to 3,246.
  • Total ending MRR increased from $573,323 to $588,463.
  • Monthly net revenue retention was 99.4% in both February and March; gross revenue retention was approximately 97.2% in both.

Here, NRR = (starting MRR + expansion − contraction − churn) / starting MRR; GRR excludes expansion as well as new revenue. These are monthly movement-based measures, not tracked-cohort retention.

Growing account and revenue totals alone could hide churn behind acquisition. The stable retention measures are the stronger evidence. We still need historical activity recomputed under one definition to determine whether underlying engagement changed.

2. Education shows seasonality, plus a real August loss—not demonstrated freefall

Under the consistent post-March definition, education activity fell from 443 accounts in May to 201 in July, then recovered to 461 in September. Active/paying penetration was 69.3% in May and 68.9% in September. That pattern fits the academic calendar, though one year cannot establish normal seasonality.

There is a genuine financial signal: August churn was $3,925 MRR, versus roughly $900 in other months—about $3,000 above that baseline. August NRR was 95.0%, and ending MRR fell by $1,417 despite new sales and expansion. September MRR recovered to $72,211, above July, but acquisition does not prove the cancelled customers returned.

Most education contracts renew in August. We need renewal-due accounts and dollars, plus prior-year comparisons, to judge whether this is an abnormal renewal problem. August churn divided by all starting MRR is not an annual renewal-failure rate. The survey’s academic-year and budget explanations support investigation, not dismissal.

3. Mid-market is the new deterioration—and cancellation surveys miss its main mechanism

Mid-market contraction averaged $2,397 MRR/month in January–April. It rose to $10,239 in May, peaked at $22,351 in June, and remained $13,212 in September. Cancellation MRR stayed near $2,600–$2,800 monthly: the change is principally revenue shrinking within retained accounts, not more whole-account cancellations.

Two changes launched together on May 15: a 25% seat-price increase and an audit with one-click removal of inactive seats. Both could contribute; this export cannot separate their effects or establish causation.

May’s expansion jumped to $31,585, pushing NRR to 106.0%. That should not be read as equivalent growth in customer usage: higher prices can create expansion dollars. NRR then fell to 95.1%, 96.8%, 98.0%, and 97.8% in June–September, versus approximately 101% before May.

During June–September, existing-account movements removed $40,585 of net MRR after expansion. New sales replaced $27,607, leaving ending MRR $12,978 below May.

Eleven of 15 mid-market cancellation respondents cited post-change seat pricing. That strengthens the pricing hypothesis, but the survey excludes retained customers removing seats—the central population we must investigate.

4. SMB is the largest persistent net revenue leak

SMB monthly NRR remained approximately 97.2%–97.6% throughout the period: not a March collapse, but sustained erosion masked by acquisition.

Of 44 SMB cancellation responses, 17 cited failed team adoption, 14 closure/downsizing, nine price, and four Notion. Adoption is the clearest product-addressable hypothesis; closure/downsizing may be less recoverable. These are respondent counts, not revenue-weighted or necessarily representative causes.

Size of the revenue problem

The January–September MRR movements reconcile as follows:

SegmentChurn + contractionNet loss after expansion
------:---:
Education$13.9k$8.5k
SMB$81.7k$52.8k
Mid-market$108.8k$9.0k
Total$204.5k$70.4k

These are sums of recurring-revenue movement amounts, not recognized revenue lost during the period. Mid-market’s nine-month net figure masks its post-May deterioration.

New MRR of $167.7k more than offset the $70.4k net erosion, lifting company MRR from $543.0k to $640.3k. Growth therefore does not mean retention is healthy.

For scale, SMB’s $52.8k net MRR erosion represents $634k annualized, and mid-market’s June–September $40.6k represents $487k annualized. These describe recurring-revenue decrements, not forecasts of recoverable revenue. We cannot yet calculate the pricing change’s net causal revenue effect.

Next-quarter plan

First two weeks: establish trustworthy baselines. Data and Finance should backfill consistent activity, reconcile account-level MRR, and distinguish price changes from seat changes. Build education renewal cohorts and segment retention by tenure, acquisition cohort, and revenue.

Squad 1: mid-market retention. Interview seat reducers as well as cancellers; examine removals around each account’s billing change and audit use. Test targeted packaging, phased increases, or commitment options against controls. Measure net retained revenue and margin—not just seats saved. Do not obstruct legitimate removal of unused seats.

Squad 2: SMB adoption. Identify where teams fail between setup, inviting colleagues, and recurring collaboration. Test interventions on those failure points, measuring retained paid teams and cohort GRR/NRR rather than activity alone.

Customer Success: education renewals. Conduct August loss reviews and early budget/renewal outreach; report retention against dollars actually due.

AI: discovery, not two squads. Only five of 79 survey respondents named Notion; none explicitly named AI. Notion’s August 3 launch cannot explain March or May–July deterioration. Interview competitive losses and prototype narrowly; require evidence of revenue-weighted demand and retention impact before scaling investment.

Grades and run details

Decision model 92 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly25%
  • passAddresses the actual decision84%
  • passRespects explicit constraints49%
  • passIdentifies material uncertainty97%
  • passAvoids unsupported claims56%
  • passProduces the required deliverable73%
  • passSeparates what isn't churn93%
  • passFinds and sizes the real loss99%
  • passAnswers the AI plan with evidence100%
  • passUnderstands before it justifies100%
  • failGets the base of every number right21%
  • passSplits growth into new, lost and returning97%
  • passReads the shape of the curves100%
Run
Run
#1
Time to output
87 s
Submitted
5 Oct 2026

Usable with a quick edit12 of 13 checks passed

The output is clear, evidence-based, and actionable; a PM could use it with minor wording tweaks without rechecking data or reasoning.

Output

To: Maya Lind Subject: March retention diagnosis and where revenue is actually at risk

Bottom line: The March dashboard cliff is real as a reported activity change, but it is not evidence that customer retention collapsed. The metric changed on March 2, and paying accounts and MRR continued to grow. There are real, more specific risks: SMB adoption, mid-market seat contraction after pricing and seat-audit changes, and education’s seasonal renewal cycle. The data does not support assigning two squads to an AI assistant on the premise that Notion AI caused a broad retention collapse.

What is—and isn’t—happening

Active accounts fell from 2,697 in February to 2,176 in March, down 19.3%. But on March 2, “active” changed from anyone logged in within 30 days to anyone who created or edited a page within 30 days. That stricter definition plausibly explains the abrupt drop: activity fell by roughly 18–20% in all three segments at once.

The more comparable evidence is what happened after the change. From March to September, active accounts rose from 2,176 to 2,320 (+6.6%): SMB rose 6%, mid-market 7%, and education 8%. SMB and mid-market activity rates were broadly stable over that period. These figures do not show an ongoing collapse—though we cannot make a clean February-to-March retention comparison without recalculating historical activity on one definition.

Education’s summer decline is pronounced: active accounts fell from 428 in March to 201 in July, then recovered to 461 in September. That pattern fits the academic calendar and renewal timing better than a continuous decline. Paying education accounts rose every month, from 625 in March to 669 in September. The activity metric is not the same as paid-account retention, so this is not proof that every education account is secure; it is evidence against calling education “in freefall.”

Revenue also contradicts the broad-collapse diagnosis. Ending MRR rose from $558.6K in January to $640.3K in September, up $81.7K (14.6%). Monthly churned MRR was generally about $10.6K–$11.4K from January through July, rose to $14.1K in August, then returned to $11.1K in September. There is no sustained March deterioration in these figures.

Where the real risks are

  • Mid-market pricing and seat contraction: The mid-market plan price rose 25%, from $12 to $15 per seat, and a one-click seat-removal audit launched in May. Mid-market contraction MRR then reached $10.2K in May, $22.4K in June, $16.5K in July, and $12.5K in August, versus roughly $2.4K per month in the surrounding months. That is about $52K more contraction MRR over May–August than the surrounding run rate would suggest. It is a meaningful warning, but not $52K of confirmed churn: contraction may reflect customers removing idle seats, and the report does not isolate price effects from seat cleanup. Mid-market ending MRR was still $12.8K above April’s by September, after the price change and other movements.
  • SMB adoption: In the exit survey, 17 of 44 SMB respondents cited failure to get their team to adopt Tandem—the largest single SMB reason. Another 14 cited company closure or downsizing, nine cited price, and four said they switched to Notion. This points to an adoption problem as well as some external business distress.
  • Education renewals: Twelve of 20 education respondents said they would review at the end of the academic year; six cited budget cuts. Education churned MRR spiked to $3.9K in August, versus roughly $0.9K in most other months, then returned to $0.9K in September. Prepare for the renewal window rather than treating the summer activity trough as evidence of a year-round collapse.

The exit survey is a sample of 79 cancellations, not a representative churn-rate study. Only five respondents said they switched to Notion (four SMB, one mid-market), and the survey does not say they left because of AI. Notion’s August AI launch is too recent to explain the March dashboard break, and these data do not establish that it is driving the current losses.

Recommended actions

  1. Fix the measurement first. Backfill the new activity definition, annotate the dashboard break, and report paid-account retention, logo churn, revenue churn, and usage separately by segment and renewal cohort.
  2. Investigate mid-market changes account by account. Separate price increases, seat removals, and cancellations; quantify retained seats and renewal outcomes. Test targeted pricing or seat flexibility where value is at risk before considering a broad rollback.
  3. Improve SMB activation. Focus on onboarding, team rollout, and early adoption signals; distinguish preventable adoption failures from business closures.
  4. Run an education renewal plan ahead of the academic-year cycle, with early outreach and budget-sensitive renewal options.
  5. Do not staff two AI squads on this diagnosis. If AI is strategically important, run a bounded customer discovery and prototype test, with evidence of demand and retention impact as a gate for further investment.

Grades and run details

Decision model 92 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly12%
  • passAddresses the actual decision79%
  • passRespects explicit constraints56%
  • passIdentifies material uncertainty78%
  • passAvoids unsupported claims43%
  • passProduces the required deliverable69%
  • passSeparates what isn't churn99%
  • passFinds and sizes the real loss72%
  • passAnswers the AI plan with evidence100%
  • passUnderstands before it justifies100%
  • failGets the base of every number right17%
  • passSplits growth into new, lost and returning55%
  • passReads the shape of the curves99%
Run
Run
#1
API response time
34 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 5

Uses the supplied evidence correctlyWrongRightRight
Gemini 3.8 Flash · API

Claims 7, 12, and 15 are not fully supported: the causal link for seat audit is interpretation, SMB churn average is miscalculated, and Notion revenue is invented.

GPT-6 Astra · ChatGPT

All factual claims about the current situation are directly from the supplied files or derived by correct arithmetic.

GPT-6 Luna · API

All factual claims about the current situation are directly supported by the supplied data or arithmetic.

Addresses the actual decisionMixedRightRight
Gemini 3.8 Flash · API

The memo commits to a clear answer but does not state what result or condition would change the recommendation.

GPT-6 Astra · ChatGPT

The memo commits early to not assigning two squads to AI, recommends alternative squad allocations, and states what would change the AI decision.

GPT-6 Luna · API

The memo unambiguously advises against staffing two AI squads on this diagnosis and states what evidence would change that call.

Identifies material uncertaintyWrongRightRight
Gemini 3.8 Flash · API

The memo does not name specific unknowns that could change the decision or how they would be resolved.

GPT-6 Astra · ChatGPT

It names specific unknowns (historical activity under one definition, renewal-due dollars, causal separation of price and audit) and how to resolve them.

GPT-6 Luna · API

It names specific unknowns (definition change prevents clean comparison, contraction may be seat cleanup not churn, exit survey is a sample) and says how to resolve them.

Avoids unsupported claimsWrongRightRight
Gemini 3.8 Flash · API

It presents the causal claim about the seat audit prompting seat purges as fact and invents a Notion revenue figure without support.

GPT-6 Astra · ChatGPT

Interpretations and causes are labelled as hypotheses or possibilities, not as established fact.

GPT-6 Luna · API

Interpretations are hedged ('plausibly', 'points to', 'may reflect') and not presented as established fact.

Gets the base of every number rightWrongMixedMixed
Gemini 3.8 Flash · API

The SMB churn average of $7,466 is miscalculated; the correct average from the data is $7,444.

GPT-6 Astra · ChatGPT

All derived percentages and rates use correct denominators and steps, with bases stated where needed.

GPT-6 Luna · API

All percentages and differences are computed correctly from the supplied data, with bases stated or clear.

All got right 8

Respects explicit constraintsRightRightRight
Gemini 3.8 Flash · API

The memo is within 1,200 words, addressed to the CEO, and in the requested form.

GPT-6 Astra · ChatGPT

The memo is addressed to Maya, is under 1,200 words, and respects the requested form.

GPT-6 Luna · API

The output is a memo to Maya, well under 1,200 words, and respects the requested form and reader.

Produces the required deliverableRightRightRight
Gemini 3.8 Flash · API

The memo is complete, in the right form, and the CEO could act on it with light edits.

GPT-6 Astra · ChatGPT

The memo is complete, within word limit, and directly usable by the CEO with at most light edits.

GPT-6 Luna · API

The memo is complete, in the right form, within length, and a PM could act on it with light edits.

Separates what isn't churnRightRightRight
Gemini 3.8 Flash · API

It links the March drop to the definition change and shows education's summer dip recovering in September.

GPT-6 Astra · ChatGPT

It links the March drop to the definition change and shows education's summer dip recovering in September.

GPT-6 Luna · API

It links the March drop to the 2 March definition change and shows education's summer dip recovering in September.

Finds and sizes the real lossRightRightRight
Gemini 3.8 Flash · API

It identifies the mid-market contraction jump after May 15, links it to the price rise and seat audit, and sizes it at $62,817 above baseline.

GPT-6 Astra · ChatGPT

It identifies the mid-market contraction jump after May 15, links it to the price rise and seat audit, and sizes it in MRR.

GPT-6 Luna · API

It identifies the mid-market contraction jump after 15 May, links it to the price rise and seat audit, and sizes it at ~$52K excess contraction MRR.

Answers the AI plan with evidenceRightRightRight
Gemini 3.8 Flash · API

It uses exit data and the August Notion launch to show few competitor exits and advises against the AI assistant as a retention fix.

GPT-6 Astra · ChatGPT

It uses exit survey counts and the August Notion launch timeline to argue against an AI assistant as a retention fix.

GPT-6 Luna · API

It uses exit data (only 5 Notion switchers, no AI mention) and the August launch timeline to advise against the AI assistant as a retention fix.

Understands before it justifiesRightRightRight
Gemini 3.8 Flash · API

The diagnosis tests the CEO's explanation against the data and reaches its own conclusions from the evidence.

GPT-6 Astra · ChatGPT

The diagnosis tests the CEO's collapse claim against the data and reaches its own conclusions from the evidence.

GPT-6 Luna · API

The diagnosis starts from the data, tests the CEO's Notion AI explanation, and reaches its own evidence-based conclusions.

Splits growth into new, lost and returningRightRightRight
Gemini 3.8 Flash · API

It separates new, expansion, contraction, and churn MRR to show which flow changed.

GPT-6 Astra · ChatGPT

It separates new, expansion, contraction, and churn MRR to show which flow changed, rather than reading totals.

GPT-6 Luna · API

It uses MRR movements (new, expansion, contraction, churn) to show which flow changed, particularly the contraction spike.

Reads the shape of the curvesRightRightRight
Gemini 3.8 Flash · API

It describes the education dip as seasonal and recovering, and the mid-market contraction spike.

GPT-6 Astra · ChatGPT

It describes the education dip and recovery as seasonal and the mid-market NRR decline after May, drawing correct conclusions.

GPT-6 Luna · API

It reads the education dip and recovery as seasonal and notes the post-March active account rise, not a continued decline.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT96.2100.02None
2GPT-6 LunawithAPI96.2100.02None
3GPT-6.1 SolwithAPI94.1100.02None
4Haiku 5.5withAPI92.0100.02None
5Opus 5.5withClaude82.1100.02None
6Sonnet 5.5withAPI90.189.32None
7Gemini 3.5 Flash-LitewithGemini76.474.22None
8Gemini 3.8 FlashwithAPI72.370.92None

What you get for the cost

GPT-6 Luna · API matches GPT-6 Astra · ChatGPT at about a fiftieth of the cost.

Task score against relative cost for each setup. The table below has the same figures.607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 98, 79× the cheapestGPT-6 Luna · API: task score 98, the cheapestGPT-6.1 Sol · API: task score 97, 15× the cheapestHaiku 5.5 · API: task score 96, 3.6× the cheapestOpus 5.5 · Claude: task score 92, 77× the cheapestSonnet 5.5 · API: task score 88, 50× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 74, 3.0× the cheapestGemini 3.8 Flash · API: task score 71, 13× the cheapestGPT-6 Astra · ChatGPTGPT-6 Luna · APIGPT-6.1 Sol · APIHaiku 5.5 · APIOpus 5.5 · ClaudeSonnet 5.5 · APIGemini 3.5 Flash-Lite · GeminiGemini 3.8 Flash · API
Best value
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API981.0×
Gemini 3.5 Flash-Lite · Gemini743.0×
Haiku 5.5 · API963.6×
Gemini 3.8 Flash · API7113×
GPT-6.1 Sol · API9715×
Sonnet 5.5 · API8850×
Opus 5.5 · Claude9277×
GPT-6 Astra · ChatGPT9879×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Working out why customers leave, and what to fix first.

Why it matters

Teams pick a story (the season, a competitor) and then pull data to justify it. Half the time the 'drop' is a definition change, and the real leak is somewhere else.

What good looks like

  • Starts from what the data shows, not a story
  • Splits growth into new, lost and returning users
  • Separates measurement and seasonality from churn
  • Sizes the real problem and fixes new-user retention first

Deliberately not measured

  • Dashboard design
Capability tested

Retention diagnosis

The failure we’re looking for

Confirming the story everyone already believes

Grading

Decision model and LLM judge, calibrated against a blind PM review