Tasks / Growth

Diagnose a retention problem

Can the model work out from raw data what's really happening to retention, separating churn from measurement and seasonality, and find the cause?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 78% were usable with at most a quick edit.

Reliably right

  1. Respects explicit constraints100% pass
    The output is a diagnosis under 800 words, addressed to Leo, and proposes actions that would enforce the constraint of holding acquisition spend.
    GPT-6.1 Sol · API · Flat, while sign-ups grow
  2. Produces the required deliverable100% pass
    Provides a complete diagnosis with evidence and recommendations, in memo form for Leo, within 800 words, and is actionable as-is.
    GPT-6.1 Sol · API · Flat, while sign-ups grow
  3. Understands before it justifies100% pass
    Tests the summer explanation against the data, finds it insufficient, and reaches its own diagnosis from the cohort and accounting evidence.
    GPT-6.1 Sol · API · Flat, while sign-ups grow

Where it slips

  1. Uses the supplied evidence correctly67% pass
    The SMB adoption churn calculation is incorrect; 39% of ~$7.4k/month churn is ~$2.9k/month, not $34k MRR, and the annualized figure is ~$34.8k ARR, not $0.4M.
    Sonnet 5.5 · API · The collapse that wasn't
  2. Avoids unsupported claims72% pass
    The output states as fact that removing goal-setting 'stripped away the user’s commitment and personal investment' and caused the drop, without labelling this as a hypothesis.
    Gemini 3.8 Flash · API · Flat, while sign-ups grow
  3. Gets the base of every number right75% pass
    The SMB adoption churn calculation is incorrect; 39% of ~$7.4k/month churn is ~$2.9k/month, not $34k MRR, and the annualized figure is ~$34.8k ARR, not $0.4M.
    Sonnet 5.5 · API · The collapse that wasn't

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Understands before it justifies

    Does the diagnosis start from what the data shows (which users, when, at what point in their life with the product) before accepting or proposing an explanation, including the one it was handed?

    Passes when Tests the explanation it was given against the data, and reaches its own diagnosis from the evidence.

  2. Splits growth into new, lost and returning

    Does the diagnosis separate new users or revenue from what was lost and what came back, rather than reading a single total?

    Passes when Uses growth accounting (new, retained, churned, returning, or new, expansion, contraction, churn) to show which flow changed.

  3. Reads the shape of the curves

    Does the diagnosis read how retention curves behave over time (flattening, still falling, or dipping and recovering) and what each shape means?

    Passes when Describes the curve shapes in the data and draws the right conclusion from each, such as a dip that recovers being seasonal.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 3 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a Staff PM at Tandem. Our CEO, Maya Lind, believes retention collapsed in March and wants two squads on an AI assistant next quarter (her note is below). She's asked you to confirm the diagnosis and find where else we're losing customers. Write her a memo of no more than 1,200 words: what is and isn't happening to retention, the evidence for each, the size of the real problem in revenue, and what to do about it. The files are attached, as exported.

What the model was given6 items: About Tandem, ceo_note.md, active_accounts_by_month.csv (from the main dashboard), mrr_movements_by_month.csv, exit_survey_2026.csv (accounts that cancelled), product_and_pricing_changes.md
About TandemA shared workspace for teams: docs, wikis and project pages. Sold per seat per month. Three segments: education (schools and universities, mostly annual contracts renewing in August), SMB, and mid-market.
ceo_note.md1 lines · Download
From Maya (CEO): "Retention collapsed in March and hasn't recovered: active accounts are down by a fifth and education is in freefall. Notion AI is eating us. I want two squads on an AI assistant next quarter. Can you confirm the diagnosis and tell me where else we're bleeding?"
active_accounts_by_month.csv (from the main dashboard)month,segment,paying_accounts,active_accounts 2026-01,Education,610,525 2026-01,SMB,2140,1758 2026-01,Mid-market,420,378 2026-02,Education,617,531 2026-02,SMB,2166,1785 2026-02,Mid-market,425,381 2026-03,Education,625,428 2026-03,SMB,2191,1434 2026-03,Mid-market,430,314 2026-04,Education,632,439 2026-04,SMB,2217,1447 2026-04,Mid-market,435,317 2026-05,Education,639,443 2026-05,SMB,2243,1448 2026-05,Mid-market,440,322 2026-06,Education,647,293 2026-06,SMB,2268,1476 2026-06,Mid-market,445,325 2026-07,Education,654,201 2026-07,SMB,2294,1497 2026-07,Mid-market,450,327 2026-08,Education,661,296 2026-08,SMB,2320,1514 2026-08,Mid-market,455,330 2026-09,Education,669,461 2026-09,SMB,2345,1524 2026-09,Mid-market,460,335
mrr_movements_by_month.csv27 rows · Download
month,segment,starting_mrr_usd,new_mrr_usd,expansion_mrr_usd,contraction_mrr_usd,churned_mrr_usd,ending_mrr_usd
2026-01,Education,61000,2237,603,311,874,62655
2026-01,SMB,214000,9047,3152,1616,7154,217429
2026-01,Mid-market,268000,7253,8246,2374,2613,278512
2026-02,Education,62655,2092,584,300,934,64097
2026-02,SMB,217429,9117,3227,1592,7555,220626
2026-02,Mid-market,278512,6797,8384,2324,2769,288600
…
exit_survey_2026.csv (accounts that cancelled)response_id,segment,main_reason X501,Mid-market,Too expensive per seat after the price change X502,Mid-market,Too expensive per seat after the price change X503,Mid-market,Too expensive per seat after the price change X504,Mid-market,Too expensive per seat after the price change X505,Mid-market,Too expensive per seat after the price change X506,Mid-market,Too expensive per seat after the price change X507,Mid-market,Too expensive per seat after the price change X508,Mid-market,Too expensive per seat after the price change X509,Mid-market,Too expensive per seat after the price change X510,Mid-market,Too expensive per seat after the price change X511,Mid-market,Too expensive per seat after the price change X512,Mid-market,Consolidating tools X513,Mid-market,Consolidating tools X514,Mid-market,Consolidating tools X515,Mid-market,Switched to Notion X516,SMB,Company closed or downsized X517,SMB,Company closed or downsized X518,SMB,Company closed or downsized X519,SMB,Company closed or downsized X520,SMB,Company closed or downsized X521,SMB,Company closed or downsized X522,SMB,Company closed or downsized X523,SMB,Company closed or downsized X524,SMB,Company closed or downsized X525,SMB,Company closed or downsized X526,SMB,Company closed or downsized X527,SMB,Company closed or downsized X528,SMB,Company closed or downsized X529,SMB,Company closed or downsized X530,SMB,Too expensive X531,SMB,Too expensive X532,SMB,Too expensive X533,SMB,Too expensive X534,SMB,Too expensive X535,SMB,Too expensive X536,SMB,Too expensive X537,SMB,Too expensive X538,SMB,Too expensive X539,SMB,Switched to Notion X540,SMB,Switched to Notion X541,SMB,Switched to Notion X542,SMB,Switched to Notion X543,SMB,Didn't get the team to adopt it X544,SMB,Didn't get the team to adopt it X545,SMB,Didn't get the team to adopt it X546,SMB,Didn't get the team to adopt it X547,SMB,Didn't get the team to adopt it X548,SMB,Didn't get the team to adopt it X549,SMB,Didn't get the team to adopt it X550,SMB,Didn't get the team to adopt it X551,SMB,Didn't get the team to adopt it X552,SMB,Didn't get the team to adopt it X553,SMB,Didn't get the team to adopt it X554,SMB,Didn't get the team to adopt it X555,SMB,Didn't get the team to adopt it X556,SMB,Didn't get the team to adopt it X557,SMB,Didn't get the team to adopt it X558,SMB,Didn't get the team to adopt it X559,SMB,Didn't get the team to adopt it X560,Education,"End of academic year, will review" X561,Education,"End of academic year, will review" X562,Education,"End of academic year, will review" X563,Education,"End of academic year, will review" X564,Education,"End of academic year, will review" X565,Education,"End of academic year, will review" X566,Education,"End of academic year, will review" X567,Education,"End of academic year, will review" X568,Education,"End of academic year, will review" X569,Education,"End of academic year, will review" X570,Education,"End of academic year, will review" X571,Education,"End of academic year, will review" X572,Education,Budget cut X573,Education,Budget cut X574,Education,Budget cut X575,Education,Budget cut X576,Education,Budget cut X577,Education,Budget cut X578,Education,Switched to Google Workspace X579,Education,Switched to Google Workspace
product_and_pricing_changes.md6 lines · Download
# Product and pricing changes, 2026

- 2 March: Dashboard metric "active account" changed from "anyone logged in within 30 days" to "anyone created or edited a page within 30 days", to measure real use.
- 15 May: Mid-market plan from $12 to $15 per seat per month, at each account's next monthly bill.
- 15 May: Seat audit launched for admins: a page listing seats with no activity in 60 days, with a one-click "remove seat".
- 3 August: Notion launched its AI assistant on every paid plan.
What a strong answer doesThe answer key the graders mark against

Separates what isn't churn from what is. (1) March: active accounts fall about 17 points in every segment at once while paying accounts keep growing, because on 2 March 'active' changed from logging in to editing: a definition change, not a collapse. (2) Education: active accounts dip June to August and recover in September, the summer break; education MRR grows, with a modest churn bump at August renewals. (3) Logo churn is roughly flat throughout, and Notion is the main reason in only 5 of 79 exit responses, and Notion's AI launched in August, months after the 'collapse'. The real loss is mid-market contraction: after the 15 May rise from $12 to $15 per seat, shipped the same day as the seat audit, contraction jumps from about $2,400 a month to $12,000 to $22,000 a month (about $75,000 in May to September against about $9,600 in January to April), and mid-market MRR stops growing (flat at about $322,000 from June after rising about $10,000 a month), while 11 of 15 mid-market exits cite price per seat. A secondary issue is SMB adoption ('didn't get the team to adopt it' is the top SMB reason). Recommends fixing the mid-market seat economics (such as pricing active seats or a seat-band discount, and rethinking how the seat audit is presented), restating the dashboard on a consistent definition, and not putting two squads on an AI assistant as a retention fix without evidence it would help.

Critical failures (cap the score at 40)
  • Confirms that retention collapsed in March
Task

Staff-level · v1.1

The outputs

9 setups answered this brief.

Needs more than a quick edit8 of 13 checks passed

The memo contains a couple of minor arithmetic mistakes (SMB active ratio 65.4% vs. 65.5%, education churn baseline ~$906) that would require a double-check; otherwise it is well-structured and actionable.

Output

MEMORANDUM

TO: Maya Lind, CEO FROM: Staff Product Manager DATE: October 2026 SUBJECT: Retention Diagnosis, Revenue Impact, and Q4 Squad Allocation

Executive Summary

Retention did not collapse in March, Education is not in freefall, and Notion AI is not driving customer losses.

The apparent ~20% drop in active accounts in March was a reporting artifact caused by a stricter analytics definition launched on March 2. Furthermore, the summer dip in Education usage is normal academic seasonality, which fully rebounded in September. We should not allocate two squads to an AI assistant next quarter.

However, we are bleeding significant revenue in two other areas: 1. Mid-Market Contraction ($62,800/mo in excess contraction since May): On May 15, we simultaneously raised Mid-market prices by 25% ($12 to $15/seat) and launched a "one-click remove seat" audit tool for admins. This prompted admins to aggressively purge read-only seats, wiping out the gains from the price hike and stalling Mid-market growth. 2. SMB Gross Churn (~$7,400/mo in lost MRR): SMB is our largest source of full account cancellations, driven primarily by poor team onboarding and macro business closures.

Instead of building an AI assistant, we should redirect those two squads toward Mid-market pricing/packaging (introducing free "Viewer" seats) and SMB onboarding and activation.

---

1. What Isn’t Happening (Evaluating the Initial Diagnosis)

The March "Collapse" Was a Metric Definition Change * The Evidence: On March 2, we changed the dashboard definition of an "active account" from anyone logging in within 30 days to anyone creating or editing a page within 30 days. Immediately, reported active-to-paying ratios dropped by roughly 20% across all segments simultaneously: * Education: 86.1% (Feb) $\rightarrow$ 68.5% (Mar) * SMB: 82.4% (Feb) $\rightarrow$ 65.4% (Mar) * Mid-market: 89.6% (Feb) $\rightarrow$ 73.0% (Mar) * The Reality: Underlying retention remained completely stable. Paying accounts grew steadily in March (from 3,208 to 3,246 total), and total Churned MRR was flat ($11,258 in Feb vs. $11,403 in Mar). Under the new definition, engagement rates have remained completely stable from March through September (~65% in SMB, ~73% in Mid-market).

Education is Experiencing Normal Seasonality, Not Freefall * The Evidence: Education active accounts dropped to 293 in June and 201 in July, before rebounding to 461 in September (68.9% of paying accounts, matching March's 68.5% rate). Meanwhile, paying Education accounts grew every single month from January (610) to September (669). * The Reality: Schools and universities go on break over the summer. Because most Education contracts are annual and renew in August, we saw a predictable annual churn spike in August ($3,925 vs. a ~$906/mo baseline), with exit surveys citing "End of academic year, will review" (12/20 responses) and "Budget cut" (6/20). By September, Education MRR hit an all-time high of $72,211.

Notion AI is Not Eating Our Customer Base * The Evidence: Notion launched its AI assistant on August 3. However, our total Churned MRR across SMB and Mid-market actually decreased slightly after the launch ($10,386 in July $\rightarrow$ $10,189 in Aug $\rightarrow$ $10,173 in Sep). In our 2026 exit surveys (79 total responses), 0 customers mentioned AI, and only 5 (6.3%) switched to Notion (4 SMB, 1 Mid-market).

---

2. Where We Are Actually Bleeding (and Size of the Problem)

While logo churn is stable, revenue growth has stalled due to two distinct problems in Mid-market and SMB.

#### Problem A: Mid-Market Seat Contraction (The May 15 Self-Inflicted Wound) On May 15, we increased Mid-market pricing from $12 to $15/seat (+25%) and simultaneously launched the Admin Seat Audit page, which flags seats with no activity in 60 days and offers a one-click "remove seat" button.

  • The Dynamics: The price hike generated a one-time $31,585 expansion spike in May (~$23,200 above our ~$8,360/mo baseline). However, because we don't differentiate between "Editors" and "Viewers," the price increase gave admins sticker shock, and the audit tool gave them an immediate mechanism to cut costs by purging passive readers.
  • The Evidence: Mid-market Contraction MRR averaged $2,397/mo from Jan–Apr. Following the May 15 changes, it jumped to $10,239 in May, peaked at $22,351 in June, and remains elevated at $13,212 in September (over 5x the pre-May baseline). In exit surveys, 11 of 15 (73%) Mid-market cancellations cited "Too expensive per seat after the price change."
  • Revenue Impact:
  • From May to September (5 months), total Mid-market contraction was $74,802—representing ~$62,816 in excess contraction above our pre-May baseline.
  • This completely wiped out the ~$23.2k May price-hike expansion, resulting in a net revenue loss of ~$39,600 from the May 15 initiatives.
  • Mid-market MRR is now actively shrinking (falling from $334,910 in May to $321,932 in September), and we are still bleeding ~$10,800/mo in excess contraction ($129,600/yr annualized run-rate).

#### Problem B: SMB Churn & The Activation Gap SMB is our highest-churning segment in both percentage and absolute dollar terms.

  • The Evidence: SMB Churned MRR averages $7,444/mo (~$89,300/yr in lost ARR, or ~3.3% monthly gross MRR churn), accounting for 65% of all company Churned MRR.
  • The Drivers: Out of 44 SMB exit survey responses:
  • 39% (17/44) left because they "Didn't get the team to adopt it."
  • 32% (14/44) left because the "Company closed or downsized" (uncontrollable macro).
  • 20% (9/44) cited "Too expensive."
  • Further, under our new active metric, 35% of paying SMB accounts (~821 accounts in September) did not create or edit a single page in the last 30 days. These dormant accounts are prime candidates for future churn.

---

3. Recommendations: What to Do Next Quarter

Instead of putting two squads on an AI assistant, we should deploy them against the two problems costing us nearly $220,000/yr in combined annualized MRR losses:

#### Squad 1: Mid-Market Packaging, Pricing & Admin Flows Goal: Stop the $10.8k/mo excess contraction bleed by aligning our pricing model with how teams actually use docs and wikis. 1. Introduce Free "Viewer / Commenter" Roles: Wikis and project pages require broad read access across a company, even if only a core group edits them. Charging $15/mo for passive readers forces admins to remove them, which shrinks our footprint inside the account. Make read/comment-only seats free, and charge $15/mo strictly for Editors/Creators. 2. Redesign the Admin Seat Audit Flow: Replace the destructive "one-click remove seat" button with a "Downgrade to Free Viewer" default action. Additionally, ensure the 60-day inactivity check tracks page views rather than just edits, so admins don't accidentally purge executives and cross-functional stakeholders who rely on Tandem to read documentation. 3. Win-Back & Annual Lock-In: Have Customer Success reach out to Mid-market accounts that contracted heavily since May, offering to restore removed employees as free Viewers and offering a $12/Editor rate in exchange for an annual contract commitment.

#### Squad 2: SMB Onboarding, Activation & Health Intervention Goal: Reduce the $7.4k/mo SMB churn rate by solving the #1 controllable churn driver ("Didn't get the team to adopt it") and activating the 35% of paying SMBs that are currently read-only/dormant. 1. Template-Driven Multiplayer Onboarding: New SMB workspaces fail when the admin faces a blank page. Build guided setup flows with pre-built templates (e.g., Company Wiki, Weekly Sprint Tracker) that prompt the admin to invite teammates and assign them specific sections to edit during the first 7 days. 2. Automated Zero-Edit Playbooks: Trigger automated lifecycle nudges and in-app guidance for paying SMB accounts that hit 14 days without creating or editing a page, helping admins migrate existing docs from Google Docs/Notion before renewal. 3. Pause/Downsize Flows: For the 32% of SMBs churning due to downsizing or closures, introduce a self-serve "Pause Subscription" or "Startup Relief" discount tier in the cancellation flow to retain logos through tough macro periods.

Claims the judge couldn’t find in the brief

  • Active-to-paying ratios dropped: Education 86.1% → 68.5%, SMB 82.4% → 65.4%, Mid-market 89.6% → 73.0%.
  • Education contracts are annual and renew in August; saw an annual churn spike in August of $3,925 vs. a ~$906/mo baseline.

Grades and run details

Decision model 62 · LLM judge 64
Decision model checks
  • failUses the supplied evidence correctly31%
  • failAddresses the actual decision89%
  • passRespects explicit constraints43%
  • failIdentifies material uncertainty95%
  • failAvoids unsupported claims22%
  • passProduces the required deliverable86%
  • passSeparates what isn't churn100%
  • passFinds and sizes the real loss98%
  • passAnswers the AI plan with evidence100%
  • passUnderstands before it justifies100%
  • failGets the base of every number right22%
  • passSplits growth into new, lost and returning88%
  • passReads the shape of the curves100%
Run
Run
#1
API response time
22 s
Submitted
13 Oct 2026

Needs more than a quick edit9 of 13 checks passed

The memo contains factual errors (mid-market exit reasons and education churn drivers) that would require re-checking the evidence before use. A PM would need to correct those claims before acting on it.

Output

MEMORANDUM

TO: Maya Lind, CEO FROM: Staff Product Manager DATE: October 1, 2026 SUBJECT: Retention Analysis & Strategic Realignment

*

Maya,

You asked me to investigate your hypothesis that retention collapsed in March due to Notion AI, and to find where else we are bleeding customers.

The short answer: Your diagnosis of a retention collapse in March is incorrect, and Notion AI is not the primary culprit. What happened on March 2nd was a definition change in our analytics dashboard.

However, we do have a severe, bleeding problem in our Mid-market segment that started quietly in May. It has nothing to do with AI competitors and everything to do with a self-inflicted wound in our pricing and seat-management mechanics.

Below is the breakdown of what is and isn't happening, the true revenue impact, and where we should actually deploy our engineering squads next quarter.

---

1. What Is NOT Happening (The March "Collapse")

The Claim: Active accounts dropped sharply in March across all segments, indicating a market-wide retention failure. * Education active accounts: 531 $\rightarrow$ 428 * SMB active accounts: 1,785 $\rightarrow$ 1,434 * Mid-market active accounts: 381 $\rightarrow$ 314

The Reality: This was an artifact of measurement, not a behavioral collapse.

On March 2nd, engineering updated the dashboard metric for an "active account" from “anyone logged in within 30 days” to “anyone created or edited a page within 30 days.” Passive logins (users opening the app and reading without contributing) were stripped out overnight.

The Evidence: * MRR Movements tell the truth: If active accounts had "collapsed by a fifth" in March, revenue would have followed. Yet March MRR grew healthily across the board (SMB ending MRR grew from \$220k to \$223k; Mid-market from \$288k to \$298k). Churning MRR in March (SMB: \$7,728; Mid-market: \$2,779) was virtually identical to February. * The Notion AI Red Herring: Notion launched its AI assistant on August 3rd. March predates this launch by five months. While exit surveys show a tiny trickle of churn to Notion (4 SMB, 1 Mid-market accounts total across the year), it is not a systemic threat driving our numbers down.

---

2. What IS Happening (The Mid-Market Bleed)

While March was a reporting illusion, May 15th brought a real, severe revenue contraction driven by two concurrent product updates: 1. The Mid-market plan price increased from \$12 to \$15 per seat. 2. A "Seat Audit" tool launched for admins, surfacing seats with no activity in 60 days with a one-click "remove seat" button.

The Evidence: * Massive Contraction Spikes: Look at Mid-market contraction MRR (voluntary license shedding/downgrades): * April: \$2,501 * May: \$10,239 (a 309% jump the month of the price hike/audit tool) * June: \$22,351 * July: \$16,514 * August: \$12,486 * September: \$13,212 * Exit Surveys Confirm Price Sensitivity: Every single Mid-market cancellation in the exit log cites “Too expensive per seat after the price change” (combined with tool consolidation). * The Education Annual Cliff: Education accounts show a predictable annual churn spike in August (churned MRR jumped from \$935 in July to \$3,925 in August), entirely driven by the end of the academic cycle (“End of academic year, will review”), not competitive displacement.

---

3. The Size of the Real Problem in Revenue

Our core business is stable, but we are actively amputating our Mid-market expansion engine.

  • The Mid-Market Leak: Prior to May, Mid-market expansion MRR outpaced contraction by roughly \$6,000/month (e.g., Feb expansion was \$8,384 vs contraction of \$2,324).
  • The Reversal: In June, following the price hike and seat audit tool, Mid-market contraction (\$22,351) completely crushed expansion (\$8,565). We hemorrhaged net expansion capability in our highest-value tier.
  • The Net Result: While Mid-market paying accounts continue to creep up slowly (435 in April $\rightarrow$ 460 in September), the revenue per account is being aggressively optimized downward by our own customers using the admin seat audit tool to purge dormant seats they were previously paying for.

---

4. Strategic Recommendations for Next Quarter

Maya, your instinct to mobilize two squads around AI is understandable given market noise, but building an AI assistant will not solve a pricing-induced contraction problem or an academic seasonality cycle.

Instead, I recommend the following adjustment to next quarter’s roadmap:

1. Scrap the Notion AI panic response. Do not put two squads on an AI assistant. Notion’s bundle play works for them because of their architecture, but copying them reactively without product-market validation in our core workspace loops is a trap. 2. Deploy Squad 1 to Mid-Market Value Realization & Packaging. * The \$12 $\rightarrow$ \$15 price jump triggered a violent contraction because enterprise buyers immediately audited and purged unproven seats. * Instead of raw seat-pricing friction, we need tiered packaging (e.g., introducing advanced security, admin controls, or compliance features at the \$15 tier) rather than a naked price hike on the base tier that incentivizes seat-cutting. 3. Deploy Squad 2 to Education Onboarding & Workspace Activation. * Education is entering its post-August renewal trough. Squad 2 should focus on automated workspace templates and cross-semester continuity features that reduce the friction of the annual academic reset, preventing schools from hitting the "pause" button every summer.

We do not have a top-of-funnel retention collapse. We have a self-inflicted Mid-market pricing shock. Let’s fix the bucket before we build new ways to pour water into it.

Claims the judge couldn’t find in the brief

  • Every single Mid-market cancellation in the exit log cites 'Too expensive per seat after the price change' (combined with tool consolidation).
  • Education accounts show a predictable annual churn spike in August (churned MRR jumped from $935 in July to $3,925 in August), entirely driven by the end of the academic cycle.
  • The revenue per account is being aggressively optimized downward by our own customers using the admin seat audit tool to purge dormant seats they were previously paying for.

Grades and run details

Decision model 73 · LLM judge 71
Decision model checks
  • failUses the supplied evidence correctly25%
  • failAddresses the actual decision85%
  • passRespects explicit constraints58%
  • failIdentifies material uncertainty86%
  • partialAvoids unsupported claims38%
  • passProduces the required deliverable46%
  • passSeparates what isn't churn25%
  • passFinds and sizes the real loss94%
  • passAnswers the AI plan with evidence98%
  • passUnderstands before it justifies98%
  • passGets the base of every number right10%
  • passSplits growth into new, lost and returning85%
  • passReads the shape of the curves67%
Run
Run
#1
Time to output
7 s
Submitted
5 Oct 2026

Usable with a quick edit12 of 13 checks passed

The memo is well-structured, evidence-based, and directly actionable with only minor wording tweaks needed.

Output

To: Maya Lind From: [Staff PM] Re: Retention diagnosis and where we're actually losing revenue

---

Bottom line

Retention did not collapse in March, and education is not in freefall. The March drop comes from a change in how we count active accounts. The education dips follow the school calendar. The data doesn't support Notion AI as the driver either: it launched five months after the March drop and shows up in 6% of exit responses.

We do have a real problem, and it's in mid-market. Since the 15 May price rise and seat audit, mid-market accounts have been removing seats faster than we add revenue. The segment went from growing about $10k MRR a month to shrinking slightly. It is now about $40k MRR (about $475k ARR) below where it was heading, and the excess seat removals haven't stopped.

I recommend we don't put two squads on an AI assistant yet. Put one on mid-market pricing and seat value, and one on SMB adoption.

---

1. What isn't happening

The March "collapse" is a metric redefinition. On 2 March the dashboard changed "active" from logged in within 30 days to created or edited a page within 30 days.

  • Active accounts fell from 2,697 to 2,176 (−19%, your "fifth") in exactly that month.
  • Paying accounts rose every single month, from 3,170 in January to 3,474 in September.
  • The active share dropped evenly in every segment at once, which is the signature of a definition change rather than customer behaviour:
SegmentActive share, FebActive share, MarActive share, Sep
Education86%68%69%
SMB82%65%65%
Mid-market90%73%73%

Since March the active share has been flat. The bar went up and nothing has deteriorated against it.

Education is seasonal, not in freefall. Education active accounts fell to 293 in June and 201 in July, then recovered to 461 in September. That September share (69%) matches March. Paying education accounts grew every month, from 610 to 669. Teachers don't edit pages in July.

August renewals did cost us something. Education churned MRR was $3,925 against a normal ~$900, so about $3k MRR (~4% of the segment) went at renewal. We have no August 2025 figure to compare against, so I can't yet say whether that's worse than usual. Education MRR still ended September at $72.2k, up 18% since January.

Notion AI isn't visible in the data. - It launched on 3 August, five months after the March drop. - "Switched to Notion" accounts for 5 of 79 exit responses (6%): 4 SMB and 1 mid-market. Education has none (2 cited Google Workspace). - Total company MRR grew from $543k in January to $640k in September, up 18%.

Notion is a competitor to watch, but it isn't what is costing us money today.

2. What is happening

Mid-market: price rise plus seat audit, both on 15 May

On 15 May we raised mid-market from $12 to $15 per seat (+25%). On the same day we gave admins a one-click way to remove seats with no activity in 60 days.

MonthExpansionContractionChurnEnding MRR
Jan–Apr avg$8.4k$2.4k$2.7k+$10.3k/mo growth
May$31.6k$10.2k$2.8k$334.9k
Jun$8.6k$22.4k$2.6k$325.4k
Jul$8.8k$16.5k$2.8k$321.6k
Aug$8.7k$12.5k$2.7k$322.0k
Sep$8.7k$13.2k$2.7k$321.9k

Three things stand out:

  • Customers aren't leaving. Churn is flat and paying accounts keep rising. Instead, customers are cutting seats.
  • Excess contraction since May totals ~$63k MRR, measured against the $2.4k baseline. The price rise added only a one-off ~$23k of extra expansion. The net loss is ~$40k MRR. This matches the gap between actual September MRR ($322k) and the pre-May trend (~$361k).
  • It isn't a one-time cleanup. September contraction is still 5.5× baseline. We're losing roughly $11k MRR a month above normal, which adds about $130k ARR of loss each month it continues.

The exit survey points the same way: 11 of 15 mid-market leavers cite "too expensive per seat after the price change."

My read is that the two changes reinforce each other. We made each seat 25% more expensive and, on the same day, handed admins a list of idle seats with a delete button. I can't yet separate the two effects. That's the first thing to measure.

SMB: stable, but a persistent leak we can fix

SMB is steady. MRR is growing about $3.6k a month and churn is flat at about $7.5k a month. Still, that's ~3.1% of SMB MRR lost every month, roughly a third a year, and it's our largest absolute churn.

Of 44 SMB exit responses:

ReasonResponses
Didn't get the team to adopt it17 (39%)
Company closed or downsized14 (32%), not addressable
Too expensive9 (20%)
Switched to Notion4 (9%)

Adoption is the biggest reason we can actually act on. It fits a broader signal: about a third of paying accounts don't create or edit anything in a given month.

Education: winnable lapses

Of 20 education exits, 12 said "end of academic year, will review." These are deferrals, not rejections, and we should win them back before term starts. Another 6 cited budget cuts.

3. Recommendations

1. Squad 1: mid-market pricing and seat value (urgent). - Within two weeks, split contraction by source: seat-audit removals versus other seat reductions. - Interview 10–15 contracting accounts. - Options to test: - Make the audit default to "reassign" or "nudge user" rather than "remove". - Offer annual or volume pricing to soften the per-seat increase. - Introduce a cheaper light or viewer seat. - Proactively offer at-risk accounts a temporary rate. - Target: bring contraction back toward ~$3k a month.

2. Squad 2: SMB activation. - Focus on first-30-day team adoption: invites, templates, a first shared project. - Measure success as the active share in month two and SMB churn.

3. AI assistant: discovery, not two squads. - Run a small spike or customer research. - Track Notion mentions in exit surveys from August onward. - Revisit next quarter if "switched to Notion" or AI-related reasons start rising.

4. Education win-back. - Contact the 12 "will review" accounts now. - Pull August 2025 renewal churn to judge whether this year's ~$3k was unusual.

5. Fix the dashboard. - Restate January–February on the new active definition, or annotate the break, so nobody reads March as a collapse again. - Show education with a year-on-year view.

4. Caveats

  • The exit survey is undated, single-choice and small (79 responses). We can't see whether Notion mentions rose after August. Responses should carry a date going forward.
  • The recorded price uplift (~$23k) is smaller than a full 25% on ~$309k of MRR would imply (~$77k). Finance should confirm how the increase was booked. If some of it was netted into contraction, the mid-market loss could look somewhat different.
  • Education accounts rose in August even as churned MRR quadrupled. The account and MRR files may count things differently, and they need reconciling.

None of these caveats change the conclusion: no March collapse, no education freefall, no evidence yet of a Notion problem, and a real, ongoing mid-market seat loss we caused ourselves and can fix.

Grades and run details

Decision model 92 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly9%
  • passAddresses the actual decision88%
  • passRespects explicit constraints56%
  • passIdentifies material uncertainty95%
  • passAvoids unsupported claims28%
  • passProduces the required deliverable80%
  • passSeparates what isn't churn100%
  • passFinds and sizes the real loss98%
  • passAnswers the AI plan with evidence100%
  • passUnderstands before it justifies100%
  • passGets the base of every number right9%
  • passSplits growth into new, lost and returning98%
  • passReads the shape of the curves100%
Run
Run
#1
Time to output
67 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 5

Uses the supplied evidence correctlyMixedWrongMixed
Gemini 4 Argon · API

All key facts are taken correctly from the supplied context; minor arithmetic rounding issues do not invent new facts.

Gemini 3.5 Flash-Lite · Gemini

Two claims are factually wrong: 'Every single Mid-market cancellation' cites price (only 11 of 15 do), and education churn is 'entirely driven by the end of the academic cycle' (budget cuts and Google Workspace switches also appear).

Opus 5.5 · Claude

All facts and figures are taken correctly from the supplied data, with no invented information.

Addresses the actual decisionWrongMixedRight
Gemini 4 Argon · API

The output commits to a clear call (no AI squads, redirect) but does not state what result or condition would change that recommendation.

Gemini 3.5 Flash-Lite · Gemini

The memo commits early and unambiguously that the March collapse is incorrect and Notion AI is not the culprit, and recommends specific squad deployments instead of the AI plan.

Opus 5.5 · Claude

Commits to a clear answer (no March collapse, real problem is mid-market contraction, don't put two squads on AI) and states what would change the call (rising Notion mentions).

Identifies material uncertaintyWrongWrongRight
Gemini 4 Argon · API

The memo does not name any unknowns that could change the decision or say how they would be resolved.

Gemini 3.5 Flash-Lite · Gemini

The memo does not name specific unknowns that could change the decision, nor does it say what would change the call or how to resolve them.

Opus 5.5 · Claude

Caveats section names unknowns (exit survey dating, price uplift booking, education account reconciliation) and how to resolve them.

Avoids unsupported claimsWrongWrongRight
Gemini 4 Argon · API

The causal claim that the seat audit 'prompted admins to aggressively purge read-only seats' is presented as established fact without direct evidence beyond timing.

Gemini 3.5 Flash-Lite · Gemini

It presents 'Every single Mid-market cancellation' and 'entirely driven by the end of the academic cycle' as established facts, and states the seat audit tool is the cause of contraction without labeling it as a hypothesis.

Opus 5.5 · Claude

Interpretations are labelled as such (e.g., 'My read is...'), and confident claims are supported by evidence.

Gets the base of every number rightWrongRightRight
Gemini 4 Argon · API

The SMB active-to-paying ratio for March is given as 65.4% instead of the correct 65.5%; the Education churn baseline is ~$906 not the computed ~$912, making figures slightly off.

Gemini 3.5 Flash-Lite · Gemini

Derived figures like the 309% jump and the $6,000/month expansion-contraction gap are computed correctly from the supplied data.

Opus 5.5 · Claude

All percentages and ratios (active share, 6%, 39%, 5.5×) are computed correctly with clear denominators.

All got right 8

Respects explicit constraintsRightRightRight
Gemini 4 Argon · API

The memo is well under 1,200 words, addressed to CEO Maya Lind, and covers the required topics.

Gemini 3.5 Flash-Lite · Gemini

The output is a memo addressed to the CEO, under 1,200 words, and respects the requested form and length.

Opus 5.5 · Claude

Memo is under 1,200 words, addressed to Maya Lind, and respects the requested form.

Produces the required deliverableRightRightRight
Gemini 4 Argon · API

The output is a complete, actionable memo that the CEO could use with light edits.

Gemini 3.5 Flash-Lite · Gemini

The memo is complete, in the correct form for the CEO, within the word limit, and actionable with clear recommendations.

Opus 5.5 · Claude

The memo is complete, actionable, and within the word limit for the CEO.

Separates what isn't churnRightRightRight
Gemini 4 Argon · API

Clearly links the March drop to the 2 March definition change and shows education's summer dip recovering in September.

Gemini 3.5 Flash-Lite · Gemini

It clearly links the March drop to the definition change on 2 March and shows education's summer dip recovering in September, using the dashboard change log and active account data.

Opus 5.5 · Claude

Shows March drop is the 2 March definition change and education summer dip recovers in September.

Finds and sizes the real lossRightRightRight
Gemini 4 Argon · API

Identifies the mid-market contraction spike after the 15 May price rise and seat audit, and sizes it in MRR with specific numbers.

Gemini 3.5 Flash-Lite · Gemini

It identifies the mid-market contraction spike after 15 May, links it to the price rise and seat audit, and sizes it with monthly contraction MRR figures.

Opus 5.5 · Claude

Identifies mid-market contraction jump after 15 May, links to price rise and seat audit, and sizes it at ~$40k MRR net loss.

Answers the AI plan with evidenceRightRightRight
Gemini 4 Argon · API

Uses exit surveys and the August Notion launch timeline to show AI is not behind the losses and advises against the AI squad plan.

Gemini 3.5 Flash-Lite · Gemini

It uses the exit data (few Notion switchers) and the August launch timeline to argue against the AI assistant as a retention fix, and advises against two squads on it.

Opus 5.5 · Claude

Uses exit data (5/79) and timeline (August launch) to advise against AI assistant as a retention fix.

Understands before it justifiesRightRightRight
Gemini 4 Argon · API

Tests the CEO's diagnosis against the data and reaches its own conclusions from the evidence.

Gemini 3.5 Flash-Lite · Gemini

The memo tests the CEO's hypothesis against the data, rejects the March collapse narrative, and reaches its own diagnosis from the evidence.

Opus 5.5 · Claude

Tests the CEO's collapse diagnosis against the data and reaches its own conclusion from the evidence.

Splits growth into new, lost and returningRightRightRight
Gemini 4 Argon · API

Uses new, expansion, contraction and churn MRR to show which flows changed (e.g., expansion spike, contraction jump).

Gemini 3.5 Flash-Lite · Gemini

It uses MRR components (new, expansion, contraction, churn) to separate contraction from churn and show which flow changed.

Opus 5.5 · Claude

Splits MRR into new, expansion, contraction, churn, and shows contraction, not churn, is the problem.

Reads the shape of the curvesRightRightRight
Gemini 4 Argon · API

Describes the dip-and-recover shape of education and the stable ratios after the definition change.

Gemini 3.5 Flash-Lite · Gemini

It reads the education dip and September recovery as seasonal, and the mid-market contraction as a spike after May, drawing correct conclusions from the curve shapes.

Opus 5.5 · Claude

Describes education dip as seasonal recovery and mid-market contraction as ongoing, not a one-time cleanup.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstrawithChatGPT100.0100.02None
2Haiku 5.5withAPI100.0100.02None
3GPT-6 LunawithAPI96.2100.02None
4GPT-6.1 SolwithAPI92.3100.02None
5Opus 5.5withClaude85.7100.02None
6Sonnet 5.5withAPI92.389.32None
7Gemini 3.5 Flash-LitewithGemini80.374.22None
8Gemini 4 ArgonwithAPI70.482.12None
9Gemini 3.8 FlashwithAPI76.470.92None

What you get for the cost

GPT-6 Luna · API scores within 2 points of GPT-6 Astra · ChatGPT at about a fiftieth of the cost.

Task score against relative cost for each setup. The table below has the same figures.607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 100, 79× the cheapestHaiku 5.5 · API: task score 100, 3.6× the cheapestGPT-6 Luna · API: task score 98, the cheapestGPT-6.1 Sol · API: task score 96, 15× the cheapestOpus 5.5 · Claude: task score 93, 77× the cheapestSonnet 5.5 · API: task score 89, 50× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 76, 3.0× the cheapestGemini 4 Argon · API: task score 76, 17× the cheapestGemini 3.8 Flash · API: task score 73, 13× the cheapestGPT-6 Astra · ChatGPTHaiku 5.5 · APIGPT-6 Luna · APIGPT-6.1 Sol · APIOpus 5.5 · ClaudeSonnet 5.5 · APIGemini 3.5 Flash-Lite · GeminiGemini 4 Argon · APIGemini 3.8 Flash · API
Best valueNobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API981.0×
Gemini 3.5 Flash-Lite · Gemini763.0×
Haiku 5.5 · API1003.6×
Gemini 3.8 Flash · API7313×
GPT-6.1 Sol · API9615×
Gemini 4 Argon · API7617×
Sonnet 5.5 · API8950×
Opus 5.5 · Claude9377×
GPT-6 Astra · ChatGPT10079×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Working out why customers leave, and what to fix first.

Why it matters

Teams pick a story (the season, a competitor) and then pull data to justify it. Half the time the 'drop' is a definition change, and the real leak is somewhere else.

What good looks like

  • Starts from what the data shows, not a story
  • Splits growth into new, lost and returning users
  • Separates measurement and seasonality from churn
  • Sizes the real problem and fixes new-user retention first

Deliberately not measured

  • Dashboard design
Capability tested

Retention diagnosis

The failure we’re looking for

Confirming the story everyone already believes

Grading

Decision model and LLM judge, calibrated against a blind PM review