Humanity’s Last PRD
Latest drop: Haiku 5.5 · 8 Oct
Tasks / Growth

Diagnose a retention problem

Can the model work out from raw data what's really happening to retention, separating churn from measurement and seasonality, and find the cause?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 81% were usable with at most a quick edit.

Reliably right

  1. Produces the required deliverable100% pass
    The diagnosis is complete, actionable, and in the right format for Leo; a PM could act on it with light edits.
    GPT-6 Astra · ChatGPT · Flat, while sign-ups grow
  2. Understands before it justifies100% pass
    It tests the summer explanation against the data, finds the break aligns with the onboarding change not the paid spend increase, and reaches its own diagnosis from the evidence.
    GPT-6 Astra · ChatGPT · Flat, while sign-ups grow
  3. Splits growth into new, lost and returning100% pass
    It splits growth into new, retained, churned, and returning users, and uses the net growth equation to show which flow changed.
    GPT-6 Astra · ChatGPT · Flat, while sign-ups grow

Where it slips

  1. Gets the base of every number right61% pass
    The SMB adoption churn calculation is incorrect; 39% of ~$7.4k/month churn is ~$2.9k/month, not $34k MRR, and the annualized figure is ~$34.8k ARR, not $0.4M.
    Sonnet 5.5 · API · The collapse that wasn't
  2. Avoids unsupported claims67% pass
    The output presents causal interpretations (e.g., onboarding changes broke retention, low-intent users) as established fact without labelling them as hypotheses.
    Gemini 3.5 Flash-Lite · Gemini · Flat, while sign-ups grow
  3. Identifies material uncertainty77% pass
    The output does not name specific unknowns that could change the decision or say how they would be resolved.
    Gemini 3.5 Flash-Lite · Gemini · Flat, while sign-ups grow

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Understands before it justifies

    Does the diagnosis start from what the data shows (which users, when, at what point in their life with the product) before accepting or proposing an explanation, including the one it was handed?

    Passes when Tests the explanation it was given against the data, and reaches its own diagnosis from the evidence.

    Bangaly KabaLenny’s Podcast

  2. Splits growth into new, lost and returning

    Does the diagnosis separate new users or revenue from what was lost and what came back, rather than reading a single total?

    Passes when Uses growth accounting (new, retained, churned, returning, or new, expansion, contraction, churn) to show which flow changed.

    Naomi GleitLenny’s Podcast

  3. Reads the shape of the curves

    Does the diagnosis read how retention curves behave over time (flattening, still falling, or dipping and recovering) and what each shape means?

    Passes when Describes the curve shapes in the data and draws the right conclusion from each, such as a dip that recovers being seasonal.

    Product and growth leadersOn Lenny’s Podcast

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 3 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're a Staff PM at Tandem. Our CEO, Maya Lind, believes retention collapsed in March and wants two squads on an AI assistant next quarter (her note is below). She's asked you to confirm the diagnosis and find where else we're losing customers. Write her a memo of no more than 1,200 words: what is and isn't happening to retention, the evidence for each, the size of the real problem in revenue, and what to do about it. The files are attached, as exported.

What the model was given6 items: About Tandem, ceo_note.md, active_accounts_by_month.csv (from the main dashboard), mrr_movements_by_month.csv, exit_survey_2026.csv (accounts that cancelled), product_and_pricing_changes.md
About TandemA shared workspace for teams: docs, wikis and project pages. Sold per seat per month. Three segments: education (schools and universities, mostly annual contracts renewing in August), SMB, and mid-market.
ceo_note.md1 lines · Download
From Maya (CEO): "Retention collapsed in March and hasn't recovered: active accounts are down by a fifth and education is in freefall. Notion AI is eating us. I want two squads on an AI assistant next quarter. Can you confirm the diagnosis and tell me where else we're bleeding?"
active_accounts_by_month.csv (from the main dashboard)month,segment,paying_accounts,active_accounts 2026-01,Education,610,525 2026-01,SMB,2140,1758 2026-01,Mid-market,420,378 2026-02,Education,617,531 2026-02,SMB,2166,1785 2026-02,Mid-market,425,381 2026-03,Education,625,428 2026-03,SMB,2191,1434 2026-03,Mid-market,430,314 2026-04,Education,632,439 2026-04,SMB,2217,1447 2026-04,Mid-market,435,317 2026-05,Education,639,443 2026-05,SMB,2243,1448 2026-05,Mid-market,440,322 2026-06,Education,647,293 2026-06,SMB,2268,1476 2026-06,Mid-market,445,325 2026-07,Education,654,201 2026-07,SMB,2294,1497 2026-07,Mid-market,450,327 2026-08,Education,661,296 2026-08,SMB,2320,1514 2026-08,Mid-market,455,330 2026-09,Education,669,461 2026-09,SMB,2345,1524 2026-09,Mid-market,460,335
mrr_movements_by_month.csv27 rows · Download
month,segment,starting_mrr_usd,new_mrr_usd,expansion_mrr_usd,contraction_mrr_usd,churned_mrr_usd,ending_mrr_usd
2026-01,Education,61000,2237,603,311,874,62655
2026-01,SMB,214000,9047,3152,1616,7154,217429
2026-01,Mid-market,268000,7253,8246,2374,2613,278512
2026-02,Education,62655,2092,584,300,934,64097
2026-02,SMB,217429,9117,3227,1592,7555,220626
2026-02,Mid-market,278512,6797,8384,2324,2769,288600
…
exit_survey_2026.csv (accounts that cancelled)response_id,segment,main_reason X501,Mid-market,Too expensive per seat after the price change X502,Mid-market,Too expensive per seat after the price change X503,Mid-market,Too expensive per seat after the price change X504,Mid-market,Too expensive per seat after the price change X505,Mid-market,Too expensive per seat after the price change X506,Mid-market,Too expensive per seat after the price change X507,Mid-market,Too expensive per seat after the price change X508,Mid-market,Too expensive per seat after the price change X509,Mid-market,Too expensive per seat after the price change X510,Mid-market,Too expensive per seat after the price change X511,Mid-market,Too expensive per seat after the price change X512,Mid-market,Consolidating tools X513,Mid-market,Consolidating tools X514,Mid-market,Consolidating tools X515,Mid-market,Switched to Notion X516,SMB,Company closed or downsized X517,SMB,Company closed or downsized X518,SMB,Company closed or downsized X519,SMB,Company closed or downsized X520,SMB,Company closed or downsized X521,SMB,Company closed or downsized X522,SMB,Company closed or downsized X523,SMB,Company closed or downsized X524,SMB,Company closed or downsized X525,SMB,Company closed or downsized X526,SMB,Company closed or downsized X527,SMB,Company closed or downsized X528,SMB,Company closed or downsized X529,SMB,Company closed or downsized X530,SMB,Too expensive X531,SMB,Too expensive X532,SMB,Too expensive X533,SMB,Too expensive X534,SMB,Too expensive X535,SMB,Too expensive X536,SMB,Too expensive X537,SMB,Too expensive X538,SMB,Too expensive X539,SMB,Switched to Notion X540,SMB,Switched to Notion X541,SMB,Switched to Notion X542,SMB,Switched to Notion X543,SMB,Didn't get the team to adopt it X544,SMB,Didn't get the team to adopt it X545,SMB,Didn't get the team to adopt it X546,SMB,Didn't get the team to adopt it X547,SMB,Didn't get the team to adopt it X548,SMB,Didn't get the team to adopt it X549,SMB,Didn't get the team to adopt it X550,SMB,Didn't get the team to adopt it X551,SMB,Didn't get the team to adopt it X552,SMB,Didn't get the team to adopt it X553,SMB,Didn't get the team to adopt it X554,SMB,Didn't get the team to adopt it X555,SMB,Didn't get the team to adopt it X556,SMB,Didn't get the team to adopt it X557,SMB,Didn't get the team to adopt it X558,SMB,Didn't get the team to adopt it X559,SMB,Didn't get the team to adopt it X560,Education,"End of academic year, will review" X561,Education,"End of academic year, will review" X562,Education,"End of academic year, will review" X563,Education,"End of academic year, will review" X564,Education,"End of academic year, will review" X565,Education,"End of academic year, will review" X566,Education,"End of academic year, will review" X567,Education,"End of academic year, will review" X568,Education,"End of academic year, will review" X569,Education,"End of academic year, will review" X570,Education,"End of academic year, will review" X571,Education,"End of academic year, will review" X572,Education,Budget cut X573,Education,Budget cut X574,Education,Budget cut X575,Education,Budget cut X576,Education,Budget cut X577,Education,Budget cut X578,Education,Switched to Google Workspace X579,Education,Switched to Google Workspace
product_and_pricing_changes.md6 lines · Download
# Product and pricing changes, 2026

- 2 March: Dashboard metric "active account" changed from "anyone logged in within 30 days" to "anyone created or edited a page within 30 days", to measure real use.
- 15 May: Mid-market plan from $12 to $15 per seat per month, at each account's next monthly bill.
- 15 May: Seat audit launched for admins: a page listing seats with no activity in 60 days, with a one-click "remove seat".
- 3 August: Notion launched its AI assistant on every paid plan.
What a strong answer doesThe answer key the graders mark against

Separates what isn't churn from what is. (1) March: active accounts fall about 17 points in every segment at once while paying accounts keep growing, because on 2 March 'active' changed from logging in to editing: a definition change, not a collapse. (2) Education: active accounts dip June to August and recover in September, the summer break; education MRR grows, with a modest churn bump at August renewals. (3) Logo churn is roughly flat throughout, and Notion is the main reason in only 5 of 79 exit responses, and Notion's AI launched in August, months after the 'collapse'. The real loss is mid-market contraction: after the 15 May rise from $12 to $15 per seat, shipped the same day as the seat audit, contraction jumps from about $2,400 a month to $12,000 to $22,000 a month (about $75,000 in May to September against about $9,600 in January to April), and mid-market MRR stops growing (flat at about $322,000 from June after rising about $10,000 a month), while 11 of 15 mid-market exits cite price per seat. A secondary issue is SMB adoption ('didn't get the team to adopt it' is the top SMB reason). Recommends fixing the mid-market seat economics (such as pricing active seats or a seat-band discount, and rethinking how the seat audit is presented), restating the dashboard on a consistent definition, and not putting two squads on an AI assistant as a retention fix without evidence it would help.

Critical failures (cap the score at 40)
  • Confirms that retention collapsed in March
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

GPT-6 AstraChatGPT96Usable with a quick editGPT-6 LunaAPI96Usable with a quick editGPT-6.1 SolAPI96Usable with a quick editHaiku 5.5API96Usable with a quick editOpus 5.5Claude91Usable with a quick editSonnet 5.5API80Needs more workGemini 3.5 Flash-LiteGemini66Needs more workGemini 3.8 FlashAPI59Needs more work
Gemini 3.5 Flash-LiteGemini

Needs more than a quick edit9 of 13 checks passed

The memo contains factual errors (mid-market exit reasons and education churn drivers) that would require re-checking the evidence before use. A PM would need to correct those claims before acting on it.

Output

MEMORANDUM

TO: Maya Lind, CEO FROM: Staff Product Manager DATE: October 1, 2026 SUBJECT: Retention Analysis & Strategic Realignment

*

Maya,

You asked me to investigate your hypothesis that retention collapsed in March due to Notion AI, and to find where else we are bleeding customers.

The short answer: Your diagnosis of a retention collapse in March is incorrect, and Notion AI is not the primary culprit. What happened on March 2nd was a definition change in our analytics dashboard.

However, we do have a severe, bleeding problem in our Mid-market segment that started quietly in May. It has nothing to do with AI competitors and everything to do with a self-inflicted wound in our pricing and seat-management mechanics.

Below is the breakdown of what is and isn't happening, the true revenue impact, and where we should actually deploy our engineering squads next quarter.

---

1. What Is NOT Happening (The March "Collapse")

The Claim: Active accounts dropped sharply in March across all segments, indicating a market-wide retention failure. * Education active accounts: 531 $\rightarrow$ 428 * SMB active accounts: 1,785 $\rightarrow$ 1,434 * Mid-market active accounts: 381 $\rightarrow$ 314

The Reality: This was an artifact of measurement, not a behavioral collapse.

On March 2nd, engineering updated the dashboard metric for an "active account" from “anyone logged in within 30 days” to “anyone created or edited a page within 30 days.” Passive logins (users opening the app and reading without contributing) were stripped out overnight.

The Evidence: * MRR Movements tell the truth: If active accounts had "collapsed by a fifth" in March, revenue would have followed. Yet March MRR grew healthily across the board (SMB ending MRR grew from \$220k to \$223k; Mid-market from \$288k to \$298k). Churning MRR in March (SMB: \$7,728; Mid-market: \$2,779) was virtually identical to February. * The Notion AI Red Herring: Notion launched its AI assistant on August 3rd. March predates this launch by five months. While exit surveys show a tiny trickle of churn to Notion (4 SMB, 1 Mid-market accounts total across the year), it is not a systemic threat driving our numbers down.

---

2. What IS Happening (The Mid-Market Bleed)

While March was a reporting illusion, May 15th brought a real, severe revenue contraction driven by two concurrent product updates: 1. The Mid-market plan price increased from \$12 to \$15 per seat. 2. A "Seat Audit" tool launched for admins, surfacing seats with no activity in 60 days with a one-click "remove seat" button.

The Evidence: * Massive Contraction Spikes: Look at Mid-market contraction MRR (voluntary license shedding/downgrades): * April: \$2,501 * May: \$10,239 (a 309% jump the month of the price hike/audit tool) * June: \$22,351 * July: \$16,514 * August: \$12,486 * September: \$13,212 * Exit Surveys Confirm Price Sensitivity: Every single Mid-market cancellation in the exit log cites “Too expensive per seat after the price change” (combined with tool consolidation). * The Education Annual Cliff: Education accounts show a predictable annual churn spike in August (churned MRR jumped from \$935 in July to \$3,925 in August), entirely driven by the end of the academic cycle (“End of academic year, will review”), not competitive displacement.

---

3. The Size of the Real Problem in Revenue

Our core business is stable, but we are actively amputating our Mid-market expansion engine.

  • The Mid-Market Leak: Prior to May, Mid-market expansion MRR outpaced contraction by roughly \$6,000/month (e.g., Feb expansion was \$8,384 vs contraction of \$2,324).
  • The Reversal: In June, following the price hike and seat audit tool, Mid-market contraction (\$22,351) completely crushed expansion (\$8,565). We hemorrhaged net expansion capability in our highest-value tier.
  • The Net Result: While Mid-market paying accounts continue to creep up slowly (435 in April $\rightarrow$ 460 in September), the revenue per account is being aggressively optimized downward by our own customers using the admin seat audit tool to purge dormant seats they were previously paying for.

---

4. Strategic Recommendations for Next Quarter

Maya, your instinct to mobilize two squads around AI is understandable given market noise, but building an AI assistant will not solve a pricing-induced contraction problem or an academic seasonality cycle.

Instead, I recommend the following adjustment to next quarter’s roadmap:

1. Scrap the Notion AI panic response. Do not put two squads on an AI assistant. Notion’s bundle play works for them because of their architecture, but copying them reactively without product-market validation in our core workspace loops is a trap. 2. Deploy Squad 1 to Mid-Market Value Realization & Packaging. * The \$12 $\rightarrow$ \$15 price jump triggered a violent contraction because enterprise buyers immediately audited and purged unproven seats. * Instead of raw seat-pricing friction, we need tiered packaging (e.g., introducing advanced security, admin controls, or compliance features at the \$15 tier) rather than a naked price hike on the base tier that incentivizes seat-cutting. 3. Deploy Squad 2 to Education Onboarding & Workspace Activation. * Education is entering its post-August renewal trough. Squad 2 should focus on automated workspace templates and cross-semester continuity features that reduce the friction of the annual academic reset, preventing schools from hitting the "pause" button every summer.

We do not have a top-of-funnel retention collapse. We have a self-inflicted Mid-market pricing shock. Let’s fix the bucket before we build new ways to pour water into it.

Claims the judge couldn’t find in the brief

  • Every single Mid-market cancellation in the exit log cites 'Too expensive per seat after the price change' (combined with tool consolidation).
  • Education accounts show a predictable annual churn spike in August (churned MRR jumped from $935 in July to $3,925 in August), entirely driven by the end of the academic cycle.
  • The revenue per account is being aggressively optimized downward by our own customers using the admin seat audit tool to purge dormant seats they were previously paying for.

Grades and run details

Decision model 65 · LLM judge 71
Decision model checks
  • failUses the supplied evidence correctly27%
  • passAddresses the actual decision43%
  • passRespects explicit constraints37%
  • failIdentifies material uncertainty85%
  • failAvoids unsupported claims33%
  • passProduces the required deliverable45%
  • partialSeparates what isn't churn25%
  • passFinds and sizes the real loss93%
  • passAnswers the AI plan with evidence99%
  • passUnderstands before it justifies99%
  • failGets the base of every number right33%
  • passSplits growth into new, lost and returning86%
  • passReads the shape of the curves61%
Run
Run
#1
Time to output
7 s
Submitted
5 Oct 2026
Haiku 5.5API

Usable with a quick edit12 of 13 checks passed

The memo correctly diagnoses the problems using the available data, gives a clear recommendation, and is structured for immediate use with only light polish.

Output

To: Maya Lind, CEO From: Staff PM Re: Retention diagnosis and where we're losing revenue

Bottom line

The March drop is mostly a metric change, not customers leaving. Paying accounts rose every month. The real leaks are mid-market seat contraction after the May price change and seat audit, a steady SMB churn problem, and an August education renewal spike. Notion AI doesn't show up in the numbers. I don't think two squads on an AI assistant next quarter is the right bet. I'd fund one squad on SMB activation and fix the mid-market problem now.

1. What isn't happening

The March "collapse" is a definition change. On 2 March, "active account" changed from "logged in within 30 days" to "created or edited a page within 30 days." The dashboard shows the step in March and doesn't restate January and February. Paying accounts rose from 3,208 in February to 3,246 in March and reached 3,474 in September. Active accounts as a share of paying accounts were 84% in February. They were 67% in March and have stayed at about 67% since. The series is flat after the change, so it hasn't "failed to recover." It rebased.

Education isn't in freefall. Active education accounts dip in June–August (293, 201, 296) and recover to 461 in September, above March's 428. Paying education accounts grew from 625 to 669, and education MRR grew from $64k to $72k. We only have 2026 data, so I can't confirm the summer dip is seasonal, but the September recovery is clear.

Notion AI isn't visibly eating us. The AI assistant launched on 3 August, five months after the March step, so it can't explain March. Of 79 exit-survey responses, 5 cite Notion (4 SMB, 1 mid-market). Mid-market churned MRR was $2.7k in both August and September, in line with January–July. SMB churned MRR was $7.5k in August and September, also in line. The survey has no dates, so I can't rule out a Notion effect in a given month, but nothing in the MRR series changes after 3 August.

2. What is happening

A. Mid-market: the price change is costing more in seats than it's adding. - On 15 May the plan went from $12 to $15 per seat (+25%). The seat audit launched the same day, with a one-click "remove seat" action. - Logo churn hasn't moved. Churned MRR is $2.6–2.8k a month before and after. Eleven of 15 mid-market exits cite price. - Contraction jumped from about $2.4k a month to $10–22k: $10.2k in May, $22.4k in June, $16.5k in July, $12.5k in August, $13.2k in September. That's about $63k of excess contraction since May. - May expansion rose by about $23k, which is probably the price uplift. That's well short of the roughly $75k a full 25% increase on a ~$309k base would produce. Billing needs to confirm how the increase was applied. - Mid-market MRR peaked at $335k in May and has been flat at about $322k since July. Before the changes it was growing about $10k a month. Against that trend, it's about $37k MRR behind, roughly $0.45M ARR. - The export can't separate seat-audit removals from price-driven downgrades. We need to pull remove-seat events and plan changes separately.

B. SMB: a chronic leak, not a new one. - SMB churned MRR has been flat at $7.1–7.7k a month all year. In September it was 3.1% of starting MRR, or about $90k of ARR a year. - Of 44 SMB exits: 17 cite "didn't get the team to adopt it," 14 say the company closed or downsized, 9 say too expensive, and 4 switched to Notion. - Paying-but-inactive SMB accounts grew from 757 in March to 821 in September. That's the pool adoption work would target. - SMB is still growing: MRR is $246k, up 15% since January, because new MRR of $9.5–9.9k a month exceeds churn and contraction.

C. Education: one renewal cliff in August. - Churned education MRR was $3.9k in August against a typical $0.9k. That's about $3k of excess, roughly $36k ARR, in a single month. - Of 20 education exits: 12 say "end of academic year, will review," 6 cite budget cuts, and 2 switched to Google Workspace. - The 12 "will review" accounts are the live risk. Their decision is still open.

3. Size of the problem

IssueRevenue impactStatus
Mid-market gap to trend~$37k MRR (~$0.45M ARR)Ongoing since May; flat since July
SMB churn~$7.5k MRR/month (~$90k ARR/year)Chronic, not rising
Education August renewals~$3k excess MRR (~$36k ARR)Renewal-timed; 12 accounts still open
Notion AINot visible in MRR5 of 79 exits
March "active" drop$0Metric change

Total MRR is up 18% since January, to $640k (about $7.7M ARR). The mid-market gap is an estimate, based on extrapolating pre-May growth, so treat it as a range rather than a precise figure.

4. What to do

  1. Restate the active-account series before it goes anywhere else. Recompute January and February on the new definition, or show both. Drop the "retention collapsed in March" claim. (Data team, two weeks.)
  1. Don't fund two AI-assistant squads yet. Fund one squad on SMB activation. Adoption is the largest controllable SMB exit reason, and the inactive-paying pool is growing. If the AI assistant can help new teams reach their first shared page, test it inside that squad's scope for six weeks against an activation metric. Fund a second squad only if that test moves activation.
  1. Run a 30-day review of the mid-market price change and seat audit. Split contraction into seat removals and plan downgrades. Check whether the audit is removing seats customers still use. Options include making the audit recommend rather than remove, offering right-sizing at renewal, and protecting accounts that contract. Judge the price change on net MRR, not the headline uplift. Ask billing why the May step was about $23k rather than about $75k.
  1. Work the education accounts now. Contact the 12 "will review" and 6 budget-cut accounts this month with a renewal offer, such as term or seat flexibility. Set up a renewal review about 60 days before each August cohort, so we're not finding out in the renewal window.
  1. Fix the reporting. We need monthly logo churn by segment and renewal cohort, contraction reasons, and an exit survey with dates and a response rate. Right now 79 exit responses can't be matched to MRR movements, and the Notion share can't be measured by month.

Grades and run details

Decision model 92 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly25%
  • passAddresses the actual decision94%
  • passRespects explicit constraints58%
  • passIdentifies material uncertainty97%
  • passAvoids unsupported claims30%
  • passProduces the required deliverable86%
  • passSeparates what isn't churn30%
  • passFinds and sizes the real loss100%
  • passAnswers the AI plan with evidence100%
  • passUnderstands before it justifies100%
  • failGets the base of every number right18%
  • passSplits growth into new, lost and returning94%
  • passReads the shape of the curves100%
Run
Run
#1
API response time
65 s
Submitted
8 Oct 2026
GPT-6 LunaAPI

Usable with a quick edit12 of 13 checks passed

The output is clear, evidence-based, and actionable; a PM could use it with minor wording tweaks without rechecking data or reasoning.

Output

To: Maya Lind Subject: March retention diagnosis and where revenue is actually at risk

Bottom line: The March dashboard cliff is real as a reported activity change, but it is not evidence that customer retention collapsed. The metric changed on March 2, and paying accounts and MRR continued to grow. There are real, more specific risks: SMB adoption, mid-market seat contraction after pricing and seat-audit changes, and education’s seasonal renewal cycle. The data does not support assigning two squads to an AI assistant on the premise that Notion AI caused a broad retention collapse.

What is—and isn’t—happening

Active accounts fell from 2,697 in February to 2,176 in March, down 19.3%. But on March 2, “active” changed from anyone logged in within 30 days to anyone who created or edited a page within 30 days. That stricter definition plausibly explains the abrupt drop: activity fell by roughly 18–20% in all three segments at once.

The more comparable evidence is what happened after the change. From March to September, active accounts rose from 2,176 to 2,320 (+6.6%): SMB rose 6%, mid-market 7%, and education 8%. SMB and mid-market activity rates were broadly stable over that period. These figures do not show an ongoing collapse—though we cannot make a clean February-to-March retention comparison without recalculating historical activity on one definition.

Education’s summer decline is pronounced: active accounts fell from 428 in March to 201 in July, then recovered to 461 in September. That pattern fits the academic calendar and renewal timing better than a continuous decline. Paying education accounts rose every month, from 625 in March to 669 in September. The activity metric is not the same as paid-account retention, so this is not proof that every education account is secure; it is evidence against calling education “in freefall.”

Revenue also contradicts the broad-collapse diagnosis. Ending MRR rose from $558.6K in January to $640.3K in September, up $81.7K (14.6%). Monthly churned MRR was generally about $10.6K–$11.4K from January through July, rose to $14.1K in August, then returned to $11.1K in September. There is no sustained March deterioration in these figures.

Where the real risks are

  • Mid-market pricing and seat contraction: The mid-market plan price rose 25%, from $12 to $15 per seat, and a one-click seat-removal audit launched in May. Mid-market contraction MRR then reached $10.2K in May, $22.4K in June, $16.5K in July, and $12.5K in August, versus roughly $2.4K per month in the surrounding months. That is about $52K more contraction MRR over May–August than the surrounding run rate would suggest. It is a meaningful warning, but not $52K of confirmed churn: contraction may reflect customers removing idle seats, and the report does not isolate price effects from seat cleanup. Mid-market ending MRR was still $12.8K above April’s by September, after the price change and other movements.
  • SMB adoption: In the exit survey, 17 of 44 SMB respondents cited failure to get their team to adopt Tandem—the largest single SMB reason. Another 14 cited company closure or downsizing, nine cited price, and four said they switched to Notion. This points to an adoption problem as well as some external business distress.
  • Education renewals: Twelve of 20 education respondents said they would review at the end of the academic year; six cited budget cuts. Education churned MRR spiked to $3.9K in August, versus roughly $0.9K in most other months, then returned to $0.9K in September. Prepare for the renewal window rather than treating the summer activity trough as evidence of a year-round collapse.

The exit survey is a sample of 79 cancellations, not a representative churn-rate study. Only five respondents said they switched to Notion (four SMB, one mid-market), and the survey does not say they left because of AI. Notion’s August AI launch is too recent to explain the March dashboard break, and these data do not establish that it is driving the current losses.

Recommended actions

  1. Fix the measurement first. Backfill the new activity definition, annotate the dashboard break, and report paid-account retention, logo churn, revenue churn, and usage separately by segment and renewal cohort.
  2. Investigate mid-market changes account by account. Separate price increases, seat removals, and cancellations; quantify retained seats and renewal outcomes. Test targeted pricing or seat flexibility where value is at risk before considering a broad rollback.
  3. Improve SMB activation. Focus on onboarding, team rollout, and early adoption signals; distinguish preventable adoption failures from business closures.
  4. Run an education renewal plan ahead of the academic-year cycle, with early outreach and budget-sensitive renewal options.
  5. Do not staff two AI squads on this diagnosis. If AI is strategically important, run a bounded customer discovery and prototype test, with evidence of demand and retention impact as a gate for further investment.

Grades and run details

Decision model 92 · LLM judge 100
Decision model checks
  • passUses the supplied evidence correctly12%
  • passAddresses the actual decision79%
  • passRespects explicit constraints56%
  • passIdentifies material uncertainty78%
  • passAvoids unsupported claims43%
  • passProduces the required deliverable69%
  • passSeparates what isn't churn99%
  • passFinds and sizes the real loss72%
  • passAnswers the AI plan with evidence100%
  • passUnderstands before it justifies100%
  • failGets the base of every number right17%
  • passSplits growth into new, lost and returning55%
  • passReads the shape of the curves99%
Run
Run
#1
API response time
34 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 3

Uses the supplied evidence correctlyWrongRightRight
Gemini 3.5 Flash-Lite · Gemini

Two claims are factually wrong: 'Every single Mid-market cancellation' cites price (only 11 of 15 do), and education churn is 'entirely driven by the end of the academic cycle' (budget cuts and Google Workspace switches also appear).

Haiku 5.5 · API

All factual claims are drawn directly from the supplied data or derived by correct arithmetic, with no invented facts about the current situation.

GPT-6 Luna · API

All factual claims about the current situation are directly supported by the supplied data or arithmetic.

Identifies material uncertaintyWrongRightRight
Gemini 3.5 Flash-Lite · Gemini

The memo does not name specific unknowns that could change the decision, nor does it say what would change the call or how to resolve them.

Haiku 5.5 · API

Identifies key unknowns (summer seasonality, Notion month effect, billing application, seat-audit vs downgrade split) and says how to resolve them or what condition changes the recommendation.

GPT-6 Luna · API

It names specific unknowns (definition change prevents clean comparison, contraction may be seat cleanup not churn, exit survey is a sample) and says how to resolve them.

Avoids unsupported claimsWrongRightRight
Gemini 3.5 Flash-Lite · Gemini

It presents 'Every single Mid-market cancellation' and 'entirely driven by the end of the academic cycle' as established facts, and states the seat audit tool is the cause of contraction without labeling it as a hypothesis.

Haiku 5.5 · API

Causal interpretations are hedged ('probably', 'not visibly'), estimates are labelled, and conclusions stick to what the evidence directly supports.

GPT-6 Luna · API

Interpretations are hedged ('plausibly', 'points to', 'may reflect') and not presented as established fact.

All mixed 1

Gets the base of every number rightMixedMixedMixed
Gemini 3.5 Flash-Lite · Gemini

Derived figures like the 309% jump and the $6,000/month expansion-contraction gap are computed correctly from the supplied data.

Haiku 5.5 · API

Derived percentages (84%/67% active, 3.1% SMB churn, 18% MRR growth) are computed correctly, and bases are stated or clear from context.

GPT-6 Luna · API

All percentages and differences are computed correctly from the supplied data, with bases stated or clear.

All got right 9

Addresses the actual decisionRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The memo commits early and unambiguously that the March collapse is incorrect and Notion AI is not the culprit, and recommends specific squad deployments instead of the AI plan.

Haiku 5.5 · API

The memo commits early to a clear call (one squad on SMB activation, fix mid-market, not two AI squads) and states what would change it (a successful activation test).

GPT-6 Luna · API

The memo unambiguously advises against staffing two AI squads on this diagnosis and states what evidence would change that call.

Respects explicit constraintsRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The output is a memo addressed to the CEO, under 1,200 words, and respects the requested form and length.

Haiku 5.5 · API

The memo is a properly formatted note to the CEO, stays well under 1,200 words, and addresses the required topics.

GPT-6 Luna · API

The output is a memo to Maya, well under 1,200 words, and respects the requested form and reader.

Produces the required deliverableRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The memo is complete, in the correct form for the CEO, within the word limit, and actionable with clear recommendations.

Haiku 5.5 · API

Produces a complete, actionable memo for Maya that diagnoses retention, sizes the revenue impact, and lays out specific next steps.

GPT-6 Luna · API

The memo is complete, in the right form, within length, and a PM could act on it with light edits.

Separates what isn't churnRightRightRight
Gemini 3.5 Flash-Lite · Gemini

It clearly links the March drop to the definition change on 2 March and shows education's summer dip recovering in September, using the dashboard change log and active account data.

Haiku 5.5 · API

Clearly links the March drop to the 2 March active-account definition change and shows education's summer dip recovering in September.

GPT-6 Luna · API

It links the March drop to the 2 March definition change and shows education's summer dip recovering in September.

Finds and sizes the real lossRightRightRight
Gemini 3.5 Flash-Lite · Gemini

It identifies the mid-market contraction spike after 15 May, links it to the price rise and seat audit, and sizes it with monthly contraction MRR figures.

Haiku 5.5 · API

Identifies the mid-market contraction spike after the 15 May price rise and seat audit, quantifies it in MRR, and contrasts it with flat logo churn.

GPT-6 Luna · API

It identifies the mid-market contraction jump after 15 May, links it to the price rise and seat audit, and sizes it at ~$52K excess contraction MRR.

Answers the AI plan with evidenceRightRightRight
Gemini 3.5 Flash-Lite · Gemini

It uses the exit data (few Notion switchers) and the August launch timeline to argue against the AI assistant as a retention fix, and advises against two squads on it.

Haiku 5.5 · API

Uses exit data (5 of 79 mention Notion) and the August launch date to show Notion AI doesn't explain the March 'collapse', and advises against two AI squads as a retention fix.

GPT-6 Luna · API

It uses exit data (only 5 Notion switchers, no AI mention) and the August launch timeline to advise against the AI assistant as a retention fix.

Understands before it justifiesRightRightRight
Gemini 3.5 Flash-Lite · Gemini

The memo tests the CEO's hypothesis against the data, rejects the March collapse narrative, and reaches its own diagnosis from the evidence.

Haiku 5.5 · API

Tests the CEO's collapse claim against paying accounts and the definition change before offering explanations, then builds its own diagnosis from the data.

GPT-6 Luna · API

The diagnosis starts from the data, tests the CEO's Notion AI explanation, and reaches its own evidence-based conclusions.

Splits growth into new, lost and returningRightRightRight
Gemini 3.5 Flash-Lite · Gemini

It uses MRR components (new, expansion, contraction, churn) to separate contraction from churn and show which flow changed.

Haiku 5.5 · API

Breaks MRR changes into new, expansion, contraction and churn across segments to isolate which flow changed.

GPT-6 Luna · API

It uses MRR movements (new, expansion, contraction, churn) to show which flow changed, particularly the contraction spike.

Reads the shape of the curvesRightRightRight
Gemini 3.5 Flash-Lite · Gemini

It reads the education dip and September recovery as seasonal, and the mid-market contraction as a spike after May, drawing correct conclusions from the curve shapes.

Haiku 5.5 · API

Describes the active-account series as a rebased flat line and the education curve as a dip that recovers, drawing the correct conclusions (definition change, seasonality).

GPT-6 Luna · API

It reads the education dip and recovery as seasonal and notes the post-March active account rise, not a continued decline.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6 AstraChatGPT96.2100.02None
2GPT-6 LunaAPI96.2100.02None
3GPT-6.1 SolAPI94.1100.02None
4Haiku 5.5API92.0100.02None
5Opus 5.5Claude82.1100.02None
6Sonnet 5.5API90.189.32None
7Gemini 3.5 Flash-LiteGemini76.474.22None
8Gemini 3.8 FlashAPI72.370.92None

What you get for the cost

GPT-6 Luna · API matches GPT-6 Astra · ChatGPT at about a fiftieth of the cost.

Task score against relative cost for each setup. The table below has the same figures.607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6 Astra · ChatGPT: task score 98, 79× the cheapestGPT-6 Luna · API: task score 98, the cheapestGPT-6.1 Sol · API: task score 97, 15× the cheapestHaiku 5.5 · API: task score 96, 3.6× the cheapestOpus 5.5 · Claude: task score 92, 77× the cheapestSonnet 5.5 · API: task score 88, 50× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 74, 3.0× the cheapestGemini 3.8 Flash · API: task score 71, 13× the cheapestGPT-6 Astra · ChatGPTGPT-6 Luna · APIGPT-6.1 Sol · APIHaiku 5.5 · APIOpus 5.5 · ClaudeSonnet 5.5 · APIGemini 3.5 Flash-Lite · GeminiGemini 3.8 Flash · API
Best value

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

The PM job

Working out why customers leave, and what to fix first.

Why it matters

Teams pick a story (the season, a competitor) and then pull data to justify it. Half the time the 'drop' is a definition change, and the real leak is somewhere else.

What good looks like

  • Starts from what the data shows, not a story
  • Splits growth into new, lost and returning users
  • Separates measurement and seasonality from churn
  • Sizes the real problem and fixes new-user retention first

Deliberately not measured

  • Dashboard design
Capability tested

Retention diagnosis

The failure we’re looking for

Confirming the story everyone already believes

Grading

Decision model and LLM judge, calibrated against a blind PM review