Tasks / Strategy

Competitor analysis

Can the model find where a competitor can actually hurt you, from win/loss and churn data, rather than list features?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 56% were usable with at most a quick edit.

Reliably right

  1. Respects explicit constraints100% pass
    Delivers a competitive analysis memo to Dan within 900 words, addressing all requested topics without proposing a full feature-by-feature match.
    Opus 5.5 · Claude · The competitor isn't who you think
  2. Reads where Crestline is heading100% pass
    It reads Crestline's job ads as signalling payroll, mid-market, and possible accountant partnerships, and adjusts the response accordingly.
    GPT-6.1 Sol · API · Free forever, funded
  3. Produces the required deliverable97% pass
    The memo is complete, in the right form for the CEO, within the word limit, and actionable with light edits.
    Opus 5.5 · Claude · The competitor isn't who you think

Where it slips

  1. Uses the supplied evidence correctly56% pass
    Two factual claims are wrong: the number of unconnected multi-user accounts going to Crestline is 40, not 42, and the number of connected 6-20 cancellations is 5, not 4.
    Sonnet 5.5 · API · Free forever, funded
  2. Gets the base of every number right63% pass
    Derived figures for unconnected multi-user cancellations (42 vs actual 40) and connected 6-20 cancellations (4 vs actual 5) are incorrect.
    Sonnet 5.5 · API · Free forever, funded
  3. Separates what a rival can copy68% pass
    Does not explicitly separate advantages a well-funded competitor could copy from those it couldn't, nor anchor the response in durable advantages with evidence.
    GPT-6 Luna · API · The competitor isn't who you think

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Competes with what customers really choose

    Does the analysis treat the status quo (spreadsheets, manual work, doing nothing) as a competitor, and draw at least one implication from where competitors are investing rather than only from today's features?

    Passes when Analyses the status quo, strengths included, and gives a direction-of-travel insight that affects the recommendation.

  2. Separates what a rival can copy

    Does the analysis separate advantages a well-funded competitor could copy (features, price, being first) from ones it couldn't easily (switching costs, networks, relationships), and build the response on the latter?

    Passes when Names which advantages are copyable and which aren't, with evidence, and anchors the response in the durable ones.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the VP Product at Harbourline. Crestline raised $120M in August and launched a free plan for solo businesses. Our board chair wants to know whether we should launch a free plan for everyone to match it (the note is below). Write the board a competitive analysis of no more than 1,200 words: where Crestline can hurt us and where it can't, with the evidence; which of our advantages it could copy and which it couldn't; where it's heading; and what we should do. The files are attached.

What the model was given6 items: About Harbourline, board_chair_note.md, customer_base.md, cancellations_aug_sep.csv (every account that cancelled since Crestline's launch), cancellation_summary.md, crestline_public_info.md
About HarbourlineBookkeeping, invoicing and expenses for small businesses, with a partner network that connects customers to their accounting firm.
board_chair_note.md1 lines · Download
From the board chair: "Crestline's free tier is an existential threat. Should we launch a free plan for everyone to match it? We need a view on where they can hurt us and where they can't."
customer_base.md9 lines · Download
# Harbourline customers (end of September)

| Account size | Paying accounts | Share connected to an accounting firm | Average revenue per month | Annual churn before Crestline's free tier |
|---|---|---|---|---|
| 1 user | 11,400 | 9% | $14 | 31% |
| 2-5 users | 4,900 | 46% | $52 | 14% |
| 6-20 users | 1,300 | 81% | $170 | 6% |
…
cancellations_aug_sep.csv (every account that cancelled since Crestline's launch)cancelled_account,account_users,connected_to_accounting_firm,mrr_usd,went_to H-1200,1,no,14,Crestline (free tier) H-1201,1,no,10,Crestline (free tier) H-1202,2-5,no,58,Crestline (free tier) H-1203,1,no,11,Crestline (free tier) H-1204,1,no,17,Crestline (free tier) H-1205,1,no,15,Spreadsheets / no tool H-1206,1,no,15,Spreadsheets / no tool H-1207,1,no,9,Crestline (free tier) H-1208,2-5,no,66,Crestline (free tier) H-1209,2-5,no,44,Crestline (free tier) H-1210,1,no,11,Crestline (free tier) H-1211,1,no,9,Crestline (free tier) H-1212,1,no,12,Crestline (free tier) H-1213,1,no,16,Crestline (free tier) H-1214,2-5,yes,73,Crestline (free tier) H-1215,1,no,9,Crestline (free tier) H-1216,2-5,no,56,Crestline (free tier) H-1217,1,no,14,Crestline (free tier) H-1218,2-5,no,47,Business closed H-1219,1,no,16,Crestline (free tier) H-1220,6-20,no,177,Crestline (free tier) H-1221,2-5,yes,51,Crestline (free tier) H-1222,2-5,no,54,Business closed H-1223,6-20,no,168,Crestline (free tier) H-1224,1,no,18,Crestline (free tier) H-1225,1,no,13,Spreadsheets / no tool H-1226,1,no,11,Crestline (free tier) H-1227,1,no,15,Crestline (free tier) H-1228,1,no,12,Crestline (free tier) H-1229,1,no,13,Crestline (free tier) H-1230,2-5,no,51,Business closed H-1231,1,no,18,Crestline (free tier) H-1232,1,no,16,Spreadsheets / no tool H-1233,1,no,12,Crestline (free tier) H-1234,1,no,16,Spreadsheets / no tool H-1235,1,no,12,Spreadsheets / no tool H-1236,1,no,13,Crestline (free tier) H-1237,1,no,17,Spreadsheets / no tool H-1238,6-20,yes,188,Business closed H-1239,1,no,17,Crestline (free tier) H-1240,1,no,15,Crestline (free tier) H-1241,6-20,yes,238,Crestline (free tier) H-1242,1,no,11,Spreadsheets / no tool H-1243,2-5,no,66,Crestline (free tier) H-1244,2-5,no,63,Business closed H-1245,6-20,no,199,Crestline (free tier) H-1246,1,no,18,Crestline (free tier) H-1247,1,no,11,Crestline (free tier) H-1248,1,no,15,Crestline (free tier) H-1249,2-5,no,60,Crestline (free tier) H-1250,2-5,no,66,Crestline (free tier) H-1251,2-5,no,34,Crestline (free tier) H-1252,1,no,12,Crestline (free tier) H-1253,1,no,18,Crestline (free tier) H-1254,1,no,18,Spreadsheets / no tool H-1255,2-5,no,31,Crestline (free tier) H-1256,1,no,17,Crestline (free tier) H-1257,1,no,16,Crestline (free tier) H-1258,1,no,18,Spreadsheets / no tool H-1259,1,no,9,Crestline (free tier) H-1260,1,no,15,Crestline (free tier) H-1261,1,no,18,Crestline (free tier) H-1262,6-20,yes,170,Business closed H-1263,1,no,18,Crestline (free tier) H-1264,2-5,no,41,Business closed H-1265,1,no,14,Spreadsheets / no tool H-1266,1,no,14,Crestline (free tier) H-1267,1,no,12,Crestline (free tier) H-1268,6-20,no,112,Crestline (free tier) H-1269,1,no,9,Crestline (free tier) H-1270,2-5,no,71,Business closed H-1271,1,no,12,Crestline (free tier) H-1272,1,no,11,Crestline (free tier) H-1273,1,no,15,Crestline (free tier) H-1274,1,no,9,Crestline (free tier) H-1275,6-20,no,209,Crestline (free tier) H-1276,2-5,no,36,Crestline (free tier) H-1277,1,no,13,Crestline (free tier) H-1278,1,no,11,Crestline (free tier) H-1279,1,no,15,Crestline (free tier) H-1280,1,no,18,Spreadsheets / no tool H-1281,1,no,16,Crestline (free tier) H-1282,1,no,17,Crestline (free tier) H-1283,1,no,9,Crestline (free tier) H-1284,1,no,10,Crestline (free tier) H-1285,6-20,yes,128,Business closed H-1286,1,no,17,Crestline (free tier) H-1287,1,no,12,Spreadsheets / no tool H-1288,2-5,no,39,Crestline (free tier) H-1289,1,no,14,Crestline (free tier) H-1290,1,no,12,Crestline (free tier) H-1291,1,no,16,Crestline (free tier) H-1292,2-5,no,62,Crestline (free tier) H-1293,1,no,12,Crestline (free tier) H-1294,1,no,12,Crestline (free tier) H-1295,1,no,9,Spreadsheets / no tool H-1296,1,no,11,Crestline (free tier) H-1297,1,no,16,Crestline (free tier) H-1298,1,no,17,Crestline (free tier) H-1299,1,no,13,Spreadsheets / no tool H-1300,1,no,13,Crestline (free tier) H-1301,1,no,17,Crestline (free tier) H-1302,1,no,12,Crestline (free tier) H-1303,1,no,17,Crestline (free tier) H-1304,1,no,14,Crestline (free tier) H-1305,1,no,9,Crestline (free tier) H-1306,1,no,15,Crestline (free tier) H-1307,1,no,16,Crestline (free tier) H-1308,1,no,16,Crestline (free tier) H-1309,1,no,12,Crestline (free tier) H-1310,2-5,no,47,Business closed H-1311,2-5,no,78,Crestline (free tier) H-1312,1,no,11,Crestline (free tier) H-1313,1,no,14,Crestline (free tier) H-1314,1,no,15,Spreadsheets / no tool H-1315,1,no,14,Spreadsheets / no tool H-1316,1,no,9,Crestline (free tier) H-1317,2-5,no,52,Business closed H-1318,1,no,13,Crestline (free tier) H-1319,2-5,no,43,Business closed H-1320,1,no,11,Spreadsheets / no tool H-1321,1,no,15,Crestline (free tier) H-1322,2-5,no,55,Business closed H-1323,1,no,12,Spreadsheets / no tool H-1324,6-20,no,166,Crestline (free tier) H-1325,2-5,no,67,Business closed H-1326,1,no,16,Crestline (free tier) H-1327,1,no,15,Crestline (free tier) H-1328,1,no,13,Crestline (free tier) H-1329,1,no,17,Crestline (free tier) H-1330,1,no,12,Crestline (free tier) H-1331,1,no,13,Crestline (free tier) H-1332,2-5,no,68,Crestline (free tier) H-1333,1,no,14,Spreadsheets / no tool H-1334,1,no,18,Crestline (free tier) H-1335,1,no,9,Crestline (free tier) H-1336,1,no,14,Crestline (free tier) H-1337,1,no,13,Crestline (free tier) H-1338,1,no,10,Crestline (free tier) H-1339,1,no,18,Spreadsheets / no tool H-1340,2-5,no,70,Crestline (free tier) H-1341,1,no,13,Spreadsheets / no tool H-1342,2-5,no,34,Crestline (free tier) H-1343,1,no,11,Crestline (free tier) H-1344,1,no,17,Spreadsheets / no tool H-1345,6-20,no,159,Crestline (free tier) H-1346,6-20,no,214,Crestline (free tier) H-1347,2-5,no,67,Crestline (free tier) H-1348,2-5,no,40,Crestline (free tier) H-1349,1,no,18,Crestline (free tier) H-1350,1,no,17,Crestline (free tier) H-1351,1,no,11,Crestline (free tier) H-1352,1,no,17,Crestline (free tier) H-1353,2-5,no,48,Business closed H-1354,1,no,12,Crestline (free tier) H-1355,1,no,16,Crestline (free tier) H-1356,1,no,14,Crestline (free tier) H-1357,1,no,18,Crestline (free tier) H-1358,1,no,11,Crestline (free tier) H-1359,1,no,17,Spreadsheets / no tool H-1360,1,no,12,Spreadsheets / no tool H-1361,2-5,no,39,Business closed H-1362,1,no,18,Crestline (free tier) H-1363,2-5,no,39,Crestline (free tier) H-1364,2-5,no,41,Business closed H-1365,2-5,no,51,Crestline (free tier) H-1366,1,no,11,Crestline (free tier) H-1367,2-5,no,76,Business closed H-1368,1,no,11,Crestline (free tier) H-1369,1,no,18,Crestline (free tier) H-1370,2-5,no,37,Business closed H-1371,1,no,9,Crestline (free tier) H-1372,1,no,18,Crestline (free tier) H-1373,2-5,no,36,Crestline (free tier) H-1374,2-5,no,77,Crestline (free tier) H-1375,2-5,no,64,Crestline (free tier) H-1376,1,no,9,Crestline (free tier) H-1377,2-5,no,71,Crestline (free tier) H-1378,1,no,10,Crestline (free tier) H-1379,1,no,12,Spreadsheets / no tool H-1380,6-20,yes,235,Business closed H-1381,1,no,9,Crestline (free tier) H-1382,1,no,15,Spreadsheets / no tool H-1383,2-5,yes,65,Crestline (free tier) H-1384,1,no,16,Crestline (free tier) H-1385,1,no,17,Spreadsheets / no tool H-1386,1,no,13,Spreadsheets / no tool H-1387,2-5,no,66,Business closed H-1388,1,no,17,Crestline (free tier) H-1389,1,no,18,Crestline (free tier) H-1390,1,no,16,Crestline (free tier) H-1391,1,no,18,Spreadsheets / no tool H-1392,1,no,15,Crestline (free tier) H-1393,1,no,10,Crestline (free tier) H-1394,1,no,11,Crestline (free tier) H-1395,2-5,no,46,Business closed H-1396,1,no,16,Crestline (free tier) H-1397,1,no,15,Crestline (free tier) H-1398,2-5,no,34,Crestline (free tier) H-1399,1,no,13,Crestline (free tier) H-1400,1,no,13,Crestline (free tier) H-1401,1,no,10,Crestline (free tier) H-1402,1,no,11,Crestline (free tier) H-1403,2-5,no,49,Crestline (free tier) H-1404,2-5,no,58,Crestline (free tier) H-1405,1,no,17,Crestline (free tier) H-1406,1,no,17,Crestline (free tier) H-1407,1,no,12,Crestline (free tier) H-1408,2-5,no,37,Crestline (free tier) H-1409,1,no,15,Spreadsheets / no tool H-1410,2-5,no,31,Crestline (free tier) H-1411,1,no,13,Spreadsheets / no tool H-1412,1,no,15,Crestline (free tier) H-1413,1,no,13,Crestline (free tier) H-1414,1,no,15,Crestline (free tier) H-1415,1,no,18,Crestline (free tier) H-1416,1,no,14,Crestline (free tier) H-1417,1,no,11,Crestline (free tier) H-1418,6-20,no,131,Crestline (free tier) H-1419,1,no,17,Crestline (free tier) H-1420,1,no,18,Crestline (free tier) H-1421,1,no,16,Crestline (free tier) H-1422,1,no,11,Crestline (free tier) H-1423,2-5,no,78,Business closed H-1424,1,no,16,Spreadsheets / no tool H-1425,1,no,18,Spreadsheets / no tool H-1426,1,no,18,Crestline (free tier) H-1427,1,no,15,Crestline (free tier) H-1428,1,no,14,Spreadsheets / no tool H-1429,1,no,14,Spreadsheets / no tool H-1430,2-5,no,63,Business closed H-1431,1,no,18,Crestline (free tier) H-1432,1,no,15,Crestline (free tier) H-1433,1,no,17,Spreadsheets / no tool H-1434,1,no,14,Crestline (free tier) H-1435,1,no,14,Crestline (free tier) H-1436,1,no,12,Crestline (free tier) H-1437,1,no,11,Crestline (free tier) H-1438,1,no,13,Crestline (free tier) H-1439,1,no,13,Spreadsheets / no tool H-1440,2-5,no,54,Business closed H-1441,1,no,14,Crestline (free tier) H-1442,1,no,16,Crestline (free tier) H-1443,2-5,no,74,Crestline (free tier) H-1444,1,no,14,Crestline (free tier) H-1445,2-5,no,65,Crestline (free tier) H-1446,1,no,18,Crestline (free tier) H-1447,1,no,11,Crestline (free tier) H-1448,1,no,10,Crestline (free tier) H-1449,2-5,no,69,Business closed
cancellation_summary.md1 lines · Download
# Cancellations, August to September: 250 accounts, about $8,100 of MRR. Before Crestline's launch we lost about 95 accounts a month.
crestline_public_info.md6 lines · Download
# Crestline (news and public pages)

- 1 August: raised $120M. 15 August: launched a free plan for 1 user (invoicing, expenses, bank feeds).
- Paid plans from $25/month; no accounting-firm partner programme.
- Open roles (September): 14 engineers for payroll; 6 for "mid-market multi-entity"; 3 for "accountant partnerships (exploring)".
- Pricing page: free plan "free forever for solo businesses".
What a strong answer doesThe answer key the graders mark against

Shows from the cancellations that Crestline is taking single-user accounts not connected to an accounting firm: 186 of the 250 cancellations went to Crestline, 142 of them single-user accounts at $9 to $19 a month, and only 4 of the 186 were connected to an accounting firm; the $5,600 of MRR lost to Crestline is a small share of a base where 1-user accounts already churned at 31% a year. The partner network is the advantage Crestline can't easily copy: 81% of 6-20 user accounts and 46% of 2-5 user accounts are connected to a firm, and they barely leave; Crestline has no partner programme. Free and price are easy to copy, and so is matching features. Reads Crestline's hiring (payroll, mid-market multi-entity, accountant partnerships 'exploring') as heading upmarket and possibly at the partner channel, the real future threat. Recommends against a free plan for everyone: it would give away revenue from 11,400 single-user accounts paying about $14 a month (about $160,000 MRR) to defend against losses that are a fraction of that. Instead: let the least valuable solo accounts go or offer a narrow solo option tied to an accounting firm, deepen the partner network (bring firms' clients onto Harbourline, lock in the firm workflow), and get ahead on payroll and multi-entity where Crestline is heading. Sets the signals that would change the call (losses spreading to connected or multi-user accounts, or Crestline launching a partner programme).

Critical failures (cap the score at 40)
  • Recommends launching a free plan for every account to match Crestline
Task

Staff-level · v1.2

The outputs

9 setups answered this brief.

Needs more than a quick edit6 of 11 checks passed

Missing a stated condition that would change the recommendation, and an unsupported causal claim would need to be rephrased before use.

Output

To: Board of Directors, Harbourline From: VP Product Subject: Competitive Analysis: Crestline’s Free Tier and Our Strategic Response

Executive Summary

Crestline’s new free tier is not an existential threat to Harbourline, and we should not launch a free plan for everyone to match it. Doing so would destroy the unit economics of our low-ARPU solo tier while unnecessarily subsidizing our most profitable, sticky multi-user segments.

An analysis of our August–September churn data shows that while Crestline has accelerated cancellations among unconnected, solo businesses, our core revenue engine—accounts connected to our 2,100 accounting-firm partners—remains almost entirely insulated. Rather than engaging in a capital-destroying race to the bottom on price, we must accelerate connections to our accounting network, tighten our seat-utilization value proposition, and preempt Crestline’s upcoming product expansion.

---

1. Where Crestline Can Hurt Us—and Where It Can’t (The Evidence)

Our Revenue Base vs. Churn Reality Harbourline has 17,600 paying accounts generating ~$635,400 in total MRR ($7.6M ARR): * 1 user: 11,400 accounts (65% of base), $14 ARPU, $159,600 MRR (25% of revenue). Pre-Crestline annual churn: 31%. Only 9% are connected to an accounting firm. * 2–5 users: 4,900 accounts (28% of base), $52 ARPU, $254,800 MRR (40% of revenue). Pre-Crestline annual churn: 14%. 46% are connected to an accounting firm. * 6–20 users: 1,300 accounts (7% of base), $170 ARPU, $221,000 MRR (35% of revenue). Pre-Crestline annual churn: 6%. 81% are connected to an accounting firm.

Across August and September, we lost 250 accounts ($8,113 MRR), up from our historical baseline of ~190 accounts over two months (~95/month). An examination of all 250 cancellations reveals a stark divide:

SegmentTotal Cancelled (Aug–Sep)Went to Crestline (Free)Went to Spreadsheets / ClosedConnected to Firm & Went to CrestlineMRR Lost to Crestline
:---:---:---:---:---:---
1 user18314538 (Spreadsheets)0 (out of 1,026 connected)$2,021
2–5 users543123 (Business closed)3 (out of 2,254 connected)$1,683
6–20 users1394 (Business closed)1 (out of 1,053 connected)$1,605
Total250185654 (out of 4,333 connected)$5,309

Where Crestline Can Hurt Us: 1. Unconnected Solo Accounts: 78% of the accounts lost to Crestline (145 of 185) are 1-user accounts with no accounting firm attached. However, this segment already suffered 31% annual churn and represents just $2,021 of lost MRR over two months. Furthermore, given that total two-month churn only rose by ~60 accounts above our historical baseline (190 vs. 250), a large majority of the 185 accounts moving to Crestline are substituted churn—price-sensitive micro-businesses that previously churned to spreadsheets, now opting for a free tool. 2. Over-provisioned Multi-User Accounts: We lost 31 accounts in the 2–5 user tier and 9 accounts in the 6–20 user tier to Crestline’s 1-user free plan. Because Crestline’s free tier only supports a single user, these 40 businesses ($3,288 MRR) were paying us $52–$170/month while only truly needing single-user access for invoicing and expenses.

Where Crestline Cannot Hurt Us: * Firm-Connected Accounts: Approximately 4,333 of our customers (~25% of accounts, but ~48% of our total MRR at ~$305k/month) are connected to an accounting firm. Out of those 4,333 accounts, only 4 defected to Crestline ($427 MRR)—a two-month attrition rate to Crestline of 0.09%. * When an accountant reconciles, files, and advises inside Harbourline, the switching cost is no longer governed by the software price; a business owner will not sever their workflow with their trusted advisor to save $14 to $52 a month.

---

2. Our Competitive Moat: What Crestline Can Copy vs. What It Can’t

  • What They Can Copy (and Already Have): Core standalone bookkeeping, invoicing, expense tracking, and bank feeds. With $120M in fresh capital, Crestline can easily match or outspend us on standalone UI, automated bank rules, and customer acquisition marketing for solo operators.
  • What They Cannot Easily Copy: Our 2,100-firm partner network and two-sided workflow. Accounting firms are notoriously risk-averse and standardize their internal staff training and reconciliation processes around a primary ledger. Convincing 2,100 firms to migrate their practices and clients takes years of relationship-building, compliance verification, and practice-management tooling—not just venture capital. Crestline currently has zero partner program and is only just hiring 3 people to "explore" accountant partnerships.

---

3. Where Crestline Is Heading

Crestline’s public job postings and pricing structure reveal their actual playbook: 1. The Free Tier is a Loss-Leader for Payroll & Up-Market Expansion: Crestline is hiring 14 engineers for payroll and 6 for mid-market multi-entity, versus just 3 for accountant partnerships. They are using the $120M raise to subsidize free solo acquisition, planning to monetize those users as they hire employees (via payroll attach) and grow in complexity (via $25+/month plans and multi-entity features). 2. They Are Bypassing Accountants for Now: With only 3 exploratory roles in accountant partnerships, Crestline is at least 18–24 months away from fielding a competitive partner network. They are betting on direct-to-SMB product-led growth.

---

4. What We Should Do: Strategic Recommendations

1. Reject the "Free Plan for Everyone" Launching a universal free tier would instantly cannibalize up to $159,600 in monthly solo revenue while funding free bank-feed and support costs—playing directly into Crestline’s $120M balance sheet advantage. Worse, it would invite our 2–5 user accounts ($254.8k MRR) to downgrade.

2. Double Down on the Accounting Network (Our True Moat) Since firm-connected accounts have near-zero defection to Crestline (4 out of 4,333), increasing network attachment is our highest-ROI defense: * Drive 2–5 User Attachment from 46% to 70%+: Launch an in-app "Match with an Advisor" onboarding flow and co-marketing subsidies with our 2,100 firms to connect our 2,646 unconnected 2–5 user accounts. * Use a Targeted "Firm-Sponsored" Starter Tier Instead of Public Freemium: Allow our 2,100 partner firms to provision a free or $5/month "Ledger-Only" seat exclusively for their solo clients. This neutralizes Crestline’s price advantage only when an account is locked into our partner moat, turning our 2,100 firms into an aggressive acquisition channel.

3. Stop Multi-User Downgrades by Strengthening Collaboration We lost 40 multi-user accounts ($3,288 MRR—62% of our MRR lost to Crestline) to a single-user free plan because those customers weren't getting multi-user value. We need to audit multi-user accounts that only have 1 active login, proactively right-size them or drive adoption of multi-user workflows (role-based expense approvals, team receipt capture) that a 1-user free tier cannot support.

4. Preempt Crestline on Payroll Crestline’s 14 payroll engineering hires show how they plan to monetize SMBs as they grow from 1 to 2–5 users. Rather than spending 12 months building native payroll from scratch, we should immediately integrate deeply with (or white-label) a best-in-class payroll provider—specifically building accountant-friendly payroll reconciliation tools inside our partner portal before Crestline’s payroll product launches.

Grades and run details

Decision model 64 · LLM judge 67
Decision model checks
  • failUses the supplied evidence correctly46%
  • failAddresses the actual decision48%
  • passRespects explicit constraints33%
  • partialIdentifies material uncertainty31%
  • partialAvoids unsupported claims27%
  • passProduces the required deliverable76%
  • passFinds who's actually leaving83%
  • passReads where Crestline is heading100%
  • passCompetes with what customers really choose94%
  • failGets the base of every number right52%
  • passSeparates what a rival can copy100%
Run
Run
#1
API response time
22 s
Submitted
13 Oct 2026

Needs more than a quick edit7 of 11 checks passed

Contains an unsupported factual claim about the number of partner firms and speculates about data collection method; a PM would need to verify and correct these before use.

Output

Crestline: Competitive Analysis and Recommendation

To: Board of Directors From: VP Product

Recommendation

Do not launch a free plan for everyone. Crestline's free tier is a real threat to our solo accounts, but it does not reach the part of the business that earns most of our revenue. Respond by selling the accountant connection to solo and small accounts, protecting unconnected mid-size accounts, and watching three specific signals that would change this view.

What the August–September data shows

Of the 250 cancellations (about $8,100 MRR), 186 went to Crestline's free tier, 38 to spreadsheets, and 26 to closed businesses. Crestline's pull falls almost entirely on accounts without an accounting firm.

SegmentAccountsLost to Crestline (2 months)RateMRR lost
1 user, unconnected10,3741421.4%~$2,010
2–5 users, unconnected2,646311.2%~$1,660
6–20 users, unconnected24793.6%~$1,535
2–5 users, connected2,25430.1%~$190
6–20 users, connected1,05310.1%~$240
1 user, connected1,02600%$0

Connected accounts show the clearest pattern. Eight connected accounts cancelled: four closed their businesses and four went to Crestline. None went to spreadsheets.

Data caveat: The churn column in the customer base file (31% / 14% / 6% annual, roughly 360 cancellations a month) does not match the cancellation summary (about 95 a month before launch). Also, "went_to" appears to be self-reported. Finance needs to reconcile both before these figures go into any external materials. The rates above use the cancellation log as recorded.

Where Crestline can hurt us

  • Unconnected solo accounts. These are 65% of our accounts but about 25% of MRR. Crestline is taking roughly 70 a month, or about $1,000 of MRR per month. At this pace, that compounds to roughly $12k of lost ARR each month.
  • Unconnected mid-size accounts. This is our most exposed group per account: 3.6% lost in two months, at $170 MRR each. Crestline's six multi-entity engineering roles aim directly at this segment.
  • Product scope. Crestline is hiring 14 engineers for payroll. If payroll becomes a reason to stay on their platform, small employers with staff could move too.

Where it can't

  • Connected accounts are 19% of accounts but 49% of MRR (about $311k of $635k). Crestline took 4 of 3,307 in two months.
  • The firm network. We have 2,100 firms working inside our platform on reconciliation, filing and advice. Crestline has no partner programme and only three "exploring" roles.
  • Firm lock-in. Once a firm runs a client's books in our system, leaving means the firm changing its workflow, not just the customer changing software.

What Crestline could copy, and what it couldn't

Easy to copy: a free solo tier, bank feeds, invoicing and expenses. Crestline already has these, and we should assume they will match any feature we ship.

Hard to copy: the firm network, firm-side workflow, trust between firms and their clients, and years of connected books. Crestline is at "exploring" on accountant partnerships. My estimate is that a comparable network would take them at least 12–18 months to build, and their hiring suggests they have not yet committed to it.

Where Crestline is heading

  1. Up from solo. The free plan is a funnel. Paid plans start at $25 and the real money is in payroll and multi-entity work.
  2. Into our mid-market. Six engineers on multi-entity features target our 6–20 user accounts, which carry the highest MRR per account.
  3. Toward firms. The accountant partnership roles are the signal that matters most. If Crestline builds a firm network, our moat is much weaker.

What we should do

  1. Do not launch a free plan for everyone. Our 1-user unconnected base alone is about $145k MRR ($1.7M ARR). A free-for-all would also give away the 2–5 and 6–20 segments, about $476k MRR ($5.7M ARR), where Crestline is not competing. Crestline's observed impact in two months was about $5.6k MRR. Even if that understates the effect considerably, a free-for-all is the wrong response.
  1. Sell the connection to solo accounts. Connected 1-user accounts had zero churn; unconnected ones are where Crestline is winning. Launch a firm-connected solo offer and aim to move connected solo accounts from 9% toward 15% within six months. Partner economics need to be confirmed before we commit to a number.
  1. Protect unconnected 6–20 accounts now. There are 247 of them, worth about $42k MRR. Offer each a firm introduction and a named contact, and prioritise them for the multi-entity roadmap.
  1. Test any price response before rolling it out. If the connection offer underperforms, test a lower-priced solo tier in one market with a holdout group. Do not roll out a free plan to all accounts.
  1. Lock in partner firms. Offer multi-year terms or referral incentives to our top firms. Their switching cost is what protects us from a Crestline firm network.
  1. Fix the data within two weeks. Reconcile the churn baseline, and run a short exit survey to validate the "went_to" tags.

7. Set triggers to revisit free. I propose we reconsider a broader free plan if any of these occur: - Crestline launches an accountant partner programme. - Crestline-attributed churn among connected accounts exceeds 0.5% a month for two consecutive months. - Crestline-attributed churn among unconnected 2–5 user accounts exceeds 2% a month for two consecutive months.

Claims the judge couldn’t find in the brief

  • 'went_to' appears to be self-reported
  • We have 2,100 firms working inside our platform

Grades and run details

Decision model 73 · LLM judge 67
Decision model checks
  • failUses the supplied evidence correctly31%
  • passAddresses the actual decision100%
  • passRespects explicit constraints55%
  • passIdentifies material uncertainty100%
  • partialAvoids unsupported claims33%
  • passProduces the required deliverable66%
  • passFinds who's actually leaving86%
  • passReads where Crestline is heading100%
  • partialCompetes with what customers really choose69%
  • failGets the base of every number right43%
  • passSeparates what a rival can copy100%
Run
Run
#1
API response time
82 s
Submitted
8 Oct 2026

Needs more than a quick edit9 of 11 checks passed

The output contains minor but factual errors in the cancellation counts (unconnected multi-user and connected 6-20) that would need correction before use.

Output

Crestline: where it can hurt us, and what we should do

To: Board | From: VP Product

Recommendation

Don't launch a free plan for everyone. Crestline is taking our solo and unconnected customers, not our firm-connected ones. Accounts with 2-20 users make up 75% of our MRR (about $476k of $635k), and a free-for-all plan would put that revenue at risk to defend the 25% that is solo. Instead we should connect more customers to accounting firms, test a narrow solo response, and lock in firms before Crestline builds a partner programme.

What the cancellations show

Since launch, 250 accounts (about $8,100 MRR) have cancelled:

Went toAccountsMRR
Crestline186 (74%)~$5,580 (69%)
Business closed26 (10%)~$1,930 (24%)
Spreadsheets / no tool38 (15%)~$555 (7%)

The 250 overstates the damage. Normal churn is about 95 accounts a month, so the excess is roughly 60-110 accounts, depending on whether we count from 15 August or 1 August. There is also a data problem: the segment churn rates in our customer table imply about 360 cancellations a month, not 95. At those rates, no segment is churning above normal. Finance needs to reconcile the two before anyone quotes an "increase." Until then, treat 250 as an upper bound.

Where Crestline can hurt us

Solo accounts (11,400 accounts, 65% of the base, $14 average, 9% connected to a firm). - Free beats $14, and this segment already churned at 31% a year. - 142 solo accounts ($1,960 MRR) went to Crestline. - We should expect more losses here, but the whole segment is only about $160k MRR.

Unconnected multi-user accounts, our largest dollar leak. - 42 accounts with 2-20 users and no firm connection went to Crestline, taking $3,320 MRR. That is 59% of the MRR we lost to Crestline, more than the solo losses. - Crestline's free plan is one user only, so these customers either took a paid plan (from $25, against our $52 and $170 averages) or cut down to one seat. The files can't tell us which, and the cancellations are labelled "free tier." We need exit calls this week. - A free plan wouldn't win them back, because they didn't leave for a free plan.

Where it can't (so far)

  • Connected accounts are barely leaving. They are about 25% of our base (roughly 4,300 accounts) but only 8 of 250 cancellations (3%).
  • Only 4 connected accounts went to Crestline ($427 MRR). The other 4 were businesses that closed.
  • Larger accounts show the same pattern. In the 6-20 segment, 81% of the base is connected, but only 4 of 14 cancellations were. The 10 that went to Crestline were nearly all unconnected; one was connected.
  • Caveats. This is six to eight weeks of data. Connected accounts may also simply be stickier customers, so the data shows correlation, not proof. Switching costs show up at renewal, so this is an early read.

Which advantages Crestline could copy

Easy to copy: - A free plan (they've done it) and feature parity. Their open roles are 14 engineers for payroll and 6 for mid-market multi-entity, 20 of 23 openings. Assume they match our core product and add payroll within a year. - Price. They have $120M to subsidise it.

Hard to copy: - Our firm network. 2,100 firms reconcile, file and advise from inside Harbourline. A firm standardises dozens of clients on one platform, so a switch means moving the firm's whole workflow, not one customer. Building that two-sided trust takes years. Crestline has no partner programme today, and "accountant partnerships" is three roles marked "exploring."

The window is limited. With $120M they can pay firms, buy a practice-management tool, or acquire a partner network. My judgement, not a data point, is that we have 12-18 months of lead. That is a window to use, not a permanent moat.

Where Crestline is heading

Crestline looks to be using the free solo plan as a funnel and monetising later through payroll and multi-entity, which means moving into our 2-5 and 6-20 accounts. Today it competes at the bottom of our base. Within a year it will compete in the middle, and the accountant-partnership hiring is the signal that it's going after our moat.

What we should do

  1. Don't match with a free plan for everyone. It would put about $476k of MRR at risk. It also wouldn't answer why multi-user accounts are leaving.
  1. Connect the unconnected. About 2,900 multi-user accounts have no firm connection (roughly 2,650 in 2-5 and 250 in 6-20), worth about $180k MRR on segment averages. Launch an in-product invite flow, and ask firms to onboard their clients. This is the best protection we have, and Crestline can't match it today.
  1. Test, don't commit, on solo. Run a 60-day test of a limited or lower-priced solo plan, limited to one user so larger accounts can't downgrade into it. Offer it to new sign-ups and as a save offer for at-risk solo accounts, with a route to a firm. The downside is capped at the solo segment. Success means solo churn back toward its 31% baseline and a rising share of solo accounts connected to a firm.
  1. Lock in firms in the next 90 days. Give firms better client-migration tools, firm-side incentives and a commitment from the largest ones. Move before Crestline's partnership hiring turns into a programme.

5. Fix the data and set tripwires. - Reconcile the churn baseline. - Run exit calls on the 42 unconnected multi-user losses. - Review the free-plan question if any of these happen: - connected-account cancellations to Crestline rise from about 2 a month to more than 10; - Crestline announces a firm programme or general availability of payroll; - multi-user unconnected churn keeps rising after the exit calls.

I'll report back at the next board meeting with the reconciled churn baseline and the exit-call findings.

Claims the judge couldn’t find in the brief

  • 42 accounts with 2-20 users and no firm connection went to Crestline, taking $3,320 MRR
  • In the 6-20 segment, 81% of the base is connected, but only 4 of 14 cancellations were

Grades and run details

Decision model 77 · LLM judge 75
Decision model checks
  • failUses the supplied evidence correctly16%
  • passAddresses the actual decision100%
  • passRespects explicit constraints53%
  • passIdentifies material uncertainty100%
  • passAvoids unsupported claims41%
  • passProduces the required deliverable74%
  • passFinds who's actually leaving31%
  • passReads where Crestline is heading100%
  • partialCompetes with what customers really choose72%
  • failGets the base of every number right19%
  • passSeparates what a rival can copy100%
Run
Run
#1
API response time
1.7 min
Submitted
6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 6

Uses the supplied evidence correctlyMixedWrongWrong
Gemini 4 Argon · API

All key numbers and facts are taken correctly from the supplied context or derived by arithmetic from it.

Haiku 5.5 · API

States that Harbourline has 2,100 partner firms, which is not in the supplied context and is invented.

Sonnet 5.5 · API

Two factual claims are wrong: the number of unconnected multi-user accounts going to Crestline is 40, not 42, and the number of connected 6-20 cancellations is 5, not 4.

Addresses the actual decisionWrongRightRight
Gemini 4 Argon · API

The output does not state what result or condition would change the recommendation, as required.

Haiku 5.5 · API

Clear early recommendation not to launch a free plan, and sets triggers to revisit.

Sonnet 5.5 · API

Commits early to 'Don't launch a free plan for everyone' and sets tripwires (connected cancellations >10/month, Crestline firm programme or payroll GA, rising multi-user unconnected churn) that would change the call.

Identifies material uncertaintyWrongRightRight
Gemini 4 Argon · API

It does not name specific unknowns that could change the decision or say how they would be resolved.

Haiku 5.5 · API

Identifies data discrepancies and sets specific churn thresholds and Crestline actions as triggers.

Sonnet 5.5 · API

Names the churn baseline discrepancy, the early data window, the need for exit calls, and sets specific tripwires that would trigger a review.

Avoids unsupported claimsWrongWrongRight
Gemini 4 Argon · API

It presents the interpretation that a large majority of Crestline defections are 'substituted churn' as fact, without labelling it as hypothesis.

Haiku 5.5 · API

Presents 'went_to is self-reported' and '2,100 firms' as facts without evidence.

Sonnet 5.5 · API

Interpretations and forecasts are labelled as such (e.g., 'My judgement, not a data point,' 'looks to be'), and confident claims are grounded in the evidence.

Competes with what customers really chooseRightWrongRight
Gemini 4 Argon · API

It mentions spreadsheets as destination and draws an implication from Crestline's investment direction (payroll) for its own response.

Haiku 5.5 · API

Does not treat spreadsheets/manual work as a competitor with a direction-of-travel insight; merely counts them as a destination.

Sonnet 5.5 · API

Treats spreadsheets/no tool as a competitor and draws a direction-of-travel insight from Crestline's investment in payroll and multi-entity.

Gets the base of every number rightMixedMixedWrong
Gemini 4 Argon · API

All derived figures check out against the supplied data; the approximate $305k connected MRR is 48% of total MRR and consistent.

Haiku 5.5 · API

All derived percentages and MRR figures check out against the base data.

Sonnet 5.5 · API

Derived figures for unconnected multi-user cancellations (42 vs actual 40) and connected 6-20 cancellations (4 vs actual 5) are incorrect.

All got right 5

Respects explicit constraintsRightRightRight
Gemini 4 Argon · API

The output is addressed to the board, is under 1,200 words, and does not propose a free plan for every account.

Haiku 5.5 · API

Memo is under 1,200 words and answers the board chair's question.

Sonnet 5.5 · API

The output is a board memo under 1,200 words, addresses the board chair's question, and does not propose a free plan for every account.

Produces the required deliverableRightRightRight
Gemini 4 Argon · API

The competitive analysis is present, in memo form for the board, within word limit, and usable with edits.

Haiku 5.5 · API

Complete board memo with recommendation, analysis, and actions within length; the core decision is actionable.

Sonnet 5.5 · API

A complete competitive analysis memo for the board, within the word limit, with a clear recommendation and actionable next steps.

Finds who's actually leavingRightRightRight
Gemini 4 Argon · API

It counts cancellations by size and partner connection, sizes MRR, and sets losses against the base, showing the pattern.

Haiku 5.5 · API

Counts cancellations by size and connection, sizes MRR lost to Crestline, and shows it is mostly solo unconnected accounts.

Sonnet 5.5 · API

Correctly identifies that losses to Crestline are overwhelmingly solo, unconnected accounts, sizes the MRR, and shows connected accounts are barely leaving.

Reads where Crestline is headingRightRightRight
Gemini 4 Argon · API

It reads Crestline's job ads to signal a move into payroll, mid-market, and possible accountant partnerships, and flags the threat.

Haiku 5.5 · API

Uses hiring information to infer Crestline is heading into payroll, mid-market, and possibly accountant partnerships.

Sonnet 5.5 · API

Uses Crestline's hiring (payroll, mid-market multi-entity, accountant partnerships exploring) to forecast an upmarket move and a future threat to the partner channel.

Separates what a rival can copyRightRightRight
Gemini 4 Argon · API

It clearly separates copyable advantages (features, price) from durable ones (partner network, switching costs) and anchors response on the latter.

Haiku 5.5 · API

Clearly separates copyable features (free tier, bank feeds) from durable advantages (firm network, switching costs) and anchors response accordingly.

Sonnet 5.5 · API

Separates copyable advantages (free plan, price, features) from durable ones (firm network, switching costs) and anchors the response in the latter.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6.1 SolwithAPI93.295.82None
2Opus 5.5withClaude88.6100.02None
3GPT-6 AstrawithChatGPT86.495.82None
4GPT-6 LunawithAPI93.283.32None
5Sonnet 5.5withAPI86.483.32None
6Haiku 5.5withAPI81.879.22None
7Gemini 3.8 FlashwithAPI68.254.22None
8Gemini 4 ArgonwithAPI65.954.22None
9Gemini 3.5 Flash-LitewithGemini54.525.02None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6.1 Sol · API: task score 95, 14× the cheapestOpus 5.5 · Claude: task score 95, 75× the cheapestGPT-6 Astra · ChatGPT: task score 92, 87× the cheapestGPT-6 Luna · API: task score 86, the cheapestSonnet 5.5 · API: task score 83, 44× the cheapestHaiku 5.5 · API: task score 79, 3.0× the cheapestGemini 3.8 Flash · API: task score 60, 12× the cheapestGemini 4 Argon · API: task score 57, 23× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 38, 2.9× the cheapestGPT-6.1 Sol · APIOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTGPT-6 Luna · APISonnet 5.5 · APIHaiku 5.5 · APIGemini 3.8 Flash · APIGemini 4 Argon · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API861.0×
Gemini 3.5 Flash-Lite · Gemini382.9×
Haiku 5.5 · API793.0×
Gemini 3.8 Flash · API6012×
GPT-6.1 Sol · API9514×
Gemini 4 Argon · API5723×
Sonnet 5.5 · API8344×
Opus 5.5 · Claude9575×
GPT-6 Astra · ChatGPT9287×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Deciding how to respond to competitors.

Why it matters

Feature grids hide the real competition, which is often the customer's current workaround, and the response that feels urgent is usually to copy everything.

What good looks like

  • Includes the status quo and non-obvious alternatives
  • Finds where losses really go, by segment
  • Separates advantages a rival can copy from ones it can't
  • Reads where competitors are heading

Deliberately not measured

  • Market sizing
Capability tested

Competitive insight

The failure we’re looking for

A feature checklist and a plan to match everything

Grading

Decision model and LLM judge, calibrated against a blind PM review