Tasks / Strategy

Competitor analysis

Can the model find where a competitor can actually hurt you, from win/loss and churn data, rather than list features?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 18 graded outputs by 9 models. 56% were usable with at most a quick edit.

Reliably right

  1. Respects explicit constraints100% pass
    Delivers a competitive analysis memo to Dan within 900 words, addressing all requested topics without proposing a full feature-by-feature match.
    Opus 5.5 · Claude · The competitor isn't who you think
  2. Reads where Crestline is heading100% pass
    It reads Crestline's job ads as signalling payroll, mid-market, and possible accountant partnerships, and adjusts the response accordingly.
    GPT-6.1 Sol · API · Free forever, funded
  3. Produces the required deliverable97% pass
    The memo is complete, in the right form for the CEO, within the word limit, and actionable with light edits.
    Opus 5.5 · Claude · The competitor isn't who you think

Where it slips

  1. Uses the supplied evidence correctly56% pass
    Two factual claims are wrong: the number of unconnected multi-user accounts going to Crestline is 40, not 42, and the number of connected 6-20 cancellations is 5, not 4.
    Sonnet 5.5 · API · Free forever, funded
  2. Gets the base of every number right63% pass
    Derived figures for unconnected multi-user cancellations (42 vs actual 40) and connected 6-20 cancellations (4 vs actual 5) are incorrect.
    Sonnet 5.5 · API · Free forever, funded
  3. Separates what a rival can copy68% pass
    Does not explicitly separate advantages a well-funded competitor could copy from those it couldn't, nor anchor the response in durable advantages with evidence.
    GPT-6 Luna · API · The competitor isn't who you think

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Competes with what customers really choose

    Does the analysis treat the status quo (spreadsheets, manual work, doing nothing) as a competitor, and draw at least one implication from where competitors are investing rather than only from today's features?

    Passes when Analyses the status quo, strengths included, and gives a direction-of-travel insight that affects the recommendation.

  2. Separates what a rival can copy

    Does the analysis separate advantages a well-funded competitor could copy (features, price, being first) from ones it couldn't easily (switching costs, networks, relationships), and build the response on the latter?

    Passes when Names which advantages are copyable and which aren't, with evidence, and anchors the response in the durable ones.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the VP Product at Harbourline. Crestline raised $120M in August and launched a free plan for solo businesses. Our board chair wants to know whether we should launch a free plan for everyone to match it (the note is below). Write the board a competitive analysis of no more than 1,200 words: where Crestline can hurt us and where it can't, with the evidence; which of our advantages it could copy and which it couldn't; where it's heading; and what we should do. The files are attached.

What the model was given6 items: About Harbourline, board_chair_note.md, customer_base.md, cancellations_aug_sep.csv (every account that cancelled since Crestline's launch), cancellation_summary.md, crestline_public_info.md
About HarbourlineBookkeeping, invoicing and expenses for small businesses, with a partner network that connects customers to their accounting firm.
board_chair_note.md1 lines · Download
From the board chair: "Crestline's free tier is an existential threat. Should we launch a free plan for everyone to match it? We need a view on where they can hurt us and where they can't."
customer_base.md9 lines · Download
# Harbourline customers (end of September)

| Account size | Paying accounts | Share connected to an accounting firm | Average revenue per month | Annual churn before Crestline's free tier |
|---|---|---|---|---|
| 1 user | 11,400 | 9% | $14 | 31% |
| 2-5 users | 4,900 | 46% | $52 | 14% |
| 6-20 users | 1,300 | 81% | $170 | 6% |
…
cancellations_aug_sep.csv (every account that cancelled since Crestline's launch)cancelled_account,account_users,connected_to_accounting_firm,mrr_usd,went_to H-1200,1,no,14,Crestline (free tier) H-1201,1,no,10,Crestline (free tier) H-1202,2-5,no,58,Crestline (free tier) H-1203,1,no,11,Crestline (free tier) H-1204,1,no,17,Crestline (free tier) H-1205,1,no,15,Spreadsheets / no tool H-1206,1,no,15,Spreadsheets / no tool H-1207,1,no,9,Crestline (free tier) H-1208,2-5,no,66,Crestline (free tier) H-1209,2-5,no,44,Crestline (free tier) H-1210,1,no,11,Crestline (free tier) H-1211,1,no,9,Crestline (free tier) H-1212,1,no,12,Crestline (free tier) H-1213,1,no,16,Crestline (free tier) H-1214,2-5,yes,73,Crestline (free tier) H-1215,1,no,9,Crestline (free tier) H-1216,2-5,no,56,Crestline (free tier) H-1217,1,no,14,Crestline (free tier) H-1218,2-5,no,47,Business closed H-1219,1,no,16,Crestline (free tier) H-1220,6-20,no,177,Crestline (free tier) H-1221,2-5,yes,51,Crestline (free tier) H-1222,2-5,no,54,Business closed H-1223,6-20,no,168,Crestline (free tier) H-1224,1,no,18,Crestline (free tier) H-1225,1,no,13,Spreadsheets / no tool H-1226,1,no,11,Crestline (free tier) H-1227,1,no,15,Crestline (free tier) H-1228,1,no,12,Crestline (free tier) H-1229,1,no,13,Crestline (free tier) H-1230,2-5,no,51,Business closed H-1231,1,no,18,Crestline (free tier) H-1232,1,no,16,Spreadsheets / no tool H-1233,1,no,12,Crestline (free tier) H-1234,1,no,16,Spreadsheets / no tool H-1235,1,no,12,Spreadsheets / no tool H-1236,1,no,13,Crestline (free tier) H-1237,1,no,17,Spreadsheets / no tool H-1238,6-20,yes,188,Business closed H-1239,1,no,17,Crestline (free tier) H-1240,1,no,15,Crestline (free tier) H-1241,6-20,yes,238,Crestline (free tier) H-1242,1,no,11,Spreadsheets / no tool H-1243,2-5,no,66,Crestline (free tier) H-1244,2-5,no,63,Business closed H-1245,6-20,no,199,Crestline (free tier) H-1246,1,no,18,Crestline (free tier) H-1247,1,no,11,Crestline (free tier) H-1248,1,no,15,Crestline (free tier) H-1249,2-5,no,60,Crestline (free tier) H-1250,2-5,no,66,Crestline (free tier) H-1251,2-5,no,34,Crestline (free tier) H-1252,1,no,12,Crestline (free tier) H-1253,1,no,18,Crestline (free tier) H-1254,1,no,18,Spreadsheets / no tool H-1255,2-5,no,31,Crestline (free tier) H-1256,1,no,17,Crestline (free tier) H-1257,1,no,16,Crestline (free tier) H-1258,1,no,18,Spreadsheets / no tool H-1259,1,no,9,Crestline (free tier) H-1260,1,no,15,Crestline (free tier) H-1261,1,no,18,Crestline (free tier) H-1262,6-20,yes,170,Business closed H-1263,1,no,18,Crestline (free tier) H-1264,2-5,no,41,Business closed H-1265,1,no,14,Spreadsheets / no tool H-1266,1,no,14,Crestline (free tier) H-1267,1,no,12,Crestline (free tier) H-1268,6-20,no,112,Crestline (free tier) H-1269,1,no,9,Crestline (free tier) H-1270,2-5,no,71,Business closed H-1271,1,no,12,Crestline (free tier) H-1272,1,no,11,Crestline (free tier) H-1273,1,no,15,Crestline (free tier) H-1274,1,no,9,Crestline (free tier) H-1275,6-20,no,209,Crestline (free tier) H-1276,2-5,no,36,Crestline (free tier) H-1277,1,no,13,Crestline (free tier) H-1278,1,no,11,Crestline (free tier) H-1279,1,no,15,Crestline (free tier) H-1280,1,no,18,Spreadsheets / no tool H-1281,1,no,16,Crestline (free tier) H-1282,1,no,17,Crestline (free tier) H-1283,1,no,9,Crestline (free tier) H-1284,1,no,10,Crestline (free tier) H-1285,6-20,yes,128,Business closed H-1286,1,no,17,Crestline (free tier) H-1287,1,no,12,Spreadsheets / no tool H-1288,2-5,no,39,Crestline (free tier) H-1289,1,no,14,Crestline (free tier) H-1290,1,no,12,Crestline (free tier) H-1291,1,no,16,Crestline (free tier) H-1292,2-5,no,62,Crestline (free tier) H-1293,1,no,12,Crestline (free tier) H-1294,1,no,12,Crestline (free tier) H-1295,1,no,9,Spreadsheets / no tool H-1296,1,no,11,Crestline (free tier) H-1297,1,no,16,Crestline (free tier) H-1298,1,no,17,Crestline (free tier) H-1299,1,no,13,Spreadsheets / no tool H-1300,1,no,13,Crestline (free tier) H-1301,1,no,17,Crestline (free tier) H-1302,1,no,12,Crestline (free tier) H-1303,1,no,17,Crestline (free tier) H-1304,1,no,14,Crestline (free tier) H-1305,1,no,9,Crestline (free tier) H-1306,1,no,15,Crestline (free tier) H-1307,1,no,16,Crestline (free tier) H-1308,1,no,16,Crestline (free tier) H-1309,1,no,12,Crestline (free tier) H-1310,2-5,no,47,Business closed H-1311,2-5,no,78,Crestline (free tier) H-1312,1,no,11,Crestline (free tier) H-1313,1,no,14,Crestline (free tier) H-1314,1,no,15,Spreadsheets / no tool H-1315,1,no,14,Spreadsheets / no tool H-1316,1,no,9,Crestline (free tier) H-1317,2-5,no,52,Business closed H-1318,1,no,13,Crestline (free tier) H-1319,2-5,no,43,Business closed H-1320,1,no,11,Spreadsheets / no tool H-1321,1,no,15,Crestline (free tier) H-1322,2-5,no,55,Business closed H-1323,1,no,12,Spreadsheets / no tool H-1324,6-20,no,166,Crestline (free tier) H-1325,2-5,no,67,Business closed H-1326,1,no,16,Crestline (free tier) H-1327,1,no,15,Crestline (free tier) H-1328,1,no,13,Crestline (free tier) H-1329,1,no,17,Crestline (free tier) H-1330,1,no,12,Crestline (free tier) H-1331,1,no,13,Crestline (free tier) H-1332,2-5,no,68,Crestline (free tier) H-1333,1,no,14,Spreadsheets / no tool H-1334,1,no,18,Crestline (free tier) H-1335,1,no,9,Crestline (free tier) H-1336,1,no,14,Crestline (free tier) H-1337,1,no,13,Crestline (free tier) H-1338,1,no,10,Crestline (free tier) H-1339,1,no,18,Spreadsheets / no tool H-1340,2-5,no,70,Crestline (free tier) H-1341,1,no,13,Spreadsheets / no tool H-1342,2-5,no,34,Crestline (free tier) H-1343,1,no,11,Crestline (free tier) H-1344,1,no,17,Spreadsheets / no tool H-1345,6-20,no,159,Crestline (free tier) H-1346,6-20,no,214,Crestline (free tier) H-1347,2-5,no,67,Crestline (free tier) H-1348,2-5,no,40,Crestline (free tier) H-1349,1,no,18,Crestline (free tier) H-1350,1,no,17,Crestline (free tier) H-1351,1,no,11,Crestline (free tier) H-1352,1,no,17,Crestline (free tier) H-1353,2-5,no,48,Business closed H-1354,1,no,12,Crestline (free tier) H-1355,1,no,16,Crestline (free tier) H-1356,1,no,14,Crestline (free tier) H-1357,1,no,18,Crestline (free tier) H-1358,1,no,11,Crestline (free tier) H-1359,1,no,17,Spreadsheets / no tool H-1360,1,no,12,Spreadsheets / no tool H-1361,2-5,no,39,Business closed H-1362,1,no,18,Crestline (free tier) H-1363,2-5,no,39,Crestline (free tier) H-1364,2-5,no,41,Business closed H-1365,2-5,no,51,Crestline (free tier) H-1366,1,no,11,Crestline (free tier) H-1367,2-5,no,76,Business closed H-1368,1,no,11,Crestline (free tier) H-1369,1,no,18,Crestline (free tier) H-1370,2-5,no,37,Business closed H-1371,1,no,9,Crestline (free tier) H-1372,1,no,18,Crestline (free tier) H-1373,2-5,no,36,Crestline (free tier) H-1374,2-5,no,77,Crestline (free tier) H-1375,2-5,no,64,Crestline (free tier) H-1376,1,no,9,Crestline (free tier) H-1377,2-5,no,71,Crestline (free tier) H-1378,1,no,10,Crestline (free tier) H-1379,1,no,12,Spreadsheets / no tool H-1380,6-20,yes,235,Business closed H-1381,1,no,9,Crestline (free tier) H-1382,1,no,15,Spreadsheets / no tool H-1383,2-5,yes,65,Crestline (free tier) H-1384,1,no,16,Crestline (free tier) H-1385,1,no,17,Spreadsheets / no tool H-1386,1,no,13,Spreadsheets / no tool H-1387,2-5,no,66,Business closed H-1388,1,no,17,Crestline (free tier) H-1389,1,no,18,Crestline (free tier) H-1390,1,no,16,Crestline (free tier) H-1391,1,no,18,Spreadsheets / no tool H-1392,1,no,15,Crestline (free tier) H-1393,1,no,10,Crestline (free tier) H-1394,1,no,11,Crestline (free tier) H-1395,2-5,no,46,Business closed H-1396,1,no,16,Crestline (free tier) H-1397,1,no,15,Crestline (free tier) H-1398,2-5,no,34,Crestline (free tier) H-1399,1,no,13,Crestline (free tier) H-1400,1,no,13,Crestline (free tier) H-1401,1,no,10,Crestline (free tier) H-1402,1,no,11,Crestline (free tier) H-1403,2-5,no,49,Crestline (free tier) H-1404,2-5,no,58,Crestline (free tier) H-1405,1,no,17,Crestline (free tier) H-1406,1,no,17,Crestline (free tier) H-1407,1,no,12,Crestline (free tier) H-1408,2-5,no,37,Crestline (free tier) H-1409,1,no,15,Spreadsheets / no tool H-1410,2-5,no,31,Crestline (free tier) H-1411,1,no,13,Spreadsheets / no tool H-1412,1,no,15,Crestline (free tier) H-1413,1,no,13,Crestline (free tier) H-1414,1,no,15,Crestline (free tier) H-1415,1,no,18,Crestline (free tier) H-1416,1,no,14,Crestline (free tier) H-1417,1,no,11,Crestline (free tier) H-1418,6-20,no,131,Crestline (free tier) H-1419,1,no,17,Crestline (free tier) H-1420,1,no,18,Crestline (free tier) H-1421,1,no,16,Crestline (free tier) H-1422,1,no,11,Crestline (free tier) H-1423,2-5,no,78,Business closed H-1424,1,no,16,Spreadsheets / no tool H-1425,1,no,18,Spreadsheets / no tool H-1426,1,no,18,Crestline (free tier) H-1427,1,no,15,Crestline (free tier) H-1428,1,no,14,Spreadsheets / no tool H-1429,1,no,14,Spreadsheets / no tool H-1430,2-5,no,63,Business closed H-1431,1,no,18,Crestline (free tier) H-1432,1,no,15,Crestline (free tier) H-1433,1,no,17,Spreadsheets / no tool H-1434,1,no,14,Crestline (free tier) H-1435,1,no,14,Crestline (free tier) H-1436,1,no,12,Crestline (free tier) H-1437,1,no,11,Crestline (free tier) H-1438,1,no,13,Crestline (free tier) H-1439,1,no,13,Spreadsheets / no tool H-1440,2-5,no,54,Business closed H-1441,1,no,14,Crestline (free tier) H-1442,1,no,16,Crestline (free tier) H-1443,2-5,no,74,Crestline (free tier) H-1444,1,no,14,Crestline (free tier) H-1445,2-5,no,65,Crestline (free tier) H-1446,1,no,18,Crestline (free tier) H-1447,1,no,11,Crestline (free tier) H-1448,1,no,10,Crestline (free tier) H-1449,2-5,no,69,Business closed
cancellation_summary.md1 lines · Download
# Cancellations, August to September: 250 accounts, about $8,100 of MRR. Before Crestline's launch we lost about 95 accounts a month.
crestline_public_info.md6 lines · Download
# Crestline (news and public pages)

- 1 August: raised $120M. 15 August: launched a free plan for 1 user (invoicing, expenses, bank feeds).
- Paid plans from $25/month; no accounting-firm partner programme.
- Open roles (September): 14 engineers for payroll; 6 for "mid-market multi-entity"; 3 for "accountant partnerships (exploring)".
- Pricing page: free plan "free forever for solo businesses".
What a strong answer doesThe answer key the graders mark against

Shows from the cancellations that Crestline is taking single-user accounts not connected to an accounting firm: 186 of the 250 cancellations went to Crestline, 142 of them single-user accounts at $9 to $19 a month, and only 4 of the 186 were connected to an accounting firm; the $5,600 of MRR lost to Crestline is a small share of a base where 1-user accounts already churned at 31% a year. The partner network is the advantage Crestline can't easily copy: 81% of 6-20 user accounts and 46% of 2-5 user accounts are connected to a firm, and they barely leave; Crestline has no partner programme. Free and price are easy to copy, and so is matching features. Reads Crestline's hiring (payroll, mid-market multi-entity, accountant partnerships 'exploring') as heading upmarket and possibly at the partner channel, the real future threat. Recommends against a free plan for everyone: it would give away revenue from 11,400 single-user accounts paying about $14 a month (about $160,000 MRR) to defend against losses that are a fraction of that. Instead: let the least valuable solo accounts go or offer a narrow solo option tied to an accounting firm, deepen the partner network (bring firms' clients onto Harbourline, lock in the firm workflow), and get ahead on payroll and multi-entity where Crestline is heading. Sets the signals that would change the call (losses spreading to connected or multi-user accounts, or Crestline launching a partner programme).

Critical failures (cap the score at 40)
  • Recommends launching a free plan for every account to match Crestline
Task

Staff-level · v1.2

The outputs

9 setups answered this brief.

Needs more than a quick edit6 of 11 checks passed

Missing a stated condition that would change the recommendation, and an unsupported causal claim would need to be rephrased before use.

Output

To: Board of Directors, Harbourline From: VP Product Subject: Competitive Analysis: Crestline’s Free Tier and Our Strategic Response

Executive Summary

Crestline’s new free tier is not an existential threat to Harbourline, and we should not launch a free plan for everyone to match it. Doing so would destroy the unit economics of our low-ARPU solo tier while unnecessarily subsidizing our most profitable, sticky multi-user segments.

An analysis of our August–September churn data shows that while Crestline has accelerated cancellations among unconnected, solo businesses, our core revenue engine—accounts connected to our 2,100 accounting-firm partners—remains almost entirely insulated. Rather than engaging in a capital-destroying race to the bottom on price, we must accelerate connections to our accounting network, tighten our seat-utilization value proposition, and preempt Crestline’s upcoming product expansion.

---

1. Where Crestline Can Hurt Us—and Where It Can’t (The Evidence)

Our Revenue Base vs. Churn Reality Harbourline has 17,600 paying accounts generating ~$635,400 in total MRR ($7.6M ARR): * 1 user: 11,400 accounts (65% of base), $14 ARPU, $159,600 MRR (25% of revenue). Pre-Crestline annual churn: 31%. Only 9% are connected to an accounting firm. * 2–5 users: 4,900 accounts (28% of base), $52 ARPU, $254,800 MRR (40% of revenue). Pre-Crestline annual churn: 14%. 46% are connected to an accounting firm. * 6–20 users: 1,300 accounts (7% of base), $170 ARPU, $221,000 MRR (35% of revenue). Pre-Crestline annual churn: 6%. 81% are connected to an accounting firm.

Across August and September, we lost 250 accounts ($8,113 MRR), up from our historical baseline of ~190 accounts over two months (~95/month). An examination of all 250 cancellations reveals a stark divide:

SegmentTotal Cancelled (Aug–Sep)Went to Crestline (Free)Went to Spreadsheets / ClosedConnected to Firm & Went to CrestlineMRR Lost to Crestline
:---:---:---:---:---:---
1 user18314538 (Spreadsheets)0 (out of 1,026 connected)$2,021
2–5 users543123 (Business closed)3 (out of 2,254 connected)$1,683
6–20 users1394 (Business closed)1 (out of 1,053 connected)$1,605
Total250185654 (out of 4,333 connected)$5,309

Where Crestline Can Hurt Us: 1. Unconnected Solo Accounts: 78% of the accounts lost to Crestline (145 of 185) are 1-user accounts with no accounting firm attached. However, this segment already suffered 31% annual churn and represents just $2,021 of lost MRR over two months. Furthermore, given that total two-month churn only rose by ~60 accounts above our historical baseline (190 vs. 250), a large majority of the 185 accounts moving to Crestline are substituted churn—price-sensitive micro-businesses that previously churned to spreadsheets, now opting for a free tool. 2. Over-provisioned Multi-User Accounts: We lost 31 accounts in the 2–5 user tier and 9 accounts in the 6–20 user tier to Crestline’s 1-user free plan. Because Crestline’s free tier only supports a single user, these 40 businesses ($3,288 MRR) were paying us $52–$170/month while only truly needing single-user access for invoicing and expenses.

Where Crestline Cannot Hurt Us: * Firm-Connected Accounts: Approximately 4,333 of our customers (~25% of accounts, but ~48% of our total MRR at ~$305k/month) are connected to an accounting firm. Out of those 4,333 accounts, only 4 defected to Crestline ($427 MRR)—a two-month attrition rate to Crestline of 0.09%. * When an accountant reconciles, files, and advises inside Harbourline, the switching cost is no longer governed by the software price; a business owner will not sever their workflow with their trusted advisor to save $14 to $52 a month.

---

2. Our Competitive Moat: What Crestline Can Copy vs. What It Can’t

  • What They Can Copy (and Already Have): Core standalone bookkeeping, invoicing, expense tracking, and bank feeds. With $120M in fresh capital, Crestline can easily match or outspend us on standalone UI, automated bank rules, and customer acquisition marketing for solo operators.
  • What They Cannot Easily Copy: Our 2,100-firm partner network and two-sided workflow. Accounting firms are notoriously risk-averse and standardize their internal staff training and reconciliation processes around a primary ledger. Convincing 2,100 firms to migrate their practices and clients takes years of relationship-building, compliance verification, and practice-management tooling—not just venture capital. Crestline currently has zero partner program and is only just hiring 3 people to "explore" accountant partnerships.

---

3. Where Crestline Is Heading

Crestline’s public job postings and pricing structure reveal their actual playbook: 1. The Free Tier is a Loss-Leader for Payroll & Up-Market Expansion: Crestline is hiring 14 engineers for payroll and 6 for mid-market multi-entity, versus just 3 for accountant partnerships. They are using the $120M raise to subsidize free solo acquisition, planning to monetize those users as they hire employees (via payroll attach) and grow in complexity (via $25+/month plans and multi-entity features). 2. They Are Bypassing Accountants for Now: With only 3 exploratory roles in accountant partnerships, Crestline is at least 18–24 months away from fielding a competitive partner network. They are betting on direct-to-SMB product-led growth.

---

4. What We Should Do: Strategic Recommendations

1. Reject the "Free Plan for Everyone" Launching a universal free tier would instantly cannibalize up to $159,600 in monthly solo revenue while funding free bank-feed and support costs—playing directly into Crestline’s $120M balance sheet advantage. Worse, it would invite our 2–5 user accounts ($254.8k MRR) to downgrade.

2. Double Down on the Accounting Network (Our True Moat) Since firm-connected accounts have near-zero defection to Crestline (4 out of 4,333), increasing network attachment is our highest-ROI defense: * Drive 2–5 User Attachment from 46% to 70%+: Launch an in-app "Match with an Advisor" onboarding flow and co-marketing subsidies with our 2,100 firms to connect our 2,646 unconnected 2–5 user accounts. * Use a Targeted "Firm-Sponsored" Starter Tier Instead of Public Freemium: Allow our 2,100 partner firms to provision a free or $5/month "Ledger-Only" seat exclusively for their solo clients. This neutralizes Crestline’s price advantage only when an account is locked into our partner moat, turning our 2,100 firms into an aggressive acquisition channel.

3. Stop Multi-User Downgrades by Strengthening Collaboration We lost 40 multi-user accounts ($3,288 MRR—62% of our MRR lost to Crestline) to a single-user free plan because those customers weren't getting multi-user value. We need to audit multi-user accounts that only have 1 active login, proactively right-size them or drive adoption of multi-user workflows (role-based expense approvals, team receipt capture) that a 1-user free tier cannot support.

4. Preempt Crestline on Payroll Crestline’s 14 payroll engineering hires show how they plan to monetize SMBs as they grow from 1 to 2–5 users. Rather than spending 12 months building native payroll from scratch, we should immediately integrate deeply with (or white-label) a best-in-class payroll provider—specifically building accountant-friendly payroll reconciliation tools inside our partner portal before Crestline’s payroll product launches.

Grades and run details

Decision model 64 · LLM judge 67
Decision model checks
  • failUses the supplied evidence correctly46%
  • failAddresses the actual decision48%
  • passRespects explicit constraints33%
  • partialIdentifies material uncertainty31%
  • partialAvoids unsupported claims27%
  • passProduces the required deliverable76%
  • passFinds who's actually leaving83%
  • passReads where Crestline is heading100%
  • passCompetes with what customers really choose94%
  • failGets the base of every number right52%
  • passSeparates what a rival can copy100%
Run
Run
#1
API response time
22 s
Submitted
13 Oct 2026

Needs more than a quick edit9 of 11 checks passed

The analysis lacks the MRR lost to Crestline set against the base and does not propose proactive moves on payroll/multi-entity, requiring rework.

Output

Board view: defend the exposed segment; don’t make Harbourline free for everyone

Recommendation: Do not launch a universal free plan to match Crestline. The evidence points to a real and concentrated threat—solo businesses without an accounting-firm connection—not an existential threat to Harbourline’s full customer base. Defend that segment with a tightly bounded offer and stronger upgrade path, while reinforcing the product and partner workflows that make larger accounts stick.

Where Crestline can hurt us—and where it can’t

The clearest exposure is solo businesses outside the partner network. Harbourline has 11,400 one-user accounts; only 9% are connected to an accounting firm. That leaves roughly 10,400 solo accounts with fewer partner-based switching costs. This segment also had 31% annual churn before Crestline launched, versus 14% for 2–5-user accounts and 6% for 6–20-user accounts. Its $14 average monthly revenue is meaningful at scale, but the underlying retention problem predates Crestline.

The cancellation file shows a strong competitive signal: from August through September, 187 of 250 cancellations (75%) named Crestline’s free tier as their destination. Another 37 went to spreadsheets or no tool, and 26 were attributed to business closure. Total cancellations were about $8,100 MRR. The two-month total of 250 compares with roughly 190 accounts at the pre-launch pace of 95 per month—a possible increase of 60, although two months is too little to attribute that excess to Crestline. The destination data is more direct evidence of competitive switching than the aggregate increase, but neither proves every Crestline switch was caused by the free offer.

Crestline’s free plan is explicitly for one user, with invoicing, expenses and bank feeds—the basic needs of many of our solo customers. Its “free forever” positioning can also reset expectations about what bookkeeping software should cost. Harbourline’s $14 solo plan is below Crestline’s $25 paid-plan starting price, but that is little protection against a $0 alternative.

The threat is less established in our core, partner-connected accounts. Harbourline’s 2–5-user customers are 46% firm-connected; 6–20-user accounts are 81% connected. These cohorts have substantially lower pre-Crestline churn. Our 2,100-firm network is not just a referral channel: firms reconcile, file and advise using customers’ books inside Harbourline. That embedded working relationship creates workflow and switching costs beyond the software itself.

There is a cautionary signal: the cancellation file records 187 Crestline-bound accounts, including 42 with 2–5 or 6–20 users and four connected to an accounting firm. Yet Crestline’s public free plan is limited to one user. Those entries do not establish that Crestline’s free product supports larger teams; account size may have changed, or destination coding may be inaccurate. We should validate these cases before treating them as proof that the free tier is winning larger accounts. For now, they are a reason to investigate—not evidence that our partner advantage has failed.

What Crestline can copy—and what it can’t quickly reproduce

Copyable: Crestline can offer a free price point, build or improve core bookkeeping features, and spend to acquire solo businesses. Its $120M raise gives it room to sustain that push. Payroll hiring and a mid-market multi-entity team could broaden its product over time.

Harder to copy quickly: Harbourline’s 2,100-firm network and the operational habits built around it. Firms already work inside Harbourline to reconcile, file and advise. A competitor can recruit firms, but reproducing that distribution, trust and embedded workflow takes time. This is a durable advantage, not an untouchable one: Crestline has three open roles for “accountant partnerships (exploring),” so it is signaling interest in closing the gap.

Where Crestline appears to be heading

Crestline is not signaling a solo-only ambition. Its public hiring plans include 14 engineers for payroll, six for mid-market multi-entity, and accountant-partnership exploration. A plausible path is free solo bookkeeping as an acquisition wedge, followed by paid products and expansion into more complex businesses. That direction is a strategic risk even if the current free plan is narrow.

What Harbourline should do

  1. Do not give every account a free plan. A blanket move would put revenue at risk across customers who have not shown the same vulnerability, including partner-connected and larger accounts. It would also answer Crestline’s price move without matching its stated one-user scope.
  2. Test a targeted solo defense. Pilot a tightly limited free or low-cost offer for new, unconnected, one-user businesses, with clear paid triggers for added users, advanced workflows or partner services. Measure conversion, retention, support cost and cannibalization against a holdout before scaling. Avoid migrating existing paying customers to free by default.
  3. Strengthen the advantages Crestline must build. Improve firm onboarding and collaboration, make accountant-led filing and advice more valuable, and reinforce multi-user workflows. These matter most in the segments where our retention and partner connection are strongest.
  4. Instrument the threat. Validate the reported multi-user Crestline switches, track churn and win/loss by firm connection and account size, and watch Crestline’s payroll, multi-entity and partnership launches. Revisit the offer if verified losses spread into connected or larger cohorts.

Bottom line: Crestline has made the solo, unconnected segment more vulnerable; the evidence does not justify making Harbourline free for everyone. Respond precisely where the competitive pressure is visible, and invest in the partner-led proposition that Crestline has not yet replicated.

Grades and run details

Decision model 91 · LLM judge 75
Decision model checks
  • passUses the supplied evidence correctly30%
  • passAddresses the actual decision98%
  • passRespects explicit constraints59%
  • passIdentifies material uncertainty94%
  • passAvoids unsupported claims56%
  • passProduces the required deliverable61%
  • partialFinds who's actually leaving88%
  • passReads where Crestline is heading100%
  • partialCompetes with what customers really choose38%
  • passGets the base of every number right27%
  • passSeparates what a rival can copy100%
Run
Run
#1
API response time
36 s
Submitted
5 Oct 2026

Needs more than a quick edit4 of 11 checks passed

The analysis contains multiple factual errors in the cancellation counts and an invented partner-firm number, so a PM would need to re-check all evidence and redo the reasoning.

Output

MEMORANDUM

TO: Board Chair FROM: VP of Product, Harbourline DATE: October 2, 2023 SUBJECT: Competitive Analysis & Strategic Response to Crestline’s Free Plan Launch

---

Executive Summary

You asked whether Crestline’s new free plan is an existential threat and whether we should launch a matching free plan for everyone.

The short answer: Crestline’s free tier is an acute hazard for our solo business segment, but it is not an existential threat to Harbourline as a whole. Launching a free plan for everyone would be a strategic error that burns revenue without neutralizing Crestline’s core advantages. Instead, we must defend our high-value accounts via our proprietary partner network, accept natural churn in the solo tier, and accelerate feature development in multi-user segments where Crestline cannot follow today.

---

1. Where Crestline Can Hurt Us (and Where It Can’t)

To understand the damage, we analyzed every account cancellation since Crestline launched their free tier on August 15 through the end of September.

  • The Damage: Total cancellations spiked to 250 accounts and ~$8,100 in MRR over August and September, compared to our historical baseline of ~95 accounts per month.
  • The Culprit: Of the 250 cancellations, 187 accounts (74.8%) explicitly migrated to Crestline’s free tier. (The remaining 63 accounts cited closed businesses or reverted to spreadsheets).

#### Where they are hurting us: Solo Businesses (1-User Accounts) Crestline’s free tier is aimed squarely at solo operators (invoicing, expenses, bank feeds for 1 user). * The Evidence: Of the 187 accounts that defected to Crestline, 164 were 1-user accounts. * Historically, our 1-user segment had an annualized churn rate of 31%—already our highest. Crestline has weaponized price against this price-sensitive, feature-light cohort. Because solo businesses rarely collaborate with an external CPA inside software (only 9% of our 1-user accounts are connected to an accounting firm), they have zero switching costs to a free tool.

#### Where they CANNOT hurt us: Multi-User & Accountant-Connected Accounts Crestline cannot effectively penetrate our core revenue engines: 1. The Partner Network Moat: Crestline has no accounting-firm partner programme. Accounts connected to our 2,100-firm partner network experience near-zero churn because Harbourline is embedded in their CPA's workflow. * The Evidence: Out of 250 cancellations in August/September, only 3 accounts connected to an accounting firm left for Crestline. 2. Multi-User Collaboration: Crestline’s free tier is strictly capped at 1 user. Businesses with 2–5 users ($52/mo average revenue) and 6–20 users ($170/mo average revenue) require team access, role-based permissions, and multi-entity tracking—none of which are supported on Crestline’s solo free tier. * While a small handful of 2–5 and 6–20 user accounts downgraded or split into solo accounts to use Crestline, the vast majority of our multi-user base is completely insulated.

---

2. What Advantages Crestline Can and Cannot Copy

With $120M in fresh capital, Crestline is executing a classic land-grab play. We must be clear-eyed about what they can replicate and what remains defensible.

#### What Crestline CAN copy: * Feature Parity for Solo Tools: Basic invoicing, expense tracking, and bank feeds are commoditized. Crestline already matches our core solo functionality. * Aggressive Go-To-Market Spend: Backed by $120M, they can outspend us on top-of-funnel acquisition for solo founders indefinitely.

#### What Crestline CANNOT copy (Our Defensible Moats): * The Accounting Firm Ecosystem (The Partner Network): Accounting practices do not adopt software one-off; they standardize their practice around shared multi-client dashboards. Crestline currently has 3 open roles exploring accountant partnerships, but building a trusted network of 2,100 firms takes years of relationship-building, workflow integration, and reputation. They cannot buy this overnight. * Embedded Collaborative Bookkeeping: For 2-to-20 user businesses, accounting is a team sport involving internal bookkeepers, business owners, and external CPAs. Harbourline’s architecture bridges this operational triad. Crestline’s solo-first roots make re-architecting for mid-market, multi-entity collaboration a heavy, multi-year engineering lift.

---

3. Where Crestline is Heading

Public job openings and product signals from September reveal Crestline’s roadmap: 1. Moving Upmarket via Payroll & Multi-Entity: They have 14 engineers dedicated to payroll and 6 building "mid-market multi-entity" features. They are using the free solo tier as a top-of-funnel trap to acquire brand awareness, intending to upsell those solo businesses if and when they grow into complex, multi-user companies. 2. The Accountant Play: Their 3 open roles for "accountant partnerships" prove they recognize their blind spot. However, going from zero to a trusted partner network is their steepest hill.

---

4. Strategic Recommendations: Should We Launch a Free Plan?

Recommendation: Do NOT launch a free plan for everyone.

Matching Crestline’s free tier across our entire customer base would be catastrophic: * Revenue Cannibalization: Harbourline generates strong revenue across 4,900 mid-sized (2-5 user) accounts and 1,300 larger (6-20 user) accounts. Giving away our product to teams that are already willing and able to pay $52 to $170/month would instantly vaporize millions in ARR. * Support Cost Bloat: Free tiers attract low-intent, high-maintenance users who strain customer support without contributing to top-line growth.

Instead, we should execute a targeted, three-part defense:

#### 1. Let Go of Unprofitable Solo Churn Accept that solo businesses with no accounting-firm connection are a low-margin acquisition battleground. Do not burn capital trying to win back price-sensitive solo operators who value $0 over workflow integration. Our blended ARPU and LTV are protected by focusing upstream.

#### 2. Deepen the Partner Network Moat (Offensive Defense) Our 2,100 accounting firm partners are our ultimate shield. We should: * Introduce incentives for accounting partners to migrate their solo clients onto Harbourline (bringing them into the network, where churn drops to near zero). * Build exclusive collaboration features between firms and multi-user clients that Crestline’s solo tool cannot touch.

#### 3. Accelerate Multi-User & Payroll Roadmap Crestline is moving upmarket; we must fortify our position there first. * Double down on team permissions, audit trails, and multi-entity reporting for our 2–5 and 6–20 user cohorts. * Ensure our payroll and expense workflows are robust enough that growing businesses have zero incentive to graduate to Crestline, but rather graduate deeper into Harbourline.

Conclusion

Crestline’s free tier is an expensive customer-acquisition engine designed to capture the lowest tier of the market. By holding our pricing power, protecting our accounting partner network, and doubling down on multi-user collaboration, we will cede low-value churn while retaining the high-value businesses that drive our enterprise value.

Claims the judge couldn’t find in the brief

  • Of the 250 cancellations, 187 accounts (74.8%) explicitly migrated to Crestline’s free tier.
  • Of the 187 accounts that defected to Crestline, 164 were 1-user accounts.
  • only 3 accounts connected to an accounting firm left for Crestline.
  • 2,100-firm partner network
  • Accounts connected to our 2,100-firm partner network experience near-zero churn because Harbourline is embedded in their CPA's workflow.

Grades and run details

Decision model 59 · LLM judge 33
Decision model checks
  • failUses the supplied evidence correctly20%
  • failAddresses the actual decision53%
  • passRespects explicit constraints51%
  • partialIdentifies material uncertainty49%
  • partialAvoids unsupported claims28%
  • passProduces the required deliverable79%
  • partialFinds who's actually leaving55%
  • passReads where Crestline is heading100%
  • partialCompetes with what customers really choose35%
  • partialGets the base of every number right4%
  • passSeparates what a rival can copy100%
Run
Run
#1
Time to output
9 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 7

Uses the supplied evidence correctlyMixedRightWrong
Gemini 4 Argon · API

All key numbers and facts are taken correctly from the supplied context or derived by arithmetic from it.

GPT-6 Luna · API

All factual claims are directly from the supplied context or derived correctly.

Gemini 3.5 Flash-Lite · Gemini

Multiple factual errors: Crestline defectors count is 186 not 187, 1-user defectors 142 not 164, connected-account defectors 4 not 3, and 2,100 firms is invented.

Addresses the actual decisionWrongRightWrong
Gemini 4 Argon · API

The output does not state what result or condition would change the recommendation, as required.

GPT-6 Luna · API

Commits to not launching a universal free plan and states that verified losses spreading to connected/larger accounts would change the call.

Gemini 3.5 Flash-Lite · Gemini

The output commits to a clear answer (do not launch a free plan) but does not state what result or condition would change that call.

Identifies material uncertaintyWrongRightWrong
Gemini 4 Argon · API

It does not name specific unknowns that could change the decision or say how they would be resolved.

GPT-6 Luna · API

Identifies that the two-month data may not fully attribute to Crestline, and that multi-user switches need validation; says what would change the call.

Gemini 3.5 Flash-Lite · Gemini

The output does not name specific unknowns that could change the decision or say how they would be resolved.

Avoids unsupported claimsWrongRightWrong
Gemini 4 Argon · API

It presents the interpretation that a large majority of Crestline defections are 'substituted churn' as fact, without labelling it as hypothesis.

GPT-6 Luna · API

Interpretations and forecasts are labelled as plausible or uncertain, not presented as fact.

Gemini 3.5 Flash-Lite · Gemini

Interpretations like 'expensive customer-acquisition engine' and 'heavy, multi-year engineering lift' are presented as fact without hedging.

Finds who's actually leavingRightWrongWrong
Gemini 4 Argon · API

It counts cancellations by size and partner connection, sizes MRR, and sets losses against the base, showing the pattern.

GPT-6 Luna · API

Does not size the MRR lost specifically to Crestline nor set it against the base of 1-user account revenue, only gives total cancellations MRR.

Gemini 3.5 Flash-Lite · Gemini

The analysis miscounts the cancellations: 1-user defectors are 142, not 164, and connected-account defectors are 4, not 3, so the sizing is inaccurate.

Competes with what customers really chooseRightWrongWrong
Gemini 4 Argon · API

It mentions spreadsheets as destination and draws an implication from Crestline's investment direction (payroll) for its own response.

GPT-6 Luna · API

Mentions spreadsheets as a competitor but does not draw an implication from Crestline's investment direction that affects the recommendation (e.g., building payroll/multi-entity), only suggests watching.

Gemini 3.5 Flash-Lite · Gemini

The output mentions spreadsheets/no tool as a destination but does not analyze the status quo as a competitor or draw a direction-of-travel insight from it.

Gets the base of every number rightMixedRightWrong
Gemini 4 Argon · API

All derived figures check out against the supplied data; the approximate $305k connected MRR is 48% of total MRR and consistent.

GPT-6 Luna · API

All derived percentages and differences are correctly computed with clear bases.

Gemini 3.5 Flash-Lite · Gemini

Percentages and counts are wrong: 74.8% should be 74.4%, 164 should be 142, 3 should be 4, and the 2,100 firms base is unsupported.

All got right 4

Respects explicit constraintsRightRightRight
Gemini 4 Argon · API

The output is addressed to the board, is under 1,200 words, and does not propose a free plan for every account.

GPT-6 Luna · API

The output is a board analysis under 1,200 words, respects the reader and form.

Gemini 3.5 Flash-Lite · Gemini

The output is a memo to the board chair, within the 1,200-word limit, and addresses the requested competitive analysis.

Produces the required deliverableRightRightRight
Gemini 4 Argon · API

The competitive analysis is present, in memo form for the board, within word limit, and usable with edits.

GPT-6 Luna · API

The board view is complete, within length, and actionable with light edits.

Gemini 3.5 Flash-Lite · Gemini

The memo is complete, in the right form for the board chair, within length, and could be acted on with light edits.

Reads where Crestline is headingRightRightRight
Gemini 4 Argon · API

It reads Crestline's job ads to signal a move into payroll, mid-market, and possible accountant partnerships, and flags the threat.

GPT-6 Luna · API

Uses Crestline's hiring to infer direction toward payroll, mid-market, and partnerships, and flags the partner-programme risk.

Gemini 3.5 Flash-Lite · Gemini

The output correctly reads Crestline's hiring as moving toward payroll, mid-market multi-entity, and exploring accountant partnerships, and flags the partner-programme risk.

Separates what a rival can copyRightRightRight
Gemini 4 Argon · API

It clearly separates copyable advantages (features, price) from durable ones (partner network, switching costs) and anchors response on the latter.

GPT-6 Luna · API

Separates copyable price/features from durable partner network and anchors response in the latter.

Gemini 3.5 Flash-Lite · Gemini

The output separates copyable advantages (features, price, spend) from durable ones (partner network, embedded collaboration) and anchors the response in the latter.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1GPT-6.1 SolwithAPI93.295.82None
2Opus 5.5withClaude88.6100.02None
3GPT-6 AstrawithChatGPT86.495.82None
4GPT-6 LunawithAPI93.283.32None
5Sonnet 5.5withAPI86.483.32None
6Haiku 5.5withAPI81.879.22None
7Gemini 3.8 FlashwithAPI68.254.22None
8Gemini 4 ArgonwithAPI65.954.22None
9Gemini 3.5 Flash-LitewithGemini54.525.02None

What you get for the cost

No setup scores within 5 points of the leader at half its cost or less here, so the leader is also the best value.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreGPT-6.1 Sol · API: task score 95, 14× the cheapestOpus 5.5 · Claude: task score 95, 75× the cheapestGPT-6 Astra · ChatGPT: task score 92, 87× the cheapestGPT-6 Luna · API: task score 86, the cheapestSonnet 5.5 · API: task score 83, 44× the cheapestHaiku 5.5 · API: task score 79, 3.0× the cheapestGemini 3.8 Flash · API: task score 60, 12× the cheapestGemini 4 Argon · API: task score 57, 23× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 38, 2.9× the cheapestGPT-6.1 Sol · APIOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTGPT-6 Luna · APISonnet 5.5 · APIHaiku 5.5 · APIGemini 3.8 Flash · APIGemini 4 Argon · APIGemini 3.5 Flash-Lite · Gemini
Nobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API861.0×
Gemini 3.5 Flash-Lite · Gemini382.9×
Haiku 5.5 · API793.0×
Gemini 3.8 Flash · API6012×
GPT-6.1 Sol · API9514×
Gemini 4 Argon · API5723×
Sonnet 5.5 · API8344×
Opus 5.5 · Claude9575×
GPT-6 Astra · ChatGPT9287×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Deciding how to respond to competitors.

Why it matters

Feature grids hide the real competition, which is often the customer's current workaround, and the response that feels urgent is usually to copy everything.

What good looks like

  • Includes the status quo and non-obvious alternatives
  • Finds where losses really go, by segment
  • Separates advantages a rival can copy from ones it can't
  • Reads where competitors are heading

Deliberately not measured

  • Market sizing
Capability tested

Competitive insight

The failure we’re looking for

A feature checklist and a plan to match everything

Grading

Decision model and LLM judge, calibrated against a blind PM review