Tasks / Strategy

Competitor analysis

Can the model find where a competitor can actually hurt you, from win/loss and churn data, rather than list features?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 63% were usable with at most a quick edit.

Reliably right

  1. Respects explicit constraints100% pass
    Delivers a competitive analysis memo to Dan within 900 words, addressing all requested topics without proposing a full feature-by-feature match.
    Opus 5.5 · Claude · The competitor isn't who you think
  2. Reads where Crestline is heading100% pass
    Uses Crestline's hiring (payroll, multi-entity, accountant partnerships) to infer direction, flags the partner-programme risk, and adjusts the response accordingly.
    GPT-6 Astra · ChatGPT · Free forever, funded
  3. Produces the required deliverable97% pass
    The memo is complete, in the right form for the CEO, within the word limit, and actionable with light edits.
    Opus 5.5 · Claude · The competitor isn't who you think

Where it slips

  1. Uses the supplied evidence correctly56% pass
    Two factual claims are wrong: the number of unconnected multi-user accounts going to Crestline is 40, not 42, and the number of connected 6-20 cancellations is 5, not 4.
    Sonnet 5.5 · API · Free forever, funded
  2. Gets the base of every number right61% pass
    Derived figures for unconnected multi-user cancellations (42 vs actual 40) and connected 6-20 cancellations (4 vs actual 5) are incorrect.
    Sonnet 5.5 · API · Free forever, funded
  3. Separates what a rival can copy66% pass
    Does not explicitly separate advantages a well-funded competitor could copy from those it couldn't, nor anchor the response in durable advantages with evidence.
    GPT-6 Luna · API · The competitor isn't who you think

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Competes with what customers really choose

    Does the analysis treat the status quo (spreadsheets, manual work, doing nothing) as a competitor, and draw at least one implication from where competitors are investing rather than only from today's features?

    Passes when Analyses the status quo, strengths included, and gives a direction-of-travel insight that affects the recommendation.

  2. Separates what a rival can copy

    Does the analysis separate advantages a well-funded competitor could copy (features, price, being first) from ones it couldn't easily (switching costs, networks, relationships), and build the response on the latter?

    Passes when Names which advantages are copyable and which aren't, with evidence, and anchors the response in the durable ones.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the VP Product at Harbourline. Crestline raised $120M in August and launched a free plan for solo businesses. Our board chair wants to know whether we should launch a free plan for everyone to match it (the note is below). Write the board a competitive analysis of no more than 1,200 words: where Crestline can hurt us and where it can't, with the evidence; which of our advantages it could copy and which it couldn't; where it's heading; and what we should do. The files are attached.

What the model was given6 items: About Harbourline, board_chair_note.md, customer_base.md, cancellations_aug_sep.csv (every account that cancelled since Crestline's launch), cancellation_summary.md, crestline_public_info.md
About HarbourlineBookkeeping, invoicing and expenses for small businesses, with a partner network that connects customers to their accounting firm.
board_chair_note.md1 lines · Download
From the board chair: "Crestline's free tier is an existential threat. Should we launch a free plan for everyone to match it? We need a view on where they can hurt us and where they can't."
customer_base.md9 lines · Download
# Harbourline customers (end of September)

| Account size | Paying accounts | Share connected to an accounting firm | Average revenue per month | Annual churn before Crestline's free tier |
|---|---|---|---|---|
| 1 user | 11,400 | 9% | $14 | 31% |
| 2-5 users | 4,900 | 46% | $52 | 14% |
| 6-20 users | 1,300 | 81% | $170 | 6% |
…
cancellations_aug_sep.csv (every account that cancelled since Crestline's launch)cancelled_account,account_users,connected_to_accounting_firm,mrr_usd,went_to H-1200,1,no,14,Crestline (free tier) H-1201,1,no,10,Crestline (free tier) H-1202,2-5,no,58,Crestline (free tier) H-1203,1,no,11,Crestline (free tier) H-1204,1,no,17,Crestline (free tier) H-1205,1,no,15,Spreadsheets / no tool H-1206,1,no,15,Spreadsheets / no tool H-1207,1,no,9,Crestline (free tier) H-1208,2-5,no,66,Crestline (free tier) H-1209,2-5,no,44,Crestline (free tier) H-1210,1,no,11,Crestline (free tier) H-1211,1,no,9,Crestline (free tier) H-1212,1,no,12,Crestline (free tier) H-1213,1,no,16,Crestline (free tier) H-1214,2-5,yes,73,Crestline (free tier) H-1215,1,no,9,Crestline (free tier) H-1216,2-5,no,56,Crestline (free tier) H-1217,1,no,14,Crestline (free tier) H-1218,2-5,no,47,Business closed H-1219,1,no,16,Crestline (free tier) H-1220,6-20,no,177,Crestline (free tier) H-1221,2-5,yes,51,Crestline (free tier) H-1222,2-5,no,54,Business closed H-1223,6-20,no,168,Crestline (free tier) H-1224,1,no,18,Crestline (free tier) H-1225,1,no,13,Spreadsheets / no tool H-1226,1,no,11,Crestline (free tier) H-1227,1,no,15,Crestline (free tier) H-1228,1,no,12,Crestline (free tier) H-1229,1,no,13,Crestline (free tier) H-1230,2-5,no,51,Business closed H-1231,1,no,18,Crestline (free tier) H-1232,1,no,16,Spreadsheets / no tool H-1233,1,no,12,Crestline (free tier) H-1234,1,no,16,Spreadsheets / no tool H-1235,1,no,12,Spreadsheets / no tool H-1236,1,no,13,Crestline (free tier) H-1237,1,no,17,Spreadsheets / no tool H-1238,6-20,yes,188,Business closed H-1239,1,no,17,Crestline (free tier) H-1240,1,no,15,Crestline (free tier) H-1241,6-20,yes,238,Crestline (free tier) H-1242,1,no,11,Spreadsheets / no tool H-1243,2-5,no,66,Crestline (free tier) H-1244,2-5,no,63,Business closed H-1245,6-20,no,199,Crestline (free tier) H-1246,1,no,18,Crestline (free tier) H-1247,1,no,11,Crestline (free tier) H-1248,1,no,15,Crestline (free tier) H-1249,2-5,no,60,Crestline (free tier) H-1250,2-5,no,66,Crestline (free tier) H-1251,2-5,no,34,Crestline (free tier) H-1252,1,no,12,Crestline (free tier) H-1253,1,no,18,Crestline (free tier) H-1254,1,no,18,Spreadsheets / no tool H-1255,2-5,no,31,Crestline (free tier) H-1256,1,no,17,Crestline (free tier) H-1257,1,no,16,Crestline (free tier) H-1258,1,no,18,Spreadsheets / no tool H-1259,1,no,9,Crestline (free tier) H-1260,1,no,15,Crestline (free tier) H-1261,1,no,18,Crestline (free tier) H-1262,6-20,yes,170,Business closed H-1263,1,no,18,Crestline (free tier) H-1264,2-5,no,41,Business closed H-1265,1,no,14,Spreadsheets / no tool H-1266,1,no,14,Crestline (free tier) H-1267,1,no,12,Crestline (free tier) H-1268,6-20,no,112,Crestline (free tier) H-1269,1,no,9,Crestline (free tier) H-1270,2-5,no,71,Business closed H-1271,1,no,12,Crestline (free tier) H-1272,1,no,11,Crestline (free tier) H-1273,1,no,15,Crestline (free tier) H-1274,1,no,9,Crestline (free tier) H-1275,6-20,no,209,Crestline (free tier) H-1276,2-5,no,36,Crestline (free tier) H-1277,1,no,13,Crestline (free tier) H-1278,1,no,11,Crestline (free tier) H-1279,1,no,15,Crestline (free tier) H-1280,1,no,18,Spreadsheets / no tool H-1281,1,no,16,Crestline (free tier) H-1282,1,no,17,Crestline (free tier) H-1283,1,no,9,Crestline (free tier) H-1284,1,no,10,Crestline (free tier) H-1285,6-20,yes,128,Business closed H-1286,1,no,17,Crestline (free tier) H-1287,1,no,12,Spreadsheets / no tool H-1288,2-5,no,39,Crestline (free tier) H-1289,1,no,14,Crestline (free tier) H-1290,1,no,12,Crestline (free tier) H-1291,1,no,16,Crestline (free tier) H-1292,2-5,no,62,Crestline (free tier) H-1293,1,no,12,Crestline (free tier) H-1294,1,no,12,Crestline (free tier) H-1295,1,no,9,Spreadsheets / no tool H-1296,1,no,11,Crestline (free tier) H-1297,1,no,16,Crestline (free tier) H-1298,1,no,17,Crestline (free tier) H-1299,1,no,13,Spreadsheets / no tool H-1300,1,no,13,Crestline (free tier) H-1301,1,no,17,Crestline (free tier) H-1302,1,no,12,Crestline (free tier) H-1303,1,no,17,Crestline (free tier) H-1304,1,no,14,Crestline (free tier) H-1305,1,no,9,Crestline (free tier) H-1306,1,no,15,Crestline (free tier) H-1307,1,no,16,Crestline (free tier) H-1308,1,no,16,Crestline (free tier) H-1309,1,no,12,Crestline (free tier) H-1310,2-5,no,47,Business closed H-1311,2-5,no,78,Crestline (free tier) H-1312,1,no,11,Crestline (free tier) H-1313,1,no,14,Crestline (free tier) H-1314,1,no,15,Spreadsheets / no tool H-1315,1,no,14,Spreadsheets / no tool H-1316,1,no,9,Crestline (free tier) H-1317,2-5,no,52,Business closed H-1318,1,no,13,Crestline (free tier) H-1319,2-5,no,43,Business closed H-1320,1,no,11,Spreadsheets / no tool H-1321,1,no,15,Crestline (free tier) H-1322,2-5,no,55,Business closed H-1323,1,no,12,Spreadsheets / no tool H-1324,6-20,no,166,Crestline (free tier) H-1325,2-5,no,67,Business closed H-1326,1,no,16,Crestline (free tier) H-1327,1,no,15,Crestline (free tier) H-1328,1,no,13,Crestline (free tier) H-1329,1,no,17,Crestline (free tier) H-1330,1,no,12,Crestline (free tier) H-1331,1,no,13,Crestline (free tier) H-1332,2-5,no,68,Crestline (free tier) H-1333,1,no,14,Spreadsheets / no tool H-1334,1,no,18,Crestline (free tier) H-1335,1,no,9,Crestline (free tier) H-1336,1,no,14,Crestline (free tier) H-1337,1,no,13,Crestline (free tier) H-1338,1,no,10,Crestline (free tier) H-1339,1,no,18,Spreadsheets / no tool H-1340,2-5,no,70,Crestline (free tier) H-1341,1,no,13,Spreadsheets / no tool H-1342,2-5,no,34,Crestline (free tier) H-1343,1,no,11,Crestline (free tier) H-1344,1,no,17,Spreadsheets / no tool H-1345,6-20,no,159,Crestline (free tier) H-1346,6-20,no,214,Crestline (free tier) H-1347,2-5,no,67,Crestline (free tier) H-1348,2-5,no,40,Crestline (free tier) H-1349,1,no,18,Crestline (free tier) H-1350,1,no,17,Crestline (free tier) H-1351,1,no,11,Crestline (free tier) H-1352,1,no,17,Crestline (free tier) H-1353,2-5,no,48,Business closed H-1354,1,no,12,Crestline (free tier) H-1355,1,no,16,Crestline (free tier) H-1356,1,no,14,Crestline (free tier) H-1357,1,no,18,Crestline (free tier) H-1358,1,no,11,Crestline (free tier) H-1359,1,no,17,Spreadsheets / no tool H-1360,1,no,12,Spreadsheets / no tool H-1361,2-5,no,39,Business closed H-1362,1,no,18,Crestline (free tier) H-1363,2-5,no,39,Crestline (free tier) H-1364,2-5,no,41,Business closed H-1365,2-5,no,51,Crestline (free tier) H-1366,1,no,11,Crestline (free tier) H-1367,2-5,no,76,Business closed H-1368,1,no,11,Crestline (free tier) H-1369,1,no,18,Crestline (free tier) H-1370,2-5,no,37,Business closed H-1371,1,no,9,Crestline (free tier) H-1372,1,no,18,Crestline (free tier) H-1373,2-5,no,36,Crestline (free tier) H-1374,2-5,no,77,Crestline (free tier) H-1375,2-5,no,64,Crestline (free tier) H-1376,1,no,9,Crestline (free tier) H-1377,2-5,no,71,Crestline (free tier) H-1378,1,no,10,Crestline (free tier) H-1379,1,no,12,Spreadsheets / no tool H-1380,6-20,yes,235,Business closed H-1381,1,no,9,Crestline (free tier) H-1382,1,no,15,Spreadsheets / no tool H-1383,2-5,yes,65,Crestline (free tier) H-1384,1,no,16,Crestline (free tier) H-1385,1,no,17,Spreadsheets / no tool H-1386,1,no,13,Spreadsheets / no tool H-1387,2-5,no,66,Business closed H-1388,1,no,17,Crestline (free tier) H-1389,1,no,18,Crestline (free tier) H-1390,1,no,16,Crestline (free tier) H-1391,1,no,18,Spreadsheets / no tool H-1392,1,no,15,Crestline (free tier) H-1393,1,no,10,Crestline (free tier) H-1394,1,no,11,Crestline (free tier) H-1395,2-5,no,46,Business closed H-1396,1,no,16,Crestline (free tier) H-1397,1,no,15,Crestline (free tier) H-1398,2-5,no,34,Crestline (free tier) H-1399,1,no,13,Crestline (free tier) H-1400,1,no,13,Crestline (free tier) H-1401,1,no,10,Crestline (free tier) H-1402,1,no,11,Crestline (free tier) H-1403,2-5,no,49,Crestline (free tier) H-1404,2-5,no,58,Crestline (free tier) H-1405,1,no,17,Crestline (free tier) H-1406,1,no,17,Crestline (free tier) H-1407,1,no,12,Crestline (free tier) H-1408,2-5,no,37,Crestline (free tier) H-1409,1,no,15,Spreadsheets / no tool H-1410,2-5,no,31,Crestline (free tier) H-1411,1,no,13,Spreadsheets / no tool H-1412,1,no,15,Crestline (free tier) H-1413,1,no,13,Crestline (free tier) H-1414,1,no,15,Crestline (free tier) H-1415,1,no,18,Crestline (free tier) H-1416,1,no,14,Crestline (free tier) H-1417,1,no,11,Crestline (free tier) H-1418,6-20,no,131,Crestline (free tier) H-1419,1,no,17,Crestline (free tier) H-1420,1,no,18,Crestline (free tier) H-1421,1,no,16,Crestline (free tier) H-1422,1,no,11,Crestline (free tier) H-1423,2-5,no,78,Business closed H-1424,1,no,16,Spreadsheets / no tool H-1425,1,no,18,Spreadsheets / no tool H-1426,1,no,18,Crestline (free tier) H-1427,1,no,15,Crestline (free tier) H-1428,1,no,14,Spreadsheets / no tool H-1429,1,no,14,Spreadsheets / no tool H-1430,2-5,no,63,Business closed H-1431,1,no,18,Crestline (free tier) H-1432,1,no,15,Crestline (free tier) H-1433,1,no,17,Spreadsheets / no tool H-1434,1,no,14,Crestline (free tier) H-1435,1,no,14,Crestline (free tier) H-1436,1,no,12,Crestline (free tier) H-1437,1,no,11,Crestline (free tier) H-1438,1,no,13,Crestline (free tier) H-1439,1,no,13,Spreadsheets / no tool H-1440,2-5,no,54,Business closed H-1441,1,no,14,Crestline (free tier) H-1442,1,no,16,Crestline (free tier) H-1443,2-5,no,74,Crestline (free tier) H-1444,1,no,14,Crestline (free tier) H-1445,2-5,no,65,Crestline (free tier) H-1446,1,no,18,Crestline (free tier) H-1447,1,no,11,Crestline (free tier) H-1448,1,no,10,Crestline (free tier) H-1449,2-5,no,69,Business closed
cancellation_summary.md1 lines · Download
# Cancellations, August to September: 250 accounts, about $8,100 of MRR. Before Crestline's launch we lost about 95 accounts a month.
crestline_public_info.md6 lines · Download
# Crestline (news and public pages)

- 1 August: raised $120M. 15 August: launched a free plan for 1 user (invoicing, expenses, bank feeds).
- Paid plans from $25/month; no accounting-firm partner programme.
- Open roles (September): 14 engineers for payroll; 6 for "mid-market multi-entity"; 3 for "accountant partnerships (exploring)".
- Pricing page: free plan "free forever for solo businesses".
What a strong answer doesThe answer key the graders mark against

Shows from the cancellations that Crestline is taking single-user accounts not connected to an accounting firm: 186 of the 250 cancellations went to Crestline, 142 of them single-user accounts at $9 to $19 a month, and only 4 of the 186 were connected to an accounting firm; the $5,600 of MRR lost to Crestline is a small share of a base where 1-user accounts already churned at 31% a year. The partner network is the advantage Crestline can't easily copy: 81% of 6-20 user accounts and 46% of 2-5 user accounts are connected to a firm, and they barely leave; Crestline has no partner programme. Free and price are easy to copy, and so is matching features. Reads Crestline's hiring (payroll, mid-market multi-entity, accountant partnerships 'exploring') as heading upmarket and possibly at the partner channel, the real future threat. Recommends against a free plan for everyone: it would give away revenue from 11,400 single-user accounts paying about $14 a month (about $160,000 MRR) to defend against losses that are a fraction of that. Instead: let the least valuable solo accounts go or offer a narrow solo option tied to an accounting firm, deepen the partner network (bring firms' clients onto Harbourline, lock in the firm workflow), and get ahead on payroll and multi-entity where Crestline is heading. Sets the signals that would change the call (losses spreading to connected or multi-user accounts, or Crestline launching a partner programme).

Critical failures (cap the score at 40)
  • Recommends launching a free plan for every account to match Crestline
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Usable with a quick edit9 of 11 checks passed

The analysis is evidence-based, logically structured, and directly answers the board's question; a product manager could use it with minor wording tweaks and no rework of evidence or reasoning.

Output

Board memo: Crestline’s free plan—targeted threat, not a case for universal free

Recommendation: Do not launch a free plan for everyone. Crestline threatens customers buying standalone bookkeeping, including some larger businesses—not just solos. Our strongest protection is the accounting-firm relationship embedded in Harbourline, not account size or a feature lead. We should test a tightly bounded free offer, defend unconnected customers across sizes, and deepen our partner advantage before Crestline builds its own.

1. Where Crestline can hurt us—and where it is not yet winning

The threat is real, but headline cancellations overstate incremental damage

The cancellation file totals 250 accounts and $8,065 in lost MRR. Of these:

  • 186 accounts, representing $5,581 MRR, went to Crestline.
  • 38 returned to spreadsheets/no tool; 26 closed their businesses.
  • Crestline-associated losses equal approximately 0.88% of our current $635,400 MRR.

Against the previous run rate of 95 cancellations monthly, 250 over August–September is about 60 additional cancellations—not 186. Destination tells us where customers went, not whether they would otherwise have stayed. The mid-August launch also complicates the comparison.

This is a meaningful warning, not evidence of an existential threat. Equally, cancellations do not capture prospects we failed to acquire or future pricing pressure.

Accounting-firm connection explains exposure better than size

Calculated from `cancellationsaugsep.csv` and `customer_base.md`:

Account sizeUnconnected exits to Crestline / estimated unconnected baseConnected exits to Crestline / estimated connected baseMRR lost to Crestline
------:---:---:
1 user142 / 10,374 (1.37%)0 / 1,026 (0%)$1,958
2–5 users31 / 2,646 (1.17%)3 / 2,254 (0.13%)$1,850
6–20 users9 / 247 (3.64%)1 / 1,053 (0.09%)$1,773

These are observed-period exit ratios using September-end balances and reported connection shares, not cohort churn rates.

Where Crestline can hurt us:

  • Standalone solos: They account for 142 of 186 competitive exits. Free invoicing, expenses and bank feeds directly challenge a $14/month product. This segment already had 31% annual churn before launch.
  • Unconnected larger businesses: These are not protected simply because they have more users. The highest observed competitive exit ratio is among unconnected 6–20-user accounts, albeit from only nine exits.
  • Revenue beyond solos: Multi-user customers represent only 24% of Crestline exits but 65% of the associated lost MRR. A solo-only defensive strategy would miss most of the revenue leakage.

There is an important inconsistency to investigate: 44 multi-user Harbourline accounts report moving to a one-user free plan. They may have reduced usage, retained other tools, or reported their destination imprecisely. We should verify their actual workflows and Crestline plans before claiming full multi-user substitution. The revenue loss is nevertheless real.

Where Crestline is not yet winning materially:

Only four accounting-firm-connected accounts moved to Crestline, versus 182 unconnected accounts. Relative to estimated bases, that is approximately 0.09% versus 1.37%. Connected customers account for roughly 49% of our MRR.

An accountant reconciles, files and advises inside Harbourline. Replacing us therefore means changing a shared operating workflow, not just choosing cheaper software. Crestline currently has no accounting-firm partner programme. That is strong evidence of protection today—not immunity: four connected customers did leave, and customer selection may explain some of the retention difference.

2. Which advantages can it copy?

AdvantageAssessment
Core bookkeeping, invoicing, expenses and bank-feed functionalityCopyable; partly already matched. These are not a sufficient basis for defending subscription pricing.
Collaboration features and accountant-facing softwareCopyable with investment. The $120M raise gives Crestline resources, although funding alone does not prove execution.
Larger-business functionalityContestable. Its hiring signals an intention to close gaps; size is not a durable barrier.
Our 2,100-firm network and embedded client–accountant workflowsNot quickly reproducible by shipping features. Crestline must recruit firms, earn trust, change working practices and migrate client relationships. It can build an alternative over time, but cannot instantly copy our existing relationships.

Our moat is therefore the working network, not merely the existence of an accountant portal. We should strengthen productive firm–customer connections rather than assume the partner count itself guarantees defensibility. We have no evidence here for additional advantages in brand, data, accuracy or cost.

3. Where Crestline is heading

The public hiring mix is more informative than “free forever” alone:

  • 14 payroll engineers: a substantial signal toward an adjacent recurring workflow.
  • Six mid-market multi-entity engineers: evidence of ambition beyond solos.
  • Three accountant-partnership roles, labelled “exploring”: early recognition of the channel where we are strongest.

The likely strategy is free acquisition at the bottom, paid expansion through greater complexity and adjacent products. Paid plans already start at $25/month. Payroll and multi-entity could create monetisation opportunities while preserving the solo-free promise.

This is a directional inference, not a confirmed roadmap. Hiring does not establish launch dates or product quality. Nevertheless, we should expect a broader competitor, not permanently dismiss Crestline as a free solo tool.

4. What we should do

1. Reject universal free; authorise a controlled test. Our solo segment alone generates approximately $159,600 MRR, versus $1,958 lost to Crestline in the file. Those figures are not a forecast, but show the cannibalisation risk. Test a limited one-user offer for new self-serve prospects and selected at-risk customers, with clear usage boundaries and paid upgrade paths. Use holdouts to measure incremental activation, retained contribution, conversion, support costs and paid-plan downgrades—not free registrations.

2. Defend unconnected customers across all sizes immediately. Interview the multi-user switchers first. Establish whether price, unused seats, workflow simplicity or missing value drove departure. Improve onboarding and retention offers accordingly; do not default to discounts for every customer.

3. Invest primarily in the partner workflow. Make firm connection, shared reconciliation, filing handoffs and multi-client work easier. Help existing firms connect more suitable clients. Measure active shared workflows and cohort retention, not merely partner sign-ups.

4. Review in 90 days against explicit evidence. Track competitive losses and acquisition win rates by size and firm connection, lost MRR, partner recruitment and Crestline’s product releases. Reconcile the historical annual-churn figures with the separate 95-per-month baseline before making causal claims.

Decision requested: Approve a bounded free-plan experiment and partner-led defence—not a universal price reset. Reconsider broader action if connected-customer losses rise materially or controlled tests show that free creates sustainable incremental value.

Grades and run details

Decision model 82 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly69%
  • passAddresses the actual decision91%
  • passRespects explicit constraints42%
  • passIdentifies material uncertainty90%
  • passAvoids unsupported claims40%
  • passProduces the required deliverable76%
  • passFinds who's actually leaving84%
  • passReads where Crestline is heading100%
  • passCompetes with what customers really choose73%
  • failGets the base of every number right59%
  • passSeparates what a rival can copy100%
Run
Run
#1
Time to output
87 s
Submitted
5 Oct 2026

Needs more than a quick edit9 of 11 checks passed

The output contains minor but factual errors in the cancellation counts (unconnected multi-user and connected 6-20) that would need correction before use.

Output

Crestline: where it can hurt us, and what we should do

To: Board | From: VP Product

Recommendation

Don't launch a free plan for everyone. Crestline is taking our solo and unconnected customers, not our firm-connected ones. Accounts with 2-20 users make up 75% of our MRR (about $476k of $635k), and a free-for-all plan would put that revenue at risk to defend the 25% that is solo. Instead we should connect more customers to accounting firms, test a narrow solo response, and lock in firms before Crestline builds a partner programme.

What the cancellations show

Since launch, 250 accounts (about $8,100 MRR) have cancelled:

Went toAccountsMRR
Crestline186 (74%)~$5,580 (69%)
Business closed26 (10%)~$1,930 (24%)
Spreadsheets / no tool38 (15%)~$555 (7%)

The 250 overstates the damage. Normal churn is about 95 accounts a month, so the excess is roughly 60-110 accounts, depending on whether we count from 15 August or 1 August. There is also a data problem: the segment churn rates in our customer table imply about 360 cancellations a month, not 95. At those rates, no segment is churning above normal. Finance needs to reconcile the two before anyone quotes an "increase." Until then, treat 250 as an upper bound.

Where Crestline can hurt us

Solo accounts (11,400 accounts, 65% of the base, $14 average, 9% connected to a firm). - Free beats $14, and this segment already churned at 31% a year. - 142 solo accounts ($1,960 MRR) went to Crestline. - We should expect more losses here, but the whole segment is only about $160k MRR.

Unconnected multi-user accounts, our largest dollar leak. - 42 accounts with 2-20 users and no firm connection went to Crestline, taking $3,320 MRR. That is 59% of the MRR we lost to Crestline, more than the solo losses. - Crestline's free plan is one user only, so these customers either took a paid plan (from $25, against our $52 and $170 averages) or cut down to one seat. The files can't tell us which, and the cancellations are labelled "free tier." We need exit calls this week. - A free plan wouldn't win them back, because they didn't leave for a free plan.

Where it can't (so far)

  • Connected accounts are barely leaving. They are about 25% of our base (roughly 4,300 accounts) but only 8 of 250 cancellations (3%).
  • Only 4 connected accounts went to Crestline ($427 MRR). The other 4 were businesses that closed.
  • Larger accounts show the same pattern. In the 6-20 segment, 81% of the base is connected, but only 4 of 14 cancellations were. The 10 that went to Crestline were nearly all unconnected; one was connected.
  • Caveats. This is six to eight weeks of data. Connected accounts may also simply be stickier customers, so the data shows correlation, not proof. Switching costs show up at renewal, so this is an early read.

Which advantages Crestline could copy

Easy to copy: - A free plan (they've done it) and feature parity. Their open roles are 14 engineers for payroll and 6 for mid-market multi-entity, 20 of 23 openings. Assume they match our core product and add payroll within a year. - Price. They have $120M to subsidise it.

Hard to copy: - Our firm network. 2,100 firms reconcile, file and advise from inside Harbourline. A firm standardises dozens of clients on one platform, so a switch means moving the firm's whole workflow, not one customer. Building that two-sided trust takes years. Crestline has no partner programme today, and "accountant partnerships" is three roles marked "exploring."

The window is limited. With $120M they can pay firms, buy a practice-management tool, or acquire a partner network. My judgement, not a data point, is that we have 12-18 months of lead. That is a window to use, not a permanent moat.

Where Crestline is heading

Crestline looks to be using the free solo plan as a funnel and monetising later through payroll and multi-entity, which means moving into our 2-5 and 6-20 accounts. Today it competes at the bottom of our base. Within a year it will compete in the middle, and the accountant-partnership hiring is the signal that it's going after our moat.

What we should do

  1. Don't match with a free plan for everyone. It would put about $476k of MRR at risk. It also wouldn't answer why multi-user accounts are leaving.
  1. Connect the unconnected. About 2,900 multi-user accounts have no firm connection (roughly 2,650 in 2-5 and 250 in 6-20), worth about $180k MRR on segment averages. Launch an in-product invite flow, and ask firms to onboard their clients. This is the best protection we have, and Crestline can't match it today.
  1. Test, don't commit, on solo. Run a 60-day test of a limited or lower-priced solo plan, limited to one user so larger accounts can't downgrade into it. Offer it to new sign-ups and as a save offer for at-risk solo accounts, with a route to a firm. The downside is capped at the solo segment. Success means solo churn back toward its 31% baseline and a rising share of solo accounts connected to a firm.
  1. Lock in firms in the next 90 days. Give firms better client-migration tools, firm-side incentives and a commitment from the largest ones. Move before Crestline's partnership hiring turns into a programme.

5. Fix the data and set tripwires. - Reconcile the churn baseline. - Run exit calls on the 42 unconnected multi-user losses. - Review the free-plan question if any of these happen: - connected-account cancellations to Crestline rise from about 2 a month to more than 10; - Crestline announces a firm programme or general availability of payroll; - multi-user unconnected churn keeps rising after the exit calls.

I'll report back at the next board meeting with the reconciled churn baseline and the exit-call findings.

Claims the judge couldn’t find in the brief

  • 42 accounts with 2-20 users and no firm connection went to Crestline, taking $3,320 MRR
  • In the 6-20 segment, 81% of the base is connected, but only 4 of 14 cancellations were

Grades and run details

Decision model 77 · LLM judge 75
Decision model checks
  • failUses the supplied evidence correctly30%
  • passAddresses the actual decision100%
  • passRespects explicit constraints51%
  • passIdentifies material uncertainty100%
  • partialAvoids unsupported claims38%
  • passProduces the required deliverable77%
  • passFinds who's actually leaving31%
  • passReads where Crestline is heading100%
  • passCompetes with what customers really choose69%
  • failGets the base of every number right56%
  • passSeparates what a rival can copy100%
Run
Run
#1
API response time
1.7 min
Submitted
6 Oct 2026

Needs more than a quick edit5 of 11 checks passed

The connected-accounts MRR is incorrect and the memo lacks explicit uncertainty triggers, so a PM would need to re-check the numbers and add conditions before using it.

Output

MEMORANDUM

TO: Board of Directors, Harbourline FROM: VP Product DATE: October 6 SUBJECT: Competitive Analysis & Strategic Response: Crestline’s Free Plan and $120M Raise

---

1. Executive Summary & Recommendation

The Board Chair asked whether Crestline’s new free tier represents an existential threat and whether Harbourline should match it with a free tier of our own.

The short answer is no. Launching a free plan for everyone—or even matching Crestline’s solo free tier—would be a severe strategic mistake. It would unnecessarily cannibalize our $635K monthly recurring revenue (MRR) to defend a low-margin, high-churn segment, while playing directly into a $120M VC-subsidized price war on our competitor’s terms.

The empirical data from August and September shows that Crestline is not an existential threat to Harbourline’s core business. While Crestline is peeling away unattached, price-sensitive solo accounts, our 2,100 accounting firm partner network is an impenetrable moat.

Our strategy should not be defensive price slashing, but aggressive doubling down on our accounting network to lock in high-LTV customers and convert unattached accounts into connected, un-churnable relationships.

---

2. Empirical Evidence: Cancellation Analysis (Aug–Sep)

In the two months following Crestline’s August 15 launch, Harbourline recorded 250 cancellations (~$8,100 MRR lost), up from our baseline run-rate of ~95 cancellations per month (~190 over two months).

``` Cancellation Breakdown by Outcome (250 accounts, Aug–Sep): ├── Crestline (free tier): 186 accounts (74.4%) | ~$6,300 MRR ├── Spreadsheets / no tool: 38 accounts (15.2%) | ~$550 MRR └── Business closed: 26 accounts (10.4%) | ~$1,250 MRR ```

A granular analysis of the 186 accounts lost to Crestline reveals clear structural patterns:

1. Firm Connection Renders Accounts Virtually Immune: * 97.8% (182 of 186) of accounts lost to Crestline were not connected to an accounting firm. * Only 4 connected accounts across our entire 17,600-customer base left for Crestline (two 2–5 user accounts and one 6–20 user account). * Zero connected 1-user accounts left for Crestline. * Of the 8 total connected cancellations across the entire dataset, 4 were due to business closures, not competitive loss.

2. The Damage is Concentrated in Solo, Unconnected Accounts: * 76.3% (142 of 186) of Crestline defectors were solo (1-user), unconnected accounts paying an average of $14/month. * These accounts already had an annual baseline churn of 31%. Crestline merely accelerated a transition that was already a structural vulnerability. * While 34 small teams (2–5 users) and 10 mid-sized teams (6–20 users) churned to Crestline, almost all (40 of 44) were unattached to an accounting firm.

---

3. Where Crestline Can and Cannot Hurt Us

DimensionWhere Crestline Can Hurt UsWhere Crestline Cannot Hurt Us
:---:---:---
Customer SegmentUnconnected 1-user businesses (10,374 accounts, ~$145K MRR). Highly price-sensitive, low switching costs, and view bookkeeping purely as administrative overhead.Connected accounts across all tiers (4,333 accounts generating ~$325K MRR). Switching software requires firing or disrupting their accounting firm.
Customer AcquisitionOrganic top-of-funnel for solo businesses. Prospective solo founders searching for free invoicing/expenses will default to Crestline.Firm-referred acquisition. Clients directed to software by our 2,100 partner firms do not shop on price.
Revenue ImpactSolo ARPU expansion. It will be harder to raise prices on unattached solo tiers without driving them to Crestline.Core MRR ($475K+ in multi-user tiers). Teams of 2–20 users require collaboration, permissioning, and reconciliation workflows that a free solo tier cannot satisfy.

---

4. Strategic Moats: What Crestline Can Copy vs. What It Cannot

Crestline’s $120M war chest allows them to buy features and subsidize customer acquisition, but capital cannot instantly replicate distribution architecture.

  • What Crestline CAN copy:
  • Core Product Features: Invoicing, expenses, and bank feeds are commodities. Their engineering hiring confirms they will match basic workflows rapidly.
  • Pricing Undercutting: They can burn VC cash on 1-user infrastructure indefinitely to starve competitors of top-of-funnel volume.
  • What Crestline CANNOT easily copy:
  • Our 2,100 Accounting Firm Network: Accounting firms exhibit extreme platform inertia. An accounting practice standardizes its entire staff training, audit trails, and reconciliation workflows on a single system. They do not switch software because a solo client wants a free tool; rather, they mandate what software the client must use.
  • Embedded Practice Workflows: Harbourline allows accountants to reconcile, file taxes, and deliver advisory services directly inside the ledger. Crestline currently has no partner program and only 3 exploratory job postings in this domain. Building trusted practice tools and bilateral distribution takes 3–5 years of dedicated field sales and product integration.

---

5. Crestline’s Trajectory & Endgame

Crestline is executing a textbook freemium land-and-expand strategy: 1. The Bait (Present): Offer a "free forever" solo plan to capture massive top-of-funnel market share and depress competitor unit economics. 2. The Monetization (Near-Term): Crestline's September hiring pipeline (14 payroll engineers) indicates they plan to monetize this "free" solo cohort via high-margin ancillary services (payroll, payment processing, lending) and force upgrades when solo companies hire their first employee ($25+/month plans). 3. Upmarket Migration (Medium-Term): With 6 engineers hired for "mid-market multi-entity," Crestline intends to move directly into higher-ACV accounts (our 6–20 user base) where margins are superior.

Their exploratory hiring for accountant partnerships (3 roles) shows they realize freemium reach alone cannot win sustainable SMB accounting without firm distribution. However, they are years behind us in this arena.

---

6. What Harbourline Should Do: Recommended Strategy

Matching Crestline with a "free for everyone" plan would instantly incinerate up to $635K in monthly revenue to solve an $8K/month churn problem. Even offering a free tier for 1-user accounts risks immediately wiping out $160K/month ($1.9M ARR).

Instead, we must weaponize our network moat:

#### A. Do NOT Launch a Free Plan Maintain our $14/month entry pricing. Competing on "free" plays into Crestline’s balance sheet strength ($120M) and devalues our product. Solo customers seeking zero-cost tools generate negative margin and churn at 31%. Let Crestline absorb the support costs and hosting overhead of unprofitable solo users.

#### B. Launch the "Firm Match" Concierge (Turn Vulnerable Accounts into Moats) The single most decisive finding is that connected accounts do not churn to Crestline. * We have 10,374 unattached 1-user accounts and 2,646 unattached 2–5 user accounts. * We should launch an in-app matching service connecting unattached SMBs with certified Harbourline accounting firms, potentially offering a 3-month platform discount upon pairing. * Connecting an account instantly reduces churn from 31% to baseline firm-connected rates (<6–10%), completely neutralizing Crestline’s pitch.

#### C. Incentivize Firm-Led Client Acquisition Double down on our 2,100 accounting partners: * Provide firms with free client "starter seats" or wholesale client bundle pricing that firms can include in their monthly retainers. * When the accounting firm provisions Harbourline for the client, price sensitivity drops to zero.

#### D. Preempt Crestline Upmarket Crestline is building payroll and multi-entity capabilities to attack our lucrative 6–20 user segment ($170 ARPU, 6% churn). We must accelerate our own payroll integrations and advanced reporting modules within the accounting portal before Crestline ships their mid-market product.

Summary

Crestline’s free tier is an expensive customer acquisition test, not an existential threat to Harbourline. By cementing our position as the operating system for accounting firms and their clients, we protect our revenue, preserve our margins, and let Crestline exhaust its capital chasing low-value solo churn.

Claims the judge couldn’t find in the brief

  • Connected accounts across all tiers: 4,333 accounts generating ~$325K MRR.

Grades and run details

Decision model 64 · LLM judge 42
Decision model checks
  • failUses the supplied evidence correctly70%
  • partialAddresses the actual decision28%
  • passRespects explicit constraints31%
  • partialIdentifies material uncertainty38%
  • failAvoids unsupported claims49%
  • passProduces the required deliverable76%
  • passFinds who's actually leaving88%
  • passReads where Crestline is heading100%
  • passCompetes with what customers really choose73%
  • failGets the base of every number right52%
  • passSeparates what a rival can copy100%
Run
Run
#1
API response time
50 s
Submitted
6 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 6

Uses the supplied evidence correctlyMixedWrongWrong
GPT-6 Astra · ChatGPT

All factual claims about the current situation are drawn directly from the supplied context or computed correctly from it; no invented facts.

Sonnet 5.5 · API

Two factual claims are wrong: the number of unconnected multi-user accounts going to Crestline is 40, not 42, and the number of connected 6-20 cancellations is 5, not 4.

Gemini 3.8 Flash · API

The connected-accounts MRR is stated as ~$325K, but the data gives ~$310.6K; this figure is not correctly derived from the supplied context.

Addresses the actual decisionRightRightWrong
GPT-6 Astra · ChatGPT

Commits early to 'Do not launch a free plan for everyone' and states that rising connected-customer losses or controlled-test results would change the call.

Sonnet 5.5 · API

Commits early to 'Don't launch a free plan for everyone' and sets tripwires (connected cancellations >10/month, Crestline firm programme or payroll GA, rising multi-user unconnected churn) that would change the call.

Gemini 3.8 Flash · API

The output does not state what result or condition would change the recommendation, as required for a clear decision.

Identifies material uncertaintyRightRightWrong
GPT-6 Astra · ChatGPT

Identifies the multi-user switcher inconsistency, the limits of hiring signals, and specifies the evidence (connected losses, test results) that would change the decision.

Sonnet 5.5 · API

Names the churn baseline discrepancy, the early data window, the need for exit calls, and sets specific tripwires that would trigger a review.

Gemini 3.8 Flash · API

The output does not name specific unknowns that could change the decision or how they would be resolved.

Avoids unsupported claimsRightRightWrong
GPT-6 Astra · ChatGPT

Forecasts and interpretations are labelled as such (e.g., 'likely strategy', 'directional inference'), and confident claims are backed by the data.

Sonnet 5.5 · API

Interpretations and forecasts are labelled as such (e.g., 'My judgement, not a data point,' 'looks to be'), and confident claims are grounded in the evidence.

Gemini 3.8 Flash · API

Forecasts like 'textbook freemium land-and-expand' and 'impenetrable moat' are presented as fact without being labelled as interpretations.

Competes with what customers really chooseRightRightMixed
GPT-6 Astra · ChatGPT

Treats spreadsheets/no tool as a competitor, notes the status quo, and draws a direction-of-travel insight from Crestline's investment signals.

Sonnet 5.5 · API

Treats spreadsheets/no tool as a competitor and draws a direction-of-travel insight from Crestline's investment in payroll and multi-entity.

Gemini 3.8 Flash · API

The output mentions spreadsheets as a destination but does not analyse the status quo as a competitor or draw implications from it.

Gets the base of every number rightMixedWrongWrong
GPT-6 Astra · ChatGPT

All percentages and differences use the correct denominators (e.g., unconnected/connected bases, total MRR) and the bases are stated where ambiguous.

Sonnet 5.5 · API

Derived figures for unconnected multi-user cancellations (42 vs actual 40) and connected 6-20 cancellations (4 vs actual 5) are incorrect.

Gemini 3.8 Flash · API

The connected-accounts MRR is miscalculated (~$325K vs. ~$310.6K), so a derived figure is wrong.

All got right 5

Respects explicit constraintsRightRightRight
GPT-6 Astra · ChatGPT

Delivers a board memo under 1,200 words, addresses the board chair's question, and proposes actions that would enforce the recommendation.

Sonnet 5.5 · API

The output is a board memo under 1,200 words, addresses the board chair's question, and does not propose a free plan for every account.

Gemini 3.8 Flash · API

The output is a memo under 1,200 words, addressed to the board, and does not propose a free plan for everyone.

Produces the required deliverableRightRightRight
GPT-6 Astra · ChatGPT

The memo is complete, in the right form for the board, within the word limit, and actionable with light edits.

Sonnet 5.5 · API

A complete competitive analysis memo for the board, within the word limit, with a clear recommendation and actionable next steps.

Gemini 3.8 Flash · API

The memo is complete, within word count, and usable by the board with light edits.

Finds who's actually leavingRightRightRight
GPT-6 Astra · ChatGPT

Counts cancellations by size and partner connection, sizes the MRR lost, and sets it against the base, showing losses are overwhelmingly unconnected single-user accounts.

Sonnet 5.5 · API

Correctly identifies that losses to Crestline are overwhelmingly solo, unconnected accounts, sizes the MRR, and shows connected accounts are barely leaving.

Gemini 3.8 Flash · API

The analysis clearly shows losses are almost all single-user, unconnected accounts, sizes the MRR, and sets it against the base.

Reads where Crestline is headingRightRightRight
GPT-6 Astra · ChatGPT

Uses Crestline's hiring (payroll, multi-entity, accountant partnerships) to infer direction, flags the partner-programme risk, and adjusts the response accordingly.

Sonnet 5.5 · API

Uses Crestline's hiring (payroll, mid-market multi-entity, accountant partnerships exploring) to forecast an upmarket move and a future threat to the partner channel.

Gemini 3.8 Flash · API

The output reads Crestline's hiring to flag payroll, mid-market, and accountant partnership moves, and adjusts the response accordingly.

Separates what a rival can copyRightRightRight
GPT-6 Astra · ChatGPT

Separates copyable advantages (features, price) from the durable partner network and embedded workflows, and anchors the response in the latter.

Sonnet 5.5 · API

Separates copyable advantages (free plan, price, features) from durable ones (firm network, switching costs) and anchors the response in the latter.

Gemini 3.8 Flash · API

The output separates copyable features and pricing from the durable accounting-firm network and builds the response on the latter.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Opus 5.5withClaude86.4100.02None
2GPT-6 AstrawithChatGPT88.695.82None
3GPT-6.1 SolwithAPI88.695.82None
4GPT-6 LunawithAPI93.283.32None
5Sonnet 5.5withAPI86.483.32None
6Haiku 5.5withAPI84.179.22None
7Gemini 3.8 FlashwithAPI68.254.22None
8Gemini 3.5 Flash-LitewithGemini61.425.02None

What you get for the cost

GPT-6.1 Sol · API scores within a point of Opus 5.5 · Claude at a fifth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreOpus 5.5 · Claude: task score 94, 75× the cheapestGPT-6 Astra · ChatGPT: task score 93, 87× the cheapestGPT-6.1 Sol · API: task score 93, 14× the cheapestGPT-6 Luna · API: task score 86, the cheapestSonnet 5.5 · API: task score 83, 44× the cheapestHaiku 5.5 · API: task score 80, 3.0× the cheapestGemini 3.8 Flash · API: task score 60, 12× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 41, 2.9× the cheapestOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTGPT-6.1 Sol · APIGPT-6 Luna · APISonnet 5.5 · APIHaiku 5.5 · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API861.0×
Gemini 3.5 Flash-Lite · Gemini412.9×
Haiku 5.5 · API803.0×
Gemini 3.8 Flash · API6012×
GPT-6.1 Sol · API9314×
Sonnet 5.5 · API8344×
Opus 5.5 · Claude9475×
GPT-6 Astra · ChatGPT9387×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Deciding how to respond to competitors.

Why it matters

Feature grids hide the real competition, which is often the customer's current workaround, and the response that feels urgent is usually to copy everything.

What good looks like

  • Includes the status quo and non-obvious alternatives
  • Finds where losses really go, by segment
  • Separates advantages a rival can copy from ones it can't
  • Reads where competitors are heading

Deliberately not measured

  • Market sizing
Capability tested

Competitive insight

The failure we’re looking for

A feature checklist and a plan to match everything

Grading

Decision model and LLM judge, calibrated against a blind PM review