Tasks / Strategy

Competitor analysis

Can the model find where a competitor can actually hurt you, from win/loss and churn data, rather than list features?

Task type v1.1 · 2 tasksLast changed 6 Oct 2026 · ChangelogDifficulty

What AI gets right here, and what you’ll still have to catch

From 16 graded outputs by 8 models. 63% were usable with at most a quick edit.

Reliably right

  1. Respects explicit constraints100% pass
    Delivers a competitive analysis memo to Dan within 900 words, addressing all requested topics without proposing a full feature-by-feature match.
    Opus 5.5 · Claude · The competitor isn't who you think
  2. Reads where Crestline is heading100% pass
    Uses Crestline's hiring (payroll, multi-entity, accountant partnerships) to infer direction, flags the partner-programme risk, and adjusts the response accordingly.
    GPT-6 Astra · ChatGPT · Free forever, funded
  3. Produces the required deliverable97% pass
    The memo is complete, in the right form for the CEO, within the word limit, and actionable with light edits.
    Opus 5.5 · Claude · The competitor isn't who you think

Where it slips

  1. Uses the supplied evidence correctly56% pass
    Two factual claims are wrong: the number of unconnected multi-user accounts going to Crestline is 40, not 42, and the number of connected 6-20 cancellations is 5, not 4.
    Sonnet 5.5 · API · Free forever, funded
  2. Gets the base of every number right61% pass
    Derived figures for unconnected multi-user cancellations (42 vs actual 40) and connected 6-20 cancellations (4 vs actual 5) are incorrect.
    Sonnet 5.5 · API · Free forever, funded
  3. Separates what a rival can copy66% pass
    Does not explicitly separate advantages a well-funded competitor could copy from those it couldn't, nor anchor the response in durable advantages with evidence.
    GPT-6 Luna · API · The competitor isn't who you think

How it’s graded

Each check is built on how world-class product and growth leaders do this work, as they’ve described it on Lenny’s Podcast, and names the leader it comes from.

  1. Competes with what customers really choose

    Does the analysis treat the status quo (spreadsheets, manual work, doing nothing) as a competitor, and draw at least one implication from where competitors are investing rather than only from today's features?

    Passes when Analyses the status quo, strengths included, and gives a direction-of-travel insight that affects the recommendation.

  2. Separates what a rival can copy

    Does the analysis separate advantages a well-funded competitor could copy (features, price, being first) from ones it couldn't easily (switching costs, networks, relationships), and build the response on the latter?

    Passes when Names which advantages are copyable and which aren't, with evidence, and anchors the response in the durable ones.

Plus our standard checks

Uses the supplied evidence correctly · Addresses the actual decision · Respects explicit constraints · Identifies material uncertainty · Avoids unsupported claims · Produces the required deliverable · Gets the base of every number right · and 2 written for each task, which you’ll see in the tasks below.

The tasks

Read the brief, then compare up to three outputs, each with the LLM judge’s verdict on every check. Highlights mark what a PM had to fix.

The brief

You're the VP Product at Harbourline. Crestline raised $120M in August and launched a free plan for solo businesses. Our board chair wants to know whether we should launch a free plan for everyone to match it (the note is below). Write the board a competitive analysis of no more than 1,200 words: where Crestline can hurt us and where it can't, with the evidence; which of our advantages it could copy and which it couldn't; where it's heading; and what we should do. The files are attached.

What the model was given6 items: About Harbourline, board_chair_note.md, customer_base.md, cancellations_aug_sep.csv (every account that cancelled since Crestline's launch), cancellation_summary.md, crestline_public_info.md
About HarbourlineBookkeeping, invoicing and expenses for small businesses, with a partner network that connects customers to their accounting firm.
board_chair_note.md1 lines · Download
From the board chair: "Crestline's free tier is an existential threat. Should we launch a free plan for everyone to match it? We need a view on where they can hurt us and where they can't."
customer_base.md9 lines · Download
# Harbourline customers (end of September)

| Account size | Paying accounts | Share connected to an accounting firm | Average revenue per month | Annual churn before Crestline's free tier |
|---|---|---|---|---|
| 1 user | 11,400 | 9% | $14 | 31% |
| 2-5 users | 4,900 | 46% | $52 | 14% |
| 6-20 users | 1,300 | 81% | $170 | 6% |
…
cancellations_aug_sep.csv (every account that cancelled since Crestline's launch)cancelled_account,account_users,connected_to_accounting_firm,mrr_usd,went_to H-1200,1,no,14,Crestline (free tier) H-1201,1,no,10,Crestline (free tier) H-1202,2-5,no,58,Crestline (free tier) H-1203,1,no,11,Crestline (free tier) H-1204,1,no,17,Crestline (free tier) H-1205,1,no,15,Spreadsheets / no tool H-1206,1,no,15,Spreadsheets / no tool H-1207,1,no,9,Crestline (free tier) H-1208,2-5,no,66,Crestline (free tier) H-1209,2-5,no,44,Crestline (free tier) H-1210,1,no,11,Crestline (free tier) H-1211,1,no,9,Crestline (free tier) H-1212,1,no,12,Crestline (free tier) H-1213,1,no,16,Crestline (free tier) H-1214,2-5,yes,73,Crestline (free tier) H-1215,1,no,9,Crestline (free tier) H-1216,2-5,no,56,Crestline (free tier) H-1217,1,no,14,Crestline (free tier) H-1218,2-5,no,47,Business closed H-1219,1,no,16,Crestline (free tier) H-1220,6-20,no,177,Crestline (free tier) H-1221,2-5,yes,51,Crestline (free tier) H-1222,2-5,no,54,Business closed H-1223,6-20,no,168,Crestline (free tier) H-1224,1,no,18,Crestline (free tier) H-1225,1,no,13,Spreadsheets / no tool H-1226,1,no,11,Crestline (free tier) H-1227,1,no,15,Crestline (free tier) H-1228,1,no,12,Crestline (free tier) H-1229,1,no,13,Crestline (free tier) H-1230,2-5,no,51,Business closed H-1231,1,no,18,Crestline (free tier) H-1232,1,no,16,Spreadsheets / no tool H-1233,1,no,12,Crestline (free tier) H-1234,1,no,16,Spreadsheets / no tool H-1235,1,no,12,Spreadsheets / no tool H-1236,1,no,13,Crestline (free tier) H-1237,1,no,17,Spreadsheets / no tool H-1238,6-20,yes,188,Business closed H-1239,1,no,17,Crestline (free tier) H-1240,1,no,15,Crestline (free tier) H-1241,6-20,yes,238,Crestline (free tier) H-1242,1,no,11,Spreadsheets / no tool H-1243,2-5,no,66,Crestline (free tier) H-1244,2-5,no,63,Business closed H-1245,6-20,no,199,Crestline (free tier) H-1246,1,no,18,Crestline (free tier) H-1247,1,no,11,Crestline (free tier) H-1248,1,no,15,Crestline (free tier) H-1249,2-5,no,60,Crestline (free tier) H-1250,2-5,no,66,Crestline (free tier) H-1251,2-5,no,34,Crestline (free tier) H-1252,1,no,12,Crestline (free tier) H-1253,1,no,18,Crestline (free tier) H-1254,1,no,18,Spreadsheets / no tool H-1255,2-5,no,31,Crestline (free tier) H-1256,1,no,17,Crestline (free tier) H-1257,1,no,16,Crestline (free tier) H-1258,1,no,18,Spreadsheets / no tool H-1259,1,no,9,Crestline (free tier) H-1260,1,no,15,Crestline (free tier) H-1261,1,no,18,Crestline (free tier) H-1262,6-20,yes,170,Business closed H-1263,1,no,18,Crestline (free tier) H-1264,2-5,no,41,Business closed H-1265,1,no,14,Spreadsheets / no tool H-1266,1,no,14,Crestline (free tier) H-1267,1,no,12,Crestline (free tier) H-1268,6-20,no,112,Crestline (free tier) H-1269,1,no,9,Crestline (free tier) H-1270,2-5,no,71,Business closed H-1271,1,no,12,Crestline (free tier) H-1272,1,no,11,Crestline (free tier) H-1273,1,no,15,Crestline (free tier) H-1274,1,no,9,Crestline (free tier) H-1275,6-20,no,209,Crestline (free tier) H-1276,2-5,no,36,Crestline (free tier) H-1277,1,no,13,Crestline (free tier) H-1278,1,no,11,Crestline (free tier) H-1279,1,no,15,Crestline (free tier) H-1280,1,no,18,Spreadsheets / no tool H-1281,1,no,16,Crestline (free tier) H-1282,1,no,17,Crestline (free tier) H-1283,1,no,9,Crestline (free tier) H-1284,1,no,10,Crestline (free tier) H-1285,6-20,yes,128,Business closed H-1286,1,no,17,Crestline (free tier) H-1287,1,no,12,Spreadsheets / no tool H-1288,2-5,no,39,Crestline (free tier) H-1289,1,no,14,Crestline (free tier) H-1290,1,no,12,Crestline (free tier) H-1291,1,no,16,Crestline (free tier) H-1292,2-5,no,62,Crestline (free tier) H-1293,1,no,12,Crestline (free tier) H-1294,1,no,12,Crestline (free tier) H-1295,1,no,9,Spreadsheets / no tool H-1296,1,no,11,Crestline (free tier) H-1297,1,no,16,Crestline (free tier) H-1298,1,no,17,Crestline (free tier) H-1299,1,no,13,Spreadsheets / no tool H-1300,1,no,13,Crestline (free tier) H-1301,1,no,17,Crestline (free tier) H-1302,1,no,12,Crestline (free tier) H-1303,1,no,17,Crestline (free tier) H-1304,1,no,14,Crestline (free tier) H-1305,1,no,9,Crestline (free tier) H-1306,1,no,15,Crestline (free tier) H-1307,1,no,16,Crestline (free tier) H-1308,1,no,16,Crestline (free tier) H-1309,1,no,12,Crestline (free tier) H-1310,2-5,no,47,Business closed H-1311,2-5,no,78,Crestline (free tier) H-1312,1,no,11,Crestline (free tier) H-1313,1,no,14,Crestline (free tier) H-1314,1,no,15,Spreadsheets / no tool H-1315,1,no,14,Spreadsheets / no tool H-1316,1,no,9,Crestline (free tier) H-1317,2-5,no,52,Business closed H-1318,1,no,13,Crestline (free tier) H-1319,2-5,no,43,Business closed H-1320,1,no,11,Spreadsheets / no tool H-1321,1,no,15,Crestline (free tier) H-1322,2-5,no,55,Business closed H-1323,1,no,12,Spreadsheets / no tool H-1324,6-20,no,166,Crestline (free tier) H-1325,2-5,no,67,Business closed H-1326,1,no,16,Crestline (free tier) H-1327,1,no,15,Crestline (free tier) H-1328,1,no,13,Crestline (free tier) H-1329,1,no,17,Crestline (free tier) H-1330,1,no,12,Crestline (free tier) H-1331,1,no,13,Crestline (free tier) H-1332,2-5,no,68,Crestline (free tier) H-1333,1,no,14,Spreadsheets / no tool H-1334,1,no,18,Crestline (free tier) H-1335,1,no,9,Crestline (free tier) H-1336,1,no,14,Crestline (free tier) H-1337,1,no,13,Crestline (free tier) H-1338,1,no,10,Crestline (free tier) H-1339,1,no,18,Spreadsheets / no tool H-1340,2-5,no,70,Crestline (free tier) H-1341,1,no,13,Spreadsheets / no tool H-1342,2-5,no,34,Crestline (free tier) H-1343,1,no,11,Crestline (free tier) H-1344,1,no,17,Spreadsheets / no tool H-1345,6-20,no,159,Crestline (free tier) H-1346,6-20,no,214,Crestline (free tier) H-1347,2-5,no,67,Crestline (free tier) H-1348,2-5,no,40,Crestline (free tier) H-1349,1,no,18,Crestline (free tier) H-1350,1,no,17,Crestline (free tier) H-1351,1,no,11,Crestline (free tier) H-1352,1,no,17,Crestline (free tier) H-1353,2-5,no,48,Business closed H-1354,1,no,12,Crestline (free tier) H-1355,1,no,16,Crestline (free tier) H-1356,1,no,14,Crestline (free tier) H-1357,1,no,18,Crestline (free tier) H-1358,1,no,11,Crestline (free tier) H-1359,1,no,17,Spreadsheets / no tool H-1360,1,no,12,Spreadsheets / no tool H-1361,2-5,no,39,Business closed H-1362,1,no,18,Crestline (free tier) H-1363,2-5,no,39,Crestline (free tier) H-1364,2-5,no,41,Business closed H-1365,2-5,no,51,Crestline (free tier) H-1366,1,no,11,Crestline (free tier) H-1367,2-5,no,76,Business closed H-1368,1,no,11,Crestline (free tier) H-1369,1,no,18,Crestline (free tier) H-1370,2-5,no,37,Business closed H-1371,1,no,9,Crestline (free tier) H-1372,1,no,18,Crestline (free tier) H-1373,2-5,no,36,Crestline (free tier) H-1374,2-5,no,77,Crestline (free tier) H-1375,2-5,no,64,Crestline (free tier) H-1376,1,no,9,Crestline (free tier) H-1377,2-5,no,71,Crestline (free tier) H-1378,1,no,10,Crestline (free tier) H-1379,1,no,12,Spreadsheets / no tool H-1380,6-20,yes,235,Business closed H-1381,1,no,9,Crestline (free tier) H-1382,1,no,15,Spreadsheets / no tool H-1383,2-5,yes,65,Crestline (free tier) H-1384,1,no,16,Crestline (free tier) H-1385,1,no,17,Spreadsheets / no tool H-1386,1,no,13,Spreadsheets / no tool H-1387,2-5,no,66,Business closed H-1388,1,no,17,Crestline (free tier) H-1389,1,no,18,Crestline (free tier) H-1390,1,no,16,Crestline (free tier) H-1391,1,no,18,Spreadsheets / no tool H-1392,1,no,15,Crestline (free tier) H-1393,1,no,10,Crestline (free tier) H-1394,1,no,11,Crestline (free tier) H-1395,2-5,no,46,Business closed H-1396,1,no,16,Crestline (free tier) H-1397,1,no,15,Crestline (free tier) H-1398,2-5,no,34,Crestline (free tier) H-1399,1,no,13,Crestline (free tier) H-1400,1,no,13,Crestline (free tier) H-1401,1,no,10,Crestline (free tier) H-1402,1,no,11,Crestline (free tier) H-1403,2-5,no,49,Crestline (free tier) H-1404,2-5,no,58,Crestline (free tier) H-1405,1,no,17,Crestline (free tier) H-1406,1,no,17,Crestline (free tier) H-1407,1,no,12,Crestline (free tier) H-1408,2-5,no,37,Crestline (free tier) H-1409,1,no,15,Spreadsheets / no tool H-1410,2-5,no,31,Crestline (free tier) H-1411,1,no,13,Spreadsheets / no tool H-1412,1,no,15,Crestline (free tier) H-1413,1,no,13,Crestline (free tier) H-1414,1,no,15,Crestline (free tier) H-1415,1,no,18,Crestline (free tier) H-1416,1,no,14,Crestline (free tier) H-1417,1,no,11,Crestline (free tier) H-1418,6-20,no,131,Crestline (free tier) H-1419,1,no,17,Crestline (free tier) H-1420,1,no,18,Crestline (free tier) H-1421,1,no,16,Crestline (free tier) H-1422,1,no,11,Crestline (free tier) H-1423,2-5,no,78,Business closed H-1424,1,no,16,Spreadsheets / no tool H-1425,1,no,18,Spreadsheets / no tool H-1426,1,no,18,Crestline (free tier) H-1427,1,no,15,Crestline (free tier) H-1428,1,no,14,Spreadsheets / no tool H-1429,1,no,14,Spreadsheets / no tool H-1430,2-5,no,63,Business closed H-1431,1,no,18,Crestline (free tier) H-1432,1,no,15,Crestline (free tier) H-1433,1,no,17,Spreadsheets / no tool H-1434,1,no,14,Crestline (free tier) H-1435,1,no,14,Crestline (free tier) H-1436,1,no,12,Crestline (free tier) H-1437,1,no,11,Crestline (free tier) H-1438,1,no,13,Crestline (free tier) H-1439,1,no,13,Spreadsheets / no tool H-1440,2-5,no,54,Business closed H-1441,1,no,14,Crestline (free tier) H-1442,1,no,16,Crestline (free tier) H-1443,2-5,no,74,Crestline (free tier) H-1444,1,no,14,Crestline (free tier) H-1445,2-5,no,65,Crestline (free tier) H-1446,1,no,18,Crestline (free tier) H-1447,1,no,11,Crestline (free tier) H-1448,1,no,10,Crestline (free tier) H-1449,2-5,no,69,Business closed
cancellation_summary.md1 lines · Download
# Cancellations, August to September: 250 accounts, about $8,100 of MRR. Before Crestline's launch we lost about 95 accounts a month.
crestline_public_info.md6 lines · Download
# Crestline (news and public pages)

- 1 August: raised $120M. 15 August: launched a free plan for 1 user (invoicing, expenses, bank feeds).
- Paid plans from $25/month; no accounting-firm partner programme.
- Open roles (September): 14 engineers for payroll; 6 for "mid-market multi-entity"; 3 for "accountant partnerships (exploring)".
- Pricing page: free plan "free forever for solo businesses".
What a strong answer doesThe answer key the graders mark against

Shows from the cancellations that Crestline is taking single-user accounts not connected to an accounting firm: 186 of the 250 cancellations went to Crestline, 142 of them single-user accounts at $9 to $19 a month, and only 4 of the 186 were connected to an accounting firm; the $5,600 of MRR lost to Crestline is a small share of a base where 1-user accounts already churned at 31% a year. The partner network is the advantage Crestline can't easily copy: 81% of 6-20 user accounts and 46% of 2-5 user accounts are connected to a firm, and they barely leave; Crestline has no partner programme. Free and price are easy to copy, and so is matching features. Reads Crestline's hiring (payroll, mid-market multi-entity, accountant partnerships 'exploring') as heading upmarket and possibly at the partner channel, the real future threat. Recommends against a free plan for everyone: it would give away revenue from 11,400 single-user accounts paying about $14 a month (about $160,000 MRR) to defend against losses that are a fraction of that. Instead: let the least valuable solo accounts go or offer a narrow solo option tied to an accounting firm, deepen the partner network (bring firms' clients onto Harbourline, lock in the firm workflow), and get ahead on payroll and multi-entity where Crestline is heading. Sets the signals that would change the call (losses spreading to connected or multi-user accounts, or Crestline launching a partner programme).

Critical failures (cap the score at 40)
  • Recommends launching a free plan for every account to match Crestline
Task

Staff-level · v1.0

The outputs

8 setups answered this brief.

Usable with a quick edit9 of 11 checks passed

The analysis is evidence-based, actionable, and flags its own data caveats; a PM could use it with only minor wording tweaks.

Output

Crestline: Competitive Analysis for the Board

From: VP Product | Re: Board chair's question on a free plan

Summary

Crestline is not an existential threat today. In two months it took 186 accounts and about $5,600 of MRR, which is 0.9% of our roughly $635k MRR. Almost all of those accounts were the ones least tied to us: single users not connected to an accounting firm.

It has barely touched accounts that work with their accountant inside Harbourline. That network is the one advantage Crestline cannot quickly buy.

The real risk is where Crestline is heading. It is building payroll and multi-entity features, and it has started on accountant partnerships. It also already appears to be winning our unconnected larger accounts.

My recommendation is not to launch a free plan for everyone. It would put up to our entire MRR at risk of downgrade to counter a loss of under 1%. Instead, we should spend the next two quarters getting unconnected teams connected to firms, locking in our firm network, and closing the payroll gap.

What the cancellations show

These figures come from all 250 cancellations in August and September.

DestinationAccountsMRR lost
Crestline186$5,581
Business closed26$1,929
Spreadsheets / no tool38$555
Total250$8,065

Losses to Crestline by segment and firm connection

SegmentUnconnected baseLost to Crestline2-month rateConnected baseLost to Crestline
1 user~10,3701421.4%~1,0300
2–5 users~2,650311.2%~2,2503
6–20 users~25093.6%~1,0501
Total~13,2701821.4%~4,3304 (0.09%)

Where Crestline can hurt us

1. Unconnected solo businesses. This is the main battleground so far: 142 of the 186 accounts we lost to Crestline. These accounts pay $14 a month and already churned 31% a year before Crestline launched. We are losing them, but they were always our least durable revenue. Another 38 solo accounts left for spreadsheets, which shows that some of this segment simply doesn't value paid software.

2. Unconnected larger accounts. This is the warning sign. Nine of roughly 250 unconnected 6–20 user accounts went to Crestline in two months, about 3.6%. Annualised, that is above 20%, against a historical churn of 6% for the whole segment. These accounts pay around $170 a month.

Crestline's free plan covers only one user. So these teams either moved to Crestline's paid plans (from $25 a month) or cut down to one user. Either way, Crestline is competing on product and price for teams, not just on "free."

The numbers are small, but this pool is worth about $42k of MRR. Unconnected 2–5 user accounts are worth about $138k and are losing about 1.2% to Crestline every two months.

Where Crestline can't hurt us (yet)

Accounts connected to an accounting firm. Only 4 of about 4,330 connected accounts went to Crestline. That is roughly 0.09%, about 15 times lower than unconnected accounts.

Of the 8 connected accounts that cancelled at all, half did so because the business closed. Connected accounts make up about half our MRR (roughly $310k). Most of that is in the 2–5 and 6–20 user segments, where 46% and 81% of accounts are connected.

There is a caveat. Connection may partly reflect businesses that were already stickier, so the 15x gap is not proof that connecting an account causes retention. Still, Crestline has no partner programme. A firm that runs many clients inside Harbourline would have to move its whole practice to switch, which makes these accounts very costly to pull away.

Which advantages it could copy, and which it couldn't

Easily copied, or already matched: - Invoicing, expenses and bank feeds (already in Crestline's free plan) - Price - Payroll: 14 engineers are hiring for it, and we have no equivalent advantage - Multi-entity features for larger customers

Hard to copy: - Our 2,100-firm partner network - The fact that firms reconcile, file and advise inside Harbourline. That is a workflow habit and a two-sided network, not a feature.

Crestline has 3 open roles for accountant partnerships, labelled "exploring." With $120M it could offer firms rich incentives such as revenue share and free practice tools. But recruiting firms one at a time and changing how they work takes years, not quarters. Our advantage is durable, not permanent. Our window is now, while Crestline's effort is still exploratory.

Where Crestline is heading

Its September hiring is the clearest signal: - Payroll (14 engineers). This is its biggest bet. A bundled payroll offer would hit our 2–5 and 6–20 user segments directly. - Mid-market multi-entity (6 engineers). This moves it upmarket into the segment where we earn the most per account. - Accountant partnerships (3, exploring). This is the move against our network. It is early, but it is aimed at the right target.

The most likely pattern: free solo plans bring in new businesses cheaply, then Crestline moves those customers onto paid plans as they grow. It uses payroll and multi-entity features to win teams, and eventually goes after the firms.

Recommendations

1. Don't launch a free plan for everyone. Crestline doesn't offer one either, because its free plan is solo only. A free plan for everyone would put our ~$635k MRR at risk of downgrade to counter ~$5.6k lost over two months.

Even matching Crestline's solo-only free plan exposes about $160k of MRR (~$1.9M a year). If just 10% of solo accounts downgraded, we would lose more each month than Crestline has taken from us in total.

2. Use the firm network to defend unconnected teams. About 2,900 multi-user accounts are not connected to a firm. We should test a programme that introduces them to partner firms, for example a matching service, a free first review, or a short discount once connected.

We should measure whether connecting accounts actually reduces churn before rolling it out widely. If it does, this is the highest-return action available to us.

3. Lock in our firms before Crestline arrives. We should offer firms better tools for managing their practice, rewards for bringing clients onto Harbourline, and multi-year partner terms. Our 2,100 firms should have a reason to stay that goes beyond habit.

4. Close the payroll gap. Within a quarter, decide whether to build payroll or partner for it. If Crestline ships payroll first to our teams, our current advantage weakens.

5. Compete for solos through firms, not on price. One option is a low-cost or free solo plan available only when a partner firm sponsors or connects the client. This turns our advantage into a way to win new customers without devaluing our paid plans. We should accept some loss of unconnected solo accounts for now.

6. Set triggers to revisit this decision. Escalate to the board if any of the following happens: - Losses of connected accounts to Crestline exceed 0.5% per quarter - Losses of unconnected 6–20 user accounts stay above 3% per quarter - Crestline announces a formal accountant programme

Data issues to resolve

The baseline churn figures disagree. The summary says we lost about 95 accounts a month before Crestline launched. But the annual churn rates by segment imply roughly 350 a month. On the first figure, August–September is about 60 accounts above normal. On the second, it is below normal. Finance needs to reconcile these before we report a net Crestline impact.

The destination label looks wrong for multi-user accounts. The cancellation file records 44 multi-user accounts as going to "Crestline (free tier)," which only allows one user. We should confirm through exit interviews whether they went to paid plans or reduced to one user, because that changes how we respond.

The time window is mixed. The window includes the first half of August, before Crestline's free plan launched on 15 August, so the post-launch picture is slightly blended.

Conclusion

Crestline is winning the customers we hold most loosely, and it is investing in the areas where we are vulnerable. A free plan would trade our strongest economics for our weakest segment. The better response is to deepen the firm network, which Crestline cannot buy quickly, and to close the payroll gap before it arrives.

Grades and run details

Decision model 77 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly35%
  • passAddresses the actual decision100%
  • passRespects explicit constraints27%
  • passIdentifies material uncertainty99%
  • partialAvoids unsupported claims55%
  • passProduces the required deliverable80%
  • passFinds who's actually leaving96%
  • passReads where Crestline is heading100%
  • passCompetes with what customers really choose95%
  • failGets the base of every number right61%
  • passSeparates what a rival can copy100%
Run
Run
#1
Time to output
1.9 min
Submitted
5 Oct 2026

Needs more than a quick edit9 of 11 checks passed

The output contains minor but factual errors in the cancellation counts (unconnected multi-user and connected 6-20) that would need correction before use.

Output

Crestline: where it can hurt us, and what we should do

To: Board | From: VP Product

Recommendation

Don't launch a free plan for everyone. Crestline is taking our solo and unconnected customers, not our firm-connected ones. Accounts with 2-20 users make up 75% of our MRR (about $476k of $635k), and a free-for-all plan would put that revenue at risk to defend the 25% that is solo. Instead we should connect more customers to accounting firms, test a narrow solo response, and lock in firms before Crestline builds a partner programme.

What the cancellations show

Since launch, 250 accounts (about $8,100 MRR) have cancelled:

Went toAccountsMRR
Crestline186 (74%)~$5,580 (69%)
Business closed26 (10%)~$1,930 (24%)
Spreadsheets / no tool38 (15%)~$555 (7%)

The 250 overstates the damage. Normal churn is about 95 accounts a month, so the excess is roughly 60-110 accounts, depending on whether we count from 15 August or 1 August. There is also a data problem: the segment churn rates in our customer table imply about 360 cancellations a month, not 95. At those rates, no segment is churning above normal. Finance needs to reconcile the two before anyone quotes an "increase." Until then, treat 250 as an upper bound.

Where Crestline can hurt us

Solo accounts (11,400 accounts, 65% of the base, $14 average, 9% connected to a firm). - Free beats $14, and this segment already churned at 31% a year. - 142 solo accounts ($1,960 MRR) went to Crestline. - We should expect more losses here, but the whole segment is only about $160k MRR.

Unconnected multi-user accounts, our largest dollar leak. - 42 accounts with 2-20 users and no firm connection went to Crestline, taking $3,320 MRR. That is 59% of the MRR we lost to Crestline, more than the solo losses. - Crestline's free plan is one user only, so these customers either took a paid plan (from $25, against our $52 and $170 averages) or cut down to one seat. The files can't tell us which, and the cancellations are labelled "free tier." We need exit calls this week. - A free plan wouldn't win them back, because they didn't leave for a free plan.

Where it can't (so far)

  • Connected accounts are barely leaving. They are about 25% of our base (roughly 4,300 accounts) but only 8 of 250 cancellations (3%).
  • Only 4 connected accounts went to Crestline ($427 MRR). The other 4 were businesses that closed.
  • Larger accounts show the same pattern. In the 6-20 segment, 81% of the base is connected, but only 4 of 14 cancellations were. The 10 that went to Crestline were nearly all unconnected; one was connected.
  • Caveats. This is six to eight weeks of data. Connected accounts may also simply be stickier customers, so the data shows correlation, not proof. Switching costs show up at renewal, so this is an early read.

Which advantages Crestline could copy

Easy to copy: - A free plan (they've done it) and feature parity. Their open roles are 14 engineers for payroll and 6 for mid-market multi-entity, 20 of 23 openings. Assume they match our core product and add payroll within a year. - Price. They have $120M to subsidise it.

Hard to copy: - Our firm network. 2,100 firms reconcile, file and advise from inside Harbourline. A firm standardises dozens of clients on one platform, so a switch means moving the firm's whole workflow, not one customer. Building that two-sided trust takes years. Crestline has no partner programme today, and "accountant partnerships" is three roles marked "exploring."

The window is limited. With $120M they can pay firms, buy a practice-management tool, or acquire a partner network. My judgement, not a data point, is that we have 12-18 months of lead. That is a window to use, not a permanent moat.

Where Crestline is heading

Crestline looks to be using the free solo plan as a funnel and monetising later through payroll and multi-entity, which means moving into our 2-5 and 6-20 accounts. Today it competes at the bottom of our base. Within a year it will compete in the middle, and the accountant-partnership hiring is the signal that it's going after our moat.

What we should do

  1. Don't match with a free plan for everyone. It would put about $476k of MRR at risk. It also wouldn't answer why multi-user accounts are leaving.
  1. Connect the unconnected. About 2,900 multi-user accounts have no firm connection (roughly 2,650 in 2-5 and 250 in 6-20), worth about $180k MRR on segment averages. Launch an in-product invite flow, and ask firms to onboard their clients. This is the best protection we have, and Crestline can't match it today.
  1. Test, don't commit, on solo. Run a 60-day test of a limited or lower-priced solo plan, limited to one user so larger accounts can't downgrade into it. Offer it to new sign-ups and as a save offer for at-risk solo accounts, with a route to a firm. The downside is capped at the solo segment. Success means solo churn back toward its 31% baseline and a rising share of solo accounts connected to a firm.
  1. Lock in firms in the next 90 days. Give firms better client-migration tools, firm-side incentives and a commitment from the largest ones. Move before Crestline's partnership hiring turns into a programme.

5. Fix the data and set tripwires. - Reconcile the churn baseline. - Run exit calls on the 42 unconnected multi-user losses. - Review the free-plan question if any of these happen: - connected-account cancellations to Crestline rise from about 2 a month to more than 10; - Crestline announces a firm programme or general availability of payroll; - multi-user unconnected churn keeps rising after the exit calls.

I'll report back at the next board meeting with the reconciled churn baseline and the exit-call findings.

Claims the judge couldn’t find in the brief

  • 42 accounts with 2-20 users and no firm connection went to Crestline, taking $3,320 MRR
  • In the 6-20 segment, 81% of the base is connected, but only 4 of 14 cancellations were

Grades and run details

Decision model 77 · LLM judge 75
Decision model checks
  • failUses the supplied evidence correctly30%
  • passAddresses the actual decision100%
  • passRespects explicit constraints51%
  • passIdentifies material uncertainty100%
  • partialAvoids unsupported claims38%
  • passProduces the required deliverable77%
  • passFinds who's actually leaving31%
  • passReads where Crestline is heading100%
  • passCompetes with what customers really choose69%
  • failGets the base of every number right56%
  • passSeparates what a rival can copy100%
Run
Run
#1
API response time
1.7 min
Submitted
6 Oct 2026

Usable with a quick edit9 of 11 checks passed

The memo is well-structured, evidence-based, and directly actionable; a product manager could use it with only minor wording edits.

Output

Board memo: Crestline’s free tier and Harbourline’s response

Recommendation: do not launch a free plan for everyone. Crestline is a serious threat to businesses using Harbourline without an accounting firm—not yet an existential threat to our accountant-connected model. We should test a narrowly bounded solo free offer while investing in the partner workflows that make Harbourline difficult to replace.

1. Where Crestline can hurt us—and where it currently struggles

The threat is concentrated, but not confined to solo businesses

The cancellation file records 186 departures to Crestline out of 250 cancellations in August–September. Of those, 182 were not connected to an accounting firm.

Account sizeDepartures to CrestlineNot firm-connectedFirm-connected
------:---:---:
1 user1421420
2–5 users34313
6–20 users1091
Total1861824

Source: cancellationsaugsep.csv.

Solo customers are the obvious vulnerability: 91% lack a firm connection, average revenue is only $14/month, and annual churn was already 31% before Crestline launched. Free invoicing, expenses and bank feeds directly challenge their reason to pay us. Another 38 solo customers left for spreadsheets or no tool, reinforcing that this segment’s problem is perceived value, not just Crestline.

But account size alone is a poor defence. Among unconnected accounts, departures to Crestline represent approximately:

  • 1.37% of the end-September solo base;
  • 1.17% of the unconnected 2–5-user base;
  • 3.64% of the unconnected 6–20-user base.

These are directional comparisons using ending account counts, not precise cohort churn rates. Nevertheless, the largest unconnected segment warrants immediate attention.

The 44 multi-user departures cost $3,623 in MRR. That is substantial revenue despite being only 24% of Crestline-bound cancellations. We should not dismiss the threat as low-value solo churn.

Important qualification: Crestline publicly limits its free plan to one user. The destination labels do not establish that it supports these customers’ previous multi-user workflows for free. They may have consolidated access, accepted reduced functionality, or been misclassified. Interviewing these customers is a priority.

Accounting-firm connections are our strongest observed protection

Approximately 4,333 customers are firm-connected, versus 13,267 unconnected. Only four connected customers left for Crestline, compared with 182 unconnected: roughly 0.09% versus 1.37% of those respective bases.

This association is stronger than the apparent protection from company size. Connected customers have a firm reconciling, filing and advising inside Harbourline; switching potentially disrupts both the business and its accountant.

However, this is evidence of resistance, not immunity or proof of causation. Connected customers may differ in complexity and willingness to pay. Three connected 2–5-user accounts and one connected 6–20-user account did leave.

The evidence does not support “existential” today

Our current customer table implies approximately $635,400 MRR. Crestline-bound departures represent roughly $5,600 MRR, or 0.9% of that base.

Total cancellations rose from about 95 per month previously to 250 over August–September—about 60 above a simple two-month baseline. We must not attribute all 186 Crestline-bound departures to incremental churn: some customers would otherwise have left for another destination.

Nor should we extrapolate two months into an annual forecast. We lack acquisition, conversion and matched pre-launch cohort data. Crestline could be hurting new-customer wins more than the cancellation file reveals.

2. Which advantages can Crestline copy?

Relatively copyable: bookkeeping features, invoicing, expenses, bank feeds, pricing and much of the user experience. Crestline already offers several core functions free. Its $120M raise gives it resources to recruit, subsidise acquisition and sustain price pressure, though funding alone does not prove execution.

Copyable over time, but harder: accounting collaboration, reconciliation and filing workflows. These are software capabilities, not permanent barriers. We should improve them rather than assume they remain differentiated.

Not quickly reproducible through product development alone: our relationships with 2,100 accounting firms, their embedded working practices, customer trust and the coordination required to move shared books. Crestline would need to recruit firms, earn adoption and migrate real workflows—not merely ship an accountant dashboard.

That advantage is defensible, not uncopyable forever. Crestline is already advertising three accountant-partnership roles. Our network becomes more durable only if firms keep choosing and using it.

3. Where Crestline is heading

Its hiring signals suggest three directions:

  • Payroll: 14 engineering roles indicate a substantial adjacent-product investment. Payroll could increase usage, create additional revenue and deepen retention.
  • Larger businesses: six “mid-market multi-entity” engineering roles suggest ambitions beyond solo customers.
  • Accountant distribution: three partnership roles, explicitly “exploring,” indicate recognition of our channel’s value, but not an established programme.

The likely strategy is a free acquisition funnel feeding paid products and larger accounts; paid plans already start at $25/month. This is an inference, not a confirmed roadmap.

“Free forever for solo businesses” also makes a rapid reversal less likely. We should plan around persistent free competition. Longer term, capturing businesses at formation could deprive our partner network of future customers.

4. What we should do

Reject blanket free pricing

A universal free offer would expose revenue in segments where we currently see strong resistance, without reproducing our partner advantage. The 2–20-user segments generate 75% of current MRR. There is no evidence that making all their core functionality free is necessary or economically sustainable.

Run a controlled solo-free pilot

Test a genuinely useful, one-user entry plan for unconnected businesses, with clear limits and paid upgrades tied to additional users and higher-value workflows. Do not automatically move existing paying customers to free; measure voluntary downgrades and cannibalisation explicitly.

Use a holdout where feasible. Judge the pilot on incremental contribution, including acquisition gains, retained customers, paid conversion, lost subscription revenue and support costs—not registrations. Review after 90 days, while continuing to track longer-term conversion.

Defend unconnected multi-user customers now

Interview the 44 departed accounts and comparable retained customers. Determine whether the issue was price, unused seats, weak functionality or willingness to abandon collaboration. Test right-sized packaging and targeted retention offers before broad discounting.

Strengthen—and expand—firm connections

Prioritise shared reconciliation, filing handoffs, permissions and partner onboarding. Help firms bring more existing Harbourline customers into collaborative workflows. Track active firm usage and customer connection rates, not just partner registrations.

Board decision requested: approve the targeted experiment and partner-led defence, not universal free pricing. Review segment-level losses, acquisition win rates, contribution economics and Crestline’s partnership progress monthly. Reconsider a broader response if connected-customer losses materially accelerate or the controlled test demonstrates profitable expansion.

Grades and run details

Decision model 82 · LLM judge 100
Decision model checks
  • failUses the supplied evidence correctly15%
  • passAddresses the actual decision94%
  • passRespects explicit constraints37%
  • passIdentifies material uncertainty90%
  • passAvoids unsupported claims43%
  • passProduces the required deliverable75%
  • passFinds who's actually leaving91%
  • passReads where Crestline is heading100%
  • passCompetes with what customers really choose92%
  • failGets the base of every number right16%
  • passSeparates what a rival can copy100%
Run
Run
#1
API response time
87 s
Submitted
5 Oct 2026

Check by check

Where the outputs differ comes first. Open a check to read the judge’s reasons. “Mixed” means the two graders disagreed.

Where they differ 2

Uses the supplied evidence correctlyMixedWrongMixed
Opus 5.5 · Claude

All factual claims about the current situation are directly supported by the supplied context or arithmetic from it, with no invented facts.

Sonnet 5.5 · API

Two factual claims are wrong: the number of unconnected multi-user accounts going to Crestline is 40, not 42, and the number of connected 6-20 cancellations is 5, not 4.

GPT-6.1 Sol · API

All factual claims about the current situation are directly supported by the supplied context or derived by correct arithmetic.

Gets the base of every number rightMixedWrongMixed
Opus 5.5 · Claude

All derived percentages and MRR figures check out against the supplied data, with bases stated where needed (e.g., unconnected base sizes).

Sonnet 5.5 · API

Derived figures for unconnected multi-user cancellations (42 vs actual 40) and connected 6-20 cancellations (4 vs actual 5) are incorrect.

GPT-6.1 Sol · API

All percentages and rates are computed from the correct denominators (e.g., unconnected bases for departure rates), and ambiguous bases are clarified.

All got right 9

Addresses the actual decisionRightRightRight
Opus 5.5 · Claude

The output clearly recommends against a free plan for everyone and specifies triggers (loss thresholds, Crestline accountant programme) that would change the call.

Sonnet 5.5 · API

Commits early to 'Don't launch a free plan for everyone' and sets tripwires (connected cancellations >10/month, Crestline firm programme or payroll GA, rising multi-user unconnected churn) that would change the call.

GPT-6.1 Sol · API

The output commits early to 'do not launch a free plan for everyone' and states conditions (connected-customer losses accelerating, pilot results) that would change the call.

Respects explicit constraintsRightRightRight
Opus 5.5 · Claude

The output is a competitive analysis under 1,200 words, addressed to the board, and respects all stated constraints.

Sonnet 5.5 · API

The output is a board memo under 1,200 words, addresses the board chair's question, and does not propose a free plan for every account.

GPT-6.1 Sol · API

The output is a board memo under 1,200 words, addresses all requested topics, and proposes actions that respect the brief's scope.

Identifies material uncertaintyRightRightRight
Opus 5.5 · Claude

It names specific unknowns (baseline churn discrepancy, multi-user destination label, pre-launch blending) and says how to resolve them, plus sets escalation triggers.

Sonnet 5.5 · API

Names the churn baseline discrepancy, the early data window, the need for exit calls, and sets specific tripwires that would trigger a review.

GPT-6.1 Sol · API

It names specific unknowns (multi-user usage of free plan, acquisition impact, pilot outcomes) and says how to resolve them or what would change the recommendation.

Avoids unsupported claimsRightRightRight
Opus 5.5 · Claude

Interpretations and forecasts are labelled as such (e.g., 'appears to be winning', 'most likely pattern'), and confident claims are backed by evidence.

Sonnet 5.5 · API

Interpretations and forecasts are labelled as such (e.g., 'My judgement, not a data point,' 'looks to be'), and confident claims are grounded in the evidence.

GPT-6.1 Sol · API

Interpretations and forecasts are clearly labelled as inferences or possibilities, not presented as established fact.

Produces the required deliverableRightRightRight
Opus 5.5 · Claude

The memo is a complete competitive analysis with summary, evidence, and recommendations, ready for the board with at most light edits.

Sonnet 5.5 · API

A complete competitive analysis memo for the board, within the word limit, with a clear recommendation and actionable next steps.

GPT-6.1 Sol · API

The board memo is complete, within word limit, and provides a clear recommendation with supporting analysis that the board could act on.

Finds who's actually leavingRightRightRight
Opus 5.5 · Claude

It breaks down cancellations by segment and firm connection, shows 142 single-user unconnected lost to Crestline, sizes the MRR, and sets it against the base.

Sonnet 5.5 · API

Correctly identifies that losses to Crestline are overwhelmingly solo, unconnected accounts, sizes the MRR, and shows connected accounts are barely leaving.

GPT-6.1 Sol · API

The analysis counts cancellations by size and partner connection, sizes the MRR lost to Crestline, and sets it against the base.

Reads where Crestline is headingRightRightRight
Opus 5.5 · Claude

It reads Crestline's hiring (payroll, multi-entity, accountant partnerships) as direction, flags the partner-programme risk, and adjusts the response accordingly.

Sonnet 5.5 · API

Uses Crestline's hiring (payroll, mid-market multi-entity, accountant partnerships exploring) to forecast an upmarket move and a future threat to the partner channel.

GPT-6.1 Sol · API

It reads Crestline's job ads as signalling payroll, mid-market, and possible accountant partnerships, and adjusts the response accordingly.

Competes with what customers really chooseRightRightRight
Opus 5.5 · Claude

It treats spreadsheets/no-tool as a competitor, notes that some solos don't value paid software, and draws implications from Crestline's investment direction.

Sonnet 5.5 · API

Treats spreadsheets/no tool as a competitor and draws a direction-of-travel insight from Crestline's investment in payroll and multi-entity.

GPT-6.1 Sol · API

It treats spreadsheets/no tool as a competitor, notes the segment's perceived-value problem, and uses Crestline's hiring to infer direction of travel.

Separates what a rival can copyRightRightRight
Opus 5.5 · Claude

It separates easily copied advantages (features, price) from durable ones (partner network, workflow habits) and anchors the response in the durable ones.

Sonnet 5.5 · API

Separates copyable advantages (free plan, price, features) from durable ones (firm network, switching costs) and anchors the response in the latter.

GPT-6.1 Sol · API

It separates copyable advantages (features, pricing) from durable ones (partner network, relationships) and anchors the response in the latter.

Results

Every setup we’ve tested on this task type, across all its tasks and repeats, graded on the current checklist. Calibrated.

#Model · HarnessTask scoreDecision modelLLM judgeRunsCritical failures
1Opus 5.5withClaude86.4100.02None
2GPT-6 AstrawithChatGPT88.695.82None
3GPT-6.1 SolwithAPI88.695.82None
4GPT-6 LunawithAPI93.283.32None
5Sonnet 5.5withAPI86.483.32None
6Haiku 5.5withAPI84.179.22None
7Gemini 3.8 FlashwithAPI68.254.22None
8Gemini 3.5 Flash-LitewithGemini61.425.02None

What you get for the cost

GPT-6.1 Sol · API scores within a point of Opus 5.5 · Claude at a fifth of the cost.

Task score against relative cost for each setup. The table below has the same figures.304050607080901001×3×10×30×100×Cost per output, relative to the cheapest setup (log scale) →↑ Task scoreOpus 5.5 · Claude: task score 94, 75× the cheapestGPT-6 Astra · ChatGPT: task score 93, 87× the cheapestGPT-6.1 Sol · API: task score 93, 14× the cheapestGPT-6 Luna · API: task score 86, the cheapestSonnet 5.5 · API: task score 83, 44× the cheapestHaiku 5.5 · API: task score 80, 3.0× the cheapestGemini 3.8 Flash · API: task score 60, 12× the cheapestGemini 3.5 Flash-Lite · Gemini: task score 41, 2.9× the cheapestOpus 5.5 · ClaudeGPT-6 Astra · ChatGPTGPT-6.1 Sol · APIGPT-6 Luna · APISonnet 5.5 · APIHaiku 5.5 · APIGemini 3.8 Flash · APIGemini 3.5 Flash-Lite · Gemini
Best valueNobody beats these on both score and cost
Task score and relative cost by setup
SetupTask scoreRelative cost (cheapest = 1)
GPT-6 Luna · API861.0×
Gemini 3.5 Flash-Lite · Gemini412.9×
Haiku 5.5 · API803.0×
Gemini 3.8 Flash · API6012×
GPT-6.1 Sol · API9314×
Sonnet 5.5 · API8344×
Opus 5.5 · Claude9475×
GPT-6 Astra · ChatGPT9387×

Cost is each output’s tokens at the model’s list price, shown relative to the cheapest setup. Runs done in an app have no usage meter, so theirs are estimated from the same setup’s measured runs. How we compare cost

About the task

The PM job

Deciding how to respond to competitors.

Why it matters

Feature grids hide the real competition, which is often the customer's current workaround, and the response that feels urgent is usually to copy everything.

What good looks like

  • Includes the status quo and non-obvious alternatives
  • Finds where losses really go, by segment
  • Separates advantages a rival can copy from ones it can't
  • Reads where competitors are heading

Deliberately not measured

  • Market sizing
Capability tested

Competitive insight

The failure we’re looking for

A feature checklist and a plan to match everything

Grading

Decision model and LLM judge, calibrated against a blind PM review